# InternetLangIdent

Language Identification for internet conversations

## Dependencies
 - Python >= 3.6
 - fasttext (0.9.2)
 - pycountry (20.7.3)
 - unidecode (1.2.0)

## Setup
The following set of commands clones the repository and downloads all necessary models and datasets
```sh
$ git clone https://gitlab.fi.muni.cz/nlp/internetlangident
$ cd internetlangident
$ make all
```

## Usage

### Custom FastText wrapper
 - the module includes its own FastText module along with preprocessing pipeline and the ability to decide unknown language
 - the module is availible inside of `custom_fasttext_wrapper` directory

Example Usage:
```py
from language_identifier import LanguageIdentifier

model = LanguageIdentifier()

language, confidence = model.classify_single("ahoj")
print(language, confidence)
```

Output:
```py
sk [0.99905926]
```

### Dictionary Identifier
- the module includes word and bigram dictionaries inferred from large text corpora, where the results are aggregated using handwritten rules
- the module itself is located inside of `dict_lang_ident` directory

Example usage:
```py
from dict_lang_ident import DictLanguageIdentifier

model = DictLanguageIdentifier("dictionaries", "bigram_dicts")

language, confidence = model.predict("ahoj", debug=True)
print(language, confidence)
```

Example output:
```py
Word: ahoj
    Length modifier: 2.386294361119891x
        Occurence in other languages: [('csd', 966), ('cs', 993), ('skd', 1583), ('sk', 1693)]
        Occurence penalty: {'sk': 5.60926417417038, 'csd': 5.473283371731499, 'cs': 5.473283371731499, 'skd': 5.60926417417038}
        TOTAL SCORE: [('csd', 0.585733159600804), ('cs', 0.5807016804441587), ('skd', 0.48355245921269524), ('sk', 0.47158309910539414)]
[('csd', 0.585733159600804), ('cs', 0.5807016804441587), ('skd', 0.48355245921269524), ('sk', 0.47158309910539414), ('it', 0.0), ('fr', 0.0), ('nl', 0.0), ('da', 0.0), ('no', 0.0), ('jp', 0.0), ('pl', 0.0), ('sw', 0.0), ('fi', 0.0), ('hu', 0.0), ('en', 0.0), ('de', 0.0), ('ru', 0.0), ('zh', 0.0), ('es', 0.0), ('se', 0.0)]
('csd', 0.585733159600804)
```

### Ensemble
 - the module itself is located inside of `ensemble` directory
 - the original FastText model, custom FastText model, and the Dictionary identifier form and ensemble where the voting results are determined by a set of handcrafted rules

Example usage:

```py
from lang_ident_ensemble import LangIdentEnsemble

model = LangIdentEnsemble()

prediction, confidence = model.identify_language("hi there")
print(prediction, confidence)
```

Output:

```py
en array([1.25572915])
```

## Replicating the FastText model
- the repository also includes way to retrieve the preprocessed dataset used to train the custom FastText model (as a Makefile target) 

Model can be trained as follows:
```sh
$ make fasttext_train
```

## Evaluation Datasets
 - evaluation dataset from the OpenSubtitles corpus: https://nlp.fi.muni.cz/projekty/officebot/opensubs_lang_eval_data/
 - evaluation dataset from KMLE workspace: https://nlp.fi.muni.cz/projekty/officebot/kmle_lang_eval_set.txt
