# Language Detection

This is a comparison of language detection on the documents using the results of the OCR and modified LanguageFilter.

The repository contains following directory:
- [`reannotated_pages`]: Directory of vertical files. The files are annotated pages that have been adjusted to match up with current Ahisto OCR word split. Each file contains four attributeds: word recognized by OCR, language of the OCR, word used in annotations, language of the annotation.

The repository contains following files:

- [`annotated_dataset.vert`]: Vertical file containing the filtered dataset of annotated documents. The dataset was filtered due to discrepancies that arose from the combination of Google and Tesseract OCR results after the annotation dataset has been completed.

- [`languages_detected.vert`]: Vertical file containing results of the recognized languages. The content is divided into words, each word has eight attributes: word, Czech lemma, German lemma, Latin lemma, English lemma, correct language as annotated, language detected by OCR, language detected by LanguageFilter.

- [`languages_detected_filtered.vert`]: Vertical file containing results of the recognized languages with the following word types removed: numbers, words made up from punctuation, words not recognized during annotation. The structure is the same as in `languages_detected.vert`.

- [`mlang_filter.py`]: Python3 file containing the modified LanguageFilter.

- [`mlanguage_filter_output.vert`]: Vertical file containing the results of the modified LanguageFilter.

- [`corpora/*.frqwl.gz`]: Corpora files used in LanguageFilter to detected languages.

## Comparison of AHISTO-OCR Corpus Detected Languages (`ocr`) and Modified corpus.tools LanguageFilter (`corp`)
*word total: 43211*
| method | cs | de | la | en |
|--- |--- |--- |--- |--- |
| annotations | 13972 (32.33 %) | 26076 (60.35 %) | 3162 (7.32 %) | 1 (0.00 %) |
| ocr-correct | 12815 (29.66 %) | 23182 (53.65 %) | 2697 (6.24 %) | 0 (0.00 %) |
| ocr-incorrect | 172 (0.40 %) | 128 (0.30 %) | 524 (2.27 %) | 442 (1.02 %) |
| corp-correct | 10759 (24.90 %) | 18015 (41.69 %) | 1611 (3.73 %) | 1 (0.00 %) |
| corp-incorrect | 1397 (3.23 %) | 870 (2.01 %) | 1123 (2.60 %) | 3586 (8.30 %) |
| overlap | 10044 (23.24 %) | 16495 (38.17 %) | 1493 (3.46 %) | 0 (0.00 %) |
| ocr-adjusted | 2771 (6.41 %) | 6687 (15.48 %) | 1204 (2.79 %) | 0 (0.00 %) |
| corp-adjusted | 715 (0.00 %) | 1520 (3.52 %) | 118 (0.27 %) | 1 (0.00 %) |

### Ratio of Correctly Classified Languages
| method | cs | de | la | en |
|--- |--- |--- |--- |--- |
| ocr | 91.72 % | 88.90 % | 85.29 % | 0.00 % |
| corp | 77.00 % | 69.09 % | 50.95 % | 100.00 % |

### Ratio of Languages Other Than {cs, de, la, en} (including `unk|und`)
| | |
|--- |--- |
| ocr | 9.43 % |
| corp | 21.38 % |

Number of correctly classified words in `corp` not classified in `ocr`: 1845 (4.23 %)

## Replacement of `unk` Language in `ocr`
### Filtered Dataset
| method | cs | de | la | total |
|--- |--- |--- |--- |--- |
| `unk` replaced with majority lang of par - correct | 13714 (31.74 %) | 25351 (58.67 %) | 2802 (6.48 %) | 41867 (96.89 %) |
| `unk` replaced with majority lang of par - incorrect | 591 (1.37 %) | 170 (0.39 %) | 583 (1.35 %) | 1344 (3.11 %) |
| `unk` replaced with cs - correct | 13763 (31.85 %) | 23182 (53.65 %) | 2697 (6.24 %) | 39642 (91.74 %) |
| `unk` replaced with cs - incorrect | 2917 (6.75 %) | 128 (0.30 %) | 524 (1.21 %) | 3569 (8.26 %) |
| `unk` replaced with de - correct | 12816 (29.66 %) | 25699 (59.47 %) | 2697 (6.24 %) | 41211 (95.37 %) |
| `unk` replaced with de - incorrect | 172 (0.40 %) | 1304 (3.02 %) | 524 (1.21 %) | 2000 (4.63 %) |
| `unk` replaced with la - correct | 12815 (29.66 %) | 23182 (53.65 %) | 2925 (6.77 %) | 38922 (90.07 %) |
| `unk`replaced with la - incorrect | 172 (0.40 %) | 128 (0.30 %) | 3989 (9.23 %) | 4289 (9.93 %) |

### Unfiltered Dataset
| method | cs | de | la | total |
|--- |--- |--- |--- |--- |
| `unk` replaced with majority lang of par - correct | 14199 (30.34 %) |27519 (58.80 %)| 2861 (6.11 %) | 44579 (95.25 %) |
| `unk` replaced with majority lang of par - incorrect | 813 (1.74 %) | 687 (1.47 %) | 724 (1.55 %) | 2224 (4.75 %) |
| `unk` replaced with cs - correct | 14257 (30.46 %) | 24952 (53.31 %) | 2750 (5.88 %) | 41959 (89.65 %) |
| `unk` replaced with cs - incorrect | 3586 (7.66 %) | 607 (1.30 %) | 651 (1.39 %) | 4844 (10.35 %) |
| `unk` replaced with de - correct | 13241 (28.29 %) | 27992 (59.81 %) | 2750 (5.88 %) | 43983 (93.97 %) |
| `unk` replaced with de - incorrect | 232 (0.50 %) | 1937 (4.14 %) | 651 (1.39 %) | 2820 (6.03 %) |
| `unk` replaced with la - correct | 13241 (28.29 %) | 24952 (53.31 %) | 2998 (6.41 %) | 41191 (88.01 %) |
| `unk`replaced with la - incorrect | 232 (0.50 %) | 607 (1.30 %) | 4773 (10.20 %) | 5612 (11.99 %) |
