Commit df0bc424 authored by Vít Novotný's avatar Vít Novotný
Browse files

Add support for NTCIR-12 MathIR MathWikiFormula

parent cbfaa17b
......@@ -5,6 +5,7 @@ on a number of *tasks*:
- `task1/`[ARQMath Task1][arqmath-task1] validation dataset,
- `ntcir-11-math-2-main/`[NTCIR-11 Math-2 Task Main Subtask][ntcir-11-math-2], and
- `ntcir-12-mathir-arxiv-main/`[NTCIR-12 MathIR Task ArXiv Main Subtask][ntcir-12-mathir].
- `ntcir-12-mathir-math-wiki-formula/`[NTCIR-12 MathIR Task MathWikiFormula Subtask][ntcir-12-mathir].
Place your results in [the trec\_eval format][treceval-format] into your
dedicated directory *task/user*. To evaluate and publish your results,
......
This table contains the best result for every user.
| nDCG | User | Result name |
|:-----|------|:------------|
| 0.5970 | xstefan3 | example, key1=value1, key2=value2, etc |
| 0.5970 | xnovot32 | example, key1=value1, key2=value2, etc |
| 0.5970 | xluptak4 | example, key1=value1, key2=value2, etc |
| 0.5970 | ayetiran | example, key1=value1, key2=value2, etc |
This table contains all results for user *ayetiran* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5970 | example, key1=value1, key2=value2, etc |
This table contains all results for user *xluptak4* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5970 | example, key1=value1, key2=value2, etc |
This table contains all results for user *xnovot32* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5970 | example, key1=value1, key2=value2, etc |
This table contains all results for user *xstefan3* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5970 | example, key1=value1, key2=value2, etc |
RELEVANCE_JUDGEMENTS = {
'task1': 'qrel.V0.1-test.tsv',
'ntcir-11-math-2-main': 'NTCIR11_Math-qrels-test.dat',
'ntcir-12-mathir-arxiv-main': 'NTCIR12_Math-qrels_agg-test.dat',
'ntcir-12-mathir-math-wiki-formula': 'NTCIR12_MathWikiFrm-qrels_agg-test.dat',
}
TASK_README_HEAD = r'''
This table contains the best result for every user.
| nDCG | User | Result name |
|:-----|------|:------------|
'''.strip()
USER_README_HEAD = r'''
This table contains all results for user *%s* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
'''.strip()
......@@ -7,26 +7,7 @@ import re
import numpy as np
from pytrec_eval import RelevanceEvaluator, parse_qrel, parse_run
RELEVANCE_JUDGEMENTS = {
'task1': 'qrel.V0.1-test.tsv',
'ntcir-11-math-2-main': 'NTCIR11_Math-qrels-test.dat',
'ntcir-12-mathir-arxiv-main': 'NTCIR12_Math-qrels_agg-test.dat',
}
TASK_README_HEAD = r'''
This table contains the best result for every user.
| nDCG | User | Result name |
|:-----|------|:------------|
'''.strip()
USER_README_HEAD = r'''
This table contains all results for user *%s* in descending order of task
performance. Result names are based on the filenames of the results with
underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
'''.strip()
from .configuration import RELEVANCE_JUDGEMENTS, TASK_README_HEAD, USER_README_HEAD
if __name__ == '__main__':
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment