Commit cbfaa17b authored by Vít Novotný's avatar Vít Novotný
Browse files

Train/test split ARQMath task 1

parent 6cecc01d
Pipeline #57316 failed with stage
in 43 seconds
......@@ -13,7 +13,7 @@ execute the following commands:
``` sh
$ git add task/user/result.tsv # track your new result with Git
$ pip install -r requirements.txt # run the evaluation
$ python -m evaluate
$ python -m scripts.evaluate
$ git add -u # add the updated leaderboard to Git
$ git push # publish your new result and the updated leaderboard
```
......
......@@ -9,7 +9,7 @@ from pytrec_eval import RelevanceEvaluator, parse_qrel, parse_run
RELEVANCE_JUDGEMENTS = {
'task1': 'qrel.V0.1.tsv',
'task1': 'qrel.V0.1-test.tsv',
'ntcir-11-math-2-main': 'NTCIR11_Math-qrels-test.dat',
'ntcir-12-mathir-arxiv-main': 'NTCIR12_Math-qrels_agg-test.dat',
}
......
......@@ -2,7 +2,7 @@ This table contains the best result for every user.
| nDCG | User | Result name |
|:-----|------|:------------|
| 0.5181 | xstefan3 | example, key1=value1, key2=value2, etc |
| 0.5181 | xnovot32 | example, key1=value1, key2=value2, etc |
| 0.5181 | xluptak4 | example, key1=value1, key2=value2, etc |
| 0.5181 | ayetiran | example, key1=value1, key2=value2, etc |
| 0.5843 | xstefan3 | example, key1=value1, key2=value2, etc |
| 0.5843 | xnovot32 | example, key1=value1, key2=value2, etc |
| 0.5843 | xluptak4 | example, key1=value1, key2=value2, etc |
| 0.5843 | ayetiran | example, key1=value1, key2=value2, etc |
......@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5181 | example, key1=value1, key2=value2, etc |
| 0.5843 | example, key1=value1, key2=value2, etc |
3 0 493764 3
3 0 493782 3
3 0 1282155 0
3 0 1282180 1
3 0 1282116 1
3 0 1282112 1
3 0 1282166 3
3 0 1282114 0
3 0 2008628 2
3 0 2008712 3
3 0 2008449 1
3 0 2008650 1
3 0 2008609 2
3 0 2008616 1
3 0 2008631 2
3 0 1116378 2
3 0 1116368 1
3 0 1116370 1
......@@ -67,21 +67,3 @@
2 0 439044 1
2 0 439027 2
2 0 439055 1
3 0 493764 3
3 0 493782 3
3 0 1282155 0
3 0 1282180 1
3 0 1282116 1
3 0 1282112 1
3 0 1282166 3
3 0 1282114 0
3 0 2008628 2
3 0 2008712 3
3 0 2008449 1
3 0 2008650 1
3 0 2008609 2
3 0 2008616 1
3 0 2008631 2
3 0 1116378 2
3 0 1116368 1
3 0 1116370 1
......@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5181 | example, key1=value1, key2=value2, etc |
| 0.5843 | example, key1=value1, key2=value2, etc |
......@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5181 | example, key1=value1, key2=value2, etc |
| 0.5843 | example, key1=value1, key2=value2, etc |
......@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
| nDCG | Result name |
|------|:------------|
| 0.5181 | example, key1=value1, key2=value2, etc |
| 0.5843 | example, key1=value1, key2=value2, etc |
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment