Train/test split ARQMath task 1

README.md

@@ -13,7 +13,7 @@ execute the following commands:
 ``` sh
 $ git add task/user/result.tsv     # track your new result with Git
 $ pip install -r requirements.txt  # run the evaluation
-$ python -m evaluate
+$ python -m scripts.evaluate
 $ git add -u                       # add the updated leaderboard to Git
 $ git push                         # publish your new result and the updated leaderboard
 ```

evaluate.py → scripts/evaluate.py

@@ -9,7 +9,7 @@ from pytrec_eval import RelevanceEvaluator, parse_qrel, parse_run
 
 
 RELEVANCE_JUDGEMENTS = {
-    'task1': 'qrel.V0.1.tsv',
+    'task1': 'qrel.V0.1-test.tsv',
     'ntcir-11-math-2-main': 'NTCIR11_Math-qrels-test.dat',
     'ntcir-12-mathir-arxiv-main': 'NTCIR12_Math-qrels_agg-test.dat',
 }

task1/README.md

@@ -2,7 +2,7 @@ This table contains the best result for every user.
 
 | nDCG | User | Result name |
 |:-----|------|:------------|
-| 0.5181 | xstefan3 | example, key1=value1, key2=value2, etc |
-| 0.5181 | xnovot32 | example, key1=value1, key2=value2, etc |
-| 0.5181 | xluptak4 | example, key1=value1, key2=value2, etc |
-| 0.5181 | ayetiran | example, key1=value1, key2=value2, etc |
+| 0.5843 | xstefan3 | example, key1=value1, key2=value2, etc |
+| 0.5843 | xnovot32 | example, key1=value1, key2=value2, etc |
+| 0.5843 | xluptak4 | example, key1=value1, key2=value2, etc |
+| 0.5843 | ayetiran | example, key1=value1, key2=value2, etc |

task1/ayetiran/README.md

@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
 
 | nDCG | Result name |
 |------|:------------|
-| 0.5181 | example, key1=value1, key2=value2, etc |
+| 0.5843 | example, key1=value1, key2=value2, etc |

task1/qrel.V0.1-test.tsv

+3	0	493764	3
+3	0	493782	3
+3	0	1282155	0
+3	0	1282180	1
+3	0	1282116	1
+3	0	1282112	1
+3	0	1282166	3
+3	0	1282114	0
+3	0	2008628	2
+3	0	2008712	3
+3	0	2008449	1
+3	0	2008650	1
+3	0	2008609	2
+3	0	2008616	1
+3	0	2008631	2
+3	0	1116378	2
+3	0	1116368	1
+3	0	1116370	1

task1/qrel.V0.1.tsv → task1/qrel.V0.1-train.tsv

@@ -67,21 +67,3 @@
 2	0	439044	1
 2	0	439027	2
 2	0	439055	1
-3	0	493764	3
-3	0	493782	3
-3	0	1282155	0
-3	0	1282180	1
-3	0	1282116	1
-3	0	1282112	1
-3	0	1282166	3
-3	0	1282114	0
-3	0	2008628	2
-3	0	2008712	3
-3	0	2008449	1
-3	0	2008650	1
-3	0	2008609	2
-3	0	2008616	1
-3	0	2008631	2
-3	0	1116378	2
-3	0	1116368	1
-3	0	1116370	1

task1/xluptak4/README.md

@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
 
 | nDCG | Result name |
 |------|:------------|
-| 0.5181 | example, key1=value1, key2=value2, etc |
+| 0.5843 | example, key1=value1, key2=value2, etc |

task1/xnovot32/README.md

@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
 
 | nDCG | Result name |
 |------|:------------|
-| 0.5181 | example, key1=value1, key2=value2, etc |
+| 0.5843 | example, key1=value1, key2=value2, etc |

task1/xstefan3/README.md

@@ -4,4 +4,4 @@ underscores (`_`) replaced with a comma and a space for improved readability.
 
 | nDCG | Result name |
 |------|:------------|
-| 0.5181 | example, key1=value1, key2=value2, etc |
+| 0.5843 | example, key1=value1, key2=value2, etc |