Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
Michal Štefánik
ARQMath-eval
Commits
db2e2ef6
Commit
db2e2ef6
authored
Apr 23, 2020
by
Vít Novotný
Browse files
Extend the README
parent
21507985
Changes
1
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
db2e2ef6
...
...
@@ -10,15 +10,46 @@ on a number of *tasks*:
-
`ntcir-12-mathir-arxiv-main/`
–
[
NTCIR-12 MathIR Task ArXiv Main Subtask
][
ntcir-12-mathir
]
.
-
`ntcir-12-mathir-math-wiki-formula/`
–
[
NTCIR-12 MathIR Task MathWikiFormula Subtask
][
ntcir-12-mathir
]
.
Each task comes with a number of
*subsets*
:
-
`train`
– the training set, which you should use for parameter optimization
before publishing the results for the best parameters of your system,
-
`test`
– the test set, which you should use
*only for your best system*
after
parameter optimization on the training set,
-
`train-train`
– a subset of the training set for the
`task1-votes`
task,
which you can use for training if you also require a validation subset (e.g.
for early stopping), and
-
`train-validation`
– a subset of the training set for the
`task1-votes`
task,
which you can use for training if you also require a validation subset (e.g.
for early stopping).
### Usage
#### Evaluating zour model with various parameters
Place your results in
[
the trec\_eval format
][
treceval-format
]
into the
`results.csv`
file. To evaluate your results on the train set, execute the
following commands:
```
sh
$
pip
install
.
$
python
>>>
from arqmath_eval import get_ndcg
>>>
from pytrec_eval import parse_run
>>>
>>>
with open
(
'results.csv'
,
'rt'
)
as f:
>>>
results
=
parse_run
(
f
)
>>>
>>>
get_ndcg
(
task
=
'task'
,
subset
=
'train'
)
0.5876
```
#### Placing zour results to the leaderboard
Place your results in
[
the trec\_eval format
][
treceval-format
]
into your
dedicated directory
*task/user*
. To evaluate
and publish your results,
execute the following commands:
dedicated directory
*task/user*
. To evaluate
your results on the test set and
publish the results into the leaderboard,
execute the following commands:
```
sh
$
git add task/user/result.tsv
# track your new result with Git
$
pip
install
-e
.
# run the evaluation
$
pip
install
.
# run the evaluation
$
python
-m
scripts.evaluate
$
git add
-u
# add the updated leaderboard to Git
$
git push
# publish your new result and the updated leaderboard
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment