INJECT allows for export of logs from an exercise for analysis purposes.
The log files use the [JSONL](https://jsonlines.org/) format.
Bellow is description of the structure and schema of logs.

### Structure

```
- logs/
  - team-id/
    - uploaded_files/
        - files uploaded by team-id and for team-id during the exercise
    - inject_states.jsonl
    - questionnaire_states.jsonl
    - action_logs.jsonl
    - milestones.jsonl
    - emails.jsonl
  - team-id+1/
    ...
  - definition_files/
    - files used by the definition
  - exercise.jsonl
  - teams.jsonl
  - instructors.jsonl
  - file_infos.jsonl
  - exercise_tools.jsonl
  - exercise_questionnaires.jsonl
  - exercise_milestones.jsonl
  - exercise_learning_objectives.jsonl
  - exercise_injects.jsonl
  - exercise_channels.jsonl
  - email_participants.jsonl
```

## Description and format of individual files

Many of the fields described here do not contain a description.
That is because these fields are direct copies of the fields described in the definition format,
which can be found [here](./architecture/definitions/README.md).

### Timestamps

All fields with the `timestamp` type are timestamps in `ISO 8601` format.

### Common objects

#### Control

- **milestone_condition**: _string_
- **activate_milestone**: _list of strings_
- **deactivate_milestone**: _list of strings_
- **roles**: _list of strings_

#### Content

- **raw**: _string_ - raw text, possibly markdown
- **rendered**: _string_ - raw text converted to html elements
- **attachments**: _list of uuids_ - list of file info ids which are attached to this content

#### Confirmation

- **text**: _string_
- **control**: _control_

#### Overlay

- **duration**: _optional int_

## Exercise files

### exercise.jsonl

Contains information about the exercise.
The object has the following format:

- **exercise_id**: _int_ - id of the exercise
- **definition_id**: _int_ - id of the definition this exercise was created from
- **states**: _list of state objects_ - these objects represent the state of an exercise for the
    teams, one for normal exercises, multiple for on-demand exercises
    - **exercise_state_id**: _int_ - id of the exercise state
    - **status**: _(`Not Started`, `Stopped`, `Running`, `Finished`)_ - the status of the exercise
    - **start_time**: _optional timestamp_ - timestamp when this exercise state was started
    - **finish_time**: _optional timestamp_ - timestamp when this exercise state was finished
    - **elapsed_s**: _int_ - number of in-exercise seconds that elapsed in this exercise state

### teams.jsonl

Contains information about all the exercise teams.
Each object has the following format:

- **team_id**: _int_ - id of the team
- **name**: _string_ - name of the team
- **role**: _string_ - role of the team, empty string if no role
- **exercise_id**: _int_ - id of the exercise the team belongs to
- **finish_time**: _timestamp_ - time when the team reached one of the _final_ milestones
- **users**: _a list of user objects_ - a list of trainees assigned to this team
    - **user_id**: _uuid_ - id of the user
    - **username**: _optional string_ - username of the user, not present if the logs were anonymized
- **achieved_score**: _int_ - score achieved by the team during the exercise, includes score from
    instructor comments
- **total_score**: _int_ - total achievable score during the exercise, does not include score from
    instructor comments
- **exercise_state_id**: _int_ - id of the exercise state for this team

### instructors.jsonl

Contains all instructors assigned to the exercise.
Each object has the following format:

- **user_id**: _uuid_ - id of the user
- **username**: _optional string_ - username of the user, not present if the logs were anonymized

### exercise_injects.jsonl

Contains all injects for this exercise.
Each object has the following format:

- **inject_id**: _int_ - id of the inject
- **name**: _string_
- **display_name**: _string_
- **time**: _int_
- **delay**: _int_
- **organization**: _string_
- **type**: _(`Info`, `Email`)_
- **target**: _channel object_
    - **channel_id**: _int_ - id of the channel
    - **name**: _string_
    - **display_name**: _string_
    - **description**: _string_
    - **type**: _(`Info`, `Email`)_
- **alternatives**: _list of alternative objects_

#### _Info_ inject alternative

- **alternative_id**: _int_ - id of the alternative
- **name**: _string_
- **content**: _content_
- **control**: _control_
- **confirmation**: _optional confirmation object_

#### _Email_ inject alternative

- **alternative_id**: _int_ - id of the alternative
- **name**: _string_
- **subject**: _string_
- **content**: _content_
- **control**: _control_
- **extra_copies**: _int_

### exercise_milestones.jsonl

Contains all milestones for this exercise, which are referenced by milestones in each team.
Each object has the following format:

- **milestone_id**: _int_ - id of the milestone
- **name**: _string_
- **display_name**: _string_
- **description**: _string_
- **tags**: _list of strings_
- **file_names**: _list of strings_ - space separated list of file names
- **initial_state**: _bool_
- **score**: _int_ - score added to the team when this milestone is reached

### exercise_tools.jsonl

Contains all tools for this exercise, which are referenced by action logs in each team.
Each object has the following format:

- **tool_id**: _int_ - id of the tool
- **name**: _string_
- **tooltip_description**: _string_
- **default_response**: _content_
- **roles**: _list of strings_
- **requires_input**: _bool_
- **responses**: _list of response objects_
    - **response_id**: _int_ - id of the response
    - **param**: _string_
    - **content**: _content_
    - **control**: _control_
    - **time**: _int_
    - **regex**: _bool_

### exercise_questionnaires.jsonl

Contains all questionnaires for this exercise.
Each object has the following format:

- **questionnaire_id**: _int_ - id of the questionnaire
- **name**: _string_
- **content**: _content_
- **time**: _int_
- **control**: _control_
- **repeatable**: _optional repeatable object_
    - **max_attempts**: _int_
    - **on_fail**: _optional control_
- **questions**: _list of question objects_
    - **question_id**: _int_ - id of the question
    - **content**: _content_
    - **type**: _(`Radio`, `Free-form`, `Auto-free-form`, `Multiple-choice`)_
    - **details**: _additional details depending on the question type_

#### _Radio_ question details

- **labels**: _list of strings_
- **correct**: _int_
- **max**: _int_

#### _Free-form_ question details

- **related_milestone_ids**: _list of int_

#### _Auto-free-form_ question details

- **correct_answer**: _string_
- **correct**: _optional control_
- **incorrect**: _optional control_

#### _Multiple-choice_ question details

- **labels**: _list of strings_
- **correct**: _list of ints_
- **exact_match** _boolean_

### exercise_learning_objectives.jsonl

Contains all learning objectives for the exercise.
Each object has the following format:

- **objective_id**: _int_ - id of the learning objective
- **name**: _string_
- **description**: _string_
- **tags**: _list of strings_
- **total_score**: _int_ - total score achievable by completing all activities in this objective
- **activities**: _list of activity objects_
    - **activity_id**: _int_ - id of the activity
    - **name**: _string_
    - **description**: _string_
    - **tags**: _list of strings_
    - **milestone_ids**: _list of int_
    - **total_score**: _int_ - total score achievable by reaching all milestones linked to this
        activity

### exercise_channels.jsonl

Contains all channels that exist for the exercise.
Each object has the following format:

- **channel_id**: _int_ - id of the channel
- **name**: _string_
- **display_name**: _string_
- **description**: _string_
- **type**: _(`Info`, `Email`, `Tool`, `Form`)_

### email_participants.jsonl

Contains all email participants for this exercise, which are referenced by email threads and emails in each team.
Each object has the following format:

- **participant_id**: _int_ - id of the email participant
- **address**: _string_
- **exercise_id**: _int_ - id of the exercise this participant belongs to
- **team_id**: _optional int_ - team which this participant represents, null if it does not belong
    to any team
- **definition_address**: _optional definition address object_ - email address from the definition
    which this participant represents, null if it belongs to a team, definition address format:
    - **email_address_id**: _int_ - id of the definition address
    - **address**: _string_
    - **description**: _string_
    - **team_visible**: _boolean_
    - **organization**: _string_
    - **control**: _control_

### file_infos.jsonl

Contains all file infos for this exercise.
Each object has the following format:

- **file_id**: _uuid_ - id of the file, this is also the name of the file on the file system
- **file_name**: _string_ - original name of the file
- **uploaded_by_id**: _optional uuid_ - uuid of the user that uploaded this file, null if the file
    was not uploaded
- **uploaded_at**: _optional timestamp_ - timestamp when this file was uploaded, null if the file
    was not uploaded

## Individual team files

### inject_states.jsonl

Contains all the inject states for the team.
Each object has the following format:

- **team_id**: _int_ - id of the team
- **inject_id**: _int_ - id of the inject
- **status**: _(`Unsent`, `Delayed`, `Sent`)_ - status of the inject
- **alternative_id**: _optional int_ - id of the sent alternative

### questionnaire_states.jsonl

Contains all the questionnaire states for the team.
Each object has the following format:

- **questionnaire_state_id**: _int_ - id of the questionnaire state
- **questionnaire_id**: _int_ - id of the questionnaire
- **team_id**: _int_ - id of the team
- **status**: _(`Unsent`, `Sent`, `Answered`, `Reviewed`)_ - status of the questionnaire
- **timestamp_sent**: _optional timestamp_
- **timestamp_answered**: _optional timestamp_
- **timestamp_reviewed**: _optional timestamp_
- **reviewed_by_id**: _optional uuid_ - id of the instructor that reviewed the questionnaire
- **submissions**: _list of submission objects_
    - **answers**: _list of answer objects_
        - **question_id**: _int_ - id of the question
        - **answer**: _list of strings_ - submitted answers for this question
        - **correct**: _(`Correct`, `Incorrect`, `Partially Correct`, `Unknown`)_ - correctness of the
            question as determined by the backend
        - **attempt**: _int_ - attempt number of this answer
    - **attempt**: _int_ - attempt number of this submission
    - **accepted**: _boolean_ - flag that determines whether this submission was accepted by the
        platform, controlled by the repeatable field on the questionnaire

### milestones.jsonl

Contains all milestone states for the team.
Each object has the following format:

- **milestone_state_id**: _int_ - id of the milestone state
- **milestone_id**: _int_ - id of the milestone this state belongs to, the referenced milestone can
    be found in [exercise_milestones.jsonl](#exercisemilestonesjsonl)
- **reached**: _bool_ - state of the milestone
- **timestamp_reached**: _optional timestamp_ - time of the last state change for this milestone

### emails.jsonl

Optional file, included only if the email feature is enabled.
Contains all email threads and emails for the team.
Each object has the following format:

- **thread_id**: _int_ - id of the email thread
- **subject**: _string_
- **timestamp**: _timestamp_ - time when this email thread was created
- **participants**: _list of ints_ - list of email participant ids that belong to the thread, the
    referenced participants can be found in [email_participants.jsonl](#emailparticipantsjsonl)
- **emails**: _list of email object_ - list of emails sent to this thread, email object format:
    - **email_id**: _int_ - id of the email
    - **thread_id**: _int_ - id of the thread this email was sent to
    - **sender_id**: _int_ - id of the email participant that sent the email
    - **timestamp**: _timestamp_ - time when this email was sent
    - **content**: _content_

### action_logs.jsonl

Contains all the action logs for the team.
Each object has the following format:

- **action_log_id**: _int_ - id of the action log
- **type**: _(`Inject`, `Custom Inject`, `Tool`, `Email`, `Form`, `Form Submission`, `Form Review`,
    `Confirmation`, `File Download`, `Milestone Modification`)_ - type of the action log
- **timestamp**: _timestamp_ - time when this action log was created
- **channel_id**: _int_ - id of the channel this action log was sent to
- **instructor_comment**: _optional instructor comment_
    - **comment**: _string_ - the content of the comment
    - **score**: _int_ - score value assigned to the comment, this value is added to the team's
        achieved score, can be negative
    - **created_by_id**: _optional uuid_ - uuid of the instructor that created this comment
    - **created_at**: _timestamp_ - timestamp when this comment was created
    - **edited_by_id**: _optional uuid_ - uuid of the instructor that edited this comment
    - **edited_at**: _optional timestamp_ - timestamp when this comment was edited
- **previous_log_id**: _optional int_ - optional id of an action log that is connected to this
    action log
- **user_id**: _optional uuid_ - optional uuid of the user that performed this action, null for
    automatic actions performed by the platform
- **in_exercise_time**: _int_ - the in-exercise-time value when this action happened
- **details**: _additional details depending on the action log type_

#### _Inject_ details

- **inject_id**: _int_ - id of the inject
- **content**: _content_
- **confirmation**: _optional confirmation object_
- **overlay**: _optional overlay_

#### _Custom Inject_ details

- **content**: _content_
- **overlay**: _optional overlay_

#### _Tool_ details

- **tool_id**: _int_ - id of the used tool
- **argument**: _string_ - argument provided to the tool
- **content**: _content_
- **selected_response_id**: _optional int_ - id of the selected tool response

#### _Email_ details

Same fields as described in `emails.jsonl` in the `emails` field.

#### _Form_ details

- **questionnaire_state_id**: _int_ - id of the sent questionnaire state

#### _Form Submission_ details

Same fields as described in `questionnaire_states.jsonl` in the `submissions` field.

#### _Form Review_ details

This object currently contains no additional fields.

#### _Confirmation_ details

This object currently contains no additional fields.

#### _File Download_ details

- **file_info_id**: _uuid_ - uuid of the file info that was downloaded

#### _Milestone Modification_ details

- **activated_milestone_states**: _list of ints_ - ids of the milestone states that were activated
- **deactivated_milestone_states**: _list of ints_ - ids of the milestone states that were
    deactivated
- **cause**: _(0, 2, 4)_ - the cause for this milestone modification, 0 for trainee action, 2 for
    instructor action, 4 for automatic action

## Comparing logs from multiple exercises

The logs are constructed in a way that should allow for simple comparison of logs from multiple
exercises with the same definition.
It has to be the exact same upload of the definition,
otherwise the IDs of `injects`, `tools` and other definition data will not match.

Between different runs of the same definition, the `exercise_injects.jsonl`,
`exercise_milestones.jsonl` and `exercise_tools.jsonl` files will be the same.
The `email_participants.jsonl` will not match because email participants
need to be re-generated for every exercise.
However, the definition address they are linked to will have the same ID in all runs.
