Commit 36934e00 authored by Martin Juhas's avatar Martin Juhas
Browse files

more updates

parent 4624c832

docs/tech/log-format.md

+52 −7
Original line number Diff line number Diff line
@@ -9,6 +9,10 @@ Bellow is description of the structure and schema of logs. 
  - team-id/

    - uploaded_files/

        - files uploaded by team-id and for team-id during the exercise

    - llm-evaluations/

        - email_suggestions.jsonl

        - email_evaluations.jsonl

        - free_form_evaluations.jsonl

    - inject_states.jsonl

    - questionnaire_states.jsonl

    - action_logs.jsonl
@@ -29,6 +33,7 @@ Bellow is description of the structure and schema of logs. 
  - exercise_injects.jsonl

  - exercise_channels.jsonl

  - email_participants.jsonl

  - llm_assessments.jsonl

```



## Description and format of individual files
@@ -65,11 +70,6 @@ All fields with the `timestamp` type are timestamps in `ISO 8601` format. 


- **duration**: _optional int_



#### LLM Assessment



- **persona**: _string_ - the system prompt for the LLM defining the behaviour and persona of the LLM

- **assessment**: _string_ - prompt with specific task context and instructions for assessment



## Exercise files



### exercise.jsonl
@@ -219,7 +219,7 @@ Each object has the following format: 
#### _Free-form_ question details



- **related_milestone_ids**: _list of int_

- **llm_assessment**: _optional llm assessment_

- **assessment_id**: _optional int_ - id of the llm assessment



#### _Auto-free-form_ question details
@@ -285,7 +285,7 @@ Each object has the following format: 
    - **team_visible**: _bool_

    - **organization**: _string_

    - **control**: _control_

    - **llm_assessment**: _optional llm assessment_

    - **assessment_id**: _optional int_ - id of the llm assessment



### file_infos.jsonl
@@ -299,6 +299,15 @@ Each object has the following format: 
- **uploaded_at**: _optional timestamp_ - timestamp when this file was uploaded, null if the file

    was not uploaded



### llm_assessments.jsonl



Contains all LLM assessments for this exercise.

Each object has the following format:



- **assessment_id**: _int_ - id of the assessment

- **persona**: _string_

- **assessment**: _string_



## Individual team files



### inject_states.jsonl
@@ -444,6 +453,42 @@ This object currently contains no additional fields. 
- **cause**: _(0, 2, 4)_ - the cause for this milestone modification, 0 for trainee action, 2 for

    instructor action, 4 for automatic action



### email_suggestions.jsonl



Contains all email suggestions generated for this team.

Each object has the following format:



- **suggestion_id**: _int_ - id of the suggestion

- **thread_id**: _int_ - id of the thread

- **trigger_email_id**: _int_ - id of the email that the suggestion is responding to

- **email_participant_id**: _int_ - id of the definition participant

    that contains the assessment criteria

- **response**: _string_ - the suggested text

- **created_at**: _timestamp_ - time when this suggestion was created



### email_evaluations.jsonl



Contains all email evaluations generated for this team.

Each object has the following format:



- **evaluation_id**: _int_ - id of the email evaluation

- **action_log_id**: _int_ - id of the email action log

- **assessment_id**: _int_ - id of the llm assessment

- **response**: _string_ - the text of the evaluation

- **created_at**: _timestamp_ - time when this evaluation was created



### free_form_evaluations.jsonl



Contains all free form evaluations generated for this team.

Each object has the following format:



- **evaluation_id**: _int_ - id of the email evaluation

- **submission_id**: _int_ - id of the questionnaire submission

- **question_id**: _int_ - id of the free-form question

- **assessment_id**: _int_ - id of the llm assessment

- **response**: _string_ - the text of the evaluation

- **created_at**: _timestamp_ - time when this evaluation was created



## Comparing logs from multiple exercises



The logs are constructed in a way that should allow for simple comparison of logs from multiple