# INJECT Exercise Platform – Documentation

This repository contains the documentation for the Inject Exercise Platform (IXP).

## Table of Contents

- [Rules and Conventions](#rules-and-conventions)
- [Dependencies](#dependencies)
- [Scripts](#scripts)
- [Deploying a Local Server](#deploying-a-local-server)
- [Updating the Documentation](#updating-the-documentation)
- [GitLab Pages](#gitlab-pages)
- [Deploying a New Version of Documentation](#deploying-a-new-version-of-documentation)
- [Used CI/CD Variables](#used-cicd-variables)

## Rules and Conventions

In order to prevent potential problems, the following rules and conventions should be complied with.

### Conventional Commits

To keep the commit history of documentation repository clean and readable, the following prefixes of
the names of commits should be used:

- **docs: title** - general documentation changes/updates (usually less technical texts), structure
    change
- **feat: title** - describing behavior of new feature or technology
- **fix: title** - correction of information
- **ops: title** - changes in devops instructions/requirements
- **style: title** - formatting
- **release: title** - work needed to be done when creating new release (updating .gitlab-ci.yml,
    updating dropdown menus)
- **internal: title** - updates of the documentation infrastructure (e.g., CI/.gitlab-yml,
    README.md, scripts)
    - in order to provide more information, you can use one of the suffixes:
        - **internal-feat** - new feature of the documentation infrastructure
        - **internal-docs** - usually README.md updates
        - **internal-fix** - fix of a problem within the documentation infrastructure
        - **internal-ops** - updates of CI/CD
        - **internal-style** - formatting of internal/non-publicly visible files

### Feature Branches and Merge Requests

When making changes to the documentation, feature branches with merge requests (MRs) should be used
to provide well documented history of changes with possible comments and discussions in the MRs.

Merge requests must have always assigned a reviewer. In most cases it would be Jan V. and Martin H.,
but others who can provide feedback for the given MR can be assigned too.

Before merging the MR, above the **merge** button there is option to **Edit commit message**. Click
on it and edit the first line in the **Squash commit message** section to contain conventional
commit and _concise description_ (e.g., feat: add description for using the editor).

A **good merge** request should:

- **contain related changes** (single purpose)
- **have a clear description**
- **be complete but minimal** (avoid adding unnecessary refactoring, unrelated fixes, or multiple
    features – this should be split into multiple MRs!)

### Formatting

For consistently clean and formatted Markdown files, utilize the [format.sh script](#4-formatsh).

- **_Maintain one sentence per line._**
- Lines should ideally not exceed 100 characters (can be followed, but does not have to be strict, e.g. 105 chars is OK).
- Let the formatter handle the rest.

## Dependencies

- [Python](https://www.python.org/downloads/) version 3.10
- [Poetry](https://python-poetry.org/docs/) version 1.8.0 and higher

## Scripts

For the convenient usage of documentation and its versioning the following scripts are present:

### 1. update-submodules.sh

This script is a prerequisite to the _update-files.sh_ (unless initiation and switching to correct
branches/tags of submodules is done manually). The _update-files.sh_ script then uses files from
this submodules.

The script takes argument `-v` (or verbose `--version`) followed by a version (in format: `vX.0.0`)
of the tag. If no argument is provided it pulls the newest states for every submodule/repository.

#### Usage Examples

- We created the new branch for the new release from the main.

- Backend, definitions-registry, and docker-deployment repositories are ready and appropriate
    commits are tagged (or the latest commits contain the wanted content).

- We run the following script if the repositories contain the wanted tag (e.g., v4.0.0):

    ```bash
    poetry run ./update-submodules.sh -v v4.0.0
    ```

- If the repositories do not contain a tag, but we know that latest content is wanted, we simply
    run:

    ```bash
    poetry run ./update-submodules.sh
    ```

    This will use the newest commits/content from all repositories.

- If you require different branch/tag for each repository, you can do it manually the following way:

    ```bash
    # initialize the submodules
    git submodule init
    git submodule update --remote
    ```

    then within each submodule repository:

    ```bash
    cd ./<repository-name>
    git fetch --tags # only if you want to use a tag
    git checkout main
    git checkout tags/<your-wanted-tag> # only if you want to use a tag and it exists
    git checkout <commit-hash> # alternatively if no tag is present use commit hash of wanted state
    ```

Then, if you have each repository in the wanted state, proceed to _update-files.sh_.

### 2. update-files.sh

This script is responsible for the downloading of files that are drown from another repositories.
It is expected that either _update-submodules.sh_ was used are manual submodule initiation was done.

The following files are pulled from other repositories:

- (example) `file name` - source repository path -> documentation path
- `README.md` - _/backend/definitions/_ -> _/docs/tech/architecture/definitions/_
- `CHANGELOG.md` - _/backend/definitions/_ -> _/docs/tech/architecture/definitions/_
- `openapi.yml` - _/backend/_ -> _/docs/tech/api/_
- `showcase-definition.zip` - _/definition-registry/_ -> _/files-from-repos/showcase-definition.zip_
- `introductory-definition.zip` - _/definition-registry/_ -> _/files-from-repos/intro-definition.zip_
- `users.md` - _/docker-deployment/_ -> _/docs/tech/installation/users.md_
- `troubleshooting.md` - _/docker-deployment/_ -> _/docs/tech/installation/troubleshooting.md_
- `README.md` - _/docker-deployment/_ -> _/docs/tech/installation/setup.md_
- `installation.md` - _/docker-deployment/_ -> _/docs/tech/installation/overview.md_
- `README.md` - _/docker-deployment/https-owncert/_ -> _/docs/tech/installation/https/owncert.md_
- `README.md` - _/docker-deployment/https/_ -> _/docs/tech/installation/https/base-setup.md_

### 3. update-links.sh

This script serves for correcting all links pointing to version specific files (mostly definitions
zip archives). It is a simple sed (stream editor) script for changing the urls/paths.

It takes `-n` (or `--name`) argument followed by a name of the branch (of the given release). For
example:

- We created new branch from the main (e.g., `dux`).
- We want every link to point on the file located on this branch.
- We use:
    ```bash
    poetry run ./update-links.sh -n dux
    ```
- We should see modified files by the script via diff.

### 4. format.sh

This script formats all `.md` files according to the needs of the mkdocs site generator. If the code
was not formatted by this script the pipeline will probably fail!

The usage is simple:

```bash
poetry run ./format.sh
```

## Deploying a Local Server

To set up and run a local server for the documentation:

1. Install dependencies:

    ```bash
    poetry install
    ```

2. _(If needed)_ Run the [utilities scripts](#scripts) to update (or downgrade) files pulled from
    other repositories:

    ```bash
    # (substitute 'X' for actual version number, e.g. v3.0.0)
    poetry run ./update-submodules.sh -v vX.0.0

    poetry run ./update-files.sh

    # (substitute 'releaseName' for actual release name, e.g. dux)
    poetry run ./update-links.sh -n releaseName
    ```

3. Start the local MkDocs server:

    ```bash
    poetry run mkdocs serve
    ```

4. Access the documentation at [http://localhost:8000](http://localhost:8000). If port 8000 is taken
    by another service, you can change the default port using the `-a` or `--dev-addr` option:

    ```bash
    poetry run mkdocs serve -a <IP:PORT>
    ```

## Updating the Documentation

The following situations can happen depending on where are the updates/changes wanted.

### 1. Apply updates only to release branch

This is straight forward:

1. Create feature branch from the given release branch.
    ```bash
    git switch <release-branch-name>
    git pull
    git switch -c <feature-branch-name>
    ```
2. Apply changes (commits) to the given feature branch.
3. Create MR _(feature-branch -> release-branch)_.

### 2. Apply updates only to main branch

This is straight forward:

1. Create feature branch from the main branch.
    ```bash
    git switch main
    git pull
    git switch -c <feature-branch-name>
    ```
2. Apply changes (commits) to the given feature branch.
3. Create MR _(feature-branch -> main)_.

### 3. Apply updates to both main and release branch.

1. Firstly, use the
    **[1. Apply updates only to release branch](#1-apply-updates-only-to-release-branch)** process.
2. Wait until the changes are approved and merged. Then pick either manual or automatic guide
    (either step **3a** or **3b**) to propagate changes to main branch.

#### !!! WARNINGS and RESTRICTIONS for _**3a – Automatic copy of the updates**_ option !!!

- Automatic copies can by made only from the, so called, _preferred branch_ (that is _the most
    **recent release branch**_) to the _main branch_.
- The process applies _**the last squash commit**_ from the _preferred branch_ to the _main_. This
    effectively means that the job `copy_to_main` needs to be run after the wanted changes are
    merged into the _preferred branch_ and most importantly **BEFORE** any other changes are merged
    to the _preferred branch_!
- Manual updates (**3b – Manual copy of the updates.**) do not need to follow this rule. You can
    merge several MRs and then copy the changes via cherry pick.
- There are several reasons for this measure:
    - Simplified CI job.
    - Less room for errors that would create conflicts.
    - The history of changes in the main will mirror the history of changes in the preferred branch.
    - Less chance for creating merge conflicts (applying commits in different order can be).

#### 3a – Automatic copy of the updates.

1. Open the commit history of the release branch (where you merged the MR).

2. On the top you should see the latest commit. It should be the one created by your MR. Open the
    pipeline of the commit (gear wheel icon left to the commit hash).You should have option to run
    the `copy_to_main` stage – run it.

3. This action will create new branch from main with name `COPY-<commit-hash>` and tries to apply
    the wanted changes.

4. Two situations can happen:

    - **Success** – If the appliance of changes can be done automatically, the merge request (and the
        created branch) will contain the applied commit and MR will inform you in its description
        that commit was applied successfully. Ping someone for approve and merge it. The branch will
        delete itself.

    - **Fail** – If the appliance cannot be done automatically (due to merge conflicts), the merge
        request will contain no commits and it will contain message that appliance of the commit
        cannot be done. You will need to:

        1. Update your local repository (since new branch was created on the remote).
            ```
            git fetch
            ```
        2. Switch to the created branch (the name can be seen in the MR – `COPY-<commit-hash>`)
            ```
            git switch COPY-<commit-hash>
            ```
        3. Try to apply the commit yourself
            ```
            git cherry-pick <commit-hash>
            ```
        4. This should end with error:
            ```
            CONFLICT: Merge conflict
            error: could not apply <commit-hash>
            ```
            You will have the files with merge conflicts edited with sections looking like this:
            ```
            <<<<<<< HEAD
            Changes introduced by mainline
            ======
            Changes added by the cherry-picked commit
            >>>>>>> <commit-hash>
            ```
            Edit them manually to contain the correct changes.
        5. Add your conflict resolution and push it.

        ```
        git add .
        git commit -m "fix: resolve merge conflict"
        git push
        ```

        6. Now, the merge request should contain the wanted changes. Ping someone for approve and merge
            it. The branch will delete itself.

#### 3b – Manual copy of the updates.

1. Create temporary branch from the main.

    ```bash
    git switch main
    git pull
    git switch -c <temporary-branch-name>
    ```

2. Apply the commits from the release branch to this temporary branch (via git cherry-picking). This
    can be done via enumeration of the commit hashes (this command will apply commits with the
    given hashes to the current, release, branch):

    ```bash
    git cherry-pick <commit1-hash> <commit2-hash> <commit3-hash>
    ```

    Or alternatively (if the commits are consecutive):

    ```bash
    git cherry-pick <oldest-commit-hash>^..<newest-commit-hash>
    ```

    By one of this command the changes are added to the current branch (temporary-branch-name). It is
    possible that conflicts occur during this step. In such case, they will need to be locally
    resolved.

3. After successful addition of the wanted commits from main to the feature branch push the changes
    to remote repository:

    ```
    git push --set-upstream origin <temporary-branch-name>
    ```

4. Create a MR _temporary-branch-name -> main_. This MR should have in title: _[COPY-commit_name]_
    to show that these changes were already approved on a different branch. For this reason, you
    can request anyone for the approve.

## GitLab Pages

The documentation is built and deployed to GitLab Pages. The hosted documentation is available at:

[https://inject.pages.fi.muni.cz/inject-docs](https://inject.pages.fi.muni.cz/inject-docs)

### Pipeline

Deployment of the documentation to GitLab Pages is managed through a CI/CD pipeline defined in the
`.gitlab-ci.yml` file. GitLab CI/CD will automatically run the pipeline, which includes the
following stages. You can also manually trigger the pipeline if needed:

- Setup: Install necessary dependencies.
- Build Documentation: Use MkDocs to build the site.
- Deploy Pages (manual stage): Deploy the built documentation to GitLab Pages.

The deployment of the documentation is available via running the manual stage - Deploy Pages - to
GitLab Pages upon successful completion of all previous stages. Most importantly the _Build
Documentation_ must be successful.

### Parallel Pages and Branches

The documentation supports the deployment of multiple pages for different versions. This is done via
[GitLab Parallel Deployments](https://about.gitlab.com/blog/2024/09/23/gitlab-pages-features-review-apps-and-multiple-website-deployment/).

The way these parallel pages are to be deployed is as follows.

1. The main branch of the project is always the newest, not-yet-published page. Here we can slowly
    keep adding more and more content.
2. The main branch should never be deployed.
3. For that, we are going to create a new branch named after the release name, and this new branch
    is what we are going to release. **The main page is always the newest published release
    branch.**

The main difference between the main page of docs (the newest release) and parallel (older releases)
is that the parallel has a postfix in the URL and thus, it is not the primary site. When setting up
the version, you need to make a few adjustments to the gitlab-ci.yml.

## Deploying a New Version of Documentation

To create a new page for the documentation of the newest release you need to follow these steps:

### 1. Update and detach from main

Use the [scripts](#scripts) to pull and rewrite the files that are used from other repositories.

**Important note!** When _detaching_ from _main_, the content of the release (the state of the main
branch) should be _revised_ and _checked_. If mistakes or discrepancies are found after detaching,
it is needed to fix them _both in the detached release branch and main_ (this can be done by
following [this guide](#3-apply-update-to-both-main-and-release-branch)). For this reason it is
advised to detach from the main branch _as late as possible_.

Create a branch whose content will be used when publishing a new page of the new release.

```bash
git switch main  # new release is done from the main branch
git pull  # pull all newest changes
git switch -c <new-release-branch-name>  # create the new release branch
```

### 2. New main page creation

Since, as mentioned in the [Parallel Pages and Branches](#parallel-pages-and-branches) section, the
newest release is the main page -> we need to modify the `.gitlab-ci.yml` the following way:

```yml
pages:
[...]
  # add following:
  pages:
    path_prefix: ""
    expire_in: never
[...]
```

This means that this new page will not have any prefix (respectively postfix in the context of the
URL).

Push this change to the new release branch.

### 3. Update older releases

#### 3.1 Update path prefix in the former newest release

Since the _former newest release_ is being replaced with a new release, it needs obtain a prefix.
Until now, it had no prefix, since it was _the newest release_, which is the main page (i. e.
_https://docs.inject.muni.cz_). Now, it needs to become parallel deployment (as all older versions –
_https://docs.inject.muni.cz/release-name_).

Add the release-name to the _former newest release_ that is being replaced:

```yml
pages:
[...]
  pages:
    path_prefix: "<the-former-newest-release-name>" # TODO: rewrite the value
    expire_in: never
[...]
```

#### 3.2 Update versions combo box

Another part that needs to be adjusted for each page upon new release is the menu for switching
versions. It's located in `overrides/partials/header.html` where you can find the following section.

On the newest release it could look like this:

```html
<!-- Dropdown Menu for Page Versions -->
<div class="md-header__dropdown">
  <select onchange="window.location.href=this.value;">
    <option value="#">Currently-newest-released-release</option>
    <option value="/bastion/">Bastion</option>
    <option value="/aegis/">Aegis</option>
  </select>
</div>
```

On a parallel page (for example _bastion_):

```html
<!-- Dropdown Menu for Page Versions -->
<div class="md-header__dropdown">
  <select onchange="window.location.href=this.value;">
    <option value="#">Bastion</option>
    <option value="/">Currently-newest-released-release</option>
    <option value="/aegis/">Aegis</option>
  </select>
</div>
```

When adding a new page, you will need to add a new option to this list. The explanation of the tags
`#`, `/`, and `/{name}`:

- value="#" points to the page where the user is currently (meaning each dropdown menu of each older
    release branches should have there its own name)
- value="/" points to the main page from the parallel page (this points to the newest release)
- value="/{branch-name}/" points to an older parallel page release

### 4. Hand brake removal

To avoid accidental publishing, the tag `only:` should always be set to the name of the branch where
it is located. Also, the main branch should never have a valid gitlab-ci.yml that can publish pages.
For that reason the tag `only:` on the main branch of the project should be set to a different name
other than the main branch. On the main branch, the tag is set to `change-me`, you need to change
all of the occurrences of this tag to the current branch name.

### 5. Update PREFERRED_BRANCH CI/CD variable

To allow automatic propagation of changes from the newest release to the main, we need to update the
value of `PREFERRED_BRANCH` variable to the name of the new release branch.

### 6. Publishing

When all of the previous steps are done, you can run the manual stage `pages`, which deploys the
newest changes. It is needed to redeploy the older releases as well (due to change in the dropdown
menu). This can be done in the GitLab section `Build` -> `Pipelines` or by `Code` -> `Commits` ->
and clicking on the green tick of the latest commit that should be deployed.

## Used CI/CD Variables

In order to keep the `copy_to_main` job operational, the following CI/CD variables need to be set:

### GITLAB_TOKEN

This variable contains the GitLab `access token` with `write_repository` scope. It is needed for the
automated MR creation when copying changes from the _PREFERRED_BRANCH_ to the _main_.

### PREFERRED_BRANCH

This variable holds the name of the _latest release branch_. It allows to run the `copy_to_main` job
only on the branch with this name.
