Artifact overhaul

Tasks

KAFE types and objects

Add KafeType. It needn't replace ShardKind initially.
Add KafeObjects as artifact properties to ArtifactInfo.
Implement KafeObjectJsonConverter.
Either switch Marten to System.Text.Json or also implement KafeObjectJsonConverter for Newtonsoft.Json.

Modding API

Add IMod or some abstraction allowing new shard and blueprint validators to be registered.
Add Kafe.Core with basic KAFE types and requirements.
Turn Kafe.Media into an IMod.
Add Kafe.Blender (... or Kafe.ThreeD, Kafe.Polygons, etc.)
Add API to register KAFE mods.
Move IEntity to common.
Add IRepository<T>, IShard, and IEntity, so that requirements can be written without a reference to the Data mod.
Allow automatic assembly-wide registration of IPropertyType, IShardMetadata, IRequirement, IEntity, IDiagnosticPayload

Shards

Diagnostics

Refactor the Error abstraction into a more general Diagnostic abstraction.
Allow registration of Diagnostic types in mods.
Replace all uses of common Error types with the new Diagnostics.
Implement diagnostic message formatting (through Serilog for now)

Blueprints

Implement various blueprint IRequirements needed to validate at least the FFFI films.
Implement BlueprintInfo and add ProjectGroupInfo.BlueprintSlots and RequiredReviewerRoleIds.
Add a single blueprint for "FFFI film" or multiple for "FFFI film", "FFFI film video-annotation", and "FFFI film metadata".

Misc

Remove Kafe.Data's dependency on Kafe.Media and Kafe.Polygons.
Deploy to staging.
Deploy to production.
Close related issues that this encompasses.
- #34, #39, #43, #48, #63, #127, #218, #237, #240

Separated from this proposal

#273 Splitting of original and generated shards

Proposal

Artifacts are currently inflexible. Right now, they are essentially just named places to attach shards to. This issue discusses the current model of projects, artifacts, and blueprints and its flaws and proposes an overhaul.

Current abstraction

A. Projects

Currently, projects are defined like this (csharp-ish pseudocode):

record ProjectInfo(
    Hrib Id,
    Hrib ProjectGroupId,
    LocalizedString Name,
    ProjectArtifactInfo[] Artifacts,
    ProjectReviewInfo[] Reviews,
    Permission GlobalPermissions,
    bool IsLocked,

    // NB: FFFI-specific properties
    ProjectAuthorInfo[] Authors,
    LocalizedString Description,
    LocalizedString Genre,
    DateTimeOffset ReleasedOn
);

public record ProjectAuthorInfo(
    Hrib AuthorId,
    (Unknown | Crew | Cast) Kind,
    string[] Roles
);

public record ProjectArtifactInfo(
    Hrib ArtifactId,
    string? BlueprintSlot
);

public record ProjectReviewInfo(
    (NotReviewed | Accepted | Rejected) Kind,
    string ReviewerRole,
    LocalizedString Comment,
    DateTimeOffset AddedOn
);

When I designed this, KAFE was only used for FFFI. This model obviously isn't very accommodating to gamedev, 3D modeling, or even PV110. It has the following issues:

Description and Genre are specific to FFFI and the needs of the festival brochure. For example, 3D models might not have a need for Genre, or it can have a different meaning.
Authors are divided into Cast and Crew, which is not something that comes naturally to, for example, games.
Authors need to be bound to an AuthorInfo. This results in a lot of duplicate or otherwise redundant authors as they are not publicly available to anyone but their creator.
It's still unclear to me what to even do with ReleasedOn. I originally included it only because it was in WMA, and it seemed to be a good idea to migrate it over to KAFE. I'm not even sure what the date signifies (I guess the date of the relevant festival).
Reviews don't use the new Role abstraction (#138). They rely on a simple string instead.
Artifacts fill some kind of a "blueprint slot," which doesn't mean much since blueprints don't exist (besides the one mock blueprint for festival films).

B. Artifacts

Currently:

record ArtifactInfo(
    Hrib Id,
    CreationMethod CreationMethod,
    LocalizedString Name,
    DateTimeOffset AddedOn
);

record ShardInfoBase(
    Hrib Id,
    Hrib ArtifactId,
    DateTimeOffset CreatedAt,
    (Unknown | Video | Image | Subtitles) ShardKind
);

record VideoShardInfo(...) : ShardInfoBase;
record ImageShardInfo(...) : ShardInfoBase;
record SubtitlesShardInfo(...) : ShardInfoBase;

This abstraction also has issues:

ArtifactInfo itself offers no way of storing metadata except for Name and AddedOn.
It is impossible to add a new shard type without modifying ShardInfoBase and its ShardKind enum. This greatly limits extensibility.
There is no fallback shard kind. Something like a BlobShard for any binary file.

C. Blueprints

Blueprints don't really exist yet. The BlueprintInfo type is literally just:

record BlueprintInfo(
    Hrib Id
);

There is, however, ProjectBlueprintDto, which we use as a mockup and send it as a part of every project detail response payload:

record ProjectBlueprintDto(
    LocalizedString Name,
    LocalizedString Description,
    string[] RequiredReviewers,
    // NB: The `string` key of the dictionary is the "blueprint slot" referenced in `ProjectArtifactInfo`.
    Dictionary<string, ProjectArtifactBlueprintDto> ArtifactBlueprints
);

record ProjectArtifactBlueprintDto(
     LocalizedString Name,
     LocalizedString Description,
     (int Min, int Max) Arity,
     Dictionary<(Unknown | Video | Image | Subtitles), ProjectArtifactShardBlueprintDto> ShardBlueprints
);

record ProjectArtifactShardBlueprintDto(
     LocalizedString Name,
     LocalizedString Description,
     (int Min, int Max) Arity
);

It has... problems:

The blueprint is at the project level. Artifacts themselves have not idea they are subject to a blueprint.
The artifact blueprint has no way to declare the metadata it should contain (e.g., description, genre, etc.)
The artifact blueprint can define only a single "shape" per shard kind, completely disregarding the possibility that an artifact might have, for example, two shards of the same kind but different semantics.
The only form of declarative validation is Arity.
Like with ProjectReviewInfo above, it doesn't work with the Role abstraction (#138).

Proposed overhaul

Imagine, if you will, a museum. In a museum, exhibits are put on display in glass boxes with labels. The exhibits are also arranged into collections, subcollections, etc. Each exhibit is unique, though the labels are usually in the same format. In a gallery, the labels might contain: name, author, year, artistic movement, donor, description; in several language variants. I'm using this metaphor as a basis for the overhaul proposal.

Artifacts = Exhibits

Our artifacts are like the exhibits. Each can have a unique shape with only a few things that are common to all of them:

record ArtifactInfo(
    Hrib Id,
    LocalizedString Name,
    DateTimeOffset AddedOn,
    Dictionary<string, ArtifactProperty> Properties
);

record ArtifactProperty(
    KafeType Type,
    object Value
);

record AuthorReference(
    Hrib? AuthorId,
    string? Name,
    string[] Roles
);

record ArtifactReference(
    Hrib ArtifactId,
    bool ShouldInheritPermissions
);

// == KafeType examples ({namespace}:{type}[/{subtype}][\[\]] ==
// scalar values: core:string, core:number, core:date-time
// author references: core:author-reference
// artifact references: core:artifact-reference
// shards: core:shard/blob, media:shard/video, media:shard/image, media:shard/subtitles, 3d:shard/blender-scene

record ShardInfo(
    Hrib Id,
    Hrib ArtifactId,
    DateTimeOffset CreatedAt,
    long Size,
    string Filename,
    object Metadata,
    KafeType Type
);

In the proposed model, an artifact can have any number of properties in a flat hierarchy. (Arrays are always single-dimensional.) This would solve these issues:

Allows storing pretty much arbitrary metadata about the artifact in key-value store. Solves B.1. If we move the FFFI-specific metadata to an artifact, it would also solve A.1 and A.2 and would at least hide A.4.
Properties could be set in bulk using an event (e.g. ArtifactPropertiesSet). For each property, the event could specify whether to override or somehow augment the existing value (e.g. appending to a list).
AuthorReference would allow the user to specify the author's Hrib or use the name directly. This would solve A.3.
KafeType is not an enum but an instance of a struct that must be registered in KAFE at runtime. This would provide an extensibility point. However, it would require a custom JSON converter for ArtifactProperty, where Type would tell the converter what to deserialize.
KafeType would also replace ShardKind. Solves B.2. However, the ShardInfo projection would have to be pluggable to allow for shard-specific events. There would also need to be a custom JSON converter for ShardInfo.
Adding a fallback Blob shard would be fairly trivial. Solves B.3. This model would also hypothetically allow shards to change their type if need be. However, the current model already created a DB table for each shard kind which could be problematic to merge.
KafeType would be serializable to a syntax like: {namespace}:{type}[/{subtype}][\[\]].
~~An artifact can fulfill zero or more blueprints, specified through BlueprintIds. This moves blueprints to the artifact level (see C.1).~~
Artifact references allow artifacts to be nested. This allows us to model scenarios like FFFI registrations, where the top level artifact Registration has subartifacts Film, Videoannotation, etc.
If ArtifactReference.ShouldInheritPermissions is set to false, permissions from the superartifact are not inherited by the subartifact. This is mainly futureproofing for situations where artifacts form arbitrary graphs, and it is necessary to regulate the flow of permissions.

Blueprints = Format of labels on exhibits

A blueprint would provide a guideline on what properties should an artifact contain. In terms of the museum metaphor, a blueprint provides a template for the labels on the exhibits in the museum. Though, that's not entirely accurate, since in this proposal, a blueprint may define the shape of the exhibit itself.

record BlueprintInfo(
    Hrib Id,
    LocalizedString Name,
    LocalizedString? Description,
    Dictionary<string, BlueprintProperty> Properties,
    string[]? PropertyOrder,
    IBlueprintRequirement[] ArtifactNameRequirements,
    bool AllowAdditionalProperties
);

record BlueprintProperty(
    LocalizedString Name,
    LocalizedString? Description,
    IBlueprintRequirement[] Requirements
);

record IBlueprintRequirement(
    KafeType Type
);

// == Blueprint requirement examples ==

record TypeRequirement(
    KafeType Type = "core:requirement/type",
    KafeType PropertyType
) : IBlueprintRequirement;

record RequiredRequirement(
    KafeType Type = "core:requirement/required"
);

record ArityRequirement(
    KafeType Type = "core:requirement/arity",
    int Min,
    int Max
) : IBlueprintRequirement;

record FileSizeRequirement(
    KafeType Type = "core:requirement/file-size",
    long Min,
    long Max
) : IBlueprintRequirement;

record LanguagesRequirement(
    KafeType = "core:requirement/languages"
    string[] RequiredLanguages,
    string[] ForbiddenLanguages,
    bool AllowAdditionalLanguages
) : IBlueprintRequirement.

record BlueprintFulfilmentRequirement(
    KafeType = "core:requirement/blueprint-fulfilment",
    Hrib BlueprintId
) : IBlueprintRequirement;

record MediaDurationRequirement(
    KafeType Type = "media:requirement/duration",
    TimeSpan MinDuration,
    TimeSpan MaxDuration
) : IBlueprintRequirement;

That's quite a complex structure, yes, but it provides the flexibility KAFE needs:

Since the blueprint can force artifacts to contain arbitrary properties, it solves C.2.
There is no longer a notion of a "shard blueprint", thus C.3. is solved.
IBlueprintRequirement is flexible enough to provide arbitrary validation. (Solves C.4.) Since it relies on KafeType, it is also extensible.
The declarative nature of blueprint requirements means that they may be easily serialized and used by the front-end for client-side validation (#240).
A creator of a blueprint can decide whether the blueprint is strict, and any properties other than those in Properties are considered to be an error, by setting AllowAdditionalProperties to false.
Since ArtifactInfo.Name is the only piece of metadata built-in, a blueprint cannot attach requirements to it. Therefore, ArtifactNameRequirements is there to fill this gap.
Each blueprint requirement should generate appropriate error messages in all relevant languages. In the future, there may be additional configuration to allow overriding them.
Since you can set or unset any property, this fixes #43.
Since there may be image shard arrays (not just a blueprint slot with an arity), this fixes #39.
The BlueprintFulfilmentRequirement (I'm not sure about the name yet) can be used to force a subartifact to fulfil a blueprint (Scenario: FFFI registration has property "Film" which must fulfil blueprint "FFFI Film").
PropertyOrder is an optional array that can be used to give fixed order to the keys of Properties. This is so that the order of properties in the UI can be changed manually. Property keys not in the array should be appended at the end in alphabetic order.

Projects and project groups = Exhibit collections

Finally, projects and project groups would organize artifacts into something akin to collections in a museum.

record ProjectGroupInfo(
    Hrib Id,
    Hrib OrganizationId,
    LocalizedString Name,
    LocalizedString? Description,
    DateTimeOffset Deadline,
    bool IsOpen,
    Permission GlobalPermissions,
    Hrib ProjectBlueprintId,
    HashSet<Hrib> RequiredReviewerRoleIds
);

record ProjectInfo(
    Hrib Id,
    Hrib ProjectGroupId,
    LocalizedString Name,
    Hrib ArtifactId,
    ProjectReviewInfo[] Reviews,
    Permission GlobalPermissions,
    bool IsLocked,
);

public record ProjectReviewInfo(
    (NotReviewed | Accepted | Rejected) Kind,
    Hrib ReviewerAccountId,
    Hrib ReviewerRoleId,
    LocalizedString Comment,
    DateTimeOffset AddedOn
);

The FFFI-specific project metadata is gone. Solves A.1, A.2, and A.4.
Role HRIBs are used to identify the roles that need to review each project. They have, however, moved from blueprints directly to project groups. Solves A.5 and C.5.
Blueprint slots, surprisingly, still exist. A project group may define them in BlueprintSlots, where keys are the name of each slot. They may be ordered as they should be used to generate the UI necessary to create those artifacts. Solves A.6.
Blueprint slots are removed. Instead, each project creates exactly one artifact. The blueprint that artifact must fulfil is given by ProjectGroupInfo.ProjectBlueprintId.

Questions

What is the difference between an artifact and a shard? Why do we need both?

A shard, much like a shard of an ancient vase in a museum, is only a part of the whole. It represents a file uploaded by the user and stored by KAFE (along its potential variants). From the DB perspective, a shard represents file metadata. Compared to artifacts, shards have fixed structure given by the KAFE's implementation (and potential extensions).

How to model an artifact's authors? Should it be a property?

Yes, since films divide authors into Crew and Cast and games may or may not do that, and 3D models will likely have just a single array. In case of FFFI films, it should be two properties of KafeType core:author-reference[] called Crew and Cast.

How to keep the name of the project and its artifacts in sync? (related to #237)

Projects should optionally be able to force names upon their artifacts and keep them in sync.

Should a FFFI film one or multiple artifacts?

I don't know. I can image it being a single artifact or three: film, video-annotation, metadata and images.

Back-end implementation

This is a huge undertaking, best decompose it into several pieces, starting with changes that don't require modifying existing projects:

Edit: Tasks moved to the top. Scrolling got annoying.

Front-end implications

Hello, Jonáš.

Well, all of this needs quite a bit of front-end changes. These are just the ones I can think of:

UI for the creation and assignment of roles within an organization, so that there can actually be any reviewer roles.
Overhaul of the project detail and edit pages based on the ProjectGroupInfo.ProjectBlueprintId blueprint. I can imagine there being a tab for each ~~blueprint slot~~ artifact reference property.
The UI for uploading files can be simplified since, an artifact doesn't have to be created for almost each file. Would help with #238.
At least partial client-side implementation of the core blueprint requirements. This would allow fields that are required to be marked as such #240. The LanguagesRequirement above would help with #236.

Edited May 16, 2025 by Adam Štěpánek