Mapping Rules Versioning

[gordonmilligan-uod]

It would be good to be able to mark versions of a mapping as complete and have them saved. We can then fork existing mapping for maintenance, so we can safely apply updates to OMOP CDM or updated vocabs to existing mappings with less risk.

Being able to do compare different mapping versions would be nice but would likely be classed as a later feature.

[AndyRae]

I've moved this to a discussion - as it needs wider thought, design, and planning.
I have some initial ideas around this which I will share as well.

[AndyRae]

Versioning is “packaging” a set of rules; it is creating a snapshot at a moment in time.

Goals

• Allow users to create versioned artefacts of rulesets
• Provide underlying versioning to enable advanced features (below)

Versioning unlocks several capabilities:

• Hub/Spoke Carrot ( Hub and spoke carrot-mapper #844 )
• Publishing rules / artefacts.
• Rule provenance.
• Progress / diffing rulesets.
• Ruleset licensing

Creating a version would lock that set of rules to it. We would encourage semantic versioning, but leave it to the user to name a version.

But - Carrot should not be initially responsible for implementing version control, as in branching, forking, and merging rulesets. We can explore these possibilities in hub/spoke model, or for example through integrations with Github, and finally possibly implementing it directly.

I think it would be useful to align versioning with the standardisation (#848) and in particular with SSSOM Rule sets.

[AndyRae]

Quick user stories:

Story 1:
A user creates a rule. They pin this version with a schema name (e.g., 0.0.1). When a new rule is added, it becomes the "latest" version.

Story 2:
A user wants to view a specific version of rules, including all related past versions. They query the API to retrieve the selected version and its history.

Story 3:
A user deletes a rule from the latest version, but it remains in past versions. The rule is soft-deleted, and the version where the deletion occurred is stored.

Story 4:
A user retrieves the latest version of all rules, filtering out any that have been soft-deleted.

[AndyRae]

Deletion of mappings in versions

This should be acknowledged upfront - for a mapping to be deleted in a version, it needs to be "soft deleted", and hidden. We could enable this by adding a hidden field on the entity, which would be foreign key to the version it was deleted in.

This enables quick filtering on deleted fields, and displaying them when necessary.

[AndyRae]

This would mean something like:

# MappingRulesSet
id
parent (id to the previous version)
...

# ScanReportConcept
`hidden`: `mapping_rules_set_id`
...

Filtering:

Then if a user selects a version, we can retrieve the IDs of the relevant versions (including parents) based on it.

For example:

Id	Version	Name	Parent
1	First	0.0.1
2	Second	0.0.2	1
3	Latest	Latest	2

If a user selects to view 0.0.2, it would return [1, 2].

When querying mapping rules/concepts, we filter based on those IDs. We also filter to show hidden items if the version ID is not present.

This would also set us up for forking etc.