Versioned links for tracking suspicious links

[danwos]

Versioned links concept

Goal

Find out linked objects of changed Sphinx-Needs objects between two timestamps.
Feature is often called suspicious links in some other tools.

Use case

Some existing requirements have changed between SW release A and B.
The test team needs to know which test cases needs to be rechecked, as the underlying requirement has changed.
Therefore for all changed requirements, their linked objects must be reported.

Requirements

• Two git commits (ids, tags, latest on branch) define the "time window".
• Found "suspicious objects" can be cleared/checked. Use case: Requirement got already anaylzed and no change on linked objects is needed.
• The output shall be a pdf/md/html report
• Solution must work on CI and locally.
• A rerun of the script with the same input data shall produce the same result.

Challenges

Two kind of suspicious links

There are two kind of suspicious links:

1. Parent object got changed with knowledge

In the first scenario, the user knows what he/she is doing. And can already decide if this change is changing something in the context or the statement of the parent object. So rephrasing the content is not a problem. In this scenario not all changed parent object lead to suspicious links.

Technical outlook:
This means, a change must be marked by the user, so that technically the object is flagged as "changed" and therefore checks for suspicious links must be performed.

2. Parent object got changed without knowledge

A change by accident, without knowing that something important had changed.
Important is, that not all changes on all options need to be checked.
For instance, a status change is ok.

Technical outlook:
This can be checked by comparing the same object in Version A and B for specified attributes.

Conclusion

Both scenarios need to be treated differently, as

1. Their result/outcome is different, and the one from 2) can be much bigger as the one from 1).

Clearing findings

When a suspicious object is found and corrected, this clearing of this finding must be documented somewhere. Otherwise, a rerun of the script will report it again.

Also the clearing should not be specific to the report, but to delta timestamps of its detection.

Example: Given release A, B, C, all created in a row.

Checking for suspicious links between B and C brings up one finding, which got resolved and cleared.

Checking now for A and C would bring up this link again, even as it got solved and checked for C.

Technical outlook:
To know if wanted changes (see scenario 1) from above) are resolved, this information must be part of the document / Sphinx-Needs object itself.
This allows us to detect suspicious links with the current release (C) only. No matter if we want to have the delta from A or B, it doesn't matter (for scenario 1!) ).

Technical outcome

We separate the solution between wanted and unwanted changes.

Wanted changes -> Versioned links

The user is responsible for marking a Sphinx-Needs object as changed.
The script does no checks on content or options.

Changes are marked as Version in the Sphinx-Needs object itself.
A link contains the version and gets "suspicious" if it doesn't match

.. req: A requirement
   :id: REQ_001
   :version: 2

   Some content

.. spec: A spec
   :id: SPEC_001
   :version: 1
   :solves: REQ_001:2        <-- Version added

   Version for link can be set by "LINK:VERSION"

.. spec: Another spec
   :id: SPEC_002
   :version: 3
   :solves: REQ_001:1        <-- Links to old version 

   Some content

.. spec: Another spec
   :id: SPEC_004
   :version: 1
   :related: REQ_001         <-- No version

   Links without version are still allowed

As alternative, the object version can be already part of the need-id:

.. req: A requirement
   :id: REQ_001:2   # instead :version: 2

Benefits

• Detection happens on current documentation only (One build)
• Clearing happens on current documentation only (No diff reports/clearing between different versions)
• Feedback to developer can be given during coding (IDE extension)
• Suspicious reporting can be part of the current Sphinx build
• Backwards compatible, as no version in links is still allowed
• Constraints can be defined, that links of type X must always have a specified version.

Drawback

• Maintaining the version and link-version is extra work

Unwanted changes

Out of scope for Sphinx-Needs, as this needs to happen outside a Sphinx build because access to data from different git commits/tags is needed.

[chrisjsewell]

Thanks @danwos I will have a look more in time

As alternative, the object version can be already part of the need-id

I just wanted to comment first, since I saw this presented yesterday:
this is an absolute no for me, I would not support this; having a single string encapsulate multiple pieces of information is always problematic to parse

[danwos]

I understand this point from the technical view, but I also have the user, who wants to avoid typing too much and feel comfortable using a function.
It's then
:id: MY_REQ:1.0
versus

:id: MY_REQ
:version: 1.0

And searching for MY_REQ:1.0 has the advantage of finding links and the need location.
One search to check the traceability/topicality of a versioned link.

You must agree, from the user's point of view it has some advantages.

[chrisjsewell]

You must agree, from the user's point of view it has some advantages.

I very much do not; I doubt all the projects that may contain : in their IDs will be happy that you irreversibly broke them (very similar to #1160)

[chrisjsewell]

Also, as I've said a million times now, we need a proper syntax for links, that encapsulates multiple data fields in a clearly understandable/parsable way; ID, part ID, version, external key, ...
Similar to a URI, it needs to have distinct delimiters that can also be escaped

(also cc@ubmarco)

[stefanheinbockelbmw]

Hello @danwos, are you still looking for input on this discussion?
From my side, your proposal sounds good. The title and version number are IMHO the most important parts to identify a requirement uniquely, and thus a syntax like :id: MY_REQ:1.0 should be very useful and usable.

[ubmarco]

I want to add some pieces of information to the discussion as the topic has a broader scope.

The current need IDs are arbitrary strings. They are only safeguarded by needs_id_regex which has a default of ^[A-Z0-9_]{5,}. The versioned links would fundamentally change this and it's a major backwards incompatible change that can require a lot of migration efforts. Can't speak for everyone, but some projects will be unhappy with this. I'm not against change, just want everyone to be aware of what these changes entail.

Let's paint the bigger picture.
These are the pieces of information relevant to each need id and (outgoing) link:

• id
• part
• revision
• namespace

id and part are already part of Sphinx-Needs while revision and namespace are new concepts.

For the revision, some questions arise:

• What is the datatype and structure of this
  • major / minor / patch
  • major / minor
  • major (positive integer)
  • Dynamic form where parts can be missing
• Shall the revision allow statements about (backwards) compatibility (e.g. patch updates are always compatible)?
• What does it mean if a revision is not given for an id or in a link?

As a general requirement, the revision specification must allow for a well-defined comparison of versions.
It's also smart to follow existing standards like SemVer or the Python PEP 440.

The new namespace information makes it possible to duplicate need IDs across projects or even
parts of projects.
Just like intersphinx defines a unique identifier for each project that is used in reference links, we can do the same for Sphinx-Needs.
Namespaces would be useful to refer to imported or external need IDs.
There is an open discussion around this.
For the namespace concept, one has to think about what to do with the id_prefix information in needimport and needs_external_needs.

Considering above as a starting point, the following questions come to mind:

• What is the structure of need ids 1. for their definition and 2. when they are linked to.
• What split character(s) are used to separate the different parts of the need id?
• Are split characters open for customization?
  • If yes, when importing/exporting needs, how are (customized) split characters handled?
• Is a link allowed to not specify the revision?
• Is a link allowed to specify only a part of the revision, e.g. 1.1 where 1.1.0 is the full revision?
• Are version specifiers allowed as in PEP 440 for outgoing links?
• How are link errors handled? Stop the build or just warnings? Are there different levels of errors?
• How are warnings/errors indicated in the output?
• Do we need a new directive to report on link issues?
• Assuming versioned links are introduced to model the impact of changes and mark links as suspicious:
  • How is the impact of changes determined?
  • What is the link direction of an impact (incoming, outgoing, both)?
  • Does the impact depend on the link type, the type of linked needs or even specific option values?
  • Sphinx-Needs essentially builds a (cyclic) property graph.
    This means a change can have a circular impact.
• What's the influence of the change on auto-generated IDs?
• For bigger projects, how are links automatically updated during impact analysis?
• How are versioned ids and links mapped to connected external data sources (ALM solutions) that have a different concept of versioning / base-lining / history?
• Sphinx-Needs in its current form is stateless. State is handled by version control (Git).
  Would customer goals and core problems also be solved by generating human readable diff reports for 2 Git revisions? Those reports could be added to PRs / merge requests.

Sorry for raising more questions and not providing any answers.
I think working this out requires the core team and core users sitting down and working through the details.

From customer perspective it would be useful to get statements around the exact goals and core problems that need to be solved.
I tend to overthink problems at times, maybe someone can convince me that everything is much simpler.

[David-Le-Nir]

Just to add some user feedback, I'll share ur way of doing it. We are also dealing with suspicious links which is a critical feature for us. If it could be integrated in SN, it would be awesome.

We don't want to manage versioning at need's level because of the extra maintenance required (however, users used to do it in the former tooling and it requires a mind shift for them, which is not always easy)

Instead we internally compute a version based on need's content (a basic sha1 on a , configurable, list of fields). Each time a new version of the document is issued, the current sha1 is compared to the previous one (we have an external reference to the previous version). When changed, a field stores the version in which the need changed. This allow us to detect the list of changed items, since a given version.

Then if a need refers to a "suspicious" upward item a warning is raised for the need (displayed as an extra option).

This "detection" part works well but I admit it requires some setup, especially to the configuration of a "previous reference" for each document, allowing to detect the changes. This is setup only once because production versions are published on the stable production URL, making the detection automatic as soon as a new version is published.

The difficulty for us is more on the "resolution" phase, i.e. how a user mark the suspicious links as reviewed (i.e. either nothing to do, or adapted the child need) to make the warning disappear until the next change.

For the moment we produce a "suspicious file", listing all suspicious parents from all upward references. This file is produced locally during the build and user edit it to manually marks element as reviewed. The file is pushed in Git so it can be shared to the team and used for the document lifecycle. Each time an upward reference is changed, the related section in the suspicious file is rewritten.

The main pain point is to have user not forgetting to build their document locally and marking element as reviewed. We have some CI linter to fail-fast if not the case.

This is far from perfect but it does the trick since multiple years for us and we did not find a better solution. Maybe if it was more integrated in SN it would be easier.

Maybe also you could suggest some ideas leading to some improvements on our side.

[ubmarco]

Will read and reply to comment a bit later, but want to leave a link to the edge properties discussion here.