Add support for core platform primatives to enable better data management

[jkuester]

Is your feature request related to a problem? Please describe.

Spinning this out of this PR conversation. @fardarter has shared this in the context of trying to deal with duplicate contact data:

Additional data management features:

Caveat: I don't know the UI very well so take these as loose ideas from a UI perspective.

• Unset is_canonical. This can't be on the edit page without adding complexity as the above flow would simply reset it. Semantically, if we added it to the create/edit flow this should be something like "Create/mark as draft", which is largely unsupported. So the reset would need to take place as a data management operation somewhere on the UI, maybe close to delete.
• Mark for deletion: This should only appear if the user does not have delete permissions, otherwise it should just be the delete button.

Editing an existing record → Display the current is_canonical status (if any) and allow the user to reset it. Additionally, provide a "Mark for Deletion" option for records that need cleanup (particularly useful for users without delete permissions). From a data management perspective, in general, I think we need to see something not marked as is_canonical as being in an uncertain/undecided state rather being deleteable, so resetting the is_canonical field just move it from a "we're sure" to a "we're unsure", where mark for deletion is a "we're sure this was wrong".

For me, these are useful core platform primitives that can be evolved. Data accuracy is critical to the success of products like CHT (at least for us) and my view is that effective delegated tooling for data management would be a major contributor to success.

I think it can be really hard to correctly imagine the proper format for fundamental data structures unless you have a clear picture of the problems you are trying to solve. @fardarter could you maybe expand a bit on the data problems you are having?

[fardarter]

Thanks @jkuester. The comment was a little throwaway and was mainly arising out of my view that data quality guards should, where possible, be a platform feature rather than something implemented through form field logic.

But it looks like we've reached a different conclusion in the thread to the one I've proposed here (#9609 (comment)), but the one example I can offer is the one in the linked comment, which is that for users that don't have a delete right (which we needed and which it seems like others might need/want), it seems valuable to have the option of "Mark for deletion" on the UI. This allows CHWs or other app users to signal something about the data. Another example may be something like "Suspected duplicate", which would indicate that an investigation/follow up is required.

These could either be dealt with by a backend data team or go to a supervisor for action.

Providing ways for users on the ground to signal information about data quality to users in a position to make decisions about that data, or to the data teams, would help considerably with the overall burden of maintaining data quality.

@ChinHairSaintClair You may have more insight than I do into what data management tools would really benefit us.

[jkuester]

for users that don't have a delete right (which we needed and which it seems like others might need/want), it seems valuable to have the option of "Mark for deletion" on the UI. This allows CHWs or other app users to signal something about the data. Another example may be something like "Suspected duplicate", which would indicate that an investigation/follow up is required.

Got it. Perfect! This is just the kind of thing I was hoping we could dig into here! As I noted in the other thread, I am interested in getting your thoughts regarding using the muting functionality in this case.

The big thing to me that seems missing from the existing muting functionality is that there is no clean way to specify a cause/status for why a contact was muted. (E.g. was the contact muted because they were added by mistake and should be deleted, are they a duplicate and should actually be merged into a different contact, or did they just leave the area for awhile and the CHW should not receive notifications/tasks about them....). The context behind a muting could possibly be inferred by looking at the reports for the contact, but that is not a complete solution since a contact can be muted because one of the contacts above them in the hierarchy was muted. Processes like the dupe-checking workflow do not have time to spin through reports and ancestors to figure out if a contact was marked to be deleted or not!

One of the reasons I am approaching this "Mark for deletion" workflow from the muting perspective is because I think it helps to illustrate the level of complexity involved here. If a CHW marks a contact as deleted should the UI/UX for that contact change for the user (like what happens when muting a contact)? Does this need to work offline? Also, when marking a contact to be deleted, there is a huge question about what should happen to that contact's reports and child contacts. Things get even more interesting if we are marking a contact for deletion because it is a duplicate of a different contact. In that case, should we move all the reports to the other contact? What about the child contacts?

Recently functionality was added to cht-conf to support properly merging contacts and deleting contacts. But, those require manual intervention from a sys-admin to run.....