Topic prioritization for emergent collective agenda setting across scaleably many comments

[colinmegill]

This PR adds endpoints and user interface for topic prioritization, which allows collective agenda setting and collective attention management in polis. The hierarchical topics are computed from comments in the delphi pipeline, and user selections are persisted (pending auth integration) to the DB. It will be fed to comment routing.

[colinmegill]

Foundations of group informed consensus over mathematically rigorous emergent topics for collective attention management, in lieu of paper for the moment :)

[colinmegill]

Taking some notes about the intention of this powerful direction... allowing highly expressive, highly detailed, affordances for participants to set their own, and conversation, priorities, is the only path to getting into the millions and tens of millions of comments per conversation, given the limited number of votes. It's possible that with increased relevancy polis will see more votes -- that's possible, and even likely, but it will still be, over time, proportional to submitted comments.

[colinmegill]

Allowing group informed consensus over ranked topics allows for densifying parts of the polis vote matrix, which allows for prioritizing certain reports.

Note that this is not importance, which is meant to operate along a different, complementary axis -- for those discussing "the arts" and "music", those reports may have 1000s of comments and will still benefit from importance and ranking for substance, far too fine grained than setting the agenda for a city of millions. This coarse graining bridges that gap in the math of scaling.

[colinmegill]

The question around comment routing, and bias, is of course foundational to the technology. We'll always want random comments, and new comments, and globally divisive comments, to get in front of users. But still, to scale to millions, there must be tradeoffs. Perhaps after tuning we'll find that 30% topics personalized to the user, 30% globally important, and 30% random with weights (existing algo) is the right ratio. But this is something that would be a good test for a paper using data gathered from this feature.

[colinmegill]

The topic names will change across runs, as will the umap. So, the only option is to project down into the comment space and we'll store three archetypal comments at the centroid. As more comments are added, there will be drift, so users can prioritize and re-prioritize as the conversation grows, when they see new comments. These "anchors" in the umap space will help comment routing stay relevant, and is durable for the lifetime of a years long conversation, especially since, as they are reprojected into the topic space later on, the user can remove that topic and drill back down into others.

[colinmegill]

Since what's being stored is comments, the right intuition is group informed consensus over zones of the vector embedding space, semantically, as "priority". This would be in contrast, to, say, managing attention for advertising.

[tevko]

Looks great overall, just a few default urls to remove and table scans to refactor. Are we at all concerned about pagination in dynamo for these new queries?

[colinmegill]

A reasonable "advanced mode" default for topic prioritization is an odd number of "points" to distribute, where the number is proportional to the topic hierarchy, and it costs more to vote more in the spirit of QV but with some adaptation to it being hierarchical and across rounds. Spiritually, that direction. It's a lot more clicking though, so, it's likely an advanced opt in mode when implemented, and not implemented yet.