db-analyser: add DumpStakeDistributions pass

[nfrisby]

A new pass in db-analyser and a bash/awk script for post-processing its output.

[nfrisby]

Blue line: This graph shows that the 551 pools that have been in the top 90% of every epoch from 506 to 543 (ie 190 days) have controlled more than 88% of stake in every epoch.

Red line: Even if all of those pools' individual least-stake epochs were instead simultaneous, that lower bound is 81.77% cumulative stake. I think that suggests the stake is quite stable within these 551 pools over the past 190 days.

[amesgen]

AFAIK pdTotalActiveStake and individualTotalPoolStake are only used for Conway voting stuff, so individualTotalPoolStake / pdTotalActiveStake does not necessarily equal individualPoolStake, which is used for leader election:

ouroboros-consensus/ouroboros-consensus-protocol/src/ouroboros-consensus-protocol/Ouroboros/Consensus/Protocol/Praos.hs

Line 521 in 43c1fef

maybe 0 SL.individualPoolStake $

and the ledger peer info we give to Network

ouroboros-consensus/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/PeerSelection.hs

Line 48 in 43c1fef

. map (second (PoolStake . SL.individualPoolStake))

So I think it makes sense to either ignore them here or make this explicit in the output somehow.

I looked at all epochs until 544, and so far, we apparently didn't have a difference here, no idea how likely it is for a difference to arise in the future.

See IntersectMBO/cardano-ledger#4324 (comment), cc @lehins for confirmation.

[lehins]

AFAIK pdTotalActiveStake and individualTotalPoolStake are only used for Conway voting stuff

That statement is not quite correct.

IndividualPoolStake that is extracted from PoolDistr as it is done in consensus, will not have any voting related stake within it. However, the version of this in DRep pulser will have proposal deposits added to that stake. The issue was that the haddock for individualTotalPoolStake should not have stated anything about proposal deposits.

I looked at all epochs until 544, and so far, we apparently didn't have a difference here, no idea how likely it is for a difference to arise in the future.

So, individualTotalPoolStake / pdTotalActiveStake should always equal individualPoolStake for leader election, unless you are looking into the version from pulser.

[amesgen]

Thanks, that is good to hear! I guess then the remaining mentions of proposal deposits here and here should also be removed?

[amesgen]

Idea for another thing to plot: the statistical distance/total variation distance of the stake distributions over time (potentially eliding the top 90%). This is what the researchers use in the Praos paper (Definition 15, Theorem 11) for their notion of stake shift, as well as in their later paper "Stake Shift in Major Cryptocurrencies: An Empirical Study".

Concretely, the stake shift $\sigma$ between two stake distributions measures the largest absolute difference between the combined stake of a fixed set of pools relative to the two distributions. So when we sample pools for Genesis with total stake $s$ from a somewhat old stake distribution from a snapshot, then these pools still have at least $s-\sigma$ stake in today's stake distribution, where $\sigma$ is the stake shift between the old and today's stake distribution.

[nfrisby]

Great, thank you. I'll compute ${\displaystyle \Delta (X,Y)={\frac {1}{2}}\sum _{\alpha \in D}|\Pr[X=\alpha ]-\Pr[Y=\alpha ]|}$ for the final stake distribution versus each individual past stake distribution.

[nfrisby]

^^^ Done, as of my latest commit.

This is the result

Blue line is all pools that were in all epochs. Red line is pools that were in the top 90% of all epochs.

This is the cumulative stake of those two sets of pools per epoch: