agd throws panic: counter cannot decrease in value
#11062
Comments
michaelfig
changed the title
agd with telemetry throws panic: counter cannot decrease in valueagd throws panic: counter cannot decrease in value
siarhei-agoric
added a commit
that referenced
this issue
Feb 27, 2025
mergify bot
pushed a commit
that referenced
this issue
Feb 27, 2025
… and 'size_decrease' (#11063) refs: #11062 refs: #10938 ## Description [Despite testing](#10997 (comment)), `telemetry.IncrCounterWithLabels()` appears to only accept non-negative numbers. As a workaround, we will be reporting vstorage size increase and decrease as two separate counters. ### Security Considerations N/A ### Scaling Considerations One extra metric not an issue as long as number of keeper instances is low. ### Documentation Considerations New metrics: ``` # HELP store_size_increase store_size_increase # TYPE store_size_increase counter store_size_increase{storeKey="vstorage"} 42386 # HELP store_size_decrease store_size_decrease # TYPE store_size_decrease counter store_size_decrease{storeKey="vstorage"} 335 ``` ### Testing Considerations Manual Testing; Automated testing will be submitted as a separate PR. First terminal: ``` cd agoric-sdk/packages/cosmic-swingset make scenario2-setup scenario2-run-chain ``` Second terminal (after Block 17) ``` cd agoric-sdk/packages/cosmic-swingset make fund-provision-pool make provision-acct ACCT_ADDR=$(cat t1/8000/ag-cosmos-helper-address) ``` Third terminal (start before Block 17): ``` cd agoric-sdk/packages/cosmic-swingset while true; do curl -s -G 'http://localhost:26660/metrics' | grep 'store_size_'; sleep 10; done [...] # HELP store_size_increase store_size_increase # TYPE store_size_increase counter store_size_increase{storeKey="vstorage"} 42386 # HELP store_size_decrease store_size_decrease # TYPE store_size_decrease counter store_size_decrease{storeKey="vstorage"} 335 # HELP store_size_increase store_size_increase # TYPE store_size_increase counter store_size_increase{storeKey="vstorage"} 284 # HELP store_size_decrease store_size_decrease # TYPE store_size_decrease counter store_size_decrease{storeKey="vstorage"} 335 # HELP store_size_increase store_size_increase # TYPE store_size_increase counter store_size_increase{storeKey="vstorage"} 284 # HELP store_size_decrease store_size_decrease # TYPE store_size_decrease counter store_size_decrease{storeKey="vstorage"} 788 ``` ### Upgrade Considerations N/A
siarhei-agoric
added a commit
that referenced
this issue
Feb 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
If
agdis configured with telemetry enabled, and runs a fairly typical workload (e.g. with a chainStorage set or delete that decrease the size of the stored key), it panics the Golang side, causing an exception on the JS side.To Reproduce
Steps to reproduce the behavior:
git checkout mfig-devnet-config-steps(the PR fix(vm-config):vats/init-core.jsrelies on econCommittee #11056 branch)cd agoric-sdk/packages/cosmic-swingsetmake scenario2-setup scenario2-run-chainto start a new chain with Golang telemetry enabledmake fund-provision-pool(which triggers the SwingSet provisionPool vat to interact with the Cosmos bridge)make provision-acct ACCT_ADDR=$(cat t1/8000/ag-cosmos-helper-address)panic: counter cannot decrease in valueerrors logged by the running chain, as below2025-02-27T03:09:04.893Z SwingSet: ls: v10: Error#1: panic: counter cannot decrease in value
Expected behavior
No panics.
Platform Environment
vats/init-core.jsrelies on econCommittee #11056 , currentlygit describe --tags --alwaysyields fast-usdc-beta-1-137-gef30b2a883Additional context
#10997 introduced an idiom that looks like:
sizeDeltamay very well be negative, which is invalid fortelemetry.IncrCounter*. For the metric to go up and down over time, usetelemetry.SetGauge*instead. More detail than only the current number may need to be implemented with multiple Counters and/or Gauges using different metric keys.The text was updated successfully, but these errors were encountered: