This repository has been archived by the owner on Mar 26, 2024. It is now read-only.
Monitor Spring Boot/Tomcat Thread Utilization #45
Assignees
Labels
Comments
iliecirciumaru
added a commit
that referenced
this issue
Mar 15, 2022
Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen
PaulWen
pushed a commit
that referenced
this issue
Jan 26, 2023
Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen
geahaad
pushed a commit
that referenced
this issue
Jan 30, 2023
Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen
PaulWen
added a commit
that referenced
this issue
Jan 30, 2023
* Create LICENSE Signed-off-by: Gerhard Dinhof [email protected] * docker-compose setup to start Issuer-Verifier and Holder AcaPys as well as Postgres DBs * Clean up docker-compose setup * Add postman based load generator for testing if the local setup works as expected * choreinitialized kotlin spring boot project * chore: added copyright * added maven to git ignore * docs: removed generated help md * chore!: updated to jdk 17 * docs: added copyright * feat: added boilerplate apis * feat: added Dockerfile * docs: added README * Add submodule + script for VON-network * restructuring the files * Automatically register public DID on the ledger * code cleanup * build: Switch to Java 11 as the built target for brought compatibility * feat: Integration of an automatically generated SwaggerUI * Update README.md * Update README.md * feat(AcaPy Integration): Integration of an AcaPy Java client library for calling the AcaPy Admin API as well as receiving Webhook calls from AcaPy * Working state with permission error in tails * tails server working * added extended logging capabilities * removed gitignore * remove ignored files * chor(Postman): Restructure files * chor(.idea): Update .idea files * chor(Setup): Let user know that it will take 30 seconds to register DID * chor(Setup): Automatically load and update git submodules * build(Java Version): Switch to Java Version 11 * Added README for the setup * Fixed typo in environment files * chore: Rename WebhookController to AcaPyWebhookController and LoadGeneratorController to InfoController * refactor: Structure of controllers and Swagger UI * feat: Integrate simple webhook logging * feat(TestRunner): application.yml is used to choose and configure the test case * feat(FullProcessRunner): Write schema and credential definition to ledger during setup * feat(FullProcessRunner): Connect to Issuer/Verifier as well as Holder AcaPy * feat(FullProcessRunner): Establish connection between Issuer/Verifier and Holder * build(Restart Policy): Restart each container always * feat(FullProcessRunner): Issue credential to holder and request credential from holder via proof request * refactor(Spring Configs): Move all Spring Config files into a config package * refactor(Aries Client): Build a wrapper around the AcaPyAriesClient library to make it easier replaceable by a different AriesClient library 1. to make it easier to time the HTTP requests the AriesClient 2. to potentially be able to use this load generators to test other agents too * refactor(Logger): Move logger into their own package * refactor(Aries Event Handler): Build a wrapper around the AcaPyEventHandler to make it easier replaceable by a different AriesClient library To potentially be able to use this load generators to test other agents too. * refactor(AriesClient and AriesObserver): Move service interfaces into the domain layer * chore: Renaming of package * chore: Clean up SwaggerConfig * feat(Readme): Update the Readmes * feat(Wallet Logger): Log the size of the issuer/holder wallet database periodically * refactor(Runner): Move termination condition into own method * feat(logging): added Loki as log collector * feat(Log HTTP Requests): Log the durations and Status Codes of HTTP requests going out to the AcaPy * chore(Renaming): Rename AriesEventLogger types * chore: Remove unused volume from docker-compose * chore: Clean WalletLogger log message * refactor(Agent Implementations): Move all AcaPy agent specific code into one folder * chore: Adopt manage.sh logging * chore: Adopt manage.sh logging * feat(AcaPy Http Request Timeout): Make it possible to configure the timeout for HTTP requests to the AcaPy clients * chore(Name Containers): Name the Grafana and Loki container * chore(Logging): Remove unused Logback configuration * chore(Logging): Adopt Logback output format * feat(Provision Grafana): Add Loki as the default datasource to Grafana and deactivate authentication for Grafana * fix(Wallet Logger): remove comma from log message * feat(Dashboard): Configured a default Grafana dashboard that is automatically provisioned on startup of Grafana * feat(Dashboard): Add overviews about Connections, Credentials, and Proof Requests to the dashboard * feat(Dashboard): Visualize the HTTP Response Codes and HTTP Response Times in Grafana * feat(Dashboard): Visualize the rate in which connections are being activated, credentials are being issued, and proof requests are being verified in Grafana * fix(AcaPy Event Publisher): Fix string date to milliseconds conversion * fix(Loki Config): Add custom Loki config to avoid the "Too many outstanding requests." issue * feat(Dashboard): Add total number of connection, credential issuance, and proof request count * refactor(Dashboard): Rename the keys that are being logged * chore(.env): Remove .env file from git * chore(Dashboard): Update HTTP Response Times metric * chore(Dashboard): Fix labeling rate is not per Second but per Minute * chore(Dashboard): Provide the wallet DB size in MB instead of bytes * chore(Dashboard): Outline the number of test iterations started per minute on the dashboard to visualize the load the load generator puts on the Agents * feat(FullProcessRunner): Add a new test runner to start new full process iterations on a constant "iteration per minute" base * chore(Dashboard): Reorder dashboard panels * feat(Export PDF Report): Grafana dashboards can now be exported as a PDF * fix(Dashboard Deployment): Put dashboard and logging containers into a separate docker network to avoid error when shutting everything down * refactor(Full Process Runner): Avoid code duplication across the FullProcessRunners * feat(Full Process Runner): Schedule FullProcessConstantLoadRunner at a fixed rate leveraging a thread pool * refactor(Aries Agent): Extract connectionId from ProofRequestDo and CredentialDo in preparation for connectionless proof requests * feat(Connectionless Proof Requests): Started implementing support for connectionless proof requests - not finished yet * feat(Scale AcaPy using Nginx): Scale AcaPy horizontally by using a Nginx container as a load balancer * feat(Scale AcaPy using Nginx): Scale Issuer/Verifier as well as Holder AcaPy to 10 instances each by using a Nginx container as a load balancer * feat(AcaPy Wallet Type): Make AcaPy Wallet Type configurable via the .env file * fix(.env): Fix .env file * feat(Handle Exception on Connection Invitation Receive): Start a new iteration if the holder is not able to accept the connection invitation * feat(Dashbaord): Show number of failes "POST/connections/receive-invitation" * chore(Holder AcaPy): Increase number of Holder AcaPys to 20 * chore(Dashboard Setup): Define the containers Grafana depends on * chore(Dashboard Setup): Define the containers Grafana depends on * chore(AcaPy): Increase default AcaPy Version * fix: Circular dependency in dashboard/logging deployment * refactor(TestRunners and TestFlows): Differentiate between TestRunners and TestFlows to make it easier to implement differen TestFlows while reusing the existing TestRunners * chore(Readme and Dashboard): Adopt the Readme and Dashboard to meet the new setup * feat(Issuer Flow): Added a new test flow to simulate an issuer who is establishing connections to holders and issues credentials to them afterwards * chore(Application Config): Add new application configuration settings and rename existing ones * chore(Application Config): Add new application configuration settings and rename existing ones * chore: Readme * feat(Revocable Credentials): Configure if a credential should be revocable, the size of the revocation registry, and if a proof request should include a non revoked check * chore(OOB Credential Offer): Preparations to support OOB Credential Offers * feat(Wallet DB Connections): Configure the number of connections to the Wallet DBs * feat(Connection Flow): Test Flow that only established connections with the holder * feat(Credential Issuance Flow): Test Flow that only issues credentials to the holder * feat(Proof Request Flow): Test Flow that only requests proofs from the holder * feat(Increasing Load Test Runner): Added a new test runner that increases the peak load in predefined steps * fix: Count of iterations in the Increasing-Load-Test-Runner * feat(AcaPy): Make Storage Type used by AcaPys configurable * feat(ErrorLogger): Introduction of an error logger to log unexpected error that occurred during testing * feat: Start Load Generator as Docker Container using "./manage.sh start" - to not start it via docker use "./manage.sh startWithoutLoadGenerator" * fix: .env.example * feat(Distribute load Across many Holders): Instead of having one horizontally scaled holder agent we now start many independent holder agents and the load generator takes care of distributing the load across all the holders * feat: Increase number of stand-alone holders to 10 * feat: Increase number of issuer/verifier AcaPys to 10 * feat: Add known error to the dashboard * feat: Rework manage.sh to ensure everything starts in the correct order * feat: Offer a "restart" command via the manage.sh * fix: AcaPy provisioning * fix: AcaPy provisioning * docs: Update Readme * Prometheus set-up (#43) * #32 Initial setup (macOS only) * #32 adding cadvisor support to be docker and platform agnostic * #32 Adding node-export * #32 Adding prometheus data source provisioning config * #32 Adding prometheus / node-exporter dashboard * Remove unused redis container * Increase platform agnostic-ness * fix: remove "redis" dependency * Removed unused prometheus scraper * Added Prometheus and installation guide for node-exporter Co-authored-by: PaulWen <[email protected]> Signed-off-by: Gerhard Dinhof [email protected] * fix(manage.sh): Rebuild load-generator docker image on every start to ensure that the latest application.yaml is loaded * feat(Docker Container Metrics): Visualize the docker container metrics in Grafana that have been collected by cAdvisor * feat(Linux Localhost): Avoid having to switch between "host.docker.internal" and "172.17.0.1" depending on the host OS * feat(Postgres Metrics): Collect Postgres metrics using postgres_exporter and visualize them in Grafana * Fixed cAdvisor volume mapping for macOS #44 Signed-off-by: Gerhard Dinhof [email protected] * fix: configure postgres username and password via env-variables * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * chore(Dashboards): Improve description of graphics * chore(Dashboards): Add a download link to each dashboard * chore(Dashboards): Update test result dashboard * fix(Exception Handling): Catch and log exceptions thrown in the context of ScheduledThreadPools * refactor(Error Logging): Rework the error logging to differentiate between errors and provide more information for easier debugging * feat(Summary Dashboard): Introduce a new dashboard which lists the most interesting graphs in one single dashboard * fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included * chore: Specify ID for Grafana folder to ensure a static URL * fix: Improve error logging for failed HTTP requests to AcaPy * fix(Test Results Dashboard): Improve the error logging on the test results dashboard * feat(Test Results Dashboard): Log started iterations as well as pending and failed iterations * fix(Constant and Increasing Load Runners): Previously, the scheduler used to generate the load was not running more than one task at a time. This caused the load generator not to generate the desired iterations per minute. * chore: Fix wording on test result dashboard * chore: Update default Revocation Registry Size to 3000 * refactor: Combine test result and summary dashboards into one dashboard * chore: Add CPU and memory usage of all containers to the test result dashboard * chore: Adjust tooltips * fix: Increasing Load Test Runner * chore: Improve test result dashboard * chore: Improve test result dashboard * Revert "fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included" This reverts commit cfe7f21. * chore: Improve test result dashboard * chore: Restructure the manage.sh * fix: Cancel schedulers properly once the tests are over * chore: Update test results * docs: restructured and added architecture diagram * chore: deleted unneeded readme * docs: added nginx and small cleanups * Implement Postgres Cluster as wallet DB (#49) Issue: #48 Author: Ilie Circiumaru Reviewer: @PaulWen Description: Implement 3 instances Postgres Cluster for wallet DB for the issuer-verifier. Split docker-compose on more files to enable launch of the isolated components * chore: Update Test-Results Dashboard * chore: Remove unused grafana-pdf-exporter reference * refactor: Cleanup the setup procedure * fix: Gitmodules * fix: Gitmodules * fix: Gitmodules * fix: nginx-issuer-verifier.conf * fix: manage.sh * chore: remove unused profiles from docker-compose-agents.yml * feat(Activate Postgres Cluster via .env): Switch from many start commands to just one and configure what is started via the .env * feat(Configure Number of Issuer/Verifier Agents): Depending on the number of requested issuer/verifier AcaPy instances a different nginx.conf is loaded that matches the desired number * feat(Configure Number of Holder Agents): The number of requested holder AcaPy instances can be changed via the .env * chore: restructure .env * feat(Log Revocation Registry Metadata): Track the usage of revocation registries. In particular look out if any RevReg/Index combination has been assigned to multiple credentials. * Implement declarative deployment configuration (#58) Issue: #56 Co-authored-by: Ilie Circiumaru <[email protected]> Co-authored-by: PaulWen <[email protected]> Description: 1. Implementation of declarative deployment configuration 2. Setup metrics for Postgres Cluster * fix(Revocation Registry Logging): Do not log any revocation registry metadata if not revocation registry is used * feat(Postgres Configuration): make the "statement_timeout" configurable * fix: network handling with docker-compose * feat(Postgres configuration): Configure "idle_in_transaction_session" and "lock_timeout" via the .env * fix(Postgres configuration): "idle_in_transaction_session_timeout" * fix: Test Results Dashboard * chore: Update Test-Results Dashboard * feat(Multitenancy): Run tests with AcaPy in multitenancy mode * chore: remove unnecessary host.docker.internal * docs: Split Readme into Developer and User Readme to avoid any confusion * docs: Improve the README.md * docs: Improve the README.md * fix: add host.docker.internal reference to load-generator so that he is able to connect to the VON Network to register a public DID * chore: Update test-results.json * docs: define test cases to verify the load generator still behaves as expected * docs: Add a troubleshooting section to the README.md * Implement monitoring of the Spring Boot app (#62) Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen * docs: Update Troubleshooting section * docs: Fix spelling * Added configuration option to enable Postgres debug logging (#60) * Added flags to enable postgres deub logging * Removing postgres debug settings from clustered setup * added comprehensive description for POSTGRES_LOG_DEBUG * alphabetical order Signed-off-by: Gerhard Dinhof [email protected] * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * #63 debug acapy (#64) feat(Debug AcaPy): Running "manage.sh debug" it is now possible to attach a debugger to the issuer-verifier-acapy. * Feat 59: credential revocation (#65) feat(Full Flow): Support for credential revocation with configurable revocation batch size * Fix 66 macos compsev2 compatibility (#67) fix: Switches Postgres from port 5000 to port 5432 as MacOS Monterey already uses the post 5000 itself and extends the README to outline that Compose V2 is not supported. * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * fix(Debugging): Load generator needs to be started manually after the debugger has been attached to the issuer-verifier-acapy. * docs: Improve the troubleshooting section * fix(Database Logger): Fix the configuration of the database exporter to the metrics of the wallet DB * chore(Multitenancy): Be able to run load tests with 9 tenants * feat(Problem Reports): Log problem reports and show them on the dashboard * chore(README): Add list of required dependencies * feat(Dashboards): Update the dashboards and add a dashboard to evaluate AcaPy logs * fix(Aries Client Logging): Ensure that aries client errors can be parsed by Loki * feat(manage.sh): Wait for the ledger to start until the DID was registered successfully to the ledger * feat(Apple M1 Chip): Update the dockerfile to be able to build the Load Generator Docker image on an Apple M1 Mac * feat(.env): Update the default .env values * chore(OOB): Remove out-of-band (OOB) placeholder code --------- Signed-off-by: Gerhard Dinhof [email protected] Co-authored-by: PaulWen <[email protected]> Co-authored-by: Richard Holzeis <[email protected]> Co-authored-by: Lukas Willburger <[email protected]> Co-authored-by: Paul Wenzel <[email protected]> Co-authored-by: iliecirciumaru <[email protected]>
PaulWen
added a commit
that referenced
this issue
Jan 30, 2023
* Create LICENSE Signed-off-by: Gerhard Dinhof [email protected] * docker-compose setup to start Issuer-Verifier and Holder AcaPys as well as Postgres DBs * Clean up docker-compose setup * Add postman based load generator for testing if the local setup works as expected * choreinitialized kotlin spring boot project * chore: added copyright * added maven to git ignore * docs: removed generated help md * chore!: updated to jdk 17 * docs: added copyright * feat: added boilerplate apis * feat: added Dockerfile * docs: added README * Add submodule + script for VON-network * restructuring the files * Automatically register public DID on the ledger * code cleanup * build: Switch to Java 11 as the built target for brought compatibility * feat: Integration of an automatically generated SwaggerUI * Update README.md * Update README.md * feat(AcaPy Integration): Integration of an AcaPy Java client library for calling the AcaPy Admin API as well as receiving Webhook calls from AcaPy * Working state with permission error in tails * tails server working * added extended logging capabilities * removed gitignore * remove ignored files * chor(Postman): Restructure files * chor(.idea): Update .idea files * chor(Setup): Let user know that it will take 30 seconds to register DID * chor(Setup): Automatically load and update git submodules * build(Java Version): Switch to Java Version 11 * Added README for the setup * Fixed typo in environment files * chore: Rename WebhookController to AcaPyWebhookController and LoadGeneratorController to InfoController * refactor: Structure of controllers and Swagger UI * feat: Integrate simple webhook logging * feat(TestRunner): application.yml is used to choose and configure the test case * feat(FullProcessRunner): Write schema and credential definition to ledger during setup * feat(FullProcessRunner): Connect to Issuer/Verifier as well as Holder AcaPy * feat(FullProcessRunner): Establish connection between Issuer/Verifier and Holder * build(Restart Policy): Restart each container always * feat(FullProcessRunner): Issue credential to holder and request credential from holder via proof request * refactor(Spring Configs): Move all Spring Config files into a config package * refactor(Aries Client): Build a wrapper around the AcaPyAriesClient library to make it easier replaceable by a different AriesClient library 1. to make it easier to time the HTTP requests the AriesClient 2. to potentially be able to use this load generators to test other agents too * refactor(Logger): Move logger into their own package * refactor(Aries Event Handler): Build a wrapper around the AcaPyEventHandler to make it easier replaceable by a different AriesClient library To potentially be able to use this load generators to test other agents too. * refactor(AriesClient and AriesObserver): Move service interfaces into the domain layer * chore: Renaming of package * chore: Clean up SwaggerConfig * feat(Readme): Update the Readmes * feat(Wallet Logger): Log the size of the issuer/holder wallet database periodically * refactor(Runner): Move termination condition into own method * feat(logging): added Loki as log collector * feat(Log HTTP Requests): Log the durations and Status Codes of HTTP requests going out to the AcaPy * chore(Renaming): Rename AriesEventLogger types * chore: Remove unused volume from docker-compose * chore: Clean WalletLogger log message * refactor(Agent Implementations): Move all AcaPy agent specific code into one folder * chore: Adopt manage.sh logging * chore: Adopt manage.sh logging * feat(AcaPy Http Request Timeout): Make it possible to configure the timeout for HTTP requests to the AcaPy clients * chore(Name Containers): Name the Grafana and Loki container * chore(Logging): Remove unused Logback configuration * chore(Logging): Adopt Logback output format * feat(Provision Grafana): Add Loki as the default datasource to Grafana and deactivate authentication for Grafana * fix(Wallet Logger): remove comma from log message * feat(Dashboard): Configured a default Grafana dashboard that is automatically provisioned on startup of Grafana * feat(Dashboard): Add overviews about Connections, Credentials, and Proof Requests to the dashboard * feat(Dashboard): Visualize the HTTP Response Codes and HTTP Response Times in Grafana * feat(Dashboard): Visualize the rate in which connections are being activated, credentials are being issued, and proof requests are being verified in Grafana * fix(AcaPy Event Publisher): Fix string date to milliseconds conversion * fix(Loki Config): Add custom Loki config to avoid the "Too many outstanding requests." issue * feat(Dashboard): Add total number of connection, credential issuance, and proof request count * refactor(Dashboard): Rename the keys that are being logged * chore(.env): Remove .env file from git * chore(Dashboard): Update HTTP Response Times metric * chore(Dashboard): Fix labeling rate is not per Second but per Minute * chore(Dashboard): Provide the wallet DB size in MB instead of bytes * chore(Dashboard): Outline the number of test iterations started per minute on the dashboard to visualize the load the load generator puts on the Agents * feat(FullProcessRunner): Add a new test runner to start new full process iterations on a constant "iteration per minute" base * chore(Dashboard): Reorder dashboard panels * feat(Export PDF Report): Grafana dashboards can now be exported as a PDF * fix(Dashboard Deployment): Put dashboard and logging containers into a separate docker network to avoid error when shutting everything down * refactor(Full Process Runner): Avoid code duplication across the FullProcessRunners * feat(Full Process Runner): Schedule FullProcessConstantLoadRunner at a fixed rate leveraging a thread pool * refactor(Aries Agent): Extract connectionId from ProofRequestDo and CredentialDo in preparation for connectionless proof requests * feat(Connectionless Proof Requests): Started implementing support for connectionless proof requests - not finished yet * feat(Scale AcaPy using Nginx): Scale AcaPy horizontally by using a Nginx container as a load balancer * feat(Scale AcaPy using Nginx): Scale Issuer/Verifier as well as Holder AcaPy to 10 instances each by using a Nginx container as a load balancer * feat(AcaPy Wallet Type): Make AcaPy Wallet Type configurable via the .env file * fix(.env): Fix .env file * feat(Handle Exception on Connection Invitation Receive): Start a new iteration if the holder is not able to accept the connection invitation * feat(Dashbaord): Show number of failes "POST/connections/receive-invitation" * chore(Holder AcaPy): Increase number of Holder AcaPys to 20 * chore(Dashboard Setup): Define the containers Grafana depends on * chore(Dashboard Setup): Define the containers Grafana depends on * chore(AcaPy): Increase default AcaPy Version * fix: Circular dependency in dashboard/logging deployment * refactor(TestRunners and TestFlows): Differentiate between TestRunners and TestFlows to make it easier to implement differen TestFlows while reusing the existing TestRunners * chore(Readme and Dashboard): Adopt the Readme and Dashboard to meet the new setup * feat(Issuer Flow): Added a new test flow to simulate an issuer who is establishing connections to holders and issues credentials to them afterwards * chore(Application Config): Add new application configuration settings and rename existing ones * chore(Application Config): Add new application configuration settings and rename existing ones * chore: Readme * feat(Revocable Credentials): Configure if a credential should be revocable, the size of the revocation registry, and if a proof request should include a non revoked check * chore(OOB Credential Offer): Preparations to support OOB Credential Offers * feat(Wallet DB Connections): Configure the number of connections to the Wallet DBs * feat(Connection Flow): Test Flow that only established connections with the holder * feat(Credential Issuance Flow): Test Flow that only issues credentials to the holder * feat(Proof Request Flow): Test Flow that only requests proofs from the holder * feat(Increasing Load Test Runner): Added a new test runner that increases the peak load in predefined steps * fix: Count of iterations in the Increasing-Load-Test-Runner * feat(AcaPy): Make Storage Type used by AcaPys configurable * feat(ErrorLogger): Introduction of an error logger to log unexpected error that occurred during testing * feat: Start Load Generator as Docker Container using "./manage.sh start" - to not start it via docker use "./manage.sh startWithoutLoadGenerator" * fix: .env.example * feat(Distribute load Across many Holders): Instead of having one horizontally scaled holder agent we now start many independent holder agents and the load generator takes care of distributing the load across all the holders * feat: Increase number of stand-alone holders to 10 * feat: Increase number of issuer/verifier AcaPys to 10 * feat: Add known error to the dashboard * feat: Rework manage.sh to ensure everything starts in the correct order * feat: Offer a "restart" command via the manage.sh * fix: AcaPy provisioning * fix: AcaPy provisioning * docs: Update Readme * Prometheus set-up (#43) * #32 Initial setup (macOS only) * #32 adding cadvisor support to be docker and platform agnostic * #32 Adding node-export * #32 Adding prometheus data source provisioning config * #32 Adding prometheus / node-exporter dashboard * Remove unused redis container * Increase platform agnostic-ness * fix: remove "redis" dependency * Removed unused prometheus scraper * Added Prometheus and installation guide for node-exporter Co-authored-by: PaulWen <[email protected]> Signed-off-by: Gerhard Dinhof [email protected] * fix(manage.sh): Rebuild load-generator docker image on every start to ensure that the latest application.yaml is loaded * feat(Docker Container Metrics): Visualize the docker container metrics in Grafana that have been collected by cAdvisor * feat(Linux Localhost): Avoid having to switch between "host.docker.internal" and "172.17.0.1" depending on the host OS * feat(Postgres Metrics): Collect Postgres metrics using postgres_exporter and visualize them in Grafana * Fixed cAdvisor volume mapping for macOS #44 Signed-off-by: Gerhard Dinhof [email protected] * fix: configure postgres username and password via env-variables * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * chore(Dashboards): Improve description of graphics * chore(Dashboards): Add a download link to each dashboard * chore(Dashboards): Update test result dashboard * fix(Exception Handling): Catch and log exceptions thrown in the context of ScheduledThreadPools * refactor(Error Logging): Rework the error logging to differentiate between errors and provide more information for easier debugging * feat(Summary Dashboard): Introduce a new dashboard which lists the most interesting graphs in one single dashboard * fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included * chore: Specify ID for Grafana folder to ensure a static URL * fix: Improve error logging for failed HTTP requests to AcaPy * fix(Test Results Dashboard): Improve the error logging on the test results dashboard * feat(Test Results Dashboard): Log started iterations as well as pending and failed iterations * fix(Constant and Increasing Load Runners): Previously, the scheduler used to generate the load was not running more than one task at a time. This caused the load generator not to generate the desired iterations per minute. * chore: Fix wording on test result dashboard * chore: Update default Revocation Registry Size to 3000 * refactor: Combine test result and summary dashboards into one dashboard * chore: Add CPU and memory usage of all containers to the test result dashboard * chore: Adjust tooltips * fix: Increasing Load Test Runner * chore: Improve test result dashboard * chore: Improve test result dashboard * Revert "fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included" This reverts commit cfe7f21. * chore: Improve test result dashboard * chore: Restructure the manage.sh * fix: Cancel schedulers properly once the tests are over * chore: Update test results * docs: restructured and added architecture diagram * chore: deleted unneeded readme * docs: added nginx and small cleanups * Implement Postgres Cluster as wallet DB (#49) Issue: #48 Author: Ilie Circiumaru Reviewer: @PaulWen Description: Implement 3 instances Postgres Cluster for wallet DB for the issuer-verifier. Split docker-compose on more files to enable launch of the isolated components * chore: Update Test-Results Dashboard * chore: Remove unused grafana-pdf-exporter reference * refactor: Cleanup the setup procedure * fix: Gitmodules * fix: Gitmodules * fix: Gitmodules * fix: nginx-issuer-verifier.conf * fix: manage.sh * chore: remove unused profiles from docker-compose-agents.yml * feat(Activate Postgres Cluster via .env): Switch from many start commands to just one and configure what is started via the .env * feat(Configure Number of Issuer/Verifier Agents): Depending on the number of requested issuer/verifier AcaPy instances a different nginx.conf is loaded that matches the desired number * feat(Configure Number of Holder Agents): The number of requested holder AcaPy instances can be changed via the .env * chore: restructure .env * feat(Log Revocation Registry Metadata): Track the usage of revocation registries. In particular look out if any RevReg/Index combination has been assigned to multiple credentials. * Implement declarative deployment configuration (#58) Issue: #56 Co-authored-by: Ilie Circiumaru <[email protected]> Co-authored-by: PaulWen <[email protected]> Description: 1. Implementation of declarative deployment configuration 2. Setup metrics for Postgres Cluster * fix(Revocation Registry Logging): Do not log any revocation registry metadata if not revocation registry is used * feat(Postgres Configuration): make the "statement_timeout" configurable * fix: network handling with docker-compose * feat(Postgres configuration): Configure "idle_in_transaction_session" and "lock_timeout" via the .env * fix(Postgres configuration): "idle_in_transaction_session_timeout" * fix: Test Results Dashboard * chore: Update Test-Results Dashboard * feat(Multitenancy): Run tests with AcaPy in multitenancy mode * chore: remove unnecessary host.docker.internal * docs: Split Readme into Developer and User Readme to avoid any confusion * docs: Improve the README.md * docs: Improve the README.md * fix: add host.docker.internal reference to load-generator so that he is able to connect to the VON Network to register a public DID * chore: Update test-results.json * docs: define test cases to verify the load generator still behaves as expected * docs: Add a troubleshooting section to the README.md * Implement monitoring of the Spring Boot app (#62) Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen * docs: Update Troubleshooting section * docs: Fix spelling * Added configuration option to enable Postgres debug logging (#60) * Added flags to enable postgres deub logging * Removing postgres debug settings from clustered setup * added comprehensive description for POSTGRES_LOG_DEBUG * alphabetical order Signed-off-by: Gerhard Dinhof [email protected] * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * #63 debug acapy (#64) feat(Debug AcaPy): Running "manage.sh debug" it is now possible to attach a debugger to the issuer-verifier-acapy. * Feat 59: credential revocation (#65) feat(Full Flow): Support for credential revocation with configurable revocation batch size * Fix 66 macos compsev2 compatibility (#67) fix: Switches Postgres from port 5000 to port 5432 as MacOS Monterey already uses the post 5000 itself and extends the README to outline that Compose V2 is not supported. * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * fix(Debugging): Load generator needs to be started manually after the debugger has been attached to the issuer-verifier-acapy. * docs: Improve the troubleshooting section * fix(Database Logger): Fix the configuration of the database exporter to the metrics of the wallet DB * chore(Multitenancy): Be able to run load tests with 9 tenants * feat(Problem Reports): Log problem reports and show them on the dashboard * chore(README): Add list of required dependencies * feat(Dashboards): Update the dashboards and add a dashboard to evaluate AcaPy logs * fix(Aries Client Logging): Ensure that aries client errors can be parsed by Loki * feat(manage.sh): Wait for the ledger to start until the DID was registered successfully to the ledger * feat(Apple M1 Chip): Update the dockerfile to be able to build the Load Generator Docker image on an Apple M1 Mac * feat(.env): Update the default .env values * chore(OOB): Remove out-of-band (OOB) placeholder code --------- Signed-off-by: Gerhard Dinhof [email protected] Co-authored-by: PaulWen <[email protected]> Co-authored-by: Richard Holzeis <[email protected]> Co-authored-by: Lukas Willburger <[email protected]> Co-authored-by: Paul Wenzel <[email protected]> Co-authored-by: iliecirciumaru <[email protected]>
PaulWen
pushed a commit
that referenced
this issue
Jan 30, 2023
Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen
PaulWen
added a commit
that referenced
this issue
Jan 30, 2023
* Create LICENSE Signed-off-by: Gerhard Dinhof [email protected] * docker-compose setup to start Issuer-Verifier and Holder AcaPys as well as Postgres DBs * Clean up docker-compose setup * Add postman based load generator for testing if the local setup works as expected * choreinitialized kotlin spring boot project * chore: added copyright * added maven to git ignore * docs: removed generated help md * chore!: updated to jdk 17 * docs: added copyright * feat: added boilerplate apis * feat: added Dockerfile * docs: added README * Add submodule + script for VON-network * restructuring the files * Automatically register public DID on the ledger * code cleanup * build: Switch to Java 11 as the built target for brought compatibility * feat: Integration of an automatically generated SwaggerUI * Update README.md * Update README.md * feat(AcaPy Integration): Integration of an AcaPy Java client library for calling the AcaPy Admin API as well as receiving Webhook calls from AcaPy * Working state with permission error in tails * tails server working * added extended logging capabilities * removed gitignore * remove ignored files * chor(Postman): Restructure files * chor(.idea): Update .idea files * chor(Setup): Let user know that it will take 30 seconds to register DID * chor(Setup): Automatically load and update git submodules * build(Java Version): Switch to Java Version 11 * Added README for the setup * Fixed typo in environment files * chore: Rename WebhookController to AcaPyWebhookController and LoadGeneratorController to InfoController * refactor: Structure of controllers and Swagger UI * feat: Integrate simple webhook logging * feat(TestRunner): application.yml is used to choose and configure the test case * feat(FullProcessRunner): Write schema and credential definition to ledger during setup * feat(FullProcessRunner): Connect to Issuer/Verifier as well as Holder AcaPy * feat(FullProcessRunner): Establish connection between Issuer/Verifier and Holder * build(Restart Policy): Restart each container always * feat(FullProcessRunner): Issue credential to holder and request credential from holder via proof request * refactor(Spring Configs): Move all Spring Config files into a config package * refactor(Aries Client): Build a wrapper around the AcaPyAriesClient library to make it easier replaceable by a different AriesClient library 1. to make it easier to time the HTTP requests the AriesClient 2. to potentially be able to use this load generators to test other agents too * refactor(Logger): Move logger into their own package * refactor(Aries Event Handler): Build a wrapper around the AcaPyEventHandler to make it easier replaceable by a different AriesClient library To potentially be able to use this load generators to test other agents too. * refactor(AriesClient and AriesObserver): Move service interfaces into the domain layer * chore: Renaming of package * chore: Clean up SwaggerConfig * feat(Readme): Update the Readmes * feat(Wallet Logger): Log the size of the issuer/holder wallet database periodically * refactor(Runner): Move termination condition into own method * feat(logging): added Loki as log collector * feat(Log HTTP Requests): Log the durations and Status Codes of HTTP requests going out to the AcaPy * chore(Renaming): Rename AriesEventLogger types * chore: Remove unused volume from docker-compose * chore: Clean WalletLogger log message * refactor(Agent Implementations): Move all AcaPy agent specific code into one folder * chore: Adopt manage.sh logging * chore: Adopt manage.sh logging * feat(AcaPy Http Request Timeout): Make it possible to configure the timeout for HTTP requests to the AcaPy clients * chore(Name Containers): Name the Grafana and Loki container * chore(Logging): Remove unused Logback configuration * chore(Logging): Adopt Logback output format * feat(Provision Grafana): Add Loki as the default datasource to Grafana and deactivate authentication for Grafana * fix(Wallet Logger): remove comma from log message * feat(Dashboard): Configured a default Grafana dashboard that is automatically provisioned on startup of Grafana * feat(Dashboard): Add overviews about Connections, Credentials, and Proof Requests to the dashboard * feat(Dashboard): Visualize the HTTP Response Codes and HTTP Response Times in Grafana * feat(Dashboard): Visualize the rate in which connections are being activated, credentials are being issued, and proof requests are being verified in Grafana * fix(AcaPy Event Publisher): Fix string date to milliseconds conversion * fix(Loki Config): Add custom Loki config to avoid the "Too many outstanding requests." issue * feat(Dashboard): Add total number of connection, credential issuance, and proof request count * refactor(Dashboard): Rename the keys that are being logged * chore(.env): Remove .env file from git * chore(Dashboard): Update HTTP Response Times metric * chore(Dashboard): Fix labeling rate is not per Second but per Minute * chore(Dashboard): Provide the wallet DB size in MB instead of bytes * chore(Dashboard): Outline the number of test iterations started per minute on the dashboard to visualize the load the load generator puts on the Agents * feat(FullProcessRunner): Add a new test runner to start new full process iterations on a constant "iteration per minute" base * chore(Dashboard): Reorder dashboard panels * feat(Export PDF Report): Grafana dashboards can now be exported as a PDF * fix(Dashboard Deployment): Put dashboard and logging containers into a separate docker network to avoid error when shutting everything down * refactor(Full Process Runner): Avoid code duplication across the FullProcessRunners * feat(Full Process Runner): Schedule FullProcessConstantLoadRunner at a fixed rate leveraging a thread pool * refactor(Aries Agent): Extract connectionId from ProofRequestDo and CredentialDo in preparation for connectionless proof requests * feat(Connectionless Proof Requests): Started implementing support for connectionless proof requests - not finished yet * feat(Scale AcaPy using Nginx): Scale AcaPy horizontally by using a Nginx container as a load balancer * feat(Scale AcaPy using Nginx): Scale Issuer/Verifier as well as Holder AcaPy to 10 instances each by using a Nginx container as a load balancer * feat(AcaPy Wallet Type): Make AcaPy Wallet Type configurable via the .env file * fix(.env): Fix .env file * feat(Handle Exception on Connection Invitation Receive): Start a new iteration if the holder is not able to accept the connection invitation * feat(Dashbaord): Show number of failes "POST/connections/receive-invitation" * chore(Holder AcaPy): Increase number of Holder AcaPys to 20 * chore(Dashboard Setup): Define the containers Grafana depends on * chore(Dashboard Setup): Define the containers Grafana depends on * chore(AcaPy): Increase default AcaPy Version * fix: Circular dependency in dashboard/logging deployment * refactor(TestRunners and TestFlows): Differentiate between TestRunners and TestFlows to make it easier to implement differen TestFlows while reusing the existing TestRunners * chore(Readme and Dashboard): Adopt the Readme and Dashboard to meet the new setup * feat(Issuer Flow): Added a new test flow to simulate an issuer who is establishing connections to holders and issues credentials to them afterwards * chore(Application Config): Add new application configuration settings and rename existing ones * chore(Application Config): Add new application configuration settings and rename existing ones * chore: Readme * feat(Revocable Credentials): Configure if a credential should be revocable, the size of the revocation registry, and if a proof request should include a non revoked check * chore(OOB Credential Offer): Preparations to support OOB Credential Offers * feat(Wallet DB Connections): Configure the number of connections to the Wallet DBs * feat(Connection Flow): Test Flow that only established connections with the holder * feat(Credential Issuance Flow): Test Flow that only issues credentials to the holder * feat(Proof Request Flow): Test Flow that only requests proofs from the holder * feat(Increasing Load Test Runner): Added a new test runner that increases the peak load in predefined steps * fix: Count of iterations in the Increasing-Load-Test-Runner * feat(AcaPy): Make Storage Type used by AcaPys configurable * feat(ErrorLogger): Introduction of an error logger to log unexpected error that occurred during testing * feat: Start Load Generator as Docker Container using "./manage.sh start" - to not start it via docker use "./manage.sh startWithoutLoadGenerator" * fix: .env.example * feat(Distribute load Across many Holders): Instead of having one horizontally scaled holder agent we now start many independent holder agents and the load generator takes care of distributing the load across all the holders * feat: Increase number of stand-alone holders to 10 * feat: Increase number of issuer/verifier AcaPys to 10 * feat: Add known error to the dashboard * feat: Rework manage.sh to ensure everything starts in the correct order * feat: Offer a "restart" command via the manage.sh * fix: AcaPy provisioning * fix: AcaPy provisioning * docs: Update Readme * Prometheus set-up (#43) * #32 Initial setup (macOS only) * #32 adding cadvisor support to be docker and platform agnostic * #32 Adding node-export * #32 Adding prometheus data source provisioning config * #32 Adding prometheus / node-exporter dashboard * Remove unused redis container * Increase platform agnostic-ness * fix: remove "redis" dependency * Removed unused prometheus scraper * Added Prometheus and installation guide for node-exporter Co-authored-by: PaulWen <[email protected]> Signed-off-by: Gerhard Dinhof [email protected] * fix(manage.sh): Rebuild load-generator docker image on every start to ensure that the latest application.yaml is loaded * feat(Docker Container Metrics): Visualize the docker container metrics in Grafana that have been collected by cAdvisor * feat(Linux Localhost): Avoid having to switch between "host.docker.internal" and "172.17.0.1" depending on the host OS * feat(Postgres Metrics): Collect Postgres metrics using postgres_exporter and visualize them in Grafana * Fixed cAdvisor volume mapping for macOS #44 Signed-off-by: Gerhard Dinhof [email protected] * fix: configure postgres username and password via env-variables * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * feat(Configure Load Tests): The load test configuration can now be done via the .env file instead of the application.yml * chore(Dashboards): Improve description of graphics * chore(Dashboards): Add a download link to each dashboard * chore(Dashboards): Update test result dashboard * fix(Exception Handling): Catch and log exceptions thrown in the context of ScheduledThreadPools * refactor(Error Logging): Rework the error logging to differentiate between errors and provide more information for easier debugging * feat(Summary Dashboard): Introduce a new dashboard which lists the most interesting graphs in one single dashboard * fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included * chore: Specify ID for Grafana folder to ensure a static URL * fix: Improve error logging for failed HTTP requests to AcaPy * fix(Test Results Dashboard): Improve the error logging on the test results dashboard * feat(Test Results Dashboard): Log started iterations as well as pending and failed iterations * fix(Constant and Increasing Load Runners): Previously, the scheduler used to generate the load was not running more than one task at a time. This caused the load generator not to generate the desired iterations per minute. * chore: Fix wording on test result dashboard * chore: Update default Revocation Registry Size to 3000 * refactor: Combine test result and summary dashboards into one dashboard * chore: Add CPU and memory usage of all containers to the test result dashboard * chore: Adjust tooltips * fix: Increasing Load Test Runner * chore: Improve test result dashboard * chore: Improve test result dashboard * Revert "fix(Grafana PDF Exporter): Build grafana-pdf-exporter docker image to have the latest bug fixes and features included" This reverts commit cfe7f21. * chore: Improve test result dashboard * chore: Restructure the manage.sh * fix: Cancel schedulers properly once the tests are over * chore: Update test results * docs: restructured and added architecture diagram * chore: deleted unneeded readme * docs: added nginx and small cleanups * Implement Postgres Cluster as wallet DB (#49) Issue: #48 Author: Ilie Circiumaru Reviewer: @PaulWen Description: Implement 3 instances Postgres Cluster for wallet DB for the issuer-verifier. Split docker-compose on more files to enable launch of the isolated components * chore: Update Test-Results Dashboard * chore: Remove unused grafana-pdf-exporter reference * refactor: Cleanup the setup procedure * fix: Gitmodules * fix: Gitmodules * fix: Gitmodules * fix: nginx-issuer-verifier.conf * fix: manage.sh * chore: remove unused profiles from docker-compose-agents.yml * feat(Activate Postgres Cluster via .env): Switch from many start commands to just one and configure what is started via the .env * feat(Configure Number of Issuer/Verifier Agents): Depending on the number of requested issuer/verifier AcaPy instances a different nginx.conf is loaded that matches the desired number * feat(Configure Number of Holder Agents): The number of requested holder AcaPy instances can be changed via the .env * chore: restructure .env * feat(Log Revocation Registry Metadata): Track the usage of revocation registries. In particular look out if any RevReg/Index combination has been assigned to multiple credentials. * Implement declarative deployment configuration (#58) Issue: #56 Co-authored-by: Ilie Circiumaru <[email protected]> Co-authored-by: PaulWen <[email protected]> Description: 1. Implementation of declarative deployment configuration 2. Setup metrics for Postgres Cluster * fix(Revocation Registry Logging): Do not log any revocation registry metadata if not revocation registry is used * feat(Postgres Configuration): make the "statement_timeout" configurable * fix: network handling with docker-compose * feat(Postgres configuration): Configure "idle_in_transaction_session" and "lock_timeout" via the .env * fix(Postgres configuration): "idle_in_transaction_session_timeout" * fix: Test Results Dashboard * chore: Update Test-Results Dashboard * feat(Multitenancy): Run tests with AcaPy in multitenancy mode * chore: remove unnecessary host.docker.internal * docs: Split Readme into Developer and User Readme to avoid any confusion * docs: Improve the README.md * docs: Improve the README.md * fix: add host.docker.internal reference to load-generator so that he is able to connect to the VON Network to register a public DID * chore: Update test-results.json * docs: define test cases to verify the load generator still behaves as expected * docs: Add a troubleshooting section to the README.md * Implement monitoring of the Spring Boot app (#62) Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen * docs: Update Troubleshooting section * docs: Fix spelling * Added configuration option to enable Postgres debug logging (#60) * Added flags to enable postgres deub logging * Removing postgres debug settings from clustered setup * added comprehensive description for POSTGRES_LOG_DEBUG * alphabetical order Signed-off-by: Gerhard Dinhof [email protected] * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * fix(Configuration): Fix the default configuration to be compatible with for the debugging case where the Load Generator is run locally outside a docker-container * #63 debug acapy (#64) feat(Debug AcaPy): Running "manage.sh debug" it is now possible to attach a debugger to the issuer-verifier-acapy. * Feat 59: credential revocation (#65) feat(Full Flow): Support for credential revocation with configurable revocation batch size * Fix 66 macos compsev2 compatibility (#67) fix: Switches Postgres from port 5000 to port 5432 as MacOS Monterey already uses the post 5000 itself and extends the README to outline that Compose V2 is not supported. * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * feat(Debugging): Configure via the .env if the load-generator should be started when running "manage.sh debug" * fix(Debugging): Load generator needs to be started manually after the debugger has been attached to the issuer-verifier-acapy. * docs: Improve the troubleshooting section * fix(Database Logger): Fix the configuration of the database exporter to the metrics of the wallet DB * chore(Multitenancy): Be able to run load tests with 9 tenants * feat(Problem Reports): Log problem reports and show them on the dashboard * chore(README): Add list of required dependencies * feat(Dashboards): Update the dashboards and add a dashboard to evaluate AcaPy logs * fix(Aries Client Logging): Ensure that aries client errors can be parsed by Loki * feat(manage.sh): Wait for the ledger to start until the DID was registered successfully to the ledger * feat(Apple M1 Chip): Update the dockerfile to be able to build the Load Generator Docker image on an Apple M1 Mac * feat(.env): Update the default .env values * chore(OOB): Remove out-of-band (OOB) placeholder code --------- Signed-off-by: Gerhard Dinhof [email protected] Co-authored-by: PaulWen <[email protected]> Co-authored-by: Richard Holzeis <[email protected]> Co-authored-by: Lukas Willburger <[email protected]> Co-authored-by: Paul Wenzel <[email protected]> Co-authored-by: iliecirciumaru <[email protected]>
iliecirciumaru
added a commit
that referenced
this issue
Jan 31, 2023
…epo subject to the Developer Certificate of Origin (DCO), Version 1.1. In the past I have used emails: [email protected] 3a17d46 Implement monitoring of the Spring Boot app (#62) Author: Ilie Circiumaru <[email protected]> Issue: //issues/45 Reviewer Paul Wen 16b504f Implement declarative deployment configuration (#58) Issue: #56 Co-authored-by: Ilie Circiumaru <[email protected]> Co-authored-by: PaulWen <[email protected]> Description: 1. Implementation of declarative deployment configuration 2. Setup metrics for Postgres Cluster e4caf10 Implement Postgres Cluster as wallet DB (#49) Issue: #48 Author: Ilie Circiumaru Reviewer: @PaulWen Description: Implement 3 instances Postgres Cluster for wallet DB for the issuer-verifier. Split docker-compose on more files to enable launch of the isolated components
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
To ensure that the load generator does not become a bottleneck, monitor the utilization of the Tomcat threads.
The text was updated successfully, but these errors were encountered: