diff --git a/AGENTS.md b/AGENTS.md index 8e1995a8..2e0e049c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,23 +8,23 @@ LLM-based agents can accelerate development only if they respect our house rules ## Language & character-set policy -| Requirement | Rationale | -|--------------|-----------| -| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the University of Oxford style guide for reference. | -| **ASCII-7 only** (code-points 0-127). Avoid smart quotes, non-breaking spaces and accented characters. | ASCII-7 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect the 8th bit to be 0. | -| If a symbol is not available in ASCII-7, use a textual form such as `micro-second`, `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | +| Requirement | Rationale | +|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| +| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the University of Oxford style guide for reference. | +| **ISO-8859-1** (code-points 0-255). Avoid smart quotes, non-breaking spaces and accented characters. | ISO-8859-1 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect the 8th bit to be 0. | +| If a symbol is not available in ISO-8859-1, use a textual form such as `micro-second`, `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | ## Javadoc guidelines **Goal:** Every Javadoc block should add information you cannot glean from the method signature alone. Anything else is noise and slows readers down. -| Do | Don't | -|----|-------| -| State *behavioural contracts*, edge-cases, thread-safety guarantees, units, performance characteristics and checked exceptions. | Restate the obvious ("Gets the value", "Sets the name"). | -| Keep the first sentence short; it becomes the summary line in aggregated docs. | Duplicate parameter names/ types unless more explanation is needed. | -| Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | -| Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | +| Do | Don't | +|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------| +| State *behavioural contracts*, edge-cases, thread-safety guarantees, units, performance characteristics and checked exceptions. | Restate the obvious ("Gets the value", "Sets the name"). | +| Keep the first sentence short; it becomes the summary line in aggregated docs. | Duplicate parameter names/ types unless more explanation is needed. | +| Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | +| Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | The principle that Javadoc should only explain what is *not* manifest from the signature is well-established in the wider Java community. @@ -60,7 +60,8 @@ See the [Project Requirements](src/main/adoc/project-requirements.adoc) for deta ## Elevating the Workflow with Real-Time Documentation -Building upon our existing Iterative Workflow, the newest recommendation is to emphasise *real-time updates* to documentation. +Building upon our existing Iterative Workflow, the newest recommendation is to emphasise *real-time updates* to +documentation. Ensure the relevant `.adoc` files are updated when features, requirements, implementation details, or tests change. This tight loop informs the AI accurately and creates immediate clarity for all team members. @@ -75,41 +76,54 @@ This tight loop informs the AI accurately and creates immediate clarity for all ### Best Practices -* **Maintain Sync**: Keep documentation (AsciiDoc), tests, and code synchronised in version control. Changes in one area should prompt reviews and potential updates in the others. -* **Doc-First for New Work**: For *new* features or requirements, aim to update documentation first, then use AI to help produce or refine corresponding code and tests. For refactoring or initial bootstrapping, updates might flow from code/tests back to documentation, which should then be reviewed and finalised. -* **Small Commits**: Each commit should ideally relate to a single requirement or coherent change, making reviews easier for humans and AI analysis tools. -- **Team Buy-In**: Encourage everyone to review AI outputs critically and contribute to maintaining the synchronicity of all artefacts. +* **Maintain Sync**: Keep documentation (AsciiDoc), tests, and code synchronised in version control. Changes in one area + should prompt reviews and potential updates in the others. +* **Doc-First for New Work**: For *new* features or requirements, aim to update documentation first, then use AI to help + produce or refine corresponding code and tests. For refactoring or initial bootstrapping, updates might flow from + code/tests back to documentation, which should then be reviewed and finalised. +* **Small Commits**: Each commit should ideally relate to a single requirement or coherent change, making reviews easier + for humans and AI analysis tools. + +- **Team Buy-In**: Encourage everyone to review AI outputs critically and contribute to maintaining the synchronicity of + all artefacts. ## AI Agent Guidelines When using AI agents to assist with development, please adhere to the following guidelines: -* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ASCII-7 guidelines outlined above. - Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. -* **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. -* **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. +* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and + ISO-8859-1 guidelines outlined above. + Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present + in the code or existing documentation. +* **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it + provides additional context or clarification. +* **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's + documentation standards before committing it to the repository. ## Company-Wide Tagging -This section records **company-wide** decisions that apply to *all* Chronicle projects. All identifiers use the --xxx prefix. The `xxx` are unique across in the same Scope even if the tags are different. Component-specific decisions live in their xxx-decision-log.adoc files. +This section records **company-wide** decisions that apply to *all* Chronicle projects. All identifiers use +the --xxx prefix. The `xxx` are unique across in the same Scope even if the tags are different. +Component-specific decisions live in their xxx-decision-log.adoc files. ### Tag Taxonomy (Nine-Box Framework) -To improve traceability, we adopt the Nine-Box taxonomy for requirement and decision identifiers. These tags are used in addition to the existing ALL prefix, which remains reserved for global decisions across every project. +To improve traceability, we adopt the Nine-Box taxonomy for requirement and decision identifiers. These tags are used in +addition to the existing ALL prefix, which remains reserved for global decisions across every project. .Adopt a Nine-Box Requirement Taxonomy -|Tag | Scope | Typical examples | -|----|-------|------------------| -|FN |Functional user-visible behaviour | Message routing, business rules | -|NF-P |Non-functional - Performance | Latency budgets, throughput targets | -|NF-S |Non-functional - Security | Authentication method, TLS version | -|NF-O |Non-functional - Operability | Logging, monitoring, health checks | -|TEST |Test / QA obligations | Chaos scenarios, benchmarking rigs | -|DOC |Documentation obligations | Sequence diagrams, user guides | -|OPS |Operational / DevOps concerns | Helm values, deployment checklist | -|UX |Operator or end-user experience | CLI ergonomics, dashboard layouts | -|RISK |Compliance / risk controls | GDPR retention, audit trail | +| Tag | Scope | Typical examples | +|------|-----------------------------------|-------------------------------------| +| FN | Functional user-visible behaviour | Message routing, business rules | +| NF-P | Non-functional - Performance | Latency budgets, throughput targets | +| NF-S | Non-functional - Security | Authentication method, TLS version | +| NF-O | Non-functional - Operability | Logging, monitoring, health checks | +| TEST | Test / QA obligations | Chaos scenarios, benchmarking rigs | +| DOC | Documentation obligations | Sequence diagrams, user guides | +| OPS | Operational / DevOps concerns | Helm values, deployment checklist | +| UX | Operator or end-user experience | CLI ergonomics, dashboard layouts | +| RISK | Compliance / risk controls | GDPR retention, audit trail | `ALL-*` stays global, case-exact tags. Pick one primary tag if multiple apply. diff --git a/LICENSE.adoc b/LICENSE.adoc index 4c32dfd9..f4505664 100644 --- a/LICENSE.adoc +++ b/LICENSE.adoc @@ -1,14 +1,9 @@ - == Copyright 2016-2025 chronicle.software -Licensed under the *Apache License, Version 2.0* (the "License"); -you may not use this file except in compliance with the License. +Licensed under the *Apache License, Version 2.0* (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. \ No newline at end of file +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and limitations under the License. diff --git a/README.adoc b/README.adoc index cce3eb07..a8699078 100644 --- a/README.adoc +++ b/README.adoc @@ -7,7 +7,7 @@ Chronicle Software image:https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh] image:https://javadoc.io/badge2/net.openhft/JLBH/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-wire/latest/index.html"] -//image:https://javadoc-badge.appspot.com/net.openhft/jlbh.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/jlbh] +// image:https://javadoc-badge.appspot.com/net.openhft/jlbh.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/jlbh] image:https://img.shields.io/github/license/OpenHFT/JLBH[GitHub] image:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link="https://chronicle.software/release-notes/"] image:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_JLBH&metric=alert_status[link="https://sonarcloud.io/dashboard?id=OpenHFT_JLBH"] @@ -17,13 +17,15 @@ toc::[] == About -Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a microbenchmark. See <> for a series of articles introducing JLBH. +Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a microbenchmark. +See <> for a series of articles introducing JLBH. An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles.] link:src/main/adoc/project-requirements.adoc[The requirements document] contains detailed feature descriptions. For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_7-glossary[the Glossary (section 7)]. -Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread. To do this, use the JLBH.eventLoopHandler method rather than JLBH.start. +Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread. +To do this, use the JLBH.eventLoopHandler method rather than JLBH.start. NOTE: The JLBH harness itself runs on a single thread; benchmarked code may spawn threads, but the harness thread is unique. @@ -82,10 +84,16 @@ For a visual overview of how a benchmark progresses, see the link:src/main/adoc/ == Additional Features === OS Jitter Measurement -JLBH can optionally track operating-system jitter by running a background thread that records scheduler delays as a separate probe. This is useful when investigating latency spikes caused by kernel activity, but it incurs some overhead so it is enabled by default. Disable it via `recordOSJitter(false)` when you want to avoid the extra monitoring. The feature is listed in the requirements specification around lines 52 and 56 of `project-requirements.adoc`. + +JLBH can optionally track operating-system jitter by running a background thread that records scheduler delays as a separate probe. +This is useful when investigating latency spikes caused by kernel activity, but it incurs some overhead so it is enabled by default. +Disable it via `recordOSJitter(false)` when you want to avoid the extra monitoring. +The feature is listed in the requirements specification around lines 52 and 56 of `project-requirements.adoc`. === Adding custom probes -JLBH lets you time additional stages of your benchmark using `addProbe(String)`. The returned `NanoSampler` records its own histogram. + +JLBH lets you time additional stages of your benchmark using `addProbe(String)`. +The returned `NanoSampler` records its own histogram. [source,java] ---- @@ -113,17 +121,22 @@ class ProbedTask implements JLBHTask { For a runnable example see `src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java`. === CSV Output -To export JLBH results in CSV format, invoke `JLBHResultSerializer.runResultToCSV(jlbhResult)`. This writes the output to `result.csv` by default, as defined in `JLBHResultSerializer.RESULT_CSV`. + +To export JLBH results in CSV format, invoke `JLBHResultSerializer.runResultToCSV(jlbhResult)`. +This writes the output to `result.csv` by default, as defined in `JLBHResultSerializer.RESULT_CSV`. == Environment and Configuration CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/adoc/project-requirements.adoc[requirements document]. -Histogramming of recorded latencies is precise. As specified in the link:src/main/adoc/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. This accuracy retains sub-nanosecond resolution over a wide range, so the reported percentiles closely reflect true end-to-end timings. +Histogramming of recorded latencies is precise. +As specified in the link:src/main/adoc/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. +This accuracy retains sub-nanosecond resolution over a wide range, so the reported percentiles closely reflect true end-to-end timings. == Sample Benchmarks and Tests -The project ships with several example tasks and tests located under `src/test/java/net/openhft/chronicle/jlbh`. They can serve as a starting point when writing your own benchmarks. +The project ships with several example tasks and tests located under `src/test/java/net/openhft/chronicle/jlbh`. +They can serve as a starting point when writing your own benchmarks. |=== | Class | Purpose @@ -160,10 +173,10 @@ The following table summarises the main non-functional quality attributes derive http://www.rationaljava.com/2016/04/jlbh-introducing-java-latency.html[Introducing JLBH] -- What is JLBH -- What was the motivation for JLBH -- Differences between JMH and JLBH -- Quick start guide +* What is JLBH +* What was the motivation for JLBH +* Differences between JMH and JLBH +* Quick start guide === Sample percentile output @@ -181,52 +194,56 @@ worst: 12.56 12.56 10.61 10.93 http://www.rationaljava.com/2016/04/jlbh-examples-1-why-code-should-be.html[Why Code Should be Benchmarked in Context] - - A side by side example using JMH and JLBH for Date serialisation - - Measuring Date serialisation in a microbenchmark - - Measuring Date serialisation as part of a proper application - - How to add a probe to your JLBH benchmark -- Understanding the importance of measuring code in context +* A side by side example using JMH and JLBH for Date serialisation +* Measuring Date serialisation in a microbenchmark +* Measuring Date serialisation as part of a proper application +* How to add a probe to your JLBH benchmark +* Understanding the importance of measuring code in context Co-ordinated omission occurs when latency measurements ignore the time requests wait to be serviced, leading to unrealistically low results. http://www.rationaljava.com/2016/04/jlbh-examples-2-accounting-for.html[Accounting for Coordinated Omission] -- Running JLBH with and without accounting for coordinated omission -- An example illustrating the numerical impact of coordinated omission -- A discussion about flow control +* Running JLBH with and without accounting for coordinated omission +* An example illustrating the numerical impact of coordinated omission +* A discussion about flow control http://www.rationaljava.com/2016/04/jlbh-examples-3-affects-of-throughput.html[The Affects of Throughput on Latency] Note: the blog deliberately uses "Affects" in the title while describing the effects of throughput on latency. -- A discussion about the effects of throughput on latency -- How to use JLBH to measure TCP loopback -- Adding probes to test both halves of the TCP round trip -- Watching the effect of increasing throughput on latency -- Understanding that you have to drop throughput to achieve good latencies at high percentiles. +* A discussion about the effects of throughput on latency +* How to use JLBH to measure TCP loopback +* Adding probes to test both halves of the TCP round trip +* Watching the effect of increasing throughput on latency +* Understanding that you have to drop throughput to achieve good latencies at high percentiles. http://www.rationaljava.com/2016/04/jlbh-examples-4-benchmarking-quickfix.html[Benchmarking QuickFix vs ChronicleFix] -- Using JLBH to benchmark QuickFIX -- Observing how QuickFix latencies degrade through the percentiles -- Comparing QuickFIX with Chronicle FIX +* Using JLBH to benchmark QuickFIX +* Observing how QuickFix latencies degrade through the percentiles +* Comparing QuickFIX with Chronicle FIX == Common Questions === Using JLBH as part of automated performance/performance regression testing -- Extract latency percentiles from `net.openhft.chronicle.jlbh.JLBHTest::shouldProvideResultData` for xUnit tests. -- Run them on a production-like CI server. -- When local hardware is adequate, execute them along with other tests. -- Integrate these benchmarks into the TDD cycle to expose latency-related design issues early. +* Extract latency percentiles from `net.openhft.chronicle.jlbh.JLBHTest::shouldProvideResultData` for xUnit tests. +* Run them on a production-like CI server. +* When local hardware is adequate, execute them along with other tests. +* Integrate these benchmarks into the TDD cycle to expose latency-related design issues early. === Warm-up iterations -JLBH performs a warm-up phase before measurements start. A rule of thumb is to warm up for about 30% of the run iterations. Override this with `warmUpIterations(int)` if needed. +JLBH performs a warm-up phase before measurements start. +A rule of thumb is to warm up for about 30% of the run iterations. +Override this with `warmUpIterations(int)` if needed. === Throughput modelling -Configure the target rate using `throughput(int, TimeUnit)` and optionally a `LatencyDistributor` for bursty patterns. Start with your production rate and tune to study latency trade-offs. +Configure the target rate using `throughput(int, TimeUnit)` and optionally a `LatencyDistributor` for bursty patterns. +Start with your production rate and tune to study latency trade-offs. == Licensing and Release Information -The code is licensed under the terms described in the link:LICENSE.adoc[Apache 2.0 License]. Releases are available on link:https://github.com/OpenHFT/JLBH/releases[GitHub]. +The code is licensed under the terms described in the link:LICENSE.adoc[Apache 2.0 License]. +Releases are available on link:https://github.com/OpenHFT/JLBH/releases[GitHub]. diff --git a/pom.xml b/pom.xml index 56dc8cb1..7e32bcb3 100644 --- a/pom.xml +++ b/pom.xml @@ -15,7 +15,8 @@ ~ limitations under the License. --> - + 4.0.0 @@ -26,10 +27,19 @@ - openhft - https://sonarcloud.io + openhft + https://sonarcloud.io + 3.6.0 + 8.45.1 + 4.8.6.6 + 1.14.0 + 3.28.0 + 0.8.14 + 0.883 + 0.786 + 1.23ea6 - + jlbh 1.27ea2-SNAPSHOT OpenHFT/JLBH @@ -120,19 +130,6 @@ - - org.apache.maven.plugins - maven-compiler-plugin - - true - false - - -Xlint:all - -parameters - - - - E[Cleanup]; ---- -A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are: +A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. +The key phases are: -== 1. Setup +== Setup This initial phase prepares the benchmark environment: -* **JLBH Configuration**: The user instantiates `JLBHOptions` to define all benchmark parameters (e.g., iterations, throughput, runs, coordinated omission settings). An instance of `JLBH` is then created using these options. -* **Task Initialization**: The `JLBHTask.init(JLBH jlbh)` method of the user-provided task is invoked. -** This allows the benchmark task to perform its own one-time setup, such as initializing resources or fixtures. -** Crucially, this is where additional measurement probes can be registered by calling `jlbh.addProbe("probeName")` to get `NanoSampler` instances for specific sub-sections of the task. -* **OS Jitter Monitoring**: Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. -* **Affinity**: If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. +JLBH Configuration :: +The user instantiates `JLBHOptions` to define all benchmark parameters (e.g., iterations, throughput, runs, coordinated omission settings). +An instance of `JLBH` is then created using these options. +Task Initialization :: +The `JLBHTask.init(JLBH jlbh)` method of the user-provided task is invoked. +* This allows the benchmark task to perform its own one-time setup, such as initializing resources or fixtures. +* Crucially, this is where additional measurement probes can be registered by calling `jlbh.addProbe("probeName")` to get `NanoSampler` instances for specific sub-sections of the task. +OS Jitter Monitoring :: +Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. +Affinity :: +If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. -== 2. Warmup +== Warmup Before any measurements are formally recorded, the harness executes a warm-up phase: -* **Warmup Iterations**: The `JLBHTask.run(long startTimeNs)` method is called `warmUpIterations` times (as specified in `JLBHOptions`). The `startTimeNs` passed is typically just `System.nanoTime()` for this phase. -** The primary goal is to allow the Java Virtual Machine (JVM) to perform Just-In-Time (JIT) compilation and optimize the benchmarked code paths. -** It also helps in bringing caches to a "warm" state and letting the system reach a more stable performance profile. -* **Sample Discarding**: Latency samples generated during the warmup phase are usually recorded into the histograms but are *not* part of the final reported results. The histograms are reset after this phase. -* **Task Notification**: After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. -* **Optional Pause**: If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. - -== 3. Execution (Measurement Runs) - -This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): - -* **Per Run**: -** The following steps are repeated for each configured run. -** **Iteration Loop**: The `JLBH` orchestrator executes a loop for the number of `iterations` specified in `JLBHOptions`. -*** **Start Time Calculation**: For each iteration, JLBH calculates an ideal `startTimeNs`. This calculation considers: -**** The configured `throughput` (e.g., messages per second). -**** The `LatencyDistributor` strategy, if one is set, to potentially vary the inter-iteration delay. -**** If `accountForCoordinatedOmission` is true (the default), `startTimeNs` represents the ideal scheduled time for the operation. JLBH will then busy-wait (spin) until `System.nanoTime()` reaches this `startTimeNs`, ensuring the task is dispatched as close as possible to its intended schedule, thus accounting for delays that might otherwise be missed. -*** **Task Invocation**: The `JLBHTask.run(startTimeNs)` method is invoked with the calculated (and potentially waited-for) `startTimeNs`. -*** **Sample Recording**: -**** Within the `JLBHTask.run()` method, the user's code is executed. -**** To record the primary end-to-end latency for the iteration, the task must call `jlbh.sample(System.nanoTime() - startTimeNs)`. -**** For any custom probes registered during setup, the task can call `myProbe.sampleNanos(duration)` to record latencies for specific sub-stages. -**** Each recorded sample is added to the respective `Histogram` associated with the sampler (either the main JLBH sampler or a custom probe's sampler). -** **Run Completion**: After all iterations for the current run are finished: -*** The `JLBHTask.runComplete()` method is invoked. -*** Statistics for this specific run (e.g., percentiles derived from the histograms) are typically printed to the configured `PrintStream`. -*** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. -*** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. - -== 4. Reporting +Warmup Iterations :: +The `JLBHTask.run(long startTimeNs)` method is called `warmUpIterations` times (as specified in `JLBHOptions`). +The `startTimeNs` passed is typically just `System.nanoTime()` for this phase. +* The primary goal is to allow the Java Virtual Machine (JVM) to perform Just-In-Time (JIT) compilation and optimize the benchmarked code paths. +* It also helps in bringing caches to a "warm" state and letting the system reach a more stable performance profile. +Sample Discarding :: +Latency samples generated during the warmup phase are usually recorded into the histograms but are _not_ part of the final reported results. +The histograms are reset after this phase. +Task Notification :: +After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. +Optional Pause :: +If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. + +== Execution (Measurement Runs) + +This is the core phase where timed iterations are performed and latency data is collected. +This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): + +Per Run :: +* The following steps are repeated for each configured run. +* *Iteration Loop*: The `JLBH` orchestrator executes a loop for the number of `iterations` specified in `JLBHOptions`. +** *Start Time Calculation*: For each iteration, JLBH calculates an ideal `startTimeNs`. +This calculation considers: +*** The configured `throughput` (e.g., messages per second). +*** The `LatencyDistributor` strategy, if one is set, to potentially vary the inter-iteration delay. +*** If `accountForCoordinatedOmission` is true (the default), `startTimeNs` represents the ideal scheduled time for the operation. +JLBH will then busy-wait (spin) until `System.nanoTime()` reaches this `startTimeNs`, ensuring the task is dispatched as close as possible to its intended schedule, thus accounting for delays that might otherwise be missed. +** *Task Invocation*: The `JLBHTask.run(startTimeNs)` method is invoked with the calculated (and potentially waited-for) `startTimeNs`. +** *Sample Recording*: +*** Within the `JLBHTask.run()` method, the user's code is executed. +*** To record the primary end-to-end latency for the iteration, the task must call `jlbh.sample(System.nanoTime() - startTimeNs)`. +*** For any custom probes registered during setup, the task can call `myProbe.sampleNanos(duration)` to record latencies for specific sub-stages. +*** Each recorded sample is added to the respective `Histogram` associated with the sampler (either the main JLBH sampler or a custom probe's sampler). +* *Run Completion*: After all iterations for the current run are finished: +** The `JLBHTask.runComplete()` method is invoked. +** Statistics for this specific run (e.g., percentiles derived from the histograms) are typically printed to the configured `PrintStream`. +** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. +** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. + +== Reporting After all measurement runs are completed, the aggregated results are finalized and made available: -* **Final Summary**: A comprehensive summary, often comparing results across all runs and highlighting percentile variations, is printed to the `PrintStream`. -* **Result Object Creation**: The final, immutable `JLBHResult` object is constructed. This object encapsulates all recorded data for all probes across all runs. -* **Result Consumption**: If a `JLBHResultConsumer` was provided during JLBH setup, its `accept(JLBHResult)` method is called, passing the `JLBHResult` object. This allows for programmatic access to the detailed results, for example, for assertion in automated tests or for custom serialization. -* **External Tools**: Users can further process the `JLBHResult` using utilities like: -** `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. -** `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. - -== 5. Cleanup +Final Summary :: +A comprehensive summary, often comparing results across all runs and highlighting percentile variations, is printed to the `PrintStream`. +Result Object Creation :: +The final, immutable `JLBHResult` object is constructed. +This object encapsulates all recorded data for all probes across all runs. +Result Consumption :: +If a `JLBHResultConsumer` was provided during JLBH setup, its `accept(JLBHResult)` method is called, passing the `JLBHResult` object. +This allows for programmatic access to the detailed results, for example, for assertion in automated tests or for custom serialization. +External Tools :: +Users can further process the `JLBHResult` using utilities like: +* `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. +* `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. + +== Cleanup The final phase ensures that any resources are properly released: -* **Task Cleanup**: The `JLBHTask.complete()` method is called, allowing the user's benchmark task to perform any necessary cleanup (e.g., closing files, releasing network connections). -* **OS Jitter Monitor**: If the `OSJitterMonitor` thread was started, it is signaled to terminate. -* **Affinity Release**: If a CPU affinity lock was acquired by JLBH at the start, it is released. +Task Cleanup :: +The `JLBHTask.complete()` method is called, allowing the user's benchmark task to perform any necessary cleanup (e.g., closing files, releasing network connections). +OS Jitter Monitor :: +If the `OSJitterMonitor` thread was started, it is signaled to terminate. +Affinity Release :: +If a CPU affinity lock was acquired by JLBH at the start, it is released. diff --git a/src/main/adoc/decision-log.adoc b/src/main/adoc/decision-log.adoc index 1113684c..9e14c9b0 100644 --- a/src/main/adoc/decision-log.adoc +++ b/src/main/adoc/decision-log.adoc @@ -5,7 +5,8 @@ Chronicle Software :toc: :lang: en-GB -This document records key architectural and project choices made during the development of Chronicle JLBH. Entries follow the Decision Record Template so that contributors can trace why a change occurred. +This document records key architectural and project choices made during the development of Chronicle JLBH. +Entries follow the Decision Record Template so that contributors can trace why a change occurred. == Decisions diff --git a/src/main/adoc/jlbh-cookbook.adoc b/src/main/adoc/jlbh-cookbook.adoc index fa91cc6d..0587d334 100644 --- a/src/main/adoc/jlbh-cookbook.adoc +++ b/src/main/adoc/jlbh-cookbook.adoc @@ -1,13 +1,14 @@ = Chronicle JLBH Cookbook Chronicle Software :toc: +:sectnums: :lang: en-GB :source-highlighter: rouge A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. These recipes aim to provide practical starting points for your own benchmarks. -== 1. Timing a Simple Method Call +== Timing a Simple Method Call This is the most basic use case: measuring the latency of a self-contained Java method. @@ -81,7 +82,7 @@ Avoid including unrelated logic within the timed section. If `myMethod()` is very fast, high throughput might be achievable. If it's slower, adjust throughput accordingly. -== 2. Measuring a Network Round Trip (UDP Loopback) +== Measuring a Network Round Trip (UDP Loopback) This recipe demonstrates timing a UDP message send and receive on loopback. @@ -220,7 +221,7 @@ public class MainApp { * Consider TCP for connection-oriented tests; its performance characteristics (e.g., Nagle's algorithm, ACK delays) differ from UDP. * Packet loss and retransmissions (for UDP, if implementing reliability) can heavily affect latency distributions. -== 3. Running Within an Existing Event Loop +== Running Within an Existing Event Loop JLBH can integrate with Chronicle's event loop mechanism, useful for benchmarking components that are already part of an event-driven architecture. @@ -309,9 +310,9 @@ public class MainApp { * The `eventLoopHandler` method installs handlers that drive the benchmark lifecycle on the `EventLoop`'s thread. * This is useful when the code being benchmarked is designed to run on a specific event loop. * `JLBHOptions.accountForCoordinatedOmission(true)` (the default) is essential for `eventLoopHandler` to function correctly. ---- +''' -== 4. Using Multiple Probes for Staged Operations +== Using Multiple Probes for Staged Operations JLBH allows timing multiple stages within a single benchmark iteration using custom probes. @@ -405,9 +406,9 @@ public class MainApp { * This approach is excellent for identifying bottlenecks within a larger operation. * The JLBH output will include separate percentile summaries for "Stage1_Processing", "Stage2_Persistence", and the default "end to end" probe. ---- +''' -== 5. Recording and Exporting Results +== Recording and Exporting Results JLBH allows programmatic access to results and provides utilities for serialization, such as to CSV. @@ -489,7 +490,7 @@ public class MainApp { * The `JLBHResult` object is immutable and contains rich data for all probes and runs. * `JLBHResultSerializer` provides a convenient way to get data out for external analysis. -== 6. Enabling and Understanding OS Jitter Measurement +== Enabling and Understanding OS Jitter Measurement Operating system jitter can significantly impact low-latency applications. JLBH can measure this independently. diff --git a/src/main/adoc/project-requirements.adoc b/src/main/adoc/project-requirements.adoc index 2600c7ab..5050f001 100644 --- a/src/main/adoc/project-requirements.adoc +++ b/src/main/adoc/project-requirements.adoc @@ -3,6 +3,7 @@ Chronicle Software :revnumber: 1.0 :revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB *_Abstract_*:: @@ -10,15 +11,17 @@ Chronicle JLBH is an open-source Java library, released under the Apache Licence It allows developers to _measure_, _analyse_ and _regress-test_ latency behaviour of critical code paths "in context" rather than via isolated micro-benchmarks. Key features include compensation for *co-ordinated omission*, configurable throughput modelling, additional probes, optional operating-system jitter tracking, and serialisation of results for CI dashboards. -== 1. Introduction +== Introduction + +=== Purpose -=== 1.1 Purpose This document specifies functional and non-functional requirements for Chronicle JLBH, ensuring that contributors and integrators share a common understanding of the product's goals and constraints. -=== 1.2 Scope +=== Scope + The harness targets _low-latency_ Java applications (trading systems, micro-services, real-time analytics) that must quantify latencies at extreme percentiles under realistic load patterns. -=== 1.3 Definitions, Acronyms and Abbreviations +=== Definitions, Acronyms and Abbreviations |=== | *Term* | *Definition* @@ -30,7 +33,7 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv | CI | Continuous Integration environment (e.g. GitHub Actions, TeamCity). |=== -=== 1.4 References +=== References * Upstream repository: https://github.com/OpenHFT/JLBH[GitHub - OpenHFT/JLBH] * API reference: https://javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/core/jlbh/JLBH.html[Javadoc] @@ -38,16 +41,19 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv * Discussion of _co-ordinated omission_: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE[m.s. thread] * Chronicle blog post on JLBH in event loops: https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html[Vanilla-Java article] -=== 1.5 Overview +=== Overview + Sections 2-5 describe the product context, interfaces, detailed system features, and quality attributes. Section 6 captures licensing, whilst Section 7 presents the glossary. -== 2. Overall Description +== Overall Description -=== 2.1 Product Perspective -JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. +=== Product Perspective -=== 2.2 Product Functions +JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. +It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. + +=== Product Functions * Generate high-resolution histograms (>=35 bits) of end-to-end sample latencies. * Optionally compensate for *co-ordinated omission* by adjusting the synthetic start time. @@ -59,7 +65,7 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * Integrate with CI (TeamCity helper emits `##teamcity` statistics lines). * Allow execution from any thread or installation onto a Chronicle *MediumEventLoop*. -=== 2.3 User Classes and Characteristics +=== User Classes and Characteristics |=== | *Actor* | *Description* | *Technical Expertise* @@ -69,28 +75,28 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro | CI System | Runs automated latency regression suite, ingests CSV or TeamCity stats. | N/A (automation). |=== -=== 2.4 Operating Environment +=== Operating Environment * Java 11 LTS and later; Java 17 recommended for current builds. * Linux x86-64 (primary), other POSIX OSes supported; macOS usable but CI baselines exclude it. * Typical CPU affinity facilities available via *OpenHFT Affinity* library. -=== 2.5 Design and Implementation Constraints +=== Design and Implementation Constraints * Single-threaded harness execution; benchmarked code may span threads. * Measurements rely on `System.nanoTime()` precision; hardware timers must be invariant. * JVM must be started with `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when using async-profiler concurrently. * Library is released under the Apache Licence 2.0; contributions must comply with CLA policy. -=== 2.6 Assumptions and Dependencies +=== Assumptions and Dependencies * SUT jars are available on the same JVM classpath. * Benchmark host clocks are monotonic and not subjected to aggressive power-saving states. * CI agents provision exclusive CPU cores to minimise jitter. -== 3. External Interface Requirements +== External Interface Requirements -=== 3.1 Public API +=== Public API * `JLBHOptions` - builder for benchmark parameters (throughput, warm-up, runs, etc.). * `JLBHTask` - user-defined callback with lifecycle methods `init`, `run`, `warmedUp`, `runComplete`, `complete`. @@ -98,78 +104,97 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * `JLBHResult` - immutable projection of measured data. * `JLBHResultSerializer` - CSV writer for analytics pipelines. -=== 3.2 User Interface +=== User Interface + CLI usage is demonstrated in _ExampleJLBHMain_ and test fixtures; no dedicated GUI is provided. -=== 3.3 Hardware Interfaces -None. The harness interacts with OS timers and CPU affinity via JNI where available. +=== Hardware Interfaces -=== 3.4 Software Interfaces +None. +The harness interacts with OS timers and CPU affinity via JNI where available. + +=== Software Interfaces * Build tool: *Apache Maven* (pom.xml defines BOM imports). * Logging: SLF4J (test scope). * Testing: JUnit 4 for unit and integration tests. * CI Metrics: TeamCity service messages. -== 4. System Features +== System Features + +=== FN-001 Latency Sampling -=== 4.1 FN-001 Latency Sampling *Description*: Collect nanosecond deltas from user code via `sample(long)` or `sampleNanos(long)`. *Stimulus/Response*: Upon each iteration, `JLBHTask.run` calls `jlbh.sample(...)`; histogram updates occur; run summary printed at completion. *Priority*: *Essential*. -=== 4.2 FN-002 Coordinated-Omission Compensation -Ensures that queue back-log does not under-represent tail latency. Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. +=== FN-002 Coordinated-Omission Compensation + +Ensures that queue back-log does not under-represent tail latency. +Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. + +=== FN-003 Additional Probes -=== 4.3 FN-003 Additional Probes Developers may time sub-stages (e.g., serialisation, network round-trip) through `addProbe(String)` and record them independently. -=== 4.4 FN-004 OS Jitter Tracking -A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. Results are summarised alongside core latencies. +=== FN-004 OS Jitter Tracking + +A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). +Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. +Results are summarised alongside core latencies. + +=== FN-005 CSV Serialisation -=== 4.5 FN-005 CSV Serialisation Post-run, results can be persisted for offline analysis (`result.csv` by default). -=== 4.6 FN-006 Event Loop Integration +=== FN-006 Event Loop Integration + `eventLoopHandler(EventLoop)` allows benchmarks to operate inside Chronicle threading framework, avoiding extra threads. -== 5. Non-Functional Requirements +== Non-Functional Requirements + +=== NF-P-001 Performance -=== 5.1 NF-P-001 Performance * Overhead per sample must remain below 100 ns when no additional probes are active. * Histogram generation must support >=200 M iterations without heap pressure. -=== 5.2 NF-R-001 Reliability +=== NF-R-001 Reliability + * Harness must abort gracefully on thread interruptions or sample time-outs (`timeout(long)`). * Immutable result objects ensure thread-safe publication to external consumers. -=== 5.3 NF-UX-001 Usability +=== NF-UX-001 Usability + * Fluent builder API; sensible defaults provide a runnable benchmark in ≤10 LOC. * ASCII table outputs are human-readable and CI-friendly. -=== 5.4 NF-O-001 Portability +=== NF-O-001 Portability + * Pure-Java codebase; no native compilation steps. * JDK-specific optimisations (e.g., *Zing* support) are runtime-detected. -=== 5.5 NF-O-002 Maintainability +=== NF-O-002 Maintainability + * 80 %+ unit-test line coverage with deterministic fixtures. * Code adheres to Chronicle parent POM style and SonarCloud quality gates. -=== 5.6 NF-S-001 Security -No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code. +=== NF-S-001 Security -== 6. Licensing +No executable deserialisation; harness operates in-process. +Users remain responsible for securing benchmarked code. + +== Licensing The project is released under the *Apache Licence 2.0* (_see_ `LICENSE.adoc`). Downstream consumers must preserve licence notices and may include JLBH in commercial or OSS products, subject to the terms therein. -== 7. Glossary +== Glossary Co-ordinated Omission:: Statistical artefact causing under-reporting of worst-case latency. Histogram:: Data structure that records frequency counts in logarithmic buckets, enabling percentile extraction. Percentile:: Value below which a given percentage of observations fall (e.g., 99th percentile). -== 8. Appendix A - Example Minimal Benchmark +== Appendix A - Example Minimal Benchmark [source,java] ---- @@ -189,10 +214,15 @@ public class NothingBenchmark implements JLBHTask { } ---- -== 9. Appendix B - Footnotes - -* JLBH originated within the Chronicle Software open-source stack and is actively maintained. See https://github.com/OpenHFT/JLBH. -* The API reference highlights the focus on _co-ordinated omission_ and event-loop support. See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. -* A multi-part blog series gives practical guidance on designing realistic latency tests. See http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html. -* Vanilla-Java's micro-services article demonstrates embedding JLBH in an event-driven architecture. See https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html. -* Original discussion of co-ordinated omission by Gil Tene motivates JLBH's default settings. See https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE. +== Appendix B - Footnotes + +* JLBH originated within the Chronicle Software open-source stack and is actively maintained. +See https://github.com/OpenHFT/JLBH. +* The API reference highlights the focus on _co-ordinated omission_ and event-loop support. +See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. +* A multi-part blog series gives practical guidance on designing realistic latency tests. +See http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html. +* Vanilla-Java's micro-services article demonstrates embedding JLBH in an event-driven architecture. +See https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html. +* Original discussion of co-ordinated omission by Gil Tene motivates JLBH's default settings. +See https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE. diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/adoc/results-interpretation-guide.adoc index e2886473..830d00de 100644 --- a/src/main/adoc/results-interpretation-guide.adoc +++ b/src/main/adoc/results-interpretation-guide.adoc @@ -3,154 +3,213 @@ Chronicle Software :revnumber: 1.0 :revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB :icons: font -*_Abstract_*:: -This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. Effective interpretation is key to deriving meaningful insights from your latency benchmarks. - -== 1. Understanding JLBH Console Output Structure - -When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). Understanding this output is the first step: - -* **Warm-up Messages**: Indicates the progress and completion of the warm-up phase (e.g., "Warm up complete (X iterations took Y s)"). -* **Per-Run Detailed Output**: For each measurement run, JLBH prints a section: -** Header indicating the run number (e.g., "BENCHMARK RESULTS (RUN 1) us"). -** Run metadata: Total run time, latency distribution strategy, coordinated omission status, target throughput. -** **Probe Statistics**: For each active probe (the default "End to End" probe, any custom probes added via `jlbh.addProbe()`, and the "OS Jitter" probe if enabled), a line showing: -*** Probe name and total samples recorded for that probe in the run. -*** A compact representation of key percentiles (e.g., "50/90 99/99.9 99.99 - worst was X / Y Z / A B - C"). These values are usually in microseconds. -* **Final Summary Tables**: After all runs are complete, JLBH prints summary tables for each probe: -** Header: "SUMMARY (probe_name) us". -** Rows for key percentiles (50.0, 90.0, 99.0, 99.9, etc., up to 'worst'). -** Columns for each run, showing the latency value for that percentile in that run. -** A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). - -== 2. Interpreting Latency Histograms and Percentiles - -JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. These histograms are then used to derive percentile values. - -* **What is a Percentile?** -** A percentile indicates the value below which a given percentage of observations fall. For example, the 99th percentile latency is the value X where 99% of all measured latencies were less than or equal to X, and 1% were greater. -* **Key Percentiles in JLBH Output**: -** `50.0` (Median): The middle value; 50% of latencies are lower, 50% are higher. It gives a sense of the "typical" latency but can be misleading on its own if the distribution is skewed. +_Abstract_:: +This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. +Effective interpretation is key to deriving meaningful insights from your latency benchmarks. + +== Understanding JLBH Console Output Structure + +When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). +Understanding this output is the first step: + +Warm-up Messages :: +Indicates the progress and completion of the warm-up phase (e.g., "Warm up complete (X iterations took Y s)"). +Per-Run Detailed Output :: +For each measurement run, JLBH prints a section: +* Header indicating the run number (e.g., "BENCHMARK RESULTS (RUN 1) us"). +* Run metadata: Total run time, latency distribution strategy, coordinated omission status, target throughput. +* *Probe Statistics*: For each active probe (the default "End to End" probe, any custom probes added via `jlbh.addProbe()`, and the "OS Jitter" probe if enabled), a line showing: +** Probe name and total samples recorded for that probe in the run. +** A compact representation of key percentiles (e.g., "50/90 99/99.9 99.99 - worst was X / Y Z / A B - C"). +These values are usually in microseconds. +Final Summary Tables :: +After all runs are complete, JLBH prints summary tables for each probe: +* Header: "SUMMARY (probe_name) us". +* Rows for key percentiles (50.0, 90.0, 99.0, 99.9, etc., up to 'worst'). +* Columns for each run, showing the latency value for that percentile in that run. +* A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). + +== Interpreting Latency Histograms and Percentiles + +JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. +These histograms are then used to derive percentile values. + +* *What is a Percentile?* +** A percentile indicates the value below which a given percentage of observations fall. +For example, the 99th percentile latency is the value X where 99% of all measured latencies were less than or equal to X, and 1% were greater. +Key Percentiles in JLBH Output :: +** `50.0` (Median): The middle value; 50% of latencies are lower, 50% are higher. +It gives a sense of the "typical" latency but can be misleading on its own if the distribution is skewed. ** `90.0`: 90% of requests were faster than this value. -** `99.0` (99th or "p99"): A common metric for Service Level Objectives (SLOs). Indicates that 1 out of 100 requests experienced this latency or worse. +** `99.0` (99th or "p99"): A common metric for Service Level Objectives (SLOs). +Indicates that 1 out of 100 requests experienced this latency or worse. ** `99.9` (99.9th or "p999", "three nines"): Important for high-performance systems; 1 out of 1000 requests were this slow or slower. ** `99.99` (99.99th or "p9999", "four nines"): Critical for systems demanding very high consistency in performance. -** `worst`: The single highest latency recorded during the run. This can be affected by outliers but is important to note, especially if it's dramatically different from high percentiles. -* **Why Percentiles Matter**: +** `worst`: The single highest latency recorded during the run. +This can be affected by outliers but is important to note, especially if it's dramatically different from high percentiles. +Why Percentiles Matter :: ** Averages can be heavily skewed by a small number of very low or very high latencies, masking the true experience of most requests. -** Tail latencies (e.g., p99 and above) often drive user-perceived performance and system stability. A system with a good average latency but terrible tail latency can still feel unresponsive or unreliable. -* **Units**: Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. Pay attention to the units indicated in the table headers. -* **Custom Probes**: If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. +** Tail latencies (e.g., p99 and above) often drive user-perceived performance and system stability. +A system with a good average latency but terrible tail latency can still feel unresponsive or unreliable. +Units :: +Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. +Pay attention to the units indicated in the table headers. +Custom Probes :: +If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. +Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. -== 3. Co-ordinated Omission (CO) +== Co-ordinated Omission (CO) Co-ordinated Omission is a critical concept in latency measurement, especially for systems that process requests at a certain rate. -* **The Problem**: If a system (or the benchmark harness itself) pauses or slows down, it might temporarily stop measuring. When it resumes, it might only measure the (potentially fast) service times of requests that are processed immediately, "omitting" the time that other requests would have spent waiting *during* the pause. This leads to unrealistically optimistic latency figures, particularly at the tail. -* **How JLBH Compensates**: -** JLBH addresses this by default (`accountForCoordinatedOmission = true` in `JLBHOptions`). -** For each iteration, JLBH calculates an *ideal scheduled start time* based on the target throughput. -** It then actively waits (busy-spins using `System.nanoTime()`) until this ideal start time is reached before invoking `JLBHTask.run(startTimeNs)` with that ideal time. -** The latency is then calculated as `System.nanoTime() - idealStartTimeNs`. -** This means if the system or harness was delayed, the measured latency for that sample will correctly include that delay, reflecting the experience a request would have had if it arrived at its scheduled time. -* **Impact on Results**: -** *With CO compensation (default)*: Histograms reflect a more accurate picture of the latency a user or client would experience, including any queuing or delays caused by the system struggling to keep up with the offered load. Tails are usually higher and more realistic. -** *Without CO compensation*: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. This can be useful for understanding raw compute time but not overall system responsiveness under load. -* **Comparison**: Always state whether CO compensation was enabled when comparing benchmark results. Comparing a CO-compensated run with a non-compensated run is misleading. JLBH prints "Correcting for co-ordinated:true/false" in each run's output. - -== 4. OS Jitter Probe +The Problem :: +If a system (or the benchmark harness itself) pauses or slows down, it might temporarily stop measuring. +When it resumes, it might only measure the (potentially fast) service times of requests that are processed immediately, "omitting" the time that other requests would have spent waiting _during_ the pause. +This leads to unrealistically optimistic latency figures, particularly at the tail. +How JLBH Compensates :: +* JLBH addresses this by default (`accountForCoordinatedOmission = true` in `JLBHOptions`). +* For each iteration, JLBH calculates an _ideal scheduled start time_ based on the target throughput. +* It then actively waits (busy-spins using `System.nanoTime()`) until this ideal start time is reached before invoking `JLBHTask.run(startTimeNs)` with that ideal time. +* The latency is then calculated as `System.nanoTime() - idealStartTimeNs`. +* This means if the system or harness was delayed, the measured latency for that sample will correctly include that delay, reflecting the experience a request would have had if it arrived at its scheduled time. +Impact on Results :: +* _With CO compensation (default)_: Histograms reflect a more accurate picture of the latency a user or client would experience, including any queuing or delays caused by the system struggling to keep up with the offered load. +Tails are usually higher and more realistic. +* _Without CO compensation_: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. +This can be useful for understanding raw compute time but not overall system responsiveness under load. +Comparison :: +Always state whether CO compensation was enabled when comparing benchmark results. +Comparing a CO-compensated run with a non-compensated run is misleading. +JLBH prints "Correcting for co-ordinated:true/false" in each run's output. + +== OS Jitter Probe JLBH can optionally measure Operating System (OS) jitter, which refers to delays caused by the OS scheduler or other system activities interrupting the benchmark process. -* **Mechanism**: -** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. -** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. -* **Interpreting Jitter Output**: -** The "OS Jitter" probe will have its own percentile table in the JLBH output. -** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." -** *Low jitter values* (e.g., a few microseconds) indicate that the benchmark process was generally ableto run without significant external interference from the OS. -* **Why it's Useful**: -** Helps distinguish between latency caused by your application code versus latency caused by the environment. -** If your application shows high tail latencies, and the OS Jitter probe also shows high values around the same magnitude, it's a strong indicator that external factors are at play. -* **Actions for High Jitter**: -** Ensure the benchmark is running on a quiet machine with minimal other load. -** Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). -** Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). - -== 5. Analysing Run-to-Run Variation - -Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. JLBH's summary output includes a "% Variation" column for each percentile. - -* **The Variation Column**: This shows the percentage difference between the maximum and minimum values recorded for a given percentile across all included runs. -** By default (`JLBHOptions.skipFirstRun = NOT_SET`), if there are more than 3 runs, the first run is often excluded from this variation calculation as it might still be affected by late-stage JIT or other one-off initializations. You can control this explicitly with `skipFirstRun(boolean)`. -* **Interpreting Variation**: -** *Low Variation (e.g., < 5-10%)*: Suggests a stable benchmarking environment and that your application's performance is repeatable under the test conditions. This increases confidence in the results. -** *High Variation*: Indicates instability. Possible causes include: -*** **Insufficient Warm-up**: The JVM might still be optimizing code differently across runs. Try increasing `warmUpIterations`. -*** **Garbage Collection (GC) Pauses**: Infrequent but long GC pauses can hit some runs but not others, drastically affecting tail percentiles. Monitor GC activity. -*** **External System Noise**: Other processes on the machine, network fluctuations (if applicable), or even hardware power management. -*** **Adaptive JVM Behavior**: Some JVM optimizations are adaptive and can change behavior between runs if conditions shift slightly. -*** **Benchmarked Code Instability**: The code itself might have inherent variability or be interacting with unstable external resources. -* **Goal**: Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. Focus on understanding systematic performance rather than chasing the single "best" score from one run. Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. - -== 6. The Throughput vs. Latency Relationship +Mechanism :: +* Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. +* This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. +* If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +Interpreting Jitter Output :: +* The "OS Jitter" probe will have its own percentile table in the JLBH output. +* _High jitter values_ (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. +This is "machine noise." +* _Low jitter values_ (e.g., a few microseconds) indicate that the benchmark process was generally ableto run without significant external interference from the OS. +Why it's Useful :: +* Helps distinguish between latency caused by your application code versus latency caused by the environment. +* If your application shows high tail latencies, and the OS Jitter probe also shows high values around the same magnitude, it's a strong indicator that external factors are at play. +Actions for High Jitter :: +* Ensure the benchmark is running on a quiet machine with minimal other load. +* Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). +* Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). + +== Analysing Run-to-Run Variation + +Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. +JLBH's summary output includes a "% Variation" column for each percentile. + +The Variation Column :: +This shows the percentage difference between the maximum and minimum values recorded for a given percentile across all included runs. +* By default (`JLBHOptions.skipFirstRun = NOT_SET`), if there are more than 3 runs, the first run is often excluded from this variation calculation as it might still be affected by late-stage JIT or other one-off initializations. +You can control this explicitly with `skipFirstRun(boolean)`. +Interpreting Variation :: +* _Low Variation (e.g., < 5-10%)_: Suggests a stable benchmarking environment and that your application's performance is repeatable under the test conditions. +This increases confidence in the results. +* _High Variation_: Indicates instability. +Possible causes include: +** *Insufficient Warm-up*: The JVM might still be optimizing code differently across runs. +Try increasing `warmUpIterations`. +** *Garbage Collection (GC) Pauses*: Infrequent but long GC pauses can hit some runs but not others, drastically affecting tail percentiles. +Monitor GC activity. +** *External System Noise*: Other processes on the machine, network fluctuations (if applicable), or even hardware power management. +** *Adaptive JVM Behavior*: Some JVM optimizations are adaptive and can change behavior between runs if conditions shift slightly. +** *Benchmarked Code Instability*: The code itself might have inherent variability or be interacting with unstable external resources. +Goal :: +Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. +Focus on understanding systematic performance rather than chasing the single "best" score from one run. +Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. + +== The Throughput vs. Latency Relationship A fundamental aspect of performance is the trade-off between throughput (how much work is done per unit of time) and latency (how long each piece of work takes). -* **JLBH's Role**: JLBH allows you to set a target `throughput` via `JLBHOptions`. This controls the rate at which `JLBHTask.run()` is invoked. -* **Typical Behavior**: -** At low throughputs, a system can often maintain low latency. -** As throughput increases, contention for resources (CPU, memory, network, locks) typically rises. -** Beyond a certain point, latency will start to increase, often non-linearly. Queues may build up, and the system struggles to keep pace. -* **Using JLBH to Explore**: -** Run your benchmark with different `throughput` settings to understand how your application's latency responds to varying load. -** This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. -** The results can inform capacity planning and help set realistic performance expectations. - -== 7. Common Pitfalls in Interpretation +JLBH's Role :: +JLBH allows you to set a target `throughput` via `JLBHOptions`. +This controls the rate at which `JLBHTask.run()` is invoked. +Typical Behavior :: +* At low throughputs, a system can often maintain low latency. +* As throughput increases, contention for resources (CPU, memory, network, locks) typically rises. +* Beyond a certain point, latency will start to increase, often non-linearly. +Queues may build up, and the system struggles to keep pace. +Using JLBH to Explore :: +* Run your benchmark with different `throughput` settings to understand how your application's latency responds to varying load. +* This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. +* The results can inform capacity planning and help set realistic performance expectations. + +== Common Pitfalls in Interpretation Avoiding these common mistakes can help ensure your benchmark results are meaningful and your conclusions are sound: -* **Insufficient Warm-up**: -** *Issue*: The JVM performs JIT compilation, class loading, and cache warming during initial execution. Short or inadequate warm-up can mean you're measuring pre-optimized code or including one-off startup costs. -** *Solution*: Ensure `warmUpIterations` is sufficient. A common rule of thumb is at least the JVM's compilation threshold (e.g., `Jvm.compileThreshold()` in Chronicle Core, typically 10,000-15,000 iterations) plus some buffer for application-specific warm-up. Monitor if results stabilize after a certain number of runs. -* **Ignoring Co-ordinated Omission Setting**: -** *Issue*: Comparing results where one run had CO compensation enabled and another didn't will lead to incorrect conclusions about performance changes. -** *Solution*: Always be aware of the `accountForCoordinatedOmission` setting (default is `true`) and report it alongside your results. -* **Benchmark Overhead**: -** *Issue*: The act of measuring itself (calling `System.nanoTime()`, `jlbh.sample()`) has a small overhead. For extremely fast operations (deep sub-microsecond), this overhead can become a significant percentage of the measured time. -** *Solution*: JLBH is designed for low overhead (aiming for <100ns). Be mindful of this when benchmarking trivial operations. Consider batching very small operations if appropriate, or if the raw speed of an isolated, tiny operation is critical, a more specialized microbenchmarking tool like JMH might be used for that specific part, with JLBH testing the larger context. -* **Unrealistic Throughput Targets**: -** *Issue*: Setting `throughput` far beyond what the system can handle will just measure a system under constant overload, with latencies dominated by queue times. Setting it too low might not expose bottlenecks that appear under moderate load. -** *Solution*: Test a range of throughputs relevant to your expected operational load and capacity limits. -* **Focusing Only on Averages**: -** *Issue*: Averages hide outliers and don't reflect the experience of users hitting tail latencies. -** *Solution*: Prioritize percentile analysis (p50, p90, p99, p99.9, worst) as JLBH encourages. -* **Ignoring Custom Probe Data**: -** *Issue*: If you've set up custom probes for different stages of an operation but only look at the end-to-end numbers, you miss valuable insights into internal bottlenecks. -** *Solution*: Analyze the percentile data for each custom probe. Compare their contributions to the overall latency. -* **Environmental Inconsistency**: -** *Issue*: Running benchmarks on different hardware, with varying background OS/application load, different JVM versions, or different system configurations will yield different results. -** *Solution*: For comparative analysis, always use a consistent, controlled, and quiet environment. Document the environment meticulously. -* **Thread Pinning (Affinity) Issues**: -** *Issue*: On multi-core systems, OS thread scheduling can move your benchmark thread between cores, causing cache misses and jitter. Forgetting to pin, or pinning to a busy/shared core, can skew results. -** *Solution*: Use `JLBHOptions.acquireLock(Affinity::acquireLock)` (from OpenHFT Affinity library) to attempt to pin the main benchmark thread to an isolated core. Similarly for `jitterAffinity` if using the OS Jitter probe. Ensure these cores are genuinely isolated. -* **Not Enough Iterations/Run Duration**: -** *Issue*: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. -** *Solution*: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. This often means runs lasting at least several seconds, or even minutes for deep stability tests. - -== 8. Deriving Actionable Insights +Insufficient Warm-up :: +* _Issue_: The JVM performs JIT compilation, class loading, and cache warming during initial execution. +Short or inadequate warm-up can mean you're measuring pre-optimized code or including one-off startup costs. +* _Solution_: Ensure `warmUpIterations` is sufficient. +A common rule of thumb is at least the JVM's compilation threshold (e.g., `Jvm.compileThreshold()` in Chronicle Core, typically 10,000-15,000 iterations) plus some buffer for application-specific warm-up. +Monitor if results stabilize after a certain number of runs. +Ignoring Co-ordinated Omission Setting :: +* _Issue_: Comparing results where one run had CO compensation enabled and another didn't will lead to incorrect conclusions about performance changes. +* _Solution_: Always be aware of the `accountForCoordinatedOmission` setting (default is `true`) and report it alongside your results. +Benchmark Overhead :: +* _Issue_: The act of measuring itself (calling `System.nanoTime()`, `jlbh.sample()`) has a small overhead. +For extremely fast operations (deep sub-microsecond), this overhead can become a significant percentage of the measured time. +* _Solution_: JLBH is designed for low overhead (aiming for <100ns). +Be mindful of this when benchmarking trivial operations. +Consider batching very small operations if appropriate, or if the raw speed of an isolated, tiny operation is critical, a more specialized microbenchmarking tool like JMH might be used for that specific part, with JLBH testing the larger context. +Unrealistic Throughput Targets :: +* _Issue_: Setting `throughput` far beyond what the system can handle will just measure a system under constant overload, with latencies dominated by queue times. +Setting it too low might not expose bottlenecks that appear under moderate load. +* _Solution_: Test a range of throughputs relevant to your expected operational load and capacity limits. +Focusing Only on Averages :: +* _Issue_: Averages hide outliers and don't reflect the experience of users hitting tail latencies. +* _Solution_: Prioritize percentile analysis (p50, p90, p99, p99.9, worst) as JLBH encourages. +Ignoring Custom Probe Data :: +* _Issue_: If you've set up custom probes for different stages of an operation but only look at the end-to-end numbers, you miss valuable insights into internal bottlenecks. +* _Solution_: Analyze the percentile data for each custom probe. +Compare their contributions to the overall latency. +Environmental Inconsistency :: +* _Issue_: Running benchmarks on different hardware, with varying background OS/application load, different JVM versions, or different system configurations will yield different results. +* _Solution_: For comparative analysis, always use a consistent, controlled, and quiet environment. +Document the environment meticulously. +Thread Pinning (Affinity) Issues :: +* _Issue_: On multi-core systems, OS thread scheduling can move your benchmark thread between cores, causing cache misses and jitter. +Forgetting to pin, or pinning to a busy/shared core, can skew results. +* _Solution_: Use `JLBHOptions.acquireLock(Affinity::acquireLock)` (from OpenHFT Affinity library) to attempt to pin the main benchmark thread to an isolated core. +Similarly for `jitterAffinity` if using the OS Jitter probe. +Ensure these cores are genuinely isolated. +Not Enough Iterations/Run Duration :: +* _Issue_: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. +* _Solution_: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. +This often means runs lasting at least several seconds, or even minutes for deep stability tests. + +== Deriving Actionable Insights The ultimate goal of benchmarking is to gain insights that can lead to improvements. -* **Identify Bottlenecks**: -** High end-to-end latency? Use custom probes to break down the operation and see which stage is the culprit. -** High p99 or p99.9 values? This often points to issues like GC, network spikes, lock contention, or OS jitter. Correlate with OS Jitter probe and GC logs. -* **Validate Optimizations**: Run benchmarks before and after a code change to quantify its impact on latency across different percentiles and throughputs. -* **Capacity Planning**: Determine the throughput your system can handle while meeting latency SLOs. -* **Regression Testing**: Integrate JLBH benchmarks into your CI/CD pipeline to catch performance regressions automatically. -* **Understand Trade-offs**: Use JLBH to explore the impact of different configurations, algorithms, or architectural choices on the latency profile. +Identify Bottlenecks :: +* High end-to-end latency? +Use custom probes to break down the operation and see which stage is the culprit. +* High p99 or p99.9 values? +This often points to issues like GC, network spikes, lock contention, or OS jitter. +Correlate with OS Jitter probe and GC logs. +Validate Optimizations :: +Run benchmarks before and after a code change to quantify its impact on latency across different percentiles and throughputs. +Capacity Planning :: +Determine the throughput your system can handle while meeting latency SLOs. +Regression Testing :: +Integrate JLBH benchmarks into your CI/CD pipeline to catch performance regressions automatically. +Understand Trade-offs :: +Use JLBH to explore the impact of different configurations, algorithms, or architectural choices on the latency profile. diff --git a/src/main/config/checkstyle.xml b/src/main/config/checkstyle.xml new file mode 100644 index 00000000..844dd904 --- /dev/null +++ b/src/main/config/checkstyle.xml @@ -0,0 +1,210 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/main/config/pmd-exclude.properties b/src/main/config/pmd-exclude.properties new file mode 100644 index 00000000..9b5b06bd --- /dev/null +++ b/src/main/config/pmd-exclude.properties @@ -0,0 +1,9 @@ +# PMD exclusions with justifications +# Format: filepath=rule1,rule2 +# +# Example: +# net/openhft/jlbh/LegacyBenchmark.java=AvoidReassigningParameters,TooManyFields +net/openhft/chronicle/jlbh/JLBH.java=EmptyControlStatement +# JLBH-PMD-201: empty for-loop used to spin for GC warmup; keep structure +net/openhft/chronicle/jlbh/TeamCityHelper.java=UselessParentheses +# JLBH-PMD-202: lambda wrapping improves readability in TeamCity output helper diff --git a/src/main/config/spotbugs-exclude.xml b/src/main/config/spotbugs-exclude.xml new file mode 100644 index 00000000..2811fc6b --- /dev/null +++ b/src/main/config/spotbugs-exclude.xml @@ -0,0 +1,58 @@ + + + + + + + + + + JLBH-OPS-201: CLI mode preserves historical behaviour (stack traces, exit-on-error, exposing + percentile arrays). Refactor to soft-fail tracked under JLBH-OPS-260. + + + + + + + + JLBH-OPS-202: Pseudo-random generator only seeds synthetic load; cryptographic strength + unnecessary. + + + + + + + + JLBH-OPS-203: Deterministic jitter keeps benchmarks reproducible; no attacker input. + + + + + + + JLBH-OPS-204: Caller is the benchmark harness itself; returning backing arrays avoids millions of + allocations during tight loops. + + + + + + + + JLBH-OPS-205: Paths come from operator scripts; RuntimeException propagates IO failures without + changing the established API. + + + + + + + + + + diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java index 9170eb44..93a267aa 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java @@ -35,6 +35,7 @@ import java.util.concurrent.ConcurrentSkipListMap; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; import java.util.function.Consumer; import java.util.stream.Collectors; @@ -93,8 +94,7 @@ public class JLBH implements NanoSampler { private final AtomicBoolean abortTestRun = new AtomicBoolean(); private final long mod; private final long length; - // Todo: Remove all concurrent constructs such as volatile and AtomicBoolean - private volatile long noResultsReturned; + private final AtomicLong sampleCount = new AtomicLong(); //Use non-atomic when so thread synchronisation is necessary private boolean warmedUp; private volatile Thread testThread; @@ -146,8 +146,9 @@ public JLBH(@NotNull JLBHOptions jlbhOptions, @NotNull PrintStream printStream, : jlbhOptions.iterations > 50_000_000 ? 20_000_000_000L : jlbhOptions.iterations > 10_000_000 ? 10_000_000_000L : 5_000_000_000L; - long mod2; - for (mod2 = 1000; mod2 <= jlbhOptions.iterations / 200; mod2 *= 10) { + long mod2 = 1000; + while (mod2 <= jlbhOptions.iterations / 200) { + mod2 *= 10; } this.mod = mod2; } @@ -317,7 +318,7 @@ public void start() { * * @param startTimeNs the target time, in nanoseconds, to wait until * @return the actual timestamp returned from {@code nanoTime} once the - * wait is over + * wait is over */ private static long busyWaitUntil(long startTimeNs) { long nanoTime; @@ -339,7 +340,7 @@ private void startTimeoutCheckerIfRequired() { private void waitForWarmupToComplete(long warmupStart) { while (!warmUpComplete.get()) { Jvm.pause(500); - printStream.println("Complete: " + noResultsReturned); + printStream.println("Complete: " + sampleCount.get()); if (testThread.isInterrupted()) { return; } @@ -445,7 +446,7 @@ private void endOfRun(int run, long runStart) { jlbhOptions.jlbhTask.runComplete(); - noResultsReturned = 0; + sampleCount.set(0); additionHistograms.values().forEach(Histogram::reset); endToEndHistogram.reset(); osJitterMonitor.reset(); @@ -458,8 +459,9 @@ private void checkSampleTimeout() { while (true) { Jvm.pause(TimeUnit.SECONDS.toMillis(10)); - if (previousSampleCount < noResultsReturned) { - previousSampleCount = noResultsReturned; + long current = sampleCount.get(); + if (previousSampleCount < current) { + previousSampleCount = current; previousSampleTime = System.currentTimeMillis(); } else { if (previousSampleTime < (System.currentTimeMillis() - jlbhOptions.timeout)) { @@ -520,7 +522,7 @@ private void consumeResults() { * worst: 12.56 12.56 10.61 10.93 * ---- * - * + *

* Each percentile value is printed in microseconds with two decimal places. * The final column displays the percentage variation between the largest and * smallest values present in the row. The number of {@code run} columns is @@ -659,12 +661,12 @@ public void sampleNanos(long durationNs) { * @param durationNs latency in nanoseconds */ public void sample(long durationNs) { - noResultsReturned++; - if (noResultsReturned < jlbhOptions.warmUpIterations && !warmedUp) { + long current = sampleCount.incrementAndGet(); + if (current < jlbhOptions.warmUpIterations && !warmedUp) { endToEndHistogram.sample(durationNs); return; } - if (noResultsReturned == jlbhOptions.warmUpIterations && !warmedUp) { + if (current == jlbhOptions.warmUpIterations && !warmedUp) { warmedUp = true; endToEndHistogram.reset(); if (!additionHistograms.isEmpty()) { diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java index 471dbcba..9aef92ba 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java @@ -49,6 +49,7 @@ public class JLBHOptions { boolean jitterAffinity; Supplier acquireLock = Affinity::acquireLock; long timeout; + /** * Number of iterations per second to be pushed through the benchmark * diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHResult.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHResult.java index 083e2a9e..39852141 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHResult.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHResult.java @@ -56,7 +56,7 @@ public interface JLBHResult { * * @param probeName name of the probe for which results are requested * @return an {@code Optional} containing the probe results or - * {@link Optional#empty()} if the probe is not present + * {@link Optional#empty()} if the probe is not present */ @NotNull Optional probe(String probeName); @@ -185,7 +185,7 @@ interface RunResult { * {@code null} value will be returned.

* * @return duration of the 99.99th percentile or {@code null} when - * not available + * not available */ @Nullable Duration get9999thPercentile(); @@ -199,23 +199,41 @@ interface RunResult { Duration getWorst(); enum Percentile { - /** 50th percentile (median). */ + /** + * 50th percentile (median). + */ PERCENTILE_50TH, - /** 90th percentile. */ + /** + * 90th percentile. + */ PERCENTILE_90TH, - /** 99th percentile. */ + /** + * 99th percentile. + */ PERCENTILE_99TH, - /** 99.7th percentile. */ + /** + * 99.7th percentile. + */ PERCENTILE_99_7TH, - /** 99.9th percentile. */ + /** + * 99.9th percentile. + */ PERCENTILE_99_9TH, - /** 99.97th percentile. */ + /** + * 99.97th percentile. + */ PERCENTILE_99_97TH, - /** 99.99th percentile. */ + /** + * 99.99th percentile. + */ PERCENTILE_99_99TH, - /** 99.999th percentile. */ + /** + * 99.999th percentile. + */ PERCENTILE_99_999TH, - /** Highest (worst) observed latency. */ + /** + * Highest (worst) observed latency. + */ WORST } } diff --git a/src/main/java/net/openhft/chronicle/jlbh/LatencyDistributor.java b/src/main/java/net/openhft/chronicle/jlbh/LatencyDistributor.java index 047f9cc8..056d0b47 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/LatencyDistributor.java +++ b/src/main/java/net/openhft/chronicle/jlbh/LatencyDistributor.java @@ -29,9 +29,9 @@ public interface LatencyDistributor { * Adjust the delay between benchmark iterations. * * @param averageLatencyNS the nominal delay in nanoseconds derived from the - * configured throughput + * configured throughput * @return the actual number of nanoseconds to wait before the next - * iteration is executed + * iteration is executed */ long apply(long averageLatencyNS); } diff --git a/src/main/java/net/openhft/chronicle/jlbh/TeamCityHelper.java b/src/main/java/net/openhft/chronicle/jlbh/TeamCityHelper.java index 990bf1eb..842d2f8e 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/TeamCityHelper.java +++ b/src/main/java/net/openhft/chronicle/jlbh/TeamCityHelper.java @@ -73,13 +73,13 @@ public static void histo(@NotNull String name, @NotNull Histogram histo, @NotNul *

For each percentile a {@code buildStatisticValue} service message is * emitted so that TeamCity can record the values as build statistics.

* - * @param prefix prefix used to construct the TeamCity statistic key. The - * generated key has the form - * {@code prefix.<probe>.<percentile>} - * @param jlbh benchmark instance providing access to the percentile - * data for the last run - * @param iterations total number of iterations executed in the run; used to - * derive the set of percentiles that will be output + * @param prefix prefix used to construct the TeamCity statistic key. The + * generated key has the form + * {@code prefix.<probe>.<percentile>} + * @param jlbh benchmark instance providing access to the percentile + * data for the last run + * @param iterations total number of iterations executed in the run; used to + * derive the set of percentiles that will be output * @param printStream destination to which the TeamCity service messages are * written */ @@ -110,7 +110,7 @@ private static void printPercentiles(@NotNull String s, private static void printPercentiles(@NotNull String s, @NotNull PrintStream printStream, double[] percentages, double[] values) { PercentileSummary summary = new PercentileSummary(false, Collections.singletonList(values), percentages); String extra = Jvm.isAzulZing() ? ".zing" : Jvm.isJava15Plus() ? ".java17" : ""; - summary.forEachRow(((percentile, rowValues, variance) -> - printStream.println("##teamcity[buildStatisticValue key='" + s + "." + percentile + extra + "' value='" + rowValues[0] + "']"))); + summary.forEachRow((percentile, rowValues, variance) -> + printStream.println("##teamcity[buildStatisticValue key='" + s + "." + percentile + extra + "' value='" + rowValues[0] + "']")); } } diff --git a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java index 453b9519..8e49f030 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java +++ b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java @@ -5,9 +5,10 @@ import java.io.BufferedWriter; import java.io.IOException; -import java.io.PrintWriter; import java.io.Writer; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; +import java.nio.file.Path; import java.nio.file.Paths; import java.time.Duration; import java.util.Collections; @@ -30,7 +31,7 @@ * JLBHResult result = consumer.get(); * JLBHResultSerializer.runResultToCSV(result); * } - * + *

* which will create a {@code result.csv} file in the working directory. Other * overloads allow specifying the file name, the set of probes to export and * whether the OS jitter probe should be included. @@ -88,14 +89,15 @@ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, String * {@code namesOfProbes} a row is written if that probe exists. When * {@code includeOSJitter} is {@code true} the OS jitter probe is appended. * - * @param jlbhResult the benchmark result to serialise - * @param fileName path of the CSV file to create - * @param namesOfProbes additional probes to export - * @param includeOSJitter whether to include OS jitter metrics + * @param jlbhResult the benchmark result to serialise + * @param fileName path of the CSV file to create + * @param namesOfProbes additional probes to export + * @param includeOSJitter whether to include OS jitter metrics * @throws IOException if the file cannot be written */ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, Iterable namesOfProbes, boolean includeOSJitter) throws IOException { - try (Writer pw = new BufferedWriter(new PrintWriter(Files.newOutputStream(Paths.get(fileName))))) { + Path output = Paths.get(fileName); + try (BufferedWriter pw = Files.newBufferedWriter(output, StandardCharsets.UTF_8)) { writeHeader(pw); JLBHResult.ProbeResult probeResult = jlbhResult.endToEnd(); diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java index 6623825d..63c70a49 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java @@ -9,7 +9,8 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; public class JLBHAdditionalCoverageTest { /** diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java index 09da656e..e814d5f6 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java @@ -234,11 +234,11 @@ public void histogramSummariesAreCorrect() { System.out.println(baos); assertTrue(baos.toString().replace("\r", "").contains( "-------------------------------- SUMMARY (B) us ----------------------------------------------------\n" + - "Percentile run1 run2 run3 % Variation\n" + - "50.0: 0.10 0.10 0.10 0.00\n" + - "90.0: 0.10 0.10 0.10 0.00\n" + - "99.0: 0.10 0.10 0.10 0.00\n" + - "worst: 0.10 0.10 0.10 0.00")); + "Percentile run1 run2 run3 % Variation\n" + + "50.0: 0.10 0.10 0.10 0.00\n" + + "90.0: 0.10 0.10 0.10 0.00\n" + + "99.0: 0.10 0.10 0.10 0.00\n" + + "worst: 0.10 0.10 0.10 0.00")); } @Test diff --git a/src/test/java/net/openhft/chronicle/jlbh/PercentileSummaryTest.java b/src/test/java/net/openhft/chronicle/jlbh/PercentileSummaryTest.java index 7c6e3720..c7ca5b0a 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/PercentileSummaryTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/PercentileSummaryTest.java @@ -96,10 +96,10 @@ public void testThatVarianceIsCalculatedCorrectly() { final PercentileSummary percentileSummary = new PercentileSummary(false, percentileSummaries, percentiles); percentileSummary.printSummary(); - assertEquals((0.009 - 0.002) / (0.009 + 0.002 /2) * 100, percentileSummary.calculateVariance(0), DELTA); - assertEquals((0.009 - 0.003) / (0.009 + 0.003 /2) * 100, percentileSummary.calculateVariance(1), DELTA); - assertEquals((0.009 - 0.004) / (0.009 + 0.004 /2) * 100, percentileSummary.calculateVariance(2), DELTA); - assertEquals((0.009 - 0.005) / (0.009 + 0.005 /2) * 100, percentileSummary.calculateVariance(3), DELTA); + assertEquals((0.009 - 0.002) / (0.009 + 0.002 / 2) * 100, percentileSummary.calculateVariance(0), DELTA); + assertEquals((0.009 - 0.003) / (0.009 + 0.003 / 2) * 100, percentileSummary.calculateVariance(1), DELTA); + assertEquals((0.009 - 0.004) / (0.009 + 0.004 / 2) * 100, percentileSummary.calculateVariance(2), DELTA); + assertEquals((0.009 - 0.005) / (0.009 + 0.005 / 2) * 100, percentileSummary.calculateVariance(3), DELTA); assertEquals(0, percentileSummary.calculateVariance(percentiles.length - 2), DELTA); } @@ -119,9 +119,9 @@ public void testVarianceSkipFirst() { percentileSummary.printSummary(); // 50th percentile - assertEquals((0.009 - 0.003) / (0.009 + 0.003 /2) * 100, percentileSummary.calculateVariance(0), DELTA); + assertEquals((0.009 - 0.003) / (0.009 + 0.003 / 2) * 100, percentileSummary.calculateVariance(0), DELTA); // 90th has no value in the first run, so nothing to skip? - assertEquals((0.009 - 0.003) / (0.009 + 0.003 /2) * 100, percentileSummary.calculateVariance(1), DELTA); + assertEquals((0.009 - 0.003) / (0.009 + 0.003 / 2) * 100, percentileSummary.calculateVariance(1), DELTA); } @Test @@ -144,10 +144,10 @@ public void testForEachRow() { receivedValues.add(values); receivedVariances.add(variance); }); - assertArrayEquals(new Double[] {0.5, 0.9, 1.0}, receivedPercentiles.toArray(new Double[] {})); - assertArrayEquals(new double[] {0.002, 0.003}, receivedValues.get(0), DELTA); - assertArrayEquals(new double[] {POSITIVE_INFINITY, 0.003}, receivedValues.get(1), DELTA); - assertArrayEquals(new double[] {0.002, 0.003}, receivedValues.get(2), DELTA); - assertArrayEquals(new Double[] {25.0, 0.0, 25.0}, receivedVariances.toArray(new Double[] {})); + assertArrayEquals(new Double[]{0.5, 0.9, 1.0}, receivedPercentiles.toArray(new Double[]{})); + assertArrayEquals(new double[]{0.002, 0.003}, receivedValues.get(0), DELTA); + assertArrayEquals(new double[]{POSITIVE_INFINITY, 0.003}, receivedValues.get(1), DELTA); + assertArrayEquals(new double[]{0.002, 0.003}, receivedValues.get(2), DELTA); + assertArrayEquals(new Double[]{25.0, 0.0, 25.0}, receivedVariances.toArray(new Double[]{})); } } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java index 95e2a2b4..31807efd 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java @@ -35,7 +35,7 @@ public static void main(String[] args) { .recordOSJitter(true) .runs(4) .jlbhTask(new SimpleOSJitterBenchmark()); - new JLBH(lth,System.out, jlbhResult -> { + new JLBH(lth, System.out, jlbhResult -> { jlbhResult.osJitter().ifPresent(probeResult -> { JLBHResult.RunResult runResult = probeResult.summaryOfLastRun(); System.out.println("runResult = " + runResult); diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java index 84695357..b074da07 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java @@ -33,8 +33,8 @@ public static void main(String[] args) { .throughput(100_000) // .accountForCoordinatedOmission(true) .runs(2) - .jlbhTask(new SimpleResultSerializerBenchmark() ); - new JLBH(lth, System.out,jlbhResult -> { + .jlbhTask(new SimpleResultSerializerBenchmark()); + new JLBH(lth, System.out, jlbhResult -> { try { System.out.println("Serializing result..."); JLBHResultSerializer.runResultToCSV(jlbhResult); diff --git a/src/test/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializerTest.java b/src/test/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializerTest.java index 18acfe37..f62673ca 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializerTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializerTest.java @@ -12,7 +12,8 @@ import java.time.Duration; import java.util.*; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; public class JLBHResultSerializerTest {