diff --git a/AGENTS.md b/AGENTS.md index 8e1995a..166412d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,161 +1,8 @@ -# Guidance for AI agents, bots, and humans contributing to Chronicle Software's OpenHFT projects. - -LLM-based agents can accelerate development only if they respect our house rules. This file tells you: - -* how to run and verify the build; -* what *not* to comment; -* when to open pull requests. - -## Language & character-set policy - -| Requirement | Rationale | -|--------------|-----------| -| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the University of Oxford style guide for reference. | -| **ASCII-7 only** (code-points 0-127). Avoid smart quotes, non-breaking spaces and accented characters. | ASCII-7 survives every toolchain Chronicle uses, incl. low-latency binary wire formats that expect the 8th bit to be 0. | -| If a symbol is not available in ASCII-7, use a textual form such as `micro-second`, `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | - -## Javadoc guidelines - -**Goal:** Every Javadoc block should add information you cannot glean from the method signature alone. Anything else is -noise and slows readers down. - -| Do | Don't | -|----|-------| -| State *behavioural contracts*, edge-cases, thread-safety guarantees, units, performance characteristics and checked exceptions. | Restate the obvious ("Gets the value", "Sets the name"). | -| Keep the first sentence short; it becomes the summary line in aggregated docs. | Duplicate parameter names/ types unless more explanation is needed. | -| Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | -| Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | - -The principle that Javadoc should only explain what is *not* manifest from the signature is well-established in the -wider Java community. - -## Build & test commands - -Agents must verify that the project still compiles and all unit tests pass before opening a PR: - -```bash -# From repo root -mvn -q verify -``` - -## Commit-message & PR etiquette - -1. **Subject line <= 72 chars**, imperative mood: "Fix roll-cycle offset in `ExcerptAppender`". -2. Reference the JIRA/GitHub issue if it exists. -3. In *body*: *root cause -> fix -> measurable impact* (latency, allocation, etc.). Use ASCII bullet points. -4. **Run `mvn verify`** again after rebasing. - -## What to ask the reviewers - -* *Is this AsciiDoc documentation precise enough for a clean-room re-implementation?* -* Does the Javadoc explain the code's *why* and *how* that a junior developer would not be expected to work out? -* Are the documentation, tests and code updated together so the change is clear? -* Does the commit point back to the relevant requirement or decision tag? -* Would an example or small diagram help future maintainers? - -## Project requirements - -See the [Decision Log](src/main/adoc/decision-log.adoc) for the latest project decisions. -See the [Project Requirements](src/main/adoc/project-requirements.adoc) for details on project requirements. - -## Elevating the Workflow with Real-Time Documentation - -Building upon our existing Iterative Workflow, the newest recommendation is to emphasise *real-time updates* to documentation. -Ensure the relevant `.adoc` files are updated when features, requirements, implementation details, or tests change. -This tight loop informs the AI accurately and creates immediate clarity for all team members. - -### Benefits of Real-Time Documentation - -* **Confidence in documentation**: Accurate docs prevent miscommunications that derail real-world outcomes. -* **Reduced drift**: Real-time updates keep requirements, tests and code aligned. -* **Faster feedback**: AI can quickly highlight inconsistencies when everything is in sync. -* **Better quality**: Frequent checks align the implementation with the specified behaviour. -* **Smoother onboarding**: Up-to-date AsciiDoc clarifies the system for new developers. -* **Incremental changes**: AIDE flags newly updated files so you can keep the documentation synchronised. - -### Best Practices - -* **Maintain Sync**: Keep documentation (AsciiDoc), tests, and code synchronised in version control. Changes in one area should prompt reviews and potential updates in the others. -* **Doc-First for New Work**: For *new* features or requirements, aim to update documentation first, then use AI to help produce or refine corresponding code and tests. For refactoring or initial bootstrapping, updates might flow from code/tests back to documentation, which should then be reviewed and finalised. -* **Small Commits**: Each commit should ideally relate to a single requirement or coherent change, making reviews easier for humans and AI analysis tools. -- **Team Buy-In**: Encourage everyone to review AI outputs critically and contribute to maintaining the synchronicity of all artefacts. - -## AI Agent Guidelines - -When using AI agents to assist with development, please adhere to the following guidelines: - -* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ASCII-7 guidelines outlined above. - Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. -* **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. -* **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. - -## Company-Wide Tagging - -This section records **company-wide** decisions that apply to *all* Chronicle projects. All identifiers use the --xxx prefix. The `xxx` are unique across in the same Scope even if the tags are different. Component-specific decisions live in their xxx-decision-log.adoc files. - -### Tag Taxonomy (Nine-Box Framework) - -To improve traceability, we adopt the Nine-Box taxonomy for requirement and decision identifiers. These tags are used in addition to the existing ALL prefix, which remains reserved for global decisions across every project. - -.Adopt a Nine-Box Requirement Taxonomy - -|Tag | Scope | Typical examples | -|----|-------|------------------| -|FN |Functional user-visible behaviour | Message routing, business rules | -|NF-P |Non-functional - Performance | Latency budgets, throughput targets | -|NF-S |Non-functional - Security | Authentication method, TLS version | -|NF-O |Non-functional - Operability | Logging, monitoring, health checks | -|TEST |Test / QA obligations | Chaos scenarios, benchmarking rigs | -|DOC |Documentation obligations | Sequence diagrams, user guides | -|OPS |Operational / DevOps concerns | Helm values, deployment checklist | -|UX |Operator or end-user experience | CLI ergonomics, dashboard layouts | -|RISK |Compliance / risk controls | GDPR retention, audit trail | - -`ALL-*` stays global, case-exact tags. Pick one primary tag if multiple apply. - -### Decision Record Template - -```asciidoc -=== [Identifier] Title of Decision - -Date:: YYYY-MM-DD -Context:: -* What is the issue that this decision addresses? -* What are the driving forces, constraints, and requirements? -Decision Statement:: -* What is the change that is being proposed or was decided? -Alternatives Considered:: -* [Alternative 1 Name/Type]: -** *Description:* Brief description of the alternative. -** *Pros:* ... -** *Cons:* ... -* [Alternative 2 Name/Type]: -** *Description:* Brief description of the alternative. -** *Pros:* ... -** *Cons:* ... -Rationale for Decision:: -* Why was the chosen decision selected? -* How does it address the context and outweigh the cons of alternatives? -Impact & Consequences:: -* What are the positive and negative consequences of this decision? -* How does this decision affect the system, developers, users, or operations? -- What are the trade-offs made? -Notes/Links:: -** (Optional: Links to relevant issues, discussions, documentation, proof-of-concepts) -``` - -## Asciidoc formatting guidelines - -### List Indentation - -Do not rely on indentation for list items in AsciiDoc documents. Use the following pattern instead: - -```asciidoc -section:: Top Level Section -* first level - ** nested level -``` - -### Emphasis and Bold Text - -In AsciiDoc, an underscore `_` is _emphasis_; `*text*` is *bold*. +# Chronicle JLBH AGENTS + +- Follow repository `AGENTS.md` as the base rules; this file adds JLBH specifics. Durable docs live in `src/main/docs/` with the landing page at `README.adoc`. +- Module purpose: Java Latency Benchmark Harness for running context-aware latency/throughput benchmarks with Chronicle-friendly patterns. +- Build commands: full build `mvn -q clean verify`; module-only without tests `mvn -pl JLBH -am -DskipTests install`. +- Quality gates: keep Checkstyle/SpotBugs clean; ensure benchmarks remain deterministic and reproducible; guard against benchmark harness changes that alter measurement accuracy. +- Documentation: maintain Nine-Box IDs in `src/main/docs/project-requirements.adoc` and link decisions/tests/benchmarks to them; British English, ASCII/ISO-8859-1, `:source-highlighter: rouge`. +- Guardrails: avoid introducing benchmark code that masks coordinated omission; document any new CLI options or environment knobs in the docs. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..c2935ba --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,219 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +JLBH (Java Latency Benchmark Harness) is a specialised benchmarking library for measuring end-to-end latency in Java applications under realistic throughput conditions. Unlike JMH micro-benchmarks, JLBH is designed to benchmark code "in context" and account for coordinated omission. It is particularly suited for producer/consumer workloads where iteration start and completion may occur on different threads. + +## Build & Test Commands + +```bash +# Verify build and run all tests +mvn -q verify + +# Clean build from scratch +mvn -q clean verify + +# Run a specific test class +mvn -q test -Dtest=JLBHTest + +# Run a single test method +mvn -q test -Dtest=JLBHTest#shouldProvideResultData + +# Compile only (skip tests) +mvn -q compile + +# Skip tests during verify +mvn -q verify -DskipTests +``` + +## Language & Character-Set Requirements + +**Critical**: All code and documentation must use: +- **British English** spelling (`organisation`, `licence`) except for Java keywords like `synchronized` +- **ISO-8859-1** character encoding only (code points 0-255) +- **No Unicode characters**: Use ASCII equivalents like `>=` instead of special symbols +- Verify with: `iconv -f ascii -t ascii ` + +See AGENTS.md for complete language policy and rationale. + +## Architecture Overview + +### Core Components + +**JLBH** (src/main/java/net/openhft/chronicle/jlbh/JLBH.java) +- The main orchestrator that manages the entire benchmark lifecycle +- Runs on a single thread (annotated `@SingleThreaded`) +- Provides `sample(long durationNs)` to record end-to-end latencies +- Provides `addProbe(String)` to create additional `NanoSampler` instances for measuring sub-stages +- Can run standalone via `start()` or integrate with Chronicle EventLoop via `eventLoopHandler(EventLoop)` + +**JLBHOptions** (src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java) +- Builder-style configuration object defining all benchmark parameters +- Key settings: `throughput()`, `iterations()`, `runs()`, `warmUpIterations()`, `accountForCoordinatedOmission()`, `recordOSJitter()` +- Default throughput: 10,000 ops/sec; default runs: 3 + +**JLBHTask** (src/main/java/net/openhft/chronicle/jlbh/JLBHTask.java) +- User-implemented interface defining the workload to benchmark +- Lifecycle methods called by JLBH: + - `init(JLBH)`: Setup phase, register probes + - `run(long startTimeNs)`: Execute benchmark iteration (called for each iteration) + - `warmedUp()`: Notification after warmup completes + - `runComplete()`: Notification after each run + - `complete()`: Final cleanup + +**Histograms & Results** +- Each probe (end-to-end, custom probes, OS jitter) records samples into a `Histogram` from chronicle-core +- Histograms use logarithmic bucketing with 35-bit range and 8 significant figures +- Immutable result objects: `JLBHResult`, `ProbeResult`, `RunResult` +- Use `ThreadSafeJLBHResultConsumer` to retrieve results from another thread + +### Threading Model + +- **Single-threaded harness**: All JLBHTask lifecycle methods run on the single JLBH thread +- **User code may spawn threads**: The benchmarked code within `JLBHTask.run()` can be multi-threaded +- **Sample recording thread-safety**: Only the JLBH harness thread should call `jlbh.sample()` or probe samplers +- **Event loop integration**: Via `eventLoopHandler()` instead of `start()` (requires coordinated omission accounting enabled) + +### Execution Flow + +1. **Initialization**: User configures `JLBHOptions`, creates `JLBH` instance, calls `JLBHTask.init()` +2. **Warm-up**: Runs `warmUpIterations` times to allow JIT compilation (default ~12k iterations) +3. **Measurement runs**: Executes `runs` times (default 3), each with `iterations` samples +4. **Completion**: Prints summary, constructs immutable `JLBHResult`, calls `JLBHTask.complete()` + +See src/main/adoc/benchmark-lifecycle.adoc for visual flow diagram. + +### Coordinated Omission + +JLBH accounts for coordinated omission by default (`accountForCoordinatedOmission = true`): +- `startTimeNs` passed to `run()` is the *calculated ideal start time*, not `System.nanoTime()` +- Harness busy-waits if current time is before scheduled start time +- Disabling this feature means start time is simply based on throughput without waiting + +### OS Jitter Monitoring + +Enabled by default via `recordOSJitter(true)`: +- Background thread `OSJitterMonitor` repeatedly samples `System.nanoTime()` to detect scheduler delays +- Records jitter > `recordJitterGreaterThanNs` (default: 1000ns) into separate histogram +- Incurs overhead; disable with `recordOSJitter(false)` for minimal-overhead benchmarks + +## Key Design Patterns + +**Probes for Multi-Stage Benchmarks** +```java +class MyTask implements JLBHTask { + private JLBH jlbh; + private NanoSampler stage1Probe; + + public void init(JLBH jlbh) { + this.jlbh = jlbh; + this.stage1Probe = jlbh.addProbe("stage1"); + } + + public void run(long startTimeNs) { + long stage1Start = System.nanoTime(); + // ... stage 1 work ... + stage1Probe.sampleNanos(System.nanoTime() - stage1Start); + + // ... remaining work ... + jlbh.sample(System.nanoTime() - startTimeNs); + } +} +``` + +**Typical Benchmark Setup** +```java +JLBHOptions options = new JLBHOptions() + .warmUpIterations(100_000) + .iterations(1_000_000) + .throughput(50_000) + .runs(3) + .jlbhTask(myTask); +new JLBH(options).start(); +``` + +**Thread-Safe Result Retrieval** +```java +JLBHResultConsumer resultConsumer = JLBHResultConsumer.newThreadSafeInstance(); +JLBH jlbh = new JLBH(options, System.out, resultConsumer); +// Run on separate thread +new Thread(() -> jlbh.start()).start(); +// Retrieve results from main thread after completion +JLBHResult result = resultConsumer.get(); +``` + +## Documentation Structure + +All requirements and design decisions are in src/main/adoc/: +- **project-requirements.adoc**: Formal requirements specification with tagged requirements (FN-xxx, NF-P-xxx, etc.) +- **decision-log.adoc**: Architectural decision records (ADRs) following Nine-Box taxonomy +- **architecture.adoc**: High-level architecture, components, execution flow, threading model +- **benchmark-lifecycle.adoc**: Visual diagram of benchmark execution phases +- **jlbh-cookbook.adoc**: Common usage patterns and recipes +- **results-interpretation-guide.adoc**: How to interpret percentile output + +When making changes: +1. Update relevant .adoc files first (if changing requirements/design) +2. Update code and tests +3. Verify all three stay synchronised in same commit + +## Javadoc Standards + +Follow guidelines from AGENTS.md: +- Document *behavioural contracts*, edge cases, thread-safety, units, performance characteristics +- Do NOT restate the obvious ("Gets the value") +- First sentence must be concise (becomes summary line) +- Remove autogenerated Javadoc for trivial getters/setters +- Explain *why* and *how*, not just *what* + +## Test Examples + +Test files in src/test/java/net/openhft/chronicle/jlbh/ demonstrate usage: +- `ExampleJLBHMain`: Basic command-line harness demonstration +- `SimpleBenchmark`: Minimal benchmark example +- `SimpleOSJitterBenchmark`: Demonstrates OS jitter recording and custom probes +- `JLBHTest`: Unit test showing programmatic result extraction +- `JLBHIntegrationTest`: Example integration test + +## Performance Requirements + +From project-requirements.adoc (NF-P requirements): +- Overhead per sample: < 100 ns when no additional probes active +- Histogram generation: Support >= 200M iterations without heap pressure +- Warm-up rule of thumb: ~30% of run iterations (default uses JIT compile threshold * 6/5) + +## Dependencies + +Key dependencies from pom.xml: +- **chronicle-core**: Provides `Histogram`, `NanoSampler`, threading utilities +- **affinity**: CPU affinity control via `AffinityLock` +- **chronicle-threads**: Event loop integration (test scope) + +## Common Gotchas + +- JLBH is `@SingleThreaded` - the harness itself runs on one thread +- The `startTimeNs` parameter to `run()` is NOT `System.nanoTime()` - it's the calculated ideal start time +- Warm-up iterations use ~12k by default; adjust with `warmUpIterations()` if benchmark takes long to stabilise +- OS jitter monitoring is enabled by default and adds overhead; explicitly disable if needed +- When using `eventLoopHandler()`, coordinated omission must remain enabled +- CSV export: Use `JLBHResultSerializer.runResultToCSV(jlbhResult)` (writes to `result.csv` by default) + +## Non-Functional Quality Attributes + +From README.adoc summary: +- **Performance**: < 100 ns overhead per sample, >= 200M iterations supported +- **Reliability**: Graceful abort on interruption/timeout, immutable thread-safe results +- **Usability**: Fluent API, runnable in <= 10 LOC, human-readable ASCII output +- **Portability**: Pure Java, runtime-detected JDK optimisations +- **Maintainability**: >= 80% test coverage (current: ~83% line, ~78% branch per pom.xml) + +## Commit & PR Guidelines + +From AGENTS.md: +- Subject line <= 72 chars, imperative mood +- Body: root cause -> fix -> measurable impact +- Run `mvn -q verify` before opening PR +- Keep PRs focused; avoid bundling unrelated changes +- Re-run build after addressing review comments diff --git a/LICENSE.adoc b/LICENSE.adoc index 4c32dfd..6d68b90 100644 --- a/LICENSE.adoc +++ b/LICENSE.adoc @@ -1,5 +1,8 @@ == Copyright 2016-2025 chronicle.software +:toc: +:lang: en-GB +:source-highlighter: rouge Licensed under the *Apache License, Version 2.0* (the "License"); you may not use this file except in compliance with the License. @@ -11,4 +14,4 @@ Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. \ No newline at end of file +limitations under the License. diff --git a/README.adoc b/README.adoc index 12c47f8..337c35e 100644 --- a/README.adoc +++ b/README.adoc @@ -1,9 +1,15 @@ = Chronicle JLBH +:toc: +:lang: en-GB +:source-highlighter: rouge Chronicle Software -:css-signature: demo +:toc: :toc: macro -:toclevels: 2 :icons: font +:lang: en-GB +:toclevels: 2 +:css-signature: demo +:source-highlighter: rouge image:https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh] image:https://javadoc.io/badge2/net.openhft/JLBH/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-wire/latest/index.html"] @@ -17,12 +23,12 @@ toc::[] == About -Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a microbenchmark. +Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a micro-benchmark. See <> for a series of articles introducing JLBH. -An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles.] -link:src/main/adoc/project-requirements.adoc[The requirements document] contains detailed feature descriptions. +An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles]. +link:src/main/docs/project-requirements.adoc[The requirements document] contains detailed feature descriptions. -For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_7-glossary[the Glossary (section 7)]. +For terminology used throughout the project, see link:src/main/docs/project-requirements.adoc#_7-glossary[the Glossary (section 7)]. Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread. To do this, use the JLBH.eventLoopHandler method rather than JLBH.start. @@ -79,7 +85,7 @@ Commonly used option methods include: For a full list of configuration parameters see `src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java`. -For a visual overview of how a benchmark progresses, see the link:src/main/adoc/benchmark-lifecycle.adoc[benchmark lifecycle diagram]. +For a visual overview of how a benchmark progresses, see the link:src/main/docs/benchmark-lifecycle.adoc[benchmark lifecycle diagram]. == Additional Features @@ -127,10 +133,10 @@ This writes the output to `result.csv` by default, as defined in `JLBHResultSeri == Environment and Configuration -CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/adoc/project-requirements.adoc[requirements document]. +CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/docs/project-requirements.adoc[requirements document]. Histogramming of recorded latencies is precise. -As specified in the link:src/main/adoc/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. +As specified in the link:src/main/docs/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits. This accuracy retains sub-nanosecond resolution over a wide range, so the reported percentiles closely reflect true end-to-end timings. == Sample Benchmarks and Tests diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..5b0afd7 --- /dev/null +++ b/TODO.md @@ -0,0 +1,270 @@ +# JLBH - Repository TODO + +**📋 Part of:** [Chronicle Architecture Documentation](../ARCH_TODO.md) +**Module Layer:** Infrastructure (Benchmarking) +**Priority:** 🟢 P3 +**Last Updated:** 2025-11-16 + +## Purpose + +This TODO file tracks work specific to JLBH that feeds into the master [ARCH_TODO.md](../ARCH_TODO.md). It helps break down the architecture documentation work into manageable, repository-specific chunks. + +## Related Main TODO Files + +- [../ARCH_TODO.md](../ARCH_TODO.md) - Master architecture documentation roadmap +- [../TODO_INDEX.md](../TODO_INDEX.md) - Index of all TODO files +- [../ADOC_TODO.md](../ADOC_TODO.md) - AsciiDoc standardization (affects this module) + +## Module Information for Architecture Overview + +### Basic Information +- [x] **Module Name:** JLBH +- [x] **Maven Artifact ID:** jlbh +- [x] **Primary Purpose:** Provide a Java latency benchmark harness for measuring end-to-end response times under realistic workloads rather than micro-benchmarks. +- [x] **Layer in Chronicle Stack:** Infrastructure (Benchmarking and performance analysis) +- [x] **Dependencies (Chronicle modules):** `affinity`, `chronicle-core`, `chronicle-threads` (test-only) +- [x] **Key Classes/Interfaces:** `JLBH`, `JLBHOptions`, `JLBHTask`, `JLBHResult`, `JLBHResultConsumer` + +### ISO Alignment and Trust Zone + +- [x] **Trust zone identified (Edge/Core/Foundation):** JLBH is a *Foundation (Zone C)* benchmarking and latency measurement tool used to characterise Chronicle components rather than process untrusted production traffic directly. +- [x] **Shared standards reviewed:** Align JLBH documentation with the shared standards in `Chronicle-Quality-Rules/src/main/docs`, focusing on its role in validating latency budgets and performance envelopes rather than enforcing security policies. + +### Architecture Information for ARCH_TODO.md Stage 3 + +**Feeds into:** ARCH_TODO.md Stage 3 - Module Deep Dives (ARCH-MOD-JLBH) + +- [ ] [P2] [E:M] **Core Abstractions:** [List primary abstractions this module provides] +- [ ] [P2] [E:M] **Interactions with other modules:** [Which Chronicle modules does this use/integrate with?] +- [ ] [P2] [E:M] **Typical use cases:** [List 2-3 common scenarios where this module is used] +- [ ] [P2] [E:M] **Performance characteristics:** [Key performance metrics if applicable] +- [ ] [P2] [E:M] **Design patterns used:** [e.g., flyweight, single writer, etc.] + +### Existing Documentation Audit + +- [ ] [P2] [E:S] Check if `src/main/docs/architecture-overview.adoc` exists + - [ ] If yes: Review quality (compare to Chronicle-Bytes standard) + - [ ] If no: Note as gap for ARCH_TODO Stage 5.5 +- [ ] [P2] [E:S] Check if `src/main/docs/project-requirements.adoc` exists + - [ ] If yes: Review for ARCH_TODO Stage 1.75 (Requirements Overview) + - [ ] If no: Note as gap for FUNC_TODO.md +- [ ] [P2] [E:S] Check if `src/main/docs/decision-log.adoc` exists + - [ ] If yes: Review for ARCH_TODO Stage 1.85 (Decision Log Overview) + - [ ] If no: Note as gap for DECISION_TODO.md +- [ ] [P2] [E:S] Check if `README.adoc` provides good module overview +- [ ] [P2] [E:S] Check if `AGENTS.md` exists and follows canonical template + +### Documentation Gaps (for ARCH_TODO Stage 5.5) + +**Missing Documentation:** +- [ ] Architecture overview? [Y/N] +- [ ] Requirements documentation? [Y/N] +- [ ] Decision log? [Y/N] +- [ ] Security review? [Y/N] +- [ ] Testing strategy? [Y/N] +- [ ] Performance targets? [Y/N] + +**Documentation Quality Issues:** +- [ ] Missing `:toc:`, `:lang: en-GB`, or `:source-highlighter: rouge`? +- [ ] Manual section numbering instead of `:sectnums:`? +- [ ] Broken cross-references? +- [ ] Outdated information? + +## Requirements for Architecture Overview (ARCH_TODO Stage 1.75) + +**Feeds into:** Requirements Overview consolidation + +- [ ] [P1] [E:L] **Identify key functional requirements:** [List 3-5 most important] +- [ ] **Identify key non-functional requirements:** + - [ ] [P1] [E:M] Performance targets: [e.g., latency, throughput] + - [ ] [P1] [E:M] Security obligations: [e.g., bounds checking, input validation] + - [ ] [P1] [E:M] Operability requirements: [e.g., monitoring, logging] +- [ ] [P1] [E:L] **Map requirements to architecture patterns:** [How do requirements drive design?] + +## Decisions for Architecture Overview (ARCH_TODO Stage 1.85) + +**Feeds into:** Decision Log Overview consolidation + +- [ ] [P1] [E:M] **Identify key architectural decisions:** [List 2-4 major decisions] + - [ ] Decision ID (if in decision-log.adoc): + - [ ] Brief description: + - [ ] Rationale: + - [ ] Alternatives considered: +- [ ] **Identify decision patterns used:** + - [ ] [P1] [E:S] Off-heap memory? [Y/N - explain] + - [ ] [P1] [E:S] Single writer principle? [Y/N - explain] + - [ ] [P1] [E:S] Reference counting? [Y/N - explain] + - [ ] [P1] [E:S] Flyweight pattern? [Y/N - explain] + +## Glossary Terms (ARCH_TODO Stage 1.5) + +**Feeds into:** Cross-module glossary + +- [ ] [P2] [E:S] **Module-specific terms to include in glossary:** + - [ ] Term 1: [Definition] + - [ ] Term 2: [Definition] + - [ ] [Add more as needed] + +## ISO 9001 Quality Management Considerations + +**Reference:** [../COMPLIANCE_QUICK_REFERENCE.md](../COMPLIANCE_QUICK_REFERENCE.md) + +### Design Inputs (ISO 9001 Clause 8.3.3) +- [ ] **Functional requirements documented?** + - [ ] Location: `src/main/docs/project-requirements.adoc` + - [ ] Requirements use Nine-Box taxonomy? (JLBH-FN-NNN) + - [ ] Requirements are testable and verifiable? +- [ ] **Non-functional requirements documented?** + - [ ] Performance requirements (JLBH-NF-P-NNN) + - [ ] Security requirements (JLBH-NF-S-NNN) + - [ ] Operability requirements (JLBH-NF-O-NNN) + +### Design Outputs (ISO 9001 Clause 8.3.5) +- [ ] **Architecture documented?** + - [ ] Location: `src/main/docs/architecture-overview.adoc` + - [ ] Describes key components and their interactions? + - [ ] Includes interface specifications? +- [ ] **APIs and interfaces specified?** + - [ ] Public API documented (JavaDoc)? + - [ ] Integration points with other modules described? + +### Design Verification (ISO 9001 Clause 8.3.4) +- [ ] **Requirements traceable to tests?** + - [ ] Test classes reference requirement IDs in comments/docs? + - [ ] Coverage: What % of requirements have corresponding tests? +- [ ] **Test strategy documented?** + - [ ] Unit test approach + - [ ] Integration test approach + - [ ] Performance test approach (if applicable) +- [ ] **Code review evidence?** + - [ ] PR review process followed? + - [ ] Review comments addressed? + +### Design Changes (ISO 9001 Clause 8.3.4) +- [ ] **Architectural decisions documented?** + - [ ] Location: `src/main/docs/decision-log.adoc` + - [ ] Decisions include context, alternatives, rationale? + - [ ] Impact of changes assessed? +- [ ] **Change history maintained?** + - [ ] Git commit messages describe rationale? + - [ ] Breaking changes documented in release notes? + +## ISO 27001 Information Security Considerations + +**Reference:** [../ARCHITECTURE_RESEARCH_GUIDE.md](../ARCHITECTURE_RESEARCH_GUIDE.md) - Security Research Topics + +### Secure Coding (ISO 27001 Control A.8.28) +- [ ] **Input validation implemented?** + - [ ] Where are untrusted inputs received? [List entry points] + - [ ] How are malformed inputs handled? + - [ ] Size limits enforced? +- [ ] **Bounds checking implemented?** + - [ ] Buffer overflow prevention mechanisms? + - [ ] Array access validation? + - [ ] Off-heap memory bounds checked? +- [ ] **Static analysis performed?** + - [ ] Checkstyle violations reviewed? + - [ ] SpotBugs security patterns checked? + - [ ] Suppressions justified and documented? + +### Access Control (ISO 27001 Control A.8.3) +- [ ] **Access restrictions implemented?** + - [ ] Are there authentication/authorization mechanisms? [Y/N] + - [ ] If yes, where and how are they implemented? + - [ ] Principle of least privilege followed? +- [ ] **Privileged operations identified?** + - [ ] Which operations require elevated privileges? + - [ ] How are they protected? + +### Cryptographic Controls (ISO 27001 Control A.8.24) +- [ ] **Cryptography usage identified?** + - [ ] Is encryption used? [Y/N - where?] + - [ ] Is hashing used? [Y/N - which algorithms?] + - [ ] Is TLS/SSL used? [Y/N - configuration?] +- [ ] **Key management?** + - [ ] How are cryptographic keys managed? + - [ ] Are keys hardcoded? [Y/N - if yes, flag as risk] + +### Network Security (ISO 27001 Control A.8.22) +- [ ] **Network communication security?** + - [ ] Does this module communicate over network? [Y/N] + - [ ] If yes, is communication encrypted? + - [ ] How are network endpoints authenticated? +- [ ] **Network configuration?** + - [ ] Secure defaults configured? + - [ ] Insecure protocols disabled? + +### Vulnerability Management (ISO 27001 Control A.8.8) +- [ ] **Known vulnerabilities?** + - [ ] Any open security issues in GitHub? + - [ ] Any CVEs against dependencies? +- [ ] **Security testing?** + - [ ] Fuzz testing performed? + - [ ] Security-specific test cases? + - [ ] Penetration testing performed? + +### Security Documentation +- [ ] **Security review documented?** + - [ ] Location: `src/main/docs/security-review.adoc` + - [ ] Threat model documented? + - [ ] Security controls described? + - [ ] Known limitations documented? + +## Automation Tasks + +- [x] [P1] [E:S] Create `scripts/generate-progress.sh` to track TODO completion. +- [x] [P1] [E:S] Create `scripts/housekeeping.sh` to archive completed tasks. + +## Improvement Tasks (ARCH_TODO Stage 5.5) + +**Feeds into:** Improve Existing Module Documentation + +### High Priority +- [ ] [P1] [E:M] Create missing architecture-overview.adoc (if needed) +- [x] [P1] [E:S] Add missing front-matter to existing docs +- [ ] [P1] [E:M] Fix broken cross-references +- [x] [P1] [E:S] Add `:sectnums:` where appropriate + +### Medium Priority +- [ ] [P2] [E:L] Expand brief architecture docs (if < 75 lines) +- [ ] [P2] [E:L] Add "Trade-offs and Alternatives" section (following Chronicle-Bytes pattern) +- [ ] [P2] [E:M] Add performance characteristics section +- [ ] [P2] [E:M] Create decision log entries for undocumented decisions + +### Low Priority +- [ ] [P3] [E:L] Add diagrams (PlantUML or draw.io) +- [ ] [P3] [E:M] Create example code snippets +- [ ] [P3] [E:L] Expand requirements documentation +- [ ] [P3] [E:M] Add cross-references to other module docs + +## Code Quality Tasks + +**Reference:** [../QUALITY_PLAYBOOK.md](../QUALITY_PLAYBOOK.md) + +- [x] Run Checkstyle scan and document violations + - Latest command: `mvn checkstyle:check` from `JLBH` with Java 21 (see `verify-jlbh-java21-checkstyle-latest.log`); Checkstyle reports `You have 0 Checkstyle violations.` for this module. +- [x] Run SpotBugs scan and document issues + - A Java 21 SpotBugs run (`mvn com.github.spotbugs:spotbugs-maven-plugin:4.9.8.1:check -DskipTests`, see `verify-jlbh-java21-spotbugs-latest.log`) completes with `BugInstance size is 0` and `No errors/warnings found` for JLBH. +- [ ] [P1] [E:S] Identify any code review follow-ups from CODE_REVIEW_STATUS.md + - JLBH does not currently have a dedicated section in `CODE_REVIEW_STATUS.md`; any future review actions (for example around new benchmarks or harness features) should be recorded there and linked from this TODO. + +## Notes + +- 2025-11-18: JLBH is Checkstyle- and SpotBugs-clean on Java 21 (`verify-jlbh-java21-checkstyle-latest.log`, `verify-jlbh-java21-spotbugs-latest.log`). Remaining documentation/requirements/compliance TODOs are longer-running work and are tracked as deferred in `TODO_STATUS.md`. + +## Completion Checklist + +Before marking this repository's contribution to ARCH_TODO as complete: + +- [ ] All "Module Information" sections filled out +- [ ] Existing documentation audited +- [ ] Requirements identified for ARCH_TODO Stage 1.75 +- [ ] Decisions identified for ARCH_TODO Stage 1.85 +- [ ] Glossary terms identified for ARCH_TODO Stage 1.5 +- [ ] Documentation gaps documented +- [ ] Improvement tasks prioritized +- [ ] Information contributed to relevant ARCH_TODO stages + +--- + +**When complete, update:** [../ARCH_TODO.md](../ARCH_TODO.md) Stage 3 tracking matrix diff --git a/pom.xml b/pom.xml index 9c7aaea..fe12e54 100644 --- a/pom.xml +++ b/pom.xml @@ -31,7 +31,7 @@ net.openhft third-party-bom - 3.27ea5 + 3.27ea7 pom import @@ -142,6 +142,79 @@ 8 + + + + quality + + + + org.apache.maven.plugins + maven-checkstyle-plugin + 3.6.0 + + + validate + validate + + check + + + + + ${checkstyle.config.location} + true + true + true + ${checkstyle.violationSeverity} + + + + com.puppycrawl.tools + checkstyle + 10.26.1 + + + net.openhft + chronicle-quality-rules + 1.27.0-SNAPSHOT + + + + + com.github.spotbugs + spotbugs-maven-plugin + 4.9.8.1 + + Max + Low + true + true + + net/openhft/quality/spotbugs27/chronicle-spotbugs-include.xml + net/openhft/quality/spotbugs27/chronicle-spotbugs-exclude.xml + + + + net.openhft + chronicle-quality-rules + 1.27.0-SNAPSHOT + + + + + spotbugs-main + + process-test-classes + + check + + + + + + + diff --git a/src/main/adoc/decision-log.adoc b/src/main/adoc/decision-log.adoc deleted file mode 100644 index 1113684..0000000 --- a/src/main/adoc/decision-log.adoc +++ /dev/null @@ -1,37 +0,0 @@ -= Chronicle JLBH - Decision Log -Chronicle Software -:revnumber: 1.0 -:revdate: 24 May 2025 -:toc: -:lang: en-GB - -This document records key architectural and project choices made during the development of Chronicle JLBH. Entries follow the Decision Record Template so that contributors can trace why a change occurred. - -== Decisions - -=== [JL-DOC-001] Adopt AsciiDoc for project documentation - -Date:: 2025-05-24 -Context:: -* Project requires easily maintainable, version-controlled documentation. -* Contributors need a text format that is readable in raw form and convertible to HTML. -Decision Statement:: -* Adopt AsciiDoc as the canonical format for all project documentation. -Alternatives Considered:: -* Markdown: -** *Description:* Common lightweight markup; widely supported. -** *Pros:* Familiar to many developers; simple syntax. -** *Cons:* Limited features for complex documents; inconsistent rendering across tools. -* Plain text files: -** *Description:* Minimalistic approach with no markup. -** *Pros:* Simplest possible format; no tooling required. -** *Cons:* Hard to express structure or cross references; not suitable for large docs. -Rationale for Decision:: -* AsciiDoc provides structured markup without sacrificing readability. -* The format integrates well with our build tooling and supports the Nine-Box tagging used in requirements. -Impact & Consequences:: -* Contributors must learn basic AsciiDoc syntax. -* Build pipeline processes `.adoc` files to publish HTML. -* Some external tools may expect Markdown, requiring conversion. -Notes/Links:: -** https://asciidoctor.org[AsciiDoc project page] diff --git a/src/main/adoc/systemProperties.adoc b/src/main/adoc/systemProperties.adoc new file mode 100644 index 0000000..4ded78a --- /dev/null +++ b/src/main/adoc/systemProperties.adoc @@ -0,0 +1,24 @@ += JLBH System Properties +:toc: +:lang: en-GB +:source-highlighter: rouge + +== System Properties + +JLBH is a micro-benchmark harness that is sensitive to any extra work done by the JVM. +It checks the `jvm.resource.tracing` property to avoid running when intrusive resource tracing is enabled. + +.System properties +[cols="2a,1,3a,2a",options="header"] +|=== +| Property Key +| Default Value +| Description +| Usage (Type) + +| `jvm.resource.tracing` +| not set or `false` +| When this property is set with an empty value or `true`, JLBH prints a warning and terminates the JVM rather than running the benchmark, because resource tracing would distort latency and allocation measurements. Ensure this property is unset or explicitly `false` when running JLBH. +| Checked in the `JLBH` constructor (`String` value parsed as `boolean`) +|=== + diff --git a/src/main/adoc/architecture.adoc b/src/main/docs/architecture.adoc similarity index 98% rename from src/main/adoc/architecture.adoc rename to src/main/docs/architecture.adoc index 4644544..a6abb38 100644 --- a/src/main/adoc/architecture.adoc +++ b/src/main/docs/architecture.adoc @@ -1,15 +1,15 @@ = Chronicle JLBH - Architecture Overview -Chronicle Software :toc: +:sectnums: :lang: en-GB :source-highlighter: rouge -== 1. Introduction +== Introduction Chronicle JLBH (Java Latency Benchmark Harness) is a library designed for measuring and analyzing the latency of Java applications "in context," particularly under specific throughput conditions and accounting for coordinated omission. This document outlines its high-level architecture, major components, and key operational flows. -== 2. Major Components +== Major Components The JLBH system is composed of several key classes and interfaces that work together to orchestrate and execute benchmarks: @@ -80,7 +80,15 @@ The `startTimeNs` is the calculated ideal start time for the iteration. ** Monitoring is enabled by default (`JLBHOptions.recordOSJitter` defaults to true) and can be disabled via `recordOSJitter(false)`. ** Repeatedly calls `System.nanoTime()` to detect delays greater than `recordJitterGreaterThanNs` and records these into a dedicated `Histogram`. -== 3. Execution Flow +== Execution Flow + +```mermaid +graph TD + A[Initialization] --> B(Warm-up Phase) + B --> C{Measurement Runs} + C --> D(Completion) + C -- loop --> C +``` The harness follows a well-defined lifecycle, detailed visually in link:benchmark-lifecycle.adoc[benchmark lifecycle diagram]. A typical execution sequence is as follows: @@ -128,7 +136,7 @@ The harness may busy-wait (spin using `System.nanoTime()`) if the current time i ** The `JLBHTask.complete()` method is invoked for any final cleanup by the user's task. ** The `OSJitterMonitor` thread is terminated if it was running. -== 4. Threading Model +== Threading Model JLBH has a specific threading model that users and contributors should understand: @@ -161,7 +169,7 @@ Chronicle Core's `Histogram` has specific single-writer assumptions unless expli ** To safely access this result from a thread other than the main harness thread (e.g., an application thread wanting to query results after a test run), `JLBHResultConsumer.newThreadSafeInstance()` should be used. This creates a `ThreadSafeJLBHResultConsumer` which uses a `volatile` field to ensure proper publication of the immutable result. -== 5. Data Management and Results +== Data Management and Results JLBH pays careful attention to how latency data is recorded, structured, and reported: diff --git a/src/main/adoc/benchmark-lifecycle.adoc b/src/main/docs/benchmark-lifecycle.adoc similarity index 98% rename from src/main/adoc/benchmark-lifecycle.adoc rename to src/main/docs/benchmark-lifecycle.adoc index 322d3bf..afa893b 100644 --- a/src/main/adoc/benchmark-lifecycle.adoc +++ b/src/main/docs/benchmark-lifecycle.adoc @@ -1,7 +1,8 @@ = Benchmark Lifecycle +:toc: +:sectnums: :lang: en-GB :source-highlighter: rouge -:toc: [mermaid] ---- @@ -14,7 +15,7 @@ graph LR A typical JLBH (Java Latency Benchmark Harness) execution follows a distinct lifecycle designed to ensure accurate and repeatable latency measurements. The key phases are: -== 1. Setup +== Setup This initial phase prepares the benchmark environment: @@ -25,7 +26,7 @@ This initial phase prepares the benchmark environment: * **OS Jitter Monitoring**: Jitter tracking is enabled by default (`recordOSJitter` defaults to true); unless disabled via `recordOSJitter(false)`, the `OSJitterMonitor` background thread starts to measure operating system scheduling jitter independently. * **Affinity**: If an `acquireLock` supplier is configured in `JLBHOptions`, an attempt to acquire CPU affinity for the main benchmark thread might occur. -== 2. Warmup +== Warmup Before any measurements are formally recorded, the harness executes a warm-up phase: @@ -36,7 +37,7 @@ Before any measurements are formally recorded, the harness executes a warm-up ph * **Task Notification**: After all warmup iterations are complete, the `JLBHTask.warmedUp()` method is called. * **Optional Pause**: If `JLBHOptions.pauseAfterWarmupMS` is greater than zero, JLBH will pause for the specified duration before proceeding to the execution phase. -== 3. Execution (Measurement Runs) +== Execution (Measurement Runs) This is the core phase where timed iterations are performed and latency data is collected. This phase consists of one or more "runs" (as configured by `JLBHOptions.runs`): @@ -59,7 +60,7 @@ This is the core phase where timed iterations are performed and latency data is *** The collected histogram data for this run is processed and stored internally for the final `JLBHResult`. *** All probe histograms are then reset to ensure that measurements for the next run (if any) are independent. -== 4. Reporting +== Reporting After all measurement runs are completed, the aggregated results are finalized and made available: @@ -70,7 +71,7 @@ After all measurement runs are completed, the aggregated results are finalized a ** `JLBHResultSerializer` to write results to a CSV file for analysis in spreadsheets or other tools. ** `TeamCityHelper` to output statistics in a format suitable for TeamCity CI server integration. -== 5. Cleanup +== Cleanup The final phase ensures that any resources are properly released: diff --git a/src/main/docs/decision-log.adoc b/src/main/docs/decision-log.adoc new file mode 100644 index 0000000..ac51482 --- /dev/null +++ b/src/main/docs/decision-log.adoc @@ -0,0 +1,214 @@ += Chronicle JLBH - Decision Log +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +This document records key architectural and project choices made during the development of Chronicle JLBH. +Identifiers use the `JL` project scope with Nine-Box tags (FN, NF-P, NF-O, DOC, OPS, RISK, TEST, UX) and link back to the requirements in `project-requirements.adoc`. + +== Decision Index + +* link:#JL-DOC-001[JL-DOC-001 Adopt AsciiDoc for project documentation] +* link:#JL-FN-002[JL-FN-002 Single-threaded harness with JLBHTask lifecycle] +* link:#JL-NF-P-003[JL-NF-P-003 Default coordinated-omission compensation and histograms] +* link:#JL-OPS-004[JL-OPS-004 OS jitter tracking as an optional probe] +* link:#JL-FN-005[JL-FN-005 Additional probes and NanoSampler API] +* link:#JL-OPS-006[JL-OPS-006 CSV serialisation and result retention] +* link:#JL-OPS-007[JL-OPS-007 CI integration via TeamCity statistics] + +[[JL-DOC-001]] +=== [JL-DOC-001] Adopt AsciiDoc for project documentation + +Date:: 2025-05-24 +Context:: +* Project requires easily maintainable, version-controlled documentation. +* Contributors need a text format that is readable in raw form and convertible to HTML. +Decision Statement:: +* Adopt AsciiDoc as the canonical format for all project documentation. +Alternatives Considered:: +* Markdown: +** *Description:* Common lightweight markup; widely supported. +** *Pros:* Familiar to many developers; simple syntax. +** *Cons:* Limited features for complex documents; inconsistent rendering across tools. +* Plain text files: +** *Description:* Minimalistic approach with no markup. +** *Pros:* Simplest possible format; no tooling required. +** *Cons:* Hard to express structure or cross references; not suitable for large docs. +Rationale for Decision:: +* AsciiDoc provides structured markup without sacrificing readability. +* The format integrates well with our build tooling and supports the Nine-Box tagging used in requirements. +Impact & Consequences:: +* Contributors must learn basic AsciiDoc syntax. +* Build pipeline processes `.adoc` files to publish HTML. +* Some external tools may expect Markdown, requiring conversion. +Notes/Links:: +** https://asciidoctor.org[AsciiDoc project page] + +[[JL-FN-002]] +=== [JL-FN-002] Single-threaded harness with JLBHTask lifecycle + +Date:: 2025-11-14 +Context:: +* JLBH targets low-latency Java applications where benchmark overhead and scheduling noise must be minimised. +* The harness needs a clear lifecycle so that warm-up, steady-state runs and shutdown can be expressed consistently across benchmarks. +* Requirements: FN-001 Latency Sampling, FN-006 Event Loop Integration, NF-P-001 Performance, NF-O-001 Portability. +Decision Statement:: +* Run the JLBH harness logic on a single dedicated thread, driving user code through the `JLBHTask` lifecycle (`init`, `run`, `warmedUp`, `runComplete`, `complete`). +* Allow the harness thread to be supplied by the caller via `eventLoopHandler(EventLoop)` while preserving the single-threaded execution model. +Alternatives Considered:: +* Multi-threaded harness with worker pools :: +** *Pros:* Could saturate multi-core CPUs more easily in some scenarios. +** *Cons:* Adds scheduling variability, complicates reasoning about pauses and makes per-iteration latency harder to interpret. +* Delegating to a generic micro-benchmark framework (e.g. JMH) :: +** *Pros:* Reuses existing tooling and reporting. +** *Cons:* Less control over in-context execution, fewer hooks for probes and OS jitter, and different lifecycle assumptions than JLBH requires. +Rationale for Decision:: +* A single harness thread keeps timing behaviour predictable and avoids extra scheduling noise beyond what the benchmarked code introduces. +* The explicit lifecycle methods match the phases described in the requirements and README, making it easy to structure complex benchmarks. +Impact & Consequences:: +* Benchmarked code must respect the single-threaded driver model, spawning its own threads only when needed and understanding the impact on measurements. +* Event-loop integration is explicit: users can install JLBH onto an existing loop without changing the underlying lifecycle. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:benchmark-lifecycle.adoc[Benchmark lifecycle overview] + +[[JL-NF-P-003]] +=== [JL-NF-P-003] Default coordinated-omission compensation and high-resolution histograms + +Date:: 2025-11-14 +Context:: +* Co-ordinated omission can hide worst-case latency by skipping samples during pauses or back-pressure. +* JLBH must support high-resolution percentile analysis at extreme tails to satisfy low-latency users. +* Requirements: FN-001 Latency Sampling, FN-002 Coordinated-Omission Compensation, NF-P-001 Performance. +Decision Statement:: +* Enable compensation for co-ordinated omission by default via `accountForCoordinatedOmission(true)`, with an option to disable it when raw uncorrected timings are required. +* Use high-resolution histograms as the internal representation for recorded latencies so that percentiles can be reported accurately across a wide range of values. +Alternatives Considered:: +* No coordinated-omission compensation :: +** *Pros:* Simpler mental model; raw timings only. +** *Cons:* Under-reports tail latency in back-pressured systems, undermining the harness goals. +* External correction tooling applied after the run :: +** *Pros:* Keeps the core harness simpler. +** *Cons:* Splits logic across tools, increases risk of misconfiguration and makes it harder to reason about what the reported percentiles mean. +Rationale for Decision:: +* Default-on correction matches the expectations set in the JLBH documentation and literature on latency measurement accuracy. +* High-resolution histograms are a proven structure for representing latency distributions with minimal overhead. +Impact & Consequences:: +* Users must be aware that reported percentiles are corrected unless they explicitly disable the feature. +* Benchmark authors can rely on accurate tail reporting without bolting on extra tooling. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-004]] +=== [JL-OPS-004] OS jitter tracking as an optional probe + +Date:: 2025-11-14 +Context:: +* Kernel scheduling, interrupts and other background activity can introduce jitter that affects latency measurements. +* Users often need to distinguish between pauses caused by their code and pauses caused by the operating system or environment. +* Requirements: FN-004 OS Jitter Tracking, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide an optional OS jitter probe implemented as a background thread that records scheduler delays beyond a configured threshold, and summarise the results alongside core latency histograms. +* Enable OS jitter tracking by default, with configuration options such as `recordOSJitter(false)` to disable it when the overhead is not acceptable. +Alternatives Considered:: +* Rely on external OS-level tools or profilers :: +** *Pros:* No extra threads or complexity inside the harness. +** *Cons:* Harder to correlate jitter with benchmark iterations; adds extra setup steps for users. +* Always disabling jitter tracking in the core library :: +** *Pros:* Zero overhead from jitter measurement. +** *Cons:* Removes a key diagnostic feature that helps interpret unexpected latency spikes. +Rationale for Decision:: +* Integrating jitter measurement directly into the harness keeps benchmark setup simple and ensures results remain correlated with the workload. +* Making the feature optional and configurable manages overhead while preserving a sensible default for investigation. +Impact & Consequences:: +* When enabled, benchmarks incur some additional overhead from the jitter thread; users can trade this off against diagnostic value. +* CI and local runs can use the jitter probe to explain outliers and refine environment configuration (CPU affinity, power settings). +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-FN-005]] +=== [JL-FN-005] Additional probes and NanoSampler API + +Date:: 2025-11-14 +Context:: +* Benchmarks often need to record latencies for sub-stages (for example serialisation, network calls) without distorting the end-to-end latency distribution. +* JLBH must make it easy to capture these timings while keeping the main histogram focused on overall request latency. +* Requirements: FN-003 Additional Probes, NF-P-001 Performance, NF-UX-001 Usability. +Decision Statement:: +* Provide `addProbe(String name)` on `JLBH` that returns a `NanoSampler`, each backed by its own histogram and reported alongside the primary distribution. +* Keep the end-to-end `sample` / `sampleNanos` API unchanged so that additional probes never alter the core latency measurements. +Alternatives Considered:: +* Require users to maintain their own histograms outside the harness :: +** *Pros:* No extra API surface in JLBH. +** *Cons:* More boilerplate; higher risk of inconsistent reporting or incompatible histogram settings. +* Fold sub-stage timings into the primary histogram :: +** *Pros:* Single distribution to manage. +** *Cons:* Hides distinctions between phases and makes diagnosis of slow sub-components difficult. +Rationale for Decision:: +* A dedicated probe API gives clear separation between overall latencies and sub-stage metrics, supporting richer analysis without complicating the main flow. +* `NanoSampler` provides a lightweight abstraction that integrates cleanly with the existing sampling model. +Impact & Consequences:: +* Each additional probe incurs extra memory and processing cost; users should add probes selectively for meaningful stages. +* Documentation and examples must show how to name probes consistently so that dashboards and CI jobs can rely on stable identifiers. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc#_additional_features[Additional features in the README] + +[[JL-OPS-006]] +=== [JL-OPS-006] CSV serialisation and result retention + +Date:: 2025-11-14 +Context:: +* Users need to archive benchmark results for later analysis, trend tracking and comparison across builds or environments. +* Plain console tables are useful for humans but awkward for automated tooling and external analytics systems. +* Requirements: FN-005 CSV Serialisation, NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Expose an immutable `JLBHResult` model for completed runs and provide `JLBHResultSerializer` to write results to CSV with a stable schema. +* Default to a simple file layout suitable for CI artefacts, while allowing callers to direct output to custom streams or paths. +Alternatives Considered:: +* Rely solely on console output :: +** *Pros:* No extra types or wiring. +** *Cons:* Difficult to parse reliably; fragile across formatting changes. +* Use a binary format (for example compressed histograms) only :: +** *Pros:* More compact; potentially faster to write. +** *Cons:* Harder to inspect manually; requires custom tooling for every consumer. +Rationale for Decision:: +* CSV strikes a balance between human readability and machine processing, working well with spreadsheets, scripting languages and CI systems. +* A dedicated serializer centralises schema decisions so that downstream tools can depend on consistent column names and ordering. +Impact & Consequences:: +* Schema changes to CSV output must be versioned and documented carefully to avoid breaking consumers. +* Large or long-running benchmarks may produce sizeable CSV files; operations teams should manage retention and storage policies. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] + +[[JL-OPS-007]] +=== [JL-OPS-007] CI integration via TeamCity statistics + +Date:: 2025-11-14 +Context:: +* Many users run JLBH benchmarks in CI and need to gate builds on latency regressions or track trends over time. +* TeamCity provides a native mechanism (`##teamcity` service messages) for publishing numerical statistics from builds. +* Requirements: Product Functions (CI Metrics), NF-O-001 Portability, NF-R-001 Reliability. +Decision Statement:: +* Provide helper code to emit key JLBH metrics (for example selected percentiles and throughput) as TeamCity statistics lines, using predictable metric names derived from benchmark configuration. +* Treat CI integration as an optional layer that can be enabled or disabled without affecting the core harness behaviour. +Alternatives Considered:: +* Expect users to write their own CI integration scripts around console output or CSV files :: +** *Pros:* Keeps the core library smaller; no explicit dependency on CI conventions. +** *Cons:* Duplicated effort across projects; higher risk of inconsistent parsing and metrics. +* Integrate tightly with a single CI provider only :: +** *Pros:* Deep integration for that platform. +** *Cons:* Locks users into a specific CI and complicates usage elsewhere. +Rationale for Decision:: +* A lightweight TeamCity helper covers a common use case without constraining users who rely on other CI platforms. +* Publishing statistics directly simplifies setting latency thresholds and visualising trends in existing dashboards. +Impact & Consequences:: +* CI pipelines can fail fast when configured thresholds are exceeded, turning JLBH results into actionable quality gates. +* Additional documentation must describe metric naming conventions so that teams can map them to their CI configuration. +Notes/Links:: +* link:project-requirements.adoc[Chronicle JLBH - Software Requirements Specification] +* link:../README.adoc[Chronicle JLBH README] diff --git a/src/main/adoc/jlbh-cookbook.adoc b/src/main/docs/jlbh-cookbook.adoc similarity index 98% rename from src/main/adoc/jlbh-cookbook.adoc rename to src/main/docs/jlbh-cookbook.adoc index fa91cc6..c0d0f30 100644 --- a/src/main/adoc/jlbh-cookbook.adoc +++ b/src/main/docs/jlbh-cookbook.adoc @@ -1,13 +1,14 @@ = Chronicle JLBH Cookbook -Chronicle Software :toc: +:sectnums: :lang: en-GB :source-highlighter: rouge A collection of worked examples for common benchmarking scenarios using Chronicle JLBH. These recipes aim to provide practical starting points for your own benchmarks. +:sectnums: -== 1. Timing a Simple Method Call +== Timing a Simple Method Call This is the most basic use case: measuring the latency of a self-contained Java method. @@ -81,7 +82,7 @@ Avoid including unrelated logic within the timed section. If `myMethod()` is very fast, high throughput might be achievable. If it's slower, adjust throughput accordingly. -== 2. Measuring a Network Round Trip (UDP Loopback) +== Measuring a Network Round Trip (UDP Loopback) This recipe demonstrates timing a UDP message send and receive on loopback. @@ -193,7 +194,6 @@ import net.openhft.chronicle.jlbh.JLBHOptions; import java.io.IOException; - public class MainApp { public static void main(String[] args) { // First, ensure SimpleUDPEchoServer is running. @@ -220,7 +220,7 @@ public class MainApp { * Consider TCP for connection-oriented tests; its performance characteristics (e.g., Nagle's algorithm, ACK delays) differ from UDP. * Packet loss and retransmissions (for UDP, if implementing reliability) can heavily affect latency distributions. -== 3. Running Within an Existing Event Loop +== Running Within an Existing Event Loop JLBH can integrate with Chronicle's event loop mechanism, useful for benchmarking components that are already part of an event-driven architecture. @@ -311,7 +311,7 @@ public class MainApp { * `JLBHOptions.accountForCoordinatedOmission(true)` (the default) is essential for `eventLoopHandler` to function correctly. --- -== 4. Using Multiple Probes for Staged Operations +== Using Multiple Probes for Staged Operations JLBH allows timing multiple stages within a single benchmark iteration using custom probes. @@ -405,9 +405,9 @@ public class MainApp { * This approach is excellent for identifying bottlenecks within a larger operation. * The JLBH output will include separate percentile summaries for "Stage1_Processing", "Stage2_Persistence", and the default "end to end" probe. ---- +''' -== 5. Recording and Exporting Results +== Recording and Exporting Results JLBH allows programmatic access to results and provides utilities for serialization, such as to CSV. @@ -489,7 +489,7 @@ public class MainApp { * The `JLBHResult` object is immutable and contains rich data for all probes and runs. * `JLBHResultSerializer` provides a convenient way to get data out for external analysis. -== 6. Enabling and Understanding OS Jitter Measurement +== Enabling and Understanding OS Jitter Measurement Operating system jitter can significantly impact low-latency applications. JLBH can measure this independently. diff --git a/src/main/adoc/project-requirements.adoc b/src/main/docs/project-requirements.adoc similarity index 86% rename from src/main/adoc/project-requirements.adoc rename to src/main/docs/project-requirements.adoc index 2600c7a..3dbe1a0 100644 --- a/src/main/adoc/project-requirements.adoc +++ b/src/main/docs/project-requirements.adoc @@ -1,24 +1,23 @@ = Chronicle JLBH - Software Requirements Specification -Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB +:source-highlighter: rouge *_Abstract_*:: Chronicle JLBH is an open-source Java library, released under the Apache Licence 2.0, that provides a high-resolution latency benchmark harness. It allows developers to _measure_, _analyse_ and _regress-test_ latency behaviour of critical code paths "in context" rather than via isolated micro-benchmarks. Key features include compensation for *co-ordinated omission*, configurable throughput modelling, additional probes, optional operating-system jitter tracking, and serialisation of results for CI dashboards. -== 1. Introduction +== Introduction -=== 1.1 Purpose +=== Purpose This document specifies functional and non-functional requirements for Chronicle JLBH, ensuring that contributors and integrators share a common understanding of the product's goals and constraints. -=== 1.2 Scope +=== Scope The harness targets _low-latency_ Java applications (trading systems, micro-services, real-time analytics) that must quantify latencies at extreme percentiles under realistic load patterns. -=== 1.3 Definitions, Acronyms and Abbreviations +=== Definitions, Acronyms and Abbreviations |=== | *Term* | *Definition* @@ -30,7 +29,7 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv | CI | Continuous Integration environment (e.g. GitHub Actions, TeamCity). |=== -=== 1.4 References +=== References * Upstream repository: https://github.com/OpenHFT/JLBH[GitHub - OpenHFT/JLBH] * API reference: https://javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/core/jlbh/JLBH.html[Javadoc] @@ -38,16 +37,16 @@ The harness targets _low-latency_ Java applications (trading systems, micro-serv * Discussion of _co-ordinated omission_: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE[m.s. thread] * Chronicle blog post on JLBH in event loops: https://vanilla-java.github.io/2016/04/02/Microservices-in-the-Chronicle-World-Part-5.html[Vanilla-Java article] -=== 1.5 Overview +=== Overview Sections 2-5 describe the product context, interfaces, detailed system features, and quality attributes. Section 6 captures licensing, whilst Section 7 presents the glossary. -== 2. Overall Description +== Overall Description -=== 2.1 Product Perspective +=== Product Perspective JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chronicle Core for low-level utilities. It may be embedded in unit tests, standalone mains, or invoked as an _EventLoop_ handler. -=== 2.2 Product Functions +=== Product Functions * Generate high-resolution histograms (>=35 bits) of end-to-end sample latencies. * Optionally compensate for *co-ordinated omission* by adjusting the synthetic start time. @@ -59,7 +58,7 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * Integrate with CI (TeamCity helper emits `##teamcity` statistics lines). * Allow execution from any thread or installation onto a Chronicle *MediumEventLoop*. -=== 2.3 User Classes and Characteristics +=== User Classes and Characteristics |=== | *Actor* | *Description* | *Technical Expertise* @@ -69,28 +68,28 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro | CI System | Runs automated latency regression suite, ingests CSV or TeamCity stats. | N/A (automation). |=== -=== 2.4 Operating Environment +=== Operating Environment * Java 11 LTS and later; Java 17 recommended for current builds. * Linux x86-64 (primary), other POSIX OSes supported; macOS usable but CI baselines exclude it. * Typical CPU affinity facilities available via *OpenHFT Affinity* library. -=== 2.5 Design and Implementation Constraints +=== Design and Implementation Constraints * Single-threaded harness execution; benchmarked code may span threads. * Measurements rely on `System.nanoTime()` precision; hardware timers must be invariant. * JVM must be started with `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when using async-profiler concurrently. * Library is released under the Apache Licence 2.0; contributions must comply with CLA policy. -=== 2.6 Assumptions and Dependencies +=== Assumptions and Dependencies * SUT jars are available on the same JVM classpath. * Benchmark host clocks are monotonic and not subjected to aggressive power-saving states. * CI agents provision exclusive CPU cores to minimise jitter. -== 3. External Interface Requirements +== External Interface Requirements -=== 3.1 Public API +=== Public API * `JLBHOptions` - builder for benchmark parameters (throughput, warm-up, runs, etc.). * `JLBHTask` - user-defined callback with lifecycle methods `init`, `run`, `warmedUp`, `runComplete`, `complete`. @@ -98,78 +97,84 @@ JLBH is delivered as a *Maven* artefact (`net.openhft:jlbh`) and depends on Chro * `JLBHResult` - immutable projection of measured data. * `JLBHResultSerializer` - CSV writer for analytics pipelines. -=== 3.2 User Interface +=== User Interface CLI usage is demonstrated in _ExampleJLBHMain_ and test fixtures; no dedicated GUI is provided. -=== 3.3 Hardware Interfaces +=== Hardware Interfaces None. The harness interacts with OS timers and CPU affinity via JNI where available. -=== 3.4 Software Interfaces +=== Software Interfaces * Build tool: *Apache Maven* (pom.xml defines BOM imports). * Logging: SLF4J (test scope). * Testing: JUnit 4 for unit and integration tests. * CI Metrics: TeamCity service messages. -== 4. System Features +== System Features -=== 4.1 FN-001 Latency Sampling +=== FN-001 Latency Sampling *Description*: Collect nanosecond deltas from user code via `sample(long)` or `sampleNanos(long)`. *Stimulus/Response*: Upon each iteration, `JLBHTask.run` calls `jlbh.sample(...)`; histogram updates occur; run summary printed at completion. *Priority*: *Essential*. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. -=== 4.2 FN-002 Coordinated-Omission Compensation +=== FN-002 Coordinated-Omission Compensation Ensures that queue back-log does not under-represent tail latency. Enabled by default; opt-out via `accountForCoordinatedOmission(false)`. +*Decision reference*: link:decision-log.adoc#JL-NF-P-003[JL-NF-P-003]. -=== 4.3 FN-003 Additional Probes +=== FN-003 Additional Probes Developers may time sub-stages (e.g., serialisation, network round-trip) through `addProbe(String)` and record them independently. +*Decision reference*: link:decision-log.adoc#JL-FN-005[JL-FN-005]. -=== 4.4 FN-004 OS Jitter Tracking +=== FN-004 OS Jitter Tracking A background thread measures scheduling gaps exceeding a configured threshold (`recordJitterGreaterThanNs`). Monitoring is enabled by default and can be disabled via `recordOSJitter(false)`. Results are summarised alongside core latencies. +*Decision reference*: link:decision-log.adoc#JL-OPS-004[JL-OPS-004]. -=== 4.5 FN-005 CSV Serialisation +=== FN-005 CSV Serialisation Post-run, results can be persisted for offline analysis (`result.csv` by default). +*Decision reference*: link:decision-log.adoc#JL-OPS-006[JL-OPS-006]. -=== 4.6 FN-006 Event Loop Integration +=== FN-006 Event Loop Integration `eventLoopHandler(EventLoop)` allows benchmarks to operate inside Chronicle threading framework, avoiding extra threads. +*Decision reference*: link:decision-log.adoc#JL-FN-002[JL-FN-002]. -== 5. Non-Functional Requirements +== Non-Functional Requirements -=== 5.1 NF-P-001 Performance +=== NF-P-001 Performance * Overhead per sample must remain below 100 ns when no additional probes are active. * Histogram generation must support >=200 M iterations without heap pressure. -=== 5.2 NF-R-001 Reliability +=== NF-R-001 Reliability * Harness must abort gracefully on thread interruptions or sample time-outs (`timeout(long)`). * Immutable result objects ensure thread-safe publication to external consumers. -=== 5.3 NF-UX-001 Usability +=== NF-UX-001 Usability * Fluent builder API; sensible defaults provide a runnable benchmark in ≤10 LOC. * ASCII table outputs are human-readable and CI-friendly. -=== 5.4 NF-O-001 Portability +=== NF-O-001 Portability * Pure-Java codebase; no native compilation steps. * JDK-specific optimisations (e.g., *Zing* support) are runtime-detected. -=== 5.5 NF-O-002 Maintainability +=== NF-O-002 Maintainability * 80 %+ unit-test line coverage with deterministic fixtures. * Code adheres to Chronicle parent POM style and SonarCloud quality gates. -=== 5.6 NF-S-001 Security +=== NF-S-001 Security No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code. -== 6. Licensing +== Licensing The project is released under the *Apache Licence 2.0* (_see_ `LICENSE.adoc`). Downstream consumers must preserve licence notices and may include JLBH in commercial or OSS products, subject to the terms therein. -== 7. Glossary +== Glossary Co-ordinated Omission:: Statistical artefact causing under-reporting of worst-case latency. Histogram:: Data structure that records frequency counts in logarithmic buckets, enabling percentile extraction. Percentile:: Value below which a given percentage of observations fall (e.g., 99th percentile). -== 8. Appendix A - Example Minimal Benchmark +== Appendix A - Example Minimal Benchmark [source,java] ---- @@ -189,7 +194,7 @@ public class NothingBenchmark implements JLBHTask { } ---- -== 9. Appendix B - Footnotes +== Appendix B - Footnotes * JLBH originated within the Chronicle Software open-source stack and is actively maintained. See https://github.com/OpenHFT/JLBH. * The API reference highlights the focus on _co-ordinated omission_ and event-loop support. See https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/net/openhft/chronicle/jlbh/JLBH.html. diff --git a/src/main/adoc/results-interpretation-guide.adoc b/src/main/docs/results-interpretation-guide.adoc similarity index 96% rename from src/main/adoc/results-interpretation-guide.adoc rename to src/main/docs/results-interpretation-guide.adoc index e288647..9dd72d9 100644 --- a/src/main/adoc/results-interpretation-guide.adoc +++ b/src/main/docs/results-interpretation-guide.adoc @@ -1,15 +1,13 @@ = JLBH Results Interpretation Guide -Chronicle Software -:revnumber: 1.0 -:revdate: 23 May 2025 :toc: +:sectnums: :lang: en-GB -:icons: font +:source-highlighter: rouge *_Abstract_*:: This guide explains how to read and understand the various outputs and summaries generated by Chronicle JLBH. Effective interpretation is key to deriving meaningful insights from your latency benchmarks. -== 1. Understanding JLBH Console Output Structure +== Understanding JLBH Console Output Structure When JLBH runs, it typically prints information to the console (or a configured `PrintStream`). Understanding this output is the first step: @@ -26,7 +24,7 @@ When JLBH runs, it typically prints information to the console (or a configured ** Columns for each run, showing the latency value for that percentile in that run. ** A final "% Variation" column, indicating the percentage difference between the highest and lowest values for that percentile across the runs (excluding the first run by default if `runs > 3` or if `skipFirstRun` is explicitly true). -== 2. Interpreting Latency Histograms and Percentiles +== Interpreting Latency Histograms and Percentiles JLBH uses histograms to capture the full distribution of measured latencies, rather than just averages which can hide important details. These histograms are then used to derive percentile values. @@ -45,7 +43,7 @@ JLBH uses histograms to capture the full distribution of measured latencies, rat * **Units**: Latency values in JLBH output are typically shown in microseconds (`us`) or sometimes nanoseconds (`ns`) if the values are very small. Pay attention to the units indicated in the table headers. * **Custom Probes**: If you've added custom probes (e.g., `jlbh.addProbe("MyStage")`), each will have its own percentile table in the output. Analyzing these helps pinpoint which stage of an operation contributes most to the overall end-to-end latency. -== 3. Co-ordinated Omission (CO) +== Co-ordinated Omission (CO) Co-ordinated Omission is a critical concept in latency measurement, especially for systems that process requests at a certain rate. @@ -61,14 +59,14 @@ Co-ordinated Omission is a critical concept in latency measurement, especially f ** *Without CO compensation*: Histograms show only the raw processing time for the operations that were actually executed, potentially missing significant backpressure effects. This can be useful for understanding raw compute time but not overall system responsiveness under load. * **Comparison**: Always state whether CO compensation was enabled when comparing benchmark results. Comparing a CO-compensated run with a non-compensated run is misleading. JLBH prints "Correcting for co-ordinated:true/false" in each run's output. -== 4. OS Jitter Probe +== OS Jitter Probe JLBH can optionally measure Operating System (OS) jitter, which refers to delays caused by the OS scheduler or other system activities interrupting the benchmark process. * **Mechanism**: ** Jitter monitoring is enabled by default (`recordOSJitter` defaults to true); a dedicated background thread (`OSJitterMonitor`) starts unless you explicitly disable it with `recordOSJitter(false)`. ** This thread repeatedly calls `System.nanoTime()` in a tight loop and measures the time difference between consecutive calls. -** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 microsecond), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. +** If this difference (a "gap") exceeds a configurable threshold (`recordJitterGreaterThanNs`, default 1 µs), it's recorded as an OS jitter event in a separate "OS Jitter" histogram. * **Interpreting Jitter Output**: ** The "OS Jitter" probe will have its own percentile table in the JLBH output. ** *High jitter values* (e.g., multiple milliseconds at p99 or worst) suggest that the OS, other processes, or hardware interrupts are frequently pausing the benchmark threads for significant periods. This is "machine noise." @@ -81,7 +79,7 @@ JLBH can optionally measure Operating System (OS) jitter, which refers to delays ** Utilize CPU isolation / shielding for benchmark threads and the OS jitter thread (see `JLBHOptions.acquireLock` and `jitterAffinity`). ** Investigate system settings (e.g., power saving modes, transparent huge pages, kernel scheduler options, interrupt coalescing). -== 5. Analysing Run-to-Run Variation +== Analysing Run-to-Run Variation Latency measurements can vary between successive runs of the same benchmark on the same machine due to various factors. JLBH's summary output includes a "% Variation" column for each percentile. @@ -97,7 +95,7 @@ Latency measurements can vary between successive runs of the same benchmark on t *** **Benchmarked Code Instability**: The code itself might have inherent variability or be interacting with unstable external resources. * **Goal**: Aim for reasonably tight spreads to ensure your conclusions are based on consistent behavior. Focus on understanding systematic performance rather than chasing the single "best" score from one run. Longer runs (`iterations`) can also help smooth out some variations and capture rarer events more consistently. -== 6. The Throughput vs. Latency Relationship +== The Throughput vs. Latency Relationship A fundamental aspect of performance is the trade-off between throughput (how much work is done per unit of time) and latency (how long each piece of work takes). @@ -111,7 +109,7 @@ A fundamental aspect of performance is the trade-off between throughput (how muc ** This helps identify the "sweet spot" for your system or pinpoint throughput levels where performance starts to degrade significantly. ** The results can inform capacity planning and help set realistic performance expectations. -== 7. Common Pitfalls in Interpretation +== Common Pitfalls in Interpretation Avoiding these common mistakes can help ensure your benchmark results are meaningful and your conclusions are sound: @@ -143,7 +141,7 @@ Avoiding these common mistakes can help ensure your benchmark results are meanin ** *Issue*: Very short runs (low `iterations`) may not be statistically significant or may fail to capture infrequent, high-latency events like GC pauses or network timeouts. ** *Solution*: Run enough iterations to achieve statistical stability and to give rare events a chance to occur if they are part of the system's behavior under load. This often means runs lasting at least several seconds, or even minutes for deep stability tests. -== 8. Deriving Actionable Insights +== Deriving Actionable Insights The ultimate goal of benchmarking is to gain insights that can lead to improvements. diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java index 68df6b4..9e9ad3e 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBH.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBH.java @@ -22,6 +22,7 @@ import java.util.concurrent.ConcurrentSkipListMap; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; import java.util.function.Consumer; import java.util.stream.Collectors; @@ -81,7 +82,7 @@ public class JLBH implements NanoSampler { private final long mod; private final long length; // Todo: Remove all concurrent constructs such as volatile and AtomicBoolean - private volatile long noResultsReturned; + private final AtomicLong noResultsReturned = new AtomicLong(); //Use non-atomic when so thread synchronisation is necessary private boolean warmedUp; private volatile Thread testThread; @@ -116,8 +117,8 @@ public JLBH(@NotNull JLBHOptions jlbhOptions, @NotNull PrintStream printStream, final String resourceTracing = System.getProperty("jvm.resource.tracing"); if (resourceTracing != null && (resourceTracing.isEmpty() || Boolean.parseBoolean(resourceTracing))) { - System.out.println("***** WARNING : JLBH can not be run if jvm.resource.tracing=" + resourceTracing + ", please remove all \"jvm.resource.tracing\" as this will corrupt your stats *****"); - System.exit(-1); + throw new IllegalStateException("JLBH can not be run if jvm.resource.tracing=" + resourceTracing + + ", please remove all \"jvm.resource.tracing\" as this will corrupt your stats"); } this.jlbhOptions = jlbhOptions; @@ -133,8 +134,9 @@ public JLBH(@NotNull JLBHOptions jlbhOptions, @NotNull PrintStream printStream, : jlbhOptions.iterations > 50_000_000 ? 20_000_000_000L : jlbhOptions.iterations > 10_000_000 ? 10_000_000_000L : 5_000_000_000L; - long mod2; - for (mod2 = 1000; mod2 <= jlbhOptions.iterations / 200; mod2 *= 10) { + long mod2 = 1000; + while (mod2 <= jlbhOptions.iterations / 200) { + mod2 *= 10; } this.mod = mod2; } @@ -251,7 +253,6 @@ public void start() { } else { if (latencyBetweenTasks > 2e6) { - long end = System.nanoTime() + latencyBetweenTasks; Jvm.pause(latencyBetweenTasks / 1_000_000 - 1); // account for jitter in Thread.sleep() and wait until a fixed point in time startTimeNs = busyWaitUntil(startTimeNs); @@ -367,7 +368,7 @@ private long warmup() { private void endOfAllRuns() { printPercentilesSummary("end to end", percentileRuns, printStream); - if (additionalPercentileRuns.size() > 0) { + if (!additionalPercentileRuns.isEmpty()) { additionalPercentileRuns.forEach((label, percentileRuns1) -> printPercentilesSummary(label, percentileRuns1, printStream)); } @@ -413,13 +414,11 @@ private void endOfRun(int run, long runStart) { printStream.printf("%-48s", format("End to End: (%,d)", endToEndHistogram.totalCount())); printStream.println(endToEndHistogram.toMicrosFormat()); - if (additionHistograms.size() > 0) { + if (!additionHistograms.isEmpty()) { additionHistograms.forEach((key, value) -> { List ds = additionalPercentileRuns.computeIfAbsent(key, i -> new ArrayList<>()); ds.add(value.getPercentiles()); -// if (value.totalCount() != jlbhOptions.iterations) -// warning = " WARNING " + value.totalCount() + "!=" + jlbhOptions.iterations; printStream.printf("%-48s", format("%s (%,d)", key, value.totalCount())); printStream.println(value.toMicrosFormat()); }); @@ -432,7 +431,7 @@ private void endOfRun(int run, long runStart) { jlbhOptions.jlbhTask.runComplete(); - noResultsReturned = 0; + noResultsReturned.set(0); additionHistograms.values().forEach(Histogram::reset); endToEndHistogram.reset(); osJitterMonitor.reset(); @@ -445,8 +444,9 @@ private void checkSampleTimeout() { while (true) { Jvm.pause(TimeUnit.SECONDS.toMillis(10)); - if (previousSampleCount < noResultsReturned) { - previousSampleCount = noResultsReturned; + long currentSampleCount = noResultsReturned.get(); + if (previousSampleCount < currentSampleCount) { + previousSampleCount = currentSampleCount; previousSampleTime = System.currentTimeMillis(); } else { if (previousSampleTime < (System.currentTimeMillis() - jlbhOptions.timeout)) { @@ -529,9 +529,9 @@ public void printPercentilesSummary( .append("\n"); double[] percentiles = Histogram.percentilesFor(jlbhOptions.iterations); boolean skipFirst = percentiles.length > 3; - if (jlbhOptions.skipFirstRun == JLBHOptions.SKIP_FIRST_RUN.SKIP) { + if (jlbhOptions.skipFirstRun == JLBHOptions.SkipFirstRun.SKIP) { skipFirst = true; - } else if (jlbhOptions.skipFirstRun == JLBHOptions.SKIP_FIRST_RUN.NO_SKIP) { + } else if (jlbhOptions.skipFirstRun == JLBHOptions.SkipFirstRun.NO_SKIP) { skipFirst = false; } PercentileSummary percentileSummary = new PercentileSummary(skipFirst, percentileRuns, percentiles); @@ -581,7 +581,8 @@ private String formatPercentile(double percentile) { * @param pr formatted percentile label (e.g. {@code "99.9:"}) * @param runs number of run value placeholders to append */ - private void addPrToPrint(@NotNull StringBuilder sb, String pr, int runs) { + // Visible for tests via reflection to verify percentile summary patterns. + void addPrToPrint(@NotNull StringBuilder sb, String pr, int runs) { sb.append(pr); for (int i = 0; i < runs; i++) { sb.append("%12.2f "); @@ -646,12 +647,12 @@ public void sampleNanos(long durationNs) { * @param durationNs latency in nanoseconds */ public void sample(long durationNs) { - noResultsReturned++; - if (noResultsReturned < jlbhOptions.warmUpIterations && !warmedUp) { + long sampleIndex = noResultsReturned.incrementAndGet(); + if (sampleIndex < jlbhOptions.warmUpIterations && !warmedUp) { endToEndHistogram.sample(durationNs); return; } - if (noResultsReturned == jlbhOptions.warmUpIterations && !warmedUp) { + if (sampleIndex == jlbhOptions.warmUpIterations && !warmedUp) { warmedUp = true; endToEndHistogram.reset(); if (!additionHistograms.isEmpty()) { diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java index ecb9b39..243de10 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java @@ -31,7 +31,7 @@ public class JLBHOptions { JLBHTask jlbhTask; int pauseAfterWarmupMS = 0; @NotNull - SKIP_FIRST_RUN skipFirstRun = SKIP_FIRST_RUN.NOT_SET; + SkipFirstRun skipFirstRun = SkipFirstRun.NOT_SET; boolean jitterAffinity; Supplier acquireLock = Affinity::acquireLock; long timeout; @@ -164,8 +164,8 @@ public JLBHOptions iterations(long iterations) { * @return Instance of the JLBHOptions to be used in the builder pattern. */ @NotNull - public JLBHOptions jlbhTask(JLBHTask JLBHTask) { - this.jlbhTask = JLBHTask; + public JLBHOptions jlbhTask(JLBHTask jlbhTask) { + this.jlbhTask = jlbhTask; return this; } @@ -189,7 +189,7 @@ public JLBHOptions pauseAfterWarmupMS(int pauseMS) { */ @NotNull public JLBHOptions skipFirstRun(boolean skip) { - skipFirstRun = skip ? SKIP_FIRST_RUN.SKIP : SKIP_FIRST_RUN.NO_SKIP; + skipFirstRun = skip ? SkipFirstRun.SKIP : SkipFirstRun.NO_SKIP; return this; } @@ -235,24 +235,22 @@ public JLBHOptions timeout(long timeout) { @Override public String toString() { - final StringBuffer sb = new StringBuffer("JLBHOptions{"); - sb.append("runs=").append(runs); - sb.append(", iterations=").append(iterations); - sb.append(", warmUpIterations=").append(warmUpIterations); - sb.append(", pauseAfterWarmupMS=").append(pauseAfterWarmupMS); - sb.append(", accountForCoordinatedOmission=").append(accountForCoordinatedOmission); - sb.append(", skipFirstRun=").append(skipFirstRun); - sb.append(", recordOSJitter=").append(recordOSJitter); - sb.append(", recordJitterGreaterThanNs=").append(recordJitterGreaterThanNs); - sb.append(", throughput=").append(throughput); - sb.append(", throughputTimeUnit=").append(throughputTimeUnit); - sb.append(", latencyDistributor=").append(latencyDistributor); - sb.append(", jitterAffinity=").append(jitterAffinity); - sb.append(", timeout=").append(timeout); - sb.append(", jlbhTask=").append(jlbhTask); - sb.append(", acquireLock=").append(acquireLock); - sb.append('}'); - return sb.toString(); + return "JLBHOptions{" + "runs=" + runs + + ", iterations=" + iterations + + ", warmUpIterations=" + warmUpIterations + + ", pauseAfterWarmupMS=" + pauseAfterWarmupMS + + ", accountForCoordinatedOmission=" + accountForCoordinatedOmission + + ", skipFirstRun=" + skipFirstRun + + ", recordOSJitter=" + recordOSJitter + + ", recordJitterGreaterThanNs=" + recordJitterGreaterThanNs + + ", throughput=" + throughput + + ", throughputTimeUnit=" + throughputTimeUnit + + ", latencyDistributor=" + latencyDistributor + + ", jitterAffinity=" + jitterAffinity + + ", timeout=" + timeout + + ", jlbhTask=" + jlbhTask + + ", acquireLock=" + acquireLock + + '}'; } /** @@ -266,7 +264,7 @@ public String toString() { *
  • {@link #NO_SKIP} - always include the first run.
    • * */ - enum SKIP_FIRST_RUN { + enum SkipFirstRun { NOT_SET, SKIP, NO_SKIP } } diff --git a/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java b/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java index d083354..b52f99c 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java +++ b/src/main/java/net/openhft/chronicle/jlbh/JLBHResultConsumer.java @@ -20,7 +20,6 @@ * safe cross-thread visibility. *

    */ - public interface JLBHResultConsumer extends Consumer, Supplier { /** diff --git a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java index 1d4f35f..5ff498f 100644 --- a/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java +++ b/src/main/java/net/openhft/chronicle/jlbh/util/JLBHResultSerializer.java @@ -3,12 +3,12 @@ */ package net.openhft.chronicle.jlbh.util; +import java.io.OutputStreamWriter; import net.openhft.chronicle.jlbh.JLBHResult; import org.jetbrains.annotations.NotNull; import java.io.BufferedWriter; import java.io.IOException; -import java.io.PrintWriter; import java.io.Writer; import java.nio.file.Files; import java.nio.file.Paths; @@ -16,6 +16,8 @@ import java.util.Collections; import java.util.Optional; +import static java.nio.charset.StandardCharsets.ISO_8859_1; + /** * Utility class that writes the output of a {@link net.openhft.chronicle.jlbh.JLBH} * run to a CSV file. @@ -98,7 +100,7 @@ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, String * @throws IOException if the file cannot be written */ public static void runResultToCSV(JLBHResult jlbhResult, String fileName, Iterable namesOfProbes, boolean includeOSJitter) throws IOException { - try (Writer pw = new BufferedWriter(new PrintWriter(Files.newOutputStream(Paths.get(fileName))))) { + try (Writer pw = new BufferedWriter(new OutputStreamWriter(Files.newOutputStream(Paths.get(fileName)), ISO_8859_1))) { writeHeader(pw); JLBHResult.ProbeResult probeResult = jlbhResult.endToEnd(); diff --git a/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java b/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java index 23c9a23..da46681 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java +++ b/src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java @@ -26,7 +26,6 @@ */ public class ExampleJLBHMain implements JLBHTask { private int count = 0; - private double sin; //private NanoSampler nanoSamplerSin; //private NanoSampler nanoSamplerWait; private JLBH lth; @@ -46,13 +45,10 @@ public void run(long startTimeNS) { count++; if (count == 160_000) { System.out.println("PAUSE"); - //long now = System.nanoTime(); Jvm.pause(100); - //nanoSamplerWait.sampleNanos(System.nanoTime()-now); } - long now = System.nanoTime(); - sin = Math.sin(count); + Jvm.safepoint(); // used to keep Math.sin side-effect free path removed //nanoSamplerSin.sampleNanos(System.nanoTime()-now); lth.sample(System.nanoTime() - startTimeNS); diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java index 01e8fe1..f7ca5c2 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHAdditionalCoverageTest.java @@ -104,7 +104,11 @@ private static JLBHOptions newHarness(JLBHTask task) { } private static PrintStream silentPrintStream() { - return new PrintStream(new ByteArrayOutputStream()); + try { + return new PrintStream(new ByteArrayOutputStream(), true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } private static final class AbortOnRunTask implements JLBHTask { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java index e4b181e..a183671 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHDeterministicFixtures.java @@ -11,7 +11,7 @@ public class JLBHDeterministicFixtures { static final int ITERATIONS = 9_000; private static final int THROUGHPUT = 1_000_000; private static final int RUNS = 3; - private final static String expectedOutput = "" + + private static final String expectedOutput = "" + "Warm up complete ...\n" + "-------------------------------- BENCHMARK RESULTS (RUN 1) us --------------------------------------\n" + "Run time: ...s, distribution: NORMAL\n" + @@ -89,14 +89,13 @@ static String withoutNonDeterministicFields(String content) { static class PredictableJLBHTask implements JLBHTask { int nanoTime = 1_000_000; - private int latency; private JLBH lth; private NanoSampler additionalSamplerA; private NanoSampler additionalSamplerB; @Override public void run(long startTimeNS) { - latency = 1000 + (++this.nanoTime % 11567); + int latency = 1000 + (++this.nanoTime % 11567); lth.sample(latency); additionalSamplerA.sampleNanos(latency - 1000); if (sampleB()) diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java index 7478025..29d3e44 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHIntegrationTest.java @@ -47,7 +47,12 @@ public void shouldMeasureLatency() { jlbh.start(); // then - String stdOut = outContent.toString(); + String stdOut; + try { + stdOut = outContent.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } resetSystemOut(); assertThat(stdOut, containsString("OS Jitter")); assertThat(stdOut, containsString("Warm up complete (500 iterations took ")); @@ -59,8 +64,12 @@ public void shouldMeasureLatency() { } private void redirectSystemOut() { - System.setOut(new PrintStream(outContent)); - System.setErr(new PrintStream(errContent)); + try { + System.setOut(new PrintStream(outContent, true, "UTF-8")); + System.setErr(new PrintStream(errContent, true, "UTF-8")); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } private void resetSystemOut() { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java index 4c8063a..91cbff2 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHOptionsTest.java @@ -49,7 +49,7 @@ public void shouldApplyAllConfigurationOptions() throws Exception { assertEquals(1_000_000L, getField(options, "iterations", Long.class).longValue()); assertSame(task, getField(options, "jlbhTask", JLBHTask.class)); assertEquals(77, getField(options, "pauseAfterWarmupMS", Integer.class).intValue()); - assertEquals(JLBHOptions.SKIP_FIRST_RUN.SKIP, getField(options, "skipFirstRun", JLBHOptions.SKIP_FIRST_RUN.class)); + assertEquals(JLBHOptions.SkipFirstRun.SKIP, getField(options, "skipFirstRun", JLBHOptions.SkipFirstRun.class)); assertTrue(getField(options, "jitterAffinity", Boolean.class)); assertSame(customSupplier, getField(options, "acquireLock", Supplier.class)); assertEquals(9876L, getField(options, "timeout", Long.class).longValue()); @@ -63,7 +63,7 @@ public void shouldApplyAllConfigurationOptions() throws Exception { @Test public void shouldRespectSkipFirstRunFalse() throws Exception { JLBHOptions options = new JLBHOptions().skipFirstRun(false); - assertEquals(JLBHOptions.SKIP_FIRST_RUN.NO_SKIP, getField(options, "skipFirstRun", JLBHOptions.SKIP_FIRST_RUN.class)); + assertEquals(JLBHOptions.SkipFirstRun.NO_SKIP, getField(options, "skipFirstRun", JLBHOptions.SkipFirstRun.class)); } private static T getField(JLBHOptions options, String name, Class type) throws Exception { diff --git a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java index 97fff90..cf6bc93 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java +++ b/src/test/java/net/openhft/chronicle/jlbh/JLBHTest.java @@ -15,7 +15,6 @@ import org.junit.runners.Parameterized; import java.io.ByteArrayOutputStream; -import java.io.OutputStream; import java.io.PrintStream; import java.util.ArrayList; import java.util.Arrays; @@ -63,14 +62,25 @@ private void start(JLBH jlbh) { public void shouldWriteResultToTheOutputProvided() { // given - final OutputStream outputStream = new ByteArrayOutputStream(); - final JLBH jlbh = new JLBH(options(), new PrintStream(outputStream), resultConsumer()); + final ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); + final PrintStream ps; + try { + ps = new PrintStream(outputStream, true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } + final JLBH jlbh = new JLBH(options(), ps, resultConsumer()); // when start(jlbh); // then - String result = outputStream.toString().replace("\r", ""); + String result; + try { + result = outputStream.toString("UTF-8").replace("\r", ""); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } assertThat(result, containsString("OS Jitter")); assertThat(result, containsString("Warm up complete (500 iterations took ")); assertThat(result, containsString("Run time: ")); @@ -138,7 +148,6 @@ public void shouldProvideResultData() { assertEquals(5_106L, probeALastRunSummary.get50thPercentile().toNanos(), 20); assertEquals(8_708L, probeALastRunSummary.get90thPercentile().toNanos(), 30); assertEquals(9_516L, probeALastRunSummary.get99thPercentile().toNanos(), 30); -// assertEquals(9_604L, probeALastRunSummary.get9999thPercentile().toNanos()); assertEquals(9_604L, probeALastRunSummary.getWorst().toNanos(), 30); assertEquals(probeALastRunSummary.get50thPercentile(), probeALastRunSummary.percentiles().get(PERCENTILE_50TH)); assertEquals(probeALastRunSummary.get90thPercentile(), probeALastRunSummary.percentiles().get(PERCENTILE_90TH)); @@ -187,10 +196,20 @@ public void teamCityHelper() { // then final ByteArrayOutputStream baos = new ByteArrayOutputStream(); - try (final PrintStream printStream = new PrintStream(baos)) { - TeamCityHelper.teamCityStatsLastRun("prefix", jlbh, jlbhOptions.iterations, printStream); + try { + try (final PrintStream printStream = new PrintStream(baos, true, "UTF-8")) { + TeamCityHelper.teamCityStatsLastRun("prefix", jlbh, jlbhOptions.iterations, printStream); + } + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); } String extra = Jvm.isAzulZing() ? ".zing" : Jvm.isJava15Plus() ? ".java17" : ""; + String stats; + try { + stats = baos.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } assertEquals("##teamcity[buildStatisticValue key='prefix.end-to-end.0.5" + extra + "' value='8.072']\n" + "##teamcity[buildStatisticValue key='prefix.end-to-end.0.9" + extra + "' value='11.664']\n" + "##teamcity[buildStatisticValue key='prefix.end-to-end.0.99" + extra + "' value='12.464']\n" + @@ -204,7 +223,7 @@ public void teamCityHelper() { "##teamcity[buildStatisticValue key='prefix.B.0.5" + extra + "' value='0.100125']\n" + "##teamcity[buildStatisticValue key='prefix.B.0.9" + extra + "' value='0.100125']\n" + "##teamcity[buildStatisticValue key='prefix.B.0.99" + extra + "' value='0.100125']\n" + - "##teamcity[buildStatisticValue key='prefix.B.1.0" + extra + "' value='0.100125']\n", baos.toString().replace("\r", "")); + "##teamcity[buildStatisticValue key='prefix.B.1.0" + extra + "' value='0.100125']\n", stats.replace("\r", "")); } @Test @@ -212,14 +231,25 @@ public void histogramSummariesAreCorrect() { final JLBHResultConsumer resultConsumer = resultConsumer(); JLBHOptions jlbhOptions = options().jlbhTask(new PredictableJLBHTaskDifferentShape()).iterations(ITERATIONS * 2); final ByteArrayOutputStream baos = new ByteArrayOutputStream(); - final PrintStream printStream = new PrintStream(baos); + final PrintStream printStream; + try { + printStream = new PrintStream(baos, true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } final JLBH jlbh = new JLBH(jlbhOptions, printStream, resultConsumer); // when start(jlbh); System.out.println(baos); - assertTrue(baos.toString().replace("\r", "").contains( + String summary; + try { + summary = baos.toString("UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } + assertTrue(summary.replace("\r", "").contains( "-------------------------------- SUMMARY (B) us ----------------------------------------------------\n" + "Percentile run1 run2 run3 % Variation\n" + "50.0: 0.10 0.10 0.10 0.00\n" + @@ -307,6 +337,10 @@ private JLBHResultConsumer resultConsumer() { @NotNull private PrintStream printStream() { - return new PrintStream(new ByteArrayOutputStream()); + try { + return new PrintStream(new ByteArrayOutputStream(), true, "UTF-8"); + } catch (java.io.UnsupportedEncodingException e) { + throw new AssertionError("UTF-8 not supported", e); + } } } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java index 55988da..d4d4a9f 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleBenchmark.java @@ -15,7 +15,8 @@ public static void main(String[] args) { .warmUpIterations(20_000) .iterations(1_000_000) .throughput(100_000) -// .accountForCoordinatedOmission(true) + // enable this if you need to compensate for coordinated omission + // .accountForCoordinatedOmission(true) .runs(2) .jlbhTask(new SimpleBenchmark()); new JLBH(lth).start(); @@ -28,11 +29,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to include work done inside this method. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); } } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java index a64b46d..a611aac 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java @@ -43,11 +43,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to measure the local work. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); myProbe.sampleNanos(delta); } diff --git a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java index 3baebd0..b01b60f 100644 --- a/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java +++ b/src/test/java/net/openhft/chronicle/jlbh/SimpleResultSerializerBenchmark.java @@ -13,15 +13,16 @@ public class SimpleResultSerializerBenchmark implements JLBHTask { private JLBH jlbh; public static void main(String[] args) { - //Create the JLBH options you require for the benchmark + // Create the JLBH options you require for the benchmark JLBHOptions lth = new JLBHOptions() .warmUpIterations(20_000) .iterations(1_000_000) .throughput(100_000) -// .accountForCoordinatedOmission(true) + // enable this if you need to compensate for coordinated omission + // .accountForCoordinatedOmission(true) .runs(2) - .jlbhTask(new SimpleResultSerializerBenchmark() ); - new JLBH(lth, System.out,jlbhResult -> { + .jlbhTask(new SimpleResultSerializerBenchmark()); + new JLBH(lth, System.out, jlbhResult -> { try { System.out.println("Serializing result..."); JLBHResultSerializer.runResultToCSV(jlbhResult); @@ -40,11 +41,12 @@ public void init(JLBH jlbh) { @Override public void run(long startTimeNS) { -// long start = System.nanoTime(); // (1) - long start = startTimeNS; // (2) + // Use System.nanoTime() here if you only want to measure the local work. + // long start = System.nanoTime(); // (1) + // (2) LockSupport.parkNanos(1); - final long delta = System.nanoTime() - start; + final long delta = System.nanoTime() - startTimeNS; jlbh.sample(delta); } } diff --git a/systemProperties.adoc b/systemProperties.adoc new file mode 100644 index 0000000..4ded78a --- /dev/null +++ b/systemProperties.adoc @@ -0,0 +1,24 @@ += JLBH System Properties +:toc: +:lang: en-GB +:source-highlighter: rouge + +== System Properties + +JLBH is a micro-benchmark harness that is sensitive to any extra work done by the JVM. +It checks the `jvm.resource.tracing` property to avoid running when intrusive resource tracing is enabled. + +.System properties +[cols="2a,1,3a,2a",options="header"] +|=== +| Property Key +| Default Value +| Description +| Usage (Type) + +| `jvm.resource.tracing` +| not set or `false` +| When this property is set with an empty value or `true`, JLBH prints a warning and terminates the JVM rather than running the benchmark, because resource tracing would distort latency and allocation measurements. Ensure this property is unset or explicitly `false` when running JLBH. +| Checked in the `JLBH` constructor (`String` value parsed as `boolean`) +|=== +