Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add measureAsyncFunction function #556

Merged
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/popular-seahorses-provide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'reassure': minor
'@callstack/reassure-measure': minor
---

feat: add measureAsyncFunction
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@
- [`MeasureRendersOptions` type](#measurerendersoptions-type)
- [`measureFunction` function](#measurefunction-function)
- [`MeasureFunctionOptions` type](#measurefunctionoptions-type)
- [`measureAsyncFunction` function](#measureasyncfunction-function)
- [`MeasureAsyncFunctionOptions` type](#measureasyncfunctionoptions-type)
- [Configuration](#configuration)
- [Default configuration](#default-configuration)
- [`configure` function](#configure-function)
Expand Down Expand Up @@ -397,11 +399,40 @@ async function measureFunction(
interface MeasureFunctionOptions {
runs?: number;
warmupRuns?: number;
writeFile?: boolean;
}
```

- **`runs`**: number of runs per series for the particular test
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

#### `measureAsyncFunction` function

Allows you to wrap any **asynchronous** function, measure its execution times and write results to the output file. You can use optional `options` to customize aspects of the testing. Note: the execution count will always be one.

> **Note**: Measuring asynchronous functions can be useful when, during its execution, they rely or need to get some data from async providers e.g. storage / network and we are purposely disconsidering their impact during the test as we want only to measure the rest of the function's logic. **With that in mind, make sure these providers are properly mocked during test environment so they don't pollute your measurements.**

```ts
async function measureAsyncFunction(
fn: () => Promise<any>,
options?: MeasureAsyncFunctionOptions
): Promise<MeasureResults> {
```

#### `MeasureAsyncFunctionOptions` type

```ts
interface MeasureAsyncFunctionOptions {
runs?: number;
warmupRuns?: number;
writeFile?: boolean;
}
```

- **`runs`**: number of runs per series for the particular test
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

### Configuration

Expand Down
47 changes: 47 additions & 0 deletions docusaurus/docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,53 @@ interface MeasureFunctionOptions {
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

### `measureAsyncFunction` function {#measure-async-function}

Allows you to wrap any **asynchronous** function, measure its performance and write results to the output file. You can use optional `options` to customize aspects of the testing.

:::info

Measuring asynchronous functions can be useful when, during its execution, they rely or need to get some data from async providers e.g. storage / network and we are purposely disconsidering their impact during the test as we want only to measure the rest of the function's logic. **With that in mind, make sure these providers are properly mocked during test environment so they don't pollute your measurements.**

:::

```ts
async function measureAsyncFunction(
fn: () => Promise<any>,
options?: MeasureAsyncFunctionOptions,
): Promise<MeasureResults> {
```

#### Example {#measure-async-function-example}

```ts
// sample.perf-test.tsx
import { measureAsyncFunction } from 'reassure';
import { fib } from './fib';

test('fib 30', async () => {
await measureAsyncFunction(async () => {
const asyncLogic = () => Promise.resolve(30);
const result = await asyncLogic();
fib(result);
});
});
```

### `MeasureAsyncFunctionOptions` type {#measure-async-function-options}

```ts
interface MeasureAsyncFunctionOptions {
runs?: number;
warmupRuns?: number;
writeFile?: boolean;
}
```

- **`runs`**: number of runs per series for the particular test
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

## Configuration

### Default configuration
Expand Down
14 changes: 14 additions & 0 deletions packages/measure/src/__tests__/measure-function.test.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import stripAnsi from 'strip-ansi';
import { measureFunction } from '../measure-function';
import { measureAsyncFunction } from '../measure-async-function';
import { setHasShownFlagsOutput } from '../output';

// Exponentially slow function
Expand All @@ -20,6 +21,19 @@ test('measureFunction captures results', async () => {
expect(results.counts).toEqual([1]);
});

test('measureAsyncFunction captures results', async () => {
const fn = jest.fn(async () => {
const asyncLogic = () => Promise.resolve(5);
const result = await asyncLogic();
fib(result);
});
const results = await measureAsyncFunction(fn, { runs: 1, warmupRuns: 0, writeFile: false });

expect(fn).toHaveBeenCalledTimes(1);
expect(results.runs).toBe(1);
expect(results.counts).toEqual([1]);
});

test('measureFunction runs specified number of times', async () => {
const fn = jest.fn(() => fib(5));
const results = await measureFunction(fn, { runs: 20, warmupRuns: 0, writeFile: false });
Expand Down
2 changes: 2 additions & 0 deletions packages/measure/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
export { configure, resetToDefaults } from './config';
export { measureRenders, measurePerformance } from './measure-renders';
export { measureFunction } from './measure-function';
export { measureAsyncFunction } from './measure-async-function';
export type { MeasureRendersOptions } from './measure-renders';
export type { MeasureFunctionOptions } from './measure-function';
export type { MeasureAsyncFunctionOptions } from './measure-async-function';
export type { MeasureType, MeasureResults } from './types';
42 changes: 42 additions & 0 deletions packages/measure/src/measure-async-function.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import { config } from './config';
import type { MeasureResults } from './types';
import { type RunResult, getCurrentTime, processRunResults } from './measure-helpers';
import { showFlagsOutputIfNeeded, writeTestStats } from './output';
import { MeasureFunctionOptions } from './measure-function';

export interface MeasureAsyncFunctionOptions extends MeasureFunctionOptions {}

export async function measureAsyncFunction(
fn: () => Promise<any>,
options?: MeasureAsyncFunctionOptions
): Promise<MeasureResults> {
const stats = await measureAsyncFunctionInternal(fn, options);

if (options?.writeFile !== false) {
await writeTestStats(stats, 'function');
}

return stats;
}

async function measureAsyncFunctionInternal(
fn: () => Promise<any>,
options?: MeasureAsyncFunctionOptions
): Promise<MeasureResults> {
const runs = options?.runs ?? config.runs;
const warmupRuns = options?.warmupRuns ?? config.warmupRuns;

showFlagsOutputIfNeeded();

const runResults: RunResult[] = [];
for (let i = 0; i < runs + warmupRuns; i += 1) {
const timeStart = getCurrentTime();
await fn();
const timeEnd = getCurrentTime();

const duration = timeEnd - timeStart;
runResults.push({ duration, count: 1 });
}

return processRunResults(runResults, warmupRuns);
}
9 changes: 2 additions & 7 deletions packages/measure/src/measure-function.tsx
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { performance } from 'perf_hooks';
import { config } from './config';
import type { MeasureResults } from './types';
import { type RunResult, processRunResults } from './measure-helpers';
import { type RunResult, getCurrentTime, processRunResults } from './measure-helpers';
import { showFlagsOutputIfNeeded, writeTestStats } from './output';

export interface MeasureFunctionOptions {
Expand All @@ -11,7 +10,7 @@ export interface MeasureFunctionOptions {
}

export async function measureFunction(fn: () => void, options?: MeasureFunctionOptions): Promise<MeasureResults> {
const stats = await measureFunctionInternal(fn, options);
const stats = measureFunctionInternal(fn, options);

if (options?.writeFile !== false) {
await writeTestStats(stats, 'function');
Expand All @@ -38,7 +37,3 @@ function measureFunctionInternal(fn: () => void, options?: MeasureFunctionOption

return processRunResults(runResults, warmupRuns);
}

function getCurrentTime() {
return performance.now();
}
5 changes: 5 additions & 0 deletions packages/measure/src/measure-helpers.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { performance } from 'perf_hooks';
import * as math from 'mathjs';
import type { MeasureResults } from './types';

Expand All @@ -6,6 +7,10 @@ export interface RunResult {
count: number;
}

export function getCurrentTime() {
return performance.now();
}

export function processRunResults(inputResults: RunResult[], warmupRuns: number): MeasureResults {
const warmupResults = inputResults.slice(0, warmupRuns);
const results = inputResults.slice(warmupRuns);
Expand Down
31 changes: 31 additions & 0 deletions packages/reassure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@
- [`MeasureRendersOptions` type](#measurerendersoptions-type)
- [`measureFunction` function](#measurefunction-function)
- [`MeasureFunctionOptions` type](#measurefunctionoptions-type)
- [`measureAsyncFunction` function](#measureasyncfunction-function)
- [`MeasureAsyncFunctionOptions` type](#measureasyncfunctionoptions-type)
- [Configuration](#configuration)
- [Default configuration](#default-configuration)
- [`configure` function](#configure-function)
Expand Down Expand Up @@ -397,11 +399,40 @@ async function measureFunction(
interface MeasureFunctionOptions {
runs?: number;
warmupRuns?: number;
writeFile?: boolean;
}
```

- **`runs`**: number of runs per series for the particular test
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

#### `measureAsyncFunction` function

Allows you to wrap any **asynchronous** function, measure its execution times and write results to the output file. You can use optional `options` to customize aspects of the testing. Note: the execution count will always be one.

> **Note**: Measuring asynchronous functions can be useful when, during its execution, they rely or need to get some data from async providers e.g. storage / network and we are purposely disconsidering their impact during the test as we want only to measure the rest of the function's logic. **With that in mind, make sure these providers are properly mocked during test environment so they don't pollute your measurements.**

```ts
async function measureAsyncFunction(
fn: () => Promise<any>,
options?: MeasureAsyncFunctionOptions
): Promise<MeasureResults> {
```

#### `MeasureAsyncFunctionOptions` type

```ts
interface MeasureAsyncFunctionOptions {
runs?: number;
warmupRuns?: number;
writeFile?: boolean;
}
```

- **`runs`**: number of runs per series for the particular test
- **`warmupRuns`**: number of additional warmup runs that will be done and discarded before the actual runs.
- **`writeFile`**: (default `true`) should write output to file.

### Configuration

Expand Down
2 changes: 2 additions & 0 deletions packages/reassure/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export {
measureRenders,
measureFunction,
measureAsyncFunction,
configure,
resetToDefaults,
measurePerformance,
Expand All @@ -11,6 +12,7 @@ export type {
MeasureResults,
MeasureRendersOptions,
MeasureFunctionOptions,
MeasureAsyncFunctionOptions,
MeasureType,
} from '@callstack/reassure-measure';
export type {
Expand Down
13 changes: 12 additions & 1 deletion test-apps/native/src/fib.perf.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
import { measureFunction } from '@callstack/reassure-measure';
import {
measureFunction,
measureAsyncFunction,
} from '@callstack/reassure-measure';

function fib(n: number): number {
if (n <= 1) {
Expand All @@ -22,4 +25,12 @@ describe('`fib` function', () => {
test('fib(32)', async () => {
await measureFunction(() => fib(32));
});

test('fib(32) async', async () => {
await measureAsyncFunction(async () => {
const asyncLogic = () => Promise.resolve(32);
const result = await asyncLogic();
fib(result);
});
});
});