Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -981,6 +981,11 @@ lazy val testkit = crossProject(JSPlatform, JVMPlatform, NativePlatform)
name := "cats-effect-testkit",
libraryDependencies ++= Seq(
"org.scalacheck" %%% "scalacheck" % ScalaCheckVersion
),
mimaBinaryIssueFilters ++= Seq(
// introduced by #3309, add clockStart parameter to TestControl methods
ProblemFilters.exclude[DirectMissingMethodProblem]("cats.effect.testkit.TestControl.execute"),
ProblemFilters.exclude[DirectMissingMethodProblem]("cats.effect.testkit.TestControl.executeEmbed")
)
)
.nativeSettings(nativeTestSettings)
Expand Down
49 changes: 48 additions & 1 deletion docs/core/test-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,59 @@ test("retry at least 3 times until success") {

In this test (written using [MUnit Cats Effect](https://github.com/typelevel/munit-cats-effect)), the `action` program counts the number of `attempts` and only produces `"success!"` precisely on the third try. Every other time, it raises a `TestException`. This program is then transformed by `retry` with a `1.minute` delay and a maximum of 5 attempts.

Under the production `IO` runtime, this test could take up to 31 minutes to run! With `TestControl.executeEmbed`, it requires a few milliseconds at most. The `executeEmbed` function takes an `IO[A]`, along with an optional `IORuntimeConfig` and random seed (which will be used to govern the sequencing of "parallel" fibers during the run) and produces an `IO[A]` which runs the given `IO` fully to completion. If the `IO` under test throws an exception, is canceled, or *fails to terminate*, `executeEmbed` will raise an exception in the resulting `IO[A]` which will cause the entire test to fail.
Under the production `IO` runtime, this test could take up to 31 minutes to run! With `TestControl.executeEmbed`, it requires a few milliseconds at most. The `executeEmbed` function takes an `IO[A]`, along with an optional `IORuntimeConfig`, random seed (which will be used to govern the sequencing of "parallel" fibers during the run), and `clockStart` time offset, and produces an `IO[A]` which runs the given `IO` fully to completion. If the `IO` under test throws an exception, is canceled, or *fails to terminate*, `executeEmbed` will raise an exception in the resulting `IO[A]` which will cause the entire test to fail.

> Note: Because `TestControl` is a mock, nested `IO` runtime, it is able to detect certain forms of non-termination within the programs under test! In particular, programs like `IO.never` and similar will be correctly detected as deadlocked and reported as such. However, programs which never terminate but do *not* deadlock, such as `IO.unit.foreverM`, cannot be detected and will simply never terminate when run via `TestControl`. Unfortunately, it cannot provide a general solution to the [Halting Problem](https://en.wikipedia.org/wiki/Halting_problem).

In this case, we're testing that the program eventually retries its way to success, and we're doing it without having to wait for real clock-time `sleep`s. For *most* scenarios involving mocked time, this kind of functionality is sufficient.

### Starting with a Specific Time

In many testing scenarios, particularly when testing time-sensitive functionality like signature verification or expiration logic, you may want your program to start at a specific point in time rather than at epoch (time zero). Both `execute` and `executeEmbed` accept an optional `clockStart` parameter that allows you to set the initial time before your program begins execution.

```scala
test("verify signature at specific timestamp") {
// Unix timestamp for "Tue, 20 Apr 2021 02:07:55 GMT"
val signatureTime = 1618884475.seconds

val program = for {
now <- IO.realTime
result <- verifyHttpSignature(signedMessage)
} yield (now, result)

TestControl.executeEmbed(program, clockStart = signatureTime).flatMap { case (timestamp, isValid) =>
IO {
assertEquals(timestamp, signatureTime)
assert(isValid) // signature should be valid at this time
}
}
}
```

This is much more convenient than manually advancing time after program creation:

```scala
// Before clockStart parameter - more verbose
TestControl.execute(program).flatMap { control =>
for {
_ <- control.tick
_ <- control.advance(signatureTime)
_ <- control.tick
result <- control.results
} yield result
}

// With clockStart parameter - simpler
TestControl.executeEmbed(program, clockStart = signatureTime)
```

The `clockStart` parameter is particularly useful for:

- Testing time-bound authentication tokens or signatures
- Verifying expiration logic at specific timestamps
- Simulating programs that run at different times of day
- Testing time-sensitive business logic without complex time manipulation

### Stepping Through the Program

For more advanced cases, `executeEmbed` may not be enough to properly measure the functionality under test. For example, if we want to write a test for `retry` which shows that it always `sleep`s for some time interval that is between zero and the exponentially-growing maximum `delay`. This is relatively difficult to do in terms of `executeEmbed` without adding some side-channel to `retry` itself which reports elapsed time with each loop.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -360,13 +360,18 @@ object TestControl {
def execute[A](
program: IO[A],
config: IORuntimeConfig = IORuntimeConfig(),
seed: Option[String] = None): IO[TestControl[A]] =
seed: Option[String] = None,
clockStart: FiniteDuration = Duration.Zero): IO[TestControl[A]] =
IO {
val ctx = seed match {
case Some(seed) => TestContext(seed)
case None => TestContext()
}

if (clockStart > Duration.Zero) {
ctx.advance(clockStart)
}

val runtime: IORuntime = IORuntime(
ctx,
ctx.deriveBlocking(),
Expand Down Expand Up @@ -416,8 +421,9 @@ object TestControl {
def executeEmbed[A](
program: IO[A],
config: IORuntimeConfig = IORuntimeConfig(),
seed: Option[String] = None): IO[A] =
execute(program, config = config, seed = seed) flatMap { c =>
seed: Option[String] = None,
clockStart: FiniteDuration = Duration.Zero): IO[A] =
execute(program, config = config, seed = seed, clockStart = clockStart) flatMap { c =>
val nt = new (Id ~> IO) { def apply[E](e: E) = IO.pure(e) }

val onCancel = IO.defer(IO.raiseError(new CancellationException()))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,72 @@ class TestControlSuite extends BaseSuite {
}
}

real("execute - clockStart parameter sets initial time") {
val clockStart = 1618884473.seconds
val program = IO.realTime

TestControl.execute(program, clockStart = clockStart) flatMap { control =>
for {
_ <- control.tick
r <- control.results
_ <- IO(assertEquals(r, Some(beSucceeded(clockStart))))
} yield ()
}
}

real("executeEmbed - clockStart parameter sets initial time") {
val clockStart = 1618884473.seconds
val program = IO.realTime

TestControl.executeEmbed(program, clockStart = clockStart) flatMap { result =>
IO(assertEquals(result, clockStart))
}
}

real("execute - clockStart with monotonic time") {
val clockStart = 42.minutes
val program = IO.monotonic

TestControl.execute(program, clockStart = clockStart) flatMap { control =>
for {
_ <- control.tick
r <- control.results
_ <- IO(assertEquals(r, Some(beSucceeded(clockStart))))
} yield ()
}
}

real("execute - clockStart zero has no effect") {
val program = IO.realTime

TestControl.execute(program, clockStart = Duration.Zero) flatMap { control =>
for {
_ <- control.tick
r <- control.results
_ <- IO(assertEquals(r, Some(beSucceeded(Duration.Zero))))
} yield ()
}
}

real("execute - clockStart with sleep and time progression") {
val clockStart = 1.hour
val sleepDuration = 30.minutes
val program = for {
start <- IO.realTime
_ <- IO.sleep(sleepDuration)
end <- IO.realTime
} yield (start, end)

TestControl.execute(program, clockStart = clockStart) flatMap { control =>
for {
_ <- control.tick
_ <- control.advanceAndTick(sleepDuration)
r <- control.results
_ <- IO(assertEquals(r, Some(beSucceeded((clockStart, clockStart + sleepDuration)))))
} yield ()
}
}

private def beSucceeded[A](value: A): Outcome[Id, Throwable, A] =
Outcome.succeeded[Id, Throwable, A](value)
}
Loading