Skip to content

Commit 1b8f13e

Browse files
grynspangrynspan
committed
[ST-NNNN] Test cancellation
1 parent 227f3c2 commit 1b8f13e

File tree

1 file changed

+398
-0
lines changed

1 file changed

+398
-0
lines changed

‎proposals/testing/NNNN-test-cancellation.md

Lines changed: 398 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,398 @@
1+
# Test cancellation
2+
3+
* Proposal: [ST-NNNN](NNNN-test-cancellation.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Bug: [swiftlang/swift-testing#120](https://github.com/swiftlang/swift-testing/issues/120)
8+
* Implementation: [swiftlang/swift-testing#1284](https://github.com/swiftlang/swift-testing/pull/1284)
9+
* Review: ([pitch](https://forums.swift.org/t/pitch-test-cancellation/81847))
10+
11+
## Introduction
12+
13+
Swift Testing provides the ability to conditionally skip a test before it runs
14+
using the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)),
15+
[`.disabled(if:)`](https://developer.apple.com/documentation/testing/trait/disabled(if:_:sourcelocation:)),
16+
etc. family of traits:
17+
18+
```swift
19+
@Test(.enabled(if: Tyrannosaurus.isTheLizardKing))
20+
func `Tyrannosaurus is scary`() {
21+
let dino = Tyrannosaurus()
22+
#expect(dino.isScary)
23+
// ...
24+
}
25+
```
26+
27+
This proposal extends that feature to allow cancelling a test after it has
28+
started but before it has ended.
29+
30+
## Motivation
31+
32+
We have received feedback from a number of developers indicating that their
33+
tests have constraints that can only be checked after a test has started, and
34+
they would like the ability to end a test early and see that state change
35+
reflected in their development tools.
36+
37+
To date, we have not provided an API for ending a test's execution early because
38+
we want to encourage developers to use the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
39+
_et al._ trait. This trait can be evaluated early and lets Swift Testing plan a
40+
test run more efficiently. However, we recognize that these traits aren't
41+
sufficient. Some test constraints are dependent on data that isn't available
42+
until the test starts, while others only apply to specific test cases in a
43+
parameterized test function.
44+
45+
## Proposed solution
46+
47+
A static `cancel()` function is added to the [`Test`](https://developer.apple.com/documentation/testing/test)
48+
and [`Test.Case`](https://developer.apple.com/documentation/testing/test/case)
49+
types. When a test author calls these functions from within the body of a test
50+
(or from within the implementation of a trait, e.g. from [`prepare(for:)`](https://developer.apple.com/documentation/testing/trait/prepare(for:))),
51+
Swift Testing cancels the currently-running test or test case, respectively.
52+
53+
### Relationship between tasks and tests
54+
55+
Each test runs in its own task during a test run, and each test case in a test
56+
also runs in its own task. Cancelling the current task from within the body of a
57+
test will, therefore, cancel the current test case, but not the current test:
58+
59+
```swift
60+
@Test(arguments: Species.all(in: .dinosauria))
61+
func `Are all dinosaurs extinct?`(_ species: Species) {
62+
if species.in(.aves) {
63+
// Birds aren't extinct (I hope)
64+
withUnsafeCurrentTask { $0?.cancel() }
65+
return
66+
}
67+
// ...
68+
}
69+
```
70+
71+
Using [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
72+
here is not ideal. It's not clear that the intent is to cancel the test case,
73+
and [`UnsafeCurrentTask`](https://developer.apple.com/documentation/swift/unsafecurrenttask)
74+
is, unsurprisingly, an unsafe interface.
75+
76+
> [!NOTE]
77+
> The version of Swift Testing included with Swift 6.2 does not correctly handle
78+
> task cancellation under all conditions. See [swiftlang/swift-testing#1289](https://github.com/swiftlang/swift-testing/issues/1289).
79+
80+
## Detailed design
81+
82+
New static members are added to [`Test`](https://developer.apple.com/documentation/testing/test)
83+
and [`Test.Case`](https://developer.apple.com/documentation/testing/test/case):
84+
85+
```swift
86+
extension Test {
87+
/// Cancel the current test.
88+
///
89+
/// - Parameters:
90+
/// - comment: A comment describing why you are cancelling the test.
91+
/// - sourceLocation: The source location to which the testing library will
92+
/// attribute the cancellation.
93+
///
94+
/// - Throws: An error indicating that the current test case has been
95+
/// cancelled.
96+
///
97+
/// The testing library runs each test in its own task. When you call this
98+
/// function, the testing library cancels the task associated with the current
99+
/// test:
100+
///
101+
/// ```swift
102+
/// @Test func `Food truck is well-stocked`() throws {
103+
/// guard businessHours.contains(.now) else {
104+
/// try Test.cancel("We're off the clock.")
105+
/// }
106+
/// // ...
107+
/// }
108+
/// ```
109+
///
110+
/// If the current test is parameterized, all of its pending and running test
111+
/// cases are cancelled. If the current test is a suite, all of its pending
112+
/// and running tests are cancelled. If you have already cancelled the current
113+
/// test or if it has already finished running, this function throws an error
114+
/// but does not attempt to cancel the test a second time.
115+
///
116+
/// - Important: If the current task is not associated with a test (for
117+
/// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1))
118+
/// this function records an issue and cancels the current task.
119+
///
120+
/// To cancel the current test case but leave other test cases of the current
121+
/// test alone, call ``Test/Case/cancel(_:sourceLocation:)`` instead.
122+
public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never
123+
}
124+
125+
extension Test.Case {
126+
/// Cancel the current test case.
127+
///
128+
/// - Parameters:
129+
/// - comment: A comment describing why you are cancelling the test case.
130+
/// - sourceLocation: The source location to which the testing library will
131+
/// attribute the cancellation.
132+
///
133+
/// - Throws: An error indicating that the current test case has been
134+
/// cancelled.
135+
///
136+
/// The testing library runs each test case of a test in its own task. When
137+
/// you call this function, the testing library cancels the task associated
138+
/// with the current test case:
139+
///
140+
/// ```swift
141+
/// @Test(arguments: [Food.burger, .fries, .iceCream])
142+
/// func `Food truck is well-stocked`(_ food: Food) throws {
143+
/// if food == .iceCream && Season.current == .winter {
144+
/// try Test.Case.cancel("It's too cold for ice cream.")
145+
/// }
146+
/// // ...
147+
/// }
148+
/// ```
149+
///
150+
/// If the current test is parameterized, the test's other test cases continue
151+
/// running. If the current test case has already been cancelled, this
152+
/// function throws an error but does not attempt to cancel the test case a
153+
/// second time.
154+
///
155+
/// - Important: If the current task is not associated with a test case (for
156+
/// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1))
157+
/// this function records an issue and cancels the current task.
158+
///
159+
/// To cancel all test cases in the current test, call
160+
/// ``Test/cancel(_:sourceLocation:)`` instead.
161+
public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never
162+
}
163+
```
164+
165+
These functions behave similarly, and are distinguished by the level of the test
166+
to which they apply:
167+
168+
- `Test.cancel()` cancels the current test.
169+
- If the current test is parameterized, it implicitly cancels all running and
170+
pending test cases of said test.
171+
- If the current test is a suite (only applicable during trait evaluation), it
172+
recursively cancels all test suites and test functions within said suite.
173+
- `Test.Case.cancel()` cancels the current test case.
174+
- If the current test is parameterized, other test cases are unaffected.
175+
- If the current test is _not_ parameterized, `Test.Case.cancel()` behaves the
176+
same as `Test.cancel()`.
177+
178+
Cancelling a test or test case implicitly cancels its associated task (and any
179+
child tasks thereof) as if [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel())
180+
were called on that task.
181+
182+
### Throwing semantics
183+
184+
Unlike [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()),
185+
these functions always throw an error instead of returning. This simplifies
186+
control flow when a test is cancelled; instead of having to write:
187+
188+
```swift
189+
if condition {
190+
theTask.cancel()
191+
return
192+
}
193+
```
194+
195+
A test author need only write:
196+
197+
```swift
198+
if condition {
199+
try Test.cancel()
200+
}
201+
```
202+
203+
The errors these functions throw are of a type internal to Swift Testing that is
204+
semantically similar to [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
205+
but carries additional information (namely the `comment` and `sourceLocation`
206+
arguments to `cancel(_:sourceLocation:)`) that Swift Testing can present to the
207+
user. When Swift Testing catches an error of this type[^cancellationErrorToo],
208+
it does not record an issue for the current test or test case.
209+
210+
[^cancellationErrorToo]: Swift Testing also catches errors of type
211+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
212+
if the current task has been cancelled. If the current task has not been
213+
cancelled, errors of this type are still recorded as issues.
214+
215+
Suppressing these errors with `do`/`catch` or `try?` does not uncancel a test,
216+
test case, or task, but can be useful if you have additional local work you need
217+
to do before the test or test case ends.
218+
219+
### Support for CancellationError
220+
221+
Cancelling a test's or test case's associated task is equivalent to cancelling
222+
the test or test case. Hence, if a test or test case throws an instance of
223+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
224+
_and_ the current task has been cancelled, it is treated as if the test or test
225+
case were cancelled.
226+
227+
### Support for XCTSkip
228+
229+
XCTest has an approximate equivalent to test cancellation: throwing an instance
230+
of [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
231+
from the body of an XCTest test function causes that test function to be skipped
232+
(equivalent to cancelling it).
233+
234+
While we encourage developers to adopt `Test.cancel()` and `Test.Case.cancel()`,
235+
we recognize the need for interoperability with XCTest. As such, Swift Testing
236+
will recognize when a test or test case throws an instance of
237+
[`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
238+
and will treat it as cancelling the test or test case.
239+
240+
An instance of [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
241+
must be caught by Swift Testing in order for it to cancel the current test. It
242+
is not sufficient to create and discard an instance of this error type or to
243+
catch one before Swift Testing can catch it. This behavior is consistent with
244+
that of XCTest.
245+
246+
> [!NOTE]
247+
> This compatibility does **not** extend to Objective-C. In Objective-C, XCTest
248+
> implements [`XCTSkip()`](https://developer.apple.com/documentation/xctest/xctskip-c.macro?language=objc)
249+
> as a macro that throws an Objective-C exception. Exceptions are not supported
250+
> in Swift, and Swift Testing does not attempt to catch these exceptions.
251+
252+
### Interaction with recorded issues
253+
254+
If you cancel a test or test case that has previously recorded an issue, that
255+
issue is not overridden or nullified. In particular, if the test or test case
256+
has already recorded an issue of severity **error** when you call
257+
`cancel(_:sourceLocation:)`, the test or test case will still fail.
258+
259+
### Example usage
260+
261+
To cancel the current test case and let other test cases run:
262+
263+
```swift
264+
@Test(arguments: Species.all(in: .dinosauria))
265+
func `Are all dinosaurs extinct?`(_ species: Species) throws {
266+
if species.in(.aves) {
267+
try Test.Case.cancel("\(species) is birds!")
268+
}
269+
// ...
270+
}
271+
```
272+
273+
Or, to cancel all remaining test cases in the current test:
274+
275+
```swift
276+
@Test(arguments: Species.all(in: .dinosauria))
277+
func `Are all dinosaurs extinct?`(_ species: Species) throws {
278+
if species.is(.godzilla) {
279+
try Test.cancel("Forget about unit tests! Run for your life!")
280+
}
281+
// ...
282+
}
283+
```
284+
285+
## Source compatibility
286+
287+
This change is additive only.
288+
289+
## Integration with supporting tools
290+
291+
The JSON event stream Swift Testing provides is updated to include two new event
292+
kinds:
293+
294+
```diff
295+
<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
296+
"issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" |
297+
- "runEnded" | "valueAttached"
298+
+ "runEnded" | "valueAttached" | "testCancelled" | "testCaseCancelled"
299+
```
300+
301+
And new fields are added to event records to represent the comment and source
302+
location passed to `cancel(_:sourceLocation:)`:
303+
304+
```diff
305+
<event> ::= {
306+
"kind": <event-kind>,
307+
"instant": <instant>, ; when the event occurred
308+
["issue": <issue>,] ; the recorded issue (if "kind" is "issueRecorded")
309+
["attachment": <attachment>,] ; the attachment (if kind is "valueAttached")
310+
"messages": <array:message>,
311+
["testID": <test-id>,]
312+
+ ["comments": <array:string>,]
313+
+ ["sourceLocation": <source-location>,]
314+
}
315+
```
316+
317+
These new fields are populated for the new event kinds as well as other event
318+
kinds that can populate them.
319+
320+
These new event kinds and fields will be included in the next revision of the
321+
JSON schema (currently expected to be schema version `"6.3"`).
322+
323+
## Future directions
324+
325+
- Adding a corresponding `Test.checkCancellation()` function and/or
326+
`Test.isCancelled` static property. These are beyond the scope of this
327+
proposal, primarily because [`Task.isCancelled`](https://developer.apple.com/documentation/swift/task/iscancelled-swift.type.property)
328+
and [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
329+
already work in a test.
330+
331+
## Alternatives considered
332+
333+
- Doing nothing. While we do want test authors to use [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
334+
_et al._ trait, we recognize it does not provide the full set of functionality
335+
that test authors need.
336+
337+
- Ignoring task cancellation or treating [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
338+
as a normal error even when the current task has been cancelled. It is not
339+
possible for Swift Testing to outright ignore task cancellation, and a
340+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
341+
instance thrown from [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
342+
is not really a test issue but rather a manifestation of control flow.
343+
344+
- Using the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
345+
type from XCTest. Interoperation with XCTest is an area of exploration for us,
346+
but core functionality of Swift Testing needs to be usable without also
347+
importing XCTest.
348+
349+
- Spelling the functions `static func cancel(_:sourceLocation:) -> some Error`
350+
and requiring it be called as `throw Test.cancel()`. This is closer to how
351+
the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
352+
type is used in XCTest. We have received indirect feedback about [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
353+
indicating its usage is unclear, and sometimes need to help developers who
354+
have written:
355+
356+
```swift
357+
if x {
358+
XCTSkip()
359+
}
360+
```
361+
362+
And don't understand why it has failed to stop the test. More broadly, it is
363+
not common practice in Swift for a function to return an error that the caller
364+
is then responsible for throwing.
365+
366+
- Providing additional `cancel(if:)` and `cancel(unless:)` functions. In
367+
Objective-C, XCTest provides the [`XCTSkipIf()`](https://developer.apple.com/documentation/xctest/xctskipif)
368+
and [`XCTSkipUnless()`](https://developer.apple.com/documentation/xctest/xctskipunless)
369+
macros which capture their condition arguments as strings for display to the
370+
test author. This functionality is not available in Swift, but XCTest's Swift
371+
interface provides equivalent throwing functions as conveniences. We could
372+
provide these functions (without any sort of string-capturing ability) too,
373+
but they provide little additional clarity above an `if` or `guard` statement.
374+
375+
- Implementing cancellation using Swift macros so we can capture an `if` or
376+
`unless` argument as a string. A macro for this feature is probably the wrong
377+
tradeoff between compile-time magic and technical debt.
378+
379+
- Relying solely on [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()).
380+
Ignoring the interplay between tests and test cases, this approach is
381+
difficult for test authors to use because the current [`Task`](https://developer.apple.com/documentation/swift/task)
382+
instance isn't visible _within_ that task. Instead, a test author would need
383+
to use [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
384+
to get a temporary reference to the task and cancel _that_ value. We would
385+
also not have the ability to include a comment and source location information
386+
in the test's console output or an IDE's test result interface.
387+
388+
With that said, [`UnsafeCurrentTask.cancel()`](https://developer.apple.com/documentation/swift/unsafecurrenttask/cancel())
389+
_does_ cancel the test or test case associated with the current task.
390+
391+
## Acknowledgments
392+
393+
Thanks team!
394+
395+
Thanks Arthur! That's right, dinosaurs _do_ say "roar!"
396+
397+
And thanks to [@allevato](https://github.com/allevato) for nerd-sniping me into
398+
writing this proposal.

0 commit comments

Comments
 (0)