Split test_psa_compliance.py and update compliance version to 1.2 #453

gilles-peskine-arm · 2025-09-11T20:27:23Z

Split the PSA API compliance script test_psa_compliance.py into an engine plus a pre-branch runner, like with many similar scripts. Now each branch has the knowledge of which version of psa-arch-tests it runs against and how to adjust the results if needed.

Then, in TF-PSA-Crypto, run against v25.02_API1.8_CRYPTO_1.2.1, which allows us to claim compliance with version 1.2 of the PSA Crypto API specification. This requires a bit of preparation:

Patch a bug in the psa-arch-tests (already fixed upstream) which prevents it from building.
Provide stubs of a few more functions and macros, because psa-arch-tests requires them even for algorithms that the implementation does not provide.

Resolves #418.

Needs preceding PR: Mbed-TLS/mbedtls-framework#209.

PR checklist

Please remove the segment/s on either side of the | symbol as appropriate, and add any relevant link/s to the end of the line.
If the provided content is part of the present PR remove the # symbol.

changelog not required because: test only
development PR not required because: crypto only
TF-PSA-Crypto PR here
framework PR provided Split test_psa_compliance.py mbedtls-framework#209
3.6 PR provided Backport 3.6: Split test_psa_compliance.py mbedtls#10400
tests provided

gilles-peskine-arm · 2025-09-12T17:04:35Z

tests/scripts/test_psa_compliance.py

+import scripts_path # pylint: disable=unused-import
+from mbedtls_framework import psa_compliance
+
+PSA_ARCH_TESTS_REF = 'v25.02_API1.8_CRYPTO_1.2.1'


There's a v25.08_API1.9_ADAC_1.0.2, but with that version, test_psa_compliance.py fails at the CMake step. I'm not going to investigate this, moving to 1.9 doesn't make much of a difference for us anyway since it doesn't advance the version of the crypto API.

gilles-peskine-arm · 2025-09-26T09:59:17Z

scripts/data_files/psa-arch-tests/pal_crypto_config.patch

+diff --git a/api-tests/platform/targets/tgt_dev_apis_stdc/nspe/pal_crypto_config.h b/api-tests/platform/targets/tgt_dev_apis_stdc/nspe/pal_crypto_config.h
+index dad40ec..8d19699 100644
+--- a/api-tests/platform/targets/tgt_dev_apis_stdc/nspe/pal_crypto_config.h
+++ b/api-tests/platform/targets/tgt_dev_apis_stdc/nspe/pal_crypto_config.h


DES is disabled by default in v25.02, so no need to adjust the patch once we remove DES.

Once #475 is merged, this PR can remove the temporary definition of PSA_KEY_TYPE_DES in crypto_compat.h.

Once #475 is merged, this PR can remove the temporary definition of PSA_KEY_TYPE_DES in crypto_compat.h.

It's merged now :)

Unfortunately, no, there's some test data in psa-arch-tests where PSA_KEY_TYPE_DES isn't guarded.

…owledge Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

You should be able to run `tests/scripts/test_psa_compliance.py` without having to consult some other scripts to know what options to pass. Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

….2.1 Provide stubs that just return `PSA_ERROR_NOT_SUPPORTED` for missing functions that are necessary to build psa-arch-tests at `v25.02_API1.8_CRYPTO_1.2.1`: * `psa_key_derivation_verify_bytes()` * `psa_key_derivation_verify_key()` * `psa_pake_set_context()` Also provide a macro for the same reason: * `PSA_PAKE_STEP_CONFIRM` Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

Some of those macros are necessary to build psa-arch-tests at `v25.02_API1.8_CRYPTO_1.2.1`. Define other tightly related macros for completeness. Signed-off-by: Gilles Peskine <[email protected]>

In `generate_psa_constants.py`, we use it for anything that `macro_collector.py` analyzes as a key type constructed from a curve. This now includes `PSA_KEY_TYPE_SPAKE2P_PUBLIC_KEY(curve)` and `PSA_KEY_TYPE_SPAKE2P_KEY_PAIR(curve)`. Signed-off-by: Gilles Peskine <[email protected]>

We can now test compliance with version 1.2 of the PSA Crypto API. Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

mpg

LGTM, thanks!

mpg · 2025-09-30T09:33:41Z

tests/scripts/test_psa_compliance.py

+#!/usr/bin/env python3
+"""Run the PSA Crypto API compliance test suite.
+
+Clone the repo and check out the commit specified by PSA_ARCH_TEST_REPO and PSA_ARCH_TEST_REF,


Nit: PSA_ARCH_TEST_REPO is no longer defined here so it's a bit weird to reference it in this script's docstring.

mpg · 2025-09-30T09:33:54Z

tests/scripts/test_psa_compliance.py

+
+Clone the repo and check out the commit specified by PSA_ARCH_TEST_REPO and PSA_ARCH_TEST_REF,
+then compile and run the test suite. The clone is stored at <repository root>/psa-arch-tests.
+Known defects in either the test suite or Mbed TLS - identified by their test


Nit: TF-PSA-Crypto

bensze01

LGTM, thanks!

bensze01 · 2025-09-30T13:07:38Z

The ABI check failure is unrelated, caused by me missing that the mbedtls commit won't be a merge commit in TF-PSA-Crypto Pull requests...

Making the ABI-check work in TF-PSA-Crypto PR-jobs is non-goal of Mbed-TLS/mbedtls-test#209, so I'll open a new PR with the fix.

Since Mbed-TLS/TF-PSA-Crypto#453, psa-arch-tests crash on our CI and in similar environments because glibc detects a stack buffer overflow. Temporarily ignore the failure until we patch TF-PSA-Crypto. Revert this commit once this is done. Signed-off-by: Gilles Peskine <[email protected]>

gilles-peskine-arm added this to Roadmap pull requests (new board) Sep 11, 2025

gilles-peskine-arm added size-s Estimated task size: small (~2d) priority-high High priority - will be reviewed soon needs-preceding-pr Requires another PR to be merged first needs-ci Needs to pass CI tests labels Sep 11, 2025

github-project-automation bot moved this to In Development in Roadmap pull requests (new board) Sep 11, 2025

This was referenced Sep 11, 2025

Split test_psa_compliance.py Mbed-TLS/mbedtls-framework#209

Merged

Implement SPAKE2+ public key import and export #448

Draft

gilles-peskine-arm force-pushed the compliance-split-crypto branch 4 times, most recently from 587f31a to 0a4ed4c Compare September 12, 2025 16:39

gilles-peskine-arm commented Sep 12, 2025

View reviewed changes

gilles-peskine-arm force-pushed the compliance-split-crypto branch from 0a4ed4c to 006adb9 Compare September 12, 2025 20:08

gilles-peskine-arm added needs-review Every commit must be reviewed by at least two team members needs-reviewer This PR needs someone to pick it up for review and removed needs-ci Needs to pass CI tests labels Sep 13, 2025

gilles-peskine-arm requested a review from bensze01 September 13, 2025 15:23

gilles-peskine-arm mentioned this pull request Sep 13, 2025

Backport 3.6: Split test_psa_compliance.py Mbed-TLS/mbedtls#10400

Open

5 tasks

gilles-peskine-arm commented Sep 26, 2025

View reviewed changes

gilles-peskine-arm added 10 commits September 30, 2025 11:05

Update framework with psa_compliance.py module and minimal SPAKE2+ kn…

c8935a4

…owledge Signed-off-by: Gilles Peskine <[email protected]>

Import PSA compliance test script entry point from the framework

067df1a

Signed-off-by: Gilles Peskine <[email protected]>

Store the list of expected failures in the compliance script

ea9ae21

You should be able to run `tests/scripts/test_psa_compliance.py` without having to consult some other scripts to know what options to pass. Signed-off-by: Gilles Peskine <[email protected]>

Disable unsupported mechanisms instead of skipping tests

28d1ecf

Signed-off-by: Gilles Peskine <[email protected]>

Fix JPAKE Doxygen documentation

9195d29

Signed-off-by: Gilles Peskine <[email protected]>

Declare SPAKE2+ key types and algorithms

2925625

Some of those macros are necessary to build psa-arch-tests at `v25.02_API1.8_CRYPTO_1.2.1`. Define other tightly related macros for completeness. Signed-off-by: Gilles Peskine <[email protected]>

Update psa-arch-tests to v25.02_API1.8_CRYPTO_1.2.1

e9cde2c

We can now test compliance with version 1.2 of the PSA Crypto API. Signed-off-by: Gilles Peskine <[email protected]>

Update the claimed PSA Crypto API version from 1.0 to 1.2

daa17da

Signed-off-by: Gilles Peskine <[email protected]>

gilles-peskine-arm force-pushed the compliance-split-crypto branch from 006adb9 to daa17da Compare September 30, 2025 09:22

gilles-peskine-arm removed the needs-preceding-pr Requires another PR to be merged first label Sep 30, 2025

mpg approved these changes Sep 30, 2025

View reviewed changes

mpg removed the needs-reviewer This PR needs someone to pick it up for review label Sep 30, 2025

bensze01 approved these changes Sep 30, 2025

View reviewed changes

github-project-automation bot moved this from In Development to Has Approval in Roadmap pull requests (new board) Sep 30, 2025

mpg added approved Design and code approved - may be waiting for CI or backports and removed needs-review Every commit must be reviewed by at least two team members labels Sep 30, 2025

mpg added this pull request to the merge queue Sep 30, 2025

Merged via the queue into Mbed-TLS:development with commit cf4c26d Sep 30, 2025
3 of 7 checks passed

github-project-automation bot moved this from Has Approval to Done in Roadmap pull requests (new board) Sep 30, 2025

This was referenced Oct 3, 2025

test_psa_compliance.py doesn't detect if psa-arch-tests crashes Mbed-TLS/mbedtls-framework#220

Open

Cleanups in psa_compliance.py Mbed-TLS/mbedtls-framework#221

Open

Fix crash in compliance tests #495

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Split test_psa_compliance.py and update compliance version to 1.2 #453

Split test_psa_compliance.py and update compliance version to 1.2 #453

Uh oh!

gilles-peskine-arm commented Sep 11, 2025

Uh oh!

gilles-peskine-arm Sep 12, 2025

Uh oh!

gilles-peskine-arm Sep 26, 2025

Uh oh!

mpg Sep 30, 2025

Uh oh!

gilles-peskine-arm Sep 30, 2025

Uh oh!

mpg left a comment

Uh oh!

mpg Sep 30, 2025

Uh oh!

mpg Sep 30, 2025

Uh oh!

bensze01 left a comment

Uh oh!

bensze01 commented Sep 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Split test_psa_compliance.py and update compliance version to 1.2 #453

Split test_psa_compliance.py and update compliance version to 1.2 #453

Uh oh!

Conversation

gilles-peskine-arm commented Sep 11, 2025

PR checklist

Uh oh!

gilles-peskine-arm Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

gilles-peskine-arm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

mpg Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

gilles-peskine-arm Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

mpg left a comment

Choose a reason for hiding this comment

Uh oh!

mpg Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

mpg Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

bensze01 left a comment

Choose a reason for hiding this comment

Uh oh!

bensze01 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bensze01 commented Sep 30, 2025 •

edited

Loading