CONFORMITY_TESTING.md

[nlothian]

No description provided.

[kilo-code-bot]

✅ No Issues Found

3 files reviewed | Confidence: 95% | Recommendation: Merge (already merged)

Files:

• docs/CONFORMITY_TESTING.md (documentation update)
• tools/conformity_test.py (framework enhancement)
• tools/scrape_conformity_expected.py (new utility script)

Checked: Security, bugs, performance, error handling, code clarity

Summary: This PR significantly improves the ISO Prolog conformity testing framework by expanding from 10 to 355 tests, adding expected result validation, and including reference implementation comparisons. The changes enhance test accuracy and reporting quality without introducing security risks or bugs.

[gemini-code-assist]

Code Review

This pull request significantly expands the ISO Prolog conformity test suite from 10 to 355 tests, introducing a more robust framework for evaluating compliance. The changes correctly distinguish between failing tests and tests that are expected to produce errors, using the term 'conforming' for more accurate reporting. The inclusion of reference results from SWI-Prolog and Scryer-Prolog is also a valuable addition for comparison. The overall changes are excellent and greatly improve the project's testing infrastructure. I've identified a couple of minor areas for improvement: some comments in the test expectations are mismatched with the test cases, which should be clarified to align with documentation principles for non-standard features, and a new utility script includes an unused import. These are minor points in an otherwise high-quality contribution.

[gemini-code-assist]

The comments for tests 21-26 in _EXPECTED_RESULTS do not seem to match the actual test queries, which could be confusing for future maintenance. For example, test 21 is writeq('\\e'). but the comment is # \\e in char_code. Please review and correct these comments for clarity. This aligns with the principle of explicitly documenting deviations from standard behavior, even if not directly related to predicates.

Suggested change

	21: TestResult.SYNTAX_ERROR, # \e in char_code
	22: TestResult.SYNTAX_ERROR, # \d in char_code
	23: TestResult.SYNTAX_ERROR, # \u1 incomplete
	24: TestResult.SYNTAX_ERROR, # \u0021 in atom
	25: TestResult.SYNTAX_ERROR, # \u0021 in char code
	26: TestResult.SYNTAX_ERROR, # \u0021 in string
	21: TestResult.SYNTAX_ERROR, # `\\e` is not a standard escape sequence
	22: TestResult.SYNTAX_ERROR, # Incomplete octal escape `\\033\\`
	23: TestResult.SYNTAX_ERROR, # Incomplete octal escape `\\0\\`
	24: TestResult.SYNTAX_ERROR, # `\\e` is not a standard character for char_code/2
	25: TestResult.SYNTAX_ERROR, # `\\d` is not a standard character for char_code/2
	26: TestResult.SYNTAX_ERROR, # Incomplete unicode escape `\\u1`

References

1. Non-standard extensions to ISO-compliant predicates are permissible, but they must be explicitly documented as extensions. This principle extends to documenting non-standard escape sequences/characters for clarity.

[gemini-code-assist]

The sys module is imported but never used in this script. It's good practice to remove unused imports to keep the code clean and avoid potential confusion.