Skip to content

Add Unicode character support to parser for identifiers #295

Description

@nlothian

Problem

The parser fails to parse library/numerics/testutils.pl because it uses Unicode characters in predicate names:

  • δ_inverses_t/5

Error:

syntax_error(No terminal matches 'δ' in the current parser context)
,δ_inverses_t/5
 ^

Root Cause

The Lark grammar in vibeprolog/parser.py defines atoms/identifiers using ASCII character patterns only. Greek letters and other Unicode characters are not recognized as valid atom characters.

Solution

Extend the parser's atom pattern to include Unicode letter categories:

  1. Update the ATOM token pattern in the Lark grammar to accept Unicode letters
  2. Use Unicode character classes in the regex: [\p{L}] (letters) or similar
  3. Ensure the pattern still maintains valid Prolog identifier semantics
  4. Test against both ASCII and Unicode predicate names

Implementation Details

  • Review vibeprolog/parser.py for the ATOM token definition
  • Update the regex pattern to include Unicode letters
  • Ensure backward compatibility with ASCII-only identifiers
  • Handle Unicode in quoted and unquoted atoms consistently

Testing Requirements

  1. Create comprehensive test file tests/test_unicode_identifiers.py covering:

    • Greek letters in predicate names (α, β, γ, δ, etc.)
    • Other Unicode letters (é, ñ, ü, etc.)
    • Unicode in compound terms
    • Unicode in quoted atoms
    • Unicode in variable names (should still be uppercase or underscore)
    • Mixed ASCII and Unicode identifiers
    • Ensure proper distinction between atoms and variables
  2. Verify library/numerics/testutils.pl loads successfully

  3. Ensure no regressions in existing parser tests

  4. Test that semantics are unchanged (Unicode atoms still unify correctly, etc.)

Documentation

Update FEATURES.md to document:

  • Parser now supports Unicode letters in atom identifiers
  • Examples of valid Unicode atom names
  • Any restrictions on which Unicode characters are supported

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions