Skip to content

Make ExpectColumnValuesToMatchStrftimeFormat a fully-supported, Gallery-published Expectation #12004

Description

@joshua-stauffer

Use case

I want to validate that a text column's values match a given strftime date/time format (e.g. "%Y-%m-%d" or "%Y-%m-%dT%H:%M:%S%z") — a common check on date columns that arrive as strings before parsing. Users have asked for exactly this and reported that it seemed to disappear from the docs/Gallery: see Discourse: "Does GX support date format check?" ("Can we have it added in the newer versions?") and issue #11603.

ExpectColumnValuesToMatchStrftimeFormat already exists and is exported from great_expectations.expectations, but it is not in the Expectation Gallery, has no generated schema, carries none of the support metadata, and its backend coverage is incomplete — so it's effectively invisible and its supported surface is undefined.

Proposed capability

Bring ExpectColumnValuesToMatchStrftimeFormat up to the supported-core bar: Gallery-published (schema + metadata), a correct docstring, and a defined, tested backend matrix.

import great_expectations.expectations as gxe

gxe.ExpectColumnValuesToMatchStrftimeFormat(column="event_date", strftime_format="%Y-%m-%d")

Current backend state: the backing metric implements Pandas and Spark but has no SQL (SqlAlchemy) implementation. So the target matrix requires a decision (see Additional context): either add a SQL implementation, or ship Pandas + Spark as the supported set with SQL explicitly declared out of scope, documented as a dialect limitation.

Implementation Guide

Source class: great_expectations/expectations/core/expect_column_values_to_match_strftime_format.py. Backing metric: great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py (pandas + Spark; no SqlAlchemy branch). Reference template for the supported bar: great_expectations/expectations/core/expect_column_values_to_match_regex.py + its metric + its schema great_expectations/expectations/core/schemas/ExpectColumnValuesToMatchRegex.json.

Decision to make first: SQL support

Generic strftime-format validation in SQL isn't straightforward — strftime format tokens do not map cleanly onto the format models of Snowflake, PostgreSQL, BigQuery, etc., and each dialect's date-parse/TRY_CAST behavior differs. There is precedent for shipping supported-but-not-on-every-dialect: ExpectColumnValuesToMatchRegex raises NotImplementedError on dialects without regex support and omits SQL Server + Snowflake from its SUPPORTED_DATA_SOURCES.

Two acceptable outcomes — pick one and make it the definition of done:

  • (A) Ship Pandas + Spark supported, SQL declared out. Lowest risk. SUPPORTED_DATA_SOURCES = Pandas + Spark only; document the SQL limitation in the docstring. This unblocks the Gallery and the two working engines immediately.
  • (B) Add a SQL implementation. Implement a _sqlalchemy branch (per-dialect date-format validation via TRY_CAST/TO_DATE/TRY_TO_TIMESTAMP, raising NotImplementedError on dialects that can't express it), and add only the dialects you actually validate to SUPPORTED_DATA_SOURCES. This is the hardest single piece of work here — scope it deliberately; it's fine to land (A) first and pursue (B) as a follow-up.

Definition of done (both outcomes)

  1. Docstring fix (bug): the current docstring is a plain string containing the literal token {FAILURE_SEVERITY_DESCRIPTION}, which is never interpolated and renders raw. Convert to an __doc__ = f"""...""" f-string (like the regex exemplar) and bring it to the standard Gallery format (short description, Args, Supported Data Sources, Data Quality Issues, Example Data, passing + failing Code Examples).
  2. Support metadata: add module constants EXPECTATION_SHORT_DESCRIPTION, DATA_QUALITY_ISSUES ([DataQualityIssues.VALIDITY.value]), SUPPORTED_DATA_SOURCES (matching your chosen matrix); a library_metadata ClassVar ("maturity": "production", has_full_test_suite: True, manually_reviewed_code: True, with _library_metadata alias); and a Config.schema_extra that stamps them into the schema. Copy the exact shape from the regex exemplar.
  3. Schema: add the class to the supported_expectations list in tasks.py, run invoke schemas --sync, and commit great_expectations/expectations/core/schemas/ExpectColumnValuesToMatchStrftimeFormat.json. (CI's test_schemas_updated enforces it.)
  4. Renderers: at least a prescriptive renderer if missing.
  5. Integration tests at tests/integration/data_sources_and_expectations/expectations/test_expect_column_values_to_match_strftime_format.py covering the full target matrix via @parameterize_batch_for_data_sources. The current test file exists but is Spark-only — extend it to cover pandas (and SQL, if outcome B), with passing and failing cases, including a timezone-aware format (%z), which was a real regression area (issue Validating datetime strings with timezone produces an error #9203).

Testing note for community contributors. Pandas, sqlite, spark, postgres, and mysql run locally (some via Docker Compose); BigQuery, Snowflake, Databricks, and Redshift run only in maintainer CI with credentials. Write tests against the full target matrix, run what you can locally, and CI covers the cloud dialects. Run a single module locally by marker from the repo root, e.g. pytest tests/integration/data_sources_and_expectations/expectations/test_expect_column_values_to_match_strftime_format.py -m unit (pandas), or -m sqlite / -m spark (see AGENTS.md for marker usage).

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood issues for new contributors. Join #contributors-contributing in our Slack for help!help wantedIssues we'd love to see community contributions for. Join #contributors-contributing in our Slack!ready-for-workTriaged, accepted, and specified well enough to start today -- the claiming gate

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions