Skip to content

fix(adhoc): support Map-typed columns in ad-hoc filters#1793

Open
adamyeats wants to merge 5 commits intomainfrom
fix/adhoc-map-type-support
Open

fix(adhoc): support Map-typed columns in ad-hoc filters#1793
adamyeats wants to merge 5 commits intomainfrom
fix/adhoc-map-type-support

Conversation

@adamyeats
Copy link
Copy Markdown
Contributor

Summary

Fixes #1434 — ad-hoc filters on Map(...) columns previously failed in two ways, both entirely in the plugin (despite the issue's waiting-on-clickhouse label, no upstream change is required):

  1. Filter values dropdown showed [object Object]. fetchTagValuesFromSchema() issued SELECT DISTINCT <col> against a Map column, returning whole Map values that the frontend stringified via String(value).
  2. Filters silently produced invalid SQL for non-OTel Map columns. escapeKey() used a hard-coded allowlist of three OTel names (ResourceAttributes, ScopeAttributes, LogAttributes) and treated every other dotted key as a table prefix, so user-defined Map columns and OTel variants like SpanAttributes didn't work.

What this changes

1. Map-aware tag-keys expansion (src/data/CHDatasource.ts)

getTagKeys() now inspects the type column from system.columns, collects Map-typed columns per-table, and — when the adhoc context points at a specific db.table — fans out parallel fetchUniqueMapKeys() probes and expands each Map column into one tag key per discovered map key. Falls back to the flat entry when no table context is available (avoids N-table fan-out on database-wide adhoc variables).

Detection handles Map(...), Nullable(Map(...)), and LowCardinality(Map(...)).

2. Map-aware tag-values fetch (src/data/CHDatasource.ts)

fetchTagValuesFromSchema() recognises dotted keys whose head refers to a Map-typed column (from the cache populated in step 1) and rewrites the SELECT to SELECT DISTINCT col['key'] FROM … instead of SELECT DISTINCT col FROM …. Fixes the [object Object] dropdown.

3. Schema-driven escapeKey (src/data/adHocFilter.ts)

Replaces the three-name allowlist with a schema-driven set published by the datasource via AdHocFilter.setMapColumns(). The OTel names remain as fallback defaults so behavior does not regress for users who hand-construct AdHocFilter or whose tables use the canonical names. Now correctly handles:

  • table.MapCol.key1.key2MapCol['key1.key2'] (table-prefixed)
  • MapCol.keyMapCol['key'] (no prefix, e.g. hideTableNameInAdhocFilters=true)
  • table.plain_colplain_col (unchanged, non-Map dotted path)

Tests

Scope notes

  • Probes are gated to specific db.table adhoc contexts. Database-wide or schema-wide adhoc variables still show the flat Map-column entry to avoid a per-table fan-out on every dashboard render. A follow-up can relax this if needed.
  • OTel fallback preserved. setMapColumns() is additive — DEFAULT_MAP_COLUMNS (ResourceAttributes, ScopeAttributes, LogAttributes) is always merged in. Tests that construct AdHocFilter directly without a datasource still pass.
  • Aligned with #1461 — both this and the JSON-path discovery bug share a "sample map-like keys from a large table" pattern. This PR reuses the existing fetchUniqueMapKeys() implementation; if #1461's sampling strategy changes, this will inherit it for free.

Related

  • Touches the same escapeKey() / dotted-column-path area as #798 (dots in column names). Not fixed here but adjacent.
  • #1063 (adhoc filter no-ops in Logs view) may also be resolved by the Bug 2 fix if the underlying cause is SQL rejection — worth a repro after this lands.

Closes #1434

Ad-hoc filters on Map columns previously failed in two ways:

1. Filter values dropdown showed `[object Object]`. Root cause:
   fetchTagValuesFromSchema issued `SELECT DISTINCT <col>` against a Map
   column, returning whole Map values that the frontend stringified via
   `String(value)`.

2. Filters silently produced invalid SQL for non-OTel Map columns. Root
   cause: escapeKey used a hard-coded allowlist of three OTel names
   (ResourceAttributes, ScopeAttributes, LogAttributes) and treated every
   other dotted key as a table prefix.

This change:

- getTagKeys() now inspects the `type` column from system.columns,
  collects Map-typed columns per-table, and when the adhoc context points
  at a specific `db.table` it fans out parallel fetchUniqueMapKeys()
  probes and expands each Map column into one tag key per discovered
  map key. Falls back to the flat entry when no table context is
  available (avoids N-table fan-out).

- fetchTagValuesFromSchema() recognises dotted keys whose head refers to
  a Map-typed column (from the new cache populated in step 1) and
  rewrites the SELECT to `SELECT DISTINCT col['key'] FROM …` instead of
  `SELECT DISTINCT col FROM …`.

- AdHocFilter.escapeKey() replaces the three-name allowlist with a
  schema-driven set published by the datasource via setMapColumns().
  The OTel names remain as fallback defaults so behavior does not
  regress for users who hand-construct AdHocFilter or whose tables use
  those canonical names.

Adds unit tests for all three code paths plus an e2e regression guard
that runs the generated SQL shapes (`mapKeys` discovery, bracket-access
values, full `additional_table_filters` apply) against a real ClickHouse
fixture (`e2e_test.map_events`).
Used in an adHocFilter.ts doc comment; CI spellcheck was failing on it.
…nflicts

Multiple in-flight PRs append their own fixtures to seed.sql, producing
mechanical merge conflicts even when the appended blocks are semantically
independent.

Change the e2e-data-loader entrypoint to iterate over /data/*.sql in
lexicographic order so each feature can drop a self-contained fixture
alongside seed.sql without touching it. Move the Map-column fixture
introduced for #1434 into tests/e2e/fixtures/map_events.sql; the file
re-declares the database so it does not depend on seed.sql running
first.
@adamyeats adamyeats marked this pull request as ready for review April 19, 2026 06:35
@adamyeats adamyeats requested a review from a team as a code owner April 19, 2026 06:35
…low it

Docker compose was interpolating the shell loop variable as a compose
variable ("The \"f\" variable is not set. Defaulting to a blank string."),
making the loader run `clickhouse-client ... < ""` and exit 1. Escape as
`$$f` so compose passes a literal `$` through to the shell.
The previous SETTINGS additional_table_filters={...} SQL was mangled
by Monaco editor's brace-auto-close when typed via keyboard, so the
filter never reached ClickHouse and all 6 fixture rows came back.

Unit tests in src/data/adHocFilter.test.ts cover the exact SETTINGS
shape AdHocFilter.apply() produces; this E2E test now verifies what
unit tests cannot — that ClickHouse actually executes the
`labels['key'] = 'value'` bracket-access predicate end-to-end.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Map type broken for Ad-Hoc filters

1 participant