fix(adhoc): support Map-typed columns in ad-hoc filters#1793
Open
fix(adhoc): support Map-typed columns in ad-hoc filters#1793
Conversation
Ad-hoc filters on Map columns previously failed in two ways: 1. Filter values dropdown showed `[object Object]`. Root cause: fetchTagValuesFromSchema issued `SELECT DISTINCT <col>` against a Map column, returning whole Map values that the frontend stringified via `String(value)`. 2. Filters silently produced invalid SQL for non-OTel Map columns. Root cause: escapeKey used a hard-coded allowlist of three OTel names (ResourceAttributes, ScopeAttributes, LogAttributes) and treated every other dotted key as a table prefix. This change: - getTagKeys() now inspects the `type` column from system.columns, collects Map-typed columns per-table, and when the adhoc context points at a specific `db.table` it fans out parallel fetchUniqueMapKeys() probes and expands each Map column into one tag key per discovered map key. Falls back to the flat entry when no table context is available (avoids N-table fan-out). - fetchTagValuesFromSchema() recognises dotted keys whose head refers to a Map-typed column (from the new cache populated in step 1) and rewrites the SELECT to `SELECT DISTINCT col['key'] FROM …` instead of `SELECT DISTINCT col FROM …`. - AdHocFilter.escapeKey() replaces the three-name allowlist with a schema-driven set published by the datasource via setMapColumns(). The OTel names remain as fallback defaults so behavior does not regress for users who hand-construct AdHocFilter or whose tables use those canonical names. Adds unit tests for all three code paths plus an e2e regression guard that runs the generated SQL shapes (`mapKeys` discovery, bracket-access values, full `additional_table_filters` apply) against a real ClickHouse fixture (`e2e_test.map_events`).
Used in an adHocFilter.ts doc comment; CI spellcheck was failing on it.
…nflicts Multiple in-flight PRs append their own fixtures to seed.sql, producing mechanical merge conflicts even when the appended blocks are semantically independent. Change the e2e-data-loader entrypoint to iterate over /data/*.sql in lexicographic order so each feature can drop a self-contained fixture alongside seed.sql without touching it. Move the Map-column fixture introduced for #1434 into tests/e2e/fixtures/map_events.sql; the file re-declares the database so it does not depend on seed.sql running first.
…low it
Docker compose was interpolating the shell loop variable as a compose
variable ("The \"f\" variable is not set. Defaulting to a blank string."),
making the loader run `clickhouse-client ... < ""` and exit 1. Escape as
`$$f` so compose passes a literal `$` through to the shell.
The previous SETTINGS additional_table_filters={...} SQL was mangled
by Monaco editor's brace-auto-close when typed via keyboard, so the
filter never reached ClickHouse and all 6 fixture rows came back.
Unit tests in src/data/adHocFilter.test.ts cover the exact SETTINGS
shape AdHocFilter.apply() produces; this E2E test now verifies what
unit tests cannot — that ClickHouse actually executes the
`labels['key'] = 'value'` bracket-access predicate end-to-end.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1434 — ad-hoc filters on
Map(...)columns previously failed in two ways, both entirely in the plugin (despite the issue'swaiting-on-clickhouselabel, no upstream change is required):[object Object].fetchTagValuesFromSchema()issuedSELECT DISTINCT <col>against a Map column, returning whole Map values that the frontend stringified viaString(value).escapeKey()used a hard-coded allowlist of three OTel names (ResourceAttributes,ScopeAttributes,LogAttributes) and treated every other dotted key as a table prefix, so user-defined Map columns and OTel variants likeSpanAttributesdidn't work.What this changes
1. Map-aware tag-keys expansion (src/data/CHDatasource.ts)
getTagKeys()now inspects thetypecolumn fromsystem.columns, collects Map-typed columns per-table, and — when the adhoc context points at a specificdb.table— fans out parallelfetchUniqueMapKeys()probes and expands each Map column into one tag key per discovered map key. Falls back to the flat entry when no table context is available (avoids N-table fan-out on database-wide adhoc variables).Detection handles
Map(...),Nullable(Map(...)), andLowCardinality(Map(...)).2. Map-aware tag-values fetch (src/data/CHDatasource.ts)
fetchTagValuesFromSchema()recognises dotted keys whose head refers to a Map-typed column (from the cache populated in step 1) and rewrites the SELECT toSELECT DISTINCT col['key'] FROM …instead ofSELECT DISTINCT col FROM …. Fixes the[object Object]dropdown.3. Schema-driven
escapeKey(src/data/adHocFilter.ts)Replaces the three-name allowlist with a schema-driven set published by the datasource via
AdHocFilter.setMapColumns(). The OTel names remain as fallback defaults so behavior does not regress for users who hand-constructAdHocFilteror whose tables use the canonical names. Now correctly handles:table.MapCol.key1.key2→MapCol['key1.key2'](table-prefixed)MapCol.key→MapCol['key'](no prefix, e.g. hideTableNameInAdhocFilters=true)table.plain_col→plain_col(unchanged, non-Map dotted path)Tests
additional_table_filtersapply) against a real ClickHouse fixture added to tests/e2e/fixtures/seed.sql (e2e_test.map_eventswith aMap(String, String)column).Scope notes
db.tableadhoc contexts. Database-wide or schema-wide adhoc variables still show the flat Map-column entry to avoid a per-table fan-out on every dashboard render. A follow-up can relax this if needed.setMapColumns()is additive —DEFAULT_MAP_COLUMNS(ResourceAttributes,ScopeAttributes,LogAttributes) is always merged in. Tests that constructAdHocFilterdirectly without a datasource still pass.fetchUniqueMapKeys()implementation; if #1461's sampling strategy changes, this will inherit it for free.Related
escapeKey()/ dotted-column-path area as #798 (dots in column names). Not fixed here but adjacent.Closes #1434