Databricks sql warehouses #18273
Databricks sql warehouses #18273
Conversation
- Added create SQL warehouse action - Added list SQL warehouses action - Added get SQL warehouse action - Added edit SQL warehouse action - Added delete SQL warehouse action - Added start/stop SQL warehouse actions - Added get/set SQL warehouse config actions - Added get/set SQL warehouse permissions actions Implements 11 SQL Warehouses endpoints as discussed in the enhancement issue.
Summary This PR adds support for Databricks Vector Search Index APIs by implementing 9 new actions. Changes - Added create index action - Added get index action - Added list indexes action - Added delete index action - Added query index action - Added sync index action - Added scan index action - Added upsert index data action - Added delete index data action Resolves PipedreamHQ#18126
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
WalkthroughAdds Databricks SQL Warehouse management and Vector Search support: new app methods and dynamic prop for vector indexes, many new actions for warehouse lifecycle, config and permissions, plus comprehensive vector-index actions (create/get/list/delete/query/sync/scan/upsert/delete-data). Also several action version bumps and package version update. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor U as User
participant Action as Databricks Action
participant App as Databricks App
participant API as Databricks REST API
U->>Action: Provide inputs (indexName / warehouseId / payload)
Action->>Action: Parse & validate inputs
Action->>App: Call app method (e.g., createVectorSearchIndex / editSQLWarehouse / queryVectorSearchIndex)
App->>API: HTTP request to Databricks endpoint
API-->>App: Response
App-->>Action: Response
Action-->>U: Return result + $.export("$summary", ...)
sequenceDiagram
autonumber
actor U as User
participant VAction as Vector Index Action
participant App as Databricks App
participant API as Databricks REST API
U->>VAction: Provide indexName + rows/query/params
VAction->>VAction: parse JSON / validate PKs / build payload
VAction->>App: upsert/query/scan/sync methods
App->>API: POST/GET /vector-search/indexes/{indexName}/...
API-->>App: Response
App-->>VAction: Response
VAction-->>U: Return response + $summary
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Assessment against linked issues
Out-of-scope changes
Possibly related PRs
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you so much for submitting this! We've added it to our backlog to review, and our team has been notified. |
|
Thanks for submitting this PR! When we review PRs, we follow the Pipedream component guidelines. If you're not familiar, here's a quick checklist:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
components/databricks/actions/list-runs/list-runs.mjs (2)
53-53: Bug: array compared to number in loop condition.
allRuns < this.maxResultsalways evaluates unexpectedly; should beallRuns.length.- } while (total === params.limit && allRuns < this.maxResults); + } while (total === params.limit && allRuns.length < this.maxResults);
1-66: Fix array-to-number comparison in loop condition ChangeallRuns < this.maxResultstoallRuns.length < this.maxResultsin components/databricks/actions/list-runs/list-runs.mjs around line 53.
🧹 Nitpick comments (32)
components/databricks/actions/run-job-now/run-job-now.mjs (1)
49-57: Guard against mutually exclusive params (jarParams vs notebookParams).The API rejects requests specifying both. Add a quick check to fail fast with a clear error.
async run({ $ }) { - const response = await this.databricks.runJobNow({ + const hasJar = Array.isArray(this.jarParams) && this.jarParams.length > 0; + const hasNotebook = this.notebookParams && Object.keys(this.notebookParams).length > 0; + if (hasJar && hasNotebook) { + throw new Error("jarParams and notebookParams are mutually exclusive. Provide only one."); + } + + const response = await this.databricks.runJobNow({ data: { job_id: this.jobId, jar_params: this.jarParams, notebook_params: this.notebookParams, python_params: this.pythonParams, spark_submit_params: this.sparkSubmitParams, idempotency_token: this.idempotencyToken, }, $, });components/databricks/actions/list-runs/list-runs.mjs (2)
33-38: Respect maxResults during pagination to avoid over-fetching.Adjust
limitto the remaining budget and stop early when the cap is reached.async run({ $ }) { - const params = { + const params = { job_id: this.jobId, active_only: this.activeOnly, - limit: 100, + limit: 100, offset: 0, }; + const max = this.maxResults ?? 100;
42-47: Cap each page by remaining results.Prevents extra API calls and trimming later.
- do { - const { runs } = await this.databricks.listRuns({ + do { + // Stop if we've already reached the cap + if (allRuns.length >= max) break; + // Limit this page to the remaining budget + params.limit = Math.min(params.limit, Math.max(1, max - allRuns.length)); + const { runs } = await this.databricks.listRuns({ params, $, });components/databricks/package.json (1)
3-3: Version bump — LGTM. Please ensure release notes.Given the sizable feature set, add/update CHANGELOG and fill in the PR description before merge.
components/databricks/common/constants.mjs (1)
1-11: Prevent accidental mutation of shared constants.Freeze the array to avoid runtime edits.
-export const CLUSTER_SIZES = [ +export const CLUSTER_SIZES = Object.freeze([ "2X-Small", "X-Small", "Small", "Medium", "Large", "X-Large", "2X-Large", "3X-Large", "4X-Large", -]; +]);components/databricks/common/utils.mjs (3)
16-24: Limit recursion to plain objects to avoid mangling Dates/Maps/Sets/etc.Current
typeof obj === "object"will traverse non-POJOs and strip their contents. Guard for plain objects.Apply:
- if (typeof obj === "object") { + if (typeof obj === "object" && Object.getPrototypeOf(obj) === Object.prototype) { return Object.fromEntries(Object.entries(obj).map(([ key, value, ]) => [ key, parseObject(value), ])); }
1-27: Protect against cyclic structures to prevent infinite recursion.Add a
WeakSetto track seen objects.One way:
-const parseObject = (obj) => { +const parseObject = (obj, _seen = new WeakSet()) => { @@ - if (Array.isArray(obj)) { - return obj.map((item) => parseObject(item)); + if (Array.isArray(obj)) { + return obj.map((item) => parseObject(item, _seen)); } @@ - if (typeof obj === "object" && Object.getPrototypeOf(obj) === Object.prototype) { + if (typeof obj === "object" && Object.getPrototypeOf(obj) === Object.prototype) { + if (_seen.has(obj)) return obj; + _seen.add(obj); return Object.fromEntries(Object.entries(obj).map(([ key, value, - ]) => [ - key, - parseObject(value), + ]) => [ + key, + parseObject(value, _seen), ])); }
29-31: Export named helper for ergonomics and tree-shaking.Apply:
-export default { - parseObject, -}; +export { parseObject }; +export default { parseObject };components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
16-17: Prefer nullish coalescing for count and keep concise pluralization.Apply:
- const count = response?.vector_indexes?.length || 0; - $.export("$summary", `Found ${count} vector search index${count === 1 ? "" : "es"}`); + const count = response?.vector_indexes?.length ?? 0; + $.export("$summary", `Found ${count} vector search index${count === 1 ? "" : "es"}`);components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
6-6: Clarify that deletion is irreversible in the description.Apply:
- description: "Deletes a vector search index in Databricks. [See the documentation](https://docs.databricks.com/api/workspace/vectorsearchindexes/deleteindex)", + description: "Irreversibly deletes a vector search index in Databricks. [See the documentation](https://docs.databricks.com/api/workspace/vectorsearchindexes/deleteindex)",components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs (1)
19-27: Optionally support waiting until the warehouse is ready.Starting is async; add a
waitForReadyflag to poll status viagetSQLWarehouseuntil RUNNING (with timeout).Apply:
props: { databricks, warehouseId: { description: "The ID of the SQL Warehouse to start", propDefinition: [ databricks, "warehouseId", ], }, + waitForReady: { + type: "boolean", + label: "Wait Until Ready", + description: "Polls the warehouse state until it's running (or times out).", + optional: true, + default: false, + }, + pollIntervalMs: { + type: "integer", + label: "Poll Interval (ms)", + optional: true, + default: 5000, + }, + timeoutMs: { + type: "integer", + label: "Timeout (ms)", + optional: true, + default: 300000, + }, }, async run({ $ }) { const response = await this.databricks.startSQLWarehouse({ warehouseId: this.warehouseId, $, }); $.export("$summary", `Successfully started SQL Warehouse with ID ${this.warehouseId}`); - return response; + if (!this.waitForReady) return response; + + const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + const start = Date.now(); + while (Date.now() - start < this.timeoutMs) { + const { state } = await this.databricks.getSQLWarehouse({ warehouseId: this.warehouseId, $ }); + if (state === "RUNNING") return { ...response, state }; + await sleep(this.pollIntervalMs); + } + throw new Error(`Timed out waiting for SQL Warehouse ${this.warehouseId} to be RUNNING`); },components/databricks/actions/delete-sql-warehouse/delete-sql-warehouse.mjs (2)
9-18: Add an optional “idempotent delete” switchAllow users to treat 404 as success to make deletes safe to re-run.
Apply this diff to introduce an
ignoreMissingprop:props: { databricks, warehouseId: { description: "The ID of the SQL Warehouse to delete", propDefinition: [ databricks, "warehouseId", ], }, + ignoreMissing: { + type: "boolean", + label: "Ignore Missing", + description: "Treat 404 (not found) as success.", + optional: true, + }, },
19-29: Make deletion idempotent and improve error messagingCatch 404s when
ignoreMissingis enabled; keep other errors bubbling.Apply this diff:
- async run({ $ }) { - await this.databricks.deleteSQLWarehouse({ - warehouseId: this.warehouseId, - $, - }); - - $.export("$summary", `Successfully deleted SQL Warehouse with ID ${this.warehouseId}`); - return { - success: true, - }; - }, + async run({ $ }) { + try { + await this.databricks.deleteSQLWarehouse({ + warehouseId: this.warehouseId, + $, + }); + $.export("$summary", `Successfully deleted SQL Warehouse with ID ${this.warehouseId}`); + return { success: true }; + } catch (err) { + const status = err?.response?.status ?? err?.status; + if (this.ignoreMissing && status === 404) { + $.export("$summary", `Warehouse ${this.warehouseId} not found. Treated as success (ignored).`); + return { success: true, ignored: true }; + } + throw err; + } + },components/databricks/actions/set-sql-warehouse-config/set-sql-warehouse-config.mjs (1)
131-152: Validate configuration pair items before sendingGuard against malformed entries to fail fast with actionable errors.
Apply this minimal validation:
- const configParam = utils.parseObject(this.configParam); + const configParam = utils.parseObject(this.configParam); if (Array.isArray(configParam) && configParam.length) { + configParam.forEach((p, i) => { + if (!p || typeof p !== "object" || typeof p.key !== "string") { + throw new ConfigurationError(`configParam[${i}] must be an object like { "key": string, "value": any }`); + } + }); payload.config_param = { configuration_pairs: configParam, }; } - const globalParam = utils.parseObject(this.globalParam); + const globalParam = utils.parseObject(this.globalParam); if (Array.isArray(globalParam) && globalParam.length) { + globalParam.forEach((p, i) => { + if (!p || typeof p !== "object" || typeof p.key !== "string") { + throw new ConfigurationError(`globalParam[${i}] must be an object like { "key": string, "value": any }`); + } + }); payload.global_param = { configuration_pairs: globalParam, }; } - const sqlConfigurationParameters = utils.parseObject(this.sqlConfigurationParameters); + const sqlConfigurationParameters = utils.parseObject(this.sqlConfigurationParameters); if (Array.isArray(sqlConfigurationParameters) && sqlConfigurationParameters.length) { + sqlConfigurationParameters.forEach((p, i) => { + if (!p || typeof p !== "object" || typeof p.key !== "string") { + throw new ConfigurationError(`sqlConfigurationParameters[${i}] must be an object like { "key": string, "value": any }`); + } + }); payload.sql_configuration_parameters = { configuration_pairs: sqlConfigurationParameters, }; } - const dataAccessConfig = utils.parseObject(this.dataAccessConfig); + const dataAccessConfig = utils.parseObject(this.dataAccessConfig); if (Array.isArray(dataAccessConfig) && dataAccessConfig.length) { payload.data_access_config = dataAccessConfig; }components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs (1)
19-27: Add user-friendly summaries for 404/403 (optional).Export clearer summaries on not found / forbidden without hiding the error.
- async run({ $ }) { - const response = await this.databricks.getSQLWarehousePermissions({ - warehouseId: this.warehouseId, - $, - }); - - $.export("$summary", `Retrieved permissions for SQL Warehouse ID ${this.warehouseId}`); - return response; - }, + async run({ $ }) { + try { + const response = await this.databricks.getSQLWarehousePermissions({ + warehouseId: this.warehouseId, + $, + }); + $.export("$summary", `Retrieved permissions for SQL Warehouse ID ${this.warehouseId}`); + return response; + } catch (err) { + if (err?.response?.status === 404) { + $.export("$summary", `SQL Warehouse ${this.warehouseId} not found.`); + } else if (err?.response?.status === 403) { + $.export("$summary", `Access denied to SQL Warehouse ${this.warehouseId}.`); + } + throw err; + } + },components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (1)
25-26: Use resource name in summary when available (nit).Improves UX in runs list.
- $.export("$summary", `Retrieved details for SQL Warehouse ID ${this.warehouseId}`); + $.export("$summary", `Retrieved SQL Warehouse ${response?.name ?? this.warehouseId}`);components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (2)
9-18: Optional: add wait-for-stop toggle.Stopping is often async; offer a wait flag for convenience.
props: { databricks, warehouseId: { description: "The ID of the SQL Warehouse to stop", propDefinition: [ databricks, "warehouseId", ], }, + waitForStop: { + type: "boolean", + label: "Wait Until Stopped", + description: "Polls the warehouse status until STOPPED (or timeout).", + optional: true, + default: false, + }, + pollIntervalMs: { + type: "integer", + label: "Poll Interval (ms)", + optional: true, + default: 5000, + }, },
19-27: Optional: implement simple polling when wait is enabled.Poll status until STOPPED with a bounded loop.
async run({ $ }) { const response = await this.databricks.stopSQLWarehouse({ warehouseId: this.warehouseId, $, }); - $.export("$summary", `Successfully stopped SQL Warehouse with ID ${this.warehouseId}`); - return response; + if (!this.waitForStop) { + $.export("$summary", `Stop requested for SQL Warehouse ${this.warehouseId}`); + return response; + } + // Poll for terminal state + const maxTries = 60; // ~5 min with 5s default interval + for (let i = 0; i < maxTries; i++) { + const { state } = await this.databricks.getSQLWarehouse({ warehouseId: this.warehouseId, $ }); + if (state === "STOPPED") { + $.export("$summary", `SQL Warehouse ${this.warehouseId} is STOPPED.`); + return response; + } + await new Promise((r) => setTimeout(r, this.pollIntervalMs ?? 5000)); + } + $.export("$summary", `Timed out waiting for SQL Warehouse ${this.warehouseId} to stop.`); + return response; },components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (2)
39-41: Validate numResults > 0 (defensive).Prevent accidental zero/negative values.
- if (this.numResults !== undefined) { - body.num_results = this.numResults; - } + if (this.numResults !== undefined) { + if (!Number.isInteger(this.numResults) || this.numResults <= 0) { + throw new Error("numResults must be a positive integer."); + } + body.num_results = this.numResults; + }
35-47: Minor: rename body -> payload for consistency.Matches other actions’ naming.
- const body = {}; + const payload = {}; - if (typeof this.lastPrimaryKey === "string" && this.lastPrimaryKey.trim().length > 0) { - body.last_primary_key = this.lastPrimaryKey.trim(); + if (typeof this.lastPrimaryKey === "string" && this.lastPrimaryKey.trim().length > 0) { + payload.last_primary_key = this.lastPrimaryKey.trim(); } - if (this.numResults !== undefined) { - if (!Number.isInteger(this.numResults) || this.numResults <= 0) { - throw new Error("numResults must be a positive integer."); - } - body.num_results = this.numResults; + if (this.numResults !== undefined) { + if (!Number.isInteger(this.numResults) || this.numResults <= 0) { + throw new Error("numResults must be a positive integer."); + } + payload.num_results = this.numResults; } @@ - data: body, + data: payload,components/databricks/actions/sync-vector-search-index/sync-vector-search-index.mjs (1)
18-30: LGTM with a minor enhancement suggestionLooks good. Consider an optional boolean prop like
waitForCompletionto poll until the sync job reaches a terminal state and surface failures earlier, defaulting to fire-and-forget to preserve current behavior.components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs (1)
42-53: Potential large-payload risk: consider chunking large upserts
inputs_jsoncan get big; APIs often have size limits. Consider batching (e.g., chunks of N rows or bytes) and invoking the endpoint per chunk, aggregating results.components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (2)
64-70: Stricter validation for queryVector content (optional)After
JSON.parse, ensure it’s a non-empty array of numbers to catch silent shape issues early.if (this.queryVector) { try { payload.query_vector = JSON.parse(this.queryVector); + if (!Array.isArray(payload.query_vector) || !payload.query_vector.length || payload.query_vector.some((v) => typeof v !== "number")) { + throw new Error("`queryVector` must be a non-empty JSON array of numbers."); + } } catch (err) { throw new Error(`Invalid queryVector JSON: ${err.message}`); } }
86-88: Result count path may vary across API versionsIf responses use
resultsinstead ofresult.data_array, consider a fallback.- const count = response?.result?.data_array?.length || 0; + const count = response?.result?.data_array?.length + ?? response?.results?.length + ?? 0;components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs (1)
33-41: Optional: deduplicate ACL entries to avoid redundant updatesYou can collapse duplicates by principal + permission_level before sending.
components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs (4)
117-133: Validation is solid; consider aligning defaults and cross-field constraints
- You already enforce
maxNumClusters ≥ minNumClusters. Consider also failing fast whenmaxNumClustersis omitted butminNumClusters > 1and the backend requires both for autoscaling.- Optionally cap
num_clustersat backend-specific limits if they differ by edition.
149-158: Enforce PRO + serverless pairing (optional but user-friendly)When
enableServerlessCompute === true, validatewarehouseType === "PRO"to avoid backend errors.if (this.enableServerlessCompute !== undefined) payload.enable_serverless_compute = this.enableServerlessCompute; + if (this.enableServerlessCompute && this.warehouseType && this.warehouseType !== "PRO") { + throw new ConfigurationError("Serverless compute requires warehouseType = PRO."); + }
135-147: Tags parsing: guard undefined to avoid surprisesIf
this.tagsis undefined andutils.parseObjectexpects a string, default to{}.- const parsedTags = utils.parseObject(this.tags); + const parsedTags = utils.parseObject(this.tags || {});
156-158: Channel is already an object; parsing is optionalIf the prop type enforces an object,
utils.parseObjectis unnecessary. Harmless but can be dropped.components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (2)
140-146: Serverless + CLASSIC guard is good; consider auto-upgrading type (optional)Optional: if
enableServerlessCompute = trueandwarehouseTypenot provided, consider defaultingwarehouse_typetoPROto reduce user error, or add a message prompting users to set it.
148-160: Harden tags parsing and enforce tag count limit (< 45)
parseObjectonundefinedrelies on helper behavior; also enforce documented tag-count constraint and type-safety.Apply this diff:
- const parsedTags = utils.parseObject(this.tags); - const tagArray = Object.entries(parsedTags).map(([ - key, - value, - ]) => ({ - key, - value, - })); - if (tagArray.length) { - payload.tags = { - custom_tags: tagArray, - }; - } + const parsedTags = this.tags ? utils.parseObject(this.tags) : {}; + if (parsedTags && (typeof parsedTags !== "object" || Array.isArray(parsedTags))) { + throw new ConfigurationError("tags must be an object of key-value pairs."); + } + const tagEntries = Object.entries(parsedTags); + if (tagEntries.length > 45) { + throw new ConfigurationError("tags must contain fewer than 45 entries."); + } + const tagArray = tagEntries.map(([key, value]) => ({ key, value })); + if (tagArray.length) { + payload.tags = { custom_tags: tagArray }; + }components/databricks/databricks.app.mjs (1)
47-61: Nit: clarify prop descriptionThis looks copy-pasted from runs; it’s about Warehouses.
Apply this diff:
- description: "The ID of the SQL Warehouse to get runs from", + description: "The ID of the SQL Warehouse",
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (27)
components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs(1 hunks)components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs(1 hunks)components/databricks/actions/delete-sql-warehouse/delete-sql-warehouse.mjs(1 hunks)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs(1 hunks)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs(1 hunks)components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs(1 hunks)components/databricks/actions/get-run-output/get-run-output.mjs(1 hunks)components/databricks/actions/get-sql-warehouse-config/get-sql-warehouse-config.mjs(1 hunks)components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs(1 hunks)components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs(1 hunks)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs(1 hunks)components/databricks/actions/list-runs/list-runs.mjs(1 hunks)components/databricks/actions/list-sql-warehouses/list-sql-warehouses.mjs(1 hunks)components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs(1 hunks)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs(1 hunks)components/databricks/actions/run-job-now/run-job-now.mjs(1 hunks)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs(1 hunks)components/databricks/actions/set-sql-warehouse-config/set-sql-warehouse-config.mjs(1 hunks)components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs(1 hunks)components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs(1 hunks)components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs(1 hunks)components/databricks/actions/sync-vector-search-index/sync-vector-search-index.mjs(1 hunks)components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs(1 hunks)components/databricks/common/constants.mjs(1 hunks)components/databricks/common/utils.mjs(1 hunks)components/databricks/databricks.app.mjs(2 hunks)components/databricks/package.json(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (18)
components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (3)
components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (1)
response(80-84)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
response(43-47)
components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (6)
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs (1)
response(109-112)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
response(14-14)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (1)
response(80-84)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
response(43-47)
components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (4)
components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (1)
response(174-178)components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs (1)
response(20-23)components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs (1)
response(20-23)components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (1)
response(20-23)
components/databricks/actions/set-sql-warehouse-config/set-sql-warehouse-config.mjs (1)
components/databricks/actions/get-sql-warehouse-config/get-sql-warehouse-config.mjs (1)
response(13-15)
components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (2)
components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (1)
response(20-23)components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs (1)
response(20-23)
components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs (3)
components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (1)
response(174-178)components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (1)
response(20-23)components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (1)
response(20-23)
components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
response(20-23)
components/databricks/common/utils.mjs (1)
components/databricks/actions/set-sql-warehouse-config/set-sql-warehouse-config.mjs (2)
obj(110-110)obj(119-121)
components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (4)
components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
response(20-23)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (1)
response(80-84)
components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (5)
components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs (1)
payload(42-45)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
response(20-23)components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
response(14-14)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
response(43-47)
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs (7)
components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (2)
payload(57-60)response(80-84)components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs (1)
payload(42-45)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
response(20-23)components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
response(14-14)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
response(43-47)
components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (1)
components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs (4)
payload(105-108)parsedTags(135-135)tagArray(136-142)response(160-163)
components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs (2)
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs (2)
payload(68-73)response(109-112)components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (4)
payload(106-106)parsedTags(148-148)tagArray(149-155)response(174-178)
components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs (3)
components/databricks/actions/get-sql-warehouse-config/get-sql-warehouse-config.mjs (1)
response(13-15)components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (1)
response(20-23)components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs (1)
response(33-39)
components/databricks/actions/sync-vector-search-index/sync-vector-search-index.mjs (5)
components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
response(20-23)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (1)
response(80-84)components/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
response(43-47)
components/databricks/actions/get-sql-warehouse-config/get-sql-warehouse-config.mjs (1)
components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
response(14-14)
components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs (4)
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs (2)
payload(68-73)response(109-112)components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs (2)
payload(57-60)response(80-84)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs (1)
response(34-38)components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
response(19-22)
components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs (2)
components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (1)
response(174-178)components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs (1)
response(20-23)
🔇 Additional comments (25)
components/databricks/actions/run-job-now/run-job-now.mjs (1)
7-7: Version bump only — LGTM.components/databricks/actions/list-runs/list-runs.mjs (1)
55-57: Trimming fallback still fine — keep as a safety net.components/databricks/actions/get-run-output/get-run-output.mjs (1)
7-7: Version bump only — LGTM.components/databricks/common/constants.mjs (1)
13-15: Default export — LGTM.components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs (1)
13-20: LGTM.Straightforward call/summary/return flow is correct.
components/databricks/actions/delete-vector-search-index/delete-vector-search-index.mjs (1)
18-26: LGTM.Uses propDefinition, invokes app method, and exports a clear summary.
components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs (1)
19-30: LGTM.Consistent with other actions; summary is helpful.
components/databricks/actions/start-sql-warehouse/start-sql-warehouse.mjs (1)
7-7: Version bump sanity check.New action files elsewhere use 0.0.1; confirm 0.0.3 is intentional for this action.
components/databricks/actions/get-sql-warehouse-config/get-sql-warehouse-config.mjs (1)
12-18: LGTM — action wiring and summary are correctStraightforward call-through to the app method with a clear summary. Versioning and metadata look consistent with the suite.
components/databricks/actions/set-sql-warehouse-config/set-sql-warehouse-config.mjs (1)
77-88: Sanity-check allowed keys against Databricks SQL Warehouse API contract
No JSDoc or schema forsetSQLWarehouseConfigindatabricks.app.mjs; manually compare yourallowedlist with the official/sql/config/warehousesPUT request schema in the Databricks REST API docs or OpenAPI spec to ensure you’re not dropping any supported fields.components/databricks/actions/list-sql-warehouses/list-sql-warehouses.mjs (2)
12-16: LGTM on the listing call + shape.The call pattern and destructuring look good.
13-16: No action needed: listSQLWarehouses returns { warehouses } as expected
GET /2.0/sql/warehouses returns JSON of the form{ "warehouses": [ … ] }(learn.microsoft.com), andlistSQLWarehouses()simply returns that raw JSON.components/databricks/actions/get-sql-warehouse/get-sql-warehouse.mjs (1)
20-23: getSQLWarehouse signature validated
MethodgetSQLWarehouseincomponents/databricks/databricks.app.mjs:138destructures{ warehouseId, ...args }, so invoking it with{ warehouseId: this.warehouseId, $ }is correct.components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (1)
6-6: Confirm and align Stop API reference & states
- Update description link in components/databricks/actions/stop-sql-warehouse/stop-sql-warehouse.mjs (line 6) to the canonical Stop API docs: https://docs.databricks.com/sql/api/warehouses/stop
- Verify polling or summary logic uses
response.state === "STOPPED"as a terminal state alongside STARTING, RUNNING, and STOPPINGcomponents/databricks/actions/scan-vector-search-index/scan-vector-search-index.mjs (1)
49-53: No changes needed for response shape
The ScanVectorIndexResponse schema returns the entries inresponse.dataas a list, so the existing summary andresponse.data.lengthcheck are correct.components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs (1)
26-41: Confirmedutils.parseObjecthandles arrays—no changes needed
parseObjectusesArray.isArrayto recursively map arrays, soparsedRowswill be an array whenrowsis an array or a JSON string thereof. Keep the existing array check.components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs (1)
26-33: No action required:utils.parseObjectalready performs recursive array mapping, so the subsequent filter covers empty or falsy entries correctly.components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs (1)
110-115: Good guard on autoStopMinutesClear validation and mapping to
auto_stop_minslooks correct.components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs (3)
75-81: Good: required field enforcement for DELTA_SYNCClear, early validation of
sourceTableforDELTA_SYNC.
114-118: Good: informative success summaryExports a concise, useful
$summary.
50-56: Fix prop type: embeddingSourceColumns should be object[] (not string[])The API expects an array of objects, but the prop is declared as
string[]and later treated as objects. This will yield a malformed payload at runtime.Apply this diff:
- embeddingSourceColumns: { - type: "string[]", + embeddingSourceColumns: { + type: "object[]", label: "Embedding Source Columns", description: - "List of embedding source column configs. Each entry should be a JSON object string like `{ \"embedding_model_endpoint_name\": \"e5-small-v2\", \"name\": \"text\" }`", + "List of embedding source column configs. Example: `{ \"embedding_model_endpoint_name\": \"e5-small-v2\", \"name\": \"text\" }`", optional: true, },⛔ Skipped due to learnings
Learnt from: GTFalcao PR: PipedreamHQ/pipedream#13961 File: components/fakturoid/actions/create-invoice/create-invoice.mjs:95-99 Timestamp: 2024-10-15T15:07:38.361Z Learning: An "object array" type is not yet supported for Pipedream props.components/databricks/actions/edit-sql-warehouse/edit-sql-warehouse.mjs (3)
108-121: Validation reads wellName length and
autoStopMinsconstraints are clear and enforceable.
123-138: Cluster bounds validation looks correctRange checks and
max >= minconstraint are accurate.
180-181: Good: succinct success summaryClear confirmation of the edited warehouse.
components/databricks/databricks.app.mjs (1)
62-75: LGTM: dynamic index optionsSimple, correct mapping of index names to option labels.
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs
Outdated
Show resolved
Hide resolved
...nents/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs
Outdated
Show resolved
Hide resolved
components/databricks/actions/get-sql-warehouse-permissions/get-sql-warehouse-permissions.mjs
Show resolved
Hide resolved
components/databricks/actions/query-vector-search-index/query-vector-search-index.mjs
Show resolved
Hide resolved
components/databricks/actions/set-sql-warehouse-permissions/set-sql-warehouse-permissions.mjs
Outdated
Show resolved
Hide resolved
...nents/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs
Show resolved
Hide resolved
...nents/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
components/databricks/databricks.app.mjs (1)
270-278: Resolved: path vs url bug for deleteVectorSearchDataSwitching to
pathaligns with_makeRequestand base URL construction. Looks good.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs(1 hunks)components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs(1 hunks)components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs(1 hunks)components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs(1 hunks)components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs(1 hunks)components/databricks/databricks.app.mjs(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
- components/databricks/actions/delete-vector-search-index-data/delete-vector-search-index-data.mjs
- components/databricks/actions/upsert-vector-search-index-data/upsert-vector-search-index-data.mjs
- components/databricks/actions/get-vector-search-index/get-vector-search-index.mjs
- components/databricks/actions/list-vector-search-indexes/list-vector-search-indexes.mjs
- components/databricks/actions/create-vector-search-index/create-vector-search-index.mjs
🧰 Additional context used
🧬 Code graph analysis (1)
components/databricks/databricks.app.mjs (1)
components/picqer/picqer.app.mjs (1)
warehouses(91-95)
🔇 Additional comments (6)
components/databricks/databricks.app.mjs (6)
47-61: Consider pagination for large orgsIf the List Warehouses API paginates, options() should accept
{ page }and pass pagination params; otherwise users will only see the first page. Please confirm the API’s pagination shape before changing.
62-73: Vector index options likely need paginationList Vector Search Indexes may paginate. Expose
{ page }and forward pagination params so users can browse all indexes. Please verify the API’s tokens/limits first.
151-159: Confirm HTTP method for “edit warehouse”Docs for this endpoint have varied between POST/PATCH in the past. Please confirm the current spec for
/sql/warehouses/{id}/edit.
195-213: Verify permissions endpoint path for warehousesDouble-check whether the correct path is
/permissions/warehouses/{id}or/permissions/sql/warehouses/{id}for your workspace/API version. Adjust if needed.
214-238: Vector Search endpoints API version checkAll paths are built under
/api/2.0. Confirm that Vector Search endpoints (create/list/get/delete) are indeed v2.0 for your workspace; some tenants expose newer versions.
290-298: Confirm HTTP method for “scan index”Some “scan” endpoints are GET; others are POST with a body. Please verify the expected method/payload for
/vector-search/indexes/{indexName}/scan.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (2)
4-6: Disambiguate action key/name to avoid confusion with Serving EndpointsThis action targets Vector Search Endpoints; the current key/name are generic and easy to confuse with ML Serving “endpoints.”
- key: "databricks-delete-endpoint", - name: "Delete Endpoint", + key: "databricks-delete-vector-search-endpoint", + name: "Delete Vector Search Endpoint",Note: renaming
keyis a breaking change for existing users; consider adding a new action instead if this is already published.
18-26: Consider idempotent delete + clearer summaryIf the API returns 404 for missing endpoints, allow users to treat that as success. Also, deletion may be async; “Requested deletion” is safer wording.
Apply within this block:
- async run({ $ }) { - const response = await this.databricks.deleteEndpoint({ - endpointName: this.endpointName, - $, - }); - - $.export("$summary", `Successfully deleted endpoint "${this.endpointName}".`); - return response; - }, + async run({ $ }) { + try { + const response = await this.databricks.deleteEndpoint({ + endpointName: this.endpointName, + $, + }); + $.export("$summary", `Requested deletion of endpoint "${this.endpointName}".`); + return response; + } catch (err) { + if (this.ignoreNotFound && err?.response?.status === 404) { + $.export("$summary", `Endpoint "${this.endpointName}" not found; treating as already deleted.`); + return { status: 404, message: "Not Found" }; + } + throw err; + } + },Add the optional prop (outside the selected range):
props: { databricks, endpointName: { propDefinition: [ databricks, "endpointName", ], }, + ignoreNotFound: { + type: "boolean", + label: "Ignore Not Found (404)", + description: "Treat a missing endpoint as success to make deletes idempotent.", + optional: true, + default: true, + }, },Please confirm whether the Databricks Delete Vector Search Endpoint API is async (returns 202/operation state). If synchronous, keep the original summary text.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
components/databricks/actions/create-endpoint/create-endpoint.mjs(1 hunks)components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs(1 hunks)components/databricks/actions/delete-endpoint/delete-endpoint.mjs(1 hunks)components/databricks/actions/get-endpoint/get-endpoint.mjs(1 hunks)components/databricks/actions/list-endpoints/list-endpoints.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (3)
- components/databricks/actions/list-endpoints/list-endpoints.mjs
- components/databricks/actions/get-endpoint/get-endpoint.mjs
- components/databricks/actions/create-endpoint/create-endpoint.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- components/databricks/actions/create-sql-warehouse/create-sql-warehouse.mjs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pnpm publish
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: Lint Code Base
🔇 Additional comments (2)
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (2)
7-7: Version bump looks goodMetadata-only change. Aligned with a broader release bump.
6-6: Verify the doc URL pathDatabricks doc paths change occasionally; double-check this specific endpoint URL remains accurate and public.
|
Hii @jcortes could you please verify this PR . |
| type: "string[]", | ||
| label: "Primary Keys", | ||
| description: | ||
| "Values of the index’s primary key column to delete (e.g. `1`, `2`). These are the values for the column you set as `primary_key` when the index was created.", |
There was a problem hiding this comment.
| "Values of the index’s primary key column to delete (e.g. `1`, `2`). These are the values for the column you set as `primary_key` when the index was created.", | |
| "Values of the index's primary key column to delete (e.g. `1`, `2`). These are the values for the column you set as `primary_key` when the index was created.", |
jcortes
moved this from In Review
to Ready for QA
in Component (Source and Action) Backlog
vunguyenhung
moved this from Ready for QA
to In QA
in Component (Source and Action) Backlog
Summary
This PR adds support for Databricks Vector Search Index APIs by implementing 9 new actions.
Changes
Implements 9 Vector Search Index endpoints as discussed in the enhancement issue.
Resolves #18126"
Summary by CodeRabbit