sync for 1.11 with internal repo

CHANGELOG.md

@@ -1,8 +1,11 @@
 ## dbt-teradata 1.0.0a
 
 ### Features
+* Added support for the dbt `function` resource type (dbt 1.11+), enabling `REPLACE FUNCTION` materialization of Teradata SQL scalar UDFs. Aggregate UDFs (`type: aggregate`) raise a clear compile-time error since they are not supported.
+* Implement `persist_docs` support ([IDE-26225](https://teradata-pe.atlassian.net/browse/IDE-26225)): model and column `description:` text is now written to the Teradata catalog as native `COMMENT ON TABLE/VIEW/FUNCTION/COLUMN` metadata for table, view, incremental, seed, and snapshot materializations; for `function` materializations, only the relation-level `description:` is applied via `COMMENT ON FUNCTION` (argument/column-level docs are not supported). Comments are escaped, capped at Teradata's 255-character limit, only re-issued when changed, and skipped gracefully for OTF/Iceberg relations. Relation comments now also surface in `dbt docs generate` catalog output for relational resources (tables/views/etc.); dbt-core excludes `function` nodes from the catalog fetch, so function comments do not appear there.
 
 ### Fixes
+* Fixed snapshots with `hard_deletes='new_record'` reprocessing already-deleted records on every subsequent run, generating a new "deletion" record (with a reused `dbt_scd_id`) each time instead of only once when the deletion is first detected.
 
 ### Docs
 

README.md

@@ -272,6 +272,7 @@ Refer to [connection parameters](https://github.com/Teradata/python-driver#conne
 * `table`
 * `ephemeral`
 * `incremental`
+* `function` (User-Defined Functions — see [UDF materialization support](#user-defined-function-udf-materialization-support))
 
 #### Incremental Materialization
 The following incremental materialization strategies are supported:
@@ -634,6 +635,63 @@ Another e.g. for adding multiple grants:
 
 More on Grants can be found at https://docs.getdbt.com/reference/resource-configs/grants
 
+#### Persist docs
+
+`persist_docs` writes the `description:` text from your model/column YAML into the Teradata data dictionary as native object comments (`COMMENT ON TABLE`, `COMMENT ON VIEW`, `COMMENT ON FUNCTION`, `COMMENT ON COLUMN`). This makes dbt the single source of truth for documentation and surfaces those descriptions in Teradata Studio, BI tools, and data catalogs (e.g. Collibra, Alation) that read metadata from `DBC`. It is supported for `table`, `view`, `incremental`, `seed`, `snapshot`, and (relation-level only — see below) `function` materializations.
+
+It is disabled by default. Enable it — independently for relations (tables/views) and columns — in `dbt_project.yml`, per folder, or per model:
+
+  dbt_project.yml
+  ```yaml
+  models:
+    <project-name>:
+      +persist_docs:
+        relation: true
+        columns: true
+  ```
+
+  or per model:
+  ```sql
+  {{ config(persist_docs={"relation": true, "columns": true}) }}
+  ```
+
+The comment text comes from the `description:` fields:
+
+  models/schema.yml
+  ```yaml
+  version: 2
+  models:
+    - name: customers
+      description: "Cleaned customer dimension, one row per customer."
+      columns:
+        - name: customer_id
+          description: "Primary key of the customer."
+  ```
+
+You can verify the persisted comments directly in Teradata:
+  ```sql
+  SELECT CommentString FROM DBC.TablesV
+   WHERE DatabaseName = '<schema>' AND TableName = 'customers';
+
+  SELECT ColumnName, CommentString FROM DBC.ColumnsV
+   WHERE DatabaseName = '<schema>' AND TableName = 'customers';
+  ```
+They also appear in `dbt docs generate` output (`catalog.json`) under each node's `metadata.comment` and per-column `comment`.
+
+Teradata-specific behavior:
+
+* **Length limit** – Teradata comments are stored in `DBC ...CommentString` (`VARCHAR(255)`), so comments are capped at **255 characters**. Longer descriptions are truncated with a warning. Override the cap (if your platform permits) with a project var:
+  ```yaml
+  vars:
+    teradata_max_comment_length: 255
+  ```
+* **Special characters** – single quotes are escaped by doubling (`'` → `''`). Double quotes, newlines, `--`, and `/* */` are preserved as-is and are safe inside Teradata single-quoted string literals, so they round-trip correctly.
+* **Idempotency** – re-running with unchanged descriptions issues no comment DDL; only changed text is re-written (for relations that persist across runs, such as incremental and snapshot — and, per the `REPLACE FUNCTION` behavior noted below, `function`).
+* **Open Table Format (OTF/Iceberg)** – models created via `catalog_name` are skipped gracefully (no error), since native `COMMENT ON` does not apply to them.
+* **Functions (UDFs)** – only the relation-level comment is supported (`COMMENT ON FUNCTION`, driven by the function's own `description:`). Unlike `REPLACE VIEW`/`CREATE OR REPLACE TABLE`, Teradata's `REPLACE FUNCTION` preserves the existing comment across a rebuild, so unchanged descriptions issue no DDL on re-run. Teradata has no DDL for commenting individual arguments, so argument/column-level docs cannot be applied; if a function node carries column/argument metadata and `persist_docs: {columns: true}` is enabled, the adapter emits a warning and skips those comments rather than raising an error (in practice, dbt-core 1.11's `functions.yml` schema does not currently parse a `columns:` block for functions, so this warning path is rarely reached). Function comments are also not shown in `dbt docs generate` output — dbt-core does not treat `function` nodes as "relational", so they are excluded from the catalog fetch that populates `catalog.json` — but they are visible directly in Teradata (e.g. via `DBC.TablesV.CommentString`) and in tools that read the data dictionary. See [UDF materialization support](#user-defined-function-udf-materialization-support) for details.
+
+More on persist_docs can be found at https://docs.getdbt.com/reference/resource-configs/persist_docs
+
 ### Cross DB macros
 Starting with release 1.3, some macros were migrated from [teradata-dbt-utils](https://github.com/Teradata/dbt-teradata-utils) dbt package to the connector. See the table below for the macros supported from the connector.
 
@@ -1071,6 +1129,58 @@ Teradata raises error **7825** ("OTF table not found in external catalog") — o
 * **OTF cannot be used with the `snapshot` materialization.** Setting `catalog_name` on a snapshot raises a compile-time error (snapshots require update/merge semantics OTF does not provide).
 * **Column metadata is not available for OTF tables in dbt docs.** OTF tables are not registered in Teradata's `DBC.ColumnsV` view (only native tables are), so column-level metadata (descriptions, data types, constraints) will not appear in dbt docs when you run `dbt docs generate` and serve with `dbt docs serve`. Table-level metadata is still available and functional.
 
+## User-Defined Function (UDF) materialization support
+
+Starting with dbt 1.11, dbt introduces a `function` resource type. dbt-teradata implements this resource type to materialize Teradata **SQL scalar UDFs** via `REPLACE FUNCTION`.
+
+Functions are defined the same way as any other dbt `function` resource — a `.sql` file containing the function body, plus a `functions.yml` declaring its arguments, return type, and config:
+
+```sql
+-- functions/add_two_ints.sql
+RETURN a + b;
+```
+
+```yaml
+# functions/functions.yml
+functions:
+  - name: add_two_ints
+    config:
+      type: scalar
+      volatility: deterministic
+    language: sql
+    arguments:
+      - name: a
+        data_type: INTEGER
+      - name: b
+        data_type: INTEGER
+    returns:
+      data_type: INTEGER
+```
+
+The function body must be a bare Teradata `RETURN` statement — dbt-teradata wraps it with the rest of the `REPLACE FUNCTION` DDL (argument list, `RETURNS`, `LANGUAGE SQL`, data access clause, volatility, `COLLATION INVOKER`, `INLINE TYPE 1`). Call the function from a model with the `function()` Jinja helper:
+
+```sql
+-- models/use_udf.sql
+select {{ function('add_two_ints') }}(10, 32) as total
+```
+
+### What is supported
+
+* **Scalar SQL UDFs** (`type: scalar`, `language: sql`) — the only function type/language dbt-teradata materializes.
+* **`volatility` config** — `deterministic` maps to `DETERMINISTIC`; `stable` or `non-deterministic` maps to `NOT DETERMINISTIC`. If omitted, neither clause is emitted (Teradata's own default applies). An unrecognized value emits a warning and is ignored.
+* **`grants`** — supported via the `execute` privilege, which is rendered as Teradata's `GRANT EXECUTE FUNCTION` DCL (plain `EXECUTE` is for macros/stored procedures, so the adapter adds the required `FUNCTION` keyword automatically for UDFs). Configure it like any other resource, e.g. `grants: {execute: ['reporting_role']}`. Table DML privileges (`select`/`insert`/`update`/`delete`) do not apply to UDFs.
+* **`persist_docs` (relation-level only)** — a function's `description:` is written as a native `COMMENT ON FUNCTION` when `persist_docs: {relation: true}` is set (project-wide or per-function `config:`). Unlike `REPLACE VIEW`/`CREATE OR REPLACE TABLE` (which drop-and-recreate the object, wiping any existing comment), Teradata's `REPLACE FUNCTION` preserves the existing comment across a rebuild, so change detection is meaningful here: unchanged descriptions issue no DDL on re-run, and only a changed description re-issues it. Column/argument-level `persist_docs` (`columns: true`) is not supported — Teradata has no DDL for commenting individual arguments — so if a function node carries column/argument metadata, those comments are skipped with a warning instead of raising an error (dbt-core 1.11's `functions.yml` schema does not currently parse a `columns:` block for functions, so this warning path is rarely reached in practice). Function comments do not appear in `dbt docs generate`'s `catalog.json` (dbt-core does not treat `function` nodes as "relational"), but they are visible directly in Teradata, e.g. via `SELECT CommentString FROM DBC.TablesV WHERE TableName = '<function_name>'`.
+* Functions participate in the DAG like any other resource and can be depended on with `ref()`/`function()` from downstream models; `dbt list --resource-type function` and `dbt build --select +<model>` work as expected.
+* Re-running is idempotent: the materialization always issues `REPLACE FUNCTION`, so re-running a project with an unchanged function definition does not error.
+
+### What is not supported
+
+* **Aggregate UDFs** (`type: aggregate`) — not implemented. Declaring one raises a clear compile-time error rather than emitting invalid DDL; use `type: scalar` for SQL scalar UDFs.
+* **Languages other than SQL** — dbt-teradata only emits `LANGUAGE SQL` / `INLINE TYPE 1` DDL. Note that dbt 1.11 itself does not yet parse the `language:` key in `functions.yml` (it always defaults to `sql`), so this is currently a moot point regardless of what you set there.
+* **SQL data access clauses other than `CONTAINS SQL`** — Teradata's `LANGUAGE SQL` / `INLINE TYPE 1` UDFs only accept `CONTAINS SQL`; `NO SQL`, `READS SQL DATA`, and `MODIFIES SQL DATA` are for external (C/Java) UDFs and stored procedures and are not applicable here.
+* **Reserved-word or special-character argument names.** Argument names in `functions.yml` are emitted unquoted into the `REPLACE FUNCTION` signature (e.g. `RETURN a + b;` requires an unquoted `a INTEGER` parameter, since Teradata folds unquoted identifiers to upper case and the function body references arguments unquoted). This means an argument named after a Teradata reserved word, or containing spaces/special characters, will produce invalid DDL. Use plain, non-reserved identifier names for UDF arguments.
+* The dbt user must already have `CREATE FUNCTION` privilege on the target schema; dbt-teradata does not grant it automatically.
+
 ## temporary_metadata_generation_schema (earlier fallback_schema)
 dbt-teradata internally created temporary tables to fetch the metadata of views for manifest and catalog creation. 
 In case if user does not have permission to create tables on the schema they are working on, they can define a temporary_metadata_generation_schema(to which they have proper create and drop privileges) in dbt_project.yml as variable.

dbt/adapters/teradata/column.py

@@ -13,6 +13,7 @@ class TeradataColumn(Column):
     table_name: Optional[str] = None
     table_type: Optional[str] = None
     column_index: Optional[int] = None
+    comment: Optional[str] = None
 
     @property
     def quoted(self) -> str:

dbt/include/teradata/macros/adapters.sql

@@ -284,7 +284,10 @@
           END AS table_type,
       {%- endif -%}
 
-      ColumnsV.ColumnID AS column_index
+      ColumnsV.ColumnID AS column_index,
+      {#-- "comment" is a Teradata reserved word, so the alias must be quoted. Column
+          mapping is positional (api.Column(*row)), so the alias name itself is cosmetic. --#}
+      ColumnsV.CommentString AS "comment"
     FROM
     {% if use_qvci == True -%}
       {{ information_schema_name(relation.schema) }}.ColumnsJQV AS ColumnsV

dbt/include/teradata/macros/apply_grants.sql

@@ -12,6 +12,13 @@
 {% set TD_db_name= relation.schema %}
 {% set TD_table_name=relation.identifier %}
 
+{%- if relation.type == 'function' -%}
+{#-- UDFs carry the EXECUTE FUNCTION access right ('EF') in DBC.AllRightsV, not the
+     table DML rights (R/U/I/D). Report it as the 'execute' privilege so it matches
+     the `grants: {execute: [...]}` config key when dbt diffs current vs. desired grants. --#}
+SEL t.Username as grantee, 'execute' as privilege_type FROM DBC.AllRightsV t
+WHERE t.DatabaseName='{{TD_db_name}}' and t.Username <> current_user and t.AccessRight = 'EF' and t.tablename='{{TD_table_name}}';
+{%- else -%}
 with privilege as(
 SELECT privilege_type, abbreviation FROM (sel 'select' as privilege_type, 'R' as abbreviation) As "DUAL"
 UNION ALL
@@ -24,10 +31,40 @@ SELECT privilege_type, abbreviation FROM (sel 'delete' as privilege_type, 'D' as
 
 SEL t.Username as grantee , p.privilege_type FROM DBC.AllRights t cross join privilege p
 WHERE t.DatabaseName='{{TD_db_name}}' and t.Username <> current_user and t.AccessRight IN ('R','RF','I','U','D') and p.abbreviation=t.AccessRight and t.tablename='{{TD_table_name}}';
+{%- endif -%}
 
 {% endmacro %}
 
 
+{#
+  -- Teradata requires the `EXECUTE FUNCTION` privilege keyword for UDFs; plain `EXECUTE`
+  -- is for macros / stored procedures. dbt's grant config key for functions is `execute`,
+  -- so for function relations we render it as `EXECUTE FUNCTION`. Tables/views are
+  -- unaffected and keep dbt's default DCL.
+#}
+{%- macro teradata__function_privilege(privilege) -%}
+    {%- if privilege | lower == 'execute' -%}execute function{%- else -%}{{ privilege }}{%- endif -%}
+{%- endmacro -%}
+
+
+{%- macro teradata__get_grant_sql(relation, privilege, grantees) -%}
+    {%- if relation.type == 'function' -%}
+        grant {{ teradata__function_privilege(privilege) }} on {{ relation.render() }} to {{ grantees | join(', ') }}
+    {%- else -%}
+        grant {{ privilege }} on {{ relation.render() }} to {{ grantees | join(', ') }}
+    {%- endif -%}
+{%- endmacro -%}
+
+
+{%- macro teradata__get_revoke_sql(relation, privilege, grantees) -%}
+    {%- if relation.type == 'function' -%}
+        revoke {{ teradata__function_privilege(privilege) }} on {{ relation.render() }} from {{ grantees | join(', ') }}
+    {%- else -%}
+        revoke {{ privilege }} on {{ relation.render() }} from {{ grantees | join(', ') }}
+    {%- endif -%}
+{%- endmacro -%}
+
+
 {% macro teradata__call_dcl_statements(dcl_statement_list) %}
     {#
       -- We have overridden this macro as teradata doesn't support running multiple dcl statement as a single statement

dbt/include/teradata/macros/catalog.sql

@@ -160,6 +160,7 @@
 {%- endmacro %}
 
 {% macro teradata__get_catalog_results_sql(view_tmp_tables_mapping) -%}
+    {% set use_qvci = var("use_qvci", False) | as_bool %}
     ,
     columns_transformed AS (
         SELECT
@@ -259,9 +260,44 @@
         tables.table_schema = columns_transformed.table_schema
         AND tables.table_name = columns_transformed.table_name
     )
-    SELECT *
+    SELECT
+        joined.table_database,
+        joined.table_schema,
+        joined.table_name,
+        joined.table_type,
+        {#-- Pull the real relation comment from DBC.TablesV keyed on the remapped
+            (real) names, so both tables and views (whose columns come from temp
+            tables) surface their COMMENT ON TABLE/VIEW text in the catalog. --#}
+        rel_comments.CommentString AS table_comment,
+        joined.table_owner,
+        joined.column_name,
+        joined.column_index,
+        joined.column_type,
+        {#-- For views under use_qvci=False, joined.column_comment comes from the temp
+            table (always NULL); the real COMMENT ON COLUMN text lives in DBC.ColumnsV
+            under the view's own name, so we backfill it via the col_comments join below.
+            For tables joined.column_comment is already the real value, so COALESCE keeps
+            it. Under use_qvci=True the columns come from ColumnsJQV (real view columns),
+            so joined.column_comment is already correct and the extra join is skipped. --#}
+        {% if use_qvci -%}
+        joined.column_comment AS column_comment
+        {%- else -%}
+        COALESCE(joined.column_comment, col_comments.CommentString) AS column_comment
+        {%- endif %}
     FROM joined
-    ORDER BY table_schema, table_name, column_index
+    LEFT OUTER JOIN DBC.TablesV AS rel_comments
+      ON rel_comments.DatabaseName = joined.table_schema (NOT CASESPECIFIC)
+      AND rel_comments.TableName = joined.table_name (NOT CASESPECIFIC)
+    {% if not use_qvci -%}
+    {#-- Restricted to views (joined.table_type = 'view'); table columns already carry
+        the real comment in joined.column_comment, so no join is needed for them. --#}
+    LEFT OUTER JOIN DBC.ColumnsV AS col_comments
+      ON joined.table_type = 'view'
+      AND col_comments.DatabaseName = joined.table_schema (NOT CASESPECIFIC)
+      AND col_comments.TableName = joined.table_name (NOT CASESPECIFIC)
+      AND col_comments.ColumnName = joined.column_name (NOT CASESPECIFIC)
+    {% endif -%}
+    ORDER BY joined.table_schema, joined.table_name, joined.column_index
 {%- endmacro %}
 
 --get_catalog_schemas_where_clause_sql(schemas) copied straight from pre-existing get_catalog(). This uses jinja to loop through the provided schema list and make a big WHERE clause of the form: