Skip to content

Commit 824af25

Browse files
authored
Merge branch 'main' into vs255034_IDE-25057
2 parents d891fc5 + 1fcb9e7 commit 824af25

32 files changed

Lines changed: 2136 additions & 38 deletions

File tree

.github/workflows/ci-integration-tests-csae.yml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,6 +265,18 @@ jobs:
265265
timeout_seconds: 300
266266
priority: interactive
267267
retries: 1
268+
dbt_external_tables:
269+
type: teradata
270+
host: $DBT_TERADATA_SERVER_NAME
271+
user: $DBT_TERADATA_USERNAME
272+
password: $DBT_TERADATA_PASSWORD
273+
logmech: TD2
274+
schema: dbt_external_tables
275+
tmode: TERA
276+
threads: 4
277+
timeout_seconds: 300
278+
priority: interactive
279+
retries: 1
268280
EOF
269281
env:
270282
DBT_TERADATA_SERVER_NAME: ${{ steps.create-csae-environments.outputs.teradata-server-name }}
@@ -291,6 +303,12 @@ jobs:
291303
cd $GITHUB_WORKSPACE/test/valid_history_test
292304
chmod 777 run.sh
293305
./run.sh
306+
307+
- name: Run external tables tests
308+
run: |
309+
cd $GITHUB_WORKSPACE/test/dbt_external_tables_test/integration_tests
310+
chmod 777 run.sh
311+
./run.sh
294312
295313
- name: Run nopi tests
296314
run: |

README.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -773,6 +773,92 @@ If no query_band is set by user, default query_band will come in play that is :
773773

774774
> In Teradata, reusing the same alias across multiple common table expressions (CTEs) or subqueries within a single model is not permitted, as it results in parsing errors; therefore, it is essential to assign unique aliases to each CTE or subquery to ensure proper query execution.
775775

776+
## dbt-external-tables
777+
* [dbt-external-tables](https://github.com/dbt-labs/dbt-external-tables) are supported with dbt-teradata from dbt-teradata v1.9.3 onwards.
778+
* Under the hood, dbt-teradata uses the concept of foreign tables to create tables from external sources. More information can be found [here](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Table-Statements/CREATE-FOREIGN-TABLE)
779+
* User need to add the dbt-external-tables packages as dependency and can be resolved with `dbt deps` command
780+
```yaml
781+
packages:
782+
- package: dbt-labs/dbt_external_tables
783+
version: [">=0.9.0", "<1.0.0"]
784+
```
785+
* User need to add dispatch config for the project to pick the overridden macros from dbt-teradata package
786+
```yaml
787+
dispatch:
788+
- macro_namespace: dbt_external_tables
789+
search_order: ['dbt', 'dbt_external_tables']
790+
```
791+
* To define `STOREDAS` and `ROWFORMAT` for in dbt-external tables, one of the below options can be used:
792+
* user can use the standard dbt-external-tables config `file_format` and `row_format` respectively
793+
* Or user can just add it in `USING` config as mentioned in the Teradata's [documentation](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Table-Statements/CREATE-FOREIGN-TABLE/CREATE-FOREIGN-TABLE-Syntax-Elements/USING-Clause)
794+
795+
* For external source, which requires authentication, user needs to create authentication object and pass it in `tbl_properties` as `EXTERNAL SECURITY` object.
796+
For more information on Authentication object please follow this [link](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Authorization-Statements-for-External-Routines/CREATE-AUTHORIZATION-and-REPLACE-AUTHORIZATION)
797+
798+
* Sample external sources are provided below as references
799+
```yaml
800+
version: 2
801+
sources:
802+
- name: teradata_external
803+
schema: "{{ target.schema }}"
804+
loader: S3
805+
806+
tables:
807+
- name: people_csv_partitioned
808+
external:
809+
location: "/s3/s3.amazonaws.com/dbt-external-tables-testing/csv/"
810+
file_format: "TEXTFILE"
811+
row_format: '{"field_delimiter":",","record_delimiter":"\n","character_set":"LATIN"}'
812+
using: |
813+
PATHPATTERN ('$var1/$section/$var3')
814+
tbl_properties: |
815+
MAP = TD_MAP1
816+
,EXTERNAL SECURITY MyAuthObj
817+
partitions:
818+
- name: section
819+
data_type: CHAR(1)
820+
columns:
821+
- name: id
822+
data_type: int
823+
- name: first_name
824+
data_type: varchar(64)
825+
- name: last_name
826+
data_type: varchar(64)
827+
- name: email
828+
data_type: varchar(64)
829+
```
830+
831+
```yaml
832+
version: 2
833+
sources:
834+
- name: teradata_external
835+
schema: "{{ target.schema }}"
836+
loader: S3
837+
838+
tables:
839+
- name: people_json_partitioned
840+
external:
841+
location: '/s3/s3.amazonaws.com/dbt-external-tables-testing/json/'
842+
using: |
843+
STOREDAS('TEXTFILE')
844+
ROWFORMAT('{"record_delimiter":"\n", "character_set":"cs_value"}')
845+
PATHPATTERN ('$var1/$section/$var3')
846+
tbl_properties: |
847+
MAP = TD_MAP1
848+
,EXTERNAL SECURITY MyAuthObj
849+
partitions:
850+
- name: section
851+
data_type: CHAR(1)
852+
```
853+
854+
## Fallback Schema
855+
dbt-teradata internally created temporary tables to fetch the metadata of views for manifest and catalog creation.
856+
In case if user does not have permission to create tables on the schema they are working on, they can define a fallback_schema(to which they have proper create/drop privileges) in dbt_project.yml as variable.
857+
```yaml
858+
vars:
859+
fallback_schema: <schema-name>
860+
```
861+
776862
## Credits
777863

778864
The adapter was originally created by [Doug Beatty](https://github.com/dbeatty10). Teradata took over the adapter in January 2022. We are grateful to Doug for founding the project and accelerating the integration of dbt + Teradata.

dbt/adapters/teradata/connections.py

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -84,14 +84,24 @@ def __post_init__(self):
8484
# When logmech is "browser", username and password should not be provided.
8585
if self.username is not None or self.password is not None:
8686
raise dbt_common.exceptions.DbtRuntimeError(
87-
"Username and password should not be specified when logmech is 'browser'")
87+
"Couldn’t connect to Teradata Vantage SQL Engine. Neither username nor password parameters can be "
88+
"specified in the profile when the logon mechanism (logmech) is ‘BROWSER'. Correct the profile "
89+
"and retry.")
8890
else:
8991
if self.username is None:
90-
raise dbt_common.exceptions.DbtRuntimeError("Must specify `user` in profile")
92+
raise dbt_common.exceptions.DbtRuntimeError("Couldn’t connect to Teradata Vantage SQL Engine. The "
93+
"‘user’ parameter in the profile must be specified when "
94+
"the logon mechanism (logmech) is ‘TD2'. Correct the "
95+
"profile and retry.")
9196
elif self.password is None:
92-
raise dbt_common.exceptions.DbtRuntimeError("Must specify `password` in profile")
97+
raise dbt_common.exceptions.DbtRuntimeError("Couldn’t connect to Teradata Vantage SQL Engine. The "
98+
"‘password’ parameter in the profile must be specified "
99+
"when the logon mechanism (logmech) is ‘TD2'. Correct the "
100+
"profile and retry.")
93101
if self.schema is None:
94-
raise dbt_common.exceptions.DbtRuntimeError("Must specify `schema` in profile")
102+
raise dbt_common.exceptions.DbtRuntimeError("Couldn’t connect to Teradata Vantage SQL Engine. The "
103+
"‘schema’ parameter in the profile must be specified . "
104+
"Correct the profile and retry.")
95105
# teradata classifies database and schema as the same thing
96106
if (
97107
self.database is not None and
@@ -100,8 +110,9 @@ def __post_init__(self):
100110
raise dbt_common.exceptions.DbtRuntimeError(
101111
f" schema: {self.schema} \n"
102112
f" database: {self.database} \n"
103-
f"On Teradata, database must be omitted or have the same value as"
104-
f" schema."
113+
f"Couldn’t connect to Teradata Vantage SQL Engine. The ‘database’ parameter in the profile is "
114+
f"specified and does not match the ‘schema’ parameter value. Correct the profile by removing the "
115+
f"‘database’ parameter or changing it to same value as ‘schema’ parameter and then retry."
105116
)
106117
if self.tmode == "TERA":
107118
note_for_tera = '''
@@ -345,8 +356,8 @@ def connect():
345356
return cls.apply_query_band(connection.handle, credentials.query_band)
346357

347358
except teradatasql.Error as e:
348-
logger.debug("Got an error when attempting to open a teradata "
349-
"connection: '{}'"
359+
logger.debug("Couldn’t connect to Teradata Vantage SQL Engine. The Teradata driver error message is: '{}'"
360+
"Correct the problem and retry."
350361
.format(e))
351362

352363
connection.handle = None
@@ -380,7 +391,8 @@ def exception_handler(self, sql):
380391
raise dbt_common.exceptions.DbtDatabaseError(str(e).strip()) from e
381392

382393
except Exception as e:
383-
logger.debug("Error running SQL: {}", sql)
394+
logger.debug("Couldn’t execute a SQL request against the Teradata Vantage SQL Engine. The SQL that "
395+
"failed is: {}".format(sql))
384396
logger.debug("Rolling back transaction.")
385397
self.rollback_if_open()
386398
if isinstance(e, dbt_common.exceptions.DbtRuntimeError):
@@ -474,7 +486,8 @@ def apply_query_band(cls, handle, query_band_text):
474486
logger.debug("Query Band set to {}".format(rows)) # To log in dbt.log
475487
except teradatasql.Error as ex:
476488
logger.debug(ex)
477-
logger.info("Please verify query_band parameter in profiles.yml file")
489+
logger.info("Couldn’t set the query_band using the specified value. Correct the query_band parameter in "
490+
"the profile and retry.")
478491
raise dbt.exceptions.DbtRuntimeError(str(ex))
479492

480493
return handle # returning the connection handle

dbt/adapters/teradata/impl.py

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -134,16 +134,18 @@ def list_relations_without_caching(
134134
if f"Teradata database '{schema_relation}' not found" in errmsg:
135135
return []
136136
else:
137-
description = "Error while retrieving information about"
138-
logger.debug(f"{description} {schema_relation}: {e.msg}")
137+
logger.debug(f"Couldn’t retrieve relations from database {schema_relation}. An unexpected error was "
138+
f"encountered. The Teradata error message is: {e.msg}. You may create a github issue "
139+
f"with relevant details.")
139140
return []
140141

141142
relations = []
142143
for row in results:
143144
if len(row) != 4:
144145
raise dbt_common.exceptions.DbtRuntimeError(
145-
f'Invalid value from "teradata__list_relations_without_caching({kwargs})", '
146-
f'got {len(row)} values, expected 4'
146+
f"Invalid value from 'teradata__list_relations_without_caching({kwargs})', "
147+
f"We expected 4 attributes but received {len(row)}. You may create a github issue with relevant "
148+
f"details."
147149
)
148150
_, name, _schema, relation_type = row
149151
relation: BaseRelation = self.Relation.create(
@@ -188,8 +190,8 @@ def _get_one_catalog(
188190
) -> agate.Table:
189191
if len(schemas) != 1:
190192
raise dbt_common.exceptions.CompilationError(
191-
f'Expected only one schema in _get_one_catalog() for Teradata adapter, found '
192-
f'{schemas}'
193+
f'Couldn’t retrieve catalog. We expected 1 input schema in _get_one_catalog() but received {schemas}. '
194+
f'You may create a github issue with relevant details.'
193195
)
194196

195197
return super()._get_one_catalog(information_schema, schemas, relation_config)
@@ -243,7 +245,8 @@ def string_add_sql(
243245
return f"concat('{value}', cast(trim({add_to}) as varchar(63800))"
244246
else:
245247
raise dbt_common.exceptions.DbtRuntimeError(
246-
f'Got an unexpected location value of "{location}"'
248+
f'Couldn’t modify SQL string. The location parameter was neither "append" nor "prepend". Correct the '
249+
f'model and retry.'
247250
)
248251

249252
def get_rows_different_sql(

dbt/include/teradata/macros/catalog.sql

Lines changed: 55 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,10 @@
1+
{% macro teradata_get_current_timestamp() %}
2+
{%- call statement('current_timestamp', fetch_result=True) -%}
3+
select cast(current_timestamp(0) as timestamp(0));
4+
{% endcall %}
5+
{{ return(load_result('current_timestamp').table.columns['Current TimeStamp(0)'].values()[0] | trim) }}
6+
{% endmacro %}
7+
18
{% macro teradata__get_views_from_relations(relations) -%}
29
{% set view_relations = [] %}
310
{%- for relation in relations -%}
@@ -15,30 +22,42 @@
1522
{%- endmacro %}
1623

1724
{% macro teradata__create_tmp_tables_of_views(view_relations) -%}
25+
26+
{% set fallback_schema = var("fallback_schema", null) %}
27+
{{ log("fallback_schema set to : " ~ fallback_schema) }}
28+
1829
{% set view_tmp_tables_mapping = {} %}
1930
{%- for relation in view_relations -%}
20-
{% set temp_relation_for_view = relation.identifier ~ '_tmp_viw_tbl' %}
31+
{% set timestamp = teradata_get_current_timestamp() %}
32+
{% set rand = range(1, 100000) | random %}
33+
{% set uuid = timestamp.replace(":", "").replace("-", "").replace(" ","").replace("+","").replace(".","") ~ rand %}
34+
{% set temp_relation_for_view = relation.identifier ~ '_tmp_viw_tbl_' ~ uuid %}
2135
{% set view_tmp_tables_mapping = view_tmp_tables_mapping.update({relation: temp_relation_for_view}) %}
2236
{%- endfor %}
2337

24-
{{ teradata__drop_tmp_tables_of_views(view_tmp_tables_mapping) }}
2538

2639
{%- for relation, temp_relation_for_view in view_tmp_tables_mapping.items() %}
40+
{% if fallback_schema==null %}
41+
{% set schema_name = relation.schema %}
42+
{% else %}
43+
{% set schema_name = fallback_schema %}
44+
{% endif %}
45+
46+
{{ teradata__drop_tmp_tables_of_views(schema_name, temp_relation_for_view) }}
47+
2748
{% call statement('creating_table_from_view', fetch_result=False) %}
28-
CREATE TABLE "{{ relation.schema }}"."{{ temp_relation_for_view }}" AS (SELECT * FROM "{{ relation.schema }}"."{{ relation.identifier }}") WITH NO DATA;
49+
CREATE TABLE "{{ schema_name }}"."{{ temp_relation_for_view }}" AS (SELECT * FROM "{{ relation.schema }}"."{{ relation.identifier }}") WITH NO DATA;
2950
{% endcall %}
3051
load_result('creating_table_from_view')
3152
{%- endfor %}
3253
{{ return(view_tmp_tables_mapping) }}
3354
{%- endmacro %}
3455

35-
{% macro teradata__drop_tmp_tables_of_views(view_tmp_tables_mapping) -%}
36-
{% for relation, temp_table in view_tmp_tables_mapping.items() %}
37-
{% call statement('drop_existing_table', fetch_result=False) %}
38-
DROP table /*+ IF EXISTS */ "{{ relation.schema }}"."{{ temp_table }}";
39-
{% endcall %}
40-
load_result('drop_existing_table')
41-
{% endfor %}
56+
{% macro teradata__drop_tmp_tables_of_views(schema_name, temp_relation_for_view) -%}
57+
{% call statement('drop_existing_table', fetch_result=False) %}
58+
DROP table /*+ IF EXISTS */ "{{ schema_name }}"."{{ temp_relation_for_view }}";
59+
{% endcall %}
60+
load_result('drop_existing_table')
4261
{%- endmacro %}
4362

4463

@@ -93,7 +112,15 @@
93112

94113
{% set catalog_table = load_result('catalog').table %}
95114

96-
{{ teradata__drop_tmp_tables_of_views(view_tmp_tables_mapping) }}
115+
{% set fallback_schema = var("fallback_schema", null) %}
116+
{%- for relation, temp_relation_for_view in view_tmp_tables_mapping.items() %}
117+
{% if fallback_schema==null %}
118+
{% set schema_name = relation.schema %}
119+
{% else %}
120+
{% set schema_name = fallback_schema %}
121+
{% endif %}
122+
{{ teradata__drop_tmp_tables_of_views(schema_name, temp_relation_for_view) }}
123+
{%- endfor %}
97124

98125
{{ return(catalog_table) }}
99126
{%- endmacro %}
@@ -194,15 +221,21 @@
194221
joined AS (
195222
SELECT
196223
columns_transformed.table_database,
197-
columns_transformed.table_schema,
198224
{% if view_tmp_tables_mapping is not none and view_tmp_tables_mapping|length > 0 %}
225+
CASE
226+
{% for relation, temp_table in view_tmp_tables_mapping.items() %}
227+
WHEN columns_transformed.table_schema = upper('{{ var("fallback_schema", null) }}') THEN '{{ relation.schema }}'
228+
{% endfor %}
229+
ELSE columns_transformed.table_schema
230+
END as table_schema,
199231
CASE
200232
{% for relation, temp_table in view_tmp_tables_mapping.items() %}
201233
WHEN columns_transformed.table_name = '{{ temp_table }}' THEN '{{ relation.identifier }}'
202234
{% endfor %}
203235
ELSE columns_transformed.table_name
204236
END AS table_name,
205237
{% else %}
238+
columns_transformed.table_schema,
206239
columns_transformed.table_name,
207240
{% endif %}
208241
{% if view_tmp_tables_mapping is not none and view_tmp_tables_mapping|length > 0 %}
@@ -259,15 +292,23 @@
259292
{%- for relation in relations -%}
260293
{% if relation.schema and relation.identifier %}
261294
(
262-
upper("table_schema") = upper('{{ relation.schema }}')
263295
{% if view_tmp_tables_mapping is not none and view_tmp_tables_mapping|length > 0 %}
296+
264297
{% set temp_table = view_tmp_tables_mapping[relation] %}
265298
{% if not temp_table %}
299+
upper("table_schema") = upper('{{ relation.schema }}')
266300
and upper("table_name") = upper('{{ relation.identifier }}')
267301
{% else %}
268-
and upper("table_name") = upper('{{ temp_table }}')
302+
{% if var("fallback_schema", null) != null %}
303+
upper("table_schema") = upper('{{ var("fallback_schema", null) }}')
304+
and upper("table_name") = upper('{{ temp_table }}')
305+
{% else %}
306+
upper("table_schema") = upper('{{ relation.schema }}')
307+
and upper("table_name") = upper('{{ temp_table }}')
308+
{% endif %}
269309
{% endif %}
270310
{% else %}
311+
upper("table_schema") = upper('{{ relation.schema }}')
271312
and upper("table_name") = upper('{{ relation.identifier }}')
272313
{% endif %}
273314
)
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{%- macro teradata__create_external_schema(source_node) -%}
2+
{{ exceptions.raise_compiler_error(
3+
"Creating external schema is not implemented for the Teradata adapter"
4+
) }}
5+
{%- endmacro -%}

0 commit comments

Comments
 (0)