As explained in os-climate/os_c_data_commons#232 the dbt directory needs to now be created for ingestion pipelines that want to use Open Metadata. Cell #18 fails without such a directory:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Input In [18], in <cell line: 8>()
1 shutil.rmtree("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", ignore_errors=True)
2 #path = "/opt/app-root/src/sovereign-footprint"
3 #dir_list = os.listdir(path)
4 #print("Files and directories in '", path, "' :")
5 # print the list
6 #print(dir_list)
----> 8 os.mkdir("/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models", mode=0o755)
9 custom_meta_content = {
10 'data provider': df[['data_provider']],
11 'description': 'Description',
12 }
13 custom_meta_fields= {}
FileNotFoundError: [Errno 2] No such file or directory: '/opt/app-root/src/PCAF-sovereign-footprint/dbt/pcaf_transform/models'
Actually, more must be done than merely creating the dbt directory. The pcaf_transform subdirectory must be created, copying the pattern of https://github.com/os-climate/wri-gppd-ingestion-pipeline and/or https://github.com/os-climate/essd-ingest-pipeline/tree/iceberg-dbt
There has also been some motion between sqlalchemy, dbt-trino, and the trino client. sqlalchemy versions >= 1.4.42 must be matched with trino>=0.319.0. Presently the latest version of dbt-trino (1.3.1) forces the installation of trino==0.318.0, which won't work. I will report the inconsistencies to the upstream projects and I expect it will all resolve relatively soon. Until then, best to use sqlalchemy<1.4.42.
As explained in os-climate/os_c_data_commons#232 the
dbtdirectory needs to now be created for ingestion pipelines that want to use Open Metadata. Cell #18 fails without such a directory:Actually, more must be done than merely creating the
dbtdirectory. The pcaf_transform subdirectory must be created, copying the pattern of https://github.com/os-climate/wri-gppd-ingestion-pipeline and/or https://github.com/os-climate/essd-ingest-pipeline/tree/iceberg-dbtThere has also been some motion between sqlalchemy, dbt-trino, and the trino client. sqlalchemy versions >= 1.4.42 must be matched with trino>=0.319.0. Presently the latest version of dbt-trino (1.3.1) forces the installation of trino==0.318.0, which won't work. I will report the inconsistencies to the upstream projects and I expect it will all resolve relatively soon. Until then, best to use sqlalchemy<1.4.42.