dags don't run in demo

[dschneiderch]

I'm a trying to better understand how airflow and dbt can be used together but I can't actually run the demo.
when I trigger dbt_basic_dag I get airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined
so i appears that the docker container doesn't have the environment variables defined needed to connect to postgres. is there a simple fix for this?

[dschneiderch]

The output from astro dev start suggests the u/p should be admin/admin but that doesn't seem to work:

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default credentials are admin:admin
~/b/i/airflow-dbt-demo ❯❯❯ pgcli -h localhost -d postgres -u admin                                                                                           
Password for admin:
connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "admin"

[bmonteiro]

The output from astro dev start suggests the u/p should be admin/admin but that doesn't seem to work:

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default credentials are admin:admin
~/b/i/airflow-dbt-demo ❯❯❯ pgcli -h localhost -d postgres -u admin                                                                                           
Password for admin:
connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "admin"

Project is running! All components are now available.

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default Airflow UI credentials are: admin:admin
The default Postgres DB credentials are: postgres:postgres

===> postgres:postgres

pgcli -h localhost -d postgres -u postgres
Password for postgres:
Server: PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1)
Version: 3.5.0
Home: http://pgcli.com
postgres@localhost:postgres>

[JKCai]

@dschneiderch Not sure if you've fixed this, by manually adding connection, it works for me.
However, a new error pops up and not sure if others know how to fix.

[2022-12-01, 16:23:03 AEDT] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt']
[2022-12-01, 16:23:03 AEDT] {subprocess.py:86} INFO - Output:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Error importing adapter: No module named 'dbt.adapters.*****'
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error while reading profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05    ERROR: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Defined profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05   - jaffle_shop
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - For more information on configuring profiles, please consult the dbt docs:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - https://docs.getdbt.com/docs/configure-your-profile
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Could not run dbt
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Traceback (most recent call last):
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     mod: Any = import_module("." + name, "dbt.adapters")
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     return _bootstrap._gcd_import(name[level:], package, level)
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - ModuleNotFoundError: No module named 'dbt.adapters.*****'

[JKCai]

For anyone who failed to run due to connection error, you should set hostname to postgres instead of localhost in airflow connection. I now can run this demo.

[chenna09]

@dschneiderch Not sure if you've fixed this, by manually adding connection, it works for me. However, a new error pops up and not sure if others know how to fix.

[2022-12-01, 16:23:03 AEDT] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt']
[2022-12-01, 16:23:03 AEDT] {subprocess.py:86} INFO - Output:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Error importing adapter: No module named 'dbt.adapters.*****'
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error while reading profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05    ERROR: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Defined profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05   - jaffle_shop
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - For more information on configuring profiles, please consult the dbt docs:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - https://docs.getdbt.com/docs/configure-your-profile
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Could not run dbt
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Traceback (most recent call last):
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     mod: Any = import_module("." + name, "dbt.adapters")
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     return _bootstrap._gcd_import(name[level:], package, level)
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - ModuleNotFoundError: No module named 'dbt.adapters.*****'

add dbt-postgres adapter in the requirement file, it fixed my issue.
dbt-postgres==1.3.0