Define db sqlalchemy

[rshewitt]

Pull Request

Related to 4400

About

• Design harvest tables using SQLAlchemy
• Create an ERD using PlantUML

PR TASKS

• The actual code changes.
• Tests written and passed.
• Any changes to docs?

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
harvester
__init__.py	3	0	100%
harvester/db/models
__init__.py	5	0	100%
models.py	53	0	100%
harvester/extract
__init__.py	19	2	2	89%
dcatus.py	11	2	2	82%
harvester/utils
__init__.py	0	0	100%
json.py	22	6	6	73%
pg.py	35	4	4	89%
s3.py	24	6	6	75%
harvester/validate
__init__.py	0	0	100%
dcat_us.py	24	0	100%
TOTAL	196	20	90%

Tests	Skipped	Failures	Errors	Time
29	0 💤	0 ❌	0 🔥	22.927s ⏱️

[robert-bryson]

All of the added tests fail. I'm not sure why the check is reporting success.

[jbrown-xentity]

I found the "tests failing but the jobs passes problem". tee eats the exit code, so as long as it writes the output then the command "succeeds". We need to change line 45 in the github project to something like set -o pipefail; poetry run pytest --junitxml=pytest.xml --cov=harvester | tee pytest-coverage.txt. You can confirm locally by running the above command, and then checking the exit code by running echo $?. You can confirm the current behavior by running the current pytest command, and then running the echo command to check the exit code. @rshewitt can you implement that in this PR?

[rshewitt]

yes, i'll add this!

[nickumia-reisys]

I think it's fine as an initial pass. Still lots of open questions. I think this may be stuck for the same reason S3 is stuck, pending how Airflow integrates with it. Will we be initializing the DB with a python package like this? Or does Airflow handle DB setup and init?

I don't mind merging this in the meantime. ⏳

[btylerburton]

when it's fitting, i'd like to have a deeper discussion about the data model given the pivot to Airflow. i'm not sure if we need to merge this, but i don't want to lose this work.

EDIT: let's merge this as we have no reason not to

[rshewitt]

I think it's fine as an initial pass. Still lots of open questions. I think this may be stuck for the same reason S3 is stuck, pending how Airflow integrates with it. Will we be initializing the DB with a python package like this? Or does Airflow handle DB setup and init?

I don't mind merging this in the meantime. ⏳

I don't think we'll use a python package like this for DB setup and init. Based on what i've read it looks like the only kind of DDL expected from us is to create a database and user with the necessary privileges for airflow to use ( and possibly updating pg_hba.conf to add the user to the database access control list ). airflow comes with a db init utility for its metadata database. here's the ERD of the airflow metadata database.

[nickumia-reisys]

⌛ ⌛