Feat: Ignore source tables missing in target model

[Straightpool]

Thanks a lot for this very helpful script! The following is a feature request to make this even easier to new users.

Issue
When running the migration script with extra tables in the source which are not in the target model the migration script will exit with a fatal error

Expected behavior
The migration script should ignore tables in the source model not needed in the target model

Steps to reproduce

• Create fresh Grafana sqlite3 db
• Create fresh Grafana postgre db
• Add extra tables to Grafana sqlite3 db not existing in postgre schema with some random data
• Attempt migration

Sample Error:
FATAL[2023-02-06T10:31:14Z] ❌ pq: relation "dashboard_usage_sums" does not exist INSERT INTO "dashboard_usage_sums" VALUES(0,'2022-01-20 03:56:51',0,0,0,0,0,0,0,0,0,0,0,0) - failed to import dump file to Postgres.

Cause
Apparently, Grafana does not create all tables on the first run but some non critical ones at a later time and/or does not clean up obsolete tables when migrating between different Grafana DB versions.

Version info
Grafana v9.3.6 sqlite3 DB migrated over multiple versions of Grafana from version 7.x to 9.3.6

In my sample case to be dropped tables

• dashboard_usage_by_day
• dashboard_usage_sums
• data_source_acl
• data_source_cache
• data_source_usage_by_day
• license_token
• report
• report_settings
• setting
• team_group
• user_dashboard_views
• user_stats

After dropping above tables in my own sqlite3 data file the migration completed without error and everything still work fine as expected, so nothing of value was lost in migration, as expected.

[stafot]

Is there a way to create them and not drop these tables?

[Straightpool]

Is there a way to create them and not drop these tables?

Sure, if you know the schema of the no longer used tables.

I took a quick look, but didn't find a description to quickly re-create the tables in the target model, so it was much quicker to drop them in the source model. I saw no point in keeping this data around anyway, as it is clearly no longer used.

[stafot]

@Straightpool In my use case I don't want to lose these stats after migration, so I will try to work around it.

[stafot]

@Straightpool The way to avoide this to happen and have these tables created is to use grafana-enterprise docker image instead of grafana. So probably we just need to add a small clarification on the instructions to make the life of future users easier.