build: using bake to build all images in a single step

[chakru-r]

Building via bake makes better use of remote container builder resources## Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[github-actions]

🚫 [actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 81 in c1f7e6f

run: |

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3194	2	3192	112

View the top 1 failed test(s) by shortest run time

tests.read_only.test_services_up::test_gms_config_accessible

Stack Traces | 0.006s run time

auth_session = <tests.utils.TestSessionWrapper object at 0x7f912dc1afb0>

    @pytest.mark.read_only
    def test_gms_config_accessible(auth_session) -> None:
        gms_config = auth_session.get(f"{auth_session.gms_url()}/config").json()
        assert gms_config is not None
    
        if DATAHUB_VERSION is not None:
            assert gms_config["versions"]["acryldata/datahub"]["version"] == DATAHUB_VERSION
        else:
            print("[WARN] TEST_DATAHUB_VERSION is not set")
    
        # Make sure that the default CLI version gets generated properly.
        # While we don't want to hardcode the actual value, we can make
        # sure it mostly looks like a version string.
        default_cli_version: str = gms_config["managedIngestion"]["defaultCliVersion"]
        print(f"Default CLI version: {default_cli_version}")
        assert not default_cli_version.startswith("@")
>       assert "." in default_cli_version or looks_like_a_short_sha(default_cli_version), (
            "Default CLI version does not look like a version string"
        )
E       AssertionError: Default CLI version does not look like a version string
E       assert ('.' in 'pr-build' or False)
E        +  where False = looks_like_a_short_sha('pr-build')

tests/read_only/test_services_up.py:30: AssertionError

View the full list of 1 ❄️ flaky tests

com.linkedin.metadata.entity.EbeanEntityServiceOptimizationTest testUpsertOptimization

Flake rate in main: 37.70% (Passed 38 times, Failed 23 times)

Stack Traces | 0.125s run time

java.lang.AssertionError: (initial: single insert) Expected SELECT SQL count mismatch filtering for (urn:li:dataPlatform:opt):
  1. txn[] select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE (urn, aspect, version) IN ((?, ?, ?)) FOR UPDATE; --bind(urn:li:dataset:(urn:li:dataPlatform:opt,testOpt..., datasetKey, 0) --micros(2,101)
  2. txn[] select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE (urn, aspect, version) IN ((?, ?, ?),(?, ?, ?)) FOR UPDATE; --bind(urn:li:dataset:(urn:li:dataPlatform:opt,testOpt..., dataPlatformInstance, 0, urn:li:dataset:(urn:li:dataPlatform:opt,testOpt..., browsePathsV2, 0) --micros(602)
  3. txn[] select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE (urn, aspect, version) IN ((?, ?, ?)); --bind(urn:li:dataPlatform:opt, dataPlatformInfo, 0) --micros(4,559)
  4. txn[] select t0.urn, t0.aspect, t0.version, t0.urn, t0.aspect, t0.version, t0.createdOn, t0.createdBy, t0.createdFor, t0.systemmetadata from metadata_aspect_v2 t0 where (t0.urn,t0.aspect,t0.version) in ((?,?,?), (?,?,?), (?,?,?), (?,?,?))  for update; --bind(urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),datasetKey,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),browsePathsV2,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),dataPlatformInstance,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),status,0) --micros(809)
  5. txn[] select t0.urn, t0.aspect, t0.version, t0.urn, t0.aspect, t0.version, t0.createdOn, t0.createdBy, t0.createdFor, t0.systemmetadata from metadata_aspect_v2 t0 where (t0.urn,t0.aspect,t0.version) in ((?,?,?), (?,?,?), (?,?,?), (?,?,?))  for update; --bind(urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),datasetKey,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),browsePathsV2,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),dataPlatformInstance,0,urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),status,0) --micros(187)
  6. txn[] select t0.urn, t0.aspect, t0.version from metadata_aspect_v2 t0 where t0.urn in (?) and ((t0.urn = ? and t0.aspect in (?,?,?,?))); --bind(Array[1]={urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),...},urn:li:dataset:(urn:li:dataPlatform:opt,testOptimization,PROD),Array[4]={datasetKey,browsePathsV2,dataPlatformInstance,...}) --micros(569)
  7. txn[] select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE (urn, aspect, version) IN ((?, ?, ?)); --bind(urn:li:dataPlatform:looker, dataPlatformInfo, 0) --micros(170)
txn[]  -- bind(urn:li:dataset:(urn:li:dataPlatform:opt,testOpt...,dataPlatformInstance,0,[LOB],1970-01-01 00:00:00.123,urn:li:corpuser:tester,null,{"lastObserved":1744469379734,"version":"1"})
 expected [6] but found [7]
	at org.testng.Assert.fail(Assert.java:111)
	at org.testng.Assert.failNotEquals(Assert.java:1578)
	at org.testng.Assert.assertEqualsImpl(Assert.java:150)
	at org.testng.Assert.assertEquals(Assert.java:132)
	at org.testng.Assert.assertEquals(Assert.java:1419)
	at org.testng.Assert.assertEquals(Assert.java:1383)
	at com.linkedin.metadata.entity.EbeanEntityServiceOptimizationTest.assertSQL(EbeanEntityServiceOptimizationTest.java:284)
	at com.linkedin.metadata.entity.EbeanEntityServiceOptimizationTest.testUpsertOptimization(EbeanEntityServiceOptimizationTest.java:99)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:664)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:227)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:957)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:200)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[github-actions]

📝 [actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2059:info:2:24: Don't use variables in the printf format string. Use printf '..%s..' "$foo" [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 1064 in dcb24a2

run: |

[github-actions]

📝 [actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2059:info:5:25: Don't use variables in the printf format string. Use printf '..%s..' "$foo" [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 1064 in dcb24a2

run: |

[github-actions]

🚫 [actionlint] _{reported by reviewdog 🐶}
property "datahub_ingestion_slim_build" is not defined in object type {base_build: {outputs: {build_id: string}; result: string}; setup: {outputs: {backend_change: string; backend_only: string; branch_name: string; datahub_executor_change: string; docker-login: string; elasticsearch_setup_change: string; frontend_change: string; frontend_only: string; full_tag: string; ingestion_base_change: string; ingestion_change: string; ingestion_only: string; integrations_service_change: string; kafka_setup_change: string; mysql_setup_change: string; postgres_setup_change: string; pr-publish: string; publish: string; python_release_version: string; repository_name: string; short_sha: string; slim_tag: string; smoke_test_change: string; tag: string; unique_full_tag: string; unique_slim_tag: string; unique_tag: string}; result: string}; smoke_test_matrix: {outputs: {cypress_batch_count: string; matrix: string; python_batch_count: string}; result: string}} [expression]