Skip to content

Commit b66aa28

Browse files
almorgantealmorgante
authored
Merge pull request #9 from MuttData/master
Update branch source-fb-marketing-learningstage
2 parents e5b49e8 + a6b160b commit b66aa28

File tree

5,147 files changed

+437547
-421808
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

5,147 files changed

+437547
-421808
lines changed

‎.devcontainer/python-connectors-generic/devcontainer.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,14 @@
1010
"version": "3.10",
1111
"installTools": true
1212
},
13-
"ghcr.io/devcontainers-contrib/features/poetry:2": {}
13+
"ghcr.io/devcontainers-extra/features/poetry:2": {}
1414
},
1515

1616
// Deterministic order reduces cache busting
1717
"overrideFeatureInstallOrder": [
1818
"ghcr.io/devcontainers/features/docker-in-docker",
1919
"ghcr.io/devcontainers/features/python",
20-
"ghcr.io/devcontainers-contrib/features/poetry"
20+
"ghcr.io/devcontainers-extra/features/poetry"
2121
],
2222

2323
// Configure tool-specific properties.

‎.github/CODEOWNERS‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Vector db connectors
2-
/airbyte-integrations/connectors/destination-pinecone @airbytehq/ai-language-models
3-
/airbyte-integrations/connectors/destination-weaviate @airbytehq/ai-language-models
4-
/airbyte-integrations/connectors/destination-milvus @airbytehq/ai-language-models
5-
/airbyte-integrations/connectors/destination-qdrant @airbytehq/ai-language-models
6-
/airbyte-integrations/connectors/destination-chroma @airbytehq/ai-language-models
7-
/airbyte-integrations/connectors/destination-snowflake-cortex @airbytehq/ai-language-models
2+
/airbyte-integrations/connectors/destination-pinecone @airbytehq/dev-extensibility
3+
/airbyte-integrations/connectors/destination-weaviate @airbytehq/dev-extensibility
4+
/airbyte-integrations/connectors/destination-milvus @airbytehq/dev-extensibility
5+
/airbyte-integrations/connectors/destination-qdrant @airbytehq/dev-extensibility
6+
/airbyte-integrations/connectors/destination-chroma @airbytehq/dev-extensibility
7+
/airbyte-integrations/connectors/destination-snowflake-cortex @airbytehq/dev-extensibility
88

99
# CI/CD
1010
/.github/ @airbytehq/dev-extensibility

‎.github/actions/connector-image-build-push/README.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Connector Image Build and Push Action
2+
3+
This GitHub Action builds and pushes Docker images for Airbyte connectors (Python and manifest-only connectors). It was converted from the original `connector-image-build-push.yml` workflow to provide a reusable action that can be used across multiple workflows.
4+
5+
## Features
6+
7+
- **Multi-language Support**: Supports Python and manifest-only connectors
8+
- **Smart Image Management**: Checks if images already exist before building/pushing
9+
- **Multi-architecture Builds**: Builds images for both `linux/amd64` and `linux/arm64`
10+
- **Security Scanning**: Includes vulnerability scanning with Anchore
11+
- **Flexible Tagging**: Supports custom tags and automatic latest tag pushing
12+
- **Dry Run Mode**: Build without pushing for testing purposes
13+
- **Force Publishing**: Override existing images when needed
14+
15+
## Usage
16+
17+
### Basic Usage
18+
19+
```yaml
20+
- name: Build and Push Connector
21+
uses: ./.github/actions/connector-image-build-push
22+
with:
23+
connector-name: 'source-faker'
24+
docker-hub-username: ${{ secrets.DOCKER_HUB_USERNAME }}
25+
docker-hub-password: ${{ secrets.DOCKER_HUB_PASSWORD }}
26+
dry-run: 'false'
27+
```
28+
29+
### Advanced Usage
30+
31+
```yaml
32+
- name: Build and Push Connector with Custom Settings
33+
uses: ./.github/actions/connector-image-build-push
34+
with:
35+
connector-name: 'source-postgres'
36+
registry: 'docker.io/airbyte'
37+
tag-override: 'v1.2.3'
38+
push-latest: 'true'
39+
force-publish: 'true'
40+
docker-hub-username: ${{ secrets.DOCKER_HUB_USERNAME }}
41+
docker-hub-password: ${{ secrets.DOCKER_HUB_PASSWORD }}
42+
dry-run: 'false'
43+
```
44+
45+
### Dry Run for Testing
46+
47+
```yaml
48+
- name: Test Build Connector Image
49+
uses: ./.github/actions/connector-image-build-push
50+
with:
51+
connector-name: 'destination-bigquery'
52+
dry-run: 'true'
53+
```
54+
55+
## Inputs
56+
57+
| Input | Description | Required | Default |
58+
|-------|-------------|----------|---------|
59+
| `connector-name` | Connector name (e.g., source-faker, destination-postgres) | **Yes** | - |
60+
| `registry` | Docker registry | No | `docker.io/airbyte` |
61+
| `tag-override` | Override the image tag. If not provided, uses tag from metadata.yaml | No | `""` |
62+
| `push-latest` | Also push image with 'latest' tag | No | `false` |
63+
| `dry-run` | Build but don't push (for testing) | No | `true` |
64+
| `force-publish` | Force publish even if image exists (CAUTION!) | No | `false` |
65+
| `docker-hub-username` | Docker Hub username for authentication | Yes | - |
66+
| `docker-hub-password` | Docker Hub password for authentication | Yes | - |
67+
68+
## Outputs
69+
70+
| Output | Description |
71+
|--------|-------------|
72+
| `connector-name` | The validated connector name |
73+
| `connector-type` | The connector type (python or manifest-only) |
74+
| `connector-dir` | The connector directory path |
75+
| `base-image` | The base image from metadata.yaml |
76+
| `connector-version-tag` | The version tag used for the image |
77+
| `connector-image-name` | The full image name with tag |
78+
| `docker-tags` | All Docker tags used for pushing |
79+
| `image-exists` | Whether the image already exists on the registry |
80+
| `build-executed` | Whether the build was executed |
81+
| `push-executed` | Whether the push was executed |
82+
83+
## How It Works
84+
85+
### 1. Validation Phase
86+
- Validates that the connector directory exists
87+
- Checks for `metadata.yaml` file
88+
- Ensures connector language is supported (Python or manifest-only)
89+
- Extracts base image and version information
90+
- Validates registry and input combinations
91+
92+
### 2. Image Existence Check
93+
- Queries Docker Hub API to check if the image already exists
94+
- Uses exponential backoff retry logic for API reliability
95+
- Makes build/push decisions based on existence and input flags
96+
97+
### 3. Build Phase (if needed)
98+
- Sets up Docker Buildx for multi-architecture builds
99+
- First builds a local image for testing
100+
- Runs a `spec` command test to validate the image
101+
- Then builds the multi-architecture image for publishing
102+
103+
### 4. Security Scan
104+
- Runs Anchore vulnerability scanning on the built image
105+
- Uses "high" severity cutoff but doesn't fail the build
106+
107+
### 5. Push Phase (if enabled)
108+
- Authenticates with Docker Hub (if credentials provided)
109+
- Pushes the multi-architecture image with specified tags
110+
- Can include both versioned and latest tags
111+
112+
## Requirements
113+
114+
### Repository Structure
115+
- Connector must be located in `airbyte-integrations/connectors/<connector-name>/`
116+
- Must have a valid `metadata.yaml` file
117+
- Must be a Python or manifest-only connector type
118+
119+
### Dependencies
120+
- The action installs `uv`, `poethepoet`, and `yq` automatically
121+
- Requires appropriate Dockerfile templates in `docker-images/`
122+
- Needs access to `poe` tasks for connector metadata extraction
123+
124+
### Permissions
125+
- `contents: read` - To checkout code

0 commit comments

Comments
 (0)