Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
285 changes: 155 additions & 130 deletions .circleci/config.yml

Large diffs are not rendered by default.

53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,59 @@ The following repository is used to store the OpenAEV injectors for the platform

This repository is used to host injectors that are supported by the core development team of OpenAEV. Nevertheless, the community is also developing a lot of injectors, third-parties modules directly linked to OpenAEV. You can find the list of all available injectors and plugins in the [OpenAEV ecosystem dedicated space](https://filigran.notion.site/OpenAEV-Ecosystem-30d8eb73d7d04611843e758ddef8941b).

### Creating a new injector

#### Project setup
Assuming a new collector by the name of `new_injector`, create a skeleton directory with:
```shell
poetry new new_injector
```

#### `pyoaev` dependency
We wish to retain the possibility to develop simultaneously on `pyoaev` and collectors. We rely on PEP 508 environment
markers to alternatively install a local path `pyoaev` dependency or a released version from PyPI; specifically the `extra`
marker.

Navigate to the new directory and edit `pyproject.toml`.
```shell
vim new_injector/pyproject.toml
```
(or open the file in your favourite editor).

Here's the expression for the pyoaev dependency, including the `extra` definition:
```toml
[tool.poetry.dependencies]
pyoaev = [
{ markers = "extra == 'prod' and extra != 'dev'", version = "<latest pyoaev release on PyPI>", source = "pypi" },
{ markers = "extra == 'dev' and extra != 'prod'", path = "../../client-python", develop = true },
]

[tool.poetry.extras]
prod = ["pyoaev"]
dev = ["pyoaev"]
```

### Simultaneous development on pyoaev and an injector
The injectors repository is set to assume that in the event of a simultaneous development work on both `pyoaev`
and injectors, the `pyoaev` repository is cloned in a directory at the same level as the injectors root directory,
and is named strictly `client-python`.

Here's an example layout:
```
.
├── client-python <= mandatory dir name
│ ├── docs
│ ├── pyoaev
│ ├── scripts
│ └── test
└── injectors <= this repo root dir
├── aws
├── http-query
├── nmap
└── nuclei
```


## Contributing

If you want to help use improve or develop new injector, please check out the **[development documentation for new injectors](https://docs.openaev.io/latest/development/injectors)**. If you want to make your injectors available to the community, **please create a Pull Request on this repository**, then we will integrate it to the CI and in the [OpenAEV ecosystem](https://filigran.notion.site/OpenAEV-Ecosystem-30d8eb73d7d04611843e758ddef8941b).
Expand Down
51 changes: 28 additions & 23 deletions aws/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,35 +1,40 @@
FROM python:3.11-slim
FROM python:3.13-alpine AS builder

# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
gcc \
python3-dev \
libssl-dev \
libffi-dev \
&& rm -rf /var/lib/apt/lists/*
RUN apk update && apk upgrade

# Create working directory
WORKDIR /opt/openaev-injector-aws
# poetry version available on Ubuntu 24.04
RUN pip3 install poetry==2.1.3

# Copy the injector source code
COPY src /opt/openaev-injector-aws
ARG installdir=/opt/injector
ADD . ${installdir}
WORKDIR ${installdir}
RUN poetry build

# Install Python dependencies
RUN pip3 install --no-cache-dir -r requirements.txt
FROM python:3.13-alpine AS runner

ARG installdir=/opt/injector
WORKDIR ${installdir}
COPY --from=builder ${installdir} ${installdir}
RUN pip3 install --no-cache-dir "$(ls dist/*.whl)[prod]"

# Declare the build argument
ARG PYOAEV_GIT_BRANCH_OVERRIDE

RUN if [[ ${PYOAEV_GIT_BRANCH_OVERRIDE} ]] ; then \
echo "Forcing specific version of client-python" && \
apk add --no-cache git && \
pip install pip3-autoremove && \
pip-autoremove pyoaev -y && \
pip install git+https://github.com/OpenAEV-Platform/client-python@${PYOAEV_GIT_BRANCH_OVERRIDE} ; \
fi

# Verify AWS CLI is installed
RUN aws --version || echo "AWS CLI installation verification"

# Create AWS data directory
RUN mkdir -p /root/.local/share/aws

# Set environment variables for AWS
ENV AWS_HOME=/root/.local/share/aws
# Create AWS data directory
RUN mkdir -p ${AWS_HOME}

# Copy and set up entrypoint script
COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh

# Set the entrypoint
ENTRYPOINT ["/entrypoint.sh"]
CMD ["python3", "-m", "aws.openaev_aws"]
88 changes: 47 additions & 41 deletions aws/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,32 @@ The injector supports all current AWS regions including:

**Other Regions**: Middle East (Bahrain, UAE, Tel Aviv), Africa (Cape Town), South America (São Paulo), China (Beijing, Ningxia), AWS GovCloud (US-East, US-West)

## Installation
## Configuration variables

There are a number of configuration options, which are set either in `docker-compose.yml` (for Docker) or
in `config.yml` (for manual deployment).

### OpenAEV environment variables

Below are the parameters you'll need to set for OpenAEV:

| Parameter | config.yml | Docker environment variable | Mandatory | Description |
|---------------|------------|-----------------------------|-----------|------------------------------------------------------|
| OpenAEV URL | url | `OPENAEV_URL` | Yes | The URL of the OpenAEV platform. |
| OpenAEV Token | token | `OPENAEV_TOKEN` | Yes | The default admin token set in the OpenAEV platform. |

### Base injector environment variables

Below are the parameters you'll need to set for running the injector properly:

| Parameter | config.yml | Docker environment variable | Default | Mandatory | Description |
|------------------|------------|-----------------------------|---------|-----------|----------------------------------------------------------------------------------------|
| Injector ID | id | `INJECTOR_ID` | / | Yes | A unique `UUIDv4` identifier for this injector instance. |
| Collector Name | name | `INJECTOR_NAME` | | Yes | Name of the injector. |
| Log Level | log_level | `INJECTOR_LOG_LEVEL` | info | Yes | Determines the verbosity of the logs. Options are `debug`, `info`, `warn`, or `error`. |


## Deployment

### Using Docker

Expand All @@ -102,54 +127,35 @@ docker-compose up -d

### Manual Installation

1. Install Python dependencies:
```bash
cd aws/src
pip install -r requirements.txt
```

2. Install AWS:
```bash
pip install aws
```

3. Configure the injector:
```bash
cp config.yml.sample config.yml
# Edit config.yml with your OpenAEV connection details
```

4. Run the injector:
```bash
python openaev_aws.py
```
Create a file `config.yml` based on the provided `config.yml.sample`.

## Configuration
Replace the configuration variables with the appropriate configurations for
you environment.

### Environment Variables
The poetry package management system (version 2.1 or later) must also be available: https://python-poetry.org/

- `OPENAEV_URL`: URL of your OpenAEV instance
- `OPENAEV_TOKEN`: Authentication token for OpenAEV API
- `INJECTOR_ID`: Unique identifier for this injector instance
- `INJECTOR_NAME`: Display name for the injector (default: "AWS")
- `INJECTOR_LOG_LEVEL`: Logging level (info, warning, error) - debug logging has been removed for production use
Install the environment:

### Configuration File
**Production**:
```shell
# production environment
poetry install --extras prod
```

Create a `config.yml` file based on the provided sample:
**Development** (note that you should also clone the [pyoaev](OpenAEV-Platform/client-python) repository [according to
these instructions](../README.md#simultaneous-development-on-pyoaev-and-an-injector))
```shell
# development environment
poetry install --extras dev
```

```yaml
openaev:
url: 'http://localhost:3001'
token: 'your-openaev-token'
Then, start the collector:

injector:
id: 'unique-injector-id'
name: 'AWS'
log_level: 'info'
```shell
poetry run python -m aws.openaev_aws
```

## Usage in OpenAEV
## Behaviour

1. **Deploy the Injector**: Start the AWS injector using Docker or manual installation
2. **Verify Registration**: Check that the injector appears in OpenAEV under Integrations > Injectors
Expand Down Expand Up @@ -221,7 +227,7 @@ To add support for additional AWS modules:

Run the injector with info logging to test new modules:

```python
```yaml
# In config.yml
injector:
log_level: 'info'
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
11 changes: 7 additions & 4 deletions aws/src/openaev_aws.py → aws/aws/openaev_aws.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
import time
from typing import Dict

from contracts_aws import (
from pyoaev.helpers import OpenAEVConfigHelper, OpenAEVInjectorHelper

from aws.contracts_aws import (
CLOUDTRAIL_ENUM_CONTRACT,
COGNITO_ENUM_CONTRACT,
DYNAMODB_ENUM_CONTRACT,
Expand Down Expand Up @@ -32,8 +34,7 @@
VPC_ENUM_CONTRACT,
AWSContracts,
)
from helpers.pacu_executor import PacuExecutor
from pyoaev.helpers import OpenAEVConfigHelper, OpenAEVInjectorHelper
from aws.helpers.pacu_executor import PacuExecutor


class OpenAEVAWS:
Expand Down Expand Up @@ -67,7 +68,9 @@ def __init__(self):
},
)

self.helper = OpenAEVInjectorHelper(self.config, open("img/icon-aws.png", "rb"))
self.helper = OpenAEVInjectorHelper(
self.config, open("aws/img/icon-aws.png", "rb")
)
self.pacu_executor = PacuExecutor(logger=self.helper.injector_logger)

def aws_execution(self, start: float, data: Dict) -> Dict:
Expand Down
7 changes: 0 additions & 7 deletions aws/entrypoint.sh

This file was deleted.

Loading