diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index 197dd97..14e8dc2 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,42 +1,59 @@ # Changelog + +## [5.1.0](https://github.com/intelowlproject/pyintelowl/releases/tag/5.1.0) + +Added support for investigation framework and implemented remaining endpoints for playbooks. + ## [5.0.2](https://github.com/intelowlproject/pyintelowl/releases/tag/5.0.2) + Fixed previous broken release ## [5.0.1](https://github.com/intelowlproject/pyintelowl/releases/tag/5.0.1) + - Updated documentation - Removed old endpoints ## [5.0.0](https://github.com/intelowlproject/pyintelowl/releases/tag/5.0.0) + - Fixes for Playbook Analysis ## [4.4.7](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.7) + - Fixed Running Playbook without TLP set ## [4.4.6](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.6) -- Readded default TLP for analysis as TLP:CLEAR for "classic" analyses only (the ones that do not leverage a Playbook) +- Readded default TLP for analysis as TLP:CLEAR for "classic" analyses only (the ones that do not leverage a Playbook) ## [4.4.5](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.5) -- Default TLP for analysis is not TLP:CLEAR anymore. For instance, this prevents the client to overwrite the TLP configuration of a Playbook. + +- Default TLP for analysis is not TLP:CLEAR anymore. For instance, this prevents the client to overwrite the TLP + configuration of a Playbook. ## [4.4.4](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.4) + - Little fixes ## [4.4.3](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.3) + - Fixed client results management in case of errors - Removed support for Python 3.7 ## [4.4.2](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.2) + - Added support for TLP:CLEAR ## [4.4.1](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.1) + - Analyzing a File with a Playbook now works correctly - other little bug fixing ## [4.4.0](https://github.com/intelowlproject/pyintelowl/releases/tag/4.4.0) + - this version supports the usage of a proxy while connecting to IntelOwl via Python code. ## [4.3.0](https://github.com/intelowlproject/pyintelowl/releases/tag/4.3.0) + - this version supports the new Playbooks feature released with IntelOwl v4.1.0 ## [4.2.0](https://github.com/intelowlproject/pyintelowl/releases/tag/4.2.0) @@ -57,7 +74,8 @@ Fixed previous broken release ## [4.1.3](https://github.com/intelowlproject/pyintelowl/releases/tag/4.1.3) -- Library: `IntelOwl.ask_analysis_availability` now accepts an argument `minutes_ago`. Use to specify number of minutes to go back when searching for a previous analysis. +- Library: `IntelOwl.ask_analysis_availability` now accepts an argument `minutes_ago`. Use to specify number of minutes + to go back when searching for a previous analysis. - CLI: `-m/--check-minutes-ago` flag in `analyse`. ## [4.1.2](https://github.com/intelowlproject/pyintelowl/releases/tag/4.1.2) @@ -76,7 +94,10 @@ Fixed previous broken release **Breaking Changes:**: -- Library: The `tags: List[int]` argument has been deprecated in favor of `tags_labels: List[str]` in the methods, `IntelOwl.send_observable_analysis_request` and `IntelOwl.send_file_analysis_request`. Previously, the `tags` argument would accept a list of tag indices, now the `tags_labels` accepts a list of tag labels (non-existing `Tag` objects are created automatically with a randomly generated color). +- Library: The `tags: List[int]` argument has been deprecated in favor of `tags_labels: List[str]` in the methods, + `IntelOwl.send_observable_analysis_request` and `IntelOwl.send_file_analysis_request`. Previously, the `tags` argument + would accept a list of tag indices, now the `tags_labels` accepts a list of tag labels (non-existing `Tag` objects are + created automatically with a randomly generated color). - CLI: Due to above change the `-tl/--tags-list` flag in `analyse` now also accepts a list of tag labels. **Others:** @@ -89,7 +110,8 @@ Fixed previous broken release **Changes:** -- Refactored argument names and ordering for `ask_analysis_availability`, `send_file_analysis_request`, `send_observable_analysis_request` methods to comply with latest changes in IntelOwl's REST API. +- Refactored argument names and ordering for `ask_analysis_availability`, `send_file_analysis_request`, + `send_observable_analysis_request` methods to comply with latest changes in IntelOwl's REST API. - Deprecate `run_all_available_analyzers` argument/flag. **New Features:** @@ -98,7 +120,9 @@ Fixed previous broken release - Ability to request and view "Connector Reports" for a job. - Ability to request `connector_config.json` file and view in either JSON or tabular format. - Ability to request download of sample associated with a job. -- Added `kill`, `retry` and `healthcheck` features to analyzers and connectors. See [Managing Analyzers and Connectors](https://intelowl.readthedocs.io/en/master/Usage.html#managing-analyzers-and-connectors) section of the documentation. +- Added `kill`, `retry` and `healthcheck` features to analyzers and connectors. + See [Managing Analyzers and Connectors](https://intelowl.readthedocs.io/en/master/Usage.html#managing-analyzers-and-connectors) + section of the documentation. **Others:** @@ -145,22 +169,31 @@ Other changes: _Note: Incompatible with previous versions_ -This version brings a complete rewrite of the pyintelowl library as well as command line client. We very much recommend you to update to the latest version to enjoy all new features. +This version brings a complete rewrite of the pyintelowl library as well as command line client. We very much recommend +you to update to the latest version to enjoy all new features. -- The new CLI is written with [pallets/click](https://github.com/pallets/click) and supports all IntelOwl API endpoints. The CLI is well-documented and will help you navigate different commands; you can use it to request new analysis, view an old analysis, view `analyzer_config.json`, view list of tags, list of jobs, etc. -- Complete type-hinting and sphinx docs for the `pyintelowl.IntelOwl` class with helper member functions for each IntelOwl API endpoint. +- The new CLI is written with [pallets/click](https://github.com/pallets/click) and supports all IntelOwl API endpoints. + The CLI is well-documented and will help you navigate different commands; you can use it to request new analysis, view + an old analysis, view `analyzer_config.json`, view list of tags, list of jobs, etc. +- Complete type-hinting and sphinx docs for the `pyintelowl.IntelOwl` class with helper member functions for each + IntelOwl API endpoint. ## [2.0.0](https://github.com/intelowlproject/pyintelowl/releases/tag/2.0.0) -**This version supports only IntelOwl versions >=1.8.0 (about to be released). To interact with previous IntelOwl versions programmatically please refer to pyintelowl version 1.3.5** +**This version supports only IntelOwl versions >=1.8.0 (about to be released). To interact with previous IntelOwl +versions programmatically please refer to pyintelowl version 1.3.5** -- we forced [black](https://github.com/psf/black) style, added linters and precommit configuration. In this way pyintelowl is aligned to IntelOwl. -- we have updated the authentication method from a JWT Token to a simple Token. In this way, it is easier to use pyintelowl for integrations with other products and there are no more concurrency problems on multiple simultaneous requests. +- we forced [black](https://github.com/psf/black) style, added linters and precommit configuration. In this way + pyintelowl is aligned to IntelOwl. +- we have updated the authentication method from a JWT Token to a simple Token. In this way, it is easier to use + pyintelowl for integrations with other products and there are no more concurrency problems on multiple simultaneous + requests. If you were using pyintelowl and IntelOwl before this version, you have to: - update IntelOwl to version>=1.8.0 -- retrieve a new API token from the Django Admin Interface for your user: you have to go in the _Durin_ section (click on `Auth tokens`) and generate a key there. This token is valid until manually deleted. +- retrieve a new API token from the Django Admin Interface for your user: you have to go in the _Durin_ section (click + on `Auth tokens`) and generate a key there. This token is valid until manually deleted. ## [1.3.5](https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.5) @@ -204,7 +237,9 @@ PR #16 for details. ## [1.1.0](https://github.com/intelowlproject/pyintelowl/releases/tag/1.1.0) -Added an option when executing pyintelowl as CLI: `-sc` will show the results in a colorful and organized way that helps the user in looking for useful information. By default, the results are still shown in the JSON format. Thanks to tsale to his idea and contribution. +Added an option when executing pyintelowl as CLI: `-sc` will show the results in a colorful and organized way that helps +the user in looking for useful information. By default, the results are still shown in the JSON format. Thanks to tsale +to his idea and contribution. **Example:** diff --git a/.github/release_template.md b/.github/release_template.md index 868f541..5bf4f04 100644 --- a/.github/release_template.md +++ b/.github/release_template.md @@ -1,8 +1,10 @@ # Checklist for creating a new release +- [ ] I have already checked if all Dependabot issues have been solved before creating this PR. - [ ] Update `CHANGELOG.md` for the new version - [ ] Change version number in `pyintelowl/version.py` - [ ] Verify CI Tests +- [ ] Verify that this PR is for `master` branch from the `develop` branch and that is called with the version number. Example: "5.1.0". This is important because this value is used to auto-build the pyintelowl package and push it in Pypi. - [ ] Merge the PR to the `master` branch. **Note:** Only use "Merge and commit" as the merge strategy and not "Squash and merge". Using "Squash and merge" makes history between branches misaligned. diff --git a/pyintelowl/cli/__init__.py b/pyintelowl/cli/__init__.py index 89c8bfc..a65c851 100644 --- a/pyintelowl/cli/__init__.py +++ b/pyintelowl/cli/__init__.py @@ -1,7 +1,9 @@ from .analyse import analyse from .commands import analyzer_healthcheck, connector_healthcheck from .config import config +from .investigations import investigations from .jobs import jobs +from .playbooks import playbooks from .tags import tags groups = [ @@ -9,9 +11,10 @@ config, jobs, tags, + playbooks, + investigations, ] - cmds = [ analyzer_healthcheck, connector_healthcheck, diff --git a/pyintelowl/cli/_utils.py b/pyintelowl/cli/_utils.py index 536cd6a..2e99224 100644 --- a/pyintelowl/cli/_utils.py +++ b/pyintelowl/cli/_utils.py @@ -31,6 +31,7 @@ def get_status_text(status: str, as_text=True): "pending": ("#CE5C00", str(Emoji("gear"))), "running": ("#CE5C00", str(Emoji("gear"))), "reported_without_fails": ("#73D216", str(Emoji("heavy_check_mark"))), + "concluded": ("#73D216", str(Emoji("heavy_check_mark"))), "reported_with_fails": ("#CC0000", str(Emoji("warning"))), "failed": ("#CC0000", str(Emoji("cross_mark"))), "killed": ("#CC0000", str(Emoji("cross_mark"))), diff --git a/pyintelowl/cli/investigations.py b/pyintelowl/cli/investigations.py new file mode 100644 index 0000000..9d7f9de --- /dev/null +++ b/pyintelowl/cli/investigations.py @@ -0,0 +1,222 @@ +import json + +import click +from rich import box +from rich import print as rprint +from rich.console import Console, Group +from rich.panel import Panel +from rich.table import Table + +from pyintelowl import IntelOwlClientException +from pyintelowl.cli._utils import ( + ClickContext, + add_options, + get_json_syntax, + get_status_text, + json_flag_option, +) + + +@click.group(help="Manage investigations") +def investigations(): + pass + + +def _display_all_investigations(logger, rows): + console = Console() + table = Table(show_header=True, title="List of Investigations", box=box.DOUBLE_EDGE) + header_style = "bold blue" + headers: [] = [ + "Id", + "Name", + "Tags", + "Description", + "Owner", + "TLP", + "Total jobs", + "Jobs", + "Status", + ] + [table.add_column(header=header, header_style=header_style) for header in headers] + try: + for el in rows: + table.add_row( + str(el["id"]), + el["name"], + ", ".join([str(tag) for tag in el["tags"]]), + el["description"], + el["owner"], + el["tlp"], + str(el["total_jobs"]), + ", ".join([str(job_id) for job_id in el["jobs"]]), + el["status"], + ) + console.print(table, justify="center") + except Exception as e: + logger.fatal(e, exc_info=True) + + +@investigations.command(help="Delete job from investigation by their ID") +@click.argument("investigation_id", type=int) +@click.argument("job_id", type=int) +@click.pass_context +def rm(ctx: ClickContext, investigation_id: int, job_id: int): + ctx.obj.logger.info( + f"Requesting delete for Job [underline blue]#{job_id}[/] " + f"from Investigation #[underline blue]#{investigation_id}[/].." + ) + try: + ctx.obj.delete_job_from_investigation(investigation_id, job_id) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) + + +@investigations.command( + help="Add existing job to an existing investigation by their ID" +) +@click.argument("investigation_id", type=int) +@click.argument("job_id", type=int) +@click.pass_context +def add(ctx: ClickContext, investigation_id: int, job_id: int): + ctx.obj.logger.info( + f"Requesting add for Job [underline blue]#{job_id}[/] " + f"to Investigation #[underline blue]#{investigation_id}[/].." + ) + try: + ctx.obj.add_job_to_investigation(investigation_id, job_id) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) + + +def _render_investigation_attributes(data): + style = "[bold #31DDCF]" + tags = ", ".join( + data["tags"] + ) # this is a [str], not a complex object like in job API + status: str = get_status_text(data["status"], as_text=False) + console = Console() + console.print(data) + r = Group( + f"{style}Investigation ID:[/] {str(data['id'])}", + f"{style}Name:[/] {data['name']}", + f"{style}Tags:[/] {tags}", + f"{style}Status:[/] {status}", + f"{style}TLP:[/] {data['tlp']}", + f"{style}Total jobs:[/] {data['total_jobs']}", + f"{style}Jobs ID:[/] {data['jobs']}", + f"{style}Description:[/] {data['description']}", + ) + return Panel(r, title="Investigation attributes") + + +def _render_investigation_table(data, title: str): + headers = ["Name", "Owner", "Jobs"] + table = Table( + show_header=True, + title=title, + box=box.DOUBLE_EDGE, + show_lines=True, + ) + # add headers + for h in headers: + table.add_column(h, header_style="bold blue") + + # retrieve all jobs and childrens + table.add_row( + data.get("name", ""), + str(data.get("owner", "")), + get_json_syntax(data.get("jobs", [])), + ) + return table + + +def _display_investigation(data): + console = Console() + attrs = _render_investigation_attributes(data) + with console.pager(styles=True): + console.print(attrs) + + +def _display_investigation_tree(data): + console = Console() + table = _render_investigation_table(data, title="Investigation report") + with console.pager(styles=True): + console.print(table, justify="center") + + +@investigations.command( + help="Tabular print investigation attributes and results for an investigation ID" +) +@click.argument("investigation_id", type=int) +@add_options(json_flag_option) +@click.pass_context +def view( + ctx: ClickContext, + investigation_id: int, + as_json: bool, +): + ctx.obj.logger.info( + f"Requesting Investigation [underline blue]#{investigation_id}[/].." + ) + try: + ans = ctx.obj.get_investigation_by_id(investigation_id) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) + + if as_json: + rprint(json.dumps(ans, indent=4)) + else: + _display_investigation(ans) + + +@investigations.command( + help="Tabular print investigation's tree structure for an investigation ID" +) +@click.argument("investigation_id", type=int) +@add_options(json_flag_option) +@click.pass_context +def view_tree( + ctx: ClickContext, + investigation_id: int, + as_json: bool, +): + ctx.obj.logger.info( + f"Requesting Investigation tree [underline blue]#{investigation_id}[/].." + ) + try: + ans = ctx.obj.get_investigation_tree_by_id(investigation_id) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) + + if as_json: + rprint(json.dumps(ans, indent=4)) + else: + _display_investigation_tree(ans) + + +@investigations.command(help="List all investigations") +@click.option( + "--status", + type=click.Choice( + ["created", "running", "concluded"], + case_sensitive=False, + ), + show_choices=True, + help="Only show investigations having a particular status", +) +@add_options(json_flag_option) +@click.pass_context +def ls(ctx: ClickContext, status: str, as_json: bool): + ctx.obj.logger.info("Requesting list of investigations..") + try: + ans = ctx.obj.get_all_investigations() + results = ans.get("results", []) + ctx.obj.logger.info(results) + if status: + results = [el for el in results if el["status"].lower() == status.lower()] + if as_json: + rprint(json.dumps(results, indent=4)) + else: + _display_all_investigations(ctx.obj.logger, results) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) diff --git a/pyintelowl/cli/playbooks.py b/pyintelowl/cli/playbooks.py new file mode 100644 index 0000000..7a98f84 --- /dev/null +++ b/pyintelowl/cli/playbooks.py @@ -0,0 +1,126 @@ +import json + +import click +from rich import box +from rich import print as rprint +from rich.console import Console, Group +from rich.panel import Panel +from rich.table import Table + +from pyintelowl import IntelOwlClientException +from pyintelowl.cli._utils import ( + ClickContext, + add_options, + get_success_text, + json_flag_option, +) + + +@click.group(help="Manage playbooks") +def playbooks(): + pass + + +def _display_playbook(data): + style = "[bold #31DDCF]" + tags = ", ".join( + data["tags"] + ) # this is a [str], not a complex object like in job API + console = Console() + console.print(data) + r = Group( + f"{style}Playbook ID:[/] {str(data['id'])}", + f"{style}Name:[/] {data['name']}", + f"{style}Tags:[/] {tags}", + f"{style}TLP:[/] {data['tlp']}", + f"{style}Analyzers:[/] {data['analyzers']}", + f"{style}Connectors:[/] {data['connectors']}", + f"{style}Pivots:[/] {data['pivots']}", + f"{style}Visualizers:[/] {data['visualizers']}", + f"{style}Runtime configuration:[/] {data['runtime_configuration']}", + f"{style}For Organizations:[/] {data['for_organization']}", + f"{style}Disabled:[/] {data['disabled']}", + f"{style}Starting:[/] {data['starting']}", + f"{style}Description:[/] {data['description']}", + ) + return Panel(r, title="Playbook attributes") + + +@playbooks.command(help="Tabular print playbook attributes for a playbook name") +@click.argument("playbook_name", type=str) +@add_options(json_flag_option) +@click.pass_context +def view( + ctx: ClickContext, + playbook_name: str, + as_json: bool, +): + ctx.obj.logger.info(f"Requesting Playbook [underline blue]{playbook_name}[/]..") + try: + ans = ctx.obj.get_playbook_by_name(playbook_name) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) + + if as_json: + rprint(json.dumps(ans, indent=4)) + else: + _display_playbook(ans) + + +def _display_all_playbooks(logger, rows): + console = Console() + table = Table(show_header=True, title="List of Playbooks", box=box.DOUBLE_EDGE) + header_style = "bold blue" + headers: [] = [ + "id", + "name", + "tags", + "tlp", + "analyzers", + "connectors", + "pivots", + "visualizers", + "runtime_configuration", + "for_organization", + "disabled", + "starting", + "description", + ] + [table.add_column(header=header, header_style=header_style) for header in headers] + try: + for el in rows: + table.add_row( + str(el["id"]), + el["name"], + ", ".join([str(tag) for tag in el["tags"]]), + el["tlp"], + ", ".join([str(tag) for tag in el["analyzers"]]), + ", ".join([str(tag) for tag in el["connectors"]]), + ", ".join([str(tag) for tag in el["pivots"]]), + ", ".join([str(tag) for tag in el["visualizers"]]), + str(el["runtime_configuration"]), + get_success_text(el["for_organization"]), + get_success_text(el["disabled"]), + get_success_text(el["starting"]), + el["description"], + ) + console.print(table, justify="center") + except Exception as e: + logger.fatal(e, exc_info=True) + + +@playbooks.command(help="List all playbooks") +@add_options(json_flag_option) +@click.pass_context +def ls(ctx: ClickContext, as_json: bool): + ctx.obj.logger.info("Requesting list of playbooks..") + try: + ans = ctx.obj.get_all_playbooks() + results = ans.get("results", []) + ctx.obj.logger.info(results) + if as_json: + rprint(json.dumps(results, indent=4)) + else: + _display_all_playbooks(ctx.obj.logger, results) + except IntelOwlClientException as e: + ctx.obj.logger.fatal(str(e)) diff --git a/pyintelowl/pyintelowl.py b/pyintelowl/pyintelowl.py index b9e9afa..bc7fea6 100644 --- a/pyintelowl/pyintelowl.py +++ b/pyintelowl/pyintelowl.py @@ -616,6 +616,102 @@ def get_job_by_id(self, job_id: Union[int, str]) -> Dict[str, Any]: response = self.__make_request("GET", url=url) return response.json() + def add_job_to_investigation( + self, investigation_id: Union[int, str], job_id: Union[int, str] + ): + """Add an existing job to an existing investigation. + Endpoint: ``/api/investigation/{job_id}/add_job`` + + Args: + job_id (Union[int, str]): Job ID + investigation_id (Union[int, str]): Investigation ID + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url: str = self.instance + f"/api/investigation/{str(investigation_id)}/add_job" + data: dict = {"job": job_id} + response = self.__make_request("POST", url=url, data=data) + return response.json() + + def delete_job_from_investigation( + self, investigation_id: Union[int, str], job_id: Union[int, str] + ): + """Delete a job from an existing investigation. + Endpoint: ``/api/investigation/{job_id}/remove_job`` + + Args: + job_id (Union[int, str]): Job ID + investigation_id (Union[int, str]): Investigation ID + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url: str = ( + self.instance + f"/api/investigation/{str(investigation_id)}/remove_job" + ) + data: dict = {"job": job_id} + response = self.__make_request("POST", url=url, data=data) + return response.json() + + def get_all_investigations(self) -> Dict[str, Any]: + """Fetch all investigations info. + Endpoint: ``/api/investigation/`` + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url = self.instance + "/api/investigation" + response = self.__make_request("GET", url=url) + return response.json() + + def get_investigation_by_id( + self, investigation_id: Union[int, str] + ) -> Dict[str, Any]: + """Fetch investigation info by ID. + Endpoint: ``/api/investigation/{job_id}`` + + Args: + investigation_id (Union[int, str]): Investigation ID to retrieve + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url = self.instance + "/api/investigation/" + str(investigation_id) + response = self.__make_request("GET", url=url) + return response.json() + + def get_investigation_tree_by_id( + self, investigation_id: Union[int, str] + ) -> Dict[str, Any]: + """Fetch investigation tree info by ID. + Endpoint: ``/api/investigation/{job_id}/tree`` + + Args: + investigation_id (Union[int, str]): Investigation ID to retrieve + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url = self.instance + "/api/investigation/" + str(investigation_id) + "/tree" + response = self.__make_request("GET", url=url) + return response.json() + @staticmethod def get_md5( to_hash: AnyStr, @@ -1085,3 +1181,48 @@ def connector_healthcheck(self, connector_name: str) -> Optional[bool]: url = self.instance + f"/api/connector/{connector_name}/healthcheck" response = self.__make_request("GET", url=url) return response.json().get("status", None) + + def get_playbook_by_name(self, playbook_name: str) -> Dict[str, Any]: + """Fetch playbook info by its name. + Endpoint: ``/api/playbook/{playbook_name}`` + + Args: + playbook_name (str): Playbook name to retrieve + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url = self.instance + "/api/playbook/" + playbook_name + response = self.__make_request("GET", url=url) + return response.json() + + def get_all_playbooks(self) -> Dict[str, Any]: + """Fetch all playbooks info. + Endpoint: ``/api/playbook`` + + Raises: + IntelOwlClientException: on client/HTTP error + + Returns: + Dict[str, Any]: JSON body. + """ + url = self.instance + "/api/playbook" + response = self.__make_request("GET", url=url) + return response.json() + + def disable_playbook_for_org(self, playbook_name: str): + """Disables the plugin for the organization of the user. + Endpoint: ``/api/playbook/{playbook_name}/organization`` + + Args: + playbook_name (str): Playbook name to disable for org + + Raises: + IntelOwlClientException: on client/HTTP error + """ + url = self.instance + "/api/playbook/" + playbook_name + "/organization" + # this call doesn't have a response + self.__make_request("POST", url=url) diff --git a/pyintelowl/version.py b/pyintelowl/version.py index 3a223dd..0d72820 100644 --- a/pyintelowl/version.py +++ b/pyintelowl/version.py @@ -1 +1 @@ -__version__ = "5.0.2" +__version__ = "5.1.0" diff --git a/test-requirements.txt b/test-requirements.txt index f430201..51adeb2 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -1,7 +1,7 @@ black==24.8.0 flake8==7.1.1 isort==5.12.0 -pre-commit==3.8.0 +pre-commit==4.0.1 tox==3.25.1 tox-gh-actions==2.9.1 codecov==2.1.13 \ No newline at end of file diff --git a/tests/__init__.py b/tests/__init__.py index c5db119..72c6c50 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -1,5 +1,7 @@ MOCK_CONNECTIONS = True TEST_JOB_ID = 1 +TEST_PLAYBOOK_NAME = "Playbook1" +TEST_INVESTIGATION_ID = 1 TEST_IP = "8.8.8.8" TEST_DOMAIN = "www.google.com" TEST_URL = "https://www.google.com/search?test" diff --git a/tests/mocked_requests.py b/tests/mocked_requests.py index cb58b4c..d03c230 100644 --- a/tests/mocked_requests.py +++ b/tests/mocked_requests.py @@ -215,3 +215,287 @@ def mocked_delete_tag_by_id(*args, **kwargs): 200, "/api/tags/1", ) + + +def mocked_get_investigation_by_id(*args, **kwargs): + return MockResponse( + { + "id": 1, + "tags": [], + "tlp": "CLEAR", + "total_jobs": 2, + "jobs": [1], + "status": "concluded", + "for_organization": True, + "name": "Analyzer1: https://www.test.com", + "description": "test_description", + "start_time": "2024-11-13T07:42:17.534614Z", + "end_time": "2024-11-13T07:42:35.861687Z", + "owner": "admin", + }, + 200, + "/api/investigation/1", + ) + + +def mocked_get_investigation_tree_by_id(*args, **kwargs): + return MockResponse( + { + "name": "InvestigationName: https://www.test.com", + "owner": 1, + "jobs": [ + { + "pk": 1, + "analyzed_object_name": "https://www.test.com", + "playbook": "Playbook1", + "status": "reported_without_fails", + "received_request_time": "2024-11-13T07:42:17.534614Z", + "is_sample": False, + "children": [ + { + "pk": 2, + "analyzed_object_name": "test.0", + "pivot_config": "Pivot1", + "playbook": "Playbook2", + "status": "reported_without_fails", + "received_request_time": "2024-11-13T07:42:35.243833Z", + "is_sample": True, + } + ], + } + ], + }, + 200, + "/api/investigation/1/tree", + ) + + +def mocked_delete_job_from_investigation(*args, **kwargs): + return MockResponse( + { + "id": 1, + "tags": [], + "tlp": "CLEAR", + "total_jobs": 1, + "jobs": [2], + "status": "concluded", + "for_organization": True, + "name": "InvestigationName: https://www.test.com", + "description": "", + "start_time": "2024-11-14T10:38:05.459358Z", + "end_time": "2024-11-14T10:38:17.638776Z", + "owner": "admin", + }, + 200, + "/api/investigation/1/remove_job", + ) + + +def mocked_add_job_to_investigation(*args, **kwargs): + return MockResponse( + { + "id": 1, + "tags": [], + "tlp": "CLEAR", + "total_jobs": 2, + "jobs": [2], + "status": "concluded", + "for_organization": True, + "name": "InvestigationName: https://www.test.com", + "description": "", + "start_time": "2024-11-14T10:38:05.459358Z", + "end_time": "2024-11-14T10:38:17.638776Z", + "owner": "admin", + }, + 200, + "/api/investigation/1/add_job", + ) + + +def mocked_get_investigation_tree_by_id_two_results(*args, **kwargs): + return MockResponse( + { + "name": "InvestigationName: https://www.test.com", + "owner": 1, + "jobs": [ + { + "pk": 1, + "analyzed_object_name": "https://www.test.com", + "playbook": "Playbook1", + "status": "reported_without_fails", + "received_request_time": "2024-11-13T07:42:17.534614Z", + "is_sample": False, + }, + { + "pk": 2, + "analyzed_object_name": "https://www.test2.com", + "playbook": "Playbook2", + "status": "reported_without_fails", + "received_request_time": "2024-11-14T10:38:05.459358Z", + "is_sample": False, + }, + ], + }, + 200, + "/api/investigation/1/tree", + ) + + +def mocked_get_investigation_tree_by_id_one_result(*args, **kwargs): + return MockResponse( + { + "name": "InvestigationName: https://www.test.com", + "owner": 1, + "jobs": [ + { + "pk": 2, + "analyzed_object_name": "https://www.test2.com", + "playbook": "Playbook2", + "status": "reported_without_fails", + "received_request_time": "2024-11-14T10:38:05.459358Z", + "is_sample": False, + }, + ], + }, + 200, + "/api/investigation/1/tree", + ) + + +def mocked_get_all_investigations(*args, **kwargs): + return MockResponse( + { + "count": 2, + "total_pages": 1, + "results": [ + { + "id": 2, + "tags": [], + "tlp": "CLEAR", + "total_jobs": 2, + "jobs": [2], + "status": "concluded", + "for_organization": True, + "name": "investigation2", + "description": "", + "start_time": "2024-11-13T07:42:17.534614Z", + "end_time": "2024-11-13T07:42:35.861687Z", + "owner": "admin", + }, + { + "id": 1, + "tags": [], + "tlp": "CLEAR", + "total_jobs": 2, + "jobs": [1], + "status": "concluded", + "for_organization": True, + "name": "investigation1", + "description": "", + "start_time": "2024-11-12T11:10:42.887446Z", + "end_time": "2024-11-12T11:10:49.632430Z", + "owner": "admin", + }, + ], + }, + 200, + "/api/investigation", + ) + + +def mocked_get_playbook_by_name(*args, **kwargs): + return MockResponse( + { + "id": 1, + "type": ["test"], + "analyzers": ["Analyzer1"], + "connectors": [], + "pivots": [], + "visualizers": [], + "runtime_configuration": { + "pivots": {}, + "analyzers": {}, + "connectors": {}, + "visualizers": {}, + }, + "scan_mode": 2, + "scan_check_time": "1:00:00:00", + "tags": [], + "tlp": "CLEAR", + "weight": 1, + "is_editable": False, + "for_organization": False, + "name": "Playbook1", + "description": "test", + "disabled": False, + "starting": True, + "owner": None, + }, + 200, + "/api/playbook/Playbook1", + ) + + +def mocked_get_all_playbooks(*args, **kwargs): + return MockResponse( + { + "count": 2, + "total_pages": 1, + "results": [ + { + "id": 1, + "type": ["test"], + "analyzers": ["Analyzer1"], + "connectors": [], + "pivots": [], + "visualizers": [], + "runtime_configuration": { + "pivots": {}, + "analyzers": {}, + "connectors": {}, + "visualizers": {}, + }, + "scan_mode": 2, + "scan_check_time": "1:00:00:00", + "tags": [], + "tlp": "CLEAR", + "weight": 1, + "is_editable": False, + "for_organization": False, + "name": "Playbook1", + "description": "test", + "disabled": False, + "starting": True, + "owner": None, + }, + { + "id": 2, + "type": ["test2"], + "analyzers": ["Analyzer2"], + "connectors": [], + "pivots": ["Pivot1"], + "visualizers": ["Visualizer"], + "runtime_configuration": { + "pivots": {}, + "analyzers": {}, + "connectors": {}, + "visualizers": {}, + }, + "scan_mode": 2, + "scan_check_time": "1:00:00:00", + "tags": [], + "tlp": "AMBER", + "weight": 1, + "is_editable": False, + "for_organization": False, + "name": "Playbook2", + "description": "test", + "disabled": False, + "starting": True, + "owner": None, + }, + ], + }, + 200, + "/api/playbook", + ) diff --git a/tests/test_investigations.py b/tests/test_investigations.py new file mode 100644 index 0000000..c6919f1 --- /dev/null +++ b/tests/test_investigations.py @@ -0,0 +1,265 @@ +from unittest.mock import patch + +from pyintelowl import IntelOwlClientException +from tests.mocked_requests import ( + mocked_delete_job_from_investigation, + mocked_get_all_investigations, + mocked_get_investigation_by_id, + mocked_get_investigation_tree_by_id, + mocked_get_investigation_tree_by_id_one_result, + mocked_get_investigation_tree_by_id_two_results, + mocked_raise_exception, +) +from tests.utils import BaseTest, mock_connections + + +class TestInvestigations(BaseTest): + @mock_connections( + patch("requests.Session.get", side_effect=mocked_get_investigation_by_id) + ) + def test_get_investigation_by_id(self, mock_requests): + investigation = self.client.get_investigation_by_id(self.investigation_id) + self.assertEqual(investigation.get("id", None), 1) + self.assertEqual(investigation.get("tags"), []) + self.assertEqual(investigation.get("total_jobs", 2), 2) + self.assertEqual(investigation.get("jobs", []), [1]) + self.assertEqual(investigation.get("status", ""), "concluded") + self.assertTrue(investigation.get("for_organization", False)) + self.assertEqual( + investigation.get("name", ""), "Analyzer1: https://www.test.com" + ) + self.assertEqual(investigation.get("description", ""), "test_description") + self.assertEqual( + investigation.get("start_time", ""), "2024-11-13T07:42:17.534614Z" + ) + self.assertEqual( + investigation.get("end_time", ""), "2024-11-13T07:42:35.861687Z" + ) + self.assertEqual(investigation.get("owner", ""), "admin") + + investigation = self.client.get_investigation_by_id(str(self.investigation_id)) + self.assertEqual(investigation.get("id", None), 1) + self.assertEqual(investigation.get("tags"), []) + self.assertEqual(investigation.get("total_jobs", 2), 2) + self.assertEqual(investigation.get("jobs", []), [1]) + self.assertEqual(investigation.get("status", ""), "concluded") + self.assertTrue(investigation.get("for_organization", False)) + self.assertEqual( + investigation.get("name", ""), "Analyzer1: https://www.test.com" + ) + self.assertEqual(investigation.get("description", ""), "test_description") + self.assertEqual( + investigation.get("start_time", ""), "2024-11-13T07:42:17.534614Z" + ) + self.assertEqual( + investigation.get("end_time", ""), "2024-11-13T07:42:35.861687Z" + ) + self.assertEqual(investigation.get("owner", ""), "admin") + + @mock_connections(patch("requests.Session.get", side_effect=mocked_raise_exception)) + def test_get_investigation_by_id_invalid(self, mock_requests): + investigation_id = 999 + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_by_id, + investigation_id, + ) + + investigation_id = "999" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_by_id, + investigation_id, + ) + + investigation_id = "" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_by_id, + investigation_id, + ) + + investigation_id = None + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_by_id, + investigation_id, + ) + + investigation_id = "a" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_by_id, + investigation_id, + ) + + @mock_connections( + patch("requests.Session.get", side_effect=mocked_get_investigation_tree_by_id) + ) + def test_get_investigation_tree_by_id(self, mock_requests): + investigation = self.client.get_investigation_tree_by_id(self.investigation_id) + self.assertEqual( + investigation.get("name", ""), "InvestigationName: https://www.test.com" + ) + self.assertEqual(investigation.get("owner", ""), 1) + self.assertNotEqual(investigation.get("jobs", []), []) + + jobs = investigation.get("jobs")[0] + self.assertEqual(jobs.get("pk", None), 1) + self.assertEqual(jobs.get("analyzed_object_name", ""), "https://www.test.com") + self.assertEqual(jobs.get("playbook", ""), "Playbook1") + self.assertEqual(jobs.get("status", ""), "reported_without_fails") + self.assertEqual( + jobs.get("received_request_time", ""), + "2024-11-13T07:42:17.534614Z", + ) + self.assertFalse(jobs.get("is_sample", True)) + + children = jobs.get("children")[0] + self.assertEqual(children.get("pk", 0), 2) + self.assertEqual(children.get("analyzed_object_name", ""), "test.0") + self.assertEqual(children.get("pivot_config", ""), "Pivot1") + self.assertEqual(children.get("playbook", ""), "Playbook2") + self.assertEqual(children.get("status", ""), "reported_without_fails") + self.assertEqual( + children.get("received_request_time", ""), "2024-11-13T07:42:35.243833Z" + ) + self.assertTrue(children.get("is_sample", False)) + + investigation = self.client.get_investigation_tree_by_id( + str(self.investigation_id) + ) + self.assertEqual( + investigation.get("name", ""), "InvestigationName: https://www.test.com" + ) + self.assertEqual(investigation.get("owner", ""), 1) + self.assertNotEqual(investigation.get("jobs", []), []) + + jobs = investigation.get("jobs")[0] + self.assertEqual(jobs.get("pk", None), 1) + self.assertEqual(jobs.get("analyzed_object_name", ""), "https://www.test.com") + self.assertEqual(jobs.get("playbook", ""), "Playbook1") + self.assertEqual(jobs.get("status", ""), "reported_without_fails") + self.assertEqual( + jobs.get("received_request_time", ""), + "2024-11-13T07:42:17.534614Z", + ) + self.assertFalse(jobs.get("is_sample", True)) + + children = jobs.get("children")[0] + self.assertEqual(children.get("pk", 0), 2) + self.assertEqual(children.get("analyzed_object_name", ""), "test.0") + self.assertEqual(children.get("pivot_config", ""), "Pivot1") + self.assertEqual(children.get("playbook", ""), "Playbook2") + self.assertEqual(children.get("status", ""), "reported_without_fails") + self.assertEqual( + children.get("received_request_time", ""), "2024-11-13T07:42:35.243833Z" + ) + self.assertTrue(children.get("is_sample", False)) + + @mock_connections(patch("requests.Session.get", side_effect=mocked_raise_exception)) + def test_get_investigation_tree_by_id_invalid(self, mock_requests): + investigation_id = 999 + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_tree_by_id, + investigation_id, + ) + + investigation_id = "999" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_tree_by_id, + investigation_id, + ) + + investigation_id = "" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_tree_by_id, + investigation_id, + ) + + investigation_id = None + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_tree_by_id, + investigation_id, + ) + + investigation_id = "a" + self.assertRaises( + IntelOwlClientException, + self.client.get_investigation_tree_by_id, + investigation_id, + ) + + @mock_connections( + patch("requests.Session.get", side_effect=mocked_get_all_investigations) + ) + def test_get_all_investigations(self, mock_requests): + investigation = self.client.get_all_investigations() + self.assertEqual(investigation.get("count", 0), 2) + self.assertEqual(investigation.get("total_pages", 0), 1) + self.assertTrue(investigation.get("results", [])) + self.assertEqual(len(investigation.get("results", [])), 2) + + @mock_connections( + patch( + "requests.Session.get", + side_effect=[ + # if more than one side effect you have to actually call the function + mocked_get_investigation_tree_by_id_two_results(), + mocked_get_investigation_tree_by_id_one_result(), + ], + ) + ) + @mock_connections( + patch( + "requests.Session.post", + side_effect=[mocked_delete_job_from_investigation()], + ) + ) + def test_delete_job_from_investigation(self, *mock_requests): + # before delete there are two jobs + investigation = self.client.get_investigation_tree_by_id(self.investigation_id) + self.assertEqual(len(investigation.get("jobs", [])), 2) + + del_response = self.client.delete_job_from_investigation( + self.investigation_id, self.job_id + ) + self.assertEqual(del_response.get("total_jobs", 0), 1) + + # after delete only one job is left + investigation = self.client.get_investigation_tree_by_id(self.investigation_id) + self.assertEqual(len(investigation.get("jobs", [])), 1) + + @mock_connections( + patch( + "requests.Session.get", + side_effect=[ + # if more than one side effect you have to actually call the function + mocked_get_investigation_tree_by_id_one_result(), + mocked_get_investigation_tree_by_id_two_results(), + ], + ) + ) + @mock_connections( + patch( + "requests.Session.post", + side_effect=[mocked_delete_job_from_investigation()], + ) + ) + def add_job_to_investigation(self, *mock_requests): + # before add there is 1 job + investigation = self.client.get_investigation_tree_by_id(self.investigation_id) + self.assertEqual(len(investigation.get("jobs", [])), 1) + + add_response = self.client.add_job_to_investigation( + self.investigation_id, self.job_id + ) + self.assertEqual(add_response.get("total_jobs", 0), 2) + + # after add there are 2 jobs + investigation = self.client.get_investigation_tree_by_id(self.investigation_id) + self.assertEqual(len(investigation.get("jobs", [])), 2) diff --git a/tests/test_playbooks.py b/tests/test_playbooks.py new file mode 100644 index 0000000..c298a1c --- /dev/null +++ b/tests/test_playbooks.py @@ -0,0 +1,34 @@ +from unittest.mock import patch + +from tests.mocked_requests import mocked_get_all_playbooks, mocked_get_playbook_by_name +from tests.utils import BaseTest, mock_connections + + +class TestPlaybooks(BaseTest): + @mock_connections( + patch("requests.Session.get", side_effect=mocked_get_playbook_by_name) + ) + def test_get_playbook_by_name(self, mock_requests): + playbook = self.client.get_playbook_by_name(self.playbook_name) + self.assertEqual(playbook.get("name", ""), "Playbook1") + self.assertEqual(playbook.get("type", []), ["test"]) + self.assertTrue(playbook.get("analyzers", [])) + self.assertEqual(playbook.get("analyzers", []), ["Analyzer1"]) + self.assertFalse(playbook.get("connectors", [])) + self.assertFalse(playbook.get("pivots", [])) + self.assertFalse(playbook.get("visualizers", [])) + self.assertEqual(playbook.get("description", ""), "test") + + @mock_connections( + patch("requests.Session.get", side_effect=mocked_get_all_playbooks) + ) + def test_get_all_playbooks(self, mock_requests): + playbooks = self.client.get_all_playbooks() + self.assertEqual(playbooks.get("count", 0), 2) + self.assertEqual(playbooks.get("total_pages", 0), 1) + self.assertTrue(playbooks.get("results", [])) + self.assertEqual(len(playbooks.get("results", [])), 2) + + results = playbooks.get("results", []) + self.assertEqual(results[0].get("name", ""), "Playbook1") + self.assertEqual(results[1].get("name", ""), "Playbook2") diff --git a/tests/utils.py b/tests/utils.py index de990d0..d921b15 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -12,8 +12,10 @@ TEST_FILE_HASH, TEST_GENERIC, TEST_HASH, + TEST_INVESTIGATION_ID, TEST_IP, TEST_JOB_ID, + TEST_PLAYBOOK_NAME, TEST_URL, ) @@ -55,6 +57,8 @@ class BaseTest(TestCase): def setUp(self): self.client = IntelOwl("test-token", "test-url") self.job_id = TEST_JOB_ID + self.playbook_name = TEST_PLAYBOOK_NAME + self.investigation_id = TEST_INVESTIGATION_ID self.ip = TEST_IP self.url = TEST_URL self.domain = TEST_DOMAIN