[IDEV-1964_1988] Implement nad and domaindiscovery feed; update readme docs

packages/ti_domaintools/_dev/build/docs/README.md

@@ -1,44 +1,73 @@
-# DomainTools Real Time Unified Feeds
+# DomainTools Feeds
 
-The DomainTools Real Time Unified Feeds integration allows you to monitor DomainTools Newly Observed Domains. 
-The DomainTools NOD Feed provides real-time access to newly registered and observed domains, enabling proactive threat detection and defense. 
+DomainTools Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs. Some feeds also offer data for DNS firewalls in Response Policy Zone (RPZ) format.
 
-With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized. 
-Ideal for threat hunting, phishing prevention, and brand protection, the NOD Feed delivers unparalleled visibility into emerging domain activity to stay ahead of evolving threats.
+Summary of Available Feeds:
 
-For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed. 
-Then you can reference domaintools.nod_feed when using visualizations or alerts.
+- `Domain Discovery`: New domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.
+- `Newly Active Domains (NAD)`: Apex-level domains (e.g. example.com but not <www.example.com>) that we observe based on the latest lifecycle of the domain. A domain may be seen either for the first time ever, or again after at least 10 days of inactivity (no observed resolutions in DNS). Populated with our global passive DNS (pDNS) sensor network.
+- `Newly Observed Domains (NOD)`: Apex-level domains (e.g. example.com but not <www.example.com>) that we observe for the first time, and have not observed previously with our global DNS sensor network.
+
+With over 300,000 new domains observed daily, the feed empowers security teams to identify and block potentially malicious domains before they can be weaponized.
+Ideal for threat hunting, phishing prevention, and brand protection.
+
+For example, if you wanted to monitor Newly Observed Domains (NOD) feed, you could ingest the DomainTools NOD feed.
+Then you can reference ti_domaintools.nod_feed when using visualizations or alerts.
 
 ## Data streams
 
-The DomainTools Real Time Unified Feeds integration collects one type of data streams: logs
+The DomainTools Feeds integration collects one type of data streams: **logs**
 
-Log data streams collected by the DomainTools integration include the Newly Observed Domains (NOD) feed: Apex-level domains (e.g. Example Domain  but not www.example.com) that we observe for the first time, and have not observed previously. 
-Populated with our global DNS sensor network.
+Log data streams collected by the DomainTools integration include the following feeds:
+
+- `Domain Discovery`
+- `Newly Observed Domains (NOD)`
+- `Newly Active Domains (NAD)`
 
 ## Requirements
 
-You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. 
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
 You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
 
-You will require a license to one or more DomainTools feeds, and API credentials. 
-Your required API credentials will vary with your authentication method, detailed below. 
+You will require a license to one or more DomainTools feeds, and API credentials.
+Your required API credentials will vary with your authentication method, detailed below.
 
-Obtain your API credentials from your group’s API administrator. 
+Obtain your API credentials from your group’s API administrator.
 API administrators can manage their API keys at research.domaintools.com, selecting the drop-down account menu and choosing API admin.
 
 ## Setup
 
 For step-by-step instructions on how to set up an integration, see the Getting started guide.
 
-### Newly Observed Domains (NOD) Feed 
+### Newly Observed Domains (NOD) Feed
 
 The `nod_feed` data stream provides events from [DomainTools Newly Observed Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).
-This data is collected via the [DomainTools Real Time Feeds API](https://docs.domaintools.com/feeds/realtime/).
+This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/).
 
 #### Example
 
 {{event "nod_feed"}}
 
 {{fields "nod_feed"}}
 
+### Newly Active Domains (NAD) Feed
+
+The `nod_feed` data stream provides events from [DomainTools Newly Active Domains Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).
+This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/).
+
+#### Example
+
+{{event "nad_feed"}}
+
+{{fields "nad_feed"}}
+
+### Domain Discovery Feed
+
+The `domaindiscovery feed` data stream provides events from [DomainTools Domain Discovery Feed](https://www.domaintools.com/products/threat-intelligence-feeds/).
+This data is collected via the [DomainTools Feeds API](https://docs.domaintools.com/feeds/realtime/).
+
+#### Example
+
+{{event "domaindiscovery"}}
+
+{{fields "domaindiscovery"}}

packages/ti_domaintools/_dev/deploy/docker/files/config.yml

@@ -6,3 +6,17 @@ rules:
         body: |-
           {"timestamp":"2025-01-11T08:42:46Z","domain":"test1.com"}
           {"timestamp":"2025-01-11T08:42:46Z","domain":"test2.com"}
+  - path: /v1/feed/nad/
+    methods: [GET]
+    responses:
+      - status_code: 200
+        body: |-
+          {"timestamp":"2025-01-11T08:42:46Z","domain":"test3.com"}
+          {"timestamp":"2025-01-11T08:42:46Z","domain":"test4.com"}
+  - path: /v1/feed/domaindiscovery/
+    methods: [GET]
+    responses:
+      - status_code: 200
+        body: |-
+          {"timestamp":"2025-01-11T08:42:46Z","domain":"test5.com"}
+          {"timestamp":"2025-01-11T08:42:46Z","domain":"test6.com"}

packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json

@@ -0,0 +1,13 @@
+{
+    "events": [
+        {
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}"
+        },
+        {
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}"
+        },
+        {
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}"
+        }
+    ]
+}

packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/pipeline/test-event.json-expected.json

@@ -0,0 +1,94 @@
+{
+    "expected": [
+        {
+            "domaintools": {
+                "domain": "ccnitsolution.com",
+                "timestamp": "2025-01-11T08:42:46Z"
+            },
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "threat"
+                ],
+                "kind": "enrichment",
+                "type": [
+                    "indicator"
+                ]
+            },
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution.com\"}",
+            "threat": {
+                "feed": {
+                    "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.",
+                    "name": "DomainTools domaindiscovery",
+                    "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/"
+                },
+                "indicator": {
+                    "name": "ccnitsolution.com",
+                    "type": "domain-name"
+                }
+            }
+        },
+        {
+            "domaintools": {
+                "domain": "ccnitsolution2.com",
+                "timestamp": "2025-01-11T08:42:46Z"
+            },
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "threat"
+                ],
+                "kind": "enrichment",
+                "type": [
+                    "indicator"
+                ]
+            },
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution2.com\"}",
+            "threat": {
+                "feed": {
+                    "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.",
+                    "name": "DomainTools domaindiscovery",
+                    "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/"
+                },
+                "indicator": {
+                    "name": "ccnitsolution2.com",
+                    "type": "domain-name"
+                }
+            }
+        },
+        {
+            "domaintools": {
+                "domain": "ccnitsolution3.com",
+                "timestamp": "2025-01-11T08:42:46Z"
+            },
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "threat"
+                ],
+                "kind": "enrichment",
+                "type": [
+                    "indicator"
+                ]
+            },
+            "message": "{\"timestamp\":\"2025-01-11T08:42:46Z\",\"domain\":\"ccnitsolution3.com\"}",
+            "threat": {
+                "feed": {
+                    "description": "Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.",
+                    "name": "DomainTools domaindiscovery",
+                    "reference": "https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/"
+                },
+                "indicator": {
+                    "name": "ccnitsolution3.com",
+                    "type": "domain-name"
+                }
+            }
+        }
+    ]
+}

packages/ti_domaintools/data_stream/domaindiscovery/_dev/test/system/test-default-config.yml

@@ -0,0 +1,11 @@
+input: cel
+service: domaintools
+vars:
+data_stream:
+  vars:
+    api_url: http://{{Hostname}}:{{Port}}/v1
+    interval: 10m
+    api_username: xxx
+    api_key: xxx
+assert:
+  hit_count: 2

packages/ti_domaintools/data_stream/domaindiscovery/agent/stream/cel.yml.hbs

@@ -0,0 +1,63 @@
+config_version: "2"
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+resource.url: {{api_url}}
+state:
+  api_username: {{api_username}}
+  api_key: {{api_key}}
+  session_id: {{session_id}}
+  app_name: elastic_feeds
+  app_partner: elastic
+  app_version: 0.1.0
+redact:
+  fields:
+    - api_key
+program: |
+    state.with(
+        request(
+            "GET",
+            state.url.trim_right("/") + "/feed/domaindiscovery/?" + {
+            "api_username": [state.api_username],
+            "api_key": [state.api_key],
+            "sessionID": [state.session_id],
+            "app_name": [state.app_name],
+            "app_partner": [state.app_partner],
+            "app_version": [state.app_version],
+        }.format_query()
+        ).with(
+            {
+                "Header": {
+                    "Accept": ["application/x-ndjson"],
+                },
+            }
+        ).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ?
+            {
+                "events": string(resp.Body).split("\n").filter(x,x!="").map(e,
+                    {
+                        "message": e,
+                    }
+                ),
+                "want_more": resp.StatusCode == 206
+            }
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET:" +
+                        (
+                            (size(resp.Body) != 0) ?
+                                string(resp.Body)
+                            :
+                                string(resp.Status) + " (" + string(resp.StatusCode) + ")"
+                        ),
+                    },
+                },
+                "want_more": false,
+            }
+        )
+    )

packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ilm/default_policy.json

@@ -0,0 +1,23 @@
+{
+    "policy": {
+        "phases": {
+            "hot": {
+                "actions": {
+                    "rollover": {
+                        "max_age": "2d",
+                        "max_size": "50gb"
+                    },
+                    "set_priority": {
+                        "priority": 100
+                    }
+                }
+            },
+            "delete": {
+                "min_age": "3d",
+                "actions": {
+                    "delete": {}
+                }
+            }
+        }
+    }
+}

packages/ti_domaintools/data_stream/domaindiscovery/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,59 @@
+---
+description: Pipeline for processing domaindiscovery feed
+processors:
+- json:
+    if: ctx?.message != null
+    field: message
+    target_field: domaintools
+- set:
+    field: domaintools.feed
+    value: 'domaindiscovery'
+
+  ############################
+  # Generic indicator fields #
+  ############################
+
+- set:
+    field: threat.indicator.type
+    value: domain-name
+- set:
+    if: ctx.domaintools != null && ctx.domaintools?.domain != null
+    field: threat.indicator.name
+    copy_from: domaintools.domain
+
+  ######################
+  # Threat feed fields #
+  ######################
+
+- set:
+    field: threat.feed.description
+    value: 'Apex-level domains (e.g. example.com but not www.example.com) that we observe for the first time, and have not observed previously with our global DNS sensor network.'
+- set:
+    field: threat.feed.name
+    value: 'DomainTools domaindiscovery'
+- set:
+    field: threat.feed.reference
+    value: https://docs.techdocs.ci.domaintools.cloud/feeds/realtime/userguide/
+
+  ####################
+  # Event ECS fields #
+  ####################
+
+- set:
+    field: ecs.version
+    value: '8.11.0'
+- set:
+    field: event.kind
+    value: enrichment
+- set:
+    field: event.category
+    value: ['threat']
+- set:
+    field: event.type
+    value: ['indicator']
+
+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'
+

packages/ti_domaintools/data_stream/domaindiscovery/fields/base-fields.yml

@@ -0,0 +1,8 @@
+- name: data_stream.type
+  external: ecs
+- name: data_stream.dataset
+  external: ecs
+- name: data_stream.namespace
+  external: ecs
+- name: "@timestamp"
+  external: ecs