ENH: Caching, advanced mapping and separating events for MISP Feed output bot

CHANGELOG.md

@@ -24,6 +24,8 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 - `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
 - `intelmq.lib.mixins.http`: Only set HTTP header 'Authorization' if username or password are set and are not both empty string as they are by default in the Manager (fixes #2590, PR#2634 by Sebastian Wagner).
 - `intelmq.lib.message.Message.from_dict`: Do not modify the dict parameter by adding the `__type` field and raise an error when type is not determinable (PR#2545 by Sebastian Wagner).
+- `intelmq.lib.mixins.cache.CacheMixin` was extended to support temporary storing messages in a cache queue
+  (PR#2509 by Kamil Mankowski).
 
 ### Development
 
@@ -219,7 +221,15 @@ This is short list of the most important known issues. The full list can be retr
   - Treat value `false` for parameter `filter_regex` as false (PR#2499 by Sebastian Wagner).
 
 #### Outputs
-- `intelmq.bots.outputs.misp.output_feed`: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
+- `intelmq.bots.outputs.misp.output_feed`:
+  - Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
+  - Allow saving messages in bulks instead of refreshing the feed immediately (PR#2509 by Kamil Mankowski).
+  - Regenerate only modified events (PR#2509 by Kamil Mankowski).
+  - Add `attribute_mapping` parameter to allow selecting a subset of event attributes as well as additional attribute parameters (PR#2509 by Kamil Mankowski).
+  - Add `grouping_key` parameter to allow keeping IntelMQ events in separated MISP Events based on a given field (PR#2509 by Kamil Mankowski).
+  - Add `tagging` parameter to allow adding tags to MISP events (PR#2509 by Kamil Mankowski).
+  - Add `additional_info` parameter to extend the default description of MISP Events (PR#2509 by Kamil Mankowski).
+  - Add `flat_events` parameter to allow skipping creating objects in MISP Events (PR#2509 by Kamil Mankowski).
 - `intelmq.bots.outputs.smtp_batch.output`: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar).
 
 ### Documentation

docs/dev/bot-development.md

@@ -197,14 +197,47 @@ The `CacheMixin` provides methods to cache values for bots in a Redis database.
 - `redis_cache_ttl: int = 15`
 - `redis_cache_password: Optional[str] = None`
 
-and provides the methods:
+and provides the methods to cache key-value pairs:
 
 - `cache_exists`
 - `cache_get`
 - `cache_set`
 - `cache_flush`
 - `cache_get_redis_instance`
 
+and following methods to cache objects in a queue:
+
+- `cache_lpush`
+- `cache_rpop`
+- `cache_llen`.
+
+Caching key-value pairs and queue caching are two different mechanisms. The functions in the
+ first list are designed for arbitrary values, while the latter ones are primarily for temporarily
+ storing messages, but can also handle other data types. You won't see caches from one in the other.
+ For example, if adding a key-value pair using `cache_set`, it does not change the value from
+ `cache_llen`, and if adding an element using `cache_lpush` you cannot use `check_exists` to look for it.
+
+When using queue-based caching, you have to serialize object to a format accepted by Redis/Valkey
+as the underlying storage. For example, to store a message in a queue using bot ID as key, you can
+use code like:
+
+```python
+self.cache_lpush(self.bot_id, self.receive_message().to_json(jsondict_as_string=True))
+```
+
+and to retrieve a message from the cache:
+
+```python
+data = self.cache_pop()
+if data is None:
+    return # handle empty cache
+message = json.loads(data)
+# to use it as Message object
+message_obj = MessageFactory.from_dict(
+    message, harmonization=self.harmonization, default_type="Event"
+)
+```
+
 ### Pipeline Interactions
 
 We can call three methods related to the pipeline:

docs/user/bots.md

@@ -4736,6 +4736,12 @@ Create a directory layout in the MISP Feed format.
 The PyMISP library >= 2.4.119.1 is required, see
 [REQUIREMENTS.txt](https://github.com/certtools/intelmq/blob/master/intelmq/bots/outputs/misp/REQUIREMENTS.txt).
 
+Note: please test the produced feed before using in production. This bot allows you to do an
+extensive customisation of the MISP feed, including creating multiple events and tags, but it can
+be tricky to configure properly. Misconfiguration can prevent bot from starting or have bad
+consequences for your MISP Instance (e.g. spaming with events). Use `intelmqctl check` command
+to validate your configuration against common mistakes.
+
 **Module:** `intelmq.bots.outputs.misp.output_feed`
 
 **Parameters:**
@@ -4760,6 +4766,144 @@ The PyMISP library >= 2.4.119.1 is required, see
 () The output bot creates one event per each interval, all data in this time frame is part of this event. Default "1
 hour", string.
 
+**`bulk_save_count`**
+
+(optional, int) If set to a non-0 value, the bot won't refresh the MISP feed immediately, but will cache
+incoming messages until the given number of them. Use it if your bot proceeds a high number of messages
+and constant saving to the disk is a problem. Reloading or restarting bot as well as generating
+a new MISP event based on `interval_event` triggers regenerating MISP feed regardless of the cache size.
+
+**`attribute_mapping`**
+
+(optional, dict) If set, allows selecting which IntelMQ event fields are mapped to MISP attributes
+as well as attribute parameters (like e.g. a comment). The expected format is a *dictionary of dictionaries*:
+first-level key represents an IntelMQ field that will be directly translated to a MISP attribute; nested
+dictionary represents additional parameters PyMISP can take when creating an attribute. They can use
+names of other IntelMQ fields (then the value of such field will be used), or static values. If not needed,
+leave empty dict.
+
+For available attribute parameters, refer to the
+[PyMISP documentation](https://pymisp.readthedocs.io/en/latest/_modules/pymisp/mispevent.html#MISPObjectAttribute)
+for the `MISPObjectAttribute`.
+
+For example:
+
+```yaml
+attribute_mapping:
+  source.ip: {}
+  feed.name:
+    comment: event_description.text
+  destination.ip:
+    to_ids: False
+```
+
+would create a MISP object with three attributes `source.ip`, `feed.name` and `destination.ip`
+and set their values as in the IntelMQ event. In addition, the `feed.name` would have a comment
+as given in the `event_description.text` from IntelMQ event, and `destination.ip` would be set
+as not usable for IDS. You can use `type` key to overwrite the attribute type.
+
+**`grouping_key`
+
+(optional, string): If set to a field name from IntelMQ event, the bot will work in parallel on a few
+events instead of saving all incoming messages to a one. Each unique value from the field will
+use its own MISP Event. This is useful if your feed provides data about multiple entities you would
+like to group, for example IPs of C2 servers from different botnets. For a given value, the bot will
+use the same MISP Event as long as it's allowed by the `interval_event`.
+
+**`additional_info`
+
+(optional, string): If set, the generated MISP Event will use it in the `info` field of the event,
+in addition to the standard IntelMQ description with the time frame (you cannot remove it as the bot
+depends of datetimes saved there). If you use `grouping_key`, you may want to use `{key}`
+placeholder which will be then replaced with the value of the grouping key.
+
+For example, the following configuration can be used to create MISP Feed with IPs of C2 servers
+of different botnets, having each botnet in a separated MISP Events with an appropriate description.
+Each MISP Event will contain objects with the `source.ip` field only, and the events' info will look
+like *C2 Servers for botnet-1. IntelMQ event 2024-07-09T14:51:10.825123 - 2024-07-10T14:51:10.825123*
+
+```yaml
+grouping_key: malware.name
+additional_info: C2 Servers for {key}.
+attribute_mapping:
+  source.ip:
+```
+
+**`tagging`
+
+(optional, dict): Allows setting MISP tags to MISP events. The structure is a *dict of list of dicts*.
+The keys refers to which MISP events you want to tag. If you want to tag all of them, use `__all__`.
+If you use `event_separator` and want to add additional tags to some events, use the expected values
+of the separation field. The *list of dicts* defines MISP tags as parameters to create `MISPTag`
+objects from. Each dictionary has to have at least `name`. For all available parameters refer to the
+[PyMISP documentation](https://pymisp.readthedocs.io/en/latest/_modules/pymisp/abstract.html#MISPTag)
+for `MISPTag`.
+
+Note: setting `name` is enough for MISP to match a correct tag from the global collection. You may
+see it lacking the colour in the MISP Feed view, but it will be retriven after importing to your
+instance.
+
+Example 1 - set two tags for every MISP event:
+
+```yaml
+tagging:
+  __all__:
+    - name: tlp:red
+    - name: source:intelmq
+```
+
+Example 2 - create separated events based on `malware.name` and set additional family tag:
+
+```yaml
+event_separator: malware.name
+tagging:
+  __all__:
+    - name: tlp:red
+  njrat:
+    - name: njrat
+```
+
+** `flat_events`
+
+(optional, bool): instead of creating an object for every incoming IntelMQ message, it will add
+attributes directly to the MISP event. Useful if your want to export just a list of data, e.g.
+C2 domains, without having to group some attributes together. When using flat events, you
+have to define custom mapping to ensure the correct attribute types. By default set to `False`.
+
+**Example**
+
+For example, if you have a source that sends C2 domains for multiple malware families,
+you can use the following bot's configuration:
+
+```yaml
+parameters:
+  destination_queues: {}
+  # you have to configure your webserver to expose this path for MISP
+  output_dir: "/var/lib/intelmq/bots/your_feed/"
+  misp_org_name: My Organisation
+  misp_org_uuid: Your-Org-UUID
+  interval_event: 1 day
+  grouping_key: "malware.name"
+  bulk_save_count: 100
+  additional_info: "{key} - "
+  flat_events: true
+  attribute_mapping:
+    source.fqdn:
+      comment: malware.name
+      type: domain
+      category: "Network activity"
+      to_ids: true
+  tagging:
+    __all__:
+      - name: tlp:amber
+```
+
+As a result, you will get MISP feed that creates one event per malware family every day. In the event,
+there will be just C2 domains with the IDS flag set and the malware name as comment. In addition, all
+events will be tagged with `tlp:amber` and also have the malware name in the comment, together with
+the information about the time period. The MISP Feed will be saved to disk after accumulating 100 C2
+domains or on reload/restart.
+
 **Usage in MISP**
 
 Configure the destination directory of this feed as feed in MISP, either as local location, or served via a web server.