Error while matching wildcards on fields without keyword subfield

[abulhol]

Using:
Elasticsearch 8.16.1
pysigma-pipeline-sysmon==1.0.4
Winlogbeat 8.16.1 on Windows 10

I converted all the rules from sigma/rules/windows in the SigmaHQ repository to EQL like so:

    pipeline = sigma.pipelines.elasticsearch.ecs_windows()
    backend = es.EqlBackend(pipeline)

   for root, dirs, files in os.walk(args.indir):
        for filename in files:
            if filename.endswith(".yml"):                
                with open(os.path.join(root, filename), 'r') as f:
                    try: 
                        rule = sigma.rule.SigmaRule.from_yaml(f)
                        eql_query = backend.convert_rule(rule)
                        with open(os.path.join(args.outdir, filename.rsplit('.', 1)[0]), 'w') as o:
                            o.write(json.dumps({
                            "query" : eql_query[0]
                            })) 

Now when I run the 2277 resulting rules against my Elasticsearch index (coming from Winlogbeat on Windows 10), a total of 787 rules fail.
A frequent error I get for these is:
[:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead
For example for:

• posh_ps_xml_iex.yml
• posh_ps_invoke_obfuscation_via_use_rundll32.yml

Am I doing something wrong in the conversion code?

[abulhol]

For references, this is the query that the converter produces and which I try to run against the ES index:

{
  "query": "any where (powershell.file.script_block_text:\"*New-Object*\" and powershell.file.script_block_text:\"*System.Xml.XmlDocument*\" and powershell.file.script_block_text:\"*.Load*\") and (powershell.file.script_block_text like~ (\"*IEX *\", \"*Invoke-Expression *\", \"*Invoke-Command *\", \"*ICM -*\"))"
}

[abulhol]

I investigated a bit further:
The issue above seems to be that powershell.file.script_block_text is of type text but has no keyword field (also according to the ES mapping).
The general issue seems to be that field types between the index generated by Winlogbeat do not work with the EQL queries. Here is another example:
second argument of [:] must be [string], found value [4697] type [integer]

[thomaspatzke]

It appears that the usage of the like operator also changes the semantics of the query in a way that it can't search for the value anymore because like relies on keyword typed fields while this is a text typed field. : works on text fields. Waiting for confirmation of this assumption.

The general issue seems to be that field types between the index generated by Winlogbeat do not work with the EQL queries. Here is another example:
second argument of [:] must be [string], found value [4697] type [integer]

Seems that the backend needs to distinguish between string and numeric values to choose operators because according to EQL docs : can only compare strings. This could be implemented in a convert_condition_field_eq_val_num method. Will take care of this after first is confirmed to fix both at once.

[Mat0vu]

Good morning,

there is a similar issue regarding the field mapping of keyword fields which contain numerical data (e.g. event.code) using the ESQL backend:
Searching for

detection:
  condition: selection
  selection:
    EventID:
      # - 1
      - 4688

converted by sigma convert rules/test-rule.yaml -t esql -p ecs_windows results in ... | where event.code==4688, which causes the error
first argument of [event.code==4688] is [keyword] so second argument must also be [keyword] but was [integer] because event.code is mapped as a keyword field in ecs.

Currently I´ve solved this issue by the following pipeline:

transformations:
- id: change_field_mapping_to_str
  type: convert_type
  target_type: str
  field_name_conditions:
    - type: include_fields
      fields: 
        - event.id
        - event.code

Applying this pipeline converts the rule into ... | where event.code=="4688" which runs without problems

Since most users probably except that as soon as they apply the ecs_windows pipeline, the query should work without the need of an extra pipeline in an environment with ecs, maybe it would be better to solve this problem also directly in the code?

[thomaspatzke]

Separated EventID string/integer type confusion and renamed original issue. Both are separate issues.