Data Management - invalid field type har for index pattern c.request-*

[Mato816]

Hi,

we are facing issue with Data Management. We deployed Data Management in Kubernetes cluster. Data Management was successfully deployed, but there is issue with c.request-* index pattern specifically fields har.time and har.wait.
According to logs from Kubernetes pod we observed this issue:

⚙️ Create Rollup Job c.request-log.trends: {"error":{"root_cause":[{"type":"rollup_action_request_validation_exception","reason":"Validation Failed: 1: The field referenced by a metric group must be a [numeric] or [date,date_nanos] type, but found [flattened] for field [har.wait];2: The field referenced by a metric group must be a [numeric] or [date,date_nanos] type, but found [flattened] for field [har.time];"}],"type":"rollup_action_request_validation_exception","reason":"Validation Failed: 1: The field referenced by a metric group must be a [numeric] or [date,date_nanos] type, but found [flattened] for field [har.wait];2: The field referenced by a metric group must be a [numeric] or [date,date_nanos] type, but found [flattened] for field [har.time];"},"status":400}
Enter host password for user '':

⚙️ Run Rollup Job c.request-log.trends: {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Task for Rollup Job [c.request-log.trends] not found"}],"type":"resource_not_found_exception","reason":"Task for Rollup Job [c.request-log.trends] not found"},"status":404}
Enter host password for user '':

When i checked this field har in Kibana it has type flattened.

Can you help us with this issue please?

Thank you in advance.

[rdpanek]

Hi @Mato816 please try to remove these jobs via Kibana. Probably these fields has a different data types.

[Mato816]

Hi @rdpanek ,

when i checked Jobs in Kibana i see these:

Do i have to remove all jobs?

[rdpanek]

And rolluped indices contains some data? Problem with c.request-log.trends is, that dynamic mapping for this index saved for first time numeric type and next round want store the different type for har.wait.

In your c.request-log index probably contains har.wait in more types, right?

[Mato816]

Just c.audit.trends contains some data.

Data in c.request-log are saved in following structure:

{
  "receive": 0.9,
  "wait": 69.499,
  "blocked": 0.192,
  "dns": -1,
  "comment": "",
  "time": 70.697,
  "send": 0.106,
  "ssl": -1,
  "connect": -1
}

[rdpanek]

Hi @Mato816 did removing rollup jobs help?

[Mato816]

Hi @rdpanek
i've deleted Rollup jobs in Kibana and issue still persists.
Now ended all jobs with error code:

⚙️ Create Rollup Job c.audit.trends: {"error":{"root_cause":[{"type":"status_exception","reason":"Cannot create rollup job [c.audit.trends] because job was previously created (existing metadata)."}],"type":"status_exception","reason":"Cannot create rollup job [c.audit.trends] because job was previously created (existing metadata)."},"status":409} Enter host password for user '':

⚙️ Run Rollup Job c.audit.trends: {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Task for Rollup Job [c.audit.trends] not found"}],"type":"resource_not_found_exception","reason":"Task for Rollup Job [c.audit.trends] not found"},"status":404} Enter host password for user '':

[rdpanek]

This error, when using Elasticsearch and a rollup job, indicates that the creation of the rollup job named "[c.audit.trends]" failed because the job has already been created and there are existing metadata for this job.

The error message states that the reason for the failure is that there are existing metadata for this rollup job. It means that a rollup job with the same name has already been created and configured. Elasticsearch does not allow creating duplicate rollup jobs with the same name.

To resolve this error, you can consider the following steps:

Check for the existence of an already created rollup job: Verify if a rollup job with the name "[c.audit.trends]" already exists. If it does, you can use or modify that job instead of creating a new one with the same name.

Rename the rollup job: If you want to create a new rollup job with the same functionality but a different name, you can rename the existing rollup job or create a new one with a distinct name.

Delete the existing rollup job: If you no longer need the existing rollup job, you can delete it using the appropriate command or API for removing rollup jobs in Elasticsearch.

[Mato816]

Hi @rdpanek ,

I've tried all options you mentioned in previous comment and every time I ended up with HAR error mentioned in my first comment.
Do you have any ideas how can I fix it?
Thank you