[postgres] Postgres logs pipeline misses some queries

[mathroc]

I have configured postgres and the datadog agent to collect logs from postgres into Datadog Logs (beta)

some query are correctly identified (db.operation and db.statement are ok) but some not. here is an exemple of a log that does not get correctly grok-ed:

2018-02-16 20:28:37.365 UTC [5768] docker app:auth app 10.0.1.250 5a873ef5.1688 LOG:  execute <unnamed>: 
	      select id
	      from public.user
	      where id = $1

neither the operation nor the statement is identifier

(coming from DataDog/integrations-core#1129)

[NBParis]

Hello, @mathroc

Thanks for sharing this example. We will look into this and see where the integration pipeline is failing to make sure those are supported as well.

Thanks,
Nils

[NBParis]

Hello again @mathroc ,

I had a look and here is the reason why the statement is not correctly extracted.

The Postgresql integration pipeline expects the word statement in the log to know there is a statement to extract as in the below example:

2017-11-08 18:11:35.727 UTC [5237] postgres psql postgres [local] 5a0348cd.1475 LOG:  statement: SELECT * FROM playground;

However in your log you have execute <unnamed> instead which is why it goes into the fallback parsing and set the rest as the message instead of the statement.

Is it something that you see a lot in your logs? Like a regular pattern that should be included in the parsing rule?
If it was the case you could clone the integration pipeline and change the parsing rule to:

suggested_format %{_prefix} %{_severity}:\s+(%{regex("statement:|execute <unnamed>:")}\s+%{_raw_query}|%{data:msg})

Regarding the operation attribute, once you have the statement attribute, it will automatically be extracted from there by the next parser of the integration pipeline.