Update google genai instrumentation to work with latest semantic convention, allow for uploading content.

[argaj]

Description

Accomodation of semantic conventions changes made in open-telemetry/semantic-conventions#2179.

Also, when logging the completion details, the upload hook is called, so if user has configured OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH and OTEL_INSTRUMENTATION_GENAI_UPLOAD_HOOK env vars, completion details will also be logged as refs (if appropriate capture content env var is set).

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Unit tests, manual tests.

Does This PR Require a Core Repo Change?

No?

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: argaj / name: Artur Gajowniczek (9ea2278, 166c081, b3a6efb, e1738c2, ee911f5, 13b6f2e, 17b9c0a, 3ac198d, 8583327, eeed7d5, ae4da64, f1feaf7, 47d5728, 058da13, 057d956, 2eefc39)
• ✅ login: aabmass / name: Aaron Abbott (6276543)

[aabmass]

@DylanRussell should we still do this if there is an upload hook? One practical concern for GCP logging exporter is that if you include the JSON in the log directly and it exceeds the size limit (256kB) the whole log gets dropped. Since the upload hook works on the same log, that would mean everything gets dropped.

Alternatively, maybe we could emit two separate logs to avoid this problem

[DylanRussell]

The user can both configure whether they want the upload and whether they want the content in the log right ? So maybe lets just leave it to the user and recommend one way or the other ? Probably should discuss it more internally..

[aabmass]

Ah thanks I think you're right from semconv

The hook SHOULD operate independently of the opt-in flags that control capturing of
gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages.

I kind of read this the opposite way at first but it makes sense.

[aabmass]

So the code should always call the hook

[argaj]

So, IIUC, the hook operates independently, is called always (potentially a noop), potentially stamps the attributes with refs, and if span/event logging is enabled, that then is logged, yes?

[aabmass]

@DylanRussell can you share the code you did since you had a good look at the specs?

I think we would always create the event and span as well and pass them to the hook. But just not add the content directly to them depending on the environment variable.

[DylanRussell]

The user can both configure whether they want the upload and whether they want the content in the log right ? So maybe lets just leave it to the user and recommend one way or the other ? Probably should discuss it more internally..