Skip to content

Utilize instruments-any in fastapi, kafka, and psycopg2 #3612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

Utilize instruments-any in fastapi, kafka, and psycopg2 #3612

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bf72030
Utilize instruments_either in fastapi, kafka, and psycopg2
jeremydvoss Jul 3, 2025
8e2ece4
changelog entries
jeremydvoss Jul 3, 2025
a9bd19c
Add instruments_either to bootstrap
jeremydvoss Jul 7, 2025
efd2e33
generate bootstrap code
jeremydvoss Jul 7, 2025
d38dd3c
generate readme fix
jeremydvoss Jul 7, 2025
4048c8e
instruments-any and _instruments_any
jeremydvoss Jul 9, 2025
19390ed
sync
jeremydvoss Jul 9, 2025
0890970
uv.lock
jeremydvoss Jul 11, 2025
30e6b48
uv.lock
jeremydvoss Jul 11, 2025
de08183
gen
jeremydvoss Jul 11, 2025
90e9a7b
remove upload dates from uv.lock
jeremydvoss Jul 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `opentelemetry-resource-detector-containerid`: make it more quiet on platforms without cgroups
([#3579](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3579))

### Added

- `opentelemetry-util-http` Added support for redacting specific url query string values and url credentials in instrumentations
([#3508](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3508))
- `opentelemetry-instrumentation-pymongo` `aggregate` and `getMore` capture statements support
([#3601](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3601))
- `opentelemetry-instrumentation-fastapi` Utilize instruments-any functionality. TODO MOVE TO NEW VERSION WHEN OUT
([#3612](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3612))
- `opentelemetry-instrumentation-psycopg2` Utilize instruments-any functionality.
([#3612](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3612))
- `opentelemetry-instrumentation-kafka-python` Utilize instruments-any functionality.
([#3612](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3612))

## Version 1.34.0/0.55b0 (2025-06-04)

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion instrumentation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
| [opentelemetry-instrumentation-django](./opentelemetry-instrumentation-django) | django >= 1.10 | Yes | development
| [opentelemetry-instrumentation-elasticsearch](./opentelemetry-instrumentation-elasticsearch) | elasticsearch >= 6.0 | No | development
| [opentelemetry-instrumentation-falcon](./opentelemetry-instrumentation-falcon) | falcon >= 1.4.1, < 5.0.0 | Yes | migration
| [opentelemetry-instrumentation-fastapi](./opentelemetry-instrumentation-fastapi) | fastapi ~= 0.92 | Yes | migration
| [opentelemetry-instrumentation-fastapi](./opentelemetry-instrumentation-fastapi) | fastapi ~= 0.92,fastapi-slim ~= 0.92 | Yes | migration
| [opentelemetry-instrumentation-flask](./opentelemetry-instrumentation-flask) | flask >= 1.0 | Yes | migration
| [opentelemetry-instrumentation-grpc](./opentelemetry-instrumentation-grpc) | grpcio >= 1.42.0 | No | development
| [opentelemetry-instrumentation-httpx](./opentelemetry-instrumentation-httpx) | httpx >= 0.18.0 | Yes | migration
Expand Down
4 changes: 3 additions & 1 deletion instrumentation/opentelemetry-instrumentation-fastapi/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,10 @@ dependencies = [
]

[project.optional-dependencies]
instruments = [
instruments = []
instruments-any = [
"fastapi ~= 0.92",
"fastapi-slim ~= 0.92",
]

[project.entry-points.opentelemetry_instrumentor]
Expand Down
4 changes: 3 additions & 1 deletion ...pentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
# limitations under the License.


_instruments = ("fastapi ~= 0.92",)
# TODO: update this
_instruments = ()
_instruments_any = ("fastapi ~= 0.92", "fastapi-slim ~= 0.92")

_supports_metrics = True

Expand Down
3 changes: 2 additions & 1 deletion instrumentation/opentelemetry-instrumentation-kafka-python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ dependencies = [
]

[project.optional-dependencies]
instruments = [
instruments = []
instruments-any = [
"kafka-python >= 2.0, < 3.0",
"kafka-python-ng >= 2.0, < 3.0"
]
Expand Down
4 changes: 3 additions & 1 deletion ...telemetry-instrumentation-kafka-python/src/opentelemetry/instrumentation/kafka/package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
# limitations under the License.


# TODO: where are these used?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the python side is how we do the in code dependency checks.

The _instruments* are used by this function here

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-kafka-python/src/opentelemetry/instrumentation/kafka/__init__.py#L108-L126

Which is then loaded here. https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/opentelemetry-instrumentation/src/opentelemetry/instrumentation/instrumentor.py#L88

During instrumentation time.

This change doesn't make sense to me as the logic for "either" is already inside the instrumentation_dependencies function. (https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-kafka-python/src/opentelemetry/instrumentation/kafka/__init__.py#L108-L126)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That get_dependencies method can be overwritten to achieve your desired result, though it's not its intended purpose. The problem is really that overriding it does not solve the scenario where we want to do dependency check before we load the instrumentation. That scenario is fundamental to cloud Auto instrumentation and it was the approach autoinsteumentation used prior to your change. This PR solves the problem from the ground up. Instrumentations no longer need to overwrite methods to provide hacky solutions for narrow scenarios. Now the SDK itself supports either/or scenarios -- whether the check happens before or after loading the instrumentation. In other words, with this change, partial solutions such as those implemented by Kafka and psycopg2 aren't required anymore. However they can still exist. That's why I haven't changed those method overwrites in the pr. This way it shouldn't break existing apps that already adopted that pattern. Let me know if that clears it up. We can sync tomorrow too or discuss in sig.

Copy link
Contributor

@rjduffner rjduffner Jul 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I was mostly reacting to the comment which made it seem like this was still an open question. I do like that we may not need to use the instrumentation_dependencies method anymore but I am worried that its still there at all. Is it possible we could only do the toml method?

I do understand the need for handling more complicated scenarios, (a couple ifs and some ands etc), but in a case like this, the proposed solution in #3610 doesn't work either.

An exact example would be

  • If package a, then package b and c are needed
  • Or if package d then package e and f are needed

This logic gets super complicated super quickly which is why I went towards doing it in python via the instrumentation_dependencies method but I also totally get that we may not need any more complexity than and's and or's and we can maybe refactor more fully if when we ever need this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, i think I understand. Yeah this method works for all current cases, but I see what you mean about possible "branching" scenarios. This method does not have infinite customizability. I agree that a refactor like that would be a much bigger subject. My focus now is fixing all existing scenarios cleanly without the need for custom overwrites or hacks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just a thought for what a long term solution that could support infinite branching might look like: a json-like "instruments" value that includes the logic within and the sdk merely executes.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed that you were responding to my inline comment. I understand it but just forgot to remove my comment lol.

_instruments_kafka_python = "kafka-python >= 2.0, < 3.0"
_instruments_kafka_python_ng = "kafka-python-ng >= 2.0, < 3.0"

_instruments = (_instruments_kafka_python, _instruments_kafka_python_ng)
_instruments = ()
_instruments_any = (_instruments_kafka_python, _instruments_kafka_python_ng)
3 changes: 2 additions & 1 deletion instrumentation/opentelemetry-instrumentation-psycopg2/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ dependencies = [
]

[project.optional-dependencies]
instruments = [
instruments = []
instruments-any = [
"psycopg2 >= 2.7.3.1",
"psycopg2-binary >= 2.7.3.1",
]
Expand Down
5 changes: 4 additions & 1 deletion ...ntelemetry-instrumentation-psycopg2/src/opentelemetry/instrumentation/psycopg2/package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@
# limitations under the License.


# TODO: where are these used?
_instruments_psycopg2 = "psycopg2 >= 2.7.3.1"
_instruments_psycopg2_binary = "psycopg2-binary >= 2.7.3.1"

_instruments = (
# TODO: maybe add _instruments_any
_instruments = ()
_instruments_any = (
_instruments_psycopg2,
_instruments_psycopg2_binary,
)
4 changes: 4 additions & 0 deletions opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,10 @@
"library": "fastapi ~= 0.92",
"instrumentation": "opentelemetry-instrumentation-fastapi==0.57b0.dev",
},
{
"library": "fastapi-slim ~= 0.92",
"instrumentation": "opentelemetry-instrumentation-fastapi==0.57b0.dev",
},
{
"library": "flask >= 1.0",
"instrumentation": "opentelemetry-instrumentation-flask==0.57b0.dev",
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,20 @@ dependencies = [
"opentelemetry-instrumentation-elasticsearch[instruments]",
"opentelemetry-instrumentation-falcon[instruments]",
"opentelemetry-instrumentation-fastapi[instruments]",
"opentelemetry-instrumentation-fastapi[instruments-any]",
"opentelemetry-instrumentation-flask[instruments]",
"opentelemetry-instrumentation-grpc[instruments]",
"opentelemetry-instrumentation-httpx[instruments]",
"opentelemetry-instrumentation-jinja2[instruments]",
"opentelemetry-instrumentation-kafka-python[instruments]",
"opentelemetry-instrumentation-kafka-python[instruments-any]",
"opentelemetry-instrumentation-logging",
"opentelemetry-instrumentation-mysql[instruments]",
"opentelemetry-instrumentation-mysqlclient[instruments]",
"opentelemetry-instrumentation-pika[instruments]",
"opentelemetry-instrumentation-psycopg[instruments]",
"opentelemetry-instrumentation-psycopg2[instruments]",
"opentelemetry-instrumentation-psycopg2[instruments-any]",
"opentelemetry-instrumentation-pymemcache[instruments]",
"opentelemetry-instrumentation-pymongo[instruments]",
"opentelemetry-instrumentation-pymysql[instruments]",
Expand Down
10 changes: 9 additions & 1 deletion scripts/generate_instrumentation_bootstrap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ def main():
pkg_name = pkg.get("name")
if pkg_name in packages_to_exclude:
continue
if not pkg["instruments"]:
if not pkg["instruments"] and not pkg["instruments-any"]:
default_instrumentations.elts.append(ast.Str(pkg["requirement"]))
for target_pkg in pkg["instruments"]:
libraries.elts.append(
Expand All @@ -93,6 +93,14 @@ def main():
values=[ast.Str(target_pkg), ast.Str(pkg["requirement"])],
)
)
# instruments-any is an optional field that can be used instead of or in addition to _instruments. While _instruments is a list of dependencies, all of which are expected by the instrumentation, instruments-any is a list any of which but not all are expected.
for target_pkg in pkg["instruments-any"]:
libraries.elts.append(
ast.Dict(
keys=[ast.Str("library"), ast.Str("instrumentation")],
values=[ast.Str(target_pkg), ast.Str(pkg["requirement"])],
)
)

tree = ast.parse(_source_tmpl)
tree.body[0].value = libraries
Expand Down
13 changes: 9 additions & 4 deletions scripts/generate_instrumentation_readme.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,19 +58,24 @@ def main(base_instrumentation_path):
with open(version_filename, encoding="utf-8") as fh:
exec(fh.read(), pkg_info)

instruments = pkg_info["_instruments"]
instruments_and = pkg_info.get("_instruments", ())
# _instruments_any is an optional field that can be used instead of or in addition to _instruments. While _instruments is a list of dependencies, all of which are expected by the instrumentation, _instruments_any is a list any of which but not all are expected.
instruments_any = pkg_info.get("_instruments_any", ())
supports_metrics = pkg_info.get("_supports_metrics")
semconv_status = pkg_info.get("_semconv_status")
if not instruments:
instruments = (name,)
instruments_all = ()
if not instruments_and and not instruments_any:
instruments_all = (name,)
else:
instruments_all = tuple(instruments_and + instruments_any)

if not semconv_status:
semconv_status = "development"

metric_column = "Yes" if supports_metrics else "No"

table.append(
f"| [{instrumentation}](./{instrumentation}) | {','.join(instruments)} | {metric_column} | {semconv_status}"
f"| [{instrumentation}](./{instrumentation}) | {','.join(instruments_all)} | {metric_column} | {semconv_status}"
)

with open(
Expand Down
11 changes: 8 additions & 3 deletions scripts/otel_packaging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,12 +60,17 @@ def get_instrumentation_packages(
with open(pyproject_toml_path, "rb") as file:
pyproject_toml = tomli.load(file)

optional_dependencies = pyproject_toml["project"][
"optional-dependencies"
]
instruments = optional_dependencies.get("instruments", [])
# instruments-any is an optional field that can be used instead of or in addition to instruments. While instruments is a list of dependencies, all of which are expected by the instrumentation, instruments-any is a list any of which but not all are expected.
instruments_any = optional_dependencies.get("instruments-any", [])
instrumentation = {
"name": pyproject_toml["project"]["name"],
"version": version.strip(),
"instruments": pyproject_toml["project"]["optional-dependencies"][
"instruments"
],
"instruments": instruments,
"instruments-any": instruments_any,
}
if instrumentation["name"] in independent_packages:
specifier = independent_packages[instrumentation["name"]]
Expand Down
47 changes: 33 additions & 14 deletions uv.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.