update tracing docs

Signed-off-by: Doru Blânzeanu <[email protected]>

Author: dblnz

docs/hyperlight-metrics-logs-and-traces.md

@@ -98,7 +98,7 @@ Hyperlight provides advanced observability features for guest code running insid
 
 The following features are available for guest tracing:
 - `trace_guest`: Enables tracing for guest code, capturing function calls and execution time.
-- `mem_profile`: Enables memory profiling for guest code with stack uwinding, capturing memory allocations and usage.
+- `mem_profile`: Enables memory profiling for guest code with stack unwinding, capturing memory allocations and usage.
 
 ### Building a Guest with Tracing Support
 
@@ -109,19 +109,57 @@ just build-rust-guests debug trace_guest
 just move-rust-guests debug
 ```
 
-This will build the guest binaries with the `trace_guest` feature enabled and move them to the appropriate location for use by the host.
+This builds the guest binaries with the `trace_guest` feature enabled and move them to the appropriate location for use by the host.
+
+**NOTE**: To enable the tracing in your application you need to use the `trace_guest` feature on the `hyperlight-guest-bin` and `hyperlight-guest` crates.
 
 ### Running a Hyperlight Example with Guest Tracing
 
 Once the guest is built, you can run a Hyperlight example with guest tracing enabled. For example:
 
 ```bash
-cargo run --example hello-world --features trace_guest
+RUST_LOG="info,hyperlight_host::sandbox=info,hyperlight_guest=trace,hyperlight_guest_bin=trace" cargo run --example tracing-otlp --features trace_guest
 ```
 
-This will execute the `hello-world` example, loading the guest with tracing enabled. During execution, trace data will be collected and written to a file in the `trace` directory.
+This will execute the `tracing-otlp` example, loading the guest with tracing enabled.
+During execution, trace data will be collected on the host and exported as `opentelemetry` spans/events.
+
+You can set up a collector to gather all the traces and inspect the traces from both host and guests.
+
+Due to the nature of execution inside a Sandbox, on a call basis, the guest tracing sets up a stack of spans to keep track of the correct parents for the incoming
+guest spans.
+We start with `call-to-guest` which contains all the spans coming from a guest. Additionally, for each exit into the host, we add another layer marking it with
+a `call-to-host` span to follow the execution in the host and correctly set it as a child of the active span in the guest.
+This logic simulates the propagation of `Opentelemetry` context that is usually done between two services, but cannot be done here seamlessly because the guest side
+runs `no_std` which `opentelemetry` doesn't know.
+
+#### How it works
+
+##### Guest
+
+When the guest starts executing the `entrypoint` function, it receives a `max_log_level` parameter that tells the guest what kind of logging level is expected from it.
+
+The `trace_guest` logic takes advantage of this parameter and when the `max_log_level` is `trace`, it allocates a custom made `GuestSubscriber` that implements the `Subscriber`
+trait from `tracing_core` that allows defining a subscriber for the `tracing` crate to handle new spans and events.
+
+This custom subscriber stores the spans and events in a buffer initialized only when tracing is enabled. For each new span and event, a method is called on the custom subscriber which
+not only stores the data, but also keeps track of the hierarchy and dependencies between the other spans/events.
+
+When the storage space is filled, the guest triggers a VM Exit that sends the guest pointers to the host. The host can access the guest memory, get the data and parse it to create the `spans` and `events` using the `opentelemetry` crate which allows specifying the starting and ending timestamps
+which are captured in the guest using the `TSC`.
+
+To improve performance, for each VMExit, the guest adds metadata for the host to be able to report the tracing data and free space.
+
+##### Host
+
+When a guest exits, the host checks for metadata from the guest reporting tracing data.
+If tracing data is found, the host starts parsing it and reconstructing a tree which represents the spans hierarchy.
+
+Additionally, the host also adds new children `span`s to the guest's reported active span, emphasizing the spans created on the host as a result of a temporary VM Exit. This helps visualize a call into the guest with context propagated across the VM boundary.
+
+The host creates `opentelemetry` spans and events for each guest span and event reported.
 
-### Inspecting Guest Trace Files
+### Inspecting Guest memory Trace Files (for mem_profile)
 
 To inspect the trace file generated by the guest, use the `trace_dump` crate. You will need the path to the guest symbols and the trace file. Run the following command: