Discuss introducing a debug console facility

[jodh-intel]

Background

We have considered adding a debug console in Clear Containers in the past:

• WIP: Debug: Enable VM debug console clearcontainers/runtime#658
• Debug: Provide a way to enable a debug console to the VM clearcontainers/agent#122
• Adding a minimal shell to default rootfs clearcontainers/osbuilder#36

However, discussions stalled for two main reasons:

Security

The concern that introducing such a feature could be an attack vector.

Image size

Adding a shell (even a tiny one) is going to bloat the images slightly and that space isn't going to be used 99.99% of the time.

Rationale

It would be very useful for developers and admins to have the ability to debug a running container from the guest-side root namespace. Note that docker exec is NOT what we want as that is not running in the guest root namespace and is thus constrained.

Image support

We could generate two images - one with a shell and one without. However, the general view is that this is suboptimal since:

• The "debug-enabled" version would probably get less testing.
• If a developer/admin needs to debug a container that is already running, they might not be running the "debug" image which defeats the point of the feature.

Architecture

• runv expects the agent to be running as PID = 1 (init daemon)

  A debug console shell would either need to run as a separate thread or a child process of the agent.

• cc-runtime (virtcontainers) assumes the agent to be running with PID != 1

  A debug console shell can be launched by the init daemon (systemd by default) or could be handled as a separate thread / child of the agent.

From a testing (and security) perspective, it would be safer to have a single code path for a debug console.

Configuration and logging

If we introduce a debug console facility it:

• MUST be disabled by default.
• MUST be possible to query the status of the feature somehow to be assured that it is disabled.
• MUST generate an agent log entry when the feature is enabled to allow an admin to detect this change.
• MUST generate an agent log entry when the feature is used to allow an admin to track usage.

[jodh-intel]

/cc @mcastelino, @sboeuf, @jcvenegas.

[jodh-intel]

/cc @egernst.

[jodh-intel]

Any thoughts on this @gnawux, @bergwolf?

[gnawux]

runV/hyperstart had a method to run a process outside any container. I agree this is useful sometimes.

add @laijs

[sboeuf]

@jodh-intel I think we all agree this is really needed since we want to be able to access a VM to perform some advanced debugging.

I am not sure if we need to maintain an official debug image, instead we could provide the right documentation on how a user can create its own debug image based on the official one with osbuilder.
And also we need the documentation to explain how we should disable the debug logs from the proxy, so that we can actually connect to console.sock from a socat ... command.

Unfortunately, and as you mentioned for security purposes, we should not have a dedicated way to debug from the official image itself.

[grahamwhaley]

If possible, we could add a flag to osbuilder to automatically add and configure the bits to an image - that would make the process of generating a debug image a little easier as well.

[jodh-intel]

@mcastelino can comment further, but to be really useful, an admin would be able to debug any pod. That implies a single image type which either contains some sort of built-in but dormant debug console facility that can be enabled as desired [*], or a way to allow a debug console to be added securely by an admin to the running virtual machine.

Maybe a compromise would be to provide some way to do the following (optionally after some sort of clone operation so the debug fiddling could be performed on a copy of the real workload?):

• pause the pod.
• save the state.
• switch to a new image (same release version, but with debug machinery enabled).
• restore the state.
• enable debug console.
• start debug console.
• (optionally) unpause the pod.

---

[*] - but must be disabled by default when new containers are created.

[sboeuf]

@jodh-intel not sure I understand why we need such a complex functioning ? The user will just need to make sure the proxy(or anything else) is not consuming the dedicated socket, but other than that, it can be joined anytime, no need to pause the VM for that.

[jodh-intel]

Hi @sboeuf - Oh, sorry - I hadn't read you previous comment correctly.

I think the confusion here revolves around perceived requirements. I think we all agree that:

• we want such a facility.
• we require it to be disabled by default.

But what still doesn't seem very clear is whether we have the following requirement:

• we will create a single official image that provides a debug facility which is disabled by default.

Folk with a security hat on will tend to say "no", whilst folks who want to be able to debug running container issues will tend to say "yes".

Back to your corners y'all for a wet sponge and towel... and...

🔔 Round 1! 🔔

[grahamwhaley]

my 2p - maybe you can have a single image if you can have access to the debug port secure enough that it does not pose a major problem during deployment.
One suggestion maybe is to have the port/socket/terminal/whatever secured with a key that is only accessible to root and/or the container infra. For instance, if you expose a socket tty, make it SSH and store the key in the container manager or orchestration side. If that is compromised already then your container is already compromised etc.

[sboeuf]

Having the terminal + ssh into the image will make it bigger, should not be that big though, but might worth to take into account.
We have all our images being created from osbuilder and I think, as mentioned by @grahamwhaley, having a simple flag to generate a debug image (including what's needed) should be easy. 99% of the time, an issue discovered on a specific packaged image should be easily reproduced with an image generated from osbuilder.

Now, I agree that having the debug capability included in every image would be easier because we would never have the problem of not being able to reproduce the issue through a specific debug build.

Maybe the alternative to this would be to look at the problem upside down here, and consider having an image including debug by default, but only when people would need to use Kata in production, they would generate a "production-ready" image from osbuilder. This image would be the current image that we have (no debug included).

[mcastelino]

I am not sure if we need to maintain an official debug image, instead we could provide the right documentation on how a user can create its own debug image based on the official one with osbuilder.
And also we need the documentation to explain how we should disable the debug logs from the proxy, so that we can actually connect to console.sock from a socat ... command.

@sboeuf I do not agree that the user should generate the image. It is not hard for us to generate the image each time we create a release and then the user can simply switch to the debug image by modifying the toml. @egernst and myself have had to support multiple customers and developers to get this working.

Making our users and customer jump through hoops just to be able to debug is not a productive use of their time or ours.

[sboeuf]

@mcastelino I don't have a strong opinion on that, and based on your experience with customers, I trust your opinion about this !

[jodh-intel]

@mcastelino - what's your view though on whether we should provide a single image or two ("standard" and "debug")?