podman-logs fails with --log-driver

[ricardobranco777]

Issue Description

podman logs fails when using --log-driver. The json-file & k8s-file drivers seem to be affected.

We found this issue thanks to the upstream e2e tests in openQA when testing the new conmon v2.2.0 update. The podman logs partial log lines test fails both as root & rootless, with both crun & runc:

• runc root: http://openqa-assets.opensuse.org/tests/5646628/file/podman_e2e-localintegration.txt
• crun root: http://openqa-assets.opensuse.org/tests/5646627/file/podman_e2e-localintegration.txt
• runc rootless: http://openqa-assets.opensuse.org/tests/5646626/file/podman_e2e-localintegration.txt
• crun rootless: http://openqa-assets.opensuse.org/tests/5646608/file/podman_e2e-localintegration.txt

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run --log-driver json-file --name test quay.io/libpod/alpine:latest printf hola
2. podman logs test

Describe the results you received

holaERRO[0000] Getting new log line: '2026-02-03T11:48:54.201516484+01:00 stdout F' is not a valid container log line

Exit status: 0

Describe the results you expected

A simple hola

podman info output

{
  "host": {
    "arch": "amd64",
    "buildahVersion": "1.42.2",
    "cgroupManager": "systemd",
    "cgroupVersion": "v2",
    "cgroupControllers": [
      "pids"
    ],
    "conmon": {
      "package": "conmon-2.2.0-37.1.x86_64",
      "path": "/usr/bin/conmon",
      "version": "conmon version 2.2.0, commit: unknown"
    },
    "cpus": 8,
    "cpuUtilization": {
      "userPercent": 0.06,
      "systemPercent": 0.11,
      "idlePercent": 99.82
    },
    "databaseBackend": "sqlite",
    "distribution": {
      "distribution": "opensuse-tumbleweed",
      "version": "20251223"
    },
    "eventLogger": "journald",
    "freeLocks": 2044,
    "hostname": "jeos",
    "idMappings": {
      "gidmap": [
        {
          "container_id": 0,
          "host_id": 1000,
          "size": 1
        },
        {
          "container_id": 1,
          "host_id": 100000,
          "size": 65536
        }
      ],
      "uidmap": [
        {
          "container_id": 0,
          "host_id": 1000,
          "size": 1
        },
        {
          "container_id": 1,
          "host_id": 100000,
          "size": 65536
        }
      ]
    },
    "kernel": "6.18.2-1-default",
    "logDriver": "journald",
    "memFree": 32866586624,
    "memTotal": 33653137408,
    "networkBackend": "netavark",
    "networkBackendInfo": {
      "backend": "netavark",
      "version": "netavark 1.17.1",
      "package": "netavark-1.17.1-49.2.x86_64",
      "path": "/usr/libexec/podman/netavark",
      "dns": {
        "version": "aardvark-dns 1.17.0",
        "package": "aardvark-dns-1.17.0-38.2.x86_64",
        "path": "/usr/libexec/podman/aardvark-dns"
      }
    },
    "ociRuntime": {
      "name": "runc",
      "package": "runc-1.4.0-1.1.x86_64",
      "path": "/usr/bin/runc",
      "version": "runc version 1.4.0\ncommit: v1.4.0-0-g8bd78a9977e6\nspec: 1.3.0\ngo: go1.25.4\nlibseccomp: 2.6.0"
    },
    "os": "linux",
    "remoteSocket": {
      "path": "/run/user/1000/podman/podman.sock",
      "exists": true
    },
    "rootlessNetworkCmd": "pasta",
    "serviceIsRemote": false,
    "security": {
      "apparmorEnabled": false,
      "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
      "rootless": true,
      "seccompEnabled": true,
      "seccompProfilePath": "/usr/share/containers/seccomp.json",
      "selinuxEnabled": true
    },
    "slirp4netns": {
      "executable": "",
      "package": "",
      "version": ""
    },
    "pasta": {
      "executable": "/usr/bin/pasta",
      "package": "passt-20251215.b40f5cd-1.1.x86_64",
      "version": "pasta 20251215.b40f5cd-1.1\nCopyright Red Hat\nGNU General Public License, version 2 or later\n  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
    },
    "swapFree": 0,
    "swapTotal": 0,
    "uptime": "0h 18m 31.00s",
    "variant": "",
    "linkmode": "dynamic"
  },
  "store": {
    "configFile": "/home/ricardo/.config/containers/storage.conf",
    "containerStore": {
      "number": 4,
      "paused": 0,
      "running": 0,
      "stopped": 4
    },
    "graphDriverName": "overlay",
    "graphOptions": {},
    "graphRoot": "/home/ricardo/.local/share/containers/storage",
    "graphRootAllocated": 25229766656,
    "graphRootUsed": 1874268160,
    "graphStatus": {
      "Backing Filesystem": "btrfs",
      "Native Overlay Diff": "true",
      "Supports d_type": "true",
      "Supports shifting": "false",
      "Supports volatile": "true",
      "Using metacopy": "false"
    },
    "imageCopyTmpDir": "/var/tmp",
    "imageStore": {
      "number": 2
    },
    "runRoot": "/run/user/1000/containers",
    "volumePath": "/home/ricardo/.local/share/containers/storage/volumes",
    "transientStore": false
  },
  "registries": {
    "search": [
      "registry.opensuse.org",
      "registry.suse.com",
      "docker.io"
    ]
  },
  "plugins": {
    "volume": [
      "local"
    ],
    "network": [
      "bridge",
      "macvlan",
      "ipvlan"
    ],
    "log": [
      "k8s-file",
      "none",
      "passthrough",
      "journald"
    ],
    "authorization": null
  },
  "version": {
    "APIVersion": "5.7.1",
    "Version": "5.7.1",
    "GoVersion": "go1.25.6",
    "GitCommit": "",
    "BuiltTime": "Mon Dec 22 08:41:09 2025",
    "Built": 1766389269,
    "OsArch": "linux/amd64",
    "Os": "linux"
  }
}

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

openSUSE Tumbleweed 20260202 on a QEMU VM.

Additional information

No response

[Luap99]

This sounds like a regression from conmon v2.2.0 then.

The json-file & k8s-file drivers seem to be affected.

json-file does not really exists and is just an alias to k8s-file which is defined by kubernetes AFAIK, so we would need to check what the actual spec says about this log line

2026-02-03T11:48:54.201516484+01:00 stdout F is not considered a valid line because it has no content, it is missing a space at the end.

I guess the easiest would be to hold off the conmon update and revert whatever changed this there.
cc @jnovy @lsm5 @jankaluza

[Luap99]

k8s cannot parse such line either AFAICS, https://github.com/kubernetes/kubernetes/blob/bfafa32d90958a8fe7a2ce09ed553fdfef4edd98/staging/src/k8s.io/cri-client/pkg/logs/logs.go#L131-L176

so this must be fixed on the conmon end for sure

[jankaluza]

For the future and FYI: The conmon-v3 github CI will run subset of podman tests which will find out issues like that on PR level. There's a card about this I'm going to work on next sprint.