Most processes exit with -1: signal: terminated -> Job Failure

[iamnoah]

Hi, I'm using kubexit to run k8s Jobs with a lot of service containers and it is awesome, but noticed that my jobs frequently end up in the failure state due one or more containers having a non-zero exit. I see a lot of this in the terminate container logs:

Terminating child process...
Tombstone Watch(/graveyard): done
Child Exited(-1): signal: terminated

I'm guessing this is due to those processes not have a SIGTERM handler. In this situation (child was terminated due to a death dependency), I'd much rather they exit with code 0 regardless of the actually exit code or condition of the child process.

Would adding another flag like KUBEXIT_TERM_EXIT_CODE that overrides the child exit code if the child has been terminated be amenable?

[GMartinez-Sisti]

@iamnoah when processes that need to run on docker can't handle signals, it's common to wrap them with an init system. yelp/dumb-init is a common one, there is also krallin/tini that is even supported by docker-cli.

Would this help?

[ngealy]

In my case, I am running a bash script in a sidecar and it was getting terminated with a non-zero exit due to a death dependency.

I tried to use krallin/tini, and unfortunately, it did not help with the issue.

I got it to work by adding a SIGTERM trap. I followed the answer on this post to add it. https://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash/444676?newreg=324d02e3137940d8b3e0826e68296e40

[greg65236592]

IMO kubexit is a command wrapper so it just reflects the exit code of child processes.

How I resolve this issue is writing a script as following:

command: [ 'bash', '-c' ]
# Ignore child process error status
args:
- |
  /kubexit/kubexit /path/to/cmd || true

By using || true error status can be ignored by users choice without changing kubexit's behavior.

Further more, if we want to make sure the allowed exit code is -1 only, just adding some if else statement to check the code, which could be more robust.

See if that helps :)