feature request: `Result` or `ExecMainStatus`

[simonLeary42]

I am having trouble finding a way to make it so that I get an alert whenever a service crashes. When a service crashes and systemd intends to restart it, the service gets the activating state. I don't want to setup an alert for the activating state because then it could trigger on legitimate starts and restarts. Digging through the output from systemctl show, I see that ExecMainStatus shows the exit code of the main process, and Result has the value exit-code after the service has crashed.

ExecMainStatus seems like a good fit for prometheus since it's just one number, but I get the feeling that its value may surprise me in niche edge cases.

Here is the list of possible values of Result:

static const char* const service_result_table[_SERVICE_RESULT_MAX] = {
        [SERVICE_SUCCESS]                 = "success",
        [SERVICE_FAILURE_RESOURCES]       = "resources",
        [SERVICE_FAILURE_PROTOCOL]        = "protocol",
        [SERVICE_FAILURE_TIMEOUT]         = "timeout",
        [SERVICE_FAILURE_EXIT_CODE]       = "exit-code",
        [SERVICE_FAILURE_SIGNAL]          = "signal",
        [SERVICE_FAILURE_CORE_DUMP]       = "core-dump",
        [SERVICE_FAILURE_WATCHDOG]        = "watchdog",
        [SERVICE_FAILURE_START_LIMIT_HIT] = "start-limit-hit",
        [SERVICE_FAILURE_OOM_KILL]        = "oom-kill",
        [SERVICE_SKIP_CONDITION]          = "exec-condition",
};

There are many different result tables for different contexts.

For my own quick-and-dirty exporter, I am thinking of just making a metric called unit_result_success, which is 1 if the result is success and 0 otherwise. Maybe I will create another metric for SERVICE_SKIP_CONDITION if that becomes a problem.

related: #12

[simonLeary42]

Note: I am aware of the restarts_total metric, which would be good for detecting a restart loop, but not for detecting a single crash-and-restart event. If I set an alert at NRestarts >= 1, that would trigger alerts on legitimate restarts (I think).