Hovering around 10% CPU even when idle

[rgaufman]

I'm running:

good_job 4.9.0
rails 8.0.1
postgresql 16.4-1.pgdg24.04+1 (Ubuntu 24.04)

This is my good_job configuration:

config.good_job = {
  preserve_job_records: false,
  cleanup_discarded_jobs: true,
  cleanup_interval_seconds: 3600,  # 1 hour
  cleanup_preserved_jobs_before_seconds_ago: 28800,  # 8 hours
  retry_on_unhandled_error: false,
  on_thread_error: ->(exception) { Airbrake.notify(exception) },
  execution_mode: :external,
  queues: "snapshots:8;cloud_copy:8;filesystem:8;status:8;default:8",
  max_threads: 8,  # Reduce thread count
  max_cache: 10_000,
  poll_interval: 60,  # Increase poll interval
  shutdown_timeout: 30,
  enable_cron: false
}

Even if I just put sleep 300 in my jobs (so that good_job isn't actually doing much), I am still seeing around 10% cpu load per good_job process:

top - 13:39:05 up 9 days,  1:11,  2 users,  load average: 0.37, 0.51, 0.71
Tasks: 252 total,   2 running, 250 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.2 us,  5.5 sy,  8.7 ni, 80.0 id,  1.6 wa,  0.0 hi,  0.9 si,  0.0 st
MiB Mem :   7236.0 total,    383.4 free,   1761.9 used,   5464.3 buff/cache
MiB Swap:   4096.0 total,   4054.6 free,     41.4 used.   5474.1 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
4126406 deployer  39  19  967272 200228  23296 S  11.9   2.7   0:23.78 /usr/local/bin/good_job
4073693 deployer  38  18 1107136 216364  23296 S   6.9   2.9   1:11.57 /data/deployer/tetherbox/tasks/_run.rb

For example with workers like this:

class CameraCheckJob < ApplicationJob
  good_job_control_concurrency_with(total_limit: 1, key: -> { arguments[0] })
  queue_as :default

  def perform(camera_id)
    sleep 120 
  end

My application job is simply:

class ApplicationJob < ActiveJob::Base
  include GoodJob::ActiveJobExtensions::Concurrency
  retry_on ActiveRecord::ConnectionNotEstablished, ActiveRecord::ConnectionFailed, wait: 5.seconds, attempts: 5
end

If I start additional good_job processes, each takes from 7-12% cpu just idling.

This is on an Intel N100 Shuttle MiniPC.

Is there anything in my configuration causing this? - or anything else I can do to minimise this idle overhead?

[bensheldon]

You could use a profiler like Vernier to peek at what the process is doing. GoodJob has several threads so it's possible their wakeups are using the CPU, but I wouldn't expect it to be very much.

Are you able to compare this to your web process when it's idle?

[rgaufman]

Thank you I will try. Another strange symptom I'm getting is after running for a while, I suddenly get very high Postgresql usage too, which goes away when I restart the good_job processes. Also, it's often not showing my in progress jobs in the web interface, but I see in the logs when they finish running (in this case the are just sleeping for 120 seconds before logging).