After a while, Scrapyd stops picking up the maximum number of jobs for processing defined in "max_proc_per_cpu"

[aaronm137]

My Ubuntu server has 4-core CPU and 8GB RAM. In scrapyd.conf, I set the following:

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
items_dir   =
jobs_to_keep = 500
dbs_dir     = dbs
max_proc    = 60
#max_proc_per_cpu = 8
max_proc_per_cpu = 12
finished_to_keep = 100
#poll_interval = 5.0 # default
poll_interval = 1.0
bind_address = 0.0.0.0
http_port   = 6800
debug       = off
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher
ebroot     = scrapyd.website.Root

[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus

The max_proc_per_cpu is set as 12, the server has 4 cores => in theory, it looks like up to 48 processes should be running on the CPU at the same time. The reason why I have a server of this configuration is that I wanted to run as many spiders on the server at the same time as possible.

However, when I checked Scrapyd, I noticed that typically, there's less than 15 jobs running at the same time. After some additional observation, I noticed that if I restart the server, the number of processes jobs is in the range 45-60 (60 is the max_proc), but after some time (after 60-90 minutes), the number of processed jobs drops from 45-60 to like 10-15.

In htop, I checked the usage of the server:

- Core 0: 100%
- Core 1: 100%
- Core 2: 99%
- Core 3: 100%
- Mem: 4.05GB/7.75GB
- Swp: 1.15GB/4.00GB

I thought that the reason why there's running only 10-15 jobs and not 48 is that the server might be running out of memory, but that does not seem to be the case.
The CPUs are used to their full extend - that's what I wanted, to fully utilize it.

If I restart scrapyd, then for the next hour or so there will be again 48-60 jobs running at the same time and after another hour, it will drop again to 10-15.

Why is that? I would like the server to be ideally processing 48 jobs at any given time. Why is that not happening? What am I missing here?

[jpmckinney]

What's your Scrapyd version?

[jpmckinney]

Also, max_proc_per_cpu has no effect if max_proc is set. You can try max_proc = 0, which causes max_proc_per_cpu to be respected.

[aaronm137]

The Scrapyd version is Version: 1.4.3. Testing it now with

max_proc    = 0
max_proc_per_cpu = 12

and will post here the progress.

[jpmckinney]

Okay, you should also try with version 1.5.0 ideally.

[jpmckinney]

• Core 0: 100%
• Core 1: 100%
• Core 2: 99%
• Core 3: 100%

Your server is already at full utilization. Getting Scrapyd to run more Scrapy processes running won't make anything go faster. Your options are:

• Check your Scrapy spiders so that they use less CPU
• Add more CPUs

[aaronm137]

Alright, so after doing some experiments and observation, I noticed that if I have dozens of thousands of jobs in the queue, then after some time Scrapyd starts using less and less jobs per CPU core. In other words, if max_proc_per_cpu=12, then over time (in 4+ hours processing the jobs), the maximum number of jobs per CPU core will gradually start decreasing from 12 to 1. After 4 hours of processing, the max processes per job is (in my case), say, 9 jobs. After another 2 hours is only 6, and so on.

What helps in this case is to restart the Systemd service. And then again.

I tested turning off max_proc_per_cpu and using max_proc instead. Interestingly, using max_proc seems to solve this issue. If my original setup was max_proc_per_cpu=12 and now I removed max_proc_per_cpu and used max_proc=48, then the number of simultaneously processed jobs is in the range of 45-48.

Why is that happening - I don't know.

[jpmckinney]

Strange, the code treats both settings the same way. The only possible difference is that multiprocessing.cpu_count() reports a different number at different times. My only guess is that in a virtualized environment, the host might change the number of CPUs that the process can detect via the OS – but that seems kind of far-fetched.

scrapyd/scrapyd/launcher.py

Lines 76 to 85 in bf14898

	def _get_max_proc(self, config):
	max_proc = config.getint("max_proc", 0)
	if max_proc:
	return max_proc
	try:
	cpus = multiprocessing.cpu_count()
	except NotImplementedError: # Windows 17520a3
	cpus = 1
	return cpus * config.getint("max_proc_per_cpu", 4)

[jpmckinney]

FYI, a second user has experienced this, so I wrote up some instructions to help debug in #543.