Is the offering logging misleading?

[hansbogert]

Hi,

I can't get my cluster of 80 CPUs and 200GB+ of mem to allocate the last 8.5 CPUs.
In the logging I can see this repeatedly:

15/03/09 11:56:12 INFO mapred.ResourcePolicy: Declining offer with insufficient resources for a TaskTracker:
  cpus: offered 0.8499999940395355 needed at least 0.15000000596046448
  mem : offered 20182.0 needed at least 368.0
  disk: offered 1859053.0 needed at least 0.0
  ports:  at least 2 (sufficient)

I'm not sure why the hadoop/mesos is declining, every resource demand has been met.

[tarnfeld]

Have you seen #26? Maybe that's related?

[hansbogert]

Ahh this explains:

hadoop/src/main/java/org/apache/hadoop/mapred/ResourcePolicyVariable.java

Line 23 in eef6c53

// Is this offer too small for even the minimum slots?

Maybe we should add some log output, to make this a bit more clear.

For people experiencing the same, the 'problem' is that the mesos container running the tasktracker is probably given some value by property, 'mapred.mesos.tasktracker.cpus', in my case 0.15. So in my case 7 CPUs (per node) are completely taken by slots, plus 0.15 for the TT. Then Mesos tries to handout the remaining 0.85 CPU resources, but

hadoop/src/main/java/org/apache/hadoop/mapred/ResourcePolicyVariable.java

Line 23 in eef6c53

// Is this offer too small for even the minimum slots?

returns false to

hadoop/src/main/java/org/apache/hadoop/mapred/ResourcePolicy.java

Line 277 in eef6c53

final boolean sufficient = computeSlots();

and only then logs the output as in the this issue's start post. Though the logging does not tell it accurately.

[ashwanthkumar]

I have a similar issue

2015-08-17 06:59:14,377 INFO org.apache.hadoop.mapred.ResourcePolicy: Declining offer with insufficient resources for a TaskTracker: 
  cpus: offered 5.0 needed at least 1.0
  mem : offered 1680.0 needed at least 1024.0
  disk: offered 15038.0 needed at least 1024.0
  ports:  at least 2 (sufficient)

I don't have maximum map / reduce tasks as 0. What else could be an issue?