Rake task and bin tool have different behavior.

[askreet]

Locally (with bundler) when I use the rake task, I get very quick passing catalog generations that seem to use manifests/site.pp when I've specified configtest/site.pp (an auto-generated file that tests more modules).

While troubleshooting (using the pp gem) I determined that calling Puppet.settings.inspect would change the behavior of the rake task to match that of the command-line tool. Yes, I understand how crazy that sounds, and I haven't figured out why this is yet.

I am using Puppet 3.x

[askreet]

It turns out this is simply intermittent and has nothing to do with calling #inspect. Which is good, because that seemed really crazy. Still looking into this behavior skew.

[askreet]

This is related to the parallel processing, Shown here:

[INFO] Using current puppet parser
[INFO] Using puppet 3.7.5
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
/Users/me/path/to/catalogtest/site.pp
[FAILED]  scenarioname (compile time: 3.439781 seconds)
[FAILED]  scenarioname (compile time: 3.456922 seconds)
[FAILED]  scenarioname (compile time: 3.467263 seconds)
/dev/null/manifests/site.pp
[FAILED]  scenarioname (compile time: 3.473072 seconds)
[PASSED]  scenarioname (compile time: 0.033959 seconds)
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.008052 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.034429 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.005679 seconds)
[PASSED]  scenarioname (compile time: 0.014692 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.035811 seconds)
[FAILED]  scenarioname (compile time: 3.499797 seconds)
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.034573 seconds)
[PASSED]  scenarioname (compile time: 0.010124 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.013673 seconds)
[PASSED]  scenarioname (compile time: 0.008605 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[FAILED]  scenarioname (compile time: 3.51593 seconds)
[PASSED]  scenarioname (compile time: 0.00787 seconds)
[PASSED]  scenarioname (compile time: 0.00845 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.013241 seconds)
[PASSED]  scenarioname (compile time: 0.007926 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[FAILED]  scenarioname (compile time: 3.529687 seconds)
[PASSED]  scenarioname (compile time: 0.016549 seconds)
[PASSED]  scenarioname (compile time: 0.012943 seconds)
[FAILED]  scenarioname (compile time: 3.531746 seconds)
[PASSED]  scenarioname (compile time: 0.008211 seconds)
[PASSED]  scenarioname (compile time: 0.008745 seconds)
/dev/null/manifests/site.pp
/dev/null/manifests/site.pp
[PASSED]  scenarioname (compile time: 0.043389 seconds)
[PASSED]  scenarioname (compile time: 0.047225 seconds)

You can see the first 8 (= processor count) have the correct value for Puppet.settings[:manifest], while subsequent generations are very fast and reference an empty catalog. In some runs, all results are that fast, though I haven't been able to reproduce that yet.

When using the CLI tool (instead of the rake task) I am also unable to reproduce. Very odd.

[askreet]

Scratch that last part, the command-line tool consistently does the 8 failures behavior, I have just not seen the zero failures behavior. In all likelihood, I should be seeing all 32 tests fail at the moment.

[askreet]

With the PuppetTest helpers disabled, my tests behave as expected. Probably the goal is to use the test helpers to ensure consistent state. I think the right thing to do is extract all the puppet configuration out of the constructors and into a "configure puppet" method that is called before each scenario.

[jerith]

I think a better solution would to add :isolation => true to the Parallel.map() call in the test runner. That ensures that each test gets run in a separate process and we never inherit Puppet's mutable global state from a previous test.

[jerith]

Oh, that was added to parallel in the 1.8.0 release ten days ago.