Are reference metrics with or without compiler optimisations?

[lxssio]

We've been trying to collect performance metrics on a g3.4xlarge AWS machine and our reference implementation metric is 23.34 for the integral puzzle. This is well above the reported reference metric from the spec and we haven't made any changes to the reference implementation.

[Norbo11]

(Same team as OP)

We are getting the following scores for execution of the reference implementations on AWS:

integral: 23.3
rank: 205.8
ising: 13.9
decompose: 15.9

We can see that decompose and ising seem pretty close to what is reported in the spec, but integral is a lot higher than the reported 14.7, while our rank is completely blown out of proportion than the reported 41.8.

People seemed to experience similar issues in #13. Could we please shed some light on this?

[jjd06]

-O3. Are you timing the whole of puzzle->Execute with scales generated using 20+(i*20+U*5) for i in {1, 2, 3, ...} with U in [0,1)?. There will be some noise due to the randomness in the scale and data generation, but it shouldn't make that big a difference to the performance that you see.

[Norbo11]

Big oops: the ^ operator in Python is an XOR, not an exponentiation operator >.>

Despite this, rank is now scoring 23.0, which is now less than the reported 41.8, which I think is what people in #13 were also saying.