opt: lazily calculate sparsity for all ops (including diffsl)

[martinjrobins]

this is done upfront, even if its not required