test: account for pdb and Prometheus' staleness period

Problem: We recently introduced pod disruption budgets and improved
availability on Prometheus. This is reflected in the up metric for
Prometheus. By virtue of Prometheus staleness mechanism (it will return
samples up to 5m old for a series) the old query can appear to see 4
Prometheus instances.

Solution: Use the last_over_time aggregation with a 2m interval to only
look at the last sample in that interval. This effectively adjusts
Prometheus staleness period to 2m.

Signed-off-by: Jan Fajerski <[email protected]>

Author: jan--f

test/e2e/main_test.go

@@ -84,7 +84,7 @@ func testMain(m *testing.M) error {
 			body []byte
 			v    int
 		)
-		body, loopErr = f.ThanosQuerierClient.PrometheusQuery("count(up{job=\"prometheus-k8s\"})")
+		body, loopErr = f.ThanosQuerierClient.PrometheusQuery("count(last_over_time(up{job=\"prometheus-k8s\"}[2m]))")
 		if loopErr != nil {
 			loopErr = errors.Wrap(loopErr, "error executing prometheus query")
 			return false, nil