Computational cost issue of Task 2 Option 1

[MENG2010]

Copies from Vincent:

Good evening!

My team and I are trying to refine our projects. However, EOT attack is taking too long to generate AEs for even just one attack. How low can we go for the subsampling? Currently, we have it at 0.1.

Looking forward for your response!

[MENG2010]

Your question is very vague.

There are many factors that impact the computational cost, for example, (1) the selection of weak defenses in the ensemble; (2) the number of samples of the distribution; (3) the size of the benign sample (BS) for which you generate AEs; etc.

Therefore, there are multiple ways to reduce computational time. For example, (1) generate AEs targeted on a smaller ensemble (if currently, you built an ensemble upon all the 72 weak defenses, try fewer); (2) sample fewer samples (if the number of samples is 1000 currently, try fewer, say 500); (3) try a smaller dataset (a ratio at 0.1 will give you a subset of 1000 samples, 100 per class. A ratio at 0.05 may still give a good enough approximation).

You can tune in single or multiple dimensions.

[MENG2010]

Copied from Vincent:

If I am not wrong, (2) is this :
data_bs = data_bs[:5]
labels = labels[:5]
part of the code am I right? I will lower the data set to 0.05.

[MENG2010]

We used a tiny dataset, consisting of 5 samples, for the demo. This is for (3). Please remove those two statements when working on your assignment.

The number of samples of the distribution is a tunable parameter of the EOT attack. It is defined by num_samples in the attack configuration json file.
for example,

  "configs1": {
    "attack": "pgd",
    "description": "PGD_eps0.082-Dist_rot[-45,45]_s1000",
    "eps": 0.082,
    "distribution": {
      "num_samples": 1000,
      "transformation": "rotation",
      "min_angle": -45,
      "max_angle": 45
    }
  },