KeyError: 'AlexNet' - CIFAR (Corruptions)

[OehriSven]

Problem:
When running the corruptions benchmark on either of the CIFAR datasets I encounter the following error:

Traceback (most recent call last):
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 191, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 234, in pandas._libs.index.IndexEngine._get_loc_duplicates
  File "index.pyx", line 242, in pandas._libs.index.IndexEngine._maybe_get_bool_indexer
  File "index.pyx", line 134, in pandas._libs.index._unpack_bool_indexer
KeyError: 'AlexNet'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/labor/Schreibtisch/robustbench/claim.py", line 18, in <module>
    clean_acc, robust_acc = benchmark(
  File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 130, in benchmark
    adv_accuracy, adv_mce = corruptions_evaluation(
  File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 211, in corruptions_evaluation
    alexnet_accs_corruption = full_results.loc['AlexNet']
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/generic.py", line 4301, in xs
    loc = index.get_loc(key)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'AlexNet'

This is a result of this commit where adv_mce was introduced for ImageNet in eval.py. Due to the missing results for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions benchmark a KeyError is raised.

Possible Solution:
As suggested by Hendrycks et al. themselves "any baseline will do (even a baseline with 100% error rates, corresponding to an average of CEs)" for calculating the mCE. Hence, the easiest solution would be to insert a dummy entry for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions with 0.0 for each corruption and severity. The resulting adv_mce then simply corresponds to 1 - adv_accuracy.