Undergraduate Research Request: Access to Baker2022 Frankenstein Benchmark Data

[Mahi-mond]

Background
I am an undergraduate student conducting research on comparing visual model performance, with a particular focus on the Frankenstein model. I'm investigating the discrepancies between results from the Brain-Score website (benchmark #200) and my local implementation of model evaluation.
Issue Description
When attempting to run local evaluations using the Brain-Score package, specifically targeting the Baker2022frankenstein-accuracy_delta and Baker2022fragmented-accuracy_delta benchmarks, I consistently encounter access restrictions:
python

Error with Frankenstein benchmark: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')]

I've tried:

Setting anonymous AWS credentials
Using different benchmark identifiers
Attempting to access the stimulus sets directly
Following the package data tutorial

Request
As part of my undergraduate research project, I would greatly appreciate:

Access to the Baker2022 Frankenstein benchmark data for local evaluation
Information about the specific methodology used for benchmark #200
Guidance on how to properly compare the Brain-Score results with my local implementation

Research Context
I'm specifically interested in understanding why there are different results between the Brain-Score evaluation and my local implementation. This comparison is important for my research on model evaluation methodologies.
Additional Information

I'm using brainscore_vision version 2.2.12
I understand that some benchmarks are intentionally restricted, but having access to this specific benchmark would significantly enhance my research

Thank you for considering my request. I'm happy to provide any additional information about my research if needed.

[KartikP]

Hi @Mahi-mond! I can try to help you. It's certainly weird that you're not able to access the Baker2022 benchmarks as they should be set the public. I just tried running a model on Baker2022frankenstein-accuracy_delta without any AWS credentials and was able to score the model.

Would it be possible to share code that you are using that is attempting to load the benchmark?

Once we resolve the ability to download the benchmark, we can explore the BehaviorAssembly to see if the data you are using is the same as what is provided in the benchmark and then assess how each of us are computing the appropriate metrics.

The Baker2022 benchmark implementations can be seen here.

[Mahi-mond]

Hey @KartikP , where can I find this Baker2022frankenstein-accuracy_delta ?

[Mahi-mond]

I tried to use this codes here https://github.com/brain-score/vision/tree/master/examples score and models @KartikP

[Mahi-mond]

@KartikP I tried first to run this code You can run an existing model on an existing benchmark by calling a single function, e.g. here alexnet on the behavioral benchmark Rajalingham2018-i2n. from brainscore_vision import score

model_score = score(model_identifier='alexnet', benchmark_identifier='Rajalingham2018-i2n')
print(model_score) and then got this error NoCredentialsError Traceback (most recent call last)
File c:\Users\Mahkameh\anaconda3\Lib\site-packages\brainio\fetch.py:81, in BotoFetcher.download_boto(self)
80 self._logger.debug("attempting default download (signed)")
---> 81 self.download_boto_config(config=None)
82 except Exception as e_signed: # try without authentication

File c:\Users\Mahkameh\anaconda3\Lib\site-packages\brainio\fetch.py:97, in BotoFetcher.download_boto_config(self, config)
96 # show progress. see https://gist.github.com/wy193777/e7607d12fad13459e8992d4f69b53586
---> 97 with tqdm(total=obj.content_length, unit='B', unit_scale=True,
98 desc=self.bucketname + "/" + self.relative_path) as progress_bar:
99 def progress_hook(bytes_amount):

File c:\Users\Mahkameh\anaconda3\Lib\site-packages\boto3\resources\factory.py:381, in ResourceFactory._create_autoload_property..property_loader(self)
380 if hasattr(self, 'load'):
--> 381 self.load()
382 else:

File c:\Users\Mahkameh\anaconda3\Lib\site-packages\boto3\resources\factory.py:565, in ResourceFactory._create_action..do_action(self, *args, **kwargs)
564 def do_action(self, *args, **kwargs):
--> 565 response = action(self, *args, **kwargs)
566 self.meta.data = response

File c:\Users\Mahkameh\anaconda3\Lib\site-packages\boto3\resources\action.py:88, in ServiceAction.call(self, parent, *args, **kwargs)
81 logger.debug(
...
89 # when unsigned download also fails, raise both exceptions
90 # raise Exception instead of specific type to avoid missing init arguments
---> 91 raise Exception([e_signed, e_unsigned])

Exception: [NoCredentialsError('Unable to locate credentials'), ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')]
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[Mahi-mond]

@KartikP and then I changed to this : `import os

os.environ['AWS_ACCESS_KEY_ID'] = 'anonymous'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'anonymous'

from brainscore_vision import score

try:
model_score = score(model_identifier='alexnet', benchmark_identifier='Baker2022frankenstein-accuracy_delta')
print(f"Frankenstein benchmark score: {model_score}")
except Exception as e:
print(f"Error with Frankenstein benchmark: {e}")

try:
model_score = score(model_identifier='alexnet', benchmark_identifier='Baker2022fragmented-accuracy_delta')
print(f"Fragmented benchmark score: {model_score}")
except Exception as e:
print(f"Error with fragmented benchmark: {e}")
` and get this error

Error with Frankenstein benchmark: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')]
Error with fragmented benchmark: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')]

[KartikP]

Thank you for all this. You shouldn't have to set any AWS credentials to access the Baker2022 benchmark/datasets.

I'm noticing that you are running on Windows computer. We've had AWS access issues with non-unix based systems. I've previously replicated this issue on Windows virtual machine as well.

What has helped myself and other users was by using WSL2, which is a Unix subsystem on Windows. Once you have WSL2 installed, you can then install the WSL extension in your IDE of your choice. Then follow the Quickstart tutorial and you should be able to access the benchmarks without issue.