quickstart tutorial example not working

[mdrichards2]

https://www.brain-score.org/tutorials/models/quickstart

Tried running the quickstart tutorial linked above and couldn't get it to work. Also tried developing a custom script for scoring and ran into errors tracing back to the brainscore_vision source code.

Any insights or suggestions? Thanks!

[KartikP]

https://www.brain-score.org/tutorials/models/quickstart

Tried running the quickstart tutorial linked above and couldn't get it to work. Also tried developing a custom script for scoring and ran into errors tracing back to the brainscore_vision source code.

Any insights or suggestions? Thanks!

Hi @mdrichards2, would you be able to share any additional on the issues you ran into? Specifically, the error and details of your Conda environment.

[mdrichards2]

I work on a compute cluster and use a mamba environment (extension of conda), although I saw that virtual environments are optional?

The model_registry did not contain any registered models. I tried running the example with the model identifier / benchmark identifier provided and then tried some other the other model ids, could not get any to work

[mike-ferguson]

Hi @mdrichards2 - could you share you stack trace/error you are encountering? It will help us better pinpoint what exactly is happening

[mdrichards2]

My cluster uses slurm to manage jobs submissions, here's a snippet of my sbatch file:

---

Activate conda environment

mamba activate .dope_venv

Run the job

python brainscore_vision score --model_identifier='pixels' --benchmark_identifier='MajajHong2015public.IT-pls'

And here's the output from terminal:

---

[maceor22@holylogin05 vision]$ sbatch brain_job.sh
Submitted batch job 2800243
[maceor22@holylogin05 vision]$ tail -f slurm-2800243.out
/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/metrics/init.py:16: FutureWarning: xarray subclass Score should explicitly define slots
class Score(DataAssembly):
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 20, in
fire.Fire()
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 9, in score
result = _score_function(model_identifier, benchmark_identifier, conda_active)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_vision/init.py", line 103, in score
return wrap_score(file,
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/conda_score.py", line 88, in wrap_score
result = score_function(model_identifier, benchmark_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_vision/init.py", line 74, in _run_score
model: BrainModel = load_model(model_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_vision/init.py", line 67, in load_model
return model_registryidentifier
~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'pixels'

[mike-ferguson]

Hi @mdrichards2, sorry for all the trouble.

We sometimes see this happen on compute clusters, and is usually due to a step in the install going wrong, where the Brain-Score registry of models and benchmarks cannot be built properly. I have solved this on various clusters in the past by reinstalling Brain-Score (basically redoing the tutorial), and making sure that the pip install was successful : python -m pip install -e ..

I have also has success in fixing this by redoing the tutorial with python3 instead of python for each of the commands, mostly due to sometimes conflicting source paths on clusters for python itself:

#Create and activate a new conda environment
conda create -y -n myenv python=3.11
conda activate myenv

#Install packages
git clone https://github.com/brain-score/vision.git
cd vision
python3 -m pip install --upgrade pip .
python3 -m pip install -e .

If this does not work, please do not hesitate to reply on this thread- we are happy to help!

[mdrichards2]

Hi! This did work for the pixels model, I was able to get a brainscore. I then tried uploading my own model into the vision/brainscore_vision/models directory under the folder _test (vision/brainscore_vision/models/_test), where all the other models are located. I made the required model.py and init.py files as well. But when I tried scoring I got a similar error about the model_indentifier not being registered. Wondering how this can be fixed so I can score custom models?

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 20, in
fire.Fire()
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 9, in score
result = _score_function(model_identifier, benchmark_identifier, conda_active)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 105, in score
return wrap_score(file,
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/conda_score.py", line 88, in wrap_score
result = score_function(model_identifier, benchmark_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 76, in _run_score
model: BrainModel = load_model(model_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 67, in load_model
import_plugin('brainscore_vision', 'models', identifier)
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/import_plugin.py", line 105, in import_plugin
importer = ImportPlugin(library_root=library_root, plugin_type=plugin_type, identifier=identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/import_plugin.py", line 32, in init
self.plugin_dirname = self.locate_plugin()
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/import_plugin.py", line 59, in locate_plugin
assert plugin_registrations_count > 0, f"No registrations found for {self.identifier}"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: No registrations found for _test

[mdrichards2]

Hi @mike-ferguson @KartikP wondering if you have thoughts on the above ^? Thanks for all your help with this, very excited to use brainscore in my project

[mike-ferguson]

Hi @mdrichards2, again sorry for all the trouble. We are aware of some issues that are similar to this when running on a cluster, and usually have do do with wires being crossed between python versions or other requirements that cause all the dependencies to not install, leading to the registry not being built properly.

Can you share what your __init__.py file looks like, as well as your get_model function from your model.py?

Thanks for your patience as well get to the bottom of this.

[mdrichards2]

Thanks a ton for helping me troubleshoot this. Here's the code of my init.py file in vision/brainscore_vision/models/_test/:

from brainscore_vision.model_helpers.brain_transformation import ModelCommitment
from brainscore_vision import model_registry
from .model import get_layers, get_model

model_registry['_test'] =
lambda: ModelCommitment(identifier='_test', activations_model=get_model('_test'), layers=get_layers('_test'))

and here is the code from my model.py file:

import functools
from brainscore_vision.model_helpers.activations.pytorch import load_preprocess_images
from brainscore_vision.model_helpers.activations.pytorch import PytorchWrapper
from brainscore_vision.model_helpers.check_submission import check_models
from dopamine_perturbation.models import MultilayerPerceptron
import torch
import os

def get_model(name):
"""
This method fetches an instance of the MultilayerPerceptron model.
:param name: the name of the model to fetch
:return: the model instance
"""
assert name == '_test'

# Initialize the MLP with the same parameters as in brain_scoring.py
model = MultilayerPerceptron(
    input_dim=3072,  # 32x32x3 for CIFAR-10
    hidden_dims=[1000, 1000, 1000],
    output_dim=10,
)

# Load the trained weights
model.load_state_dict(
    torch.load(
        os.path.join('results', 'trained_mlp_cifar10_[1000, 1000, 1000]', 'models', 'state_dict.pt'),
        map_location=torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    )
)

# Define preprocessing function
preprocessing = functools.partial(load_preprocess_images, image_size=32)

# Create and return the wrapped model
wrapper = PytorchWrapper(identifier=name, model=model, preprocessing=preprocessing)
wrapper.image_size = 32
return wrapper

def get_layers(name):
"""
Returns the layer names to be used for brain score computation
"""
assert name == '_test'
return [f'layers.{i}' for i in range(3)] + ['final']

def get_bibtex(name):
"""
Returns the bibtex reference of the model
"""
return '''
@misc{your_reference_here,
title={Your Title},
author={Your Name},
year={2024},
}
'''

if name == 'main':
# Use this method to ensure the correctness of the BaseModel implementations.
# It executes a mock run of brain-score benchmarks.
check_models.check_base_models(name)

[mike-ferguson]

@mdrichards2 thanks for this! I would recommend the following:

1. Rename your model (both the directory name and the model_identifier in model.py and __init__.py) something without the leading _. Brain-Score ignores any files starting with a _ in building the registry.
2. If that does not work, after you create your model, rerun python3 -m pip install -e . (note the dot)

Sometimes on clusters the model registry needs to be rebuilt after adding a new model. If this does not work, please let me know!

[mdrichards2]

Deleting the leading underscore and rebuilding the package worked! I was able to score the hidden layers of my test model (all having 1000 neurons) but could not score the final readout layer (10 neurons) and encountered the below error. Is there a way to change the number of components used in the analysis, or should I just avoid scoring the readout layer? Thanks a ton for helping!

Traceback (most recent call last): 0/10 [00:00<?, ?it/s]
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 20, in
fire.Fire()
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 9, in score
result = _score_function(model_identifier, benchmark_identifier, conda_active)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 105, in score
return wrap_score(file,
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/conda_score.py", line 88, in wrap_score
result = score_function(model_identifier, benchmark_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 78, in _run_score
score: Score = benchmark(model)
^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/benchmark_helpers/neural_common.py", line 27, in call
source_assembly = candidate.look_at(stimulus_set, number_of_trials=self._number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 101, in look_at
return self.layer_model.look_at(stimuli, number_of_trials=number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/temporal.py", line 159, in look_at
responses = self._layer_model.look_at(stimuli, number_of_trials=number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 35, in look_at
layers = self.region_layer_map[region]
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 123, in getitem
self.commit_region(region)
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 128, in commit_region
best_layer = self.layer_selection(selection_identifier=region, benchmark=benchmark)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 85, in call
result = self._call(model_identifier=model_identifier, selection_identifier=selection_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 80, in wrapper
result = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 96, in _call
layer_scores = self._layer_scoring(benchmark=benchmark, benchmark_identifier=selection_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 115, in call
return self._call(model_identifier=self.model_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 312, in wrapper
result = function(**reduced_call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 132, in _call
score = benchmark(layer_model)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/utils/init.py", line 80, in call
return self.content(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/benchmark_helpers/neural_common.py", line 30, in call
raw_score = self._similarity_metric(source_assembly, self._assembly)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/metrics/regression_correlation/metric.py", line 25, in call
return self.cross_validation(source, target, apply=self.apply, aggregate=self.aggregate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/metric_helpers/transformations.py", line 40, in call
values = self._run_pipe(*args, apply=apply, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/metric_helpers/transformations.py", line 49, in _run_pipe
y = apply(*vals)
^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/metrics/regression_correlation/metric.py", line 28, in apply
self.regression.fit(source_train, target_train)
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/metric_helpers/xarray_utils.py", line 27, in fit
self._regression.fit(source, target)
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/cross_decomposition/_pls.py", line 719, in fit
super().fit(X, y)
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/base.py", line 1389, in wrapper
return fit_method(estimator, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/cross_decomposition/_pls.py", line 299, in fit
raise ValueError(
ValueError: n_components upper bound is 10. Got 25 instead. Reduce n_components.

[mike-ferguson]

Hi @mdrichards2, apologies for the late reply - most of our team was actually out for a few days last week. In the case of your specific error above, we usually recommend avoiding scoring layers that have less then 10 components, and this is the case for PCA reasons 👍🏼

[mdrichards2]

Another question regarding the PCA... I tried to run a larger model on imagenet and got the below traceback. It seemed like principal components were able to be computed up until the last hidden layer and then failed. How can I avoid this integer overflow issue? Is there a way I can modify the PCA settings, or something else you would suggest?

---

activations: 100%|██████████| 1024/1024 [01:58<00:00, 8.68it/s]
layer packaging: 100%|██████████| 7/7 [00:15<00:00, 2.20s/it]
activations: 0%| | 0/3200 [21:13<?, ?it/s] 9.28it/s]
layers: 0%| | 0/7 [21:36<?, ?it/s]00:01, 1.58s/it]
Traceback (most recent call last):██████▏ | 5/7 [15:54<04:40, 140.48s/it]
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 20, in
fire.Fire()
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/main.py", line 9, in score
result = _score_function(model_identifier, benchmark_identifier, conda_active)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 105, in score
return wrap_score(file,
^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/brainscore_core/plugin_management/conda_score.py", line 88, in wrap_score
result = score_function(model_identifier, benchmark_identifier)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/init.py", line 78, in _run_score
score: Score = benchmark(model)
^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/benchmark_helpers/neural_common.py", line 27, in call
source_assembly = candidate.look_at(stimulus_set, number_of_trials=self._number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 101, in look_at
return self.layer_model.look_at(stimuli, number_of_trials=number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/temporal.py", line 159, in look_at
responses = self._layer_model.look_at(stimuli, number_of_trials=number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 35, in look_at
layers = self.region_layer_map[region]
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 123, in getitem
self.commit_region(region)
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/init.py", line 128, in commit_region
best_layer = self.layer_selection(selection_identifier=region, benchmark=benchmark)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 85, in call
result = self._call(model_identifier=model_identifier, selection_identifier=selection_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 80, in wrapper
result = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 96, in _call
layer_scores = self._layer_scoring(benchmark=benchmark, benchmark_identifier=selection_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 115, in call
return self._call(model_identifier=self.model_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 312, in wrapper
result = function(**reduced_call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 132, in _call
score = benchmark(layer_model)
^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/utils/init.py", line 80, in call
return self.content(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/benchmark_helpers/neural_common.py", line 27, in call
source_assembly = candidate.look_at(stimulus_set, number_of_trials=self._number_of_trials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/brain_transformation/neural.py", line 153, in look_at
self._model(layers=self._layers, stimuli=stimuli)
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/pytorch.py", line 41, in call
return self._extractor(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 61, in call
return function_call(
^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 76, in from_stimulus_set
activations = self.from_paths(stimuli_paths=stimuli_paths, layers=layers, stimuli_identifier=stimuli_identifier,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 104, in from_paths
activations = fnc(layers=layers, stimuli_paths=reduced_paths, require_variance=require_variance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 312, in wrapper
result = function(**reduced_call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 111, in _from_paths_stored
return self._from_paths(layers=layers, stimuli_paths=stimuli_paths, require_variance=require_variance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 117, in _from_paths
layer_activations = self._get_activations_batched(stimuli_paths, layers=layers, batch_size=self._batch_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 185, in _get_activations_batched
batch_activations = hook(batch_activations)
^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/pca.py", line 22, in call
self._ensure_initialized(batch_activations.keys())
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/pca.py", line 38, in _ensure_initialized
layer_pcas = self._pcas(identifier=self._extractor.identifier, layers=missing_layers,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/result_caching/init.py", line 223, in wrapper
result = function(**reduced_call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/pca.py", line 68, in _pcas
layer_pcas = change_dict(imagenet_activations, init_and_progress, keep_name=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 546, in change_dict
results = map_fnc(apply_change, d.items())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/core.py", line 543, in apply_change
values = change_function(values) if not keep_name else change_function(layer, values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/DopaminePerturbation/dopamine_perturbation/vision/brainscore_vision/model_helpers/activations/pca.py", line 64, in init_and_progress
pca.fit(activations)
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/base.py", line 1389, in wrapper
return fit_method(estimator, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/decomposition/_pca.py", line 442, in fit
self._fit(X)
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/decomposition/_pca.py", line 542, in _fit
return self._fit_full(X, n_components, xp, is_array_api_compliant)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/sklearn/decomposition/_pca.py", line 583, in _fit_full
U, S, Vt = linalg.svd(X_centered, full_matrices=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/n/home03/maceor22/.conda/envs/.dope_venv/lib/python3.11/site-packages/scipy/linalg/_decomp_svd.py", line 148, in svd
raise ValueError(f"Indexing a matrix of {sz} elements would "
ValueError: Indexing a matrix of 3211264000 elements would incur an in integer overflow in LAPACK. Try using numpy.linalg.svd instead.
layer principal components: 71%|███████▏ | 5/7 [15:55<06:22, 191.13s/it]

[mdrichards2]

Hi @mike-ferguson , wondering what your thoughts are here? I also tested one of the open models trained on imagenet that came with the brainscore repo and it worked fine, not sure why I'm getting an integer overflow error ^.

[mike-ferguson]

Hi @mdrichards2, sorry again for the late reply. I was out on vacation then immediately got the flu. My guess is that the activation matrix is so large it exceeds the indexing limit of LAPACK’s SVD implementation. You might be able to replace Brain-Score's PCA implementation with numpy's linalg.svd() instead of scipy.linalg.svd() in your local code, but I want to loop in @mschrimpf here to see if he has seen this before/has a quick fix.