Compression of repeated datasets

[robjmcgibbon]

For the ExclusiveSphere/Projected apertures there is a lot of repetition once we get to large apertures for small objects (e.g. the 100kpc stellar mass is equal to the 50kp stellar mass for most COLIBRE galaxies). We don't recompute these properties for the larger apertures, we just copy across the value calculated from the smaller aperture. However, this means we are storing lots of redundant data as HDF5 compression doesn't work across multiple datasets.

We could maybe get round this by having a single array for each aperture-property combination, e.g. a 2D array of stellar mass where each column corresponds to a different aperture. This would compress better, and we could maybe still index into it in a way that allows the current output format of the SOAP catalogues to remain unchanged.