Faild to read Simba snapshot

[z20230220]

Hello Development Team! I am using Caesar to read the Simba snapshot from the s50: 100 Mpc/h box, 2x1024^3 particles, but I am encountering a ValueError, as shown below:

The entire code is quite short, and I would appreciate your guidance on what might be causing this issue. Thank you very much for your assistance!

The text mode code and outputs are placed in the next block.

[z20230220]

import yt
import caesar

# first we load your snapshot into yt
ds = yt.load('/data/Simba/snapshot/snap_m100n1024_151.hdf5')

# now we create a CAESAR object, and pass along the yt dataset
obj = caesar.CAESAR(ds)

# now we execute member_search(), which identifies halos, galaxies, and computes
# properties (including photometry; this requires installing FSPS) on 16 OpenMP cores
obj.member_search(haloid='snap', fof6d_file='simba_151_fof6d', nproc=16)

# finally we save the CAESAR galaxy/halo catalog to your desired filename
obj.save('simba_151_caesar_file.hdf5')

======================================

yt : [INFO ] 2024-09-12 14:15:23,119 ComovingIntegrationOn does not exist, falling back to OmegaLambda
yt : [INFO ] 2024-09-12 14:15:23,186 Calculating time from 1.000e+00 to be 4.375e+17 seconds
yt : [INFO ] 2024-09-12 14:15:23,188 Assuming length units are in kpc/h (comoving)
yt : [INFO ] 2024-09-12 14:15:23,296 Parameters: current_time = 4.3748565942961094e+17 s
yt : [INFO ] 2024-09-12 14:15:23,298 Parameters: domain_dimensions = [1 1 1]
yt : [INFO ] 2024-09-12 14:15:23,299 Parameters: domain_left_edge = [0. 0. 0.]
yt : [INFO ] 2024-09-12 14:15:23,300 Parameters: domain_right_edge = [100000. 100000. 100000.]
yt : [INFO ] 2024-09-12 14:15:23,301 Parameters: cosmological_simulation = True
yt : [INFO ] 2024-09-12 14:15:23,302 Parameters: current_redshift = 4.440892098500626e-16
yt : [INFO ] 2024-09-12 14:15:23,303 Parameters: omega_lambda = 0.7
yt : [INFO ] 2024-09-12 14:15:23,303 Parameters: omega_matter = 0.3
yt : [INFO ] 2024-09-12 14:15:23,304 Parameters: omega_radiation = 0.0
yt : [INFO ] 2024-09-12 14:15:23,304 Parameters: hubble_constant = 0.68
Generating md5 hash: 4011422it [39:30, 1692.13it/s]
yt : [INFO ] 2024-09-12 14:54:58,269 Allocating for 2.148e+09 particles
Loading particle index: 100%|██████████| 4096/4096 [00:05<00:00, 761.47it/s]
yt : [INFO ] 2024-09-12 14:55:30,456 member_search() running on 16 cores
The particle types will be loaded: ['PartType0', 'PartType1', 'PartType4', 'PartType5']
yt : [INFO ] 2024-09-12 14:58:26,828 Calculated Omega_Baryon=0.0479616 and 1024^3 effective resolution
yt : [INFO ] 2024-09-12 14:58:27,909 Using FOF Halo ID from snapshots, RAM=0.615 GB
yt : [INFO ] 2024-09-12 15:00:39,271 Initializing member search, loading particles, RAM=16.62 GB
yt : [INFO ] 2024-09-12 15:34:11,781 Loaded particle data, RAM=63.47 GB
/opt/conda/envs/Quench/lib/python3.12/site-packages/pyGadgetReader-2.6-py3.12.egg/readgadget/modules/common.py:98: RuntimeWarning: overflow encountered in cast
/opt/conda/envs/Quench/lib/python3.12/site-packages/pyGadgetReader-2.6-py3.12.egg/readgadget/modules/common.py:98: RuntimeWarning: invalid value encountered in divide
yt : [INFO ] 2024-09-12 15:52:24,988 Loaded baryon data, RAM=71.09 GB
yt : [INFO ] 2024-09-12 16:09:51,140 Found 2227671 valid halos, loaded indexes, RAM=86.76 GB
yt : [INFO ] 2024-09-12 16:11:28,891 Calculating properties for 2227671 halos (nproc=16), RAM=92.6 GB
yt : [INFO ] 2024-09-12 17:16:27,530 Computed properties for 2227671 halos, RAM=246 GB

ValueError Traceback (most recent call last)
Cell In[2], line 9
5 obj = caesar.CAESAR(ds)
7 # now we execute member_search(), which identifies halos, galaxies, and computes
8 # properties (including photometry; this requires installing FSPS) on 16 OpenMP cores
----> 9 obj.member_search(haloid='snap', fof6d_file='simba_151_fof6d', nproc=16)
11 # finally we save the CAESAR galaxy/halo catalog to your desired filename
12 obj.save('simba_151_caesar_file.hdf5')

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/main.py:340, in CAESAR.member_search(self, *args, **kwargs)
338 else:
339 from caesar.fubar_halo import fubar_halo
--> 340 fubar_halo(self)
341 assign.assign_galaxies_to_halos(self)
342 assign.assign_clouds_to_galaxies(self)

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/fubar_halo.py:72, in fubar_halo(obj)
70 mylog.warning('No valid halos found! Aborting member search')
71 return
---> 72 get_group_properties(halos,halos.obj.halo_list) # compute halo properties
73 if not obj.simulation.baryons_present: # if no baryons, we're done
74 return

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/group.py:819, in get_group_properties(self, grp_list)
816 from caesar.group_funcs import get_group_overall_properties,get_group_gas_properties,get_group_star_properties,get_group_bh_properties,get_group_dust_properties
818 get_group_overall_properties(self,grp_list)
--> 819 if 'gas' in self.obj.data_manager.ptypes: get_group_gas_properties(self,grp_list)
820 if 'star' in self.obj.data_manager.ptypes: get_group_star_properties(self,grp_list)
821 if 'dust' in self.obj.data_manager.ptypes: get_group_dust_properties(self,grp_list)

File caesar/group_funcs/group_funcs.pyx:497, in caesar.group_funcs.get_group_gas_properties()

ValueError: Buffer dtype mismatch, expected 'float' but got 'double'

[dnarayanan]

hi - thanks for reporting this! what commit hash are you on?

i'm also wondering if this may be due to a lack of memory. the error is obtuse so i'm not 100% sure, but i just tried this locally on the m25n512 snapshot, and it worked fine! i wonder if you can please:

a. report the commit you're on
b. try with a lower particle count snapshot to see if it works?

[z20230220]

hi - thanks for reporting this! what commit hash are you on?

i'm also wondering if this may be due to a lack of memory. the error is obtuse so i'm not 100% sure, but i just tried this locally on the m25n512 snapshot, and it worked fine! i wonder if you can please:

a. report the commit you're on b. try with a lower particle count snapshot to see if it works?

Thank you for your help. However, I'm not sure how to check my commit hash. I only know that the version of caesar I'm using is 0.2b0 and the version of yt is 4.3.1. I will test a snapshot with fewer particles tomorrow.

[dnarayanan]

Hi - try this within your caesar directory

https://stackoverflow.com/questions/949314/how-do-i-get-the-hash-for-the-current-commit-in-git

[z20230220]

Hi - try this within your caesar directory

https://stackoverflow.com/questions/949314/how-do-i-get-the-hash-for-the-current-commit-in-git

Ok, I have now obtained the commit hash: e713602.

Additionally, I tested the data with fewer particles: snap_m50n512_151.hdf5 and snap_m25n512_151.hdf5, but received the same error message as follows:

...
yt : [INFO ] 2024-09-20 14:25:56,768 Computed properties for 141933 halos, RAM=25.85 GB

ValueError Traceback (most recent call last)
Cell In[3], line 8
5 obj = caesar.CAESAR(ds)
7 # now we execute member_search(), which identifies halos, galaxies, and computes
----> 8 obj.member_search(nproc=16)
10 # finally we save the CAESAR galaxy/halo catalog to your desired filename
11 obj.save('simba_151_caesar_file.hdf5')

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/main.py:340, in CAESAR.member_search(self, *args, **kwargs)
338 else:
339 from caesar.fubar_halo import fubar_halo
--> 340 fubar_halo(self)
341 assign.assign_galaxies_to_halos(self)
342 assign.assign_clouds_to_galaxies(self)

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/fubar_halo.py:72, in fubar_halo(obj)
70 mylog.warning('No valid halos found! Aborting member search')
71 return
---> 72 get_group_properties(halos,halos.obj.halo_list) # compute halo properties
73 if not obj.simulation.baryons_present: # if no baryons, we're done
74 return

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/group.py:819, in get_group_properties(self, grp_list)
816 from caesar.group_funcs import get_group_overall_properties,get_group_gas_properties,get_group_star_properties,get_group_bh_properties,get_group_dust_properties
818 get_group_overall_properties(self,grp_list)
--> 819 if 'gas' in self.obj.data_manager.ptypes: get_group_gas_properties(self,grp_list)
820 if 'star' in self.obj.data_manager.ptypes: get_group_star_properties(self,grp_list)
821 if 'dust' in self.obj.data_manager.ptypes: get_group_dust_properties(self,grp_list)

File caesar/group_funcs/group_funcs.pyx:497, in caesar.group_funcs.get_group_gas_properties()

ValueError: Buffer dtype mismatch, expected 'float' but got 'double'
342 assign.assign_clouds_to_galaxies(self)

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/fubar_halo.py:72, in fubar_halo(obj)
70 mylog.warning('No valid halos found! Aborting member search')
71 return
---> 72 get_group_properties(halos,halos.obj.halo_list) # compute halo properties
73 if not obj.simulation.baryons_present: # if no baryons, we're done
74 return

File /opt/conda/envs/Quench/lib/python3.12/site-packages/caesar-0.2b0-py3.12-linux-x86_64.egg/caesar/group.py:819, in get_group_properties(self, grp_list)
816 from caesar.group_funcs import get_group_overall_properties,get_group_gas_properties,get_group_star_properties,get_group_bh_properties,get_group_dust_properties
818 get_group_overall_properties(self,grp_list)
--> 819 if 'gas' in self.obj.data_manager.ptypes: get_group_gas_properties(self,grp_list)
820 if 'star' in self.obj.data_manager.ptypes: get_group_star_properties(self,grp_list)
821 if 'dust' in self.obj.data_manager.ptypes: get_group_dust_properties(self,grp_list)

File caesar/group_funcs/group_funcs.pyx:497, in caesar.group_funcs.get_group_gas_properties()

ValueError: Buffer dtype mismatch, expected 'float' but got 'double'

[weiguangcui]

I can confirm that the latest version of Caesar on the main branch has no problem. Could you give the latest version a try?