"boost::bad_get: failed value get using boost::get" error when using UDA with certain options

[olivhoenen]

This is related to iterorganization/IMAS-Core#22 opened by @paulotex on IMAS-Core.

Trying to reproduce and understand the issue, I get the feeling (but no real evidence yet) that this issue may be due to an interplay between imas-python and the UDA backend in the core.

Snippet to reproduce the error, using IMAS-Python 2.1.0 (but I also checked with develop) and IMAS-Core 5.5.2:

import imas
dbentry = imas.DBEntry("imas://uda.iter.org:56565/uda?path=/work/imas/shared/imasdb/ITER/3/121013/50&backend=hdf5&verbose=1", "r")
cp = in_dbentry.get("core_profiles", autoconvert=False) 
print(f"Read time array: {cp.time.value}")

Traceback:

  File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.1.0-intel-2023b/lib/python3.11/site-packages/imas/backends/imas_core/al_context.py", line 149, in read_data
    status, data = ll_interface.read_data(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "_al_lowlevel.pyx", line 998, in imas_core._al_lowlevel.al_read_data
  File "_al_lowlevel.pyx", line 849, in imas_core._al_lowlevel.al_read_data_scalar
imas_core.exception.ALException: Error while reading data: al_plugin_read_data: boost::bad_get: failed value get using boost::get
Error status=-1

Note that the get operation with direct read using the hdf5 backend works as expected.

Now if I force dd_version="3.38.1" (corresponds to the version of the pulse) when opening the DBentry, I don't see the error (with or without autoconvert=False in the get). In fact all later DD3 versions work, but not DD4.

If I use lazy=True (with or without adding cache_mode=None in the URI) it also works as expected.

Now if I disable the cache and use a normal (non-lazy) get, I may be facing the same error but the reported message is different and may be more insightful:

  File "/home/ITER/hoeneno/GIT/IMAS-Python/venv/lib/python3.11/site-packages/imas/backends/imas_core/al_context.py", line 149, in read_data
    status, data = ll_interface.read_data(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "_al_lowlevel.pyx", line 992, in imas_core._al_lowlevel.al_read_data
  File "_al_lowlevel.pyx", line 975, in imas_core._al_lowlevel.al_read_data_string
imas_core.exception.ALException: Error while reading data: al_plugin_read_data: cannot find node core_profiles/ids_properties/source in data dictionary (source not found)

Any idea @maarten-ic @prasad-sawantdesai ?

[maarten-ic]

Hi @olivhoenen,

My first thought is that this is to do with changed data types in the Data Dictionary? We've had a couple of elements that changed between FLT and INT (or vice versa) with DD 3->4...

I'll have a detailed look into your repro to see what's really going on here!

[maarten-ic]

After some debugging I think the following is the case in the first reproduction scenario:

1. The issue arises when attempting to read core_profiles/profiles_1d[0]/ion[0]/element[0]/z_n from the UDA backend. This quantity is an INT_0D in Data Dictionary 4, but a FLT_0D in DD 3.x.
2. The DBEntry you constructed was using the default DD version (DD 4.0.0 or 4.1.0), so IMAS-Python made (a copy of) that DD definition XML available for UDA, see https://github.com/iterorganization/IMAS-Python/blob/develop/imas/backends/imas_core/db_entry_al.py#L138-L141
3. When executing the get() a couple of things happen:
   1. IMAS-Python reads the DD version that the IDS was stored in. This happens to be 3.38.1.
   2. IMAS-Python loads the DD definitions for 3.38.1 for fetching data from the backend. Since you set autoconvert=False the returned IDS will be in this version of the DD (but it doesn't really matter for the issue).
   3. Next, IMAS-Python starts the full get() of all data in the IDS. This, in turn, triggers the UDA caching mechanism. UDA will prefetch all data, and parses the DD definitions to get a list of all data. The problem, however, is that UDA has loaded the DD 4.x definitions!
   4. At the point where IMAS-Python fetches the value for core_profiles/profiles_1d[0]/ion[0]/element[0]/z_n from the UDA backend, it expects to find a FLT_0D stored in the backend (since the data is stored in DD 3.x). However, UDA has a INT_0D ready in the cache (remember that it used the DD 4 definitions for querying the data on the remote). This mismatch in data type will trigger the boost::get error: the underlying boost::variant is holding an integer, and boost::get<double>(...) will fail for this (UDA backend source)...

When disabling the cache, a different error pops up, but this appears to have the same root cause: In DD 3.x there was a source field inside ids_properties which has been replaced by the provenance structure in DD 4.x. So I'd attribute this error to mismatching DD version expectations as well.

---

This is all well, but how to solve this?

UDA loads the data definitions upon construction of the Backend object (see uda_backend.h). So, (without overhauling the UDA backend code), the naive approach is to:

1. When doing a get(), first fetch the ids_properties/version_put/data_dictionary as usual.
2. [New and only applicable for the UDA backend] If this version is different from the version of the current DBEntry we need to:
   1. Construct a new DBEntry explicitly with the on-disk DD version. IMAS-Python will prepare an XML file for that DD version for UDA to use, and construct a new UDABackend.
   2. The get() is out-sourced to the new DBEntry, which is destroyed immediately afterwards.

However, this approach has two limitations:

1. It won't work with lazy loading, since that needs the DBEntry object to stay alive for as long as you might request additional data.
2. It won't work with implicit conversions of IDSs between minor versions of the Data Dictionary. An explicit conversion is possible but would introduce discrepancies in observable behaviour between different backends...

So, I think the naive approach is not really an option and a more elaborate solution will be needed.

---

I hope this write-up is clear, let me know if you'd like to discuss in more detail!

Final thoughts: This issue is pretty tricky, but I suppose that's what happens when code in IMAS-Core (the UDA backend) is trying to do stuff that's DD version dependent (which for all the other backends is for the HLI to manage)... Similar issues may arise for any other HLI code (e.g. the MATLAB HLI) or plugin (if anyone will develop that) that's trying to follow NBC changes?

[olivhoenen]

Thanks for the clear explanation @maarten-ic, indeed I was missing the fact that the IDSFactory for the DBEntry is the last version by default (so DD4) even after you recent change to defer parsing of the DD to the last moment. And the other tricky part was the fact that while the problem had the same root cause with or without cache_mode, the symptoms were affecting two different levels in the stack.

The good news is that with cache_mode=None and lazy=True there is no issue, it probably covers a wide range of cases. For other cases fetch option may be the most performant one and shall also be safe from this error. The last case of interest are remotely reading a slice (or set of).

Maybe for the time being it will be enough to print a warning when we use UDA in IMAS-Python and the IDSFactory's and remote IDS's DD versions do not match?

[maarten-ic]

Great, then this problem indeed occurs for a small number of use cases anyway.

Maybe for the time being it will be enough to print a warning when we use UDA in IMAS-Python and the IDSFactory's and remote IDS's DD versions do not match?

The reality is that users don't really read warnings, and if the chance of an error happening is high enough we should maybe just disallow it and provide a workaround?

Proposed solution

1. Add a section to the online documentation to explain workarounds, in the same spirit and on the same page as https://imas-python.readthedocs.io/en/stable/multi-dd.html#loading-idss-from-a-different-major-version. Possible workarounds are:

   • Use cache_mode=none (with lazy loading) or cache_mode=fetch in the UDA URI
   • Explicitly provide the DD version in de DBEntry constructor (e.g. dbentry = DBEntry(uda_uri, "r", dd_version="3.38.1") for the repro example in this issue), do an imas.convert_ids in case you need the IDS in a different DD version.

2. Raise an exception in IMAS-Python, which links to the documentation from point 1, if all of the following are true:

   • Using UDA backend
   • Using cache_mode=none and no lazy loading OR not using cache_mode=fetch
   • On-disk DD version does not match DBEntry DD version

Let me know what you think.

[olivhoenen]

I like the proposal, on the first part I would also add a note that the value of dd_version can be obtain by a lazy read of ids_properties.version_put.data_dictionary.

On the second part, the conditions look accurate it's just that they're quite complex and difficult to grasp from the user. And we could start with only the case where DD version on-disk and DBEntry do not have the same major version? I guess it may happen within a minor change, but I still have to find a case where is actually does.

[maarten-ic]

Thanks Olivier. Indeed the DD version would be read with a lazy read (as implemented in #39).

And we could start with only the case where DD version on-disk and DBEntry do not have the same major version? I guess it may happen within a minor change, but I still have to find a case where is actually does.

I found you a problematic case within the same major version 🙂 Although this doesn't raise an exception, but results in missing data. See the following code snippet:

import imas

pth = "/work/imas/shared/imasdb/ITER/3/121013/50"
hdf5_uri = f"imas:hdf5?path={pth}"
uda_uri = f"imas://uda.iter.org:56565/uda?path={pth}&backend=hdf5"

with imas.DBEntry(hdf5_uri, "r", dd_version="3.42.0") as entry:
    cp = entry.get("core_profiles")
    print("HDF5 backend:")
    print(repr(cp.profiles_1d[0].ion[0].name))
    print()


with imas.DBEntry(uda_uri, "r", dd_version="3.42.0") as entry:
    cp = entry.get("core_profiles")
    print("UDA backend:")
    print(repr(cp.profiles_1d[0].ion[0].name))
    print()

Will output

HDF5 backend:
<IDSString0D (IDS:core_profiles, profiles_1d[0]/ion[0]/name, STR_0D)>
str('H+')

UDA backend:
<IDSString0D (IDS:core_profiles, profiles_1d[0]/ion[0]/name, empty STR_0D)>

This field was renamed from label to name when going to DD 3.42.0. The following happens in this case:

• IMAS-Python knows that the on-disk version is 3.38.1, so this quantity is stored under a different name in the HDF5 file.
• When making the request to the backend, it will therefore ask about the value of label (the old name)
• The HDF5 backend returns the value, as expected.
• However, the UDA backend gives us an empty value!
  • In the UDA caching mechanism, the UDA backend requested the name of the ion (which the HDF5 backend reports as an empty string). The UDA backend doesn't know that this quantity used to be called label and it hasn't requested the label quantity from the remote backend. When IMAS-Python requests the value of label, the UDA backend hasn't cached a value for this and indicates that there is no data.