Test that the container running on the UChicago AF JupyterHub is able to control worker nodes through Dask

[matthewfeickert]

We want to be able to run a workflow that runs a distributed analysis with dask.distributed. This means that a

from dask.distributed import Client
client = Client()

run inside of a Jupyter Notebook in the interactive JupyterLab session should be able to send jobs to the worker nodes on the AF.

[lukasheinrich]

Just pasting here some of the next steps we should be targeting

• compile @mvigl's code in standalone mode (i.e. what he's been working on for a while)
• run Nikolai's notebook in local dask worker mode
  • End user should be given control over /venv/ virtual environment analysisbase-dask-uc#2 is going to require a rebuild of the lock file that includes coffea for now, as the permissions issue won't allow for coffea to unistall any versions of software during its install (@matthewfeickert edit)
  • https://gitlab.cern.ch/gstark/pycolumnarprototype/-/issues/1 (@matthewfeickert edit)
• try to setup a dask cluster with 2 nodes or similar & repeat
  • https://gitlab.cern.ch/gstark/pycolumnarprototype/-/issues/2 (@matthewfeickert edit)
  • Test that the container running on the UChicago AF JupyterHub is able to control worker nodes through Dask #4 (comment) (@matthewfeickert edit)
• possibly to a full scale test (e.g. entire dataset)
  • Test that the container running on the UChicago AF JupyterHub is able to control worker nodes through Dask #4 (comment) (@matthewfeickert edit)
• look into integrating pybind/nanobind into athena & adjust cmake
• recompile @mvigl's code within athena
• repeat

[matthewfeickert]

For tracking purposes, @mvigl can you also comment here what the data sets are that are needed for https://gitlab.cern.ch/gstark/pycolumnarprototype (not sure what branch is being used for dev at the moment) to run the notebook demos and where those are located on the MWT2 at the moment?

[matthewfeickert]

compile @mvigl's code in standalone mode (i.e. what he's been working on for a while)

So right away we run into some permissions issues with Git in the container (will have to think of the best way to sort that out)

(venv) [bash][feickert AnalysisBase-24.2.26]:workarea > . /release_setup.sh
(venv) [bash][feickert AnalysisBase-24.2.26]:workarea > mkdir container_testing
(venv) [bash][feickert AnalysisBase-24.2.26]:workarea > cd container_testing/
(venv) [bash][feickert AnalysisBase-24.2.26]:container_testing > git clone --recursive ssh://git@gitlab.cern.ch:7999/gstark/pycolumnarprototype.git
Cloning into 'pycolumnarprototype'...
Warning: Permanently added the ECDSA host key for IP address '[188.185.35.37]:7999' to the list of known hosts.
Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
(venv) [bash][feickert AnalysisBase-24.2.26]:container_testing > git clone -v --recursive https://gitlab.cern.ch/gstark/pycolumnarprototype.git
Cloning into 'pycolumnarprototype'...
# This hangs forever

But if we avoid that for the moment and do from a login node on the UChicago AF

$ ssh feickert@login.af.uchicago.edu
[16:19] login02.af.uchicago.edu:~ $ cd workarea/container_testing/
[16:20] login02.af.uchicago.edu:~/workarea/container_testing $ git clone --branch py_el_tool_test --recursive ssh://git@gitlab.cern.ch:7999/gstark/pycolumnarprototype.git

and then flip back to JupyterLab we're able to build as expected

(venv) [bash][feickert AnalysisBase-24.2.26]:pycolumnarprototype > cmake -S src -B build
(venv) [bash][feickert AnalysisBase-24.2.26]:pycolumnarprototype > cmake --build build --clean-first
-- Setting ATLAS specific build flags
-- checker_gccplugins library not found
-- Using the LCG modules without setting up a release
-- $<BUILD_INTERFACE:/usr/AnalysisBaseExternals/24.2.26/InstallArea/x86_64-centos7-gcc11-opt/lib/libpython3.9.so>;$<INSTALL_INTERFACE:/usr/AnalysisBaseExternals/24.2.26/InstallArea/x86_64-centos7-gcc11-opt/lib/libpython3.9.so>
-- Configuring ATLAS project with name "PyColumnarPrototypeDemo" and version "1.0.0"
...
[100%] Linking CXX shared library ../x86_64-centos7-gcc11-opt/lib/libColumnarPrototypeDict.so
Detaching debug info of libColumnarPrototypeDict.so into libColumnarPrototypeDict.so.dbg
[100%] Built target ColumnarPrototypeDict
[100%] Built target Package_ColumnarPrototype
(venv) [bash][feickert AnalysisBase-24.2.26]:pycolumnarprototype > PYTHONPATH="$(dirname $(find . -type f -iname "PyColumnarPrototype*.so")):${PYTHONPATH}" python3 -c 'import PyColumnarPrototype; print(f"{PyColumnarPrototype.column_maker()=}")'
PyColumnarPrototype.column_maker()=array([1543080.6, 7524391.5], dtype=float32)

✅

I'll count that as "good enough" until we can figure out how to manage Git permissions here, and I think the SSL team can give us plenty of pointers given that they know how to do this for Coffea-casa. 👍

[mvigl]

For tracking purposes, @mvigl can you also comment here what the data sets are that are needed for https://gitlab.cern.ch/gstark/pycolumnarprototype (not sure what branch is being used for dev at the moment) to run the notebook demos and where those are located on the MWT2 at the moment?

The dev branch is https://gitlab.cern.ch/gstark/pycolumnarprototype/-/tree/py_el_tool_test?ref_type=heads, the PHYSLITE data in Zee_demo.ipynb is quite old and I believe it can only be found here now:

• /afs/cern.ch/user/e/ekourlit/public/mc20_13TeV.700322.Sh_2211_Zee_maxHTpTV2_CVetoBVeto.deriv.DAOD_PHYSLITE.e8351_s3681_r13167_p5511/mc20_13TeV/DAOD_PHYSLITE.32059997._000049.pool.root.1
• /eos/user/e/ekourlit/public/data17_13TeV.periodAllYear.physics_Main.PhysCont.DAOD_PHYSLITE.grp17_v01_p5631/DAOD_PHYSLITE.29681342._000190.pool.root.1

As for the datasets we will move to I'll paste here Vangelis' comment from another thread:

So the MC dataset for the phase 1 testing can be the: mc20_13TeV.700322.Sh_2211_Zee_maxHTpTV2_CVetoBVeto.deriv.DAOD_PHYSLITE.e8351_s3681_r13167_p5855
which is ~1 TB.

For the phase 2 testing we would need both data and MC:

• data18_13TeV.periodAllYear.physics_Main.PhysCont.DAOD_PHYSLITE.grp18_v01_p5855
• mc20_13TeV.700320.Sh_2211_Zee_maxHTpTV2_BFilter.deriv.DAOD_PHYSLITE.e8351_s3681_r13167_p5855
• mc20_13TeV.700321.Sh_2211_Zee_maxHTpTV2_CFilterBVeto.deriv.DAOD_PHYSLITE.e8351_s3681_r13167_p5855
  which is ~45 TB (data) + ~1.5 TB (MC)

[matthewfeickert]

run Nikolai's notebook in local dask worker mode

This is now done via https://gitlab.cern.ch/gstark/pycolumnarprototype/-/issues/1#note_7305996. I'll let @mvigl do a double check on this but I'm calling this complete as it ran from top top bottom with a "restart and run all" in the container on the UChicago AF.

✅

[ivukotic]

I have already made sure that data can be found at MWT2_LOCALGROUPDISK and MC partly at MWT2 and all of it at BNL.

[matthewfeickert]

try to setup a dask cluster with 2 nodes or similar & repeat

Okay, so @ivukotic has been kind enough to agree to work on setting up a kubernets cluster for us this week where he'll do some tests and see if he can get the notebook to scale out across all the data that he's transfered to the MWT2_LOCALGROUPDISK. 🚀 He'll comment back here if he has questions for us, but once he's done that then we should be able to try this out genrically and do some scaling tests on the cluster.

So this might also address

possibly to a full scale test (e.g. entire dataset)

We can then take these results to ATLAS and further motivate

look into integrating pybind/nanobind into athena & adjust cmake

👍

[matthewfeickert]

Recursive cloning without authentication

git clone --branch py_el_tool_test --recurse-submodules https://gitlab.cern.ch/gstark/pycolumnarprototype.git

has now been fixed.

Example:

(venv) [bash][atlas AnalysisBase-24.2.26]:workdir > git clone --branch py_el_tool_test --recurse-submodules https://gitlab.cern.ch/gstark/pycolumnarprototype.git
Cloning into 'pycolumnarprototype'...
remote: Enumerating objects: 515, done.
remote: Counting objects: 100% (152/152), done.
remote: Compressing objects: 100% (146/146), done.
remote: Total 515 (delta 67), reused 16 (delta 6), pack-reused 363
Receiving objects: 100% (515/515), 2.26 MiB | 1.67 MiB/s, done.
Resolving deltas: 100% (283/283), done.
Submodule 'src/columnarprototype' (https://gitlab.cern.ch/krumnack/columnarprototype.git) registered for path 'src/ColumnarPrototype'
Submodule 'src/nanobind' (https://github.com/wjakob/nanobind) registered for path 'src/nanobind'
Cloning into 'src/ColumnarPrototype'...
remote: Enumerating objects: 370, done.
remote: Counting objects: 100% (308/308), done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 370 (delta 199), reused 307 (delta 199), pack-reused 62
Receiving objects: 100% (370/370), 104.55 KiB | 0 bytes/s, done.
Resolving deltas: 100% (229/229), done.
Submodule path 'src/ColumnarPrototype': checked out '1e1537fc9669fe7425e74313200dd8bd3e4e3c64'
Cloning into 'src/nanobind'...
remote: Enumerating objects: 5502, done.
remote: Counting objects: 100% (1855/1855), done.
remote: Compressing objects: 100% (268/268), done.
remote: Total 5502 (delta 1646), reused 1681 (delta 1568), pack-reused 3647
Receiving objects: 100% (5502/5502), 2.00 MiB | 0 bytes/s, done.
Resolving deltas: 100% (3981/3981), done.
Submodule path 'src/nanobind': checked out 'c7bd406ef758c933eaf4b2d03d6d81b54bd9ad03'
Submodule 'ext/robin_map' (https://github.com/Tessil/robin-map) registered for path 'ext/robin_map'
Cloning into 'ext/robin_map'...
remote: Enumerating objects: 1098, done.
remote: Counting objects: 100% (152/152), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 1098 (delta 105), reused 115 (delta 82), pack-reused 946
Receiving objects: 100% (1098/1098), 875.43 KiB | 0 bytes/s, done.
Resolving deltas: 100% (752/752), done.
Submodule path 'src/nanobind/ext/robin_map': checked out '68ff7325b3898fca267a103bad5c509e8861144d'
(venv) [bash][atlas AnalysisBase-24.2.26]:workdir > 

c.f. https://gitlab.cern.ch/gstark/pycolumnarprototype/-/issues/2#note_7306579 for more details.

Using this I've added PyColumnarPrototype to the container image in PR #11 so that it is importable. If a user wants to build ColumnarPrototype from https://gitlab.cern.ch/gstark/pycolumnarprototype/ they still can use that by either patching PYTHONPATH

PYTHONPATH="$(dirname $(find . -type f -iname "PyColumnarPrototype*.so")):${PYTHONPATH}"

or sys.path

# Ensure importable
try:
    import PyColumnarPrototype
except ModuleNotFoundError as err:
    import sys
    from pathlib import Path
    # position 1 of sys.path will be after cwd and before the activated virtual environment's site-packages
    sys.path.insert(1, str(next(Path().cwd().glob("**/PyColumnarPrototype*.so")).parent))
    import PyColumnarPrototype

with the built version at import time.

[ivukotic]

@mvigl can you please let me know how do I change Z->ee notebook in order to run across more than 1 mc and 1 data file?

[mvigl]

@mvigl can you please let me know how do I change Z->ee notebook in order to run across more than 1 mc and 1 data file?

I don't have a clear answer yet, sry - I need to look more in detail tomorrow. Changing the 'Get data' section to something like this could be a starting point.

MC_Files = "/data/alheld/ATLAS/mc20_13TeV/DAOD_PHYSLITE.*.pool.root.1:CollectionTree"
variables = ["AnalysisElectronsAuxDyn.pt",
             "AnalysisElectronsAuxDyn.eta",
             "AnalysisElectronsAuxDyn.phi",
             "AnalysisElectronsAuxDyn.charge"
            ]
events = uproot.dask(MC_Files, 
                     library="ak", 
                     filter_name=variables,
                     open_files=False)

We don't want to specify the variables that are accessed (which is a longer list) but for some reason it doesn't work if I don't.

UnknownInterpretation: none of the rules matched
in file [/eos/user/e/ekourlit/public/mc20_13TeV/DAOD_PHYSLITE.34869306._000049.pool.root.1](https://vscode-remote+ssh-002dremote-002blxplus-002ecern-002ech.vscode-resource.vscode-cdn.net/eos/user/e/ekourlit/public/mc20_13TeV/DAOD_PHYSLITE.34869306._000049.pool.root.1)
in object [/CollectionTree](https://vscode-remote+ssh-002dremote-002blxplus-002ecern-002ech.vscode-resource.vscode-cdn.net/CollectionTree);1:xTrigDecisionAux./xTrigDecisionAux.xAOD::AuxInfoBase

[ivukotic]

@mvigl can you please let me know how do I change Z->ee notebook in order to run across more than 1 mc and 1 data file?

I don't have a clear answer yet, sry - I need to look more in detail tomorrow. Changing the 'Get data' section to something like this could be a starting point.

MC_Files = "/data/alheld/ATLAS/mc20_13TeV/DAOD_PHYSLITE.*.pool.root.1:CollectionTree"
variables = ["AnalysisElectronsAuxDyn.pt",
             "AnalysisElectronsAuxDyn.eta",
             "AnalysisElectronsAuxDyn.phi",
             "AnalysisElectronsAuxDyn.charge"
            ]
events = uproot.dask(MC_Files, 
                     library="ak", 
                     filter_name=variables,
                     open_files=False)

We don't want to specify the variables that are accessed (which is a longer list) but for some reason it doesn't work if I don't.

UnknownInterpretation: none of the rules matched
in file [/eos/user/e/ekourlit/public/mc20_13TeV/DAOD_PHYSLITE.34869306._000049.pool.root.1](https://vscode-remote+ssh-002dremote-002blxplus-002ecern-002ech.vscode-resource.vscode-cdn.net/eos/user/e/ekourlit/public/mc20_13TeV/DAOD_PHYSLITE.34869306._000049.pool.root.1)
in object [/CollectionTree](https://vscode-remote+ssh-002dremote-002blxplus-002ecern-002ech.vscode-resource.vscode-cdn.net/CollectionTree);1:xTrigDecisionAux./xTrigDecisionAux.xAOD::AuxInfoBase

it is not realistic to have 50TB locally. I need a way to give it a lot of files accessible via xroot...

[ekourlit]

I think that should be doable according to documentation: https://uproot.readthedocs.io/en/latest/uproot._dask.dask.html

[matthewfeickert]

@alexander-held points out that for the accessing local files you can use a list of xrootd-accessible URIs like is done in the IRIS-HEP Analysis Grand Challenge notebooks. Probably the most relevant bit is the utils.file_input.construct_fileset.