Lazy datafusion registration via custom table provider#100

Open

alxmrs wants to merge 74 commits intomainfrom

Commits on Jan 15, 2026

New approach: python only stream API.
alxmrs
committed
Clever strategy: use a rust-implemented datafusion provider to read from arrow streams, which are lazy, to implement the table registration, which in the default df library, is not lazy enough.
alxmrs
committed
CC Self-review. Applied lint. Using new method in the high level API (XarrayContext). This leads to two failed tests, but this could be caused by test errors.
alxmrs
committed

Commits on Jan 16, 2026

Commits on Jan 18, 2026

Commits on Jan 24, 2026

Commits on Jan 25, 2026

Commits on Jan 26, 2026