Lazy datafusion registration via custom table provider #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

alxmrs wants to merge 74 commits into main from reader

.github/workflows/ci-build.yml

-Original file line number
+Diff line change
@@ -0,0 +1,58 @@
+    name: ci-build
+    on:
+      # Triggers the workflow on push or pull request events but only for the main branch
+      push:
+        branches: [ main ]
+      pull_request:
+        branches: [ main ]
+      # Allows you to run this workflow manually from the Actions tab
+      workflow_dispatch:
+    jobs:
+      build:
+        name: "python ${{ matrix.platform.runner }}-${{ matrix.platform.target }} build"
+        runs-on: ${{ matrix.platform.runner }}
+        strategy:
+          fail-fast: false
+          matrix:
+            platform:
+              - runner: windows-latest
+                target: x86
+                python_arch: x86
+              - runner: macos-latest
+                target: aarch64
+              - runner: ubuntu-latest
+                target: aarch64
+              - runner: ubuntu-latest
+                target: x86_64
+        steps:
+          - uses: actions/checkout@v4
+          - uses: dtolnay/rust-toolchain@stable
+          - name: Setup sccache
+            uses: mozilla-actions/sccache-action@v0.0.9
+          - name: Configure sccache
+            run: |
+              echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
+              echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
+          - name: Cache Cargo
+            uses: actions/cache@v4
+            with:
+              path: ~/.cargo
+              key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ matrix.platform.runner }}-${{ matrix.platform.target }}-${{ hashFiles('Cargo.lock') }}
+          - uses: astral-sh/setup-uv@v5
+            with:
+              python-version: 3.11
+              enable-cache: true
+          # Use the --no-install-package to only install the dependencies
+          # but do not yet build the rust library
+          - name: The project builds
+            run: |
+              uv sync --dev --no-install-package xarray-sql
+              uv run --no-project maturin build

.github/workflows/ci-rust.yml

-Original file line number
+Diff line change
@@ -0,0 +1,59 @@
+    name: ci-rust
+    on:
+      push:
+        branches:
+          - main
+      pull_request:
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
+      cancel-in-progress: true
+    jobs:
+      lint-test-all-features:
+        name: Lint and Test
+        runs-on: ubuntu-latest
+        steps:
+          - uses: actions/checkout@v4
+            with:
+              submodules: "recursive"
+          - name: Install uv
+            uses: astral-sh/setup-uv@v5
+          - name: Install Rust
+            uses: dtolnay/rust-toolchain@stable
+            with:
+              components: clippy
+          - name: Setup sccache
+            uses: mozilla-actions/sccache-action@v0.0.9
+          - name: Configure sccache
+            run: |
+              echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
+              echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
+          - name: Clippy
+            run: cargo clippy --all-features --tests -- -D warnings
+          - name: Check
+            run: cargo check --all-features
+          - name: Test
+            run: cargo test --all-features
+      fmt:
+        name: Fmt
+        runs-on: ubuntu-latest
+        steps:
+          - uses: actions/checkout@v4
+            with:
+              submodules: "recursive"
+          - name: Install Rust
+            uses: dtolnay/rust-toolchain@stable
+            with:
+              components: rustfmt
+          - name: Fmt
+            run: cargo fmt -- --check --config imports_granularity=Module,group_imports=StdExternalCrate

.github/workflows/ci.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -33,19 +33,40 @@ jobs: @@
           fail-fast: false
           matrix:
             python-version: [
-              "3.9",
               "3.10",
               "3.11",
               "3.12",
               "3.13",
             ]
         steps:
           - uses: actions/checkout@v4
-          - name: Install uv and set the python version
-            uses: astral-sh/setup-uv@v5
+          - uses: dtolnay/rust-toolchain@stable
+          - name: Setup sccache
+            uses: mozilla-actions/sccache-action@v0.0.9
+          - name: Configure sccache
+            run: |
+              echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
+              echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
+          - name: Cache Cargo
+            uses: actions/cache@v4
+            with:
+              path: ~/.cargo
+              key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
+          - uses: astral-sh/setup-uv@v5
             with:
               python-version: ${{ matrix.python-version }}
+              enable-cache: true
+            # Use the --no-install-package to only install the dependencies
+            # but do not yet build the rust library
           - name: Install xarray_sql
-            run: uv sync --dev
+            run: uv sync --dev --no-install-package xarray-sql
+          - name: build rust
+            run: uv run --no-project maturin develop --uv
           - name: Run unit tests
-            run: uv run pytest -m "not integration"
+            run: uv run --no-project pytest -v . -m "not integration"

.github/workflows/publish.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,4 +1,5 @@
  
    # Taken from Xee and minimally modified: https://github.com/google/Xee/blob/main/.github/workflows/publish.yml

    # Taken from Xee and modified: https://github.com/google/Xee/blob/main/.github/workflows/publish.yml

    # Rust/maturin release adapted from https://github.com/apache/datafusion-python/blob/main/.github/workflows/build.yml

    #

    # Copyright 2023 Google LLC

    #

    @@ -13,85 +14,180 @@
  
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

    # See the License for the specific language governing permissions and

    # limitations under the License.

    #

    #

    # Copyright 2022 Apache Software Foundation

    #

    # Licensed to the Apache Software Foundation (ASF) under one

    # or more contributor license agreements.  See the NOTICE file

    # distributed with this work for additional information

    # regarding copyright ownership.  The ASF licenses this file

    # to you under the Apache License, Version 2.0 (the

    # "License"); you may not use this file except in compliance

    # with the License.  You may obtain a copy of the License at

    #

    #   http://www.apache.org/licenses/LICENSE-2.0

    #

    # Unless required by applicable law or agreed to in writing,

    # software distributed under the License is distributed on an

    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

    # KIND, either express or implied.  See the License for the

    # specific language governing permissions and limitations

    # under the License.

    #

    name: Publish to PyPi

    on:

      release:

        types: [published]

    jobs:

      build-artifacts:

        runs-on: ubuntu-latest

      build-artifacts-mac-win:

        runs-on: ${{ matrix.os }}

        strategy:

          fail-fast: false

          matrix:

            os: [ macos-latest, macos-15-intel, windows-latest ]

        steps:

          - uses: actions/checkout@v4

          - name: Set up Python

            uses: actions/setup-python@v2.3.1

            with:

              python-version: 3.11

          - name: Install dependencies

            run: |

              python -m pip install --upgrade pip

              python -m pip install setuptools setuptools-scm wheel twine check-manifest

          - uses: dtolnay/rust-toolchain@stable

          - name: Setup sccache

            uses: mozilla-actions/sccache-action@v0.0.9

          - name: Build tarball and wheels

          - name: Configure sccache

            run: |

              git clean -xdf

              git restore -SW .

              python -m build --sdist --wheel .

          - name: Check built artifacts

              echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV

              echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV

          - uses: astral-sh/setup-uv@v7

            with:

              enable-cache: true

          - name: Build wheels

            run: |

              python -m twine check dist/*

              pwd

              uv sync --dev --no-install-package xarray-sql

              uv run --no-project maturin build --release --strip

          - uses: actions/upload-artifact@v4

            with:

              name: releases

              path: dist

              name: dist-${{ matrix.os }}

              path: target/wheels/*

      test-built-dist:

        needs: build-artifacts

      build-artifacts-manylinux-x86_64:

        runs-on: ubuntu-latest

        steps:

          - uses: actions/setup-python@v2.3.1

            name: Install Python

          - uses: actions/checkout@v4

          - name: Build wheels for manylinux x86_64

            uses: PyO3/maturin-action@v1

            env:

              RUST_BACKTRACE: 1

            with:

              rust-toolchain: stable

              target: x86_64

              manylinux: 2014

              rustup-components: rust-std rustfmt

              sccache: 'true'

              args: --release

          - uses: actions/upload-artifact@v4

            with:

              name: dist-manylinux-x86_64

              path: target/wheels/*

      build-artifacts-manylinux-arm64:

        runs-on: ubuntu-latest

        steps:

          - uses: actions/checkout@v4

          - name: Build wheels for manylinux arm64

            uses: PyO3/maturin-action@v1

            env:

              RUST_BACKTRACE: 1

            with:

              python-version: 3.11

              rust-toolchain: stable

              target: aarch64

              # Use manylinux_2_28-cross because the manylinux2014-cross has GCC 4.8.5, which causes the build to fail

              manylinux: 2_28

              rustup-components: rust-std rustfmt

              sccache: 'true'

              args: --release

          - uses: actions/upload-artifact@v4

            with:

              name: dist-manylinux-aarch64

              path: target/wheels/*

      build-sdist:

        name: Source distribution

        runs-on: ubuntu-latest

        steps:

          - uses: actions/checkout@v4

          - name: Build sdist

            uses: PyO3/maturin-action@v1

            with:

              rust-toolchain: stable

              manylinux: auto

              rustup-components: rust-std rustfmt

              args: --release --sdist --out dist

          - uses: actions/upload-artifact@v4

            with:

              name: dist-sdist

              path: dist/*

      merge-build-artifacts:

        runs-on: ubuntu-latest

        needs:

          - build-artifacts-mac-win

          - build-artifacts-manylinux-x86_64

          - build-artifacts-manylinux-arm64

          - build-sdist

        steps:

          - name: Merge Build Artifacts

            uses: actions/upload-artifact/merge@v4

            with:

              name: dist

              pattern: dist-*

      verify-built-dist:

        needs: merge-build-artifacts

        runs-on: ubuntu-latest

        steps:

          - uses: actions/download-artifact@v4

            with:

              name: releases

              name: dist

              path: dist

          - name: List contents of built dist

            run: |

              ls -ltrh

              ls -ltrh dist

          - name: Publish package to TestPyPI

            if: github.event_name == 'push'

            uses: pypa/gh-action-pypi-publish@v1.13.0

            with:

              user: __token__

              password: ${{ secrets.TESTPYPI_TOKEN }}

              repository_url: https://test.pypi.org/legacy/

              verbose: true

            run: uv publish --token ${{ secrets.TESTPYPI_TOKEN }} --username __token__

          - name: Check uploaded package

            if: github.event_name == 'push'

            run: |

              sleep 3

              python -m pip install --upgrade pip

              python -m pip install --extra-index-url https://test.pypi.org/simple --upgrade de

              uv pip install --extra-index-url https://test.pypi.org/simple --upgrade de

              python -c "import xarray_sql; print(xarray_sql.__version__)"

      upload-to-pypi:

        needs: test-built-dist

        if: github.event_name == 'release'

        needs: verify-built-dist

        runs-on: ubuntu-latest

        steps:

          - uses: actions/download-artifact@v4

            with:

              name: releases

              name: dist

              path: dist

          - name: Publish package to PyPI

            uses: pypa/gh-action-pypi-publish@v1.13.0

          - uses: astral-sh/setup-uv@v7

            with:

              user: __token__

              password: ${{ secrets.PYPI_TOKEN }}

              verbose: true

              enable-cache: true

          - name: Publish package to PyPI

            run: uv publish --token ${{ secrets.PYPI_TOKEN }} --username __token__

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy datafusion registration via custom table provider #100

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Lazy datafusion registration via custom table provider #100

Are you sure you want to change the base?

Uh oh!

Lazy datafusion registration via custom table provider #100

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!