Refactor build system (#596)

* Refactor build logic

* update (simplify) cd.yml

* fix typo

* Fix ci (I think)

* deps

* typo

* not sure why but need preinstalled here

* black

* twine too

* no input

* try

* remove dev reqs

* Fix ci/cd

* do korijns earlier trick

* tweak in error msg

* Fix sdist

* Update references to download script

* black

* fix cd

* Move coverage config to pyproject.toml

* apply review suggestions

Author: almarklein

.github/ISSUE_TEMPLATE/update-wgpu.md

@@ -47,7 +47,7 @@ For context, see the [codegen readme](https://github.com/pygfx/wgpu-py/blob/main
 
 *The lines below can be copied in the PR's top post.*
 
-* [ ] Run `python download-wgpu-native.py --version xx` to download the latest webgpu.h and DLL.
+* [ ] Run `python tools/download_wgpu_native.py --version xx` to download the latest webgpu.h and DLL.
 * [ ] Run `python codegen` to apply the automatic patches to the code.
 * [ ] It may be necessary to tweak the `hparser.py` to adjust to new formatting.
 * [ ] Diff the report for new differences to take into account.

.github/workflows/cd.yml

@@ -1,15 +1,8 @@
 # Github Actions script to produce binary wheels.
 #
-# Note that a lot of the cibuildwheel config is in pyproject.toml.
-#
-# We perform one build for each wheel that we generate, i.e. one per architecture.
-# In download_wgpu_native.py, we detect CIBW_PLATFORM and CIBW_ARCHS to determine
-# the required binary from wgpu-native.
-#
-# If https://github.com/pypa/cibuildwheel/issues/944 gets implemented, we can build more wheels per build.
-#
-# Also includes the sdist build that does not include a binary.
-
+# * One build to create all wheels (cross-platform).
+# * One build (with matrix) test the wheels on a selection of platforms.
+# * One build to publish the wheels on GitHub and Pypi.
 
 name: CD
 
@@ -24,118 +17,88 @@ on:
 
 jobs:
 
-  release-builds:
-    name: Build wheel for ${{ matrix.platform }} ${{ matrix.arch }}
-    timeout-minutes: 10
+  build-wheels:
+    name: Build all wheels
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dev dependencies
+        run: |
+            python -m pip install --upgrade pip
+            pip install -U -e .[build]
+      - name: Build wheels (and sdist)
+        run: python tools/build_all_wheels.py
+      - name: Twine check
+        run: |
+            twine check dist/*
+      - name: Upload distributions
+        uses: actions/upload-artifact@v4
+        with:
+          path: dist
+          name: all_wheels
+
+  test-wheels:
+    name: Test wheel for ${{ matrix.name }}
+    needs: [build-wheels]
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
         include:
-          - platform: windows
-            arch: AMD64
-            os: windows-latest
-            testable: true
-          - platform: windows
-            arch: ARM64
+          - name: windows amd64
             os: windows-latest
-          - platform: windows
-            arch: x86
-            os: windows-latest
-          - platform: macos
-            arch: arm64
+          - name: macos arm64
             os: macos-latest
-            testable: true
-          - platform: macos
-            arch: x86_64
+          - name: macos x86_64
             os: macos-13  # last Intel MacOS
-            cibw_version: '==2.16'  # delocation does not work for later versions
-          - platform: linux
-            arch: x86_64
-            os: ubuntu-latest
-            testable: true
-          - platform: linux
-            arch: aarch64
+          - name: linux amd64
             os: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
-    - name: Set up QEMU
-      if: matrix.platform == 'linux' && matrix.arch == 'aarch64'
-      uses: docker/setup-qemu-action@v3
+    - name: Set up Python
+      uses: actions/setup-python@v5
       with:
-        platforms: arm64
-    - name: Install dev dependencies
-      run: |
-          python -m pip install --upgrade pip wheel setuptools twine cibuildwheel${{ matrix.cibw_version}}
-    - name: Build wheels
-      run: python -m cibuildwheel --output-dir dist
-      env:
-        CIBW_PLATFORM: ${{ matrix.platform }}
-        CIBW_ARCHS: ${{ matrix.arch }}
-    - name: Twine check
+        python-version: '3.12'
+    - name: Download assets
+      uses: actions/download-artifact@v4
+      with:
+        path: dist
+    - name: Flatten dist dir
+      shell: bash
       run: |
-          twine check dist/*
-    - name: Test wheel
-      if: matrix.testable
+          find dist -mindepth 2 -type f -exec mv -f '{}' dist/ ';'
+          rm -rf dist/*/
+    - name: Install and test wheel
       shell: bash
       run: |
           rm -rf ./wgpu
-          filename=$(ls dist/*.whl)
-          pip install $filename
+          # Install 'normally' to install deps, then force the install from dist-folder and nothing else
+          pip install --find-links dist wgpu
+          pip install --force-reinstall --no-deps --no-index --find-links dist wgpu
           pushd $HOME
           python -c 'import wgpu.backends.wgpu_native; print(wgpu.backends.wgpu_native._ffi.lib_path)'
           popd
           pip uninstall -y wgpu
           git reset --hard HEAD
-    - name: Upload distributions
-      uses: actions/upload-artifact@v4
-      with:
-        path: dist
-        name: ${{ matrix.platform }}-${{ matrix.arch }}-build
-
-
-  sdist-build:
-    name: Build sdist
-    timeout-minutes: 5
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-    steps:
-    - uses: actions/checkout@v4
-    - name: Set up Python
-      uses: actions/setup-python@v5
-      with:
-        python-version: '3.12'
-    - name: Install dev dependencies
-      run: |
-          python -m pip install --upgrade pip
-          pip install -U -r dev-requirements.txt
-    - name: Create source distribution
-      run: |
-          python setup.py sdist
-    - name: Test sdist
+    - name: Install from sdist
       shell: bash
       run: |
           rm -rf ./wgpu
+          rm -rf ./dist/*.whl
+          # Install sdist, but no test, because there is no wgpu-native lib now.
           filename=$(ls dist/*.tar.gz)
           pip install $filename
-          # don't run tests, we just want to know if the sdist can be installed
           pip uninstall -y wgpu
           git reset --hard HEAD
-    - name: Twine check
-      run: |
-          twine check dist/*
-    - name: Upload distributions
-      uses: actions/upload-artifact@v4
-      with:
-        path: dist
-        name: sdist-build
-
 
   publish:
     name: Publish to Github and Pypi
     runs-on: ubuntu-latest
-    needs: [release-builds, sdist-build]
+    needs: [build-wheels, test-wheels]
     if: success() && startsWith(github.ref, 'refs/tags/v')
     steps:
     - uses: actions/checkout@v4

.github/workflows/ci.yml

@@ -28,7 +28,7 @@ jobs:
     - name: Install dev dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U black flake8 flake8-black pep8-naming
+          pip install -U -e .[lint]
     - name: Flake8
       run: |
           flake8 .
@@ -48,7 +48,7 @@ jobs:
     - name: Install dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U pytest numpy black cffi
+          pip install -U -e .[codegen]
     - name: Test codegen
       run: |
           pytest -v codegen
@@ -68,9 +68,6 @@ jobs:
     - name: Install dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install requests
-          python download-wgpu-native.py
-          pip uninstall -q -y requests
           pip install -e .
     - name: Test imports
       env:
@@ -96,7 +93,7 @@ jobs:
     - name: Install dev dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U -r dev-requirements.txt
+          pip install -U -e .[docs]
     - name: Build docs
       run: |
           cd docs
@@ -121,9 +118,7 @@ jobs:
     - name: Install dev dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U -r dev-requirements.txt
-          python download-wgpu-native.py
-          pip install -e .
+          pip install -U -e .[tests,examples]
     - name: Test examples
       env:
         EXPECT_LAVAPIPE: true
@@ -145,10 +140,7 @@ jobs:
     - name: Install dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U requests numpy pytest
-          python download-wgpu-native.py
-          pip install -e .
-          pip install psutil glfw pyinstaller>=4.9
+          pip install -U -e .[tests] glfw pyinstaller
     - name: Test PyInstaller
       run: |
           pyinstaller --version
@@ -194,9 +186,7 @@ jobs:
     - name: Install dev dependencies
       run: |
           python -m pip install --upgrade pip
-          pip install -U -r dev-requirements.txt
-          python download-wgpu-native.py
-          pip install -e .
+          pip install -U -e .[tests]
     - name: Unit tests
       run: |
           pytest -v tests

.github/workflows/screenshots.yml

@@ -8,7 +8,7 @@ on:
 
 jobs:
   screenshots:
-    name: Regenerate
+    name: Regenerate screenshot
     timeout-minutes: 10
     runs-on: 'ubuntu-latest'
     steps:
@@ -26,9 +26,7 @@ jobs:
     - name: Install dev dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -U -r dev-requirements.txt
-        python download-wgpu-native.py
-        pip install -e .
+        pip install -U -e .[tests,examples]
     - name: Regenerate screenshots
       run: |
         pytest -v --regenerate-screenshots -k test_examples_screenshots examples

README.md

@@ -115,18 +115,14 @@ This code is distributed under the 2-clause BSD license.
 ## Developers
 
 * Clone the repo.
-* Install devtools using `pip install -r dev-requirements.txt` (you can replace
-  `pip` with `pipenv` to install to a virtualenv).
-* Install wgpu-py in editable mode by running `pip install -e .`, this will also
-  install runtime dependencies as needed.
-* Run `python download-wgpu-native.py` to download the upstream wgpu-native
+* Install devtools using `pip install -e .[dev]`.
+* Using `pip install -e .` will also download the upstream wgpu-native
   binaries.
-  * Or alternatively point the `WGPU_LIB_PATH` environment variable to a custom
-    build.
+  * You can use `python tools/download_wgpu_native.py` when needed.
+  * Or point the `WGPU_LIB_PATH` environment variable to a custom build of `wgpu-native`.
 * Use `black .` to apply autoformatting.
 * Use `flake8 .` to check for flake errors.
 * Use `pytest .` to run the tests.
-* Use `pip wheel --no-deps .` to build a wheel.
 
 
 ### Updating to a later version of WebGPU or wgpu-native

codegen/README.md

@@ -136,7 +136,7 @@ The majority of work in the wgpu-native backend is the conversion of  Python dic
 
 ### The update process
 
-* Download the latest `webgpu.h` and DLL using `python download-wgpu-native.py --version xx`
+* Download the latest `webgpu.h` and DLL using `python tools/download_wgpu_native.py --version xx`
 * Run `python codegen` to apply the automatic patches to the code.
 * It may be necessary to tweak the `hparser.py` to adjust to new formatting.
 * Diff the report for new differences to take into account.