-
-
Notifications
You must be signed in to change notification settings - Fork 160
Update workflow to run swift 6.2 on different machines #882
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
6000206
Update workflow to run on ubuntu
Sencudra d18758f
return to 22.04, jq now installed?
Sencudra c472ef2
setup jq
Sencudra 709a020
Don't use jq, fix workflow, add macOS test for generator CLI
Sencudra 48da557
Add proper names and remove mac os legacy xcodes.
Sencudra 614f731
Adjust workflow
Sencudra 7bac26e
Split the ci into two workflows
Sencudra ea24b13
Fix names
Sencudra File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| name: CI Exercises | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| run-exercises-tests-ubuntu: | ||
| name: Run exercise tests for swift ${{ matrix.swift }} on Ubuntu | ||
| runs-on: ubuntu-24.04 | ||
| strategy: | ||
| matrix: | ||
| swift: ["6.0", "6.1", "6.2"] | ||
| container: | ||
| image: swift:${{ matrix.swift }} | ||
| env: | ||
| RUNALL: "true" | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 | ||
| - name: Run tests | ||
| run: swift test | ||
|
|
||
| run-exercises-tests-macos: | ||
| name: Run exercise tests on macOS ${{ matrix.macOS }} with Xcode ${{ matrix.xcode }} | ||
| runs-on: macos-${{ matrix.macOS }} | ||
| strategy: | ||
| matrix: | ||
| include: | ||
| - macOS: '26' | ||
| xcode: '26.0' | ||
| env: | ||
| RUNALL: "true" | ||
| DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 | ||
| - name: Run tests | ||
| run: swift test | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| name: CI Generator | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - '.github/workflows/ci_generator.yml' | ||
| - 'generator/**' | ||
| - 'bin/test_generator.sh' | ||
| push: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| run-generator-unit-tests: | ||
| name: Run generator unit tests | ||
| runs-on: ubuntu-24.04 | ||
| container: | ||
| image: swift:6.2 | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 | ||
| - name: Run tests | ||
| run: swift test --package-path ./generator | ||
|
|
||
| run-generator-usage-tests: | ||
| name: Test generator usage on Ubuntu | ||
| runs-on: ubuntu-24.04 | ||
| container: | ||
| image: swift:6.2 | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| - name: Run generator tests | ||
| run: ./bin/test_generator.sh | ||
|
|
||
| run-generator-usage-tests-macos: | ||
| name: Test generator usage on macOS ${{ matrix.macOS }} with Xcode ${{ matrix.xcode }} | ||
| runs-on: macos-${{ matrix.macOS }} | ||
| strategy: | ||
| matrix: | ||
| include: | ||
| - macOS: '26' | ||
| xcode: '26.0' | ||
| env: | ||
| RUNALL: "true" | ||
| DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 | ||
| - name: Run generator tests | ||
| run: ./bin/test_generator.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,42 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -eo pipefail | ||
| set -euo pipefail | ||
|
|
||
| temp=$(mktemp -d) | ||
| declare TEMP_DIR | ||
| TEMP_DIR=$(mktemp -d) | ||
|
|
||
| readonly TEMP_DIR | ||
| readonly GENERATOR_DIR="./generator" | ||
| readonly PRACTICE_DIR="./exercises/practice" | ||
| readonly TEMPLATE_PATH=".meta/template.swift" | ||
|
|
||
| swift build --package-path ./generator | ||
|
|
||
| for exercise in ./exercises/practice/*; do | ||
| if [[ -r "${exercise}/.meta/template.swift" ]]; then | ||
| cp -r "$exercise" "$temp" | ||
| test_file="${exercise}/$(jq -r '.files.test[0]' ${exercise}/.meta/config.json)" | ||
| temp_test_file="${temp}/$(basename "$exercise")/$(jq -r '.files.test[0]' ${exercise}/.meta/config.json)" | ||
| swift run --package-path ./generator Generator $(basename "$exercise") "$temp/$(basename "$exercise")" | ||
| echo "Comparing $test_file with $temp_test_file" | ||
| diff "$temp_test_file" "$test_file" | ||
| if [ $? -ne 0 ]; then | ||
| exit_code=1 | ||
| fi | ||
| exit_code=0 | ||
| for exercise_path in "${PRACTICE_DIR}"/*; do | ||
| [[ -e "${exercise_path}/${TEMPLATE_PATH}" ]] || continue | ||
|
|
||
| cp -r "${exercise_path}"/. "${TEMP_DIR}" | ||
|
|
||
| # Minify json and extract test file name | ||
| test_file=$(tr -d '\n\r ' < "${exercise_path}"/.meta/config.json | grep -o '"test":\["[^"]*' | sed 's/.*\["//') | ||
| original_test="${exercise_path}/${test_file}" | ||
| temp_test="${TEMP_DIR}/${test_file}" | ||
|
|
||
| exercise_name="${exercise_path##*/}" | ||
|
|
||
| if ! swift run --package-path "${GENERATOR_DIR}" Generator "${exercise_name}" "${TEMP_DIR}"; then | ||
| printf 'Generation failed for %s\n' "${exercise_name}" | ||
| exit_code=1 | ||
| continue | ||
| fi | ||
|
|
||
| if ! diff -u -- "${original_test}" "${temp_test}"; then | ||
| printf 'Mismatch detected in %s\n' "${exercise_name}" | ||
| exit_code=1 | ||
| else | ||
| printf '%s OK\n' "${exercise_name}" | ||
| fi | ||
| done | ||
|
|
||
| exit ${exit_code} | ||
| exit "${exit_code}" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to also test outside of xcode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I fully understand the question. For CI, it’s simple to use Swift from Xcode since it’s pre-installed and comes with Xcode and macOS — version 6.2 for Xcode 26. However, it’s not using Xcode directly to run the tests. Can someone without Xcode run the tests with only the Swift toolchain installed? Probably yes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I understand is the toolchain different (if I can recall, they use a different build system). I think it would be good for just the exercise test to test if possible both xcode and doing it outside of xcode.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What value would it add to have the tests run twice as part of the CI? The purpose of the CI step is to validate the exercise tests work, not to test the Swift test runner. Adding a second test increases the cost/complexity of the CI. What concrete value does it add?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I never mentioned the Swift test runner. The swift in xcode has a different build system compared to if you install just swift. In practice these behaves differently and code are not for graunted cross platform in swift thereby. Since certain libraries doesnt exsist on certain platforms. This in practice if only tested on one platform could mean it doesnt work on another.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the purpose of this CI to validate the tests work or that the build systems work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It suppose to test that the exercises can be built on different build systems. If we just say that if the exercises pass on one system/setup, it should automatically work on every other system/setup, then we don't have to test on xcode/macos at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TL;DR After conducting some research, I think it’s overkill to test building an exercise both with and without Xcode. The difference isn’t in the build system (it’s the same — Swift Package Manager), but rather in the dependencies included with different versions of the toolchain: installed via Xcode, Swiftly, or the standalone .pkg installer.
Running tests without Xcode does not work out of the box after installing the toolchain. It throws errors about missing the
Testinglibrary. It also does not work — and probably will not — with Swiftly, because that distribution completely lacks the requiredTestinglibraries, which is strange. Additionally, it is missingXCTest, which historically has not been part of Swift and it's expected.However, the required
Testinglibraries are included in the .pkg installer, and tests work with some warnings:So, for macOS users, I think we should strongly recommend installing Swift via Xcode, which includes everything needed, and probably not even mention Swiftly in the installation guide.
Toolchain differences between Apple and non-Apple platforms
What’s even more frustrating is that the toolchain itself differs between Apple and non-Apple platforms. On Apple platforms (macOS),
Foundationcomes as a preinstalledFoundation.framework. This means that in your code you can simply write:import Foundationgraph TD; MacOS[MacOS toolchain]-->FF NA[Non-Apple Platform toolchain]-->SCLF subgraph Apple InHouse FF[Foundation.framework] end FF-->SF subgraph GitHub SCLF[swift-corelibs-foundation]-->SF SF[swift-foundation]-->FICU[swift-foundation-icu] SF-->SC[swift-collections] endHowever, on non-Apple platforms, this is not the case. These platforms use a specific lib called
swift-corelibs-foundation, which encapsulates platform-specific logic and comes with the Swift toolchain (it is not included for macOS).There is no single
Foundationimport that provides everything. This means that if you need XML or networking capabilities, you must importFoundationfunctionality as separate standalone modules.As far as I understand, this is something Apple would eventually like to achieve on macOS as well, but the ETA is unknown.
For now, on non-Apple platforms (e.g., Linux), you are able to add the following imports:
It’s hard to imagine someone on a non-Apple platform including any of these imports in an exercise, but it is technically possible. If they do, it will break the build on macOS.
For this reason, I have added one CI job to test building on macOS, assuming GitHub Actions workflows are free for Exercism.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I totally think we should have macOS and Linux ci, and I would like to add that I like the extra Linux ci for the generator, since I have noticed that it behaves differently on macOS and Linux.
I find your conclusion that it is too much work to add non xcode testing, so I think we will just leave it as it is like this. Good job!