Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 0 additions & 46 deletions .github/workflows/ci.yml

This file was deleted.

42 changes: 42 additions & 0 deletions .github/workflows/ci_exercises.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: CI Exercises

on:
pull_request:
push:
branches:
- main
workflow_dispatch:

jobs:
run-exercises-tests-ubuntu:
name: Run exercise tests for swift ${{ matrix.swift }} on Ubuntu
runs-on: ubuntu-24.04
strategy:
matrix:
swift: ["6.0", "6.1", "6.2"]
container:
image: swift:${{ matrix.swift }}
env:
RUNALL: "true"
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Run tests
run: swift test

run-exercises-tests-macos:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to also test outside of xcode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I fully understand the question. For CI, it’s simple to use Swift from Xcode since it’s pre-installed and comes with Xcode and macOS — version 6.2 for Xcode 26. However, it’s not using Xcode directly to run the tests. Can someone without Xcode run the tests with only the Swift toolchain installed? Probably yes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand is the toolchain different (if I can recall, they use a different build system). I think it would be good for just the exercise test to test if possible both xcode and doing it outside of xcode.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What value would it add to have the tests run twice as part of the CI? The purpose of the CI step is to validate the exercise tests work, not to test the Swift test runner. Adding a second test increases the cost/complexity of the CI. What concrete value does it add?

Copy link
Member

@meatball133 meatball133 Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never mentioned the Swift test runner. The swift in xcode has a different build system compared to if you install just swift. In practice these behaves differently and code are not for graunted cross platform in swift thereby. Since certain libraries doesnt exsist on certain platforms. This in practice if only tested on one platform could mean it doesnt work on another.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the purpose of this CI to validate the tests work or that the build systems work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It suppose to test that the exercises can be built on different build systems. If we just say that if the exercises pass on one system/setup, it should automatically work on every other system/setup, then we don't have to test on xcode/macos at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR After conducting some research, I think it’s overkill to test building an exercise both with and without Xcode. The difference isn’t in the build system (it’s the same — Swift Package Manager), but rather in the dependencies included with different versions of the toolchain: installed via Xcode, Swiftly, or the standalone .pkg installer.

Running tests without Xcode does not work out of the box after installing the toolchain. It throws errors about missing the Testing library. It also does not work — and probably will not — with Swiftly, because that distribution completely lacks the required Testing libraries, which is strange. Additionally, it is missing XCTest, which historically has not been part of Swift and it's expected.

However, the required Testing libraries are included in the .pkg installer, and tests work with some warnings:

swift test --toolchain <path-to-installed-toolchain>

So, for macOS users, I think we should strongly recommend installing Swift via Xcode, which includes everything needed, and probably not even mention Swiftly in the installation guide.

Toolchain differences between Apple and non-Apple platforms

What’s even more frustrating is that the toolchain itself differs between Apple and non-Apple platforms. On Apple platforms (macOS), Foundation comes as a preinstalled Foundation.framework. This means that in your code you can simply write:

import Foundation
  graph TD;
      MacOS[MacOS toolchain]-->FF
      NA[Non-Apple Platform toolchain]-->SCLF
      
      subgraph Apple InHouse
           FF[Foundation.framework]
      end
      FF-->SF
      subgraph GitHub
        SCLF[swift-corelibs-foundation]-->SF
        SF[swift-foundation]-->FICU[swift-foundation-icu]
        SF-->SC[swift-collections]
      end   
Loading

However, on non-Apple platforms, this is not the case. These platforms use a specific lib called swift-corelibs-foundation, which encapsulates platform-specific logic and comes with the Swift toolchain (it is not included for macOS).
There is no single Foundation import that provides everything. This means that if you need XML or networking capabilities, you must import Foundation functionality as separate standalone modules.
As far as I understand, this is something Apple would eventually like to achieve on macOS as well, but the ETA is unknown.
For now, on non-Apple platforms (e.g., Linux), you are able to add the following imports:

import Foundation
import FoundationEssentials
import FoundationXML
import FoundationNetworking
import FoundationInternationalization

It’s hard to imagine someone on a non-Apple platform including any of these imports in an exercise, but it is technically possible. If they do, it will break the build on macOS.
For this reason, I have added one CI job to test building on macOS, assuming GitHub Actions workflows are free for Exercism.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I totally think we should have macOS and Linux ci, and I would like to add that I like the extra Linux ci for the generator, since I have noticed that it behaves differently on macOS and Linux.

I find your conclusion that it is too much work to add non xcode testing, so I think we will just leave it as it is like this. Good job!

name: Run exercise tests on macOS ${{ matrix.macOS }} with Xcode ${{ matrix.xcode }}
runs-on: macos-${{ matrix.macOS }}
strategy:
matrix:
include:
- macOS: '26'
xcode: '26.0'
env:
RUNALL: "true"
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Run tests
run: swift test
52 changes: 52 additions & 0 deletions .github/workflows/ci_generator.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: CI Generator

on:
pull_request:
paths:
- '.github/workflows/ci_generator.yml'
- 'generator/**'
- 'bin/test_generator.sh'
push:
branches:
- main
workflow_dispatch:

jobs:
run-generator-unit-tests:
name: Run generator unit tests
runs-on: ubuntu-24.04
container:
image: swift:6.2
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Run tests
run: swift test --package-path ./generator

run-generator-usage-tests:
name: Test generator usage on Ubuntu
runs-on: ubuntu-24.04
container:
image: swift:6.2
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run generator tests
run: ./bin/test_generator.sh

run-generator-usage-tests-macos:
name: Test generator usage on macOS ${{ matrix.macOS }} with Xcode ${{ matrix.xcode }}
runs-on: macos-${{ matrix.macOS }}
strategy:
matrix:
include:
- macOS: '26'
xcode: '26.0'
env:
RUNALL: "true"
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Run generator tests
run: ./bin/test_generator.sh
47 changes: 33 additions & 14 deletions bin/test_generator.sh
Original file line number Diff line number Diff line change
@@ -1,23 +1,42 @@
#!/usr/bin/env bash

set -eo pipefail
set -euo pipefail

temp=$(mktemp -d)
declare TEMP_DIR
TEMP_DIR=$(mktemp -d)

readonly TEMP_DIR
readonly GENERATOR_DIR="./generator"
readonly PRACTICE_DIR="./exercises/practice"
readonly TEMPLATE_PATH=".meta/template.swift"

swift build --package-path ./generator

for exercise in ./exercises/practice/*; do
if [[ -r "${exercise}/.meta/template.swift" ]]; then
cp -r "$exercise" "$temp"
test_file="${exercise}/$(jq -r '.files.test[0]' ${exercise}/.meta/config.json)"
temp_test_file="${temp}/$(basename "$exercise")/$(jq -r '.files.test[0]' ${exercise}/.meta/config.json)"
swift run --package-path ./generator Generator $(basename "$exercise") "$temp/$(basename "$exercise")"
echo "Comparing $test_file with $temp_test_file"
diff "$temp_test_file" "$test_file"
if [ $? -ne 0 ]; then
exit_code=1
fi
exit_code=0
for exercise_path in "${PRACTICE_DIR}"/*; do
[[ -e "${exercise_path}/${TEMPLATE_PATH}" ]] || continue

cp -r "${exercise_path}"/. "${TEMP_DIR}"

# Minify json and extract test file name
test_file=$(tr -d '\n\r ' < "${exercise_path}"/.meta/config.json | grep -o '"test":\["[^"]*' | sed 's/.*\["//')
original_test="${exercise_path}/${test_file}"
temp_test="${TEMP_DIR}/${test_file}"

exercise_name="${exercise_path##*/}"

if ! swift run --package-path "${GENERATOR_DIR}" Generator "${exercise_name}" "${TEMP_DIR}"; then
printf 'Generation failed for %s\n' "${exercise_name}"
exit_code=1
continue
fi

if ! diff -u -- "${original_test}" "${temp_test}"; then
printf 'Mismatch detected in %s\n' "${exercise_name}"
exit_code=1
else
printf '%s OK\n' "${exercise_name}"
fi
done

exit ${exit_code}
exit "${exit_code}"