Examples and/or documentation for running iOS tests on a physical device

[jiawen]

I've been using rules_apple successfully for a few projects and can run ios_unit_test in a simulator. However, there doesn't seem to be much documentation regarding how to run tests on a physical device. A great use case would be to easily measure on-device performance.

It would be tremendously helpful to have at least some documentation, if not working examples under the //examples directory.

I'm imagining something along the lines of:

• How to run a single ios_unit_test and ios_ui_test.
• How to run ios_unit_test_suite and ios_ui_test_suite.
• Examples of different runners and test_hosts and what they mean.
  • What's required in a host ios_application.
• Tulsi example.
• How this all fits with google/xctestrunner.

[keith]

I agree the general workflows here should be documented. But the tl;dr is that you need to use Tulsi (or some other Xcode bridge) to do anything on device. Bazel and the rules themselves have no knowledge of how to run on device. (xctestrunner does, but that isn't exposed through the rules today)

[jiawen]

Thanks! The pointer about needing Tulsi is already helpful!

I also discovered today via browsing through the issues that it's possible to get bazel run to execute tests inside a simulator with automated device matching - that's awesome! We should put it in the examples (I'm an Xoogler and happy to contribute).

[keith]

Great! We can definitely start some new document in docs/ with this kinda info

[thii]

@jiawen Can you try this patch #1266?

[jiawen]

@jiawen Can you try this patch #1266?

Sorry this took me so long. I finally mustered the energy to git clone rules_apple and figure out how repository overrides work.

This works! I saw the code signing issue on #1266 and decided to sprinkle debugging into xctestrunner. To run on device, it seems both the test_host (aka app bundle, i.e., the .app directory) as well as each test bundle (.xctest dir) needs to be signed.

Once I added provisioning_profile to ios_unit_test, it all worked!

load("@build_bazel_rules_apple//apple:ios.bzl", "ios_application", "ios_unit_test")
load("@build_bazel_rules_swift//swift:swift.bzl", "swift_library")

ios_application(
    name = "test_harness",
    app_icons = glob(["Assets.xcassets/**"]),
    bundle_id = "com.adobe.nextcam.testharness",
    families = [
        "iphone",
        "ipad",
    ],
    infoplists = ["Info.plist"],
    launch_storyboard = "Resources/LaunchScreen.storyboard",
    minimum_os_version = "14.3",
    provisioning_profile = "//ios:wildcard_mobileprovision",
    visibility = ["//visibility:public"],
    deps = [":lib"],
)

swift_library(
    name = "lib",
    srcs = [
        "AppDelegate.swift",
        "ViewController.swift",
    ],
    data = [
        "Resources/Main.storyboard",
    ],
)

ios_unit_test(
    name = "unit_tests",
    minimum_os_version = "14.3",
    provisioning_profile = "//ios:wildcard_mobileprovision",
    test_host = ":test_harness",
    deps = [":unit_tests_lib"],
)

swift_library(
    name = "unit_tests_lib",
    srcs = ["ExampleTests.swift"],
)

Ship it! I'll use a locally vendored repo until then.

Thanks again!

[DoDoENT]

I did exactly the same and running test fails with

xctestrunner.shared.ios_errors.BundleError: Failed to extract signing identity from Executable

Running the very same test via Xcode generated by rules_xcodeproj works, so there is nothing wrong with my provisioning profile.

[DoDoENT]

OK, that was because I didn't append the --ios_multi_cpus=arm64 flag, so the binary was built for the simulator and not signed.

However, now I'm getting following error:

Traceback (most recent call last):
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 319, in <module>
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 313, in main
    exit_code = args.func(args)
                ^^^^^^^^^^^^^^^
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 151, in _Test
    session.Prepare(
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/xctest_session.py", line 145, in Prepare
    self._xctestrun_obj = xctestrun_factory.GenerateXctestrun()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/xctestrun.py", line 369, in GenerateXctestrun
    self._GenerateTestRootForXctest()
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/xctestrun.py", line 652, in _GenerateTestRootForXctest
    _CopyAndSignFramework(
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/sandbox/darwin-sandbox/59/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/main/hello-world-test-default-ios.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/xctestrun.py", line 742, in _CopyAndSignFramework
    shutil.copytree(src_framework, target_path)
  File "/private/var/tmp/_bazel_dodo/d37a55ce2ed2abf9c807bae8c8d24add/execroot/_main/external/rules_python++python+python_3_11_aarch64-apple-darwin/lib/python3.11/shutil.py", line 571, in copytree
    with os.scandir(src) as itr:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Applications/Xcode-16.4.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/XCUIAutomation.framework'

This appears to be here.