Using Xcode Test Plans to Stabilize CI Test Runs

Problem: Flaky CI because test environments differ

When a test suite passes on a developer’s laptop but fails in Xcode Cloud or a CI server, the root cause is often a mismatch in settings such as language, locale, or enabled sanitizers. Manually keeping those options in sync across multiple machines is error‑prone and makes reproducing a failure difficult.

Thesis: Xcode Test Plans give a single source of truth for test configuration

A Test Plan (.xctestplan) is an editable file that bundles test targets together with configurable options like execution order, language, locale, address sanitizer, thread sanitizer, and parallelism. By committing the plan to source control, every developer and CI system reads the same settings, eliminating environment drift.

Creating and assigning a Test Plan

1. Open the project in Xcode (File → New → Test Plan).
2. Choose a name, e.g., UITests.xctestplan, and save it inside the project folder.
3. Select the project file, go to the Test tab of the target that contains your UI tests, and pick the newly created plan from the Test Plan popup.
4. The plan now appears under the target in the Test Navigator (⌘6).

At this point, any Product → Test run uses the plan’s settings.

Worked example: Run Address Sanitizer only for UI tests

Suppose you want ASan enabled for UI tests but not for unit tests, because ASan adds overhead that slows down quick unit‑test feedback.

1. Create two test plans: UnitTests.xctestplan (default config) and UITests_ASan.xctestplan.
2. Open UITests_ASan.xctestplan, enable Address Sanitizer under the Diagnostics tab, and optionally set a specific locale.
3. Assign UnitTests.xctestplan to the unit‑test target and UITests_ASan.xctestplan to the UI‑test target via each target’s Test tab.
4. Commit both .xctestplan files.

To verify locally, run the UI tests and inspect the console output. You should see flags like -fsanitize=address in the compile commands or runtime messages such as ==12345==ERROR: AddressSanitizer:.

Using the plan in CI with xcodebuild

In a CI script (or Xcode Cloud workflow) you can invoke the plan directly, ensuring the same configuration is used without passing a long list of -destination or -sanitize flags.

# Run UI tests with the ASan plan from the command line
xcodebuild test-without-building \
  -scheme MyApp \
  -testPlan UITests_ASan.xctestplan \
  -destination 'platform=iOS Simulator,name=iPhone 14,OS=latest'

Where to run: Any terminal with access to the project checkout and Xcode command‑line tools (requires read permission on the repo).

Expected check: The build log should contain a line like Test Plan: UITests_ASan.xctestplan and the ASan runtime output mentioned above.

Risk: If the .xctestplan file is not added to the target’s resources (visible in the target’s Build Phases → Copy Bundle Resources), Xcode will ignore it and fall back to default settings. Verify by opening the target’s resources list.

Limitations and practical verification

• Test Plans are only honoured by Xcode’s native test runner and Xcode Cloud. Tools like fastlane or custom scripts that call xcodebuild test without the -testPlan flag will ignore the file.
• Older Xcode versions (< 11) do not understand the .xctestplan format; CI images must run Xcode 11 or newer.
• Parallel test execution (enabled in the plan) can increase memory pressure. Monitor CI machine usage; if you see frequent OOM kills, reduce the parallelism count in the plan.

Practical way to confirm the plan is active:

1. Run xcodebuild -showTestPlan -scheme MyApp -testPlan UITests_ASan.xctestplan to see the resolved settings printed as JSON.
2. In Xcode, open the Test Navigator, run a test, and look at the Test tab in the inspector (⌥⌘4) – it lists the active plan name.

Actionable closing

Start by adding a Test Plan for the test suite that fails most often in CI. Commit the plan, point the target to it, and replace any ad‑hoc xcodebuild flags with the -testPlan argument. Verify locally that the expected diagnostics appear, then push the change and watch your CI runs become more deterministic. If you encounter resource spikes, adjust the parallelism setting in the plan and re‑check the CI logs.