Running OCUnit (or Specta) Tests from Command Line

OCUnit (SenTestingKit) and test frameworks built on top of OCUnit such as Specta and Kiwi are tightly integrated with Xcode and it is super easy to set up and run tests within the IDE. However, getting it to run from the command line (for example, to automate testing in continuous integration servers) is not quite as straight forward. As far as I know, there is no official support from Apple for running tests from the command line, but it can be done.

This guide assumes the latest version of Mac OS X and Xcode at the time of writing, which is 10.7.3 and 4.3.2 respectively.

Running Tests for Mac OS X Projects

Running tests for Mac OS X projects is actually fairly simple with the use of xcodebuild command line tool. The following command cleans, builds, and then runs the tests for the MyApp scheme in your Xcode project.

xcodebuild -scheme MyApp clean test

Omit clean if you do not want to rebuild the entire codebase.

If your project is contained inside a Xcode workspace, you can specify the workspace with a -workspace option. You probably want to do this if you use CocoaPods.

xcodebuild -workspace MyApp.xcworkspace -scheme MyApp clean test

Note that the return code of the tests will always be 0 regardless of whether the test succeeds or fails, so you will have to detect the presence of ** TEST SUCCEEDED ** text in the output.

Running Tests for iOS Projects

You would naturally assume that you can use the same command for iOS projects, but in reality, that is unfortunately not the case. The command, when run, will will fail with the following error message:

xcodebuild: error: Failed to build workspace MyApp with scheme MyApp.
        Reason: The run destination My Mac 64-bit is not valid for
Testing the scheme 'MyApp'.

The run destination should be “iPhone Simulator” and not “My Mac 64-bit”, but I could not figure out how to get xcodebuild to use the correct run destination.

According to an Apple employee at Apple Developer Forums, xcodebuild in Xcode 4 does not support unit testing iOS targets yet. It is not entirely true though, as it is actually possible to run test with xcodebuild, albeit requiring some extra steps.

The first step is to create a new Scheme for your test target. This lets you use build action instead of test action to run your tests.

New Scheme...

Make sure you choose the test bundle target and not your app, and then click “OK”.

Test Target

Once the new scheme is created, navigate to “Product” → “Edit Scheme...” and have the checkbox for “Run” that belongs to the test target checked, then click “OK”

Run Checkbox

Now, open up the target configuration screen, select the test target, select “Build Phases” tab, and make sure that the “Run Script” phase is the last item in the list. You can also uncheck “Show environment variables in build log” to reduce garbage that gets printed out when you run xcodebuild.

Build Phases - Run Script

You can then attempt to run the tests via the newly created scheme, using the following command (add the -workspace option if needed):

killall -m -KILL "iPhone Simulator"
xcodebuild -scheme MyAppTests -sdk iphonesimulator TEST_AFTER_BUILD=YES clean build

You will then see that even though the build has succeeded, the test has failed to run with the following error message:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:95: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

The error message claims that the iPhoneSimulator platform does not currently support applicated-hosted tests, but how can it be true considering how they run so perfectly under Xcode? Well, turns out, this is because the shell script that is getting invoked is outdated, and this can easily be patched.

First, open the file that is causing the error in your favorite text editor (it requires sudo access):

sudo vim `xcode-select -print-path`/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests

And change line 95 from:

Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."

to the following:

export CFFIXED_USER_HOME="${BUILT_PRODUCTS_DIR}/UserHome/"
mkdir -p "${CFFIXED_USER_HOME}"
mkdir -p "${CFFIXED_USER_HOME}/Library/Caches"
mkdir "${CFFIXED_USER_HOME}/Library/Preferences"
mkdir "${CFFIXED_USER_HOME}/Documents"
export OTHER_TEST_FLAGS="${OTHER_TEST_FLAGS} -RegisterForSystemEvents"
RunTestsForApplication "${TEST_HOST}" "${TEST_BUNDLE_PATH}"

You might have to repeat the above step when Xcode is reinstalled or upgraded. Once the edit is done, save and try running the tests (again, add -workspace option if needed):

killall -m -KILL "iPhone Simulator"
xcodebuild -scheme MyAppTests -sdk iphonesimulator TEST_AFTER_BUILD=YES clean build

Voilà! The tests should have run. The output text will contain a lot of garbage, including something that goes along the lines of “Preceding build task claims to succeed in spite of generating error messages. Please file a bug report.”, but that message can be safely ignored.

The command above also appears to generate non-zero return code when the test fails, but it might be more reliable to detect the presence of ** BUILD SUCCEEDED ** text in the output.

JUnit XML Output

If you use Jenkins CI, you may wanna check out OCUnit2JUnit, which converts the output format of OCUnit to the XML format used by JUnit.

Good Luck!

Although it takes many steps to get specs running, once set up, it works pretty well. It would be really cool if Apple gets around to fixing xcodebuild command line tool to run tests for iOS projects out of the box, but the bug has been around for quite some time, so don’t get your hopes up. Filing a bug report in Apple’s Radar will really help though. Good luck!

Journey: The Unofficial Path Client for Mac

The super secret side project I’ve been working on with the guys behind Denso - Arun, Kent, and AJ is now up and it is called Journey for Mac.

The app is open-source, and you can download its source code here. Please bear in mind that this started as a hackathon project and many decisions were made based on the speed of implementation, so you may find things that are rough around the edges.

We’ve tried to test-drive the app as much as possible with Specta and Expecta.

You can learn more about the story behind its creation here.