EmceeTestRunner

18.0.0

Star
Emcee is a tool that runs iOS tests in parallel using multiple simulators across many Macs
xcode testing-tools ios xctest multiple-simulators multiple-machines hacktoberfest hacktoberfest2021
avito-tech/Emcee

What's New

Emcee 18 — Resurrecting the Clones and Pluggables

2023-06-23T16:00:07Z

18th release is concentrated on fixing bugs, speeding up things, and separating the functionality between features which become available after purchasing a licence and always-free version.

Emcee now has paid version

Starting from version 17, Emcee is distributed as-is as a closed source project and available in a form of prebuilt binaries. Version 18 goes beyond this and introduces two distinct products: free Emcee and paid Emcee.

Free version (available as a download on Github) allows you to test the product with up to 3 workers, however, you'll need to buy licenses for some richer functionality like:

  • Allowing to use more than 3 workers
  • For a technical support
  • For EmceeAdmin GUI app
  • For getting things done that you want to see in Emcee sooner than later.

Main functionality remains available in free version:

  • For using plugins in your environment
  • For using analytics, such as grafana, statsd, and kibana
  • For a bit richer test discovery flows, such as discovering via runtime

Some notes about the current Github repository

The current source code has been frozen from April 2022, which was version 16. We will keep it available as is, and we're not planning on removing it. All made forks will still be present obviously as well, as we can't control them.

Version 17 is closed source, yet fully functional. You can use it without any limitations as well.

We haven't developed any plans on upstreaming changes from v17 into open source yet, however, we are considering it.

Automatic Job Deletion

This is huge one and helpful. Emcee now tracks if job is needed at all, and automatically deletes it if it is not needed to run anymore. This helps on CI: when you cancel the test run, job gets deleted automatically.
Tech note: previously Emcee client process issued a /jobDelete request which sometimes could get lost; now queue itself tracks all jobs.

Worker Resurrection

Emcee now attempts to restart its workers in case if they die. So if you decided to reboot your machine, or if it kernel panics, it will become a worker automatically.

Simulator Cloning

Emcee worker can now clone simulators instead of booting a new one. This speeds up creation quite a bit, and saves disk space. Emcee will clone any previously shut down simulator it booted. If Emcee never booted suitable simulator, it will create one instead. We advise to use smaller values of automaticSimulatorShutdown and automaticSimulatorDelete settings, like 120 seconds, to have better effect of simulator cloning.
As an example, booting a new simulator can take 1-2 minutes easily, but cloning and booting takes around 40 seconds.

Plugins are back!

You can use the new, a little simplified plugin SDK which is available here:
https://github.com/avito-tech/EmceePluginSupport

It's easier to wire up everything, as there are only two exposed targets now: EmceePlugin and EmceePluginModels.

Boilerplate code has been reduced just to:

let plugin = try Plugin { (event: PluginAppleTestEvent) in
    // process an event
}
plugin.join()

Bugs and small features and changes

  • Emcee now uses socket API to talk to graphite and statsd instead of state of the art OutputStream APIs. Users complained about crashes related to graphite, and this change should help.
  • You can query global queue state via /globalQueueState. Maybe this will be helpful for you to debug how queue is being formed when you submit your jobs.
  • Worker ID is exposed as capability now, allowing you to pin your tests to a specific machine. Capability name is emcee.workerid and the value is actually worker ID.
  • --device and --runtime arguments now accept only fully qualified Core Simulator IDs, e.g. com.apple.CoreSimulator.SimRuntime.iOS-16-4 and com.apple.CoreSimulator.SimDeviceType.iPhone-14. Please refer to xcrun simctl list for more examples.
  • If you want to set environment variables for a test or app while using runTests command, set them in the calling environment with a EMCEE_ prefix.
  • CPU architecture is now exposed as worker capability, allowing you to pin your tests to the machine with specific CPU arch. The key is emcee.cpu.architecture, the value is CPU arch returned by uname -a, e.g. arm64.
  • testFinished event is now correctly delivered to plugins when tests times out.
  • If plugin fails to start, error message will contain its standard output and error for debugging purposes.
  • Emcee now deletes its temporary xcresult storage after test run finishes.
  • You can now specify range of ports that workers can utilize. Queue server config has a new field workerPortRange which controls exactly that. You can also set queue port range via queuePortRange.
  • New command discoverAppleTests lets you run test discovery much easier than before, without composing a test arg file
  • Speaking of test arg files, they are gone from a free version of Emcee. We decided to keep them for paid release only, as they provide more flexibility.
  • We are happy to say Version 18 supports macOS Ventura. Version 17 does not support Ventura.

Join our Telegram channels!

A decent place to discuss knowledge about Emcee and obtain a licence!

Emcee Banner

Welcome to Emcee project, an ultimate solution for running iOS tests in parallel locally and across many Macs.

Emcee allows you to run UI tests on many physical machines, distributing the work and getting the results of the test run faster. Shared queue manages the order of test execution. Emcee workers execute tests and maintain lifecycle of their simulators automatically. Emcee can generate the Junit and trace reports to make you see how the test run behaved on different machines.

Features

  • Rich test plans using simple JSON file format

  • Automatic simulator lifecycle management

  • Per-test timeouts, simulator settings, environment variables

  • Single test queue to run tests from multiple parallel pull requests

  • Prioritized jobs and job groups for different kinds of test runs

  • Load balancing of worker machines to achieve optimal parallelization performance

  • On-the-go maintenance of the workers

  • Integration into existing test management systems via plugins

  • Easy to use command line interface

  • Rich test discovery mechanism

  • Swift Package for using and extending Emcee the way you want

Getting started

In this guide will demonstrate how to use Emcee. We will use two MacOS machines to run unit and UI tests from a sample project. You can also use a single machine to try out Emcee to see if it works for your project. In this case, a single machine will act as a queue and a worker simultaneously. Alternatively, you can scale this guide to as many machines as you have.

If you encounter any issues while proceeding through the guide, please open an issue or reach out via https://t.me/emcee_ios.

Table of contents

  1. Setting up machines
  2. Building the sample project
  3. Running tests using Emcee
    1. Tests without a host application
    2. Tests with a host application
    3. XCUI tests
  4. Advanced Emcee configuration

Setting up machines

You will need to grant SSH access to your machines.

Expand to see how to set up your machines.

We will be using two machines: ios-build-machine77 and ios-build-machine78.

machines

  • ios-build-machine77 will be a worker and a queue - it will provide workers with tests to execute and execute some of those tests.
  • ios-build-machine78 will be a worker - it will only execute tests.

Both machines are set up with a standard non-administrator user emcee and a qwerty password.

Install Xcode and sudo xcode-select --switch /Applications/Xcode.app on all of your machines.

We will use Xcode 14.3 (14E222b) and the iOS 16.4 simulator runtime bundled with this Xcode. If you want to use a specific version of simulator runtime, proceed to Xcode -> Preferences... -> Components -> Simulators and install the runtime on all the worker machines, where you want the tests to execute with the specific runtime version.

Emcee uses ssh to deploy itself to the machines specified as queue and workers. Enable SSH in your System Preferences -> Sharing -> Remote Login. To open this pane execute:

$ open "x-apple.systempreferences:com.apple.preferences.sharing?Services_RemoteLogin"

Remote Login SSH Settings

Now make sure that machines are accessible by ssh. For example:

ssh emcee@ios-build-machine77

If your machines are not accessible by DNS, use their IP addresses instead. You can check IP address in System Preferences -> Sharing. Please note IP addresses may change over time. To open this pane execute:

$ open "x-apple.systempreferences:com.apple.preferences.sharing"

Building the sample project

In this step, we will build a sample project that features different types of tests. Xcode and xcodebuild will produce build artifacts in derived data.

Expand to see how to build the sample project for testing purposes.

You can run this step from either machine. Clone the sample project:

cd ~
git clone https://github.com/avito-tech/Emcee.git
cd Emcee/Samples/EmceeSample

To build the project, create a simulator:

xcrun simctl create '16.4' 'iPhone 14' 'iOS16.4'

Now run xcodebuild:

xcodebuild build-for-testing \
	-project EmceeSample.xcodeproj \
	-destination "platform=iOS Simulator,name=16.4,OS=16.4" \
	-scheme AllTests \
	-derivedDataPath derivedData

Xcodebuild will place the build products in:

derivedData/Build/Products/Debug-iphonesimulator

Running tests using Emcee

Now that the machines are ready, and the project is built, download Emcee on the same machine where you built the project by running:

curl -L https://github.com/avito-tech/Emcee/releases/download/18.0.0/emceeFree.artifact.zip -o emceeFree.artifact.zip && unzip emceeFree.artifact.zip emceeFree

With Emcee installed it is finally time to run the tests. The sample project includes 3 test types:

  • Tests that don't require a host application
  • Tests that require a host application
  • XCUI tests

Tests without a host application

Let's first run tests that don't require a host application. We will be using the runTests command:

./emceeFree runTests \
	--queue "ssh://emcee:qwerty@ios-build-machine77" \
	--worker "ssh://emcee:qwerty@ios-build-machine77" \
	--worker "ssh://emcee:qwerty@ios-build-machine78" \
	--device "com.apple.CoreSimulator.SimDeviceType.iPhone-14" \
	--runtime "com.apple.CoreSimulator.SimRuntime.iOS-16-4" \
	--test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleTestsWithoutHost.xctest \
	--junit tests_without_host_junit.xml

Here is what these options stand for:

  • --queue - is a URL of a machine that will serve workers with tests
  • --worker - is a URL of a machine that will execute tests that it queries from the queue
  • --device and --runtime - are options that specify which simulators will run the tests
  • --test-bundle - is a path to the xctest bundle
  • --junit - is a path to the JUnit xml that will contain the result of the test run

You can find more about all options accepted by runTests and how to specify them using Emcee runTests -h.

After the test finishes, Emcee will create a tests_without_host_junit.xml file. The JUnit report contains four testcase entries matching the four test methods from the EmceeSampleTestsWithoutHost.xctest test bundle.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="xctest" tests="4" failures="1">
    <testsuite name="EmceeSampleTestsWithoutHost" tests="4" failures="1">
        <testcase classname="EmceeSampleTestsWithoutHost" name="test_0___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:16+03:00" time="0.012196063995361328"></testcase>
        <testcase classname="EmceeSampleTestsWithoutHost" name="test_1___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:27+03:00" time="0.02156198024749756"></testcase>
        <testcase classname="EmceeSampleTestsWithoutHost" name="test_2___from_tests_without_host___that_always_succeeds" timestamp="2021-12-29T01:15:36+03:00" time="0.021990060806274414"></testcase>
        <testcase classname="EmceeSampleTestsWithoutHost" name="test___from_tests_without_host___that_always_fails" timestamp="2021-12-29T01:15:31+03:00" time="0.1255110502243042">
            <failure message="failed - Failure from tests without host">/Users/emcee/Emcee/SampleProject/EmceeSampleTestsWithoutHost/EmceeSampleTestsWithoutHost.swift:17</failure>
            <failure message="failed - Failure from tests without host">/Users/emcee/Emcee/SampleProject/EmceeSampleTestsWithoutHost/EmceeSampleTestsWithoutHost.swift:17</failure>
        </testcase>
    </testsuite>
</testsuites>

For a more sophisticated test reporting mechanism such as Allure, check out the Plugins documentation.

Tests with a host application

Now let's try running tests that require a host application. Host application path is specified using the --app option. For example:

./emceeFree runTests \
    --queue "ssh://emcee:qwerty@ios-build-machine77" \
    --worker "ssh://emcee:qwerty@ios-build-machine77" \
    --worker "ssh://emcee:qwerty@ios-build-machine78" \
    --device "com.apple.CoreSimulator.SimDeviceType.iPhone-14" \
    --runtime "com.apple.CoreSimulator.SimRuntime.iOS-16-4" \
    --app derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app \
    --test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app/PlugIns/EmceeSampleHostedTests.xctest \
    --junit tests_with_host_junit.xml

To get a visual confirmation that Emcee is running the tests, you can open the Simulator app on the worker machines:

open "$(xcode-select -p)"/Applications/Simulator.app

XCUI tests

Finally, we will run XCUI tests by adding a --runner option and changing the --test-bundle option to the XCUI test bundle:

./emceeFree runTests \
    --queue "ssh://emcee:qwerty@ios-build-machine77" \
    --worker "ssh://emcee:qwerty@ios-build-machine77" \
    --worker "ssh://emcee:qwerty@ios-build-machine78" \
    --device "com.apple.CoreSimulator.SimDeviceType.iPhone-14" \
    --runtime "com.apple.CoreSimulator.SimRuntime.iOS-16-4" \
    --runner derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app \
    --app derivedData/Build/Products/Debug-iphonesimulator/EmceeSample.app \
    --test-bundle derivedData/Build/Products/Debug-iphonesimulator/EmceeSampleUITests-Runner.app/PlugIns/EmceeSampleUITests.xctest \
    --junit ui_tests_junit.xml

This is how the test run will look:

running_tests

Advanced Emcee configuration

Complete documentation is available in our Wiki.

runTests command allows you to get Emcee up and running quickly; however, it doesn't allow for a lot of configuration. On the other hand, runTestsOnRemoteQueue command allows for fine-grained control of how your tests execute. To get started with runTestsOnRemoteQueue check out the Queue Server Configuration and Test Arg File wiki pages.

Publications

Getting Around the Code

Emcee uses Swift Package Manager for building, testing and exposing the Swift packages.

To start exploring code open Package.swift in Xcode 13 or execute make open to generate and open Xcode project.

Contributing

We are happy to accept your pull requests. If something does not work for you, please let us know by submitting an issue. Read the docs and suggest improvements to them as well!

General commands that help you with a development workflow:

  • To open a package in Xcode: make open
  • To generate Package.swift: make package
  • To build the binary into .build/debug/Emcee: make build
  • To run unit tests: make test

Package.swift file is generated automatically. You must update it before submitting a pull request (run make package). CI checks will fail if you forget to do so.

Description

  • Swift Tools 5.3.0
View More Packages from this Author

Dependencies

Last updated: Mon Oct 02 2023 09:57:46 GMT-0900 (Hawaii-Aleutian Daylight Time)