Testing for a team

Set up a tester's Mac and iOS device from scratch and run the team's flows without the app's source code.

A tester sets up a Mac and an iOS device and runs the team's flows without the app's source code. The team hands over a folder of flows and uploads a QA build of the app to the team's builds. The Mac setup applies to each Mac, and the iOS device setup to each iOS device.

Before you start

Who	Provides
The tester	A Mac, an iOS device, a free Apple ID
The app team	The flows folder, an uploaded QA build of the app that includes the tester's iOS device, and any secret values the flows need
The owner of the Apple Developer account	Registers the tester's iOS device so QA builds can be installed on it

A simulator needs only the first two steps of the Mac setup. The sections after that are for an iOS device.

1. Set up the Mac

1. Install Astra Scout from the DMG. See Installation. Later versions arrive through the app itself: Settings > General > Updates has Automatically check for updates (the app checks once a day and asks before it installs), and Astra Scout > Check for Updates… checks at once.
2. Install Xcode from the App Store, open it once and let it install its components.
3. In Xcode > Settings > Accounts, add your Apple ID. A free Apple ID is enough. Select the team, then click Manage Certificates…, then + > Apple Development.
4. In Astra Scout, choose Astra Scout > Settings… (Cmd+,) and click iOS in the sidebar. The Signing Team list shows Automatic and each team found in your Keychain. Click the row of your team. A check mark shows the choice.
5. If your team is not listed, check that the certificate from step 3 is created, then click Rescan. Other Team ID… lets you type a Team ID by hand.

Astra Scout uses this team only to sign WebDriverAgent, the helper app that drives the iOS device. A free Apple ID's signature lasts 7 days; when it expires, connect the iOS device again and Astra Scout signs it again. The QA build of the app is signed by the app team, not by you. See Devices for details.

2. Set up the iOS device

1. Connect the iOS device with a cable, unlock it and tap Trust when it asks "Trust This Computer?".
2. In Xcode, open Window > Devices and Simulators, select the iOS device and wait until it is ready.
3. On the iOS device, turn on Settings > Privacy & Security > Developer Mode. It restarts; unlock it and confirm.
4. Set Settings > Display & Brightness > Auto-Lock > Never while testing.
5. Send the UDID of the iOS device to the team. The UDID is the id Apple uses to allow an iOS device to install the team's builds: a QA build installs only on iOS devices whose UDID is in its provisioning profile. In step 4 of Open the workspace, click Send UDID… in the Builds window to send it. If you cannot send it from Astra Scout, copy the Identifier of the iOS device in Devices and Simulators and send it to the owner of the Apple Developer account. The app team makes a new build after your iOS device is added. See Builds.

3. Get the flows and the QA build

Ask the app team for:

• The flows folder: the .yaml flows, the subflows/ folder and the tracking-specs/ catalog. Keep the whole folder together: flows call subflows by relative path. When the team uses Team Sync, choose Clone Team Workspace… on the Welcome to Astra Scout window instead of copying the folder.
• Secret values the flows use, such as a sandbox account password. Enter them as variables, never in the flow files. See Environments and variables.

The QA build needs no file transfer. The app team uploads it to the team's builds, and you choose it in Astra Scout in the next step. A TestFlight or App Store build cannot be used for clearState. See Share builds with testers.

4. Open the workspace and choose the build

1. In Astra Scout, choose File > Open Workspace… and pick the flows folder. On the Welcome to Astra Scout window, Open Workspace… does the same, and File > Open Recent lists the folders you opened before. See Run flows.

   

2. Connect the iOS device: do steps 1 and 2 of Connect and run.

3. Choose Window > Builds (Option+Cmd+B). Sign in with your Google account if the window asks.

4. Select the app's project, then iOS, then a build. In the inspector on the right, under the name of the connected iOS device, the label In the profile or Not in the profile says whether the iOS device can install this build. If it says Not in the profile, click Send UDID… and wait for a new build. See Builds.

5. Click Use This Build.

Flows that start with clearState uninstall the app and install the build in use again, so every run starts like a fresh install. The build stays in use until you choose another. Astra Scout notifies you when a new build arrives. See Builds and clearState on an iOS device.

5. Connect and run

1. In the toolbar, click the device picker (it reads Select a device until you choose one). Type in the search field if the list is long, and click the iOS device. The first time, Astra Scout builds and installs WebDriverAgent. The mirror area shows the steps; if the screen is locked it says "Waiting for the iOS device…". Unlock the iOS device.

   

2. If the connection fails, the mirror area shows the error with Reconnect and Disconnect. For a signing error it also shows Open iOS Settings…, which takes you to the Signing Team list (see step 4 of the Mac setup).

   

3. If the iOS device shows "Untrusted Developer", go to Settings > General > VPN & Device Management, choose the Apple Development: … certificate, tap Trust, then connect again.

4. Open a flow and choose Run > Run Test (Cmd+R), or Run > Run All Tests (Shift+Cmd+R).

6. Read the results

• The Test Runs navigator shows each run as Passed, Failed or "Running test". Click the flow row to see its verdict (Passed, Failed, Incomplete, Unsure or Blocked) in the detail pane. Blocked means a problem with the device or the setup, not a bug in the app. See Verdicts.

  

• Click a run to see its steps. Show Recording (in the row's detail and its context menu) plays the video of the run; screenshots taken by the flow are kept with the run.

  

  

• If the flow checks tracking, the runner's Tracking tab shows which events arrived and which are missing or wrong. See Tracking.

  

When you report a problem, attach the run's report, the video and the server log. Choose Help > Show Server Log and copy the content of the log. Astra Scout empties this log every time it starts, so copy it before you reopen the app.

Troubleshooting

Symptom	Fix
No events arrive from the iOS device	Check each cause below. The build in use in Window > Builds is not the QA build: select the QA build and click Use This Build. Receive events from devices on the network in Settings > Amplitude is off: turn it on, then click Restart Server Now to apply it. The Mac and the iOS device are not on the same Wi-Fi: join both to the same network. The app has no Local Network access: allow it when iOS asks the first time the app opens.
"No build chosen for an iOS device" (APP_FILE_REQUIRED)	No build is in use for this iOS device. Click Open Builds…, then Use This Build on an iOS build. If the inspector says Not in the profile, the UDID of the iOS device is not in the provisioning profile: click Send UDID… and wait for a new build.
WebDriverAgent stops opening after a week	The free Apple ID's signature expired. Connect the iOS device again.
Taps on the mirror do nothing	Choose the iOS device again in the device picker (or Reconnect).
"iOS device screen is locked" or DEVICE_LOCKED	Unlock the iOS device and set Auto-Lock to Never. See When the iOS device locks itself.

More problems are listed in Troubleshooting.

For the app team

The app team prepares the QA build following Send QA build events to Astra Scout: code, Xcode setup, simulator and iOS device builds. It uploads the builds as Share builds with testers describes. The requirements:

• The QA build sends its events to Astra Scout: it reads scoutServerUrl, which Astra Scout passes when it opens the app, and uses it as the Amplitude serverUrl. One build works for every tester.
• Build for testing, signed for development or ad hoc, not TestFlight or the App Store, so testers can install it and clearState can reinstall it.
• Register each tester's iOS device (UDID) in the Apple Developer account before building. A development or ad hoc profile holds up to 100 iOS devices per year and expires after one year.
• Share the flows folder (for example as a Git repository the testers can pull) and tell testers which variables to set.