The Mobile Testing Pyramid
Mobile Test Pyramid
- Real devices
- Mobile simulators & emulators
- Desktop browsers (using mobile simulation)
Desktop browsers: mobile testing on desktop browsers
|Functional system testing||Isolated browser tests performing full functional validations|
|Responsive design||Resizing browsers and toggling user agents|
|Cross-browser||Use equivalent desktop browsers|
|Overall visual layout||No extensive visual checks because the rendering is different than devices|
- Fast execution: Matter of milliseconds to launch a browser, also headless execution is possible.
- Scalable: Easily set up 10+ browser instances per machine.
- Cross platform: Ability to use browsers on different operating systems.
- Mobile simulation uses the desktop browser engine: Mobile simulation in desktop browsers is still using the desktop browser.
- No native integration: No native keyboards, incoming calls, etc.
- Just not a device… Incredibly fast, but still not a real device.
Mobile simulators/emulators: closer to the actual mobile experience…
|Functional end-user flows||Click paths throughout the application|
|Native API integration||GPS injection, file attachments, incoming calls etc.|
|Visuals (vanilla only)||Use equivalent desktop browsers|
|Overall visual layout||Emulators are limited to vanilla versions|
|Touch interactions||Touch interactions such as swipe and tap comes closer to the user experience of a device than browser emulation|
- Easy to set up: Simulators/emulators are easy to set up, just download, install, and run.
- Scalable: Virtualization means scalable and also running in parallel on one machine.
- Native API integration: Ability to test native APIs such as incoming calls and GPS injection.
- Simulators or Intel-based emulators are fast: Simulators are fast, because they only have to simulate the software part. Emulators based on the Intel architecture are fast.
- Debugging possibilities: Easy to debug simulator/emulators, already hooked up to the machine to access logs.
- Vanilla versions only: Manufacturer’s skins are available, but the device behavior is still based on what comes stock.
- No real resource usage: CPU/memory usage of machine in case of simulators. Emulators try to simulate the hardware.
- No real interoperability: Connectivity with NFC, Bluetooth, network connections.
- Slow ARM-based emulators: Emulators based on the ARM architecture are slow, which is the main architecture for Android devices.
- Inaccurate color display in light/dark: Contrast/brightness inaccurate in light/dark environment.
Real devices: the real thing…
|Usability||Validating usability such as actual click areas, touch actions and voice over|
|Performance||CPU/memory usage, battery, network strengths|
|Native API integration||Interruption (incoming calls, push notifications), resource fighting (camera, GPS), NFC, Bluetooth|
|Visuals||Focus on devices which are not available as simulators/emulators|
|Manufacturer’s sauce||Real OS from manufacturers, e.g. Samsung’s TouchWiz and built-in browsers|
- Native APIs in real conditions: Ability to test native APIs not only with injections for automation, but also actual NFC touch for example.
- Can be faster than emulators: Some real devices are just faster than emulators due to the simulation of hardware, especially compared to the ARM-based emulators.
- Just the real thing… Actual network conditions, battery/CPU/memory usage, manufacturer’s secret sauce on top of the OS.
- Costs: Real devices come with a price, usually you pay per device/cradle.
- New device means procurement: A new device is usually not available on-the-fly, even with cloud solutions. E.g. when the new iPhone comes out, it’s not available immediately to procure. In the meantime, the iOS simulator would already be available.
- Development iOS build required for automation: iOS apps need to be signed with Development Distribution Certificate and Provisioning Profile for automation.