Photo by Simon Kadula on Unsplash

Here are some scenarios in which input simulation is a great tool to have on your side:

1. Automated Testing — simulating user input is a core function for developing a completely automated
   system testing suite.
2. Product Validation and Qualification — providing documentation records like logs, photos, and videos about the result of some user action is way easier to manage if user input is entirely automated.
3. Regression Testing — verifying that updates with new features will not introduce bugs in a system is critical when updates are frequent.
4. Stress Testing — programs can simulate rapid and repetitive inputs that are useful to detect if a system reacts correctly.
5. Stability Testing — uninterrupted operation is a mandatory requirement for systems that need to operate reliably over long periods.
6. Benchmarking — using a program to interact with a system is an objective and repeatable method of evaluating performance.

Use cases like latency measurement are even impractical for humans to perform without additional tools. As such, more often than not, a program is frequently used to both trigger and measure the response time.

Let’s look at a few solutions used to automate user interactions.

Software methods for user input simulation

Computers and laptops

PyAutoGUI is a Python module that allows you to control the mouse and keyboard programmatically. It is one of the most popular packages for simulating mouse movement, clicks, and keyboard presses.

It works on Windows, Linux, and macOS, with the only limitation being that it is only suitable for single monitor setups.

import pyautogui
    
pyautogui.moveTo(100, 100, duration=1.)

winkey = pyautogui.locateOnScreen( 'windows_button.png' )
winkey = pyautogui.center( winkey )
pyautogui.moveTo( *winkey, duration=1.)
pyautogui.click( *winkey )

pyautogui.write( 'PyAutoGUI', interval=1e-1 )
pyautogui.screenshot( 'pyautogui_result.jpg' )

Native desktop applications

Pywinauto is a Python library for automating Windows GUI applications. It can automate interactions with native Windows controls, such as buttons, menus, and dialog boxes.

Pywinauto Github stats

Robot Framework is a versatile, open-source test automation framework that executes and manages tests. With the help of the AutoIt library, it becomes capable of automatically interacting with Windows apps.

Robot Framework's getting started guide

Web applications

Puppeteer is a headless browser automation tool developed by Google. It provides a high-level API for Node.JS to control Chromium-based web browsers. It is a popular solution for generating webpage screenshots and automating web-related tasks.

const puppeteer = require( 'puppeteer' );
const url = 'https://www.adaptarobotics.com';

async function run () 
{
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto( url );
    await page.screenshot({ path: 'screenshot.png' });
    browser.close();
}

run();

Selenium is an open-source framework built from the ground up to automate interactions with web browsers. It can navigate through webpages, fill out forms, click buttons, and wait for elements to appear. Apart from web application testing, Selenium has various uses in web scraping and automation.

Selenium website

Android smartphones

AndroidDebugBridge (ADB) is a versatile and powerful command-line tool for interacting with Android devices. It can automate user interactions, including debugging and testing Android applications.

Here’s how ADB can be useful for testing Android devices:

# turn on device
adb shell input keyevent 26
# tap a position
adb shell input tap 757 1694
# get a screenshot
adb exec-out screencap -p > screen.png

iOS devices

XCTest is Apple’s official testing framework for Swift and Objective-C apps. It is integrated directly into Xcode and supports unit testing, graphical user interface testing, and performance testing.

XCTest documentation

Mobile and native applications

Appium is an open-source tool for automating app-user interactions for many platforms, including Android TV and tvOS. Its main advantage is that it offers cross-platform compatibility and a unified way of writing tests.

Hardware methods for user input simulation

The software methods described above require a separate program to run on the device under test (DUT) and handle the automation protocol. At first hand, an automation protocol such as ADB may not be available for every device. Secondly, executing the program on the same processor will interfere with its performance, which is challenging for benchmarking and performance measurement use cases. As such, we must rely more and more on hardware solutions and black-box testing.

Black box testing is a methodology in which the internal structure, design, and implementation details of the device under test are considered unknown. In this case, the testing protocol must verify the functionality and behavior of the device from an external perspective.

Computers input simulation

The USB Rubber Ducky appears to be an ordinary flash drive, but it is a tiny computer that can run commands on the target device. It does this by pretending to be a keyboard and can type commands that automatically execute when plugged in. You know what we are talking about if you have watched Mr. Robot.

Hak5's rubber ducky

Correspondingly, any microcontroller that could present itself as an HID device should work for the job, even the Arduino.

Many gaming mice nowadays have programmable buttons and support for scripting and macros.

The scripting feature can automate user input, allowing the mouse to perform a sequence of actions with just one button press. An example would be the Logitech G Pro X Superlight gaming mouse that supports LUA scripting.

Logitech G Hub macro editing

Smartwatch testing

For smartwatch testing, there are some input actions that hardware devices excel at:

• Pressing buttons: electromechanical linear actuators can simulate taps on lateral devices
• Control charging: electronic switches like relays can turn on and off power to the charging cable for battery life testing

Custom devices and communication testing

Simulating user interactions for testing purposes on custom devices like automotive infotainment systems, medical devices, consumer electronics, and smart home panels can also be implemented via communication messages.

Dedicated tools like a programmable CAN bus transceiver can allow automated tests to send and receive data to and from the device under test.

Testing Robots

They are built to automate a wide range of testing scenarios, addressing challenges that neither of the previous mentioned solutions can manage effectively. Offering continuous testing through physical interactions with devices, robots enhance the automation of testing processes. This is particularly beneficial for manufacturers, developers, or solution providers who are dealing with the growing complexity of devices, applications, features, and updates., excelling anywhere along the testing process be it R&D, performance verification, connectivity or end-of-line testing.

Takeaways

There are many methods to test and programmatically simulate user actions on any device. Testing via input simulation is a versatile and practical approach for assessing the functionality and user experience of devices with touch screens or buttons in a plethora of scenarios. It helps to ensure that the device or application performs as expected under various conditions and input methods.