Introduction

If you’ve ever tried to automate tests for a legacy application, you’ve probably found yourself wondering: “Why is this thing fighting me?” You’re not alone.

Legacy systems—those decades-old desktop apps or clunky enterprise tools—often come with no APIs, no modern frameworks, and no straightforward way in. They’re like black boxes, but with more bugs and less documentation.

Traditional test automation assumes you have access: APIs, DOM trees, or structured element hierarchies. Legacy apps typically offer none of that. So how do you test them without rewriting or reverse-engineering the whole thing?

Instead of forcing your way in, let your tools observe and interact with the UI the same way a human tester would – by using visual recognition powered by AI, along with keyboard and mouse simulation.

Why Traditional Automation Doesn’t Cut It

Most testing frameworks rely on technical access to the application – reading UI elements, triggering events, or calling APIs. That works well for modern software.

Legacy systems are another matter.

You may encounter:

• Custom UI frameworks that don’t expose any element data
• Pixel-based rendering where buttons are nothing more than painted pixels
• Platforms that predate the concept of automated testing
• Environments where a small change requires months of change control

You often can’t inspect the UI, can’t reach inside, and sometimes can’t even interact with the application safely in production. That’s where a visual, non-invasive approach becomes valuable.

Our automated testing and monitoring tool supports companies in efficiently, transparently, and audit-proof implementing the technical testing processes required by DORA.

The Visual Recognition Approach

This method flips traditional automation on its head. Rather than digging into the application internals, it simply looks at the screen and interprets what’s there – just like a human would.

The process:

• Capture the screen – Take a screenshot of the application window.
• Recognize UI elements – An AI model trained on thousands of UI examples detects components like buttons, fields, and labels.
• Simulate interaction – Using mouse and keyboard input, the tool clicks and types to navigate the application – no internal access required.

Why This Works

Real-World Use Cases

This approach fits in environments such as:

• Insurance systems from the early 2000s – or earlier
• Government platforms that can’t be modified without a procurement process
• Legacy ERP and finance apps without integration options
• Internal tools built by teams that no longer exist

In each of these cases, automated testing is necessary – but traditional tooling has no point of entry. Visual recognition fills that gap.

Low Setup, Minimal Disruption

Getting started doesn’t require a refactor or new infrastructure.

If you have:

• Access to the screen (direct display or capture)
• Ability to send keyboard/mouse input
• An AI model (off-the-shelf or custom-trained)

…then you can start automating.

This can often be quicker and more practical than forcing internal integrations onto legacy software

What About Mobile?

This approach works on mobile apps as well – without needing emulators or rooted devices.

Most modern Android and iOS devices support video output. Connect to a capture card or compatible display and you get real-time screen output for visual analysis.

Input can be simulated via touch or keyboard events. As long as the screen is visible and the device responds to user input, it’s testable – no developer mode required.

Final Thoughts

Legacy systems are deeply embedded in critical workflows across industries – and they’re not going away anytime soon. But until recently, testing them has been a major challenge.

With AI-powered visual recognition and non-invasive input control, you can now test legacy applications without modifying or accessing their internals. By treating the app as a user would – seeing the UI, recognizing components, and interacting through clicks and keystrokes – you can build meaningful test coverage, even for the most opaque systems.

Drvless Automation enables this out of the box: pre-trained AI models that understand user interfaces, combined with full keyboard and mouse interaction across desktop and mobile platforms. No plugins, no SDKs, and no code access required. Additionally, a hardware solution is available that connects directly to HDMI and USB ports, capturing screen output and injecting input signals at the hardware level – allowing testing of systems that are otherwise completely locked down or isolated from software integration.

If your application is a black box, Drvless doesn’t force it open. It observes, understands, and interacts – quietly and effectively.

---

Author: Theodor Hartmann (Product Manager)

Theodor Hartmann began his journey in software testing in 2000 as an intern. Over the past 20 years, he has gained extensive experience across various industries, including insurance, telecommunications, and banking. With a passion for the technical aspects of testing, he enjoys uncovering defects and exploring the philosophical questions surrounding the purpose of testing, while staying curious about the constants in testing amid the evolving landscape of new technologies.