DrizzVision’s AI App Testing Ambition Meets Reality

Traditional mobile app testing often falters when user interfaces are redesigned or elements are renamed, forcing developers to spend time fixing test scripts. DrizzVision's Fathom seeks to address this by using visual AI models, allowing the system to interpret what is on screen and continue testing even if elements change or unexpected pop-ups appear. This approach is intended to reduce the brittleness of automated tests, with the company reporting a decrease in "test flakiness" on an internal benchmark.

However, DrizzVision's own benchmark testing highlights the challenge of achieving consistent AI performance in varied conditions. While the AI agent achieved a 72.3% pass rate in an "easier suite" using a standard Android image, its performance fell sharply to 28.7% in a "harder suite" that involved a rooted Android image, more complex tasks, and database inspection. This substantial drop underscores the difficulty of making AI reliably navigate real-world applications.

The benchmark also revealed considerable variation among the tested AI models. For instance, GPT 5.6 Luna's pass rate dropped from 64% in the easier suite to just 4% in the harder one. Conversely, Gemini 3.5 Flash, which scored 54% in the easier suite, achieved 34% in the harder one, outperforming every GPT configuration tested there. This disparity suggests that the choice of underlying AI model significantly impacts performance, especially as testing scenarios become more challenging.