- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.

- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.

- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.

The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.

Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.

Feel free to take a look at the full writeup for more detail!

We parallelize testing across a massive fleet of browsers/devices so it should actually be much faster than Claude. The open source benchmarks prove it :)

Actually a lot of our mobile users came from Maestro, we hear the maintenance burden is a nightmare.

1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)

2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.

3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.

4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.

1. It should be pretty efficient using prior knowledge from other runs.

2. It helps expand the graph as your app evolves so that the DAG is always up to date.

That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.

1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.

2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.

3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.

- Your first point sounds like a frontend bug to me. I wonder if there's missing revalidation.

- Sounds like the defaults need a little more tuning. You can actually update the definition yourself in settings https://app.momentic.ai/settings/mo.

- I'll take a look at this "stop testing" button thing as well. Thanks for flagging.

- What kind of export would you like? We currently have integrations that allow syncing bugs to Linear or other project management software. You can also export the full results using our CLI: `qa report <session-id> --output mo-report`.

- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.

- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."

- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.

- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.

But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).

The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.

- The candidate ground truth bugs are generated using runs from all agents

- Every candidate gets reviewed by the other agents to generate a recommendation (just makes human review easier)

- A human reviewer makes the final call (no candidates are discarded just because Mo couldn't reproduce)

And in terms of grading the final runs:

- A lot of the final runs still found a small percentage of novel bugs (5-10%), because it's extremely difficult to build a comprehensive list of all bugs, even after hundreds of runs.

- Nothing that didn't match was auto-discarded (read - it wasn't rejected because Mo couldn't repro).

- All unmatched findings were run through the same human review process as above.

- All the runs were regraded after human review to get a final score with zero unmatched findings.

One interesting thing we did notice during this is that Codex and Claude tend to converge on the same findings run over run (very few initially unmatched findings on final runs), whereas Mo tends to find a lot more novel bugs on ever run.

Everything has to be reproducible and runnable by the general public