Key takeaways
- It finds its own questions. Given only a store's own data, the agent spotted problems nobody had pointed it to.
- It catches what dashboards miss. A hero product with clicks up 6× but orders up barely 15%. Thousands of sample requests expiring because nobody reviewed them.
- Its numbers hold up. We checked every finding against our own data, and every one pointed the same way.
- Push back, and it gets more solid. When challenged, it re-checked its own work instead of digging in, so we built that review step into the process.
The question
Our earlier research on samples turned up a lot that dashboards can't show, like the fact that GMV at review time says almost nothing about who will sell. But every one of those studies needed someone to ask the question and spend time digging. What a store really needs is someone doing that every week, not once a quarter.
So we wanted to find out: can the AI agent already working in a store do this research by itself, using only that store's own data? We ran it on real stores, read-only with no changes made, and checked every number it produced.
What it found on its own
- Clicks up, orders flat. A hero product's clicks rose about 6× in a month, while orders rose barely 15%. Nearly all the new clicks came from creator videos, but those videos weren't converting.
- Sample requests expiring unreviewed. Thousands of requests timed out before anyone got to them, hundreds of them from creators with over $10K in sales.
- Fewer videos came before the sales drop. In another store, the number of videos that actually sell fell by more than half, and it happened before GMV started falling.
We didn't tell it to look for any of these. We checked each one against our own data, and every finding pointed the same way.
Push back, and it gets more solid
Like any analyst new to the job, its first draft tends to overstate things. What surprised us was how it handled pushback. Asked "could this just be a coincidence?" or "can you really compare these two periods?", it went back and re-ran the numbers every time, and corrected its conclusion instead of digging in.
Leave the certain parts to code. Let the model ask the questions code can't.
A good reviewer keeps coming back to the same handful of questions. So we turned them into a checklist and added an automatic review step to the research process. Pulling and reconciling data is handled by dependable tools, so the agent never has to worry whether it got everything. It focuses on the parts that take thinking: what to look at, what might explain it, and how to test it. Whether to act is your call.
Where this is going
We found that good research depends less on how smart the model is, and more on whether data reaches it completely and smoothly. That's where we're investing now. The goal: every week, your AI operations team brings you one store question worth your attention, with the evidence, and you decide whether to act.
About this research
Figures come from OperaX internal data and estimates, aggregated and rounded to show the pattern.