Commissioned work. An AI app-building platform commissioned this evaluation. The client's identity and the identities of the two comparison platforms are confidential.
Overview
An AI app-building platform commissioned Biz-Tech Analytics to evaluate its product alongside two other platforms. We built the evaluation pipeline in one week, covering five jobs and three independent builds per job on each platform: 45 builds in total. The work produced 3,270 scored results and more than 200 defects, each documented with reproduction steps and evidence.
The client received feature-level results, consistency across repeated builds, a prioritised defect log, and cost and time reported separately from quality.
The challenge
The client wanted to see its own app builder beside other leading names in the market: to benchmark internally, guide product direction and track improvement over time.
Because one of the platforms being scored paid for the work, the comparison had to be fair by construction. Every platform got identical prompts, the same procedure and the same answer to any question its agent asked, and the criteria were fixed before the first build.
How we built the pipeline
1. Freeze the yardstick. Prompts and criteria were fixed before any build, and the criteria were held out from the apps being evaluated. Each criterion is a pass or fail check with test steps and an expected result, designed so that two scorers would reach the same verdict.
2. Run it as a production line. Operators built and published; scorers took each build at handover. Within an app, each scorer owned the same criteria across every build and platform, so each criterion was judged the same way everywhere.
3. Test consistency and regressions. Each job was built three times from one prompt to measure consistency. Follow-up prompts counted every check that previously passed and later failed as a regression.
4. Score beyond pass or fail. Nine weighted dimensions were used, covering coverage, edge cases, security, usability, design and deployment among them. Every failure was logged as a defect with reproduction steps and evidence. Cost and time were reported beside quality, never folded into it.
5. Consensus on high-severity criteria. High-severity criteria received consensus labelling, with an auditor resolving disagreements.
Scale and staffing
Seventy or more criteria were scored per app, per platform, per build, per turn. Five jobs across three platforms, built three times each, gave 45 builds, with edit rounds scored as part of each one.
For this programme the evaluation team was expanded with domain specialists to 20 trained scorers, trained before the first build and held to one standard across builds, platforms and criteria.
What was measured
Alongside quality, every build recorded cost in credits and USD, time to first preview, agent build time, prompts and agent questions, publish time and outcome, and whether the live app matched its preview.
What the client received
For each job, a full evaluation pack: pass and fail by feature on each platform, the consistency spread across the three builds, a ranked defect log with evidence, and cost and time per build. The set can be re-run on the next release.
How we keep an evaluation fair is set out in our independence policy. See also who runs Biz-Tech.