Predicting Ad Effectiveness: A Self-Optimizing Pipeline Over Six Cycles

Hover or tap underlined terms and data values for explanations.

1. Introduction

Can a commodity vision-language model watch a thirty-second commercial and predict how effective humans found it? The question belongs to a larger class of problems on which vision-language models remain unreliable out of the box: the prediction of subjective human judgment from multimodal input, where the ground truth is the aggregated opinion of a panel of raters rather than a fact sitting in the world to be recovered (Wu et al., 2024). This work reports two findings from applying an autonomous architecture engine to one such benchmark: first, that the engine extends to this new task class and constructs a more predictive pipeline without human rule-writing between cycles; and second, that the constructed pipeline measurably improves the subjective-judgment predictions of an open-weight baseline model.

The autonomous architecture engine used in this work has been described in prior studies, where it was applied to the HealthBench benchmark (Marchell, 2026a) and to a medical triage benchmark (Marchell, 2026b), matching or exceeding frontier commercial systems in both cases. Both of those earlier applications targeted medical-reasoning tasks. The application reported here is the first outside medicine, and the first in which the engine is asked to predict human preference.

Each of 583 advertisements in the evaluation set has been rated by people on a 1–5 effectiveness scale, where 1 means “not effective at all” and 5 means “highly effective.” The ratings are nearly uniformly distributed across the five values in the scale, which sets a useful reference point: a trivial model that predicts the same rating for every advertisement achieves about 20% accuracy, and one that always predicts 3 achieves an average error of 1.19 points on the five-point scale. Qwen 3.5 Flash, an open-weight vision-language model, was given a simple instructional prompt and asked to predict a rating for each of those same advertisements. Its predictions matched the human consensus on 22% of the ads (barely above the trivial guess), with an average error of 1.45 points, larger than the 1.19 produced by always predicting 3. Roughly 80% of its predictions landed at 4 or 5 regardless of how the people had rated the ad, producing output with little power to separate good ads from bad.

Six cycles of autonomous optimization later, the same model produces ratings with an average error of 1.10 points and matches the human consensus on 26% of the ads. The 1.10 average error is a 24% reduction from the baseline and the first result in the experiment that outperforms the constant-guess benchmark. The correlation between the predictions and the human consensus rises from near zero to a meaningful level.

How Much the Predictions Improved, Cycle by Cycle

Each bar shows how much closer the system's predictions moved to the human consensus compared to the starting baseline, measured as a percentage reduction in the average rating error.

Headline Numbers

2. The Task

The benchmark is the Pitt/Kovashka Ads Dataset: 3,477 real YouTube commercials rated by crowd workers on a 1–5 effectiveness scale. The rating answers a single question: how effective is this ad at persuading a viewer to act? A balanced working set of 583 ads was sampled from the full corpus so each of the five rating levels is represented roughly equally, giving the evaluation statistical power at every point on the scale. The pipeline receives only the mp4 file; its output is a single integer compared to the human consensus rating.

3. How the Self-Optimizing Pipeline Works

Every cycle follows the same four-step loop: identify where the current system is failing, propose potential improvements, build and test each candidate, and keep only the changes that work on unseen data. Over six cycles, roughly 30 candidate improvements were tested; 10 survived validation and entered the final pipeline.

The Optimization Loop

4. What the System Built

The finished pipeline makes three model calls per ad and applies four programmatic gates on top of their combined output. No model weights were changed, no external retrieval was added, and no human wrote a rule between cycles.

Emerged Pipeline Architecture

5. The Results

Each row below shows what was added in that cycle, the failure family it was meant to address, and the resulting MAE after the change was integrated into the pipeline.

Cycle-by-Cycle Progression

The loop took corrective turns in every direction. Cycle 2 produced the experiment's only full rejection: an approach that gained on the target cases while quietly damaging everything else. Cycle 3's memorability probe was originally proposed as a way to correct ratings upward, but every upward version produced unreliable answers; the one useful formulation turned the probe into a downward cap instead. Cycle 5 had to add an error-correction review specifically because Cycle 4's downward checks had mistakenly lowered some ads that humans had actually rated 4 or 5. The engine kept adjusting to the consequences of its own prior decisions.

6. Conclusion

The question this article opened with was whether a commodity vision-language model could watch a thirty-second commercial and predict how effective humans found it. With the model alone, the answer was essentially no: it rated nearly every ad a 4 or 5, and its average error was worse than the score a trivial model would earn by guessing 3 for every ad. After six cycles of autonomous optimization, the answer looks more within reach. The same model, wrapped in a seven-stage pipeline the engine built, reduced the average rating error by 24 percent and now produces ratings that meaningfully track the human consensus. The problem is not solved, but six cycles of steady improvement suggest the optimization loop could likely close more of the remaining gap with more time.

This is the third benchmark the autonomous architecture engine has been deployed on. It previously matched frontier commercial systems on HealthBench and exceeded ChatGPT Health on a medical-triage benchmark by 16 percentage points. Ad effectiveness is a different class of problem from either: the ratings are subjective, noisy, and carry legitimate disagreement among the people who produced them. The engine delivered a comparable reduction in error anyway, using the same optimization protocol that ran in the two earlier experiments without any changes.

I started this project to answer a narrower question: could AI agents build better AI systems than I could by hand, on problems I cared about? Three benchmarks in, the answer looks like a qualified yes. The pipeline that emerged on ad effectiveness is one I would not have designed. In several places, it is better than what I would have designed. That is the most interesting finding of the experiment for me, and the reason I am continuing to run it.

7. Methodology

Wu H, et al. Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision. ICLR. 2024.

Hussain Z, et al. Automatic Understanding of Image and Video Advertisements. CVPR. 2017.

Marchell J. Autonomous Optimization on Clinical Triage: Raising DeepSeek V3.1 from 84% to 97% on the Ramaswamy Factorial Benchmark. 2026.

Marchell J. Self-Optimizing Pipeline: How Iterative Experimentation Brought Open-Weight Models to Frontier Performance on HealthBench. 2026.