← Jiaqi (Evey) Gou Technical deep dive · Client project
AI data product · Evaluation · Pipelines at scale

From 50,000 messy prompts to a dataset you can trust

A multimodal foundation-model client asked for "basic tags" on 50K video-generation records — prompts plus reference images, audio and video. I reframed it as a data product: a taxonomy you can measure, a pipeline you can audit, and a dashboard and explorer people can actually use.

Role
Taxonomy, evaluation design, product decisions, QA & delivery
Built with
Gemini for classification; AI coding agents for implementation, directed and reviewed by me
When
Oct 2026
Note
Client, media and prompts are confidential; only method and scale are shown
01 — At a glance
50K
records, every one mapped to a stable ID in original order
7 × 48
dimensions and labels in a locked, versioned taxonomy
99.8%
analytics-usable coverage (49,923) — coverage, not accuracy
~2.1×
style-validation throughput from concurrency tuning
02 — Define before you model

I wrote the taxonomy before choosing any model, and kept it separate from prompts and pipeline. Changing a threshold is a semantic change, so it gets a version — not a quiet prompt tweak. The hard work was in the boundaries:

Generation mode
An attached image doesn't make it image-to-video. Is the image being animated, or used as a character or style reference?
Subject
Tag what the request is about — a richly described background is not the subject.
Rendering style
Judge the final style to keep or produce. Photorealistic and 3D/CGI can co-exist.
Explicit silence
"No music" isn't silence. Only a fully silent track or segment counts.

A 1,000-prompt coverage audit suggested six new capabilities. I chose not to add them yet: comparable metrics mattered more than a bigger ontology.

03 — Choosing an architecture

Each step earned the next, on a sample frozen in advance so I couldn't tune prompts to individual cases:

18 calls
Smoke test
Feasibility only — not an accuracy claim
12 + 28 records
Blind review
Four architectures vs independent human labels; then disagreement analysis
500 records
Pilot
497 valid — and two systematic errors found before scaling
50,000 records
Production
Simplest winner + targeted correction layers

The simplest option won: Gemini reading the prompt directly with 2–4 keyframes from any reference video. I dropped the more elaborate summarize-then-classify chain — and decided to stop experimenting once more revisions stopped showing stable gains.

04 — The pipeline
50K CSV · stable IDs
→
Media fetch · transforms · keyframes
→
Shared ready queue
→
Raw tags · frozen prompt
Scene / mode correction
Style validation
Silence check
Recovery · human review · effective labels with provenance
→
Master dataset
→
Overview + faceted explorer

Corrections never overwrite the raw output: each record keeps raw, correction and effective labels, so every number on the dashboard can be traced back.

05 — Trade-offs at scale
Fix the queue, not the model
Eight fixed partitions left five workers idle while two sat on thousands of ready records. A shared queue with atomic claims removed the artificial bottleneck.
Reliable beats fastest
Doubling downloads to 128 brought timeouts; 16 style workers ran faster but produced unknown outcomes. I settled on 64 and 8 — the highest throughput that stayed auditable.
Recover before you re-run
Saved responses were re-parsed locally first. Requests with unknown outcomes were never auto-retried — only in a separate, approved batch.
A human for the last eight
When the same records missed a required label twice, I stopped calling the API and reviewed them by hand — filling only that field, with provenance.
06 — What it shows

Most of AI product work isn't the model call. It's deciding what to measure, being honest about what a number means, and building the system so the answer can be trusted.

Back Return to the index →