Why Good Data Isn't Pass or Fail

Written by: Jaden Rose

---

Most AI pipelines force every example into "good" or "bad." Saurab Dhir argues reality is messier and built a system that scores uncertainty instead of pretending it doesn't exist. The clip in the review queue was doing everything right, and that was the problem. Someone was folding laundry, carefully, every motion clean, and the longer Saurab Dhir watched, the less sure he was whether he was seeing diligence or theft.

“Borderline data is very deceiving,” he said. Some contributors farm hours, stretching a task to pad paid time, others are simply careful people trying to clear the bar, moving so precisely they read as machines. A clip can land on the wrong side of the line by being too good.

Folding laundry alone offered several shapes of doubt: someone taking far too long on a single shirt, someone folding so that creases go in rather than come out, someone folding things nobody folds, like socks. The hardest category was not slowness at all. It was “unnatural,” a word that lives only as gut instinct, and turning gut instinct into a model is, in Dhir’s view, a genuinely hard research problem.

Every quality-control pipeline Dhir had seen resolved a clip the same way: valid or invalid, keep or reject, a coin forced to land on one side. He built Mecka AI’s system to refuse the coin flip. The break came when he audited the ground truth itself and found the labels wrong at a meaningful rate, traceable to particular reviewers. A binary gate has nowhere to record “this label is almost wrong.” A score does. So he stopped treating a reviewer’s verdict as the output and started treating it as one piece of evidence.

That shift had teeth. Once a reviewer’s verdict became evidence rather than the final word, a bad label stopped being an anonymous mistake and became a traceable one, tied to the person who made it and weighed against everything else the pipeline knew. The point was never to catch reviewers out. It was to stop pretending a single human glance at a borderline clip was ground truth when the audit said it often was not.

His quality layer ranks borderline data instead of rejecting it, scoring clips across a wide spread of measures, numbers, bands, probabilities, rather than a single verdict, and letting the pipeline decide, continuously, what bar a given batch or customer needs. The gate became a dial. Per-customer thresholds stay private, but the shape holds everywhere: quality as a control surface, not a locked door.

The reframe sounds small and changes almost everything downstream. A binary gate throws away information at the moment the operation needs it most, a graded signal keeps it. Batches can be built to a quality target the way an investor builds a portfolio to a risk target. Ambiguous clips, the hour-farmer against the perfectionist, stop being settled by whoever happened to review them that day and start gathering evidence. And the people in the loop concentrate where their judgment pays, on the uncertain middle, instead of rubber-stamping the obvious ends.

The people in the pipeline feel the change before the metrics do. Under a binary gate, a reviewer facing the perfectionist-or-farmer clip has to make a weighted guess and move on, under the graded system, that same judgment joins other evidence instead of settling the case alone. Dhir pairs the scoring with per-reviewer analytics, so the operation can tell a drifting reviewer from a drifting contributor from a genuinely hard batch, three failures that look alike from a distance and need entirely different fixes.

Dhir's path to this ran through paperwork, not robotics. He co-founded Docula, a bootstrapped medical-AI startup whose software turned inconsistent clinical records into defensible billing analyses, work where a wrong call gets read aloud by a lawyer. Mecka acquired the company this year, and the instinct carried over intact. In both jobs the deliverable was never really the document or the video. It was a decision someone else could trust without redoing the work.

That trust is the connective tissue the company is buying. "There's a lot that goes into how things come together here, and Saurab's engineering role is a critical piece of that," said Jason Chong, Mecka's chief technology officer. The work that follows is the clearest illustration of what he means.

The second piece of the work is a tool most companies would never hand to one engineer. Off-the-shelf annotation software kept failing Mecka’s computer-vision researchers in concrete ways. Some tools would not let you edit inside a video at all, only frame by frame, which destroys context. Some would not export data in the specialized forms the team needed. Some gave almost no statistics on the annotators. And underneath all of it, none offered any read on whether the annotations were actually good, which is the operational nightmare: no way to spot the strong annotators, no way to reward speed and quality together. So Dhir built the platform himself.

It took about a month, and he built it alone. It runs onboarding, project management, operations, statistics, usage, and the annotation itself. No one made a serious case for licensing instead. The team needed something shaped to its own workflow, with features that could change as fast as the pipeline did, and nothing off-the-shelf keeps that pace.

The economics followed, described at the altitude the company allows: Dhir’s faster, leaner quality models materially reduced per-recording QA cost. What made them cheaper stays undisclosed. His design philosophy cuts against the industry reflex of renting more compute to solve throughput. The field treats efficiency as a constraint to escape, he argues, he treats it as the actual work.

A signature runs across all of it, the graded signal, the ground-up platform, the leaner models: a refusal to accept the field’s standard abstractions. Pass/fail was an abstraction. The annotation-tool market was an abstraction. The compute bill was an abstraction. Inspect any one closely, Dhir found, and it turned out to be a decision someone else had made for him, which he then unmade. He tends to ask what a default is quietly costing, on the theory that the answer is usually that nobody has checked.