Mallik Cheripally
All work

Labeling Agent

An autonomous labeling agent: orchestration, vision pipeline and benchmark

Could an AI agent label a dataset and review its own work, to the standard expected of people? I took that question on as my own and built everything it needed to be answered: the agent, the orchestration, a vision pipeline with a lot of custom work between the models, and a benchmark to judge every change. On real autonomous-vehicle data, where the raw models scored 0.2 to 0.4 IoU, the agent's masks landed around 0.95.

The agent's run seen straight on: a graph on the left that branches, rejoins and loops back, with the route this run took lit, and on the right the frame it is working, its accepted masks washed over the roofs they cover and traced as polygons with a handle at every corner

My part

  • Concept
  • Agent and orchestration
  • Vision pipeline
  • Evaluation tooling

The question

Rough boxes and jagged masks are easy to generate and no use to train on. The goal was labels good enough to replace the human pass.

Plenty of tools can draft labels. The ambition here was bigger: have an agent carry the entire annotation and review process, combining vision-language models with specialist models like YOLO and SAM, and push the output toward the quality human annotators are held to.

The raw models do well on clean images, where objects are big and their edges are clear. Real autonomous-vehicle footage isn't like that. On it, the raw masks dropped to an IoU of 0.2 to 0.4.

What I built

  1. 01

    A run starts like any dataset

    Create the dataset, set the categories and label types to generate (bounding boxes, polygons, segmentation masks and attributes), and start the run.

  2. 02

    An orchestrator, not a pipeline

    The agent coordinates a graph of steps rather than a fixed sequence, deciding what runs next.

  3. 03

    Generate, refine, describe

    A multi-stage vision pipeline produces the first labels, then refines them down to the pixel: boundaries, mask edges, polygon vertices. Most of that work sits between the models, and it's the part I can't go into here.

  4. 04

    Review

    Separate reviewer models and review steps check the results before anything is accepted.

Measuring it

An experiment is only as good as its measurements, so I built a benchmarking and comparison engine over several ground-truth datasets. Every technique was logged: which one ran, which metrics went up, which went down, and what still needed work.

A results visualizer and a dashboard made the comparisons readable across datasets and environments, so each decision about the pipeline was made on evidence.

  • Mask IoUHow much of each mask overlaps the true object

    Raw models
    0.2–0.4
    Agent
    ~0.95
  • Boundary score0.95–0.98
  • Distance from the true edge2–5 px
  • Labels at human quality30–40%
Detected objects in real autonomous-vehicle scenes. The raw models' score is the range they fell into across scenes; the agent's is where it settled.

Where it landed

False positives dropped sharply. Boundaries, the first thing anyone notices in a mask, now sit 2 to 5 pixels from the object's real edge, even in moderately difficult scenes. For medium and large objects, 30 to 40% of labels came out as clean polygon masks within 2 to 4 pixels of where a human annotator would put them.

Very small and low-contrast objects improved by more than 50%, but they still fall short of the bar I set. There's too little in the pixels to go on.

Of everything I've built, this is the work I'd point to for how I approach AI systems.