---
title: "What Is Data Labeling? Data Annotation for AI Explained"
description: "Data labeling adds the correct answer to raw examples so an AI model can learn from them. Label types, the labeling process, quality checks and AI help."
url: https://proxynet.io/blog/what-is-data-labeling
date: 2026-11-18
author: "Acar Diveroli"
category: "AI"
lang: en
---

# What Is Data Labeling? Data Annotation for AI Explained

A support team wants a model that sends each new ticket to the right queue: billing, delivery or returns. It has 60,000 old tickets, but none records where it went, so a model has nothing to learn from yet. Someone has to read a sample and write the answer next to each ticket. A car maker faces the same problem with camera images, a voice assistant team with recorded speech. That work is data labeling.

This post explains what data labeling (also called data annotation) is, why AI models need it, the main label types, how a labeling project runs, how quality is measured, how models help with the work and which personal data rules apply.

> **Note: Short answer**
>
> Data labeling, or data annotation, is the work of adding the correct answer, a label, to raw examples such as images, texts, audio clips or videos, so that a machine-learning model can learn from them or be tested against them. A label can be a category ("spam"), a box drawn around a car, a highlighted company name or a written transcript. Good projects start from a written guideline, run a pilot, measure how often independent labelers agree beyond chance and fix the guideline before labeling at scale. Models can draft labels and pick the examples that need a person, but people still check the labels that matter. When the examples contain personal data, the GDPR, KVKK and similar laws apply to the labeling as well.

## What is data labeling?

Data labeling is adding the correct answer to each raw example so that a model can learn the link between the input and that answer. The input might be a photo, a sentence or a sound clip; the label is what a careful person would say about it: "cat", "negative review", "Paris is a place name". An input together with its label is a labeled example.

Labeled examples are the raw material of supervised learning, the branch of machine learning in which a model learns from inputs paired with known answers and then predicts answers for new inputs. The labels a model is measured against are called ground truth, though they are only as true as the people and rules behind them.

Some labels arrive for free: marking an email as spam labels data for your mail service, as Burr Settles' survey of active learning points out. Business data rarely works that way; nobody tags a ticket with its queue unless that is someone's task. The same work is also called annotation, tagging or, in the social sciences, coding.

## Why do AI models need labeled data?

AI models need labeled data for three jobs: to learn a task, to be tested on it and, for chat assistants, to learn which answers people prefer.

**Training.** A model that routes tickets or spots cracked parts learns only from examples whose answer is known. The labels define the task: with no "returns" label, the model never learns to recognise a return.

**Evaluation.** Even a model trained without labels, such as a large language model pretrained on raw text, is tested on examples with known answers. A small, carefully labeled test set is often a team's most valuable data, because every later change is measured against it.

**Tuning language models.** Chat assistants are tuned with answers written by people and with rankings of several model answers from best to worst. In the [InstructGPT paper](https://arxiv.org/abs/2203.02155), a team of about 40 contractors, selected with a screening test, wrote demonstrations and ranked between four and nine model outputs per prompt.

Those rankings are labels too: a person decides which answer is better by written instructions. How they adjust a trained model is explained in [What Is Fine-Tuning?](/blog/what-is-fine-tuning).

## What types of data labeling are there?

The label type follows from the kind of data and the question the model has to answer:

| Data | Common label types | Example task |
|---|---|---|
| Images | Class, bounding box, outline, keypoints | Find damaged parcels in photos |
| Text | Category, sentiment, entities, intent | Route tickets, flag abusive comments |
| Audio | Transcript, speaker turns, sound events | Train speech-to-text for a call centre |
| Video | Objects tracked across frames, actions | Follow a forklift through a clip |
| Documents | Field names and positions, page type | Read totals and dates from invoices |

**Images.** Classification gives one label to the whole image ("contains a crack"). A bounding box is a rectangle drawn around an object and saved as coordinates. Segmentation traces the exact outline, as a polygon or pixel by pixel, and keypoints mark single points such as the joints of a body. The public COCO dataset shows the scale this can reach: 328,000 images with 2.5 million labeled objects, each outlined on its own.

**Text.** A text label can cover a whole document (topic, sentiment, an intent such as "cancel my subscription") or a span inside it. Named entity recognition, a classic annotation task in natural language processing, marks the words that name people, organisations, places, products or dates. Sentiment is often the first label a team tries; [Sentiment Analysis in Python](/blog/sentiment-analysis-python) shows why a small set of hand-labeled reviews is the yardstick for any scoring method.

**Audio and video.** Audio labels range from full transcripts to who speaks when. The work is slow: Settles' survey cites a report that word-level transcription can take ten times as long as the audio itself. In video, the same object has to keep its identity from frame to frame, which is called tracking.

## How does a data labeling project work?

A labeling project runs as a loop: define each label, test the definitions on a small batch, measure where people disagree, fix the definitions and only then label at scale. A typical project has eight steps:

1. **Define the task and the label set.** Write down the question the model must answer and the allowed labels, each with a clear border to its neighbours. An "unclear" option keeps people from guessing.
1. **Write guidelines with examples.** Give each label a definition, typical examples and the hard cases, such as a ticket that asks for a refund and complains about a late delivery.
1. **Run a pilot.** Several people label the same small batch independently. Time it, so cost estimates come from measurement.
1. **Label.** Work in batches, mix in gold items whose correct label is already known and keep everyone on the same guideline version.
1. **Review.** A second person checks a sample of every batch and every item marked "unclear".
1. **Measure agreement.** Compare labels from people who labeled the same items without seeing each other's answers.
1. **Fix the guidelines.** Disagreement usually means an ambiguous rule, not careless people. Clarify it and relabel the affected items.
1. **Deliver with documentation.** Ship the labels with the guideline version and agreement scores, and keep a test set that is never used for training.

The InstructGPT team describes the same habits at scale: onboarding, detailed instructions for each task and a shared chat room for labelers' questions.

## How is labeling quality measured?

Labeling quality is measured against gold items whose answer is already settled, and by how often independent labelers agree beyond what chance alone would produce.

**Gold items** are examples with a checked label, mixed into the work unmarked. A labeler who misses many needs feedback; a gold item that many people miss may itself be wrong.

**Inter-annotator agreement** measures how consistently different people label the same items. Raw percentage agreement misleads, because some agreement happens by chance. Cohen's kappa, introduced by Jacob Cohen in 1960, corrects for it: kappa = (observed agreement - chance agreement) / (1 - chance agreement). A kappa of 1 means perfect agreement; 0 means no better than chance.

Two people label 200 tickets as "billing" or "not billing" and agree on 85% of them. If each puts half the tickets in each class, chance alone gives 50% agreement, so kappa is (0.85 - 0.50) / (1 - 0.50) = 0.70. If both put 90% of the tickets in "not billing", chance alone gives 0.9 × 0.9 + 0.1 × 0.1 = 82%, and the same 85% is worth a kappa of only 0.17.

Fleiss' kappa and Krippendorff's alpha extend the idea to more than two labelers. In computational linguistics 0.8 is the usual target, but the survey by [Artstein and Poesio](https://aclanthology.org/J08-4004/) argues that no single cutoff fits every task and recommends reporting a confusion matrix, a table showing which labels get mixed up with which.

Subjective tasks agree less: in the InstructGPT project, labelers comparing model answers agreed with each other 72.6 ± 1.5% of the time. When agreement stays low after the guideline is fixed, adjudication, a senior reviewer or a majority vote settling each disagreement, becomes part of the process.

## How do models help with labeling?

Models help in four ways: they draft labels for people to check, choose which examples deserve a person's time, turn rough rules into labels and sometimes label with no person involved. The more the model does, the more carefully its output has to be checked.

| Approach | How it works | Watch out for |
|---|---|---|
| Manual | People label every item from the guideline | Slow and costly at scale |
| Pre-labeling | A model suggests, people accept or fix | People accept wrong suggestions |
| Active learning | The model asks about items it is unsure of | Labeled set is no longer random |
| Weak supervision | Rules vote, a model combines the votes | Rules share the same blind spots |
| Fully automatic | A model or simulator outputs the label | Errors nobody checks |

**Pre-labeling** is the simplest form: an earlier model, or a large language model given the guideline as instructions, drafts a label and a person confirms or corrects it. People tend to accept a plausible suggestion, so pre-labeled batches need the same gold-item checks. This loop is what "human in the loop" usually means in labeling.

**Active learning** rests on the idea, in Settles' words, that a model can reach "greater accuracy with fewer training labels" if it may choose the data it learns from. Its usual strategy, uncertainty sampling, sends people the examples the model is least sure about, so testing still needs a separate random sample.

**Weak supervision** replaces some hand labels with rules. In the system described by [Ratner and colleagues](https://arxiv.org/abs/1711.10160) in 2017, experts write labeling functions, small heuristics such as "if the ticket contains an invoice number, vote billing", and the system estimates how reliable each rule is and combines their votes into training labels.

**Fully automatic** labels come from a model or a simulation with no person checking each one. A 3D simulator knows where every object in its scene is, so its images arrive labeled; the limits of generated data are covered in [What Is Synthetic Data?](/blog/what-is-synthetic-data). The test set still needs labels a person has checked.

## What about personal data in labeling?

When the examples contain personal data, labeling them is processing of personal data, and the GDPR in the EU, KVKK in Türkiye and similar laws apply. Tickets carry names, photos show faces and number plates, recordings carry voices.

Remove what the task does not need before the data reaches labelers: mask names, emails and phone numbers, blur faces that are not the subject and cut audio to the relevant part. Masking is not automatically anonymisation: under Recital 26 of the [GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng), pseudonymised data, where names are replaced by codes but the key still exists, remains personal data.

Labelers working for you through another company process the data on your behalf. The GDPR requires a contract with such a processor, and KVKK (Law No. 6698) makes the data controller jointly responsible with the processor for security measures. In practice: access only to the batches a person is working on, no copies on personal devices and deletion when the project ends.

Regulation is starting to name labeling directly. The [EU Artificial Intelligence Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng) requires training, validation and test data for high-risk AI systems to follow data governance practices that cover, among other things, "annotation, labelling, cleaning, updating, enrichment and aggregation".

## Where do the unlabeled examples come from?

Unlabeled examples come from your own systems first, then from licensed datasets and, for tasks about public content, from public web pages; what you collect decides what the model can handle. A ticket router trained only on English tickets stumbles on Spanish ones.

Your own tickets, photos and recordings are used within the notices and consent that cover that purpose. To classify product listings, reviews or job ads in many languages, teams also collect public web pages from the countries they serve.

That collection step is where proxies fit. Many sites change language, currency and assortment by the country of the visitor's IP address, and a [Residential Proxy](https://proxynet.io/residential-proxy) with country and city targeting lets you collect the version local visitors see. Collect public pages only, follow robots.txt and the site's terms, use an official API where one exists and leave out personal data you will not label.

Our page on [proxies for data scraping](/data-scraping) explains which proxy type and session mode suit which collection job.

## Where is data labeling used?

Data labeling sits behind most AI features that decide something about a specific input. Support teams label tickets to route them. Online shops label products with categories and rate search results for relevance. Moderation teams label posts against a written policy. Factories label images of parts as good or defective, and banks label transactions as fraud or not. Hospitals label scans, which needs qualified clinicians and strict data protection. Teams building on a large language model label test questions with acceptable answers, so they can tell whether a new prompt or model version is better.

## Common mistakes

- **Labeling before the guideline exists.** Everyone invents their own rules, and the disagreement surfaces after thousands of items.
- **No "unclear" option.** Forced guesses look exactly like confident labels.
- **Reporting only percentage agreement.** With one dominant class, high agreement can be chance; report kappa or alpha.
- **Changing the guideline without relabeling.** The data then mixes two rule sets.
- **Trusting pre-labels.** Check model suggestions against gold items as strictly as manual labels.
- **Letting test items leak into training.** A model tested on what it was trained on looks better than it is.
- **Sending raw personal data to labelers.** Mask or remove what the task does not need first.

## Decision guide

| Need | Approach |
|---|---|
| A first model for a new task | Guideline, small pilot, manual labels |
| Many items, few experts | Active learning on the uncertain items |
| Experts can state clear rules | Weak supervision, checked on a hand-labeled sample |
| Large volume, low stakes | Model pre-labels with gold-item checks |
| A test set you can trust | Two independent labelers plus adjudication |
| Subjective labels such as tone | Several labelers per item, agreement reported |
| Data with personal details | Mask first, processor contract, limited access |

## Frequently asked questions

### What is the difference between data labeling and data annotation?

In most teams there is none. Where people separate them, annotation is the broader word for any information added to raw data, such as boxes, outlines or highlighted spans, while a label is the answer itself, such as a class name.

### What is a bounding box?

A bounding box is a rectangle drawn around an object in an image and saved as coordinates, usually two corners or one corner plus width and height. It is faster to draw than a traced outline but less precise for irregular shapes.

### Can AI label data by itself?

Partly. A large language model can label text from a written guideline, but machine labels inherit the model's mistakes and biases. Use them as drafts or for low-stakes volume, and keep people on the gold items, the disagreements and the test set.

### How much labeled data do I need?

It depends on the task, the number of labels and how alike the classes are, so measure it. Train on growing parts of a first labeled batch (a quarter, half, all of it) and check the score on a fixed test set each time. If the score is still climbing, more labels will help; if it has flattened, clearer labels usually help more.

### What is inter-annotator agreement?

Inter-annotator agreement measures how often different people give the same label to the same item, corrected for chance. Cohen's kappa is the usual measure for two people, Fleiss' kappa and Krippendorff's alpha for more. Low agreement usually points to an unclear guideline, not careless labelers.

### Is data labeling a job?

Yes. Labeling is done by in-house teams, by contractors and by specialists such as linguists or medical and legal experts when a task needs their judgment. A data labeling specialist labels and reviews items, flags unclear cases and helps improve the guideline. If you are considering an offer, read its terms carefully.

## Summary

Data labeling adds the answer a model should learn, a class, a box, a highlighted name or a transcript, to raw examples. Models need it to learn supervised tasks, to be tested and, for chat assistants, to learn which answers people prefer. Good labels come from a clear guideline, a pilot, agreement measured beyond chance and a loop that fixes the guideline where people disagree. Models now draft labels and pick the examples worth a person's time, but people still check what matters. Remove personal data the task does not need, and collect raw examples from the languages and countries your model will meet.
