What Is Natural Language Processing (NLP)?

Published:

15 minute read

Acar Diveroli
Written by: Acar Diveroli
A light beam from a tilted message panel reaches three tags on an arc: entity, sentiment and a blue intent tag

An online shop delays a batch of deliveries, and by Monday morning its support inbox holds 3,000 new emails. Nobody reads them all. Software sorts them first: these ask for a refund, these mention a damaged box, these are angry enough to go to a senior agent. The software never "reads" the way a person does, yet it turns free-form text into labels a team can act on. That is natural language processing, or NLP.

This post explains what NLP is, how it developed from hand-written rules to transformer models, how a system turns text into meaning, its main tasks, how it differs from NLU, NLG and large language models, why language is hard for computers and where businesses use it.

What is natural language processing?

Natural language processing is the field that teaches computers to analyse, understand and generate human language. "Natural" separates languages that people speak and write, such as English, Spanish or Turkish, from the formal languages computers were built for, such as Python or SQL, which have strict rules and no ambiguity.

NLP sits where computer science, artificial intelligence and linguistics meet. Linguistics describes how language is built: words, grammar, meaning and context. Computer science and machine learning supply the methods that let a program learn those patterns from examples instead of being told every rule.

Most people use NLP many times a day without naming it. The keyboard that suggests the next word, the translation button under a foreign-language post and the search box that understands "cheap flights to Rome in May" all run on it.

How did NLP develop?

NLP has gone through four broad stages: hand-written rules, statistics learned from text, neural networks and, since 2017, transformer models that are pretrained on huge amounts of text.

Rules (1950s to 1980s). In January 1954, IBM and Georgetown University publicly demonstrated a system that translated Russian sentences into English. According to W. John Hutchins' history of the Georgetown-IBM experiment, it worked with just 250 words and six grammar rules, yet it raised hopes that high-quality machine translation was only a few years away. In 1966 Joseph Weizenbaum described ELIZA, a program at MIT that analysed sentences with decomposition rules triggered by key words and built its replies from matching reassembly rules. Such systems were only as good as their rules, and every new phrasing needed another one.

Statistics (late 1980s to 2000s). Instead of writing rules, researchers let programs count patterns in large collections of text. A team at IBM's T.J. Watson Research Center published five statistical models of translation in 1993, learned from French and English sentence pairs taken from the proceedings of the Canadian Parliament. The authors noted that their algorithms had minimal linguistic content: the knowledge came from the data, not from grammar books.

Neural networks (2010s). In 2013, Tomas Mikolov and colleagues at Google described word2vec, a fast way to turn every word into a list of numbers learned from very large text collections, so that words used in similar ways end up close together. Over the following years, neural networks built on word vectors became the standard approach for translation, tagging and classification.

Transformers (2017 onwards). The transformer architecture, introduced in the paper Attention Is All You Need and first tested on translation, drops the step-by-step reading of older neural networks and lets every word look at every other word at once.

In 2018 Google researchers released BERT, a transformer pretrained on unlabelled text and then fine-tuned with one extra output layer, which set new best results on eleven NLP tasks. Today's large language models grew out of the same architecture.

How does NLP work?

An NLP system turns raw text into numbers a model can work with, step by step, and then uses a model to produce a label, a structure or new text. The classic pipeline looks like this:

  1. Clean and segment. The text is stripped of markup and split into sentences and words. This sounds trivial and is not: a full stop can end a sentence, close an abbreviation or sit inside a number.
  2. Tokenise. The text is cut into tokens, the units the model reads. Older systems used whole words; modern models use pieces of words so that a rare or new word can still be represented. What Are Tokens in AI? shows how that split works.
  3. Normalise. Different forms of a word are reduced to one: "ran", "runs" and "running" become the lemma "run", the form you would look up in a dictionary.
  4. Tag. Part-of-speech tagging marks each word as a noun, verb, adjective and so on, which separates "book a table" (verb) from "read a book" (noun).
  5. Parse. Dependency parsing works out how the words relate: who did what to whom. In "the courier left the parcel with a neighbour", parsing links "left" to its subject "courier" and its object "parcel".
  6. Represent as vectors. Each word, sentence or document becomes an embedding, a long list of numbers that places it in a space where similar meanings sit close together.
  7. Run the task model. A classifier, a sequence labeller or a text generator uses those vectors to produce the result: a category, a set of tagged names, a translation or a summary.

Modern transformer models fold most of these steps into one network that learns tagging, structure and meaning internally. The pipeline is still worth knowing, because when an NLP system fails, the cause is often in an early step.

On our test machine we ran the sentence splitter built into Node.js (Intl.Segmenter, which uses the ICU library) on "Dr. Smith ordered 2.5 kg at 9 a.m. It arrived late." It handled "2.5" and "a.m." correctly but returned three sentences instead of two, cutting "Dr." off as a sentence of its own. Small errors like these travel down the pipeline and turn into wrong tags later.

What are the main NLP tasks?

Most NLP work falls into a handful of tasks, each answering a different question about a piece of text. Take one customer message: "Order 4417 to Lyon arrived three days late and the box was crushed, I want my money back."

  • Text classification puts the whole message into a category: delivery complaint. Spam filtering and topic sorting are the same task.
  • Sentiment analysis decides whether the tone is positive, neutral or negative: clearly negative here. Sentiment Analysis in Python compares two ways to score product reviews.
  • Named entity recognition (NER) finds and labels names in the text: "Lyon" as a location, "4417" as an order number, "three days" as a duration.
  • Intent detection works out what the writer wants: a refund. Chatbots and voice assistants depend on it.
  • Information extraction fills a structured record from free text, such as order number, problem type and requested action, ready for a database.
  • Machine translation turns the message into another language for a support team abroad.
  • Summarisation shortens long text, either by picking the key sentences (extractive) or by writing a new, shorter version (abstractive).
  • Question answering finds or writes the answer to a question from a document or a knowledge base.

Speech adds two more tasks at the edges: speech recognition turns spoken words into text before NLP starts, and text-to-speech turns the output back into a voice.

What is the difference between NLP, NLU, NLG and an LLM?

NLP is the whole field; natural language understanding (NLU) is the part that works out meaning, natural language generation (NLG) is the part that writes text, and a large language model (LLM) is one kind of model that can do both.

TermWhat it coversExample output
NLPThe whole field of computers and languageEvery task in this post
NLUReading: meaning, intent, entities"Intent: refund, order 4417"
NLGWriting: text from data or a prompt"Your refund has been issued."
LLMOne model type that does bothA full drafted reply

NLU and NLG are two directions of the same work: a support bot uses NLU to read the message and NLG to write the reply.

Is an LLM the same as NLP?

No: an LLM is a tool inside NLP, not a replacement for the field. A large language model is a neural network trained on a very large amount of text to predict the next token, and that single skill lets it classify, translate, summarise and answer questions when you describe the task in a prompt. What Is an LLM? explains how these models are built and trained.

Many tasks that once needed a separate model now work by asking an LLM in plain language. Smaller task-specific models still have a place: a compact classifier trained on your own labelled emails costs less per message, gives the same label for the same input every time and can run on your own servers, which matters at high volume or with personal data.

Why is human language hard for computers?

Human language is hard because the same words can mean different things, and the meaning often depends on context, tone and knowledge the text never states.

  • Ambiguity. "The bank is closed" can be about money or a river; "I saw her duck" can describe a bird or a movement. People resolve this from context without noticing; a model has to learn it.
  • Sarcasm and tone. "Great, another late delivery" uses a positive word to complain. Simple sentiment models that count positive and negative words get this wrong.
  • Domain language. "Positive" is good news in a review and bad news in a medical test result. A model trained on news articles stumbles on legal contracts, medical notes or social media slang.
  • Languages with fewer resources. Models learn from the text available to them, so they are often weaker in languages with less digital text than English. Languages that build long words from suffixes add another layer: the Turkish word "evlerinizden" (from your houses) packs a root and three suffixes into what English says in three words.
  • Bias. Models learn from human-written text, including its stereotypes, and can repeat them in their labels or answers.
  • Spelling and mixing. Typos, emoji and messages that switch languages mid-sentence are everyday input and still trip models up.

Where is NLP used?

NLP is used wherever an organisation has more text than people to read it. Customer service teams route tickets and flag urgent messages. E-commerce teams mine reviews for recurring complaints and make site search understand questions instead of exact keywords.

Legal teams search contracts for specific clauses, hospitals turn doctors' free-text notes into structured records, finance teams track news for mentions of a company and recruiters parse job ads into comparable fields. Outside business, NLP powers live captions, translation apps and voice assistants.

Where do proxies fit into NLP projects?

Proxies matter when an NLP project needs public web text as it appears in a specific country. Reviews, news coverage, job ads and product descriptions differ between markets and languages, and a site often shows a different local version depending on where the visitor connects from.

A Residential Proxy sends each request through a home internet connection in the country and city you choose, so a team analysing sentiment in German reviews or job-ad wording in Brazil collects the same public pages a local visitor sees.

Our market research page explains how teams use proxies to see prices, promotions and product ranges as buyers in a target market see them.

For larger collection jobs, our data scraping solutions page covers the proxy pools built for them, and our sales team can discuss managed page extraction. Whichever route you choose, collect public pages only, follow robots.txt and each site's terms, keep request rates low, use an official API where one exists and leave out personal data the analysis does not need.

Common mistakes

  • Starting with a model instead of a question. Decide which label or answer you need from the text before choosing a method.
  • Skipping evaluation on your own data. A model that scores well on public benchmarks can still mislabel your customers' wording; label a few hundred of your own examples and test against them.
  • Assuming English results carry over. Check quality separately for every language you process.
  • Ignoring the early pipeline steps. Broken encoding, HTML leftovers and bad sentence splits cause many "model" errors.
  • Using an LLM for every task. For a fixed, high-volume classification job, a small trained model is often cheaper and more consistent.
  • Feeding personal data into third-party tools without checking the terms. Names, emails and order details in text are personal data under laws such as the GDPR; remove what the task does not need.

Decision guide

NeedUse
Sort high volumes of text into fixed labelsA trained text classifier
Find names, places, dates and order numbersNamed entity recognition
Measure the tone of reviews or postsSentiment analysis, checked on your data
Draft replies, summaries or translationsAn LLM, with a person reviewing the output
Answers from your own documentsSearch over those documents plus an LLM
Text that must stay on your serversAn open model you host yourself
Local web text from several countriesPublic pages collected via local proxies

Frequently asked questions

What is an example of natural language processing?

A spam filter is one of the most common examples: it reads each incoming email and decides from the wording whether it is spam. Others are machine translation, voice assistants that turn speech into commands, autocomplete on a phone keyboard and chatbots that answer customer questions.

Is ChatGPT an example of NLP?

Yes: ChatGPT is an application of NLP built on large language models, which are one type of NLP model. It uses language understanding to read your prompt and language generation to write the answer, the two halves of NLP described above.

What is the difference between NLP and machine learning?

Machine learning is a general method for learning patterns from data, and NLP is a field that applies it, among other methods, to human language. Machine learning also powers image recognition, fraud detection and recommendations, which have nothing to do with language, while early NLP used hand-written rules with no machine learning at all.

What does NLP mean in psychology?

In psychology and coaching, NLP stands for neuro-linguistic programming, a communication and self-help method that has nothing to do with computers. In AI, data science and software, NLP always means natural language processing.

Which programming language is used for NLP?

Python is the usual choice, because widely used NLP libraries such as NLTK, spaCy and Hugging Face Transformers are written for it. Many teams also call an LLM through an API from whatever language their application already uses.

Does NLP work in languages other than English?

Yes, but quality varies by language. Models are usually strongest in English and other languages with a lot of digital text, and weaker in languages with less training data or with complex word structure, so test any model on real text in each language you need.

Summary

Natural language processing is the field that lets computers read, understand and write human language. It moved from hand-written rules in the 1950s to statistics around 1990, then to neural networks and, since 2017, to transformers and the large language models built on them. A system splits text into sentences and tokens, represents them as numbers and runs a model for a task such as classification, entity recognition or translation. Language stays hard because of ambiguity, sarcasm and uneven data across languages, so test any NLP system on your own text and match the tool to the task: a small model for high-volume labels, an LLM for drafting and open-ended questions.

Ask ChatGPTAsk Claude