N-gram

What is an n-gram?

An n-gram is a sequence of n adjacent items from a larger dataset. In language processing, the items are often words or characters, and n-grams help systems identify patterns in what tends to appear next.

Adjacent matters. An n-gram preserves order and does not skip items, so “machine learning” is a word bigram, while “machine useful” is not if another word occurs between them. Items can be words, characters, tokenizer-created tokens, speech sounds, website clicks, DNA bases, or protein symbols. N-grams are a general pattern-extraction method, not just a text concept.

N-gram types: unigrams, bigrams, trigrams, and beyond

The value of n sets sequence length. The sentence “birds build nests carefully” shows what each type retains.

Word n-grams help when complete words carry the signal. Character n-grams can handle typos, variations, and unfamiliar terms. Larger n values are more specific, but many sequences occur only once. This is called sparsity.

How n-grams work in natural language processing

N-gram processing turns raw material into short, countable sequences. A language model can use the counts to estimate likely continuations.

1. Collect a corpus, the body of text or other ordered data to analyze.
2. Normalize data by choosing how to handle case, punctuation, numbers, accents, and symbols.
3. Split data into tokens, such as words, subwords, or characters.
4. Move a window of size n across each sequence, one position at a time.
5. Count each resulting n-gram.
6. Use counts as machine-learning features or convert them into prediction probabilities.

An n-gram language model uses the Markov assumption: the next word depends mainly on a limited number of preceding words, not the entire passage. A bigram model estimates the chance of “nests” after “build” by comparing the count of “build nests” with the total count of “build.” Standard explanations of this probability-based approach appear in Stanford's language modeling materials.

A concrete n-gram example

Take “Cats chase small mice.” Its word unigrams are cats, chase, small, and mice. Its bigrams are cats chase, chase small, and small mice. Its trigrams are cats chase small and chase small mice. Windows overlap, so “chase” belongs to two bigrams.

Language-modeling workflows may add start and end markers. A start marker lets a bigram such as start-cats record how sentences begin. Preprocessing changes output: preserving the period can create “mice.” as a token, while lowercasing makes “Cats” and “cats” identical. The choice should match the task.

Key components of an n-gram model

A corpus supplies examples, and a vocabulary is the set of distinct tokens kept from it. A token is one counted unit. Frequency counts record occurrences, while a context window is the preceding n minus one tokens used to predict the next token. Conditional probability expresses that prediction.

An out-of-vocabulary item is absent from the model's vocabulary. Unseen n-grams create a zero-frequency problem, assigning zero probability to a plausible phrase. Smoothing reserves or redistributes a small probability for unseen events. Backoff uses a shorter n-gram when a longer one is missing, while interpolation combines estimates from several sizes, such as unigram, bigram, and trigram estimates.

Common uses of n-grams

N-grams are often one component of a broader system.

• Search autocomplete and predictive typing use frequent sequences to suggest next words or queries.
• Spelling correction and language identification often use character n-grams to compare writing patterns.
• Text classification and sentiment analysis use word or character patterns for labels such as topic, spam, or positive sentiment.
• Plagiarism and near-duplicate detection compare overlapping sequences to find similar passages.
• Speech recognition can use word-sequence probabilities to choose between acoustically similar phrases.
• Code analysis can examine token sequences for syntax patterns, repetitive code, or unusual commands. This can complement AI pair programming workflows.
• Cybersecurity tools can inspect command, log, or network-event sequences for patterns needing review.
• Bioinformatics uses n-grams called k-mers to examine DNA and protein sequences.

Benefits of using n-grams

N-grams remain useful because their behavior is visible and easy to test.

• They are transparent. Teams can inspect sequences influencing a result.
• They are fast to create and work well as a baseline before complex models.
• They fit many traditional machine-learning methods, including linear classifiers.
• They reveal local phrase patterns that single-word counts miss.
• Character n-grams are relatively resilient to misspellings, punctuation variation, and unseen word forms.

Practical limits of n-grams

The local focus that makes n-grams simple also limits what they can understand.

• Longer sequences are sparse, so storage and memory needs can grow quickly.
• They cannot reliably connect ideas far apart in a sentence or document.
• Unseen sequences require smoothing, backoff, or safeguards against zero probabilities.
• Results are highly sensitive to tokenization and normalization choices.
• Training data can carry social, linguistic, or historical bias into detected patterns.
• Frequency is not proof of meaning, intent, truth, or causation. A common phrase may be ambiguous.

How to choose the right n-gram size

Choose n by testing the smallest representation that captures the needed signal.

• Start with word unigrams and bigrams for many document-classification tasks.
• Use trigrams when short phrase order matters and enough training data is available.
• Try character 3- to 5-grams for noisy text, names, spelling variants, or language identification.
• Consider language structure. Languages with rich word endings may benefit from character features.
• Test alternatives on held-out data, not training data.
• Control feature growth with minimum-frequency thresholds, vocabulary limits, or hashing.

N-gram models vs modern neural language models

Neural models generally capture broader context, but n-grams still have practical roles.

N-grams are not obsolete. They help check data quality, create simple production systems, and show whether a complex model offers meaningful improvement. For related concepts, see the AI glossary.

Common implementation mistakes and better practices

Reliable results depend on evaluation discipline as well as the chosen n value.

• Prevent data leakage by splitting training and evaluation data before fitting vocabularies or frequency thresholds.
• Apply identical preprocessing to training, validation, and live input.
• Keep sentence boundaries when they matter, especially for language modeling and autocomplete.
• Do not select n by intuition alone. Compare candidates on held-out data.
• Use smoothing for probability models and prune rare, noisy sequences when appropriate.
• Evaluate with task-relevant measures, such as precision and recall for detection, then review errors.
• Document preprocessing choices so results can be reproduced.
• Protect sensitive text. N-grams can preserve names, addresses, or other source-data fragments.