quanteda.io

What is quanteda?

quanteda is an open-source R package for managing and analyzing text quantitatively. It is maintained by Kenneth Benoit and Kohei Watanabe, with development supported by the Quanteda Initiative CIC, and it is designed for R users who want to move from raw documents to statistical analysis of language without leaving the R environment.

Its scope covers a full workflow: building and managing a corpus, tokenizing text and forming n-grams, inspecting keywords in context, constructing sparse document-by-feature matrices, analyzing keywords, computing feature similarities and distances, applying content dictionaries, and running supervised or unsupervised machine learning. Visualization is available through the wider package family.

What it is good for

• Researchers and students who need serious text-analysis capability without commercial software costs. The project explicitly positions itself as matching or exceeding many end-user applications that are expensive and closed source.
• R programmers comfortable with code. The API aims to let you accomplish powerful analysis in few steps, and consistent design lowers the learning curve for NLP and quantitative text analysis.
• Multilingual work, since version 4 introduced Unicode and ICU-compliant tokenizer rules intended to behave consistently across more languages.

The package family

Since version 3, quanteda has been split into modular packages:

The main trade-off is the requirement to know R. If you work in Python, prefer a graphical interface, or need a hosted service rather than a library, quanteda will not fit your setup. If you already use R and want reproducible, scriptable text analysis, the modular design means you install only the components you need.

A useful first step is to visit quanteda and read the "How To..." section, then try a small corpus of your own documents to see how tokenization and a document-feature matrix look before committing to a larger project.

How does quanteda compare to other text analysis software for R users?

For R users, quanteda is best understood as a native-R framework rather than a point-and-click alternative: it handles the whole path from corpus import and tokenisation through to sparse document-feature matrices and then into models, statistics and plots. Its strongest practical advantage is that it stays inside R, so the output of text processing is a data structure you can immediately feed into the modelling or visualisation code you already use. The comparison below sets out where that helps and where it costs you.

How it compares

The design emphasis on consistency is the detail that matters most in daily use. Once you learn how quanteda names and structures things, the same logic carries from corpus creation to token handling to text models and plots, which shortens the gap between a first script and a reproducible pipeline.

What changed in version 4

quanteda 4 is a major release that improves functionality and performance, removes previously deprecated functions, and adds tokeniser rules using Unicode and ICU-compliant handling so the default tokeniser behaves more consistently across more languages. The project documents this in an article on the new external pointer tokens objects, a set of performance benchmarks, and a full changelog. If you maintain older scripts, the removal of deprecated functions is the item most likely to break them on upgrade.

The family of packages

The project split into modular packages from v2 and v3 onward:

• quanteda — core natural language processing and textual data management
• quanteda.textmodels — the textmodel_*() functions
• quanteda.textstats — the textstat_*() functions
• quanteda.textplots — the textplot_*() functions

This matters when you compare tools: a lightweight install gives you core processing, while modelling, statistics and plotting are opt-in. Budget a little time for discovering which package a given function now lives in, especially if you are following older tutorials.

A concrete reader scenario

Suppose you are a graduate researcher with a few thousand open-ended survey responses and a supervisor who wants cross-tabs by respondent group. You would build a corpus, tokenise, form a document-feature matrix, apply a dictionary or keyword analysis, then hand the matrix to a model or plot — all in one R script, with the group variable carried alongside. A GUI tool might get you to a word cloud faster, but joining results back to respondent metadata and re-running the analysis after a coding fix is where the R-native route pays off. If your team has no R users at all, the programming requirement is a genuine barrier rather than a minor one.

How to decide

Choose quanteda if you already work in R and want text processing to behave like the rest of your analysis. Choose a general-purpose R NLP library if your main need is linguistic annotation rather than corpus and matrix management. Choose a GUI or commercial application if nobody on the team codes and the analysis is exploratory rather than part of a repeatable pipeline.

A useful next step is to take one small, real dataset you already understand and run it end to end: corpus, tokens, document-feature matrix, one statistic, one plot. That single pass tells you more about fit than any feature list. The official project site is quanteda, and if you want to see how the approach compares with a broader R text-mining toolkit, the Tidy Text Mining book is a reasonable reference point.

How do I install and load the quanteda package in R?

Install quanteda from CRAN with install.packages("quanteda"), then load it in each session with library(quanteda). That is the standard path for the current R package described on quanteda.

Install

install.packages("quanteda")

If you use RStudio, you can also install it from the Packages pane, but the console command is the quickest and most reproducible.

Load

library(quanteda)

Run this at the start of every R session. Installing once is enough; loading is per session.

A quick check

packageVersion("quanteda")

This confirms which version is active. If you expect version 4 features such as the newer tokeniser rules, check that the reported version is 4.x.

The modular family

The project split quanteda into separate packages, so some functions live elsewhere and must be installed and loaded separately:

install.packages(c("quanteda.textmodels", "quanteda.textstats", "quanteda.textplots"))
library(quanteda.textstats)

If installation fails

• On Linux, a source install may need system libraries for dependencies; install binary packages where available.
• If you see "there is no package called 'quanteda'", the install step did not complete; rerun it and read the console output.
• If a textmodel_*() or textstat_*() function is missing, you likely loaded only the core package; install and load the relevant modular package.

A practical next step: build a small corpus from a vector of strings and tokenise it, which exercises the core workflow before you add models or plots.

corp <- corpus(c(d1 = "Text analysis in R.", d2 = "Quanteda makes it manageable."))
toks <- tokens(corp)

What are the differences between quanteda and its companion packages like quanteda.textmodels?

quanteda is the core R package for managing and analyzing text; the companion packages add specialized modeling, statistics and plotting functions that were split out of the main package. In practice, you install and load quanteda for the data-handling pipeline, then add a companion only when you need its particular capability.

What each package covers

• quanteda — the core natural language processing and textual data management functions: building and managing corpora, tokenizing, forming n-grams, keywords-in-context, and constructing sparse document-by-feature and feature co-occurrence matrices.
• quanteda.textmodels — the text models and their supporting functions, i.e. the textmodel_*() functions. Split from the main package at v2.
• quanteda.textstats — statistics for textual data, i.e. the textstat_*() functions. Split at v3.
• quanteda.textplots — plots for textual data, i.e. the textplot_*() functions. Split at v3.

How to think about the split

The naming convention is the practical guide: if a function starts with textmodel_, it lives in quanteda.textmodels; textstat_ functions live in quanteda.textstats; textplot_ functions live in quanteda.textplots. Everything else in the core workflow belongs to quanteda itself.

A concrete scenario

Suppose you are a graduate student with a few thousand policy documents and no budget for commercial text analysis software. You would use quanteda to read the documents into a corpus, tokenize them, remove stopwords and build a document-feature matrix. If you then want to scale documents along a dimension or classify them, you load quanteda.textmodels for the fitting step. If you want keyness scores or lexical diversity measures, you load quanteda.textstats. If you want a word cloud or a frequency plot, you load quanteda.textplots. Each companion is a separate install and a separate library() call, which keeps the core package lighter but means you must track which package owns which function.

Decision criteria

• Start with quanteda alone if your task is data preparation and exploration.
• Add a companion package only when you reach the step it covers; there is no need to install all of them up front.
• If you are maintaining older scripts, note that the split happened progressively (v2 for textmodels, v3 for textstats and textplots), so code written against a pre-split version may need its library() calls updated.
• Because the packages share the quanteda data structures, mixing them is straightforward — but function names are namespaced, so a textmodel_* call will fail if quanteda.textmodels is not loaded.

To get oriented, work through the "How To..." examples on the quanteda site, then check the documentation for the specific companion package when you reach modeling, statistics or plotting.

How can I use quanteda for keyword analysis and text scaling?

quanteda handles both keyword-in-context work and text scaling inside one R workflow: you build a corpus, tokenise it, then either inspect terms in context or convert documents into a document-feature matrix that feeds a scaling model. Keyword work is largely descriptive and exploratory; scaling is inferential and produces per-document positions on one or more dimensions. Choosing between them depends on whether you want to read passages around a term or place documents on a continuum.

Keyword analysis

The package provides keyword-in-context inspection and feature co-occurrence matrices, so you can move from a term of interest to the passages containing it, then to which terms travel with it. A practical sequence:

1. Build a corpus and tokenise, applying the Unicode-aware tokeniser rules described for version 4 if your texts include multiple scripts.
2. Select or group terms, then view keywords in context to check whether a term is used consistently or carries several senses.
3. Compute feature co-occurrence or similarity statistics to see which words cluster around your target.
4. Apply a content dictionary if you need to score predefined categories rather than discover them.

A concrete case: a policy researcher studying "risk" across regulatory consultations would first pull keyword-in-context lines to see whether the term appears in prudential, environmental or safety senses, then use co-occurrence statistics to separate those usages before counting anything. Skipping the context step is the common failure — raw frequency counts merge distinct meanings and inflate the apparent importance of a polysemous word.

Text scaling

Scaling in this ecosystem is handled by the companion modelling package, which holds the textmodel_*() functions. The usual path is corpus → tokens → document-feature matrix → scaling model, with the matrix as the shared input for both supervised and unsupervised approaches. The core package supplies the matrix construction and manipulation; the modelling package supplies the estimation.

Two practical points matter more than the choice of algorithm:

• Feature selection drives the result. Scaling solutions are sensitive to which terms enter the matrix. Removing very rare and very common terms, or restricting to a domain vocabulary, usually changes the dimensions more than switching models does.
• Validate dimensions against known documents. If you have texts whose positions you already know, check whether they land where expected before interpreting the rest.

Which to use

The trade-off is interpretability versus generality. Keyword and dictionary work stays close to the text and is easy to defend to a non-technical audience, but it does not generalise beyond the terms you specify. Scaling summarises whole documents and can reveal structure you did not anticipate, but the dimensions often need labelling and can shift when you change the feature set.

Next step

Take a small sample of your own documents — twenty to fifty is enough — and run the full path once: corpus, tokens, document-feature matrix, keyword-in-context on one term, then a single scaling model. If the scaling dimensions do not correspond to anything you can name from reading the documents, adjust feature selection before scaling up. The framework is documented at quanteda, and the modular structure means you can install the core package first and add the text models, statistics and plotting packages as your analysis requires them.

What are the new tokenizer features in quanteda version 4?

quanteda 4.0's main tokenizer change is a smarter default: the package now applies new Unicode- and ICU-compliant tokenisation rules, so the out-of-the-box tokeniser handles more languages more consistently than earlier releases. The project also documents the change alongside new "external pointer" tokens objects, which are tied to performance improvements rather than tokenisation rules themselves.

What this means in practice

• Better multilingual defaults. If you work with text containing non-Latin scripts, accents, or mixed-script documents, the v4 tokeniser is designed to segment it more reliably without custom rules. The gain is largest for teams that previously had to patch the tokeniser for each language.
• Fewer surprises when upgrading. The v4 release also removes functions deprecated in earlier versions, so the tokeniser change arrives with a broader API cleanup. Code that relied on old tokenisation behaviour, or on removed functions, is the most likely to need attention.
• Performance is a separate story. The external pointer tokens objects are an internal redesign aimed at speed and memory use; the tokeniser improvements are described in their own benchmark article. Treat "smarter tokenisation" and "faster tokens" as two distinct v4 changes.

Where to look next

The quanteda site links three v4 resources: an article on external pointer tokens objects, a benchmarks article covering both the new tokens objects and tokeniser improvements, and the v4 changelog listing all changes and deprecations. For upgrade planning, read the changelog first, then the tokeniser benchmarks if throughput matters to you. The core package is documented at quanteda, with text models, statistics, and plots split into companion packages.

A concrete check for your own corpus

If you maintain a multilingual pipeline, run a small before-and-after comparison on a sample of your documents after upgrading: count tokens per document and inspect a few keyword-in-context views in the languages you care about. A stable token count with sensible boundaries suggests the new default rules suit your data; large shifts are worth tracing to specific scripts or punctuation before you rebuild downstream models.