简体中文
EnglishWhat Are the Key Changes in quanteda Version 4?
Updated
quanteda 4.0 is a major release of the R package for quantitative text analysis, and it changes three things that affect most users: the tokeniser, the internal representation of tokens, and the set of available functions. The tokeniser now follows new Unicode and ICU-compliant rules, so it handles more languages more consistently. Tokens objects are now built on external pointers, which the project documents as a performance improvement. And previously deprecated functions have been removed, so code that still calls them will break until it is updated.
If you are starting a new project on version 4, none of this requires action beyond installing the current release. If you are upgrading an existing script, the removal of deprecated functions is the change most likely to stop your code from running.
A smarter default tokeniser
The release notes describe "significant new tokeniser rules that makes the default tokeniser smarter than ever before," with new Unicode and ICU-compliant rules that let it "work more consistently with even more languages."
In practice this means the default behaviour of tokenisation changes. If your analysis depends on exact token counts — for example, a word-frequency table or a dictionary that matches specific strings — you should re-run your pipeline and compare outputs rather than assume the old results still hold. The same text can now split differently at the margins, especially for punctuation, symbols, and non-Latin scripts.
External pointer tokens objects
Version 4 introduces a new external pointer representation for tokens objects. The project published a dedicated article explaining these objects and a separate article with performance benchmarks for them, alongside the tokeniser improvements.
The practical consequence is that tokens objects behave differently under the hood. Most user-facing functions are designed to work the same way, but code that inspects or manipulates the internal structure of a tokens object — rather than going through the API — is the kind of code most likely to need revision.
Removed deprecated functions
The release "further improves function consistency by removing previously deprecated functions." This is the change that most often breaks existing scripts, because a call that previously produced a warning now produces an error.
The full list of changes, improvements, and deprecations is in the v4 changelog, which is the authoritative place to check whether a function you rely on was removed or replaced.
What did not change
The modular structure of the quanteda family, established across earlier releases, carries into version 4. The core package holds the natural language processing and textual data management functions, while the modelling, statistics, and plotting functions live in separate packages:
| Package | Contains |
|---|---|
| quanteda | Core NLP and textual data management functions |
| quanteda.textmodels | Text models and supporting functions, the textmodel_*() functions |
| quanteda.textstats | Statistics for textual data, the textstat_*() functions |
| quanteda.textplots | Plots for textual data, the textplot_*() functions |
The split happened progressively: textmodels was separated at v2, and textstats and textplots at v3. Version 4 does not undo that structure, so if you are migrating from v2 or v3, your package-loading code should already reflect it.
How to decide what to do
- New to quanteda: install version 4 and follow the current documentation. The tokeniser improvements and performance work are the reason to be on this release rather than an older one.
- Upgrading an existing project: read the v4 changelog first, then run your pipeline end to end and diff the outputs. Treat any change in token counts or feature frequencies as expected until you have checked it, not as a bug.
- Maintaining a package that depends on quanteda: check whether you call any deprecated functions and whether you touch tokens internals directly. Those are the two areas the release explicitly changed.
- Working with many languages or non-Latin scripts: the Unicode and ICU-compliant tokeniser rules are the main reason version 4 may produce better results than earlier versions on the same input.
The project's own articles on external pointer tokens objects, the performance benchmarks, and the changelog are the sources to consult for specifics; this page summarises what they cover rather than replacing them.