parse-latin
parse-latin is a natural language tokenizer for Latin-script text. Give it a string and it returns an nlcst syntax tree: a RootNode of ParagraphNodes, containing SentenceNodes, containing WordNodes, WhiteSpaceNodes, PunctuationNodes, and SymbolNodes, each with exact position info. It is smart about the annoying cases: periods in abbreviations like e.g. do not end sentences, and words like non-profit or she's stay one token. It powers retext-latin in the retext/unified ecosystem, which is why a package with 57 stars sees millions of weekly downloads.
A quietly excellent single-purpose tokenizer: correct about the hard cases, fully typed, and effectively finished software. Use it (or parse-english) when you live in the unified/retext world or need offset-accurate tokens; look elsewhere for real linguistic analysis.
Use it if
- You need sentence and word segmentation with exact character offsets, for highlighting, readability scoring, or profanity and style checks over source text
- You are building or using retext plugins and want the same nlcst tree the whole unified ecosystem speaks
- Your text spans many Latin-script languages (French, Icelandic, Old English) and you want one tokenizer that handles them acceptably, plus passable results on Cyrillic or Georgian
- You want a small pure-JS dependency with full TypeScript types instead of a heavyweight NLP toolkit
- You expect actual NLP: there is no part-of-speech tagging, lemmatization, entities, or sentiment here, only tokenization into a tree; compromise or wink-nlp do the linguistic analysis
- You process English or Dutch specifically: parse-english and parse-dutch extend this parser with language-specific abbreviation handling and give better sentence splits
- You just need to split sentences quickly and do not care about offsets or tree structure: Intl.Segmenter is built into modern JS runtimes with zero dependencies
- You need CJK, Arabic, or other non-Latin scripts handled correctly: the name is the scope, and segmentation quality falls off outside Latin-like scripts
- You want visible active development: last push was October 2024; the flip side is that the author (wooorm) maintains a huge stable ecosystem and the issue tracker sits at zero
Setup reality
npm install parse-latin is the easy part; the package is ESM only, so CommonJS projects must use dynamic import() or stay on the old v5 line. The tree it returns is where the learning curve lives: nlcst is a unist-flavored AST, and to do anything useful you will also install helpers like nlcst-to-string to get text back out, unist-util-visit to walk nodes, and unist-util-inspect to see what you are working with. The parser itself brings six small runtime dependencies from the same ecosystem. API surface is tiny: new ParseLatin().parse(value), nothing to configure.
Patterns
Parse a string into an nlcst treeparse-text
import {ParseLatin} from 'parse-latin'
const tree = new ParseLatin().parse('A simple sentence.')
// RootNode > ParagraphNode > SentenceNode > Word/WhiteSpace/PunctuationESM only: require('parse-latin') throws in CommonJS; use await import('parse-latin') there.
Pretty-print the tree while developinginspect-tree
import {ParseLatin} from 'parse-latin'
import {inspect} from 'unist-util-inspect'
const tree = new ParseLatin().parse('Hi there. Bye now.')
console.log(inspect(tree))unist-util-inspect renders the nested nodes with positions; far easier than console.log on a deep tree.
Collect sentence strings from textextract-sentences
import {ParseLatin} from 'parse-latin'
import {visit} from 'unist-util-visit'
import {toString} from 'nlcst-to-string'
const tree = new ParseLatin().parse(text)
const sentences = []
visit(tree, 'SentenceNode', (node) => {
sentences.push(toString(node))
})Abbreviations like e.g. or 1. do not end sentences, which is the main reason to use this over splitting on periods.
Count words in a documentcount-words
import {ParseLatin} from 'parse-latin'
import {visit} from 'unist-util-visit'
let words = 0
visit(new ParseLatin().parse(text), 'WordNode', () => {
words++
})A word is one or more unicode letters or numbers, so 11:00 and N/A each count as a single WordNode.
Turn any node back into textnode-to-string
import {toString} from 'nlcst-to-string'
const sentence = tree.children[0].children[0]
console.log(toString(sentence)) // 'A simple sentence.'Nodes with children have no value property themselves; nlcst-to-string concatenates the leaf values for you.
Highlight a word using its source offsetsuse-position-offsets
visit(tree, 'WordNode', (node) => {
const {start, end} = node.position
const original = text.slice(start.offset, end.offset)
// also has line/column: start.line, start.column
})Offsets index the exact input string, so slicing the source with them always round-trips; this is the killer feature for linters and highlighters.
Use it via retext-latin in a unified pipelineretext-pipeline
import {retext} from 'retext'
import retextPos from 'retext-pos'
const file = await retext().use(retextPos).process('Hello world.')
// retext wraps parse-latin as its default parserIf you are running plugins (spell check, readability, profanity), let retext own the parsing instead of instantiating ParseLatin yourself.
Iterate paragraphs and their sentenceswalk-paragraphs
for (const para of tree.children) {
if (para.type !== 'ParagraphNode') continue
const sentences = para.children.filter(
(n) => n.type === 'SentenceNode'
)
// whitespace between sentences appears as WhiteSpaceNode siblings
}Paragraph children interleave SentenceNodes with WhiteSpaceNodes, so filter by type instead of assuming every child is a sentence.
Separate punctuation from symbolsfind-symbols
visit(tree, (node) => {
if (node.type === 'PunctuationNode') {
// . , ! ? and friends
} else if (node.type === 'SymbolNode') {
// $ % + and anything not letter/number/space/punctuation
}
})Some punctuation stays inside WordNodes (hyphens in non-profit, the apostrophe in she's), so a top-level visit will not see those.
Load in the browser without a bundlerbrowser-usage
<script type="module">
import {ParseLatin} from 'https://esm.sh/parse-latin@7?bundle'
const tree = new ParseLatin().parse('Salut tout le monde.')
</script>The ?bundle flag makes esm.sh inline the unist/nlcst dependencies; without it you get several extra module fetches.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| parse-english | npm | Your text is English; it extends parse-latin with English abbreviations and elision handling for better sentence breaks. |
| compromise | npm | You want part-of-speech tagging, matching, and text transformation in English, not just tokenization. |
| wink-nlp | npm | You need fast tokenization plus entities, sentiment, and POS in one typed package. |