robots-parser
robots-parser turns a robots.txt string into a synchronous policy object for one origin. You supply both the robots.txt URL and its already-downloaded contents, then ask whether a target URL is allowed for a crawler user agent, which source line matched, and which crawl delay, sitemap URLs, or preferred host were declared. Version 3.0.1 implements wildcard and end-of-line matching and longest-rule precedence. It parses policy text only; it does not fetch, cache, schedule, throttle, or validate a crawler.
A focused and well-tested parser when you control fetching and can stay within the published 3.0.1 surface. Do not mistake it for a crawler policy client, and avoid it if the CommonJS typing defect, undocumented tri-state trap, or lag between main and npm is unacceptable.
Use it if
- You already fetch robots.txt yourself and want a dependency-free synchronous allow or deny decision for many URLs on the same origin
- You need wildcard and $ matching plus longest-match precedence that follows the algorithm described in the package source
- You need the 1-based source line that produced a decision for crawler logs, debugging, or an operator-facing explanation
- You also need simple access to Crawl-delay, Sitemap, and Host directives even though the latter two sit outside the core allow and disallow decision
- You want a client that fetches robots.txt, applies HTTP status rules, caches results, limits response size, and schedules crawl delays: the factory accepts a string and implements none of that network policy
- You need document validation or parse diagnostics: open issue 34 asks for validation, while the parser silently ignores unknown or malformed lines and exposes no warnings array
- You plan to call isDisallowed on untrusted or cross-origin URLs: published 3.0.1 implements it as !isAllowed(), so an undefined scope result becomes true even though the README says undefined; open issue 41 reports related surprising root decisions
- You need the isExplicitlyDisallowed API shown on the repository's current README: it was added on the main branch after npm 3.0.1 and is absent from the published tarball
- You require clean modern TypeScript and ESM packaging: 3.0.1 is CommonJS, and its shipped declaration includes an empty ambient module plus an ES default export that does not accurately describe module.exports; the main branch type fix is not yet published
- You need full crawler compliance from one dependency: robots.txt rules do not replace URL canonicalization, request-rate enforcement, redirects, response-status handling, caching, or your own identification and contact policy
Setup reality
There are no runtime dependencies or native builds, and Node 10 or newer is the declared floor. The easy part is require('robots-parser') and passing two strings. The production work surrounds it. This package never makes an HTTP request, so your crawler must resolve /robots.txt, follow the redirect policy you choose, cap bytes and time, decode text, interpret 4xx and 5xx responses, cache by origin, and decide when to refetch. Construct one parser per robots.txt origin; isAllowed returns undefined when the checked URL's protocol, hostname, or effective port does not match the robots.txt URL. Do not collapse that third state into false without an explicit policy. More sharply, avoid published 3.0.1's isDisallowed for scope validation because its implementation negates undefined into true. The parser strips a user-agent version suffix after '/', lowercases it, and chooses an exact agent group or the global * group. It does not perform substring product-token matching, so pass the product token you intend to match. Empty and malformed directives are largely ignored rather than reported. Crawl-delay is returned as a number but not enforced. Sitemap values are returned as written and are not fetched or validated. Relative URL matching works only when both the robots URL and checked URL are relative. The npm release is CommonJS and its included TypeScript declaration is inaccurate; projects using Node16 or NodeNext module resolution may need esModuleInterop, a local declaration patch, or a wrapper around require until the main-branch export = fix reaches npm. Also note that the repository README is ahead of 3.0.1 and documents isExplicitlyDisallowed, which the installed release does not contain.
Patterns
Parse a robots.txt stringparse-policy
const robotsParser = require('robots-parser');
const body = [
'User-agent: *',
'Disallow: /private/',
'Allow: /private/public.html',
].join('\n');
const robots = robotsParser('https://example.com/robots.txt', body);
The first argument establishes the allowed protocol, hostname, and port scope. The package does not fetch the second argument for you.
Handle the allow decision as three statescheck-allowed
const decision = robots.isAllowed(
'https://example.com/private/report.csv',
'ExampleBot/1.0',
);
if (decision === true) enqueue();
else if (decision === false) skip();
else throw new Error('URL is outside this robots.txt scope');
undefined means the URL is invalid for this parser, commonly because origin or port differs. Do not treat it as an ordinary disallow without deciding that policy yourself.
Derive disallowed without the negation trapcheck-disallowed-safely
const allowed = robots.isAllowed(targetUrl, 'ExampleBot');
const disallowed = allowed === false;
if (allowed === undefined) {
console.error('wrong robots policy for', targetUrl);
} else if (disallowed) {
console.log('blocked by robots.txt');
}
Published 3.0.1's isDisallowed returns !isAllowed, so it turns undefined into true. Comparing the allow result explicitly preserves scope errors.
Check separate user-agent groupsmatch-user-agent
const robots = robotsParser('https://example.com/robots.txt', `
User-agent: ExampleBot
Disallow: /exports/
User-agent: *
Disallow: /admin/
`);
robots.isAllowed('https://example.com/exports/a.csv', 'ExampleBot/2.3'); // false
robots.isAllowed('https://example.com/admin/', 'OtherBot'); // false
The parser lowercases agents and removes text after the first slash. An exact agent group takes precedence over the global group rather than merging with it.
Use wildcard and end-anchored rulesuse-wildcards
const robots = robotsParser('https://example.com/robots.txt', `
User-agent: *
Disallow: /*?preview=*
Disallow: /draft.pdf$
`);
robots.isAllowed('https://example.com/post?preview=1'); // false
robots.isAllowed('https://example.com/draft.pdf'); // false
robots.isAllowed('https://example.com/draft.pdf?download=1'); // true
A trailing $ anchors the match to the end of pathname plus query. Without $, a directive is a prefix match.
Override a broad disallow with a longer allowallow-specific-path
const robots = robotsParser('https://example.com/robots.txt', `
User-agent: *
Disallow: /assets/
Allow: /assets/public/
`);
robots.isAllowed('https://example.com/assets/private/logo.svg'); // false
robots.isAllowed('https://example.com/assets/public/logo.svg'); // true
The longest matching pattern wins. For equal-length matches, Allow wins over Disallow.
Report the source line that matchedexplain-matching-rule
const line = robots.getMatchingLineNumber(
'https://example.com/private/report.csv',
'ExampleBot',
);
if (line === -1) console.log('No directive matched');
else console.log(`Decision came from robots.txt line ${line}`);
Line numbers are 1-based. The method returns -1 when no rule matched, including some invalid-scope cases, so pair it with isAllowed when scope matters.
Read and enforce Crawl-delayread-crawl-delay
const seconds = robots.getCrawlDelay('ExampleBot');
const delayMs = seconds === undefined ? 1000 : seconds * 1000;
await new Promise((resolve) => setTimeout(resolve, delayMs));
The parser only returns the number. Your scheduler must enforce it, coordinate concurrent workers, and decide a default when the directive is absent.
Collect declared sitemap URLslist-sitemaps
for (const sitemapUrl of robots.getSitemaps()) {
const url = new URL(sitemapUrl);
if (url.protocol === 'https:') sitemapQueue.add(url.href);
}
Values are returned as written. Validate protocols, origins, duplicates, and fetch limits before adding them to a crawler queue.
Read the non-standard Host directiveread-preferred-host
const preferredHost = robots.getPreferredHost();
if (preferredHost !== null) {
console.log('Declared preferred host:', preferredHost);
}
Host is not part of the core RFC 9309 allow and disallow rules. Treat it as advisory metadata, not a redirect instruction.
Parse and check relative URLs togethermatch-relative-urls
const robots = robotsParser('/robots.txt', `
User-agent: *
Disallow: /internal/
`);
robots.isAllowed('/public/index.html'); // true
robots.isAllowed('/internal/report.html'); // false
Relative matching is allowed only when both the robots.txt URL and checked URL are relative. Do not mix a relative base with an absolute target.
Fetch with explicit limits before parsingfetch-before-parsing
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 5000);
try {
const res = await fetch('https://example.com/robots.txt', { signal: controller.signal });
if (!res.ok) throw new Error(`robots.txt HTTP ${res.status}`);
const bytes = new Uint8Array(await res.arrayBuffer());
if (bytes.byteLength > 512_000) throw new Error('robots.txt too large');
const robots = robotsParser(res.url, new TextDecoder().decode(bytes));
} finally {
clearTimeout(timer);
}
This is transport scaffolding, not a complete RFC status policy. Decide redirects, 4xx, 5xx, caching, and byte handling for your crawler before shipping.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| google-robotstxt-parser | npm | Choose it for a newer JavaScript port of Google's parser that targets both Node and browsers |
| @trybyte/robotstxt-parser | npm | Choose it when an RFC 9309-focused TypeScript implementation and current ESM-era packaging matter |
| robots-txt-parser | npm | Choose it when you want fetching, caching, and promise-based checks included with parsing |
| @flyyer/robotstxt | npm | Choose it for a pure TypeScript parser and guard with ESM and tree-shaking support |