P1 — robots.txt compliance (behavior spec)

Status: ready for test authoring Phase: P1 of the inform capability port Owner artifacts: src/sources/snapshot-fetchers/robots.ts (new), src/sources/snapshot-fetchers/website-ingest.ts (modified)

This document is the single source of truth for the test author and the implementer of P1. Neither reads the parent plan's phase section; everything needed is here.

0. Summary

akm's website source provider crawls a site (crawlWebsite in website-ingest.ts) with no awareness of /robots.txt. P1 makes akm a polite crawler: before crawling an origin it fetches and parses that origin's robots.txt, skips paths the site disallows for our product token, and honors Crawl-delay (clamped).

This is a deliberate behavior change. respectRobots defaults to true, so existing website sources may return fewer pages after upgrade. The opt-out is respectRobots: false on the website source descriptor.

1. Scope

In scope

• New module src/sources/snapshot-fetchers/robots.ts: a pure parser + matcher plus a crawl-scoped, per-origin caching policy object.
• crawlWebsite() in website-ingest.ts consults the policy before every page fetch and spaces requests by Crawl-delay.
• ensureWebsiteMirror() reads options.respectRobots and threads it down.
• respectRobots added to the website descriptor in the config zod schema, with the generated JSON schema regenerated.
• CHANGELOG entry and docs describing the change and the opt-out.

Explicitly out of scope for P1

2. Constants (pin these exactly)

Exported from robots.ts unless noted.

3. Public API surface (pin these signatures)

Tests and the implementation must agree on these. No barrel exports; import by path (../../src/sources/snapshot-fetchers/robots).

export type RobotsRuleKind = "allow" | "disallow";

export interface RobotsPathRule {
  readonly kind: RobotsRuleKind;
  /** Raw pattern text as written in robots.txt, e.g. "/private/", "/*.pdf$". */
  readonly pattern: string;
  /** Match specificity = pattern.length. Used for longest-match-wins. */
  readonly specificity: number;
}

export interface RobotsRuleSet {
  /** true => the entire origin is off-limits regardless of `rules`. */
  readonly disallowAll: boolean;
  readonly rules: readonly RobotsPathRule[];
  /** Already clamped to [0, MAX_CRAWL_DELAY_MS]; null when unspecified. */
  readonly crawlDelayMs: number | null;
}

export const ALLOW_ALL_RULES: RobotsRuleSet;    // { disallowAll: false, rules: [], crawlDelayMs: null }
export const DISALLOW_ALL_RULES: RobotsRuleSet; // { disallowAll: true,  rules: [], crawlDelayMs: null }

/** Pure. Never throws, never fetches. */
export function parseRobotsTxt(text: string): RobotsRuleSet;

/**
 * Pure. `url` is an absolute http(s) URL string. Matching uses
 * `pathname + search` of the parsed URL. Returns true when the URL cannot be
 * parsed (caller has already validated it; do not fail closed on a parse bug).
 */
export function isPathAllowedByRobots(rules: RobotsRuleSet, url: string): boolean;

export type RobotsFetchOutcome =
  | { readonly kind: "body"; readonly text: string }
  | { readonly kind: "unavailable" }   // 4xx / oversized / transport failure => allow all
  | { readonly kind: "unreachable" };  // 5xx => disallow all

/** Injected by website-ingest.ts. Receives the absolute `<origin>/robots.txt` URL. */
export type RobotsTxtLoader = (robotsUrl: string) => Promise<RobotsFetchOutcome>;

export interface RobotsPolicy {
  /** Resolves (and caches) the rule set for the URL's origin. */
  rulesFor(url: string): Promise<RobotsRuleSet>;
  isAllowed(url: string): Promise<boolean>;
  /** Clamped crawl delay in ms for the URL's origin; 0 when none. */
  crawlDelayMs(url: string): Promise<number>;
}

/** Crawl-scoped. Caches per origin (including in-flight promises). */
export function createRobotsPolicy(load: RobotsTxtLoader): RobotsPolicy;

/** Never fetches. isAllowed => true, crawlDelayMs => 0. Used when respectRobots is false. */
export function createAllowAllRobotsPolicy(): RobotsPolicy;

robots.ts must not import from website-ingest.ts (circular). The guarded fetch lives in website-ingest.ts and is passed in as a RobotsTxtLoader.

4. Behavior tables

4.1 parseRobotsTxt — grouping and directives

Input is the raw robots.txt body. Output is a RobotsRuleSet.

4.2 parseRobotsTxt — Crawl-delay

4.3 isPathAllowedByRobots — pattern matching

Matching target is url.pathname + url.search. Patterns are always anchored at the start of that string. * matches any sequence including empty. A single trailing $ anchors the end; a $ anywhere else is a literal $. All other regex metacharacters in the pattern are literal.

Specificity note: specificity = pattern.length counting * and $ characters as written. This is the Google/RFC 9309 convention.

4.4 RobotsTxtLoader outcome → rule set

Produced by createRobotsPolicy's cache layer.

4.5 loadRobotsTxt (the loader implemented in website-ingest.ts)

Signature: async function loadRobotsTxt(robotsUrl: string, options?: WebsiteValidationOptions): Promise<RobotsFetchOutcome>

Order of operations (this order is load-bearing for security — see §6.2):

1. assertWebsiteRequestUrl(robotsUrl, UsageError, options) — outside the try/catch, so guard rejections propagate.
2. await assertResolvedHostAllowed(new URL(robotsUrl).hostname, options) — also outside the try/catch.
3. Inside a try/catch: await fetchWebsiteResponse(robotsUrl, 0, options) (the existing helper: fetchWithRetry with { timeout: 15_000, retries: 1 }, redirect: "manual", manual redirect following re-guarded on every hop, capped at WEBSITE_MAX_REDIRECTS).
4. Map the response per the table below, reading the body with readBodyWithByteCap(response, ROBOTS_BYTE_CAP, { bodyTimeoutMs: ROBOTS_BODY_TIMEOUT_MS }).

Redirect to a different origin is permitted and the resulting body is used for the requesting origin's rules (rules are cached under the origin we asked about, not the origin that answered).

4.6 crawlWebsite integration

crawlWebsite's options gain respectRobots?: boolean.

const robots = options.respectRobots === false
  ? createAllowAllRobotsPolicy()
  : createRobotsPolicy((robotsUrl) =>
      loadRobotsTxt(robotsUrl, { allowPrivateHosts: options.allowPrivateHosts }));

The robots.txt URL is new URL("/robots.txt", origin).toString() — always the origin root, never relative to the start path.

4.7 ensureWebsiteMirror config coercion

coerceRespectRobots(value: unknown): boolean (private to website-ingest.ts):

Rationale for G-07 being loud rather than defaulting: silently ignoring a misspelled opt-out would make a user think robots is disabled while akm keeps enforcing it (or vice versa). ConfigError exits 78.

5. Inform-parity notes (deliberate divergences)

inform's reference is src/RobotsParser.js (CC-BY-4.0). akm's is a clean TypeScript rewrite. Where we differ, and why:

6. Security requirements

6.1 No module-level state carrying crawl semantics

The per-origin cache must live on the object returned by createRobotsPolicy, constructed inside crawlWebsite. A module-level Map of origin -> RobotsRuleSet (or any other structure that records what a specific crawl observed) is forbidden: the test preload's singleton reset does not know about it, the tripwire would not catch it, and rules would leak between crawls (and between tests) — including a disallowAll from one host poisoning another run.

This does not forbid module-level state that carries no crawl semantics: a pure memoization of a function of an individual object's own fields, keyed on that object's identity (e.g. a WeakMap<RobotsPathRule, CompiledPattern> compiling rule.pattern once), cannot leak meaning between origins, crawls, or tests — distinct parseRobotsTxt calls always allocate distinct rule objects, so distinct cache slots, reclaimed by the GC once unreachable. Frozen, empty-collection constants (ALLOW_ALL_RULES, DISALLOW_ALL_RULES) are likewise fine: Object.freeze makes them immutable by construction, so they cannot become a mutation channel between callers.

6.2 SSRF: the robots.txt URL is an attacker-influenced fetch

/robots.txt is fetched automatically for whatever origin the crawl reaches, including origins arrived at through redirects. It therefore gets exactly the same treatment as a page fetch:

• No raw fetch(). All traffic goes through fetchWithRetry from src/core/common.ts with an explicit timeout.
• assertWebsiteRequestUrl before every request, including every redirect hop.
• assertResolvedHostAllowed (resolve-then-validate) before every request, including every redirect hop.
• readBodyWithByteCap with ROBOTS_BYTE_CAP and an explicit bodyTimeoutMs.
• Redirects followed manually and capped at WEBSITE_MAX_REDIRECTS.
• Reusing fetchWebsiteResponse satisfies all of the above; do not write a second fetch path.

Guard rejections must not be swallowed into "allow all" at the point of the initial URL check (§4.5 F-12). The catch block must wrap only the network phase (step 3–4), never steps 1–2. Rejections that happen inside fetchWebsiteResponse on a later redirect hop may be swallowed to { kind: "unavailable" } — the guard has already prevented the request; only the error reporting is downgraded.

6.3 allowPrivateHosts must thread through

crawlWebsite(options.allowPrivateHosts) → loadRobotsTxt(..., { allowPrivateHosts }) → assertWebsiteRequestUrl / assertResolvedHostAllowed / fetchWebsiteResponse. Without this the loopback fixture servers in the integration tests cannot serve /robots.txt and the phase is untestable end to end. The hatch is test-only (shouldAllowPrivateWebsiteHostsForTests) and must not gain a new production entry point.

6.4 ReDoS / parser DoS

robots.txt content is fully attacker-controlled.

• Patterns must NOT be compiled to a RegExp. The required implementation compiles each pattern to a flat list of literal segments split on * (consecutive * collapsed to one) and matches it with a single greedy, non-backtracking left-to-right scan: the first segment anchors the start, each interior segment is located with one forward indexOf from the current cursor (cursor only moves forward, never backtracks), and the final segment either anchors the end (trailing $) or is located the same way. This runs in time bounded by segments × target length, independent of wildcard count. A regex-based compiler (* → .*, metacharacters escaped) is explicitly rejected, including the earlier draft of this section that proposed one: collapsing consecutive * only defuses adjacent-wildcard blowup (/****), not distinct wildcards separated by short literals (e.g. /*a*a*a*a*a*a*a*a$), which still drives a regex engine into catastrophic backtracking on a long near-miss input — see the dated review log entry below and the §6.4 regression test in tests/website-robots.test.ts. Reference implementation: compilePatternSegments / matchesCompiledPattern in robots.ts.
• The 512 KiB body cap bounds the number of rules.
• Compile each pattern at most once (at parse time or memoized), not once per URL check.

6.5 Error surface

• User-input paths (respectRobots coercion, start URL blocked): UsageError / ConfigError from src/core/errors.ts. Never a bare throw new Error.
• Diagnostics: warn() / warnVerbose() from src/core/warn.ts. Never console.*.
• A 404 on /robots.txt is the normal case and must be silent — no warn().

6.6 No secrets or credentials on the robots.txt request

Send only User-Agent (and, if convenient, Accept: text/plain). No cookies, no auth headers, no query string. The URL is always <origin>/robots.txt.

7. Config surface and defaults

7.1 The knob

bundleEntryToSourceEntry in src/core/config/config-sources.ts already round-trips every non-url website-descriptor key into the runtime entry's options bag, so no plumbing change is needed there — only the schema and the read in ensureWebsiteMirror.

Example opt-out:

{
  "bundles": {
    "docs": {
      "website": { "url": "https://docs.example.com", "respectRobots": false },
      "components": { "main": { "root": ".", "adapter": "website-snapshot", "writable": false } }
    }
  }
}

7.2 Behavior change for existing users — must be called out

This is the one place P1 intentionally changes behavior for existing configs.

• Before P1: akm crawled every reachable same-origin page up to maxPages / maxDepth, ignoring robots.txt.
• After P1: pages disallowed for akm (or *) are skipped, Crawl-delay is honored, and a crawl whose start URL is disallowed fails with a UsageError instead of producing a stash.
• Consequence: existing website sources may return fewer pages, or fail outright, after upgrade. Re-running akm update on such a source is what surfaces it.
• Opt-out: respectRobots: false on the website descriptor, which restores the pre-P1 behavior exactly (no /robots.txt request at all).

A CHANGELOG.md entry under ## [Unreleased] is required and must state the default, the user-visible consequence, and the opt-out.

7.3 Schema

Add to BundleWebsiteDescriptorSchema in src/core/config/schema/sources-bundles.ts:

respectRobots: z.boolean().optional(),

schemas/akm-config.json is generated — regenerate with bun scripts/gen-config-schema.ts. bun run lint runs gen-config-schema.ts --check and will fail if the committed JSON schema is stale.

Note the asymmetry, which is why §4.7 still coerces strings: the bundle descriptor is now boolean-validated, but the legacy sources[].options bag is z.record(z.unknown()) and accepts anything, so the runtime read must still validate and raise ConfigError.

8. Files to create / modify

Create

Modify

Do not modify: src/commands/sources/source-add.ts, src/commands/sources/installed-stashes.ts, src/core/config/config-sources.ts — the existing website-descriptor passthrough already carries the new key (§7.1). No CLI flag in P1.

9. Test guidance

Hard requirements from the repo harness:

• Unit tests under tests/; end-to-end akm add coverage under tests/integration/.
• Use tests/_helpers/sandbox.ts: withMockedFetch(run, (url) => Response), sandboxStashDir(), writeSandboxConfig().
• Never assign globalThis.fetch =, mutate process.env.HOME, or call process.chdir — scripts/lint-tests-isolation.ts fails the build.
• No real network in unit tests.
• Follow the fixture patterns in tests/integration/add-website-source.test.ts and tests/integration/website-ssrf.test.ts (loopback servers reach the crawler through the test-only private-host hatch).
• MPL-2.0 header on every new test file (scripts/lint-license-headers.ts enforces it).

Testing shape that makes this cheap:

• §4.1–§4.3 are pure-function table tests on parseRobotsTxt / isPathAllowedByRobots — no fetch mocking needed at all.
• §4.4 tests createRobotsPolicy with a hand-written RobotsTxtLoader stub; also assert the loader is invoked once for two URLs on the same origin.
• §4.5 tests loadRobotsTxt through withMockedFetch (status codes, oversized body via a large content-length, a throwing mock for the transport case).
• §4.6 and §4.7 need the loopback fixture server serving both /robots.txt and pages, driven through ensureWebsiteMirror with a sandboxed stash.
• Crawl-delay timing: assert the clamped value the policy reports and that a configured delay produces measurable spacing with a small value (e.g. Crawl-delay: 0.05), not a 2-second sleep in CI.

10. Acceptance criteria

• [x] src/sources/snapshot-fetchers/robots.ts exists with the MPL-2.0 header and exports exactly the API in §3.
• [x] robots.ts contains no module-level state carrying crawl semantics and does not import from website-ingest.ts (§6.1, reworded 2026-08-02 to distinguish crawl-semantic state from pure per-object memoization — see Review log). The import half holds (only warnVerbose is imported). The WeakMap memoizing compiled patterns is keyed on rule-object identity, cannot leak between origins/crawls/tests, and is expressly allowed by the reworded §6.1. ALLOW_ALL_RULES / DISALLOW_ALL_RULES are now Object.freezed, including their shared rules: [] array, closing the stray-mutation hazard flagged in the 2026-08-01 review log.
• [x] parseRobotsTxt satisfies every row of §4.1 (P-01…P-24). bun test tests/website-robots.test.ts — 100 pass, 0 fail.
• [x] parseRobotsTxt satisfies every Crawl-delay row of §4.2 (D-01…D-12), including the 10 s clamp.
• [x] isPathAllowedByRobots satisfies every row of §4.3 (M-01…M-32), including Allow longest-match-wins and the tie-goes-to-Allow rule.
• [x] createRobotsPolicy satisfies §4.4 and fetches /robots.txt at most once per origin (in-flight promises deduped). Proven by C-06.
• [x] createAllowAllRobotsPolicy() never invokes a loader and always reports allowed / 0 ms.
• [x] loadRobotsTxt satisfies §4.5: 2xx→body, 4xx→unavailable (silent on 404), 5xx→unreachable, oversized→unavailable, transport error→unavailable.
• [x] loadRobotsTxt performs no raw fetch(); every request goes through fetchWebsiteResponse → fetchWithRetry + assertWebsiteRequestUrl + assertResolvedHostAllowed, with readBodyWithByteCap(…, ROBOTS_BYTE_CAP, { bodyTimeoutMs }). Verified by reading loadRobotsTxt in website-ingest.ts.
• [x] A guard rejection on the initial robots.txt URL propagates and is not converted to allow-all (F-12), proven by a test.
• [x] allowPrivateHosts threads from crawlWebsite into the robots fetch path (C-13), proven by the loopback integration test. Every test in tests/integration/website-robots-crawl.test.ts passes allowPrivateHosts: true through ensureWebsiteMirror and relies on the loopback robots.txt fetch succeeding, which is only possible if the flag threads all the way to loadRobotsTxt's SSRF guards.
• [x] crawlWebsite throws UsageError when the start URL is disallowed, and the message contains the start URL and the string respectRobots (C-02).
• [x] A 5xx robots.txt on the start origin produces the same UsageError with an added server-error explanation (C-03).
• [x] Disallowed non-start URLs are skipped without erroring and without being fetched (C-04).
• [x] Crawl-delay is applied between page fetches, skipped before the first fetch, clamped to 10 s, and never sleeps past the crawl wall-clock deadline (C-07…C-11).
• [x] With respectRobots: false, zero /robots.txt requests are made and the crawl result is identical to pre-P1 behavior — proven by an integration test whose fixture server records every requested path (C-05).
• [x] coerceRespectRobots satisfies §4.7 (G-01…G-07), raising ConfigError on non-boolean values.
• [x] respectRobots: z.boolean().optional() is on BundleWebsiteDescriptorSchema and schemas/akm-config.json is regenerated (bun scripts/gen-config-schema.ts --check passes). Verified: src/core/config/schema/sources-bundles.ts:95 and bun scripts/gen-config-schema.ts --check reports "up to date".
• [x] CHANGELOG.md ## [Unreleased] documents the default-on behavior change, the user-visible consequence, and the respectRobots: false opt-out.
• [x] docs/guides/sources-registries.md documents respectRobots.
• [x] No console.* anywhere in the new/changed code; diagnostics use warn() / warnVerbose(). Verified: no console. matches in robots.ts or the changed regions of website-ingest.ts.
• [x] No new dependencies (P1 adds none — turndown, a DOM selector lib, and fast-xml-parser belong to P2/P3).
• [x] Every new src/ and tests/ file carries the MPL-2.0 header; bun scripts/lint-license-headers.ts passes.
• [x] New tests use withMockedFetch / sandboxStashDir; no globalThis.fetch =, no process.env.HOME mutation, no process.chdir; bun scripts/lint-tests-isolation.ts passes.
• [x] bunx biome check --write src/ tests/ produces no further changes; bunx tsc --noEmit is clean.
• [x] bun run check passes (lint + typecheck + test:unit + test:integration).

11. Open items flagged for review

These are spec-author decisions that go beyond, or tighten, the parent plan's phase text. A reviewer may veto any of them; they are called out so nobody has to diff the plan to find them.

1. Allow: is implemented (§5 I-01). The plan text lists only User-agent / Disallow / Crawl-delay. Omitting Allow would make akm stricter than site owners intend and would zero out sites using Disallow: / + Allow: /docs/.
2. 5xx fails closed (§4.5 F-06, §5 I-05). The plan says "missing/erroring robots.txt ⇒ allow-all … for unreachable files ≠ 5xx" and defers the 5xx call to this spec. Chosen: 5xx ⇒ full disallow, per RFC 9309 §2.3.1.4, mitigated by fetchWithRetry's retry and by respectRobots: false.
3. Dedicated 512 KiB robots byte cap rather than reusing the 5 MiB page cap the plan's "same … byte cap" phrasing implies (§2, §5 I-09). RFC 9309 §2.5 only requires 500 KiB be parsed.
4. fetchWebsiteMarkdownSnapshot is not gated (§1). Single-URL, user-typed fetches stay ungated; only crawls are.
5. Product tokens are akm / akm-cli, matched exactly and case-insensitively (§5 I-03), while the User-Agent header stays akm-cli website provider as the plan requires.

Review log

2026-08-01 — Gate close-out: advisory findings from independent test review + code review of 775b3b0

P1 passed its gate (bun run check green, all §10 tests passing). The following ADVISORY findings from the test-review and code-review passes did not block the gate. Recorded here with disposition, per the close-out process. Scope note (added 2026-08-02, see below): this entry's findings were raised against 775b3b0 (feat(p1): robots.txt compliance), the only commit that existed at the time of this review. Findings 1 and 6 as originally recorded here described the WeakMap memoization cache and the non-backtracking segment-scan matcher — neither of which existed in 775b3b0; both were introduced afterward by d5d1a6b. Those two findings have been moved to the 2026-08-02 entry below, re-attributed as self-review notes written by the author of d5d1a6b alongside that change, not as output of this independent review (which could not have seen code that did not yet exist). What remains below is only the portion of the original finding 1 that was reviewable at this date — the non-frozen ALLOW_ALL_RULES / DISALLOW_ALL_RULES shared-array risk, which did exist in 775b3b0.

1. [robots.ts:57-58 at 775b3b0] ALLOW_ALL_RULES / DISALLOW_ALL_RULES are exported non-frozen consts whose rules: [] array is a single shared instance across every crawl in the process. Disposition: accepted deviation at the time, no code change in 775b3b0. Nothing mutates the shared consts, but a stray push would poison every subsequent crawl. Noted as a candidate for an Object.freeze hardening pass in a later phase. (Resolved 2026-08-02 — see the fix entry below; both constants are now frozen and the §10 checkbox is ticked.)

2. [robots.ts:37, robots.ts:43] ROBOTS_FETCH_TIMEOUT_MS / ROBOTS_USER_AGENT_HEADER are exported and asserted by tests/website-robots.test.ts:65,68 but never referenced by production code. fetchWebsiteResponse hardcodes { timeout: 15_000 } (website-ingest.ts:479) and the literal "akm-cli website provider" (website-ingest.ts:475) instead of importing the constants, so the unit tests pin values that production code does not actually read from. Disposition: logged as a follow-up, no code change in this pass. Either wiring the constants into fetchWebsiteResponse or replacing the constant-value assertions with assertions against the actual outgoing Request (via withMockedFetch) closes this; deferred to a later phase rather than blocking P1.

3. [robots.ts:363] RobotsPolicy.crawlDelayMs re-derives crawlDelayMs ?? 0 with no re-clamp at the policy boundary. The MAX_CRAWL_DELAY_MS ceiling (§5 I-10) is enforced only inside parseRobotsTxt; a RobotsRuleSet built by any other route (a hand-built test fixture today, a future caller tomorrow) would flow an unclamped delay into sleep(). Disposition: logged as a follow-up, no code change in this pass. Not exploitable today — the deadline pre-check at website-ingest.ts:377 breaks the crawl loop rather than sleeping past the wall-clock cap, so no real crawl can stall on this. A defensive re-clamp (Math.min(Math.max(rules.crawlDelayMs ?? 0, 0), MAX_CRAWL_DELAY_MS)) in createRobotsPolicy.crawlDelayMs is a candidate one-line hardening for a later phase; not required to close P1.

4. [docs/guides/sources-registries.md:39] The guide and CHANGELOG.md both say a disallowed start URL makes akm bundle add/update "fail with an error," which is accurate for a cold cache but not for bundle update on a warm one. crawlWebsite's C-02/C-03 UsageError is thrown inside withFreshnessCache's refresh callback (website-ingest.ts:170), and src/sources/freshness.ts:64-68 swallows any refresh failure whenever a usable cached mirror exists within the 7-day stale window. An existing website source whose start URL becomes robots-disallowed therefore keeps serving its stale mirror for up to 7 days before the documented error surfaces (on the next cold crawl, or once the stale window expires). Disposition: reviewed and accepted as-is; documentation nuance only, no doc or code change in this pass. This is pre-existing freshness-ladder behavior (§4.6) applied to a new error class, not a regression introduced by P1 — the integration test suite passes because its fixtures always start cold. A follow-up sentence in docs/guides/sources-registries.md (and optionally the CHANGELOG.md entry) noting the stale-fallback window is a candidate for a later documentation pass.

5. [website-ingest.ts:217] scrapeWebsiteToStash still ends in a bare throw new Error("No content could be scraped from …") on a zero-page crawl, which plan §2.7 forbids on user-facing paths (exit 70 "internal / unclassified" instead of exit 2 "usage error"). Spec §4.6 C-12 leaves this line untouched by design, and C-02 removes the most likely new way to reach it (a fully-disallowed origin now fails earlier with a clear UsageError) — but robots filtering does put a new class of "crawl produced zero pages" outcomes in front of it. Disposition: no change in P1; logged as a candidate cleanup for a later phase. Converting website-ingest.ts:217 to UsageError so a zero-page crawl exits 2 with an actionable message is out of scope for this phase (pre-existing behavior, explicitly left alone by C-12).

2026-08-02 — Self-review notes written alongside d5d1a6b (regex → non-backtracking segment scan)

The two items below were written by the author of d5d1a6b at the time of that commit, describing code the same commit introduced. They are self-review notes, not findings from an independent reviewer seeing the change after the fact — both were previously misfiled under the 2026-08-01 independent-review entry above, dated as though a reviewer had examined and accepted them a day before the code existed. Re-filed here under their actual date/authorship per the 2026-08-02 gate re-review (see the closing entry below).

1. [robots.ts:246] Module-level WeakMap (compiledPatternCache) conflicts with the §6.1 / §10 "no module-level mutable state" wording. compiledPatternCache is a module-level WeakMap introduced by this commit to memoize compiled patterns. Note: semantically safe by construction. It memoizes a pure function of an individual rule object's own pattern, keyed by that object's identity, so distinct parseRobotsTxt calls (even for identical text) get distinct cache slots and nothing leaks between origins, crawls, or tests. §6.4 explicitly sanctions memoization. This is a wording conflict with §6.1/§10, not a defect. (Resolved 2026-08-02 — see the closing entry below: §6.1/§10 reworded to "no module-level state carrying crawl semantics", which this provably satisfies.)

2. [docs/plans/specs/p1-robots.md §6.4] The spec's first §6.4 bullet prescribed compiling patterns to RegExps with * → .* and collapsing consecutive *; the implementation deliberately does something stronger and different — a non-backtracking segment scan (compilePatternSegments/matchesCompiledPattern, robots.ts:229-290) that never constructs a RegExp. Verified the segment scan against every §4.3 M-row, including trailing-$ anchored cases and the last.length > target.length - cursor overlap guard: it is correct. The risk was purely documentary — collapsing consecutive * does not defend against distinct wildcards separated by short literals (e.g. /*a*a*a*a*a*a*a*a$), which is the actual catastrophic-backtracking shape, so a future maintainer "restoring" the spec's regex approach would reintroduce the ReDoS surface. Disposition: fixed in this pass. §6.4's first bullet now describes the non-backtracking segment matcher as the required implementation and explicitly rejects regex compilation, with the rationale above. A regression test for the distinct-wildcard shape (§6.4 regression: distinct '*a' wildcards …) is already in tests/website-robots.test.ts.

2026-08-02 — Gate re-review: robots-bypass fix, §10 module-level-state resolution, review-log correction

An independent re-review of the P1 gate raised three CONFIRMED findings, addressed in this entry's commit:

1. Redirect targets were never re-checked against robots.txt. normalizeCrawlUrl strips trailing slashes before the robots gate in crawlWebsite, so a rule shaped Disallow: /secret/ let /secret through; an ordinary trailing-slash-canonicalizing 301 back to /secret/ was then fetched and stored without ever being weighed against robots.txt. Fixed: fetchWebsitePage now re-checks the final (post-redirect) URL against the crawl's RobotsPolicy — passed in from crawlWebsite — and drops the page (warnVerbose, no error) when disallowed. fetchWebsiteMarkdownSnapshot's single-URL fetch path does not pass a policy, so it stays intentionally ungated per §1/open item 4.
2. §10's module-level-state checkbox was left open with no resolution. Fixed: §6.1 reworded to "no module-level state carrying crawl semantics", explicitly distinguishing forbidden crawl-semantic state (an origin -> RobotsRuleSet cache) from the WeakMap memoization (pure function of a rule object's own field, keyed by that object's identity — cannot leak across origins/crawls/tests) and from frozen empty-collection constants. ALLOW_ALL_RULES / DISALLOW_ALL_RULES (and their shared rules: [] array) are now Object.freezed, closing the stray-mutation hazard the 2026-08-01 entry flagged. The §10 box is ticked.
3. The review log misrepresented its own evidence. The §10 test-count claim ("107 pass, 0 fail") did not match bun test tests/website-robots.test.ts at HEAD (100 pass, 0 fail) — corrected above. Separately, the 2026-08-01 entry's findings 1 and 6 described the WeakMap and the segment-scan matcher, neither of which existed until d5d1a6b, the same commit that added the review-log entry claiming to have reviewed them a day earlier; those two items are re-filed above under their real date, labeled as self-review notes rather than independent-review findings, and the 2026-08-01 entry now scopes itself to 775b3b0, the only commit it could actually have seen.