# Car Parts Price Comparator — Walled Stores via Remote Browser (Technical Design)

**Status:** Draft v1 · 2026-07-12
**Companion documents:** `TECHNICAL-SPEC.md` (the core meta-search), `MONETIZATION.md` (why this stays a private tool), `FUNCTIONAL-SPEC.md` (product).
**Scope:** How the tool reaches stores that gate their search behind a Cloudflare **interactive challenge** (e.g. Trodo) *without* evasion — by running a real browser the user solves through, on demand.

> **Supersession note (2026-07-16):** Product measurement and stats were later removed. Any statement below that measurement remained unchanged is historical.

---

## 1. Why this exists

Five stores (Partsale, XPARTS, EOLTAS, Carparts, Handler) are reachable over plain HTTP and fetch automatically. A second tier (Trodo/eParts, and others) serves its **search endpoint** only to a browser that has cleared Cloudflare's interactive "Just a moment…" challenge. Our recon established two hard facts:

- The challenge beats **real headless Chrome**, not just plain HTTP — so no fetch trick clears it.
- The *only* legitimate way past it is a **real human solving a real challenge in a real browser** (no fingerprint spoofing, no auto-solving — that stays permanently out; see `MONETIZATION.md` §7 and the no-evasion policy).

This subsystem lets a human do exactly that, once, remotely, and then reuses the session — **on demand, only for the walled store the user asks for.**

## 2. The idea in one paragraph

Run a **real, headful Chrome** inside a container against a **virtual display (Xvfb)** on a machine the user controls (a home box, residential IP). Normally nobody watches it. When the user taps "Check Trodo" and the store throws a challenge, the app streams that browser's screen to the user's phone via **noVNC**; the user taps through the challenge; the same browser — now holding a valid `cf_clearance` — fetches Trodo's open JSON API and returns offers. Because the browser that *solves* and the browser that *fetches* are the same process on the same host, IP / fingerprint / cookie all match by construction. See the full decision trail in the project memory and `MONETIZATION.md`.

### The two meanings of "headless" — do not confuse them
- **Headless *deployment*** ✅ — no monitor on the host; runs as a background Docker service; the user only ever sees it through the stream. This is what we want.
- **`chrome --headless`** ❌ — the exact fingerprint Cloudflare beats. We run **headful Chrome under Xvfb**, never `--headless`.

## 3. Architecture

```
┌─ Docker container (home box, residential IP) ─────────────────────┐
│  Xvfb            virtual display :99                               │
│  Chrome          HEADFUL, --display=:99, --remote-debugging-port   │
│                  9222, persistent profile volume (keeps cf_clearance│
│                  warm across restarts)                             │
│  x11vnc          exports :99 as a VNC server                       │
│  websockify/noVNC serves the VNC as a web page on :8080            │
│  Node/fastify    the existing app (src/server.ts)                  │
│     ├─ auto stores      → PoliteHttp (unchanged)                   │
│     └─ onDemand stores  → BrowserHub → drives the resident Chrome  │
└───────────────────────────────────────────────────────────────────┘
   Exposed ONLY on 127.0.0.1 → reach it from the phone via Tailscale/LAN.
```

**Fetch strategy — drive the resident Chrome, don't extract the cookie.** The walled adapter does not pull `cf_clearance` out and replay it with undici: Cloudflare would re-challenge on the TLS/JA3 fingerprint mismatch (undici ≠ Chrome). Instead the adapter asks the resident Chrome to make the request from a page already on the store's origin (`page.evaluate(fetch(...))`, same-origin, cookies included). Every signal — cookie, headers, fingerprint, IP — is genuinely consistent.

## 4. Request flow (on-demand, streaming only on a challenge)

1. User searches. The 5 `auto` stores fetch over HTTP as today. `onDemand` stores render **idle** ("Check Trodo").
2. User taps "Check Trodo" → `GET /api/search/trodo?q=…`.
3. `BrowserHub` ensures its Trodo page is on `trodo.lv` and warm, then `fetch()`es the search API same-origin.
   - **Warm** → JSON back → `parse()` → offers → card fills, comparison re-ranks. *No stream.*
   - **Cold (challenge)** → `BrowserHub` throws `ChallengeRequiredError(solveUrl)` → the outcome carries `needsSolve:true, solveUrl` → the UI opens the noVNC view → user taps through → the app retries the fetch → offers. *Stream, only for those seconds.*
4. Result cached 15 min by `(store, part)` like every store, so a repeat search is instant with no browser.

Streaming is therefore rare: never for the 5 open stores, never for a warm walled session, only when a walled session is cold — and with a warm persistent browser on a residential IP, Cloudflare often waves it through non-interactively.

## 5. How it grafts onto the existing code

The design is deliberately minimal — the core stays untouched:

| Concern | Change |
|---|---|
| Store metadata | `StoreAdapter` gains `fetchMode?: "auto" \| "onDemand"` (default `"auto"`). Exposed in `/api/stores`. |
| Walled fetch | New `src/core/browserHub.ts` (`BrowserHub`, puppeteer-core `connect` to `:9222`). The adapter's `fetchHtml` calls it; **`parse()` stays a pure function over a JSON fixture**, tested exactly like `handler`/`eoltas`. |
| Challenge signal | New `src/core/browserErrors.ts` → `ChallengeRequiredError(storeId, solveUrl)`. `SearchService.search` catches it and returns `needsSolve/solveUrl` in the `SearchOutcome` instead of counting a store failure. |
| Frontend | `StoreInfo` gains optional `fetchMode`; `SearchResponse` gains `needsSolve?/solveUrl?`. `auto` stores auto-fire (as now); `onDemand` render idle with a fetch button that opens the solve view on `needsSolve`. *(UX slice — see §7.)* |
| Runtime optionality | If the browser service isn't running, `BrowserHub.connect()` fails, the walled fetch errors, and the card shows "unavailable". **The 5-store tool is unaffected.** |

The five `auto` adapters, `PoliteHttp`, cache, measurement, redirect, and stats are all unchanged.

## 6. Deployment (home box, residential IP)

- `docker/Dockerfile` — Node 24 LTS (matches the dev box; npm 11 is required by `web/package-lock.json`'s shape) + headful `google-chrome-stable` + Xvfb + x11vnc + noVNC. `.dockerignore` keeps `node_modules`/`data`/`public` out of the context.
- `docker/entrypoint.sh` — boots Xvfb → Chrome (headful, `--remote-debugging-port=9222`, persistent `--user-data-dir`) → x11vnc → websockify/noVNC → the Node app (`tsx` directly — the Angular build is baked into the image). On (re)start it clears stale locks — X11 `/tmp/.X99-lock` and Chrome `Singleton*` in the profile volume, either of which otherwise crash-loops a restarted container — and creates `/app/data` (better-sqlite3 won't mkdir).
- `docker/docker-compose.yml` — one service, `chrome-profile` + `comparator-data` volumes (warm browser session; SQLite cache/stats), `shm_size: 1gb`. Ports on **all host interfaces**: app `:3000`, noVNC `:8090` host-side (host `8080` was already taken) — reachable over home LAN (`10.82.82.1`) and the tailnet (`100.82.172.121`; the host already runs tailscaled).
- **Security:** the exposure boundary is the home router's NAT + the tailnet — **never port-forward 3000/8090 to the public internet**: noVNC is a live, *passwordless* browser session (`x11vnc -nopw`). Anyone on the LAN/tailnet can drive it; that's the accepted trade-off for a single-user home network.
- **`NOVNC_URL`** is the host-side address the solve view opens; it's set to the LAN IP, so from the tailnet open `:8090` on the tailscale IP manually until slice 4 rewrites the URL's host to `window.location.hostname` in the frontend.
- **IP:** run it at home. Datacenter IPs get challenged hard; a residential IP + warm browser rarely challenges.

### Container engine: Docker or Podman (Podman is the Fedora-native fit)

The `Dockerfile` and compose file are engine-agnostic — Podman reads both as-is (`podman compose -f docker/docker-compose.yml up --build`). On the intended Fedora home box, **rootless Podman** is the default; the image is already rootless-friendly:
- Chrome runs with `--no-sandbox` (required rootless) — in `entrypoint.sh`.
- `shm_size: 1gb` **and** `--disable-dev-shm-usage` guard against Chrome's `/dev/shm` crashes.
- A **named volume** (`chrome-profile`) is used, not a bind mount, so no SELinux `:Z` relabel is needed. (If you switch to a bind mount on an SELinux-enforcing box, add `:Z`.)

**Boot persistence:** `restart: unless-stopped` covers uptime, but rootless containers don't auto-start on reboot on their own. Use a **Quadlet** unit (`~/.config/containers/systemd/comparator.container`) or `podman generate systemd`, plus `loginctl enable-linger $USER` so it starts without a login session.

## 7. Build order

1. **Infra + seam** ✅ — `fetchMode` on `StoreAdapter` + `/api/stores`; `browserErrors.ts`; `browserHub.ts`; `needsSolve` in `SearchOutcome`; Docker files; a `trodo` adapter wired to `BrowserHub`. Backend typechecks; nothing live changed.
2. **Stand up the container** ✅ (2026-07-12) — image builds on **rootless Podman** (the box's `podman` is a remote client to the host socket; see project memory), container runs on host network. Verified: app `:3000`, noVNC `:8090`, Chrome CDP `:9222` (Chrome 150, **headful**), all reachable over LAN + tailnet. Bugs fixed en route: `.dockerignore`; Node 24 base (npm-11 lockfile); entrypoint clears stale X11 + Chrome `Singleton*` locks and creates `/app/data`; `tsx` instead of re-building Angular on boot.
3. **Capture Trodo fixture + `parse()`** ✅ (2026-07-12) — the real search endpoint is **`GET /rest/V1/catalogsearch/result/0?q=<CANON>&searchby=name&…&currency=EUR&country=LV`** (the `/oem/<n>` path hard-403s; a top-level nav to the API path is WAF-*blocked*, not challengeable — only the SPA's XHR flow is allowed). It carries a **managed** Cloudflare challenge that surfaces an inline "Verify you are human" widget on the results page; a human solve issues `cf_clearance`, then the same-origin fetch returns the JSON envelope `[{ products:[…] }]`. Fixture `fixtures/trodo/0986452041.json`; `parse()` written + fixture-tested (4 tests); `enabled:true`. Live `/api/search/trodo` returns real offers through the resident browser.
4. **Frontend on-demand UX** ✅ (2026-07-12) — walled stores start **idle** with a "Pārbaudīt <store>" (Check) button in the per-store chip row (the 5 auto stores fetch as before); tapping it fires `GET /api/search/<store>`. Warm → offers merge into the existing client-side grouping/ranking (verified: Trodo's 24 offers merged, chip → "24 piedāvājumi"). Cold → the chip flips to **"Atrisināt pārbaudi"**; tapping opens a **noVNC overlay** (iframe of the resident browser) with "Gatavs — turpināt" / "Aizvērt"; the user solves, and "Gatavs" retries the fetch. Implemented in `search.service.ts` (`fetchStore`, idle/needs_solve statuses), `app.ts` (solve overlay state + host rewrite), `app.html` (chip buttons + overlay), `strings.ts`. Both refinements landed:
   - **Cold-solve navigation** ✅ — `BrowserHub.fetchJson` gained an `opts.solvePage`; on a challenged API response it navigates the resident page to that URL (Trodo passes `…/catalogsearch/result?q=<CANON>&searchby=name`) so the "Verify you are human" widget is on screen in the stream before `ChallengeRequiredError` is thrown.
   - **noVNC host rewrite** ✅ — the frontend rewrites `solveUrl`'s host to `window.location.hostname` (keeps port/path), so the same link works whether the app was opened via the LAN IP or the tailnet IP. *(Caveat: the iframe is `http://…:8090`; if the app is ever served over HTTPS via a reverse proxy, that becomes mixed content — proxy noVNC under the same origin then.)*

Verified end-to-end in the running container by driving the real Angular UI: idle button present → warm check merges offers → forced-cold (cleared `cf_clearance`) check flips to "Atrisināt pārbaudi" → overlay opens with the host-rewritten noVNC iframe.

5. **More walled stores** ✅ (2026-07-12) — probing the remaining Tier-2 candidates through the resident browser produced a key finding: **the warm headful browser on the residential IP passes Cloudflare's managed challenge non-interactively** (no evasion — it is Cloudflare's own script running to completion in a real browser; the human/noVNC path stays as the cold-session fallback). Live results: **rd24** 200 + 108 results, **euautodalas** 200 + 148 results, **autodoc** first nav 403-challenged → self-cleared in seconds → `cf_clearance` issued → 200 on retry. All three are server-rendered HTML (not JSON like Trodo), so `BrowserHub` gained **`fetchPageHtml(storeId, url)`**: navigate the store's resident page to the results URL; on a challenge, wait `challengeClearMs` (default 12 s, env `BROWSER_CHALLENGE_CLEAR_MS`) for it to clear itself, re-navigate once, and only then throw `ChallengeRequiredError` — at which point the page is already sitting on the widget, so the noVNC stream shows exactly what to solve (no separate `solvePage` needed). Adapters: `rd24` re-enabled (its 2026-07-10 parse/fixture still match live markup), new `euautodalas` + `autodoc` (both read the clean `data-price` attribute, dodging euautodalas' decorative `0,00 €` elements). Fixtures captured through the resident browser: `fixtures/euautodalas/W71252.html`, `fixtures/autodoc/0986452041.html`. All four walled stores live-verified end-to-end through `/api/search/<store>`. Store count: **9** (5 auto + 4 onDemand). Not unlocked (different walls, unchanged): rb24 (prices login-gated), topautodalas (no part-number search), sparesauto (no priced markup), dts (Turnstile token per request).

Only after the 5-store core is validated for the click-through gate does the walled phase earn priority — this subsystem is additive and strictly opt-in.