# Car Parts Price Comparator — Technical Specification (POC, Stage A)

**Status:** Draft v1 · 2026-07-10
**Companion document:** `FUNCTIONAL-SPEC.md` (what the product does). This document covers how.
**Design priorities (per owner):** best scraping performance, and — above all — **fast adjustment when a store's site changes**. Every architectural choice below is measured against those two.

---

## 1. Store reconnaissance results (probed live, 2026-07-10)

All seven candidate stores were probed with a plain HTTP client (no browser) using a common
part number (MANN oil filter `W 712/52`) . This determines the adapter strategy per store.

| Store | Search endpoint | Works with plain HTTP? | Notes |
|---|---|---|---|
| **Rezervesdalas24.lv** | `GET /spares-search.html?keyword=<q>` | ❌ **No longer** (was ✅ at probe time) | **Update, M1 build day:** now behind a Cloudflare managed challenge (`cf-mitigated: challenge`) on every path — plain HTTP gets 403 even on the homepage. Also: robots.txt disallows the search path. Moved to **Tier 2**; adapter + fixture kept, `enabled: false`. |
| **Partsale.lv** | `GET /spare-parts/meklet?q=<q>` | ✅ Yes — server-rendered HTML with prices | Easy. Has autocomplete endpoint too. JSON-LD present. |
| **EUAutodalas.lv** | `GET /meklesana-rezervesdalas?keyword=<q>` | ✅ Yes — server-rendered | Normalizes the number itself (`W712/52` → `W71252`, 148 results). Price extraction needs care (decorative `0,00 €` elements in markup). |
| **Autodoc.lv** | `GET /search?keyword=<q>` | ⚠️ Yes, with session warm-up | Bare request → **403**. Fetch homepage first, keep cookies, send Referer → 200 with full server-rendered results and prices. Most likely store to tighten defenses; affiliate feed is the long-term fallback. |
| **XPARTS.lv** | `POST /carparts/search/` with form data `CarModAjax=Y&ShortResult=Y&WithRedirects=Y&ArtSearch=<q>` → responds `REDIRECT:/carparts/search/<q>/` → `GET` that URL | ✅ Yes — two-step | PrestaShop + custom "carmodsearch" module. **Confirmed in M1:** prices are inline as `data-ga-price` on the add-to-cart button (no secondary call); part/brand in `data-artnum`/`data-brand` on `.info_bl`. Results include analogue offers across brands — their own cross-referencing, relevant to v2. Needs a bigger timeout budget (two steps + 2s politeness gap + ~1.2MB page → 15s). |
| **Trodo.lv** | Custom Vue front-end on Magento 2; `form action="javascript:"`; `/catalogsearch/result/?q=` renders but finds nothing (part numbers not in that index); GraphQL disabled | ❌ Not with plain HTTP (yet) | Needs XHR reverse-engineering with browser DevTools, or a headless-browser adapter. **Tier 2 — not in the first build.** |
| **eParts.lv** | `GET /lv/search?q=` → **302 redirect to `https://www.trodo.lv/search`** | — | **eParts is Trodo** (same platform, same static assets, redirects to trodo.lv). Fold into the Trodo adapter; verify later whether prices differ at all. |

**Conclusion (revised during M1, settled at M4):** of the 7 original candidates, only **partsale**
and **xparts** work with plain HTTP. euautodalas and autodoc were re-probed at M2 and again at M4
(headless) — both, plus rd24 and trodo/eParts, now sit behind Cloudflare's interactive challenge
that even a real headless browser can't clear without evasion (see §11 M4 recon), so all four are
shelved pending an affiliate feed. **EOLTAS.lv**, **Carparts.lv** and **Handler.lv** (none in the original 7) were added at M4 — all
reachable over plain HTTP/2 through JSON APIs. Live store set: **partsale, xparts, eoltas, carparts, handler**.
**Requests must use HTTP/2:** Cloudflare challenges HTTP/1.1 clients claiming a Chrome UA
(verified: `curl --http1.1` → 403, same request over HTTP/2 → 200). `PoliteHttp` therefore
uses undici's `Agent({ allowH2: true })`.

## 2. Stack

**TypeScript on Node.js**, single service. Rationale against the two priorities:

- **Performance:** the whole request is I/O-bound (6 parallel HTTP calls); Node's async model
  fans out naturally. Parsing with `cheerio` (htmlparser2 under the hood) is fast enough that
  parse time is noise next to network time.
- **Fast adjustment:** one typed `StoreAdapter` interface makes every store a single small file;
  when a store changes, the fix is one file + one refreshed fixture. TypeScript catches a
  half-updated adapter at compile time.
- One language end-to-end (adapters, server, Angular frontend), so no context switching.
- `playwright` can be added later for the Trodo adapter without changing anything else.

Core dependencies (deliberately few): `fastify` (HTTP server), `cheerio` (parsing),
`zod` (validating adapters and persisted documents), `better-sqlite3` (cache + user persistence),
`vitest` (tests). Native `fetch` with `undici` for outbound requests.

## 3. Architecture

```
Browser ── GET /                      → static search page
        ── GET /api/search/:store?q=  → one store lookup (browser fires all stores in parallel)
        ── GET /api/stores            → list of enabled stores (drives the parallel fetches)
        ── GET/PUT /api/search-history → shared single-user recent searches
        ── GET/PUT /api/purchase-plan → shared single-user purchase plan
        ── POST /api/offers/refresh   → exact saved-offer refresh
        ── GET /out?store=&url=       → validated 302 to store product page

Node service
 ├─ core: normalize query → adapter.run() → validate offers → cache → respond
 ├─ adapters/            one file per store, all implementing StoreAdapter
 ├─ cache: SQLite,       key (store, normalizedNumber), TTL 15 min
 └─ user persistence: SQLite singleton search-history and purchase-plan documents
```

**Key decision — per-store endpoint instead of one aggregated endpoint.** The browser requests
each store separately and renders each result the moment it lands. This gives, for free:

- incremental UI (fast stores appear in ~1 s; nobody waits for the slowest),
- per-store timeout isolation ("could not be checked right now" per §4.4 of the functional spec),
- no SSE/WebSocket machinery.

Client-side JS then does the brand grouping/sorting (§4.2 of the functional spec) over the
accumulated offers, re-rendering as stores arrive.

**No product database.** Store responses are persisted only in the 15-minute operational cache.
The search-history document contains queries and timestamps only; the purchase plan contains offers
the user deliberately selected. There are no search-result, response, click, telemetry, or analytics tables.

## 4. The adapter contract (the "fast adjustment" core)

```ts
interface StoreAdapter {
  id: string;                    // "rd24"
  displayName: string;           // "Rezervesdalas24.lv"
  storeHomepage: string;
  enabled: boolean;              // kill switch, hot-reloadable via config
  timeoutMs: number;             // per-store budget, default 8000
  search(q: NormalizedQuery, http: PoliteHttp): Promise<Offer[]>;
}

interface Offer {
  store: string;
  brand: string | null;          // "MANN-FILTER"
  partNumber: string;            // as listed by the store
  title: string;                 // "Eļļas filtrs"
  priceEur: number;              // incl. VAT
  availability: string | null;   // store's own wording
  deliveryNote: string | null;   // "3–7 dienas", "izņemšana Rīgā šodien"
  url: string;                   // deep link to the product page
  imageUrl: string | null;       // product photo from the listing page (no extra request)
  attributes: OfferAttribute[];  // spec rows from the listing: {label: "Augstums", value: "92 mm"};
                                 // flag-style rows ("Ar atgriezējvārstu") use label "" + value
}
```

Rules that keep breakage cheap to fix:

1. **One store = one file** in `src/adapters/`. All store-specific knowledge (URL pattern,
   selectors, price parsing, quirks like Autodoc's cookie warm-up) lives there and nowhere else.
2. **Selectors live in a constant block at the top of the file**, not inline in logic — when a
   store redesigns, the fix is usually editing 5 lines of selectors.
3. **Zod-validate every Offer** an adapter emits. A parse gone quietly wrong (price `NaN`,
   empty URL) is rejected and counted, never shown.
4. **`PoliteHttp` wrapper is injected**, adapters never call fetch directly — politeness,
   cookies, timeouts, and retries are implemented once (§6).

### Breakage detection (know it broke before you notice manually)

- **Runtime anomaly counters:** per store, track (a) HTTP failures, (b) `200` responses where
  parsing yielded zero offers, (c) zod rejections. A store whose zero-parse rate spikes while
  others return results is almost certainly a markup change → surface on a `/health` page.
- **Canary check:** a script (`npm run canary`) searches one known-good part number
  (`W 712/52`) against every live store and asserts ≥1 valid offer each. Run manually or via
  cron; Stage A needs nothing fancier.
- **Fixture tests:** every adapter has recorded HTML responses in `fixtures/<store>/` and a
  parse test against them. When a store changes: re-record fixture (`npm run record <store>`),
  watch the test fail, fix selectors, test green — a rehearsed, minutes-long loop.

## 5. Query normalization

- **Canonical form** (for cache keys and response-side matching): uppercase, strip everything
  non-alphanumeric — `09.C881.11` → `09C88111`, `W 712/52` → `W71252`.
- **Sent to stores:** the canonical form by default. Probes show stores normalize internally
  (EUAutodalas and Autodoc both accepted `W712/52` and searched `W71252`), so this is safe;
  an adapter may override if a store turns out to need the raw punctuation.
- **Input guard:** reject queries shorter than 4 alphanumeric characters (per §4.4, avoid noise).

## 6. Politeness and legal posture (implementation of functional spec §7)

- **Rate limiting per store:** minimum 2 s between requests to the same store (token bucket in
  `PoliteHttp`); the 15-minute cache means repeated/refreshed searches don't touch stores at all.
- **One search = one request per store** (two for XPARTS's two-step flow). No crawling, no
  pagination walks, no background prefetching — strictly on-demand, per query.
- **Timeouts** (8 s default) and **circuit breaker**: 5 consecutive failures → store auto-marked
  "could not be checked", retried after 10 min. Never hammer a struggling site.
- **User-Agent:** a normal browser UA for Stage A (a personal tool making single polite requests;
  a distinctive bot UA invites blocks without any transparency benefit at this stage). Revisit at
  Stage B, where the published store list + delist policy become the transparency mechanism.
- **robots.txt:** fetch and note each store's rules for the search paths during implementation;
  respecting them where feasible is recorded per adapter as a comment with a date.

## 7. Frontend

**Angular 22 app in `web/`** (rewritten 2026-07-10; the original no-framework page proved too
limited for filtering/variant UX — decision: owner works in Angular professionally, and the UI
is the first thing users judge). Component library: **Spartan NG 1.1** (zinc theme, "vega"
style; helm components are vendored into `web/src/app/ui/` shadcn-style, config in
`web/components.json`) + **Tailwind v4** + **Phosphor icons** via `@ng-icons/phosphor-icons`.
Application component ownership and refactor constraints are recorded in
[FRONTEND-ARCHITECTURE.md](FRONTEND-ARCHITECTURE.md).

- `ng build` (or root `npm run build:web`) outputs straight into root `public/`, which fastify
  serves — one process in "production". `npm run dev:web` = `ng serve` on :4200 proxying
  `/api` + `/health` to :3000 (`web/proxy.conf.json`) for UI work with HMR.
- `SearchService` (signals, zoneless) fires `GET /api/search/<store>?q=…` for every enabled
  store in parallel; the page re-renders per arrival. Per-store status chips: searching… /
  n offers / no results / could not be checked. Below the `md` breakpoint these collapse into
  one aggregate progress disclosure (`completed / total`, offer count, attention count); failed
  and human-check stores are duplicated as compact action rows outside the closed disclosure so
  recovery never depends on discovering or opening the full list. At `md` and above, the existing
  wrapped chip list remains visible. The mobile disclosure uses an indeterminate spinner while
  `SearchService.searching()` is true and opens the complete status list in a focus-trapped modal
  bottom sheet with enter/exit transforms and a `prefers-reduced-motion` override.
- **Variant grouping** (functional spec §4.2, sharpened): offers group into one card per
  `brand + canonical part number` — so two same-brand variants (e.g. two-piece alu-hub brake
  disc vs. plain steel) stay separate cards, while the same part across stores merges into one
  price-comparison table, cheapest row marked. The exact searched number ranks first with a
  "Meklētais numurs" badge; analogues follow under an "Analogi citiem zīmoliem" heading,
  ordered by cheapest offer (price-neutral across stores).
- Filters, all client-side over the accumulated offers: brand chips with counts (multi-select,
  from all offers so a selected chip never vanishes) and an "in stock only" switch
  (`availability` present and not "Nav …").
- **Image + specs per card** (added 2026-07-10): each card shows a 64px product thumbnail with
  a CSS-only enlarged preview on hover (`group-hover`; Tailwind gates it behind
  `@media (hover: hover)`, so touch devices just see the thumb), plus the store's spec rows
  ("Filtra izpildījums: Uzskrūvējams filtrs" …) — first 6 inline, the rest behind a per-card
  "+ vēl N parametri" toggle. Both come free from the search listing HTML of all three
  Tier-1 stores (no per-product requests). A group's image = cheapest offer that has one;
  its specs = the offer with the most rows. Partsale thumbs are upsized 200→800px via their
  image resizer (only 100/200/400/800 actually exist — other sizes return an EMPTY 200).
- Hosted-web outbound links go through `/out` only for store-domain validation before the 302.
  Native shells validate and open the original store URL directly. Neither path records clicks,
  attaches search identifiers, or adds UTM measurement parameters.
- `SearchHistoryStore` records a valid accepted query before the normal fan-out, deduplicates by
  the shared canonical form, retains 20 newest entries, caches locally, and serializes writes through
  the injected HTTP, desktop IPC, or Android Preferences backend.
- Latvian strings in one `web/src/app/core/strings.ts` map; `models.ts` mirrors the backend
  `Offer` contract by hand. Dark mode follows the OS via the `.dark` class.

## 8. Search history and privacy migration

The shared contract defines a versioned search-history document containing at most 20 entries:
latest raw formatting, canonical part number, and last-searched timestamp. Hosted web persists one
singleton document in SQLite through `GET/PUT /api/search-history`; desktop stores an atomic JSON
document in Electron `userData`; Android uses Capacitor Preferences. The Angular store also keeps a
browser fallback and a durable pending-change marker, so offline additions and removals retry after
restart instead of being overwritten by stale backend state. Persistence failure never blocks a search.

The former product-measurement system was removed on 2026-07-16. Server startup drops its legacy
`searches`, `search_results`, and `clickouts` tables, and the former `/api/searches` and `/stats`
routes no longer exist. Searches call store endpoints directly without a `sid`. `/out` retains its
store-domain open-redirect guard but stores nothing. Desktop and Android open undecorated URLs.

## 9. Per-store implementation notes

### Purchase plan and exact-offer refresh (2026-07-13)

The Angular client persists a versioned purchase plan under
`car-parts-comparator.purchase-plan`. Pure functions in `web/src/app/core/purchase-plan.domain.ts`
own candidate identity, chosen-only totals, alternative projections, store grouping, and refresh
change tracking. `PurchasePlanStore` is the signal/local-storage boundary; malformed entries are
recovered independently and future schema versions are preserved until an explicit reset.

`POST /api/offers/refresh` is deliberately separate from broad search and its response cache. It
validates the saved URL against the selected store, fetches that exact product page through the
store's existing polite HTTP or resident-browser path, and returns a fresh snapshot only after
canonical brand and part-number identity match. Different stores refresh in parallel, while exact
products within one store are checked sequentially. Challenges retain the last snapshot and resume
only the affected exact-product request through the existing noVNC flow.

Search responses carry their original `observedAt` time, including cache hits, so saving a cached
offer never makes it appear newly observed. The shared `OFFER_FRESHNESS_MS` constant controls both
the search-cache TTL and the plan's stale threshold.


- **rd24, partsale, euautodalas:** straight GET + cheerio. EUAutodalas price selector must
  target the offer card, not page furniture (decorative `0,00 €` elements observed).
- **autodoc:** `PoliteHttp` session support — on 403, GET homepage, retain cookies, retry once
  with `Referer`. Cache the session cookies per process. Expect this adapter to be the
  highest-maintenance one; if it becomes an arms race, drop to Tier 2 and consider their
  affiliate program's data access instead of escalating (per legal posture — no evasion contest).
- **xparts:** POST → parse `REDIRECT:` body → GET pretty URL. Verify with fixtures whether
  prices are inline or need one more call; the analogue-brands block on their results page is
  noted for the v2 cross-referencing investigation, not used in the POC.
- **trodo/eparts (Tier 2, after M3):** open trodo.lv with browser DevTools, capture the search
  XHR the Vue front-end makes, replicate it in a plain-HTTP adapter; only if that fails, add a
  `playwright` adapter variant. One adapter serves both brands; check if eParts prices differ.

## 10. Testing strategy

- **Unit:** normalization, brand grouping/sorting, price parsing (`"8,29 €"` → `8.29`).
- **Adapter fixture tests:** recorded real responses per store; parse must yield expected offers.
  Fixtures are checked in, so selector fixes are verifiable offline.
- **Live canary:** `npm run canary` (§4) — the only test that touches real stores; never in CI.
- No end-to-end browser tests in the POC; the UI is simple enough to eyeball.

## 11. Build order

| Milestone | Contents | Proves |
|---|---|---|
| **M1** ✅ 2026-07-10 | Skeleton: server, adapter interface, `PoliteHttp` (HTTP/2), cache, normalization + **partsale and xparts adapters** (rd24 written but Tier 2'd — Cloudflare), bare results table, fixtures + record/canary scripts | End-to-end works on two live stores |
| **M2** ✅ 2026-07-10 | Grouping UI shipped with the Angular rewrite (variant cards, chips, sorting, status chips). euautodalas + autodoc re-probed same evening over HTTP/2: still `cf-mitigated: challenge` on every path incl. robots.txt → both moved to Tier 2 (M4) per the no-evasion posture | The real comparison experience |
| **M3** retired 2026-07-16 | The former measurement tables, search IDs, `/stats`, and UTM decoration were removed. `/out` remains only as a validated hosted-web redirect. | Privacy requirement in functional spec §9 |
| **M4 (more stores)** ✅ 2026-07-10 | **Original premise (headless-browser fetch unlocks the Tier 2 stores) failed and was abandoned** — see below. Instead added **EOLTAS.lv**, a Baltic chain reachable over plain HTTP/2 via its JSON REST API; live + fixture-tested (16 offers). `PoliteHttp` gained `postJson`; `StoreAdapter` gained optional `fixtureExt`. | Restores store count — politely, no evasion |

**M4 recon verdict (2026-07-10):** the Tier 2 stores (rd24, trodo/eParts, euautodalas, autodoc)
are **not** unlockable by a plain headless browser. In a real headless Chrome 150 with no
stealth patching, all four return Cloudflare's interactive `Just a moment…` JS challenge and
stay on it (403) even after the challenge script runs to completion (polled 45s). Clearing it
would require fingerprint/stealth evasion, which the project rules out (§12, functional spec §5).
So the browser-fetch path was dropped. A fresh store hunt found **EOLTAS.lv** reachable without
any evasion: its Vue front-end calls a JSON API — `GET /lv-lv/rest/search/<code>/2` (exact +
analog products) and `POST /lv-lv/rest/product-price` (prices in integer cents, keyed by each
product's opaque `hash`). Both endpoints are gated behind `X-Requested-With: XMLHttpRequest`
(404 without it) but need no browser. The adapter mirrors xparts' two-step shape; `fetchHtml`
merges the price response into the search JSON so `parse` stays pure over a `.json` fixture.
Remaining big-catalog stores (autodoc especially) stay out until the sanctioned route — an
**affiliate data feed / API** — is available; that is the only path consistent with the no-evasion posture.

> **Superseded 2026-07-12 (M5, walled stores):** the *headless* verdict above stands, but the
> remote-browser subsystem (WALLED-STORES.md) reopened all four Tier 2 stores without evasion:
> a resident **headful** Chrome on a residential IP, streamed to the user over noVNC when a
> challenge needs a human. In practice the warm browser passes Cloudflare's managed challenge
> **non-interactively** (it is Cloudflare's own script running to completion in a real browser)
> — live-verified for rd24 (200, 108 results), euautodalas (200, 148 results) and autodoc
> (403 challenge → self-cleared in seconds → cf_clearance issued → 200). All four walled
> stores (trodo, rd24, euautodalas, autodoc) are now live as `onDemand` adapters; the noVNC
> human solve remains the cold-session fallback. Affiliate feeds stay the long-term route if
> any of them starts demanding interaction every time.

**M4 "more stores" round (2026-07-11).** Added **Carparts.lv** — a Next.js storefront over a
TecDoc-backed JSON API (`GET api.carparts.lv/article/searcharticlenr?s=<code>`), single call,
price+stock inline, no browser/auth/challenge. Then investigated five more candidates found by
market search; the recurring lesson is that **reachable ≠ usable** — a store must clear all of:
no Cloudflare/CAPTCHA wall, real part-number search, and public prices. Verdicts:

| Candidate | Reachable? | Verdict |
|---|---|---|
| **carparts.lv** | ✅ plain HTTP JSON API | **ADDED** — clean single-call TecDoc API with price+stock |
| topautodalas.lv | ✅ server-rendered | ✗ no part-number search — name/vehicle only; `0451103318` & `W712/52` return "nekas netika atrasts" |
| rb24.lv | ✅ GET results URL `/lv/meklesanas-rezultats/1/9/<code>` | ✗ prices hidden from anonymous users (login/B2B); only `0,00 €` placeholders |
| dts.lv (OEM) | ✅ homepage via curl | ✗ part search gated by **Cloudflare Turnstile CAPTCHA** (`/part/get_part_details/` needs a token); bypass = evasion |
| sparesauto.lv | ✅ `/lv/catalog/find?criteria=<q>` | ✗ no server-rendered priced products (template shell) |
| **handler.lv** | ✅ open JSON API | **ADDED** — homepage challenges headless, but the search API is wide open (see below) |

**Handler.lv (added 2026-07-11).** The Nuxt homepage throws a Cloudflare challenge at headless
browsers, but its search API needs no browser at all: `GET api.handlerparts.com/api/Parts/Search`
`?articleNumber=<code>&siteCode=hlv&sessionId=<any-uuid>&language=lv&languageCode=lv&countryCode=lv`
returns JSON with **no auth, no CAPTCHA, no Origin check**. `sessionId` is a client-minted UUID
(the API issues nothing) — we hardcode a stable one so the cache works. This is the same class of
public-SPA-API access as eoltas, not evasion. The response is a **marketplace**: `data.offers` holds
many supplier offers for the searched part and its cross-brand analogues (180 for W71252); the adapter
collapses them to the cheapest live offer per (brand, manufacturer code) → 18 comparable offers,
with stock counts, delivery days, and per-article product URLs. Prices are VAT-inclusive and public.

Net: **5 live stores** (partsale, xparts, eoltas, carparts, handler). Across ~12 Latvian/Baltic
candidates examined over M1–M4, these five are the ones simultaneously reachable, part-number-searchable,
and publicly priced without evasion. Growing further realistically means the **affiliate-feed route**
(autodoc et al.).

Runs locally (`npm run dev`, SQLite file on disk). No deployment, domain, or hosting in Stage A;
a `Dockerfile` is written only when something must run on a box other than the laptop.

## 12. Risks

| Risk | Mitigation |
|---|---|
| Store markup changes silently | Anomaly counters + canary + fixture loop (§4) — designed-for, minutes to fix |
| Autodoc hardens bot defenses | Circuit breaker → Tier 2 → affiliate program route; never an evasion arms race |
| Store objects to inclusion | Adapter `enabled=false` is a one-line delist (functional spec §5 policy) |
| Slash/format-sensitive part numbers on some store | Canonical form default + per-adapter override hook (§5) |
| Trodo XHR can't be replicated | Confirmed at M4: Cloudflare challenge blocks even headless; no evasion → shelved for affiliate feed. Store count kept up by adding politely-reachable stores like EOLTAS instead |
| Cloudflare interactive challenge on a store | Do **not** stealth-patch the browser. Either the store exposes a JSON/XHR API reachable over plain HTTP/2 (eoltas) or it waits for an affiliate feed |