Software Architecture
Behind the Architecture of a Cloud-Based Website Spell Checker
A website spell checker is a crawler, content extractor, language-analysis pipeline and reporting product. Its hardest problems are scope, identity, false positives, safe fetching and explaining change over time.
“Check this website” hides four different systems
Website SpellChecker scans sites, analyses content for spelling and grammatical issues, shows where problems occur and tracks changes between scans. The product description sounds like one AI request. A production design is closer to four cooperating systems: discovery finds pages, fetching retrieves them safely, analysis turns page content into findings, and reporting gives those findings stable identity and context.
Keeping those responsibilities separate matters. Crawling is constrained by network behaviour and politeness. Analysis is constrained by language quality, latency and cost. Reporting is constrained by user expectations and historical continuity. Coupling all three in one request makes a slow page, model timeout or report write indistinguishable to users and operators.
The first architectural insight is that scan completion is not binary. Discovery can finish while some fetches fail; analysis can succeed for most pages; a useful report can be published with explicit coverage. Pretending partial results are total results creates false confidence.
A URL is not a reliable page identity
The same content may appear through tracking parameters, trailing-slash variants, mixed host casing, redirects, print routes and session identifiers. Conversely, two responses from one URL can vary by locale, authentication, device or JavaScript state. Naively treating every string as a page wastes crawl budget and duplicates findings.
Normalisation should be conservative: lowercase the host, remove default ports, resolve relative links, discard fragments for HTTP retrieval and apply a deliberate policy to known tracking parameters. Do not sort or remove arbitrary query values; they may change the resource. Redirect targets and declared canonical URLs are evidence, not unquestionable commands.
Give each crawl attempt an identity separate from the normalised page identity. That allows retries and history without confusing “we tried this URL twice” with “the site contains two pages”. Content fingerprints can identify exact duplicate bodies, but shared navigation must be handled separately from whole-page duplication.
Crawl scope is a product promise
Following every link can escape from the submitted site into social networks, calendars, logout routes, faceted search or effectively infinite parameter spaces. Define scope using allowed hosts, path rules, redirects, maximum depth, page limits and content types. Make exclusions visible in the final coverage report.
Respect site controls and rate limits, but remember that crawl directives are not an authorisation system. A customer entering a URL does not prove ownership, and a publicly reachable page is not necessarily safe for a server to fetch. Verification may be required before deeper or repeated scanning.
The frontier-the queue of discovered but unprocessed URLs-needs deduplication, priority and bounded growth. Breadth-first discovery gives broad early coverage; depth-first can disappear into one section. A useful scheduler balances coverage with politeness and prevents one large site monopolising workers.
A cloud crawler is also an SSRF surface
A service that fetches user-supplied URLs can be tricked into requesting private addresses, cloud metadata endpoints or internal services. Validating the initial string is insufficient because DNS can resolve unexpectedly and redirects can cross the security boundary.
Allow only intended schemes, resolve and inspect destination addresses, block loopback, link-local and private ranges, and revalidate every redirect and connection. Route crawler traffic through controlled egress with network-level restrictions. Limit response size, decompression, redirect count and duration so small inputs cannot consume unbounded resources.
Fetched HTML is hostile data. Do not execute it inside the application's trusted browser context, and do not allow page-supplied filenames or markup into reports unsafely. A headless browser expands coverage for client-rendered sites but also expands attack surface, compute cost and nondeterminism; use isolation, resource limits and a clear escalation policy from simple HTTP fetching.
Rendering everything is not automatically more accurate
Many pages expose meaningful text in server-rendered HTML. Fetching and parsing that representation is fast and repeatable. JavaScript rendering becomes necessary when primary content is constructed client-side, but it may trigger analytics, personalised experiments, consent overlays and time-dependent requests.
A staged strategy works better: attempt the inexpensive parser first, estimate whether meaningful content exists, and render only when signals indicate a client application. Record which method produced the evidence because two rendering modes may expose different text.
“What a user sees” is not a single state. Viewports, cookies, region and authentication alter it. The product should make its chosen scanning context explicit instead of implying it observed every possible visitor experience.
Extraction quality limits analysis quality
Sending raw HTML to a language model is wasteful and noisy. Scripts, styles, navigation labels, cookie banners, hidden accessibility text and structured metadata have different meaning. Extract visible textual regions while preserving page location and enough surrounding context to explain each finding.
Repeated page furniture is the classic source of inflated issue counts. If a misspelling appears in the header on 500 pages, users usually need one component-level problem with affected-page evidence, not 500 independent tasks. Cross-page fingerprints can identify repeated blocks, but overly aggressive deduplication can hide genuinely separate uses.
Keep offsets anchored to the extracted representation and store a durable locator-such as a DOM path plus nearby text-because raw character positions become stale when a page changes. A finding without a reproducible location is academically correct and operationally useless.
Language analysis is a ranking problem, not a truth machine
Dictionaries miss grammar and context; statistical or AI analysis can invent issues. Brand names, domain terminology, personal names, code samples and multilingual pages turn apparently obvious spelling rules into probabilistic judgements.
Detection and presentation should be separate. An analyser can propose an issue type, explanation, suggestion and confidence; policy then decides what to show, suppress or group. Site-specific dictionaries and accepted findings reduce repeated noise, but they need scope and auditability. Globally accepting a word because one customer uses it would contaminate every tenant.
Language detection also needs uncertainty. A navigation label may be English while the article is French; a short phrase may be impossible to classify. Analyse coherent blocks where practical and prefer “unknown” over confidently applying the wrong rules. Quality should be measured on representative labelled content, including false-positive cost-not by counting how many warnings the system produces.
Queues absorb variation; they do not remove it
Fetch time, rendered-page cost and analysis latency vary widely, so an asynchronous pipeline is natural. Each stage can scale independently and apply backpressure when the next dependency is saturated. A scan request should return an identity and observable progress rather than hold an HTTP connection open.
Most managed queues deliver at least once. Workers therefore need idempotent writes keyed by scan, page, content version and analysis policy. Visibility timeout expiry can cause concurrent processing; exactly-once marketing language does not eliminate the need for deduplication.
Retries should depend on failure type. A timeout may recover; an unsupported media type will not. Exponential backoff without a limit merely turns permanent failure into delayed expense. Dead-lettered work needs diagnostic context and an operator action, otherwise the queue is a graveyard with a metric.
Differential scanning requires stable semantics
Website SpellChecker highlights new errors found since an earlier scan and helps users track fixes. Comparing two lists of message text is not enough. Analysis policies evolve, page URLs move, shared content changes and a suggested correction may be worded differently while describing the same issue.
Create a finding fingerprint from stable properties such as normalised page or component identity, rule family, affected text and contextual location. Classify findings as new, persistent, resolved or moved, and retain the analysis version. When a model or ruleset changes substantially, distinguish re-evaluation from a deterioration in the website.
“Resolved” also needs caution. A page that failed to fetch did not prove its findings were fixed. Coverage is part of every comparison: pages analysed, pages skipped, failure reasons and content changes. Absence of evidence is not evidence of corrected copy.
Operate around user journeys and cost units
Infrastructure metrics should be connected to scan outcomes: frontier size, successful fetch rate, pages excluded, analysis latency, findings produced, cost per page and time until the first useful results. Queue depth without tenant and stage context can conceal starvation or one pathological site.
Bound concurrency per site and per tenant. Global autoscaling cannot protect a target website from an accidental traffic spike, nor can it stop one customer's huge site delaying smaller scans. Fair scheduling and cancellation matter as much as worker count.
Store raw content only as long as needed and avoid logging page bodies. Websites can contain personal or confidential text even when publicly accessible. Reports, screenshots and extracted content require tenant-scoped access and retention policies just like conventional customer data.
The architecture should make uncertainty visible
A trustworthy website quality tool does not claim omniscience. It tells users what it scanned, which rendering context it used, where it could not reach, why a finding was raised and whether a change comes from the site or the analyser.
The architectural boundaries support that honesty. Discovery defines coverage; fetching records evidence; extraction preserves location; analysis expresses confidence; comparison supplies history; reporting turns it into work. Collapsing those stages may create a quick demonstration, but it makes production errors impossible to explain.
That is the deeper lesson behind the system: AI is one replaceable part of a disciplined data pipeline. Crawl correctness, security, identity and operational feedback determine whether clever language analysis becomes a dependable product.