Every property pipeline eventually meets a value that is perfectly well-formed and completely wrong. What you do at that moment decides the architecture — and it is not a job for a data-cleaning model.
- Business problem
- Every property pipeline meets values that are perfectly well-formed and completely wrong — an unmapped field, an unrecognized status, an ambiguous match, a plausible number that is a typo. Rules cannot settle those, and a model cannot sit in the path of every value.
- Why it matters
- A national platform carries billions of values per refresh cycle. Routing each through a model is not merely expensive — at that volume the throughput and latency it demands put it out of reach of a normal batch window — while leaving the judgment to rules alone lets silent corruption into curated history, where it surfaces months later with no trace of how it arrived.
- Architecture response
- Keep the spine deterministic: PySpark unpacks the vendor feed, maps it to a known schema, matches it against what you already hold, and decides update or replace while preserving temporal history. The agent runs beside that path and adjudicates only the exceptions — it flags and proposes, and never edits a row.
- What Databright Cloud Solutions does
- We design the mapping and resolution registries, the exception model, the Glue and PySpark spine, the agent insertion points, and the evidence and evaluation layer that lets autonomy be raised on evidence rather than optimism.
The arithmetic settles the architecture
A national platform might carry twelve million active and sold listings, each with a few hundred fields. That is billions of values per refresh cycle. Sending each one through a model is not simply expensive: at that volume the throughput and latency required put it beyond what a nightly batch window can absorb, at any price a data business would pay.
Run the deterministic work first and the volume collapses. A mapped field is a mapped field; a join key matches or it does not. What survives is the residue: the unmapped field, the unrecognized enum, the ambiguous parcel match, the value that passes every rule and still looks wrong.
The pipeline, end to end
Nothing about the spine is exotic, and that is the point. Overnight, a vendor delivers a delta — a RESO Web API pull, a legacy feed, a file drop — carrying a few tens of thousands of changed listings. Each one is a nested document: a media collection of photographs, a rooms collection, open-house records, and several hundred flat fields around them. It lands in S3, is never mutated, and Glue catalogs it.2 PySpark flattens the nested collections into typed columns and does the rest.
Then the harder half, which is not transformation at all but reconciliation against the millions of records you already hold. Is this the same property? The same listing? A correction, or a genuinely new event? Getting that wrong does not corrupt one row — it corrupts the history.
The agent layer runs beside this path, never inside it. Spark emits what it could not decide; the agent reasons over it asynchronously and writes a decision back.
What rules catch, and what they miss
Deterministic validation is very good at malformed data. A bathroom count that is not an integer, a status outside the enum, a postal code that fails a lookup — these fail loudly, and Spark can quarantine them without any help.2
The expensive errors are the well-formed ones. A list price of $45,000 on a $450,000 home is a valid number. Living area typed into the lot-size field is a valid integer. A listing re-keyed as new rather than edited is a valid new row. All three pass type, range, and enum checks. All three are wrong.
These are entry mistakes, made by a listing agent working quickly against a form with several hundred fields. (An unfortunate collision of vocabulary, so to be explicit: throughout this article agent means the AI component and listing agent means the person.) You cannot enumerate the rules in advance, which is precisely why pattern recognition earns its place here.
What matters is what happens next. The output is a flag, a reason, and the evidence behind it, routed to review — not a quiet rewrite of the price to what the model thinks was meant. Silent correction of a plausible value is worse than the original error, because it destroys the evidence that anything happened.
One listing, end to end
Take the price typo, because it is the most instructive failure in property data — it is trivial to describe, invisible to rules, and it does its real damage a day later.
Tuesday, 02:00. The delta arrives. One listing carries ListPrice = 45,000; yesterday it was 450,000. Every deterministic check passes: it is a positive number, correctly typed, inside any plausible global range. Nothing about the value is malformed, so nothing rejects it.
The agent sees the same row differently, because it can read the context around it. PublicRemarks still say the home is offered at four hundred fifty thousand. OriginalListPrice is 450,000. The value sits an order of magnitude below every comparable in the same MLSAreaMajor. That is a high-confidence, high-consequence entry error — and it is flagged as one, with those three pieces of evidence attached. The price is not altered. The row publishes exactly as delivered, held out of derived market metrics until someone looks.
Wednesday, 02:00. The feed sends 450,000. This is the moment that actually separates the two architectures.6
The naive pipeline is not missing a cleaning step. It is missing the distinction between a correction to a fact and a new fact — and no amount of data cleaning recovers it, because by then the evidence is gone. Every downstream consumer of price-change velocity, reduction counts, and market-heat signals inherits a phantom.
Where the agent plugs in
Four insertion points cover most of the value, and each one sits at a specific stage of the flow above. Each is also a judgment this series has already examined in depth — the agent is automating decisions that were always being made, just slowly and by hand.
Roughly a thousand source fields per new market need mapping to canonical concepts. The agent proposes a target, a relationship type — equivalent, narrower, broader, derived, approximate, non-equivalent, or unresolved — and its evidence; a human ratifies.5 That vocabulary is what stops a source Baths = 2.5 being cast into an integer bathroom count where RESO would record three.
Values that pass every rule and still look wrong: a price off by an order of magnitude, finished area typed into the lot-size field, PublicRemarks describing four bedrooms where BedroomsTotal says three. The agent ranks by confidence and consequence, and flags for review. It never corrects.
Spark decides which parcels and listings could match. The agent adjudicates the ambiguous shortlist: the same home listed in two MLSs with overlapping coverage, new construction with no parcel number assigned yet, a condo whose parcel covers the whole building.1
Is this delivery a correction to what was already true, or a new event in the property’s history? Treat a correction as an event and the timeline gains a price change that never happened; treat a genuine reduction as a correction and it silently disappears. The worked example above is this decision.6
What stays in Spark
The temptation, once an agent works, is to widen its remit. These belong in deterministic code permanently:
Set operations at scale — testable, explainable, and cheaper by orders of magnitude.
Once registered, applying a mapping is a lookup. Re-asking a model per row reintroduces nondeterminism into a settled question.
Agents propose; the pipeline writes. This is what keeps runs replayable and failures diagnosable.
One practical warning, because it is the most common way these builds fail: do not call a model from a per-row Spark UDF. Filter to the exception set first, batch at the partition level, and cache by content hash.3 That is the difference between a job that finishes and one that does not.
Why the registry is the important part
A pipeline with a probabilistic component still has to be deterministic on replay: re-running yesterday’s job must produce yesterday’s result. It does, because model output is never consumed live. It is captured in the registry with an effective date and full evidence — inputs, model and prompt version, proposal, confidence, who ratified it, and which rows it affected.
That record is also what lets autonomy rise honestly. Every proposal carries a human disposition, giving a continuous measure of acceptance and calibration. Move a class of decisions from propose-only to auto-apply when that evidence justifies it — not when the backlog gets uncomfortable.4
What this actually buys
Not a smarter pipeline. A shorter one, in the dimension that costs most: the interval between a new source arriving and that source being trustworthy in production. Deterministic pipelines are excellent at applying decisions and incapable of making them, so every property platform accumulates a queue of small interpretive judgments only humans can clear. That queue, not compute, is the real constraint on coverage.
Use Spark for everything a rule can decide. Use an agent for the residue that needs judgment. Have it write proposals into a registry rather than values into a table. The data is never cleaned behind your back — it is validated, flagged, and changed only by a decision you can point to.
Sources and industry references
- Databright Cloud Solutions — The Property Identity Problem: why APNs, parcel numbers, addresses, and MLS IDs are not enough
- AWS — AWS Glue Data Catalog and crawlers: schema registration and discovery
- AWS — Amazon Bedrock batch inference: asynchronous S3-based processing for large datasets
- Databright Cloud Solutions — The Agentic AI Transition: bounded agents, autonomy levels, and accountable execution
- Databright Cloud Solutions — The Semantic Consistency Problem: why matching field names do not mean matching definitions
- Databright Cloud Solutions — The Temporal Truth Problem: what was true, when it became true, and when we knew it
This article provides architecture perspectives, not implementation guidance for a specific environment. Cloud service capabilities, model behavior, quotas, and pricing change; validate current service limits, supported regions, model features, and license terms for your data sources before committing to a production design.