Right Property, Right Time, Right Evidence: Evaluating Real Estate AI

Ask a general benchmark whether an answer is correct and it compares text against a reference string. Ask a property platform the same question and the honest answer is harder: correct about which property, as of when, and on evidence the system was allowed to use?

The property intelligence series describes what breaks in the data: identity, semantics, temporal truth, provenance, evidence, and rights.⁸ This article is about the other half of that problem — how you prove a system handled them, before it ships and on every release afterwards.

A reference answer is not a test

Consider one sentence from a property assistant: “The home has four bedrooms and sold in March for $612,000.”

It is fluent. It carries two citations. It may also be wrong in three independent ways that no text-similarity metric can see. The bedroom count may have come from the assessor record for the adjoining parcel. The March sale may have been recorded after the as-of date the question specified. And the price may have arrived from a source the platform is not licensed to redistribute.¹⁰

None of those failures live in the text. They live in the binding between the text and a property, a moment in time, and an evidence set. A grader that reads only the answer is structurally incapable of seeing them.

This is why evaluation belongs inside the system lifecycle rather than in front of it. NIST’s Generative AI Profile frames trustworthiness as something incorporated into the design, development, use, and evaluation of AI systems — not established once before launch.¹ The product-side framing is the same: evals are what turn fuzzy goals into something explicit enough to measure, under a specify, measure, improve loop.²

The expected answer is a specification

The single most useful change is to stop storing an expected string and start storing an expected contract. For a property question it has six parts.

The last one matters more than it looks. Where the evidence is insufficient, refusal is the correct output — and a benchmark built on string similarity scores a correct refusal as total failure. Any harness that cannot express “the right answer here is to decline, and say why” will steadily train the system toward confident nonsense.

A hallucination taxonomy that fits property data

“Hallucination” is too coarse a word here. Five failure classes behave differently, are caught differently, and cost differently.

The first and last deserve emphasis, because both produce an answer that is factually true. A wrong-property fact survives any check that asks only “is this true of some property?” An unlicensed grounding survives any check that asks only “is this supported?” Catching either requires the evaluator to know which property was asked about and which sources were permitted — neither of which is in the answer text.

Provenance is what makes the last one testable at all. RESO carries originating and source system metadata precisely so a consumer can tell which system a record came from,⁵ and a harness should assert on those fields, not only on the value.

You cannot test time by filtering today’s data

Here is the most common way a point-in-time test quietly fails: the harness queries the current table with a date filter and calls the result history. It is not. Today’s table contains corrections applied retroactively, records that arrived late, and status changes backdated to when they took effect rather than when they were entered.⁹

A real fixture reconstructs both axes — what was true of the property, and what the platform knew — at the moment the question was asked.

Freezing matters as much as reconstructing. If the evidence set can move between runs, a regression suite cannot separate a model change from a data change — and a good deal of what gets reported as model drift is data drift wearing a disguise.

Grade in layers, and separate retrieval from generation

A single judge model scoring the final answer tells you the system failed without telling you where. Two separations recover that.

First, score retrieval on its own: did it return evidence for the right property, inside the right time window, from the permitted set? If retrieval handed the model the neighbouring parcel’s documents, the generation step was lost before it began, and no amount of prompt work is the fix.

Second, route each check to the cheapest grader that can decide it.

Identifiers, dates, units, and permission checks are exactly decidable. Sending them to a judge model swaps a deterministic answer for a probabilistic one at higher cost. Reserve model graders for what genuinely needs judgment — whether evidence is sufficient, whether a field was read correctly given that the same Data Dictionary concept is legitimately expressed differently across systems³⁴ — and reserve people for the severe cases.

Weight by consequence, not by count

An aggregate accuracy figure conceals the only thing that matters: what the failures were.

A system at 97% is unacceptable if the remaining three points sit in the bottom two rows, and entirely deployable if they sit in the top one. Rank by consequence and the release decision largely writes itself.

For systems that act rather than answer, the gate has to cover the action surface too: tool selection, argument correctness, authorization, approval behavior, and the safety of a sequence of individually valid calls. OWASP’s agentic top ten is a reasonable starting inventory of what to test against.⁷

Fairness is a separate test surface

Fairness is not a subset of accuracy. A housing system can be correct on every fact and still produce a discriminatory outcome through what it surfaces, to whom, and how it phrases it.

HUD issued guidance in May 2024 on applying the Fair Housing Act to AI in two specific places: tenant screening, including third-party screening that uses AI, and targeted housing advertising.⁶ Each is an evaluation surface with its own tests — whether recommendation exposure shifts materially across protected characteristics, whether conversational filters reintroduce proxies for them, whether screening explanations reflect the actual decision logic, and whether generated descriptions carry steering language.

These do not fit the fixture used for factual correctness. They compare populations of outputs rather than single answers, which means a separate scenario library and a separate cadence.

The moat is knowing when you are right

Foundation models improve, and competitors get the same ones. What does not transfer is a scenario library built from your own hard cases, fixtures that can reconstruct any past moment, and a grading stack calibrated to your own severity model. That is the durable asset, and it compounds.

Every production failure is a test case you did not have. The teams that pull ahead are the ones that convert them at that rate.