Confidence Is Not a Score: Property Valuation After the AVM Rule

A property valuation can be accurate and indefensible at the same time. Since October 2025 that distinction has had regulators attached to it.

The rule implements a Dodd-Frank mandate and took effect on 1 October 2025. It reaches mortgage originators and secondary market issuers using an AVM to determine the collateral value of a mortgage secured by a consumer’s principal dwelling.¹ Notably, it names five outcomes and then declines to prescribe structure: the agencies stated plainly that the rule “does not set specific requirements for how” those policies and control systems should be built, so that institutions can scale them to their size and risk.²

That flexibility is sensible regulation. It is also a transfer of responsibility.

A portfolio metric is not a per-property confidence

Ask a valuation provider about accuracy and you will usually receive a median absolute error and a hit rate — the share of estimates within ten percent of the eventual sale price. Both are real, useful numbers. Both describe a population.

Neither says anything about the house in front of you. A model with a four percent median error can be badly wrong on a specific property, and the standard concerns confidence in the estimate being produced, not the average estimate across a book.

The engineering form of that requirement is calibration. If a model emits a ninety percent interval, the eventual sale price should land inside that interval about ninety percent of the time. That is measurable, and it is frequently not measured.

Segmentation is where this gets uncomfortable. An interval that is well calibrated nationally is routinely overconfident on unusual property types, at the top and bottom of the price range, and anywhere transaction density is thin. A single global coverage number hides exactly the cases where a valuation is most likely to be challenged.

One thing worth checking early: whether the confidence score you are relying on is a confidence at all. Some are data-completeness proxies wearing a probability’s clothing — higher when more fields were populated, without any claim about error. That is a useful signal and it is not calibration.

The model that matters is comparable selection

Attention naturally goes to the estimator. Most of the error arrives before it.

Which sales were eligible, whether they are genuinely the same property type, whether an arm’s-length filter excluded the transfers it should have, whether “living area” means the same measurement across every comparable — these decisions bound the achievable accuracy before any coefficient is applied. Comparable selection rests on resolving whether two records describe the same parcel or unit,⁶ and on field definitions that are standardized in name more reliably than in meaning.⁷

It follows that when two valuations of the same property disagree materially, the productive question is rarely about the algorithms. It is which comparables each one used.⁸

A backtest against today’s table is not a backtest

The fourth standard calls for random sample testing and reviews. The obvious implementation is to sample past valuations and compare them against subsequent sales.

The obvious implementation is also where the result quietly becomes meaningless. Query today’s warehouse and the sample inherits everything that arrived afterwards: corrections applied to records after the valuation date, sales recorded late, characteristics revised by a permit that closed months later. The model is being graded with information it did not have, and it will look better than it was.³

Freezing matters as much as reconstructing. If the inputs behind a test can drift between runs, a quarter-over-quarter comparison cannot separate a model change from a data change — and the review record shows a trend that may be neither.

Manipulation and conflicts are lineage questions

Two of the five standards — protecting against data manipulation and avoiding conflicts of interest — read like policy commitments. Inside a data platform they are both answerable only with lineage.

The operative question is whether a party with an interest in the outcome can influence an input. Owner-supplied characteristics, agent-entered remarks, a record corrected shortly before a valuation was ordered: none of these is inherently improper, and all of them are worth being able to see. Answering that requires field-level provenance — which system supplied each value, when it was observed, and what transformed it on the way in.⁵

There is a second-order version of this that is easy to miss. If you consume several vendor valuations and select or blend among them, the selection rule is itself a model, and it belongs under the same governance as any other. Taking the highest of three estimates is not a valuation method. It is a policy, and an adverse one to have to explain later.

Aggregate accuracy hides local error

The fifth standard concerns compliance with applicable nondiscrimination laws. The measurement question underneath it is narrower and entirely tractable: error is not evenly distributed, and an aggregate figure is built to conceal that.

A national median error is perfectly compatible with systematically larger error in particular places — and model error tends to concentrate where transaction density is low, where property characteristics are recorded less completely, and where housing stock is less uniform. Those conditions are not randomly distributed across geographies.

So error has to be reported by segment, geography included, and reviewed as a distribution rather than a headline. This is the same discipline any model making property claims requires: compare populations of outputs, not individual answers.⁴ Whether a given result raises a legal issue is a question for counsel; whether you can see the result at all is a question for your data platform, and it is the one you can settle in advance.

What a valuation record has to carry

Every standard above converges on the same requirement: a valuation must be reproducible long after the moment it was produced. That is a schema decision more than a modeling one.

The two fields most often missing are the snapshot pointer and the override. Without a frozen input snapshot the estimate cannot be recomputed. Without an explicit override field, human adjustments become invisible — and an adjustment nobody recorded is indistinguishable from a model output when someone asks a year later who decided the number.

Defensible beats accurate

An accurate valuation you cannot explain is a liability. A slightly less accurate one that carries its interval, its inputs, its comparables, and its provenance is an asset, because it can be reviewed, challenged, corrected, and improved.

The rule’s flexibility is the part worth sitting with. Because it does not define what a high level of confidence is, that definition becomes yours to write down, defend, and test against — which is a better position than it sounds, provided the writing down happens before the testing does.