What 6.5 million lease records reveal about the UK's legacy data problem – Burke and Boyd

New analysis of 6,476,524 individual records from HM Land Registry’s Registered Leases dataset by the Open Data Institute (ODI) and the Leasehold Advisory Service (LEASE) has identified that as many as one in 12 leasehold properties may be affected by problematic short leases.

However, the analysis also uncovered a problem that goes well beyond leasehold.

Of the individual lease records analysed, almost one in 12 (8%) residential leases have between 80 and 90 years remaining. Together, that represented almost a quarter of the analysed sample, suggesting a cohort of over 1.35 million short leases that are or will soon become a material concern.

But the project also revealed how difficult supposedly digital public information can be to use. Lease information is not always recorded in a consistent, structured form. Lease terms are often recorded in free text rather than a standard format. The same term can be expressed in many different ways – from “99 years from 1980” to “100 years starting four days after 19 August 2008” – making it difficult for computers to interpret consistently.

Some records lacked the Unique Property Reference Number (UPRN) used to link information reliably to an individual property. Without a consistent UPRN, more complex address-matching methods may be needed to connect the address entered by a homeowner to the correct title record.

Sponsored

Your First Home: confidence to build

Sponsored by Halifax Intermediaries

While HM Land Registry stopped holding paper files after 2008 and has digitised many records, older files still exist. With these older leases disproportionately likely to have shorter remaining terms, the number of short leases almost certainly represents a conservative estimate of the true scale of the problem.

The project identified that the registered position can also lag behind the legal position. Lease extensions, variations or expirations may have taken place without yet being reflected in HM Land Registry’s data. A lease can remain on the register after it has expired if no instruction has been received to update it.

To make sense of the records, the ODI developed software that could interpret the different ways lease terms had been recorded and calculate how many years remained. The original ambition was to clean the underlying dataset and provide that information to consumers, but this presented another problem: the data is continually changing as new datasets are released. This illustrates why data quality cannot be treated as a one-off clean-up exercise. Standards must be maintained as information changes; otherwise, today’s improved dataset becomes tomorrow’s legacy data problem.

Addressing the challenges

The new lease-checking tool takes a different approach. Rather than continually trying to clean millions of records, the relevant record can be processed when someone searches for their property. It is an effective solution to an immediate problem. But it also raises a much bigger question: why are we having to use increasingly sophisticated technology to interpret basic information that the government already holds about people’s homes?

The problem is not unique to HM Land Registry.

Much of the data held by large organisations and public bodies has accumulated over decades and often lacks a clear lineage or schema documentation. All of this requires human input to investigate and resolve problems, but for artificial intelligence (AI) systems to use this information reliably, its meaning and structure need to be recorded explicitly. If the information underneath them is fragmented, inconsistent or inaccessible, you are simply automating parts of a broken process.

Data readiness for AI involves more than converting records into a machine-readable format. It requires organisations to think about data quality, the metadata that explains it, the infrastructure through which it is accessed, and the governance that determines how it can be used. Organisations need to understand where information came from, how it has changed and, increasingly, which versions of datasets have been used to train or fine-tune AI models.

This is why work on open standards for data are so important. They provide a common way to describe and structure information, making it easier to connect and use across different organisations and systems rather than locking it into incompatible formats. The ODI has long argued that such standards improve interoperability and reduce the cost and complexity of combining data from different sources. Work on open standards such as Croissant apply this thinking to AI-ready data, providing a standardised, machine-readable structure for dataset metadata, so that both people and machines can understand and reuse data more reliably.

The government’s proposed reforms to leasehold offer an opportunity to build better data foundations for leaseholders. As the new system develops, the information needed to support it should be structured and consistent from the outset, rather than requiring another clean-up years later. A more standardised system could use common definitions, structured information and consistent identifiers from the outset. Retrofitting those things years later is much harder.

For now, leaseholders can use LEASE’s free tool to check their remaining lease term and understand what it could mean for them. But the more important lesson for the government is what had to happen behind the scenes to make that possible. Having information in digital form is not the same as having usable data. As we redesign important public systems and introduce AI into public services, the data underpinning them needs to be treated as part of the infrastructure from the outset, not as an administrative problem to fix later.