Artificial Intelligence and real estate market forecasting

Forecasting real estate markets is widely considered an art that blends science and expert judgement. Forecasters rely on historical data and scientific knowledge to understand the economic and social factors that govern market dynamics. However, they also weight those factors to tweak their models, using their expert judgement to tell a meaningful story of the future. With the exponential growth of Artificial Intelligence (AI) capabilities, this blend is poised for transformation. 

While AI will be definitive in shaping forecasting models and their inputs, its integration alongside human expertise remains crucial to produce sensible and actionable insights on the future of real estate markets. A somewhat good analogy to how AI will impact forecasting in the real estate industry could be the impact of the digital spreadsheet on accounting. While we can trace the use of spreadsheets for double entry bookkeeping to merchants in 14th century Venice, it wasn’t until the late 1970s that Dan Bricklin came up with a computer programme that would allow data to be entered into a digital spreadsheet. For the first time, software existed that could do most of the job for us. Despite this clear improvement in efficiency, the human input necessary for accounting remains practically unchanged; rather than erasing and rewriting entries on a paper sheet, we crunch keys on a keyboard to fill digital cells and write functions that perform calculations that would have taken much longer to do ourselves. Essentially, it remains the expert judgement of the accountant that will determine the data that is entered into the spreadsheet and the calculations necessary to best determine the taxes due.

Similarly, the process of forecasting will not change, in essence. It will remain a combination of data and expert judgement. But going forwards, expert forecasters could have AI-enhanced data and more accurate models at their disposal to aid them in telling a better-informed and more insightful story about what the future holds for real estate markets.

Artificial intelligence’s transformative potential for real estate forecasting 

AI's ability to process and analyse vast quantities of data with remarkable speed and accuracy is unmatched. But AI is not just able to handle much more data faster; it can also extract new and meaningful insights. There are several examples of AI exploring large datasets and identifying meaningful patterns and correlations that had previously eluded human analysts [1].

Real estate markets are characterised by highly aggregated, low frequency data, posing significant challenges for investment analysis and forecasting. An AI model, however, can be trained to find relationships between real estate data and more granular, high frequency non-real estate data. By broadening their datasets in this way, forecasters could tell better, more accurate narratives of future market dynamics. A good application of this would be in achieving more accurate and faster forecasts to improve investment decision processes. Real estate markets are inherently cyclical. An eternal question in real estate investment and management – and one that is still hotly debated – is how to most accurately time the market [2]. As unexpected events can quickly turn the cycle, having an AI trained to find the best high frequency predictor of the market turning point could help the forecaster to adjust their predictions almost in real time. Investors could time the market more precisely, deploy capital more quickly, and gain the full upside potential of the cycle. However, there may never be fully algorithmic, real time trading strategies due to real estate’s high transaction costs and the time it takes to complete any transaction – another area where AI might be able to contribute a great deal. Nevertheless, efforts to predict the market turnaround faster would still be welcomed widely.

Harnessing non-traditional data sources to proxy real estate relevant data is clearly the most exciting use that forecasters can give to an AI. Geographical information maps, textual repositories, social media, credit card transactions, street and satellite imagery, sensor readings, volumetric data, street patterns, transportation metrics, and public records are some examples that can offer additional layers of information to enrich forecasting models. For instance, an AI could be trained to detect social media sentiment through posted photographs and text and create an index that could determine how appealing a particular neighbourhood or area is to live or shop in. Another example could be AI trained to predict highly localised expenditure potential using geolocation data from mobile phones, cross-referenced with credit card expenditures, crucial for predicting the success of retail spaces.

Finally, we could tackle low quality data availability for real estate supply metrics using satellite imagery to track the development pipeline, a longstanding problem in real estate forecasting. High frequency satellite imagery is very expensive. However a smaller subset of images could be used to train an AI for predicting the development state of a building at any point in time from a single image. Whichever the direction, the possibilities seem endless and probably the most exciting AI capabilities for forecasting in the real estate industry are those that we have not even thought about yet. 

The indispensable role of human expertise

Despite AI's impressive capabilities, it is not a replacement for human expertise. The craft of forecasting real estate markets requires a deep understanding of context, something that AI currently lacks. In a recent paper, scholars from Harvard Business School show that consultants using AI on tasks that we know AI excels at – things like product ideation and marketing strategy – significantly improve their productivity. However, when using AI on a task involving quantitative analysis informed by insights gathered from focus groups and interviews – a task clearly outside of AI’s capabilities and expertise – consultants achieved significantly less accurate results [3]. Human experts bring invaluable experience and nuanced judgement essential for interpreting subtleties. At the time of writing this article, local market knowledge, assessing political stability and understanding cultural trends are areas outside of AI’s capabilities frontier where human insight remains indispensable.

Another important challenge of using AI is the opacity of its data analytics processes. AI models, particularly those based on deep learning, often operate as "black boxes" with inscrutable logic. This lack of transparency can be problematic for investors who need to trust and verify the assumptions behind forecasts. Traditional models with explicit assumptions and methodologies, allow for easier scrutiny and validation. The accuracy of AI forecasts heavily depends on data quality and the methodologies used as biased datasets can lead to flawed predictions. For example, an AI model trained on data from affluent neighbourhoods may not perform well in predicting trends in lower-income areas. Ensuring high-quality, unbiased outcomes is a significant challenge that cannot be overlooked and will continue to require human verification.

A balanced approach

We advocate for a balanced approach that combines the strengths of AI with human expertise. AI can handle the heavy lifting of data processing, uncovering patterns and trends at a scale and speed unattainable by human analysts. However, the interpretation and contextual understanding provided by human experts are crucial for validating and refining AI-generated forecasts. Another recent paper concludes that there is clear tension between AI quality and human effort; as the quality and capabilities of AI increase, humans’ incentives to substitute their effort and attention for AI’s output also increase, possibly leading to reductions in output quality. By randomly varying the quality of AI responses to HR recruiters evaluating resumes, the author finds that subjects with higher quality AI outputs were less accurate in their assessments than subjects with lower quality AI. I.e., those receiving lower quality AI, especially more experienced recruiters, exerted more effort and spent more time evaluating the resumes and were less likely to automatically select the AI-recommended candidate [4].

The integration of AI into real estate forecasting represents a significant advancement, offering unprecedented capabilities in data processing and predictive analytics. However, AI's true potential is realised when it is combined with human expertise. Consider AI as a powerful tool in the hands of skilled forecasters who will not just substitute their judgement for whatever output the AI provides. The machine can process data and identify patterns, but it takes a human to understand the broader context, apply nuanced judgement, and make informed decisions based on these insights. By leveraging AI solutions to create usable data and enhance existing models, while relying on human judgement for interpretation and context, we can achieve a faster and more robust forecasting framework. This collaborative approach should provide a more comprehensive view of the market and enable better informed investment decisions across dynamics markets.