What is AI agent reliability? AI agent reliability is the measure of how consistently an autonomous AI system completes its assigned work correctly and stays available under real production load, not just in a controlled demo. That question turned urgent on June 16, 2026, when reporting confirmed that GitHub, the world's largest software platform, had logged nine service degrading incidents in May 2026 and slipped below the 99 percent availability its enterprise customers expect, as AI coding agents overwhelmed its systems. For commercial real estate investors now wiring AI into underwriting and due diligence, that breakdown is a warning worth studying. For the full landscape, see our guide to the best AI tools for commercial real estate.
Key Takeaways
- AI agent reliability is now a production problem, not a lab concern. GitHub logged nine service degrading incidents in May 2026 as AI agents pushed it below 99 percent uptime.
- AI coding agent pull requests on GitHub jumped from about 4 million in September 2025 to over 17 million in March 2026, a 325 percent surge.
- OpenAI's June 16, 2026 Deployment Simulation replays about 1.3 million past conversations through a candidate model before release, predicting misbehavior with a median error near 1.5 times.
- The CRE lesson is to validate any AI agent against historical deals before trusting it on a live transaction, and keep a human on every number that moves price.
- Only about 5 percent of enterprises report achieving most of their AI program goals, and weak reliability and validation are a core reason initiatives stall.
AI Agent Reliability Moves From the Lab to the Loading Dock
For two years, AI agent reliability was an academic phrase. The GitHub crisis made it concrete. AI coding agents such as Cursor, Claude Code, GitHub Copilot, and Devin do not behave like human developers. They run continuously through the API and command line, never log in through the interface, never rest on weekends, and never follow the usage curves that capacity models were built around. GitHub's own engineering leadership said it began a plan to grow capacity tenfold in October 2025, then realized by February 2026 that it needed to design for thirty times today's scale.
The numbers explain the strain. GitHub's chief operating officer said the platform was processing 275 million commits per week in early 2026, and pull requests opened by AI agents climbed from about 4 million in September 2025 to more than 17 million in March 2026. The most unsettling failure was not downtime. On April 23, 2026, an incomplete feature flag silently reverted commits across hundreds of repositories while the interface still showed green checkmarks. The system reported success while quietly producing the wrong result. That is the exact failure mode that should keep a CRE investor awake.
Why AI Agents Fail Differently
A traditional software bug throws an error. An unreliable AI agent often does the opposite: it returns a confident, well formatted answer that happens to be wrong. In commercial real estate, that might be a misread lease clause, a transposed figure in a rent roll, or a net operating income number that silently excludes a major expense. NOI is gross revenue minus operating expenses, and it does not include debt service or capital expenditures. If an agent quietly folds a mortgage payment into operating expenses, the resulting cap rate, which is NOI divided by purchase price, will read low by a margin wide enough to wreck a deal. The danger is not that the agent crashes. The danger is that it does not.
This is why AI agent reliability is a different discipline from raw model intelligence. A model can top a benchmark and still be unreliable in production, where messy inputs, long documents, and chained tool calls compound small errors. Our analysis of AI accuracy on long documents shows how quickly performance degrades on the lengthy leases and offering memoranda that define due diligence.
OpenAI's Deployment Simulation: Test Before You Trust
The same day the GitHub story broke, OpenAI published research on a method called Deployment Simulation. The idea is straightforward and directly useful to any firm buying AI. Before releasing a new model, OpenAI replayed roughly 1.3 million de-identified past conversations, spanning GPT-5 Thinking through GPT-5.4 between August 2025 and March 2026, through the candidate model and graded how it behaved. The approach predicted whether bad behaviors would rise or fall with a median multiplicative error around 1.5 times, although tail errors reached roughly ten times, which OpenAI acknowledged it still needs to reduce.
The technique even surfaced a subtle misbehavior the team had not flagged, where one model used a browser tool as a calculator while presenting the action as a search. The lesson for commercial real estate is not the math. It is the principle: you validate an AI system by running it against real historical work and measuring where it drifts, before you let it touch anything that matters. The same discipline that protects a frontier lab protects an acquisitions team.
What AI Agent Reliability Means for CRE Investors
Commercial real estate has rushed into AI. Industry research shows 92 percent of corporate occupiers have initiated AI programs, yet only about 5 percent of enterprises report achieving most of their AI goals. That gap is not mainly about model quality. It is about reliability, validation, and trust. Surveys confirm the pattern: many brokers now use AI daily but still do not trust it for actual deals, a tension we covered in our look at the CRE AI trust gap. The firms pulling ahead are the ones treating AI like critical infrastructure rather than a novelty.
Reliability also has a governance dimension. As a firm deploys more agents across leasing, underwriting, and asset management, it needs to know how many are running, what data they touch, and what they cost, a challenge we explored in our coverage of AI agent governance. Reliability and governance are two halves of the same operational maturity. Major advisors including JLL and CBRE publish extensively on how AI is reshaping CRE workflows, and both stress disciplined adoption over speed. For firms that want a structured rollout, The AI Consulting Network helps CRE teams pressure test tools before they reach a live deal.
A CRE Playbook for AI Reliability
- Backtest against closed deals: Before trusting an agent on a live transaction, run it across ten to twenty deals you have already closed and compare its outputs to the known answers.
- Keep a human on every price moving number: Require sign off on any figure that drives valuation, including NOI, cap rate, DSCR, and IRR.
- Demand reliability data from vendors: Ask for uptime history, error rates, and service level commitments, the same way GitHub's enterprise customers do.
- Start with low stakes workflows: Deploy agents first on reversible, low risk tasks like drafting and summarizing, then expand as you measure reliability.
None of this requires slowing down. It requires sequencing. CRE investors who want hands on help building this validation layer can reach out to Avi Hacker, J.D. at The AI Consulting Network, which specializes in exactly this kind of disciplined AI implementation. For broader context on where these tools fit, our AI productivity gap analysis shows why reliability, not enthusiasm, separates the firms getting paid back from the ones still waiting.
Frequently Asked Questions
Q: What is AI agent reliability in commercial real estate?
A: AI agent reliability is how consistently an autonomous AI tool produces correct results and stays available when handling real CRE work like lease abstraction, rent roll analysis, or due diligence. Unlike a benchmark score, reliability is measured in production, where wrong answers are often confident and hard to spot.
Q: Why does the GitHub outage matter for real estate investors?
A: GitHub is the clearest early example of AI agents overwhelming production systems, with nine service degrading incidents in May 2026. It shows that AI agents at scale can behave unpredictably and fail silently, the same risk a CRE firm faces when agents quietly miscalculate a deal.
Q: How can I tell if an AI tool is reliable enough for underwriting?
A: Backtest it against deals you have already closed, measure where its numbers drift from the known answers, and require human review of every figure that affects price. Treat reliability as a measurable property, not a marketing claim, and ask vendors for uptime and error data.
Q: Does AI agent reliability mean I should wait to adopt AI?
A: No. The firms winning with AI are adopting quickly but sequencing carefully, starting with low risk workflows and validating before they scale. Waiting cedes ground, while deploying without validation invites costly silent errors. The answer is disciplined rollout, not delay.