# Design for the defenders you care about or risk being useless *On operational constraints of critical infrastructure providers and the limits of AI-powered vulnerability discovery* AI systems are getting genuinely good at finding software vulnerabilities. That’s not necessarily good news for everyone. Research and product teams are building systems that find zero-days, and we are all very excited. Find vulnerabilities before attackers do; patch them; systems are more secure, yay. Nevertheless, we’re all a little *too* excited about vulnerability discovery. As [has been noted before](https://ifp.org/operation-patchlight/), vulnerability discovery only helps improve the state of cybersecurity if the discovered vulnerabilities have patches written, tested, and deployed in all the places where a given piece of code is used in a timely manner. This is not so easy to do in the real world. In the real world, patches have to be tested for compatibility, maintenance windows have to be negotiated, and systems have to be rebooted to apply patches. In the real world, no one wants to interrupt patient care or power delivery for an invisible threat. I’m hopeful that we can automate this whole process in a way that doesn’t force organizations to choose between uptime and security, perhaps through some combination of [runtime patching](https://arxiv.org/pdf/2203.12132v1) and [AI triage and prioritization](https://docs.gitlab.com/user/duo_agent_platform/agents/foundational_agents/security_analyst_agent/), integration testing, and deployment. But full automation is a long way off, and in the meantime, already vulnerable organizations will have to deal with increasing threats from AI-enabled cyberattacks. If organizations can't actually patch the vulnerabilities AI discovers, attackers will be the main beneficiaries of better discovery tools. Defenders need AI tools that go beyond vulnerability discovery. Taking into account the real-world operational challenges faced by the defenders we care about is the only way to make sure that whatever we build with AI in cybersecurity ends up benefiting defense more than offense. ## The vulnerability lifecycle[^1] To understand my argument, it helps to trace the full path of a vulnerability. Software vulnerabilities are flaws in code that attackers can exploit to gain unauthorized access or compromise a system. Most modern software is riddled with them, and [there's no way to know when you've found the last one](https://www.sciencedirect.com/science/article/abs/pii/S0167404823001013). This creates an asymmetry: defenders have to find and fix *every* exploitable flaw, while attackers only need to find one. As Caleb Withers puts it in his [excellent report on offense defense balance](https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/Report_Tipping-the-Scales_TECH_Sep-2025-Final.pdf), "It is currently impractical to find and fix them all \[...\]. Rather than aiming for 100 percent vulnerability-free systems, defenders can only hope that by discovering and addressing the most critical vulnerabilities, whatever they miss is sufficiently difficult for attackers to find and exploit." The promise of AI-driven vulnerability discovery is that it might even out this asymmetry. Research labs and startups are building systems designed to find vulnerabilities at scale, at various parts of the software lifecycle, such as during development, in pre-release audits, and on deployed systems. [OpenAI’s Aardvark](https://openai.com/index/introducing-aardvark/) is embedded directly into development pipelines to scan commits as they happen, validate whether findings are actually exploitable, and propose patches. [Google’s Big Sleep](https://googleprojectzero.blogspot.com/2024/10/from-naptime-to-big-sleep.html) project uses LLMs to do variant analysis: given a past vulnerability, it looks for similar bugs in a codebase. The AI system found an exploitable SQLite vulnerability before the bug ever reached a release. And [XBOW](https://xbow.com/) automates penetration testing, or simulations of real attacks against deployed systems. Their agents work to discover and exploit vulnerabilities in running applications. But finding a vulnerability is only the first part of its story. Once one is found, someone has to write a patch for it. Then that patch has to be deployed to every system running the vulnerable code. The patching step is not trivial. Imagine you learn about a new vulnerability affecting your systems. Hopefully, a vendor has already released a patch you can use. You first have to decide how dangerous the vulnerability is for your systems and whether it’s worth your effort to patch it.  If you decide to go ahead with patching, you must then test the compatibility of the patch with the rest of the systems. Then, you need to actually deploy the patch. Maybe your organization has routine maintenance windows, but you need to decide whether the risk from the vulnerability is sufficiently low that you can wait until then. Or maybe there are no maintenance windows. In any case, you’ll have to coordinate with the rest of your organization as to when you’ll be able to actually have the time to do the patching. And after all that, you’ll have to test the patch and make sure that all went as expected. It’s very likely that you’ll run into issues at some point in this process. Patches might have dependencies on other software that cause compatibility problems. Testing might reveal errors that require troubleshooting. Deployment might fail and require rollbacks. And even when everything works technically, you’ll still have to do a bunch of human coordination: patch deployment usually requires reboots, reboots cause downtime, and getting approval for downtime from customers or upper management takes time. In healthcare organizations, for example, [it seems that the majority of patching delays](https://arxiv.org/pdf/2202.09016) occur during the deployment phase, and the most common cause of delays overall is coordination. On the whole, patching is an annoying process, and even well-resourced organizations have to make trade-offs about how often to do it and which patches to prioritize. For organizations with little tolerance for downtime, such as [hospitals that operate 24/7](https://arpa-h.gov/news-and-events/arpa-h-announces-program-automate-cybersecurity-health-care-facilities#:~:text=taking%20a%20critical%20piece%20of%20hospital%20infrastructure%20offline%20for%20updates%20can%20be%20very%20disruptive) or [factories where uptime \== revenue](https://digital.txone.com/media/txone-networks-2024-annual-ics-ot-cybersecurity-report/prioritizing-vulnerabilities-and-overcoming-patching-challenges#:~:text=especially%20in%20industries%20such%20as%20manufacturing%20and%20energy%2C%20where%20business%20continuity%20is%20of%20paramount%20importance%2E), patching is at direct odds with their operational goals. Because of the difficulty of the patching process, there is a lag between when a patch is made available to a user and when an end user patches it. Verizon's annual [Data Breach Investigations Report](https://www.verizon.com/business/resources/reports/2024-dbir-data-breach-investigations-report.pdf) found that it takes around 55 days after a patch has been made available to remediate 50% of critical, actively exploited vulnerabilities.[^2] ![[verizon_dbir.png| centering | 500]] *Survival analysis of vulnerabilities. Even critical, actively exploited vulnerabilities take months to patch. Source: [Verizon DBIR 2024](https://www.verizon.com/business/resources/reports/2024-dbir-data-breach-investigations-report.pdf)* For comparison, the mean time to exploit a vulnerability once it has been publicly disclosed is [around 5 days](https://cloud.google.com/blog/topics/threat-intelligence/time-to-exploit-trends-2023). In other words, on average, by the time half of vulnerable systems are patched, attackers have had nearly two months to exploit them. Finding more vulnerabilities doesn't help improve security if the bottleneck is patching capacity. If AI gets better at finding vulnerabilities, under-resourced organizations are worse off. They'll have more vulnerabilities to deal with and the same strained patching processes. We're seeing a similar dynamic play out with [open source maintainers](https://thenewstack.io/ffmpeg-to-google-fund-us-or-stop-sending-bugs/)[^3], who say they can't keep up with AI-discovered bugs. ## Designing for constraints Organizations face constraints: in operations, in funding, in personnel, in legal obligations. To actually improve cybersecurity in the real world, AI solutions should be designed with the constraints of their intended users in mind. We may wish that the world were a certain way, where patching was easy, where organizations didn’t still use Windows XP[^4], where we knew which vulnerabilities were actually the most critical, where we could replace inherently insecure devices once alternatives became available. But that's not the world we live in. If proposed AI solutions don't account for this, they will be of no use to the institutions that actually need help. Some of this is just lock-in. Banks [still run mainframes](http://c) because rewriting decades of COBOL isn't something anyone actually wants to do. Utilities and manufacturing plants run SCADA systems built to last decades, never meant to be networked,[^5] but now they are. Even if you believe that organizations like these will eventually be fully automated, there's a long time between now and "eventually" when critical infrastructure remains vulnerable. But there’s useful work to do now, and security practitioners already have precedent for thinking like this. There's a concept called [compensating controls](https://www.fortinet.com/resources/cyberglossary/compensating-controls): security measures you put in place when you can't fix the underlying problem. In ICS/OT[^6] environments, [virtual patching](https://www.fortinet.com/resources/cyberglossary/virtual-patching) is a common example. When a vulnerability is disclosed, operators deploy firewall rules within hours to block exploitation while the real patch gets developed, tested, and scheduled for the next maintenance window.[^7] These solutions recognize that OT operators don't have the same flexibility in patching that IT teams do. Similarly, [Privileged Access Management solutions](https://www.beyondtrust.com/resources/glossary/privileged-access-management-pam) exist in part because we can't quite yet get rid of shared accounts, hardcoded credentials, and default passwords. AI opens up new avenues for this kind of constraint-aware design. GitLab has a [Security Analyst Agent](https://docs.gitlab.com/user/duo_agent_platform/agents/foundational_agents/security_analyst_agent/) that tries to help with vulnerability triage, false positive assessment, and remediation planning. I’d be really excited to see similar technologies built for other platforms. I’d also be excited to see frontier LLM cyber capabilities thrown at all of the following parts of the vulnerability lifecycle in earnest[^8]: - Asset inventory and attack surface discovery tools - Vulnerability prioritization + triage (for developing a patch in the first place) - Patch prioritization + triage (for organizations deploying vendor patches) - Automated network topology analysis to determine which of your vulnerabilities are actually critical given your specific configuration - Systems that simulate patch rollouts to predict compatibility issues before you commit - Optimal scheduling tools that minimize downtime while maximizing patch coverage - Monitoring systems that detect whether your unpatched vulnerabilities are being actively exploited ## Conclusion To be clear, I'm *also* excited about AI-driven vulnerability discovery. For well-resourced organizations with mature security teams and flexible maintenance windows, it's going to be genuinely useful. We should keep building it. But AI vulnerability discovery is just going to happen: the market incentives are there, and the technology is already emerging. What I want more of is the stuff that won't happen by default: tools designed for the stressed-out hospital IT team, the understaffed utility company, the organizations where "just patch it" isn't a realistic answer. The promise of AI is that it can make labor-intensive work cheap enough to help everyone. But for critical infrastructure and under-resourced organizations, that's only going to happen with government or philanthropic support. The market isn't big enough, and they don't have the budgets to attract commercial interest. I don't know exactly what this technology will look like, and I’m probably wrong about the real constraints these organizations face. But the design principle remains: design for the constraints of the defenders you care about, or risk being useless (or worse: actively detrimental). If we’re serious about using AI to improve security, we have to design for the world as it is. *Thank you to E.T.J., Adam Swanda, and GovAI research staff for feedback on a draft of this post.* *Published Dec 8, 2025.* ## Appendix: Medical and Operational Technology Devices Everything above is about IT systems. When I talk about downtime for hospitals or utilities, I'm still talking about their IT layer. But there are two other classes of particularly vulnerable technologies in use by CI providers: medical devices (MRI machines, infusion pumps, pacemakers) and OT devices (PLCs controlling power plants, SCADA systems in manufacturing). These devices are particularly scary attack vectors because they're the interface between the cyber and physical worlds. A tampered infusion pump means a patient actually harmed. A hijacked PLC means changing the flow of water or chemicals or worse. They share three properties that make them even harder to patch than regular IT: 1. **Long designed lifespans.** The [FBI notes](https://www.ic3.gov/CSA/2022/220912.pdf) that medical device hardware often remains active for 10-30 years. OT systems are [similar](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r3.pdf), designed to last decades. 2. **Legacy software.** Both [medical devices](https://www.ic3.gov/CSA/2022/220912.pdf) and [OT systems](https://digital.txone.com/media/txone-networks-2024-annual-ics-ot-cybersecurity-report/prioritizing-vulnerabilities-and-overcoming-patching-challenges) run on operating systems that are long past end-of-life and will never receive another patch from the vendor. 3. **Patching requires downtime.** And downtime means [patients don't get care](https://www.usenix.org/system/files/usenixsecurity25-kustosch-patching.pdf) or [production stops](https://digital.txone.com/media/txone-networks-2024-annual-ics-ot-cybersecurity-report/prioritizing-vulnerabilities-and-overcoming-patching-challenges). 4. These factors leave medical care providers and OT users exposed. But they also don't seem to be getting attacked much. An [HHS report](https://405d.hhs.gov/Documents/405d-hospital-resiliency-analysis.pdf) writes: "threat intelligence and breach data suggest medical devices are not a prominent attack vector for adversaries to disrupt hospital operations." OT devices have a bit more of a track record. [Stuxnet](https://en.wikipedia.org/wiki/Stuxnet) remains the canonical example of what's possible, and more recently, Iran-affiliated actors [targeted Israeli-made PLCs](https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-335a) in U.S. water facilities. But the descriptions of these attacks make it sound like [maybe it wasn’t a huge deal](https://www.waterworld.com/water-utility-management/article/14302077/aliquippa-pennsylvania-suffers-cyberattack-on-booster-station-plc), often not impacting operations at all. OT attacks are currently rare, resource-intensive, and not scalable cybercrime. So should medical and OT devices be a top priority? Not today, not really.[^9] But the calculus might change really quickly. [Anthropic's recent report](https://assets.anthropic.com/m/ec212e6566a0d47/original/Disrupting-the-first-reported-AI-orchestrated-cyber-espionage-campaign.pdf) of a Chinese-affiliated group using Claude to execute 80-90% of a sophisticated cyber espionage campaign shows that we're very close to autonomous hackers, much faster and much cheaper than human labor. If attack costs drop far enough, the current logic of "not worth the effort" could shift. Maybe right now the ROI of attacking a few medical devices is insufficiently disruptive for an adversary, but if you can just launch automated agents to do it, maybe we'll see more of that. And then the lifespan, legacy, and patching problems for these devices are really going to come back to bite us. [^1]: I highly recommend reading [Chris Rohlf’s excellent blog post](https://cset.georgetown.edu/article/ai-and-the-software-vulnerability-lifecycle/) for more details on many of the steps I gloss over. [^2]: Specifically, vulnerabilities in CISA’s [Known Exploited Vulnerabilities Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog). [^3]: The bottleneck I describe above is in patch deployment, where the patch is provided by a vendor; for open source projects, it's often patch creation itself. Volunteer maintainers write the patches for open source libraries. [^4]: See [CISA Cyber Risk Summary: Energy Sector](https://commerce.idaho.gov/content/uploads/2021/09/TLP-AMBER_Energy-Cyber-Risk-Summary.pdf) in 2020: “24% of enrolled Energy Sector entities ran unsupported Windows operating systems (OSs) that no longer receive routine security updates on at least one internet-accessible host at the end of Q4 of 2020.” [^5]: See [NIST SP 800-82r3 (Guide to OT Security)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r3.pdf): “Initially, OT had little resemblance to traditional information technology (IT) systems because OT systems were isolated, ran proprietary control protocols, and used specialized hardware and software.” (p. 28); “The lifespan of an OT system can exceed 20 years. “ (pg. 37\) [^6]: Industrial Control Systems / Operational Technology [^7]: Virtual patching doesn’t cover all attack vectors and attackers [can evolve around them](https://www.giac.org/paper/gcia/12447/methods-controlled-deployment-operation-virtual-patching-program/143808), which is why it’s not a permanent solution. [^8]: Credit to E.T.J. and A.S. for some of these ideas. [^9]: One piece of evidence for "yes, this matters”, though, is that [ARPA-H is pouring $43 million](https://arpa-h.gov/news-and-events/upgrade-launches-fortify-medical-devices-against-cyber-threats) into tools for hospitals to detect and remediate vulnerabilities in medical devices. Maybe they're just AI-pilled, or maybe they see something I don’t.