Full [un]prompted coverage:
Bill Brenner
Software that detects its own vulnerabilities, writes its own fixes and patches itself while running in production may be closer than conventional development practices suggest.
At [un]prompted II in San Francisco on October 28, Gadi Evron, CEO and co-founder of Knostic, outlined a vision for AI-driven software development in which autonomous agents continuously monitor applications, identify problems and modify code without waiting for traditional release cycles.
The presentation, which Gadi created with veteran CTO Imri Goldberg – From Speculative Engineering Fiction (eng-fi) to ... Now – examined what happens when software development becomes cheap, rapid and increasingly autonomous — and why existing approaches to testing, deployment and security may need to change.
The presentation raised a critical question: If AI agents can generate and deploy software changes almost instantly, how can organizations establish that those changes are safe?
Their proposed direction combines autonomous development with stronger verification, explicit behavioral guarantees and new approaches to monitoring production systems.
From automated patching to self-healing software
The presentation's central idea is that AI agents could eventually become permanent participants in running applications rather than tools used only during development.
Evron described a convergence between software factories — systems that automate software creation — and large-scale autonomous patching.
He argued that combining those capabilities could produce applications able to identify problems and repair themselves.
The model would change the relationship between development and production.
Instead of discovering a vulnerability, creating a ticket, assigning a developer, writing a patch, running tests and scheduling deployment, an AI agent could potentially perform much of that work as part of an ongoing operational process.
He pointed to existing monitoring and debugging technologies, including Rookout, Sentry, Crashlytics and Maxwell by DoubleAI, as examples of the direction software observability is taking.
The broader question was whether agents could operate within production environments and make repairs directly.
Gadi and Goldberg call this concept live mutable software.
In that model, software would no longer be treated exclusively as a fixed artifact that changes through discrete releases. It could become a system that continuously observes its own behavior and adapts.
The presentation did not establish that such an approach is ready for widespread production use. Instead, it explored the engineering capabilities and safeguards required to make it practical.
The missing piece: Trustworthy AI-generated code
The prospect of autonomous software modification creates a difficult security problem. Traditional development processes use code review, automated testing, staged deployments and other controls to reduce the risk of introducing defects.
If agents begin changing production software continuously, those controls may no longer operate at the same pace as the systems they are supposed to protect.
Evron and Goldberg propose several approaches to establishing trust.
In the short term, they point to guarantee-based verification through a contract abstraction layer.
Rather than attempting to prove every property of a complex application, developers could define specific guarantees that generated code must satisfy.
They also recommend checking for trust signals and sending changes through continuous integration for additional inspection.
Longer term, they see formal verification becoming increasingly important.
Formal verification uses mathematical techniques to establish whether software satisfies defined properties.
The challenge is determining exactly what the software is permitted to do — and what it must never do.
Those requirements can be difficult to express, especially for complex business applications involving concurrency, multithreading and unpredictable interactions.
The presentation highlighted existing work involving Lean, Rocq, Dafny, Agda and PVS, along with MIT research combining large language models with formal verification.
Evron and Goldberg referenced work associated with CEA, Airbus and the Astrée static analyzer.
A different operating model for AI agents
Evron argued that organizations should reconsider some of the assumptions underlying software operations.
The proposed model emphasizes recovery, non-determinism, control theory and inexpensive, short-lived code.
Recovery comes first because systems that change continuously cannot depend entirely on preventing every failure.
Organizations would need mechanisms to detect problems, restore safe operation and recover from unsuccessful changes.
They also call for embracing non-determinism.
Unlike conventional software components designed to produce predictable results from fixed inputs, AI agents can generate different outputs or take different paths while pursuing the same objective.
That behavior changes how reliability must be managed.
Control theory offers one possible framework, using feedback to monitor a system and adjust its behavior as conditions change.
The concept of ephemeral code adds another dimension. If software can be generated cheaply, organizations may no longer need to treat every implementation as a long-lived asset that must be maintained indefinitely.
That could encourage more frequent replacement or regeneration of software components.
These ideas remain proposals rather than established operating standards, but they illustrate how the speakers believe AI could change software engineering beyond simply accelerating existing workflows.
Do AI agents need their own programming language?
In the slides, Evron and Goldberg question whether the programming languages used by humans are necessarily the best languages for autonomous coding agents.
Python, Rust and TypeScript were designed around human development needs.
They suggest that an ideal language for large language models might instead prioritize machine readability and editability, concise expression and the ability to verify correctness.
Such a language could make it easier for agents to generate and modify code while allowing other systems to evaluate whether the result satisfies defined requirements.
The idea reflects a larger theme running through the presentation: Rather than simply inserting AI into existing software development processes, engineers may need to redesign those processes around what AI systems can do.
That includes reconsidering how code is written, tested, verified, deployed and maintained.
Human oversight could become the next bottleneck
One of the presentation's more consequential concerns involves the limits of human supervision.
As AI-driven systems become more complex and operate at greater speed, humans may struggle to understand the decisions those systems make.
Evron and Goldberg reference Alastair Reynolds' science-fiction story Great Wall of Mars to illustrate the problem of supervising systems that exceed human cognitive capacity.
A human approval step does not necessarily provide meaningful oversight if the person approving an action cannot understand the underlying system or its consequences. This creates a challenge for organizations adopting autonomous software development. Keeping a human involved may satisfy a procedural requirement, but that alone does not guarantee effective control.
The emphasis on behavioral guarantees, monitoring and verification reflects their attempt to address that problem.
As autonomous systems become faster and more complex, oversight may increasingly depend on whether organizations can define and enforce what those systems are allowed to do.
Lessons for developers and security teams
The presentation identified several areas where organizations can begin experimenting with the emerging model.
Everyday development tools offer an immediate opportunity to explore AI-assisted software generation and modification.
More sensitive applications require stronger guarantees.
They cite tenant deletion as an example of a behavior that could be described through explicit requirements, allowing a system to verify whether an operation meets its intended constraints.
At the far end of the spectrum, they envisioned dynamic software modification becoming reliable enough for environments with exceptionally demanding engineering requirements, such as aerospace systems.
That remains an ambitious goal rather than an established capability.
For organizations evaluating AI coding and remediation tools, the presentation points to practical questions:
- Can the system identify and explain the behavior it intends to change?
- What guarantees must a generated patch satisfy before deployment?
- How will the organization detect an unsuccessful change and recover?
- Can monitoring and verification keep pace with automated code generation?
- Does human approval represent meaningful oversight or merely another procedural step?
The answers will help determine whether autonomous software modification becomes a dependable engineering practice or remains confined to demonstrations and experimental environments.
From speculation to engineering
Evron and Goldberg call for practitioners to experiment rather than wait for every aspect of the technology to mature.
They encouraged developers to revisit ideas that failed even a month earlier, reflecting the rapid pace of change in AI capabilities.
But the presentation also ended with a warning against confusing imagination with a finished engineering solution.
The final slide states: "But engineering requires precision."
AI agents may make it possible to generate, modify and repair software at speeds that traditional development processes cannot match.
The harder task is building systems that can establish whether those changes are correct, safe and appropriate for production.
For security teams, the question is no longer limited to whether AI can write code or discover vulnerabilities: It's whether the next generation of software can be trusted to repair itself.
