Amazing and Terrifying: Securing an App Built By AI in a Weekend

What a routine security ticket revealed about how fast business builders are shipping software, and what still has to catch up.

Dudi Peretz, Information Security Engineer, has spent more than ten years in security review. What he saw when he opened the ticket for Orca’s new partner portal, built over a weekend by a colleague using an AI development tool, was something he now describes as routine in one sense and unusual in another.

How the security review of an AI-built app started

The app in question was built by Shannon McWilliams, a new Orca hire tasked with rethinking the company’s partner experience. Orca’s CEO wanted to launch the finished portal quickly. Peretz’s job was to make sure it was ready first, given that it would carry real partner and customer data.

He started by asking McWilliams to sync the platform’s configuration to Orca’s GitHub repository, so the company’s own AppSec tooling could scan it for vulnerabilities and exposed secrets. From there he ran additional checks, looking at the platform the way an outside attacker would, and the way an insider with too much access might: row-level security, least-privilege violations, what was visible from outside the system and what was visible once inside it.

What AI app builders get right, and what they leave unprotected

“It’s actually amazing, and terrifying,” Peretz says after reviewing AI-built software for the first time. “The AI understands what you’re asking it to do. If you tell it to build a platform, it will build you a platform, and create you username and password authentication, and you put the database in, and add users, and it’s all okay. But the missing part I’m seeing most is they just don’t feel secure.”

Peretz’s findings we’re not unique to this one case and scenario. In fact, software built this way, AI-native, typically overlooked security in the development phase. Users could see data belonging to other users. Records that should have been limited to one partner were visible more broadly. None of it, he says, reflects bad intent on the builder’s part. It reflects what an AI tool optimizes for by default: doing exactly what it was asked, not anticipating what it wasn’t. In other words, if the person behind the AI does not understand security procedures in development, there is no way for them to prompt an AI to scan the code or setup the correct security guardrails for the app in production.

Why a platform’s built-in security scan isn’t enough

“If it was being built in the traditional way, we’d have a security review, a PR, checks that we’re doing, a separation between admin, reader, and so on,” Peretz says. “But when you build with AI, it’s so fast that a human can’t really keep up with the PRs.” He estimates McWilliams had opened something close to 3,000 pull requests by the time his review began. He says he sees the same pattern across other AI development tools and hears about it often in security discussions online.

One finding stood out to him enough to repeat as a general warning. The AI platform’s own built-in security scan told them the app was fine and ready for production. Peretz’s own tools said otherwise. “I’m running the scan, I’m running the checks, and I’m like, okay, you can’t go to production,” he says, “even though the software told us we were great.”

Using AI to fix AI-generated vulnerabilities

From there, the review became a back-and-forth rather than a single audit. Peretz would flag an issue, and McWilliams would paste the finding directly into the AI platform’s interface, which would generate a fix, often within a few minutes. Peretz would rerun his scan to confirm it held, and the two repeated that cycle several times as the product kept changing under active development. An outside firm later performed penetration testing as a final layer of verification behind the internal work.

Peretz says the timeline of the fixes changed how he thinks about the review process itself. Work that a few years ago might have waited a quarter for an engineer to check a single dependency now closes in an hour or two. “It’s pretty amazing, the progress we’re making with AI,” he says.

From one portal review to Orca AI AppGen Security

Peretz says the review process built around McWilliams’ portal wasn’t a one-off fix. It shaped how Orca now thinks about securing software built outside a traditional engineering pipeline altogether. Some of Orca’s own customers, he says, are already using the company’s scanning capabilities against their own applications, and turning up the same category of misconfigurations.

That demand is part of why Orca launched AI AppGen Security in July 2026, a capability built to discover AI-generated applications across platforms like Lovable, Supabase, and Claude, map their risk across APIs, integrations, and data access, and prioritize exposures based on real business impact. The company’s own 2026 State of AI Security Report found that 52% of organizations were already building custom applications with AI, a scale Peretz says matches what he sees inside Orca itself.

How Orca now secures AI-built apps continuously

The portal review also reshaped how Peretz’s team operates day to day. He says nearly everyone at Orca is building something now, dashboards, small tools, agents, and each one raises the same set of questions: what permissions does it have, what data does it touch, where is it hosted, has anyone checked it. A one-time review before launch no longer matches how the company builds. Every change McWilliams pushes to the portal now syncs automatically to GitHub and is scanned as it lands. If something is off, Peretz flags it directly, and it is typically fixed the same day.

He also points to a subtler risk: some builders access these AI platforms using personal email accounts rather than company ones, which means the organization has no record the tool exists at all, until something leaks and no one can trace where it came from.

When asked what he would tell someone building this way for the first time, Peretz doesn’t hedge. “Don’t trust the security scan. Never upload this before running your own check,” he says. “Run a background check and scan all of your files and configuration before you click publish and share the link.”

Security for the companies that build

The biggest risk with AI-built software is the one nobody knows exists. A personal email account is all it takes for an app to stay invisible until it leaks. Multiply that across Lovable, Replit, Vercel, Bolt and chasing each platform individually stops working. Orca AI AppGen Security discovers every application across these platforms and who built it, maps its risk across APIs, integrations, and data access, and prioritizes the exposures that matter, in one place, continuously, without slowing the builder down. Built for the companies that build. 

This is the second of two pieces on the build and review of Orca’s AI-built Partner Portal. The first follows Shannon McWilliams, who built the portal, through the weekend he built it and what he learned working with Peretz’s team afterward.

If this topic piques your interest, please join us for the Builder Exchange November 4th. We’ll share research and stories from Co-Founder & CEO of Orca Security, Gil Geron; Troy Hunt, founder of Have I Been Pwned; along with analysts and security leaders across the community.