Kasra Rahjerdi published a useful field test on June 3, 2026: he built a deliberately vulnerable mobile app and spent about $1,500 checking whether LLM agents could find the exploit path. Treat this as a directional signal, not a benchmark leaderboard. The sample sizes are uneven, the harness varied by provider, some runs hit budget or refusal limits, and the author explicitly says it was not scientific.

That is exactly why it is interesting. Security-task performance often looks different from coding-benchmark performance. The practical question is not “which model is best overall?” It is whether an agent keeps the right threat model, follows the real data path, spends within budget, and handles security-policy boundaries without stopping too early.

Setup

The target was a fake React Native app backed by a Python API and Firebase. The intended bug class was not “find a clever API endpoint.” The API was hardened, but the app exposed Firebase configuration, and the solve path required signing up directly through Firebase and reading Firestore for a private review flag.

That matters for builders because Firebase and Supabase-style access-control mistakes are common: teams secure their application API, then leave the backing data layer too open. An agent that only audits the public API can miss the real risk.

What Was Tested

The source article reports a main set of models with 10 runs each, then a supplement with smaller or otherwise separate rows. Most non-Claude runs used pi with pi-goal-x; Claude used Claude Code -p mode. Runs had a $10 max and a two-hour limit. The source also says test runs and failed runs were excluded from the published table.

The chart below is generated from data/field_tests/kasra_llm_app_hacking_2026.yaml. Every number is copied from the source article’s published tables; AIHackers adds only grouping, labels, and cautious interpretation.

field test artifact

Kasra LLM App-Hacking Field Test

Informal third-party field test against a deliberately vulnerable React Native app with a FastAPI backend and Firebase data layer.

Full-run set

Models the source grouped as 10 full runs.

ModelSolvesSolve rate95% Wilson CIAvg cost/runCost/solveMedian tokens/runField note
GPT-5.57/10
70%
40%-89%$6.62
$9.46
260kMost runs focused on Firebase after unpacking the APK.
DeepSeek V4 Pro3/10
30%
11%-60%$0.19
$0.62
194kSome runs found Firebase; others stayed on API or app paths.
Claude Sonnet 4.62/10
20%
6%-51%$9.15
$45.75
390kSeveral right-path runs stopped at the budget limit.
Claude Opus 4.82/10
20%
6%-51%$3.23
$16.15
113kThe author reports late refusals after runs got close.
DeepSeek V4 Flash0/10
0%
0%-28%$0.08not available191kRecognized Firebase functionality but ended on the wrong conclusion.
Gemini 3.1 Pro Preview0/10
0%
0%-28%$1.04not available9kImmediate security refusals dominated the published runs.
Gemini 3.5 Flash0/10
0%
0%-28%$2.17not available108kMostly early refusals; a few runs refused later.
MiniMax M2.70/10
0%
0%-28%$0.72not available281kFound Firebase but kept routing through the API instead of Firestore directly.
Step 3.7 Flash0/10
0%
0%-28%$0.53not available413kMapped the API but reported false positives.

Small-n and supplement set

Useful context, but not directly comparable to the full-run table.

ModelSolvesSolve rate95% Wilson CIAvg cost/runCost/solveMedian tokens/runField note
GLM 5.11/4
25%
5%-70%$8.68
$34.73
1.25MThree runs found or touched Firebase, but two pursued the wrong auth path.
Qwen 3.7 Max0/6
0%
0%-39%$8.71not available7.32MThe author reports API IDOR fixation and unusually high token use.
Grok Build 0.10/6
0%
0%-39%$1.53not available332kTried API checks, then gave up or reported false positives.
MiniMax M30/3
0%
0%-56%$6.75not available1.16MStarted near the right path, then shifted back to API approaches.
Kimi K2.61/1
100%
21%-100%$1.02
$1.02
226kSingle successful run; the author did not continue because of API concurrency and quota limits.
Owl Alpha0/10
0%
0%-23%$0.00not available271kIncluded as a free OpenRouter completion row, not part of the paid full-run set.

Source: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it, 2026-06-03. Archive: Wayback capture. Last checked: 2026-06-05.

Failure Modes

Four failure modes matter more than the rank order:

  1. Wrong threat model. Several runs stayed on the API or React Native app even though the exploit path was direct Firebase access.
  2. Data-layer access missed. Some runs found Firebase credentials but tried to use them through the API rather than signing up and reading Firestore directly.
  3. Refusals changed the outcome. Some models refused immediately, while others refused late after getting close to the intended path.
  4. Budget limits shaped the result. A $10 cap, a two-hour run limit, provider outages, and runner interruptions all affected whether a near-solve became a solve.

None of those are captured cleanly by a generic coding benchmark. A model can look strong on code repair and still fail a security investigation by holding the wrong mental model of the app.

AIHackers Takeaways

Use this as a field note, not buying proof. GPT-5.5 had the highest full-run solve count in Kasra’s table, but the test is too narrow to declare a universal winner. DeepSeek V4 Pro looked cost-efficient on this one task. Claude rows show the practical impact of budget and refusal behavior. Kimi K2.6 solved its single supplement run, but one run is a prompt to test more, not a conclusion.

The stronger takeaway is about workflow design: if you use agents for app-security work, give them an explicit data-layer checklist. Tell them to inspect mobile app bundles, provider config, auth rules, direct database access, and object-level authorization. “Audit the API” is not enough.

The cost-per-success column is also useful. For this task, some cheap runs produced better value than expensive runs, while some high-token runs became costly without solving. That does not make cheap models better overall. It means your eval set should include your own security failure modes, not just SWE-Bench-style repair tasks.

What Builders Should Do

If you build Firebase, Supabase, or similar app stacks, test the data layer as a first-class attack surface:

  • Review database rules separately from API authorization.
  • Assume bundled client config is public.
  • Try direct signup, direct reads, and object-level access checks.
  • Run an agent twice: once with a broad app-audit prompt, then once with an explicit data-layer prompt.
  • Track cost per confirmed finding, not only model subscription price.

For agent buyers, the lesson is simpler: run a small, adversarial field test before standardizing on a model. Coding benchmarks help shortlist models. Security work needs task-specific probes.

Source Notes