Vision LLMs and CAPTCHA Solving: Comprehensive Findings

Executive Summary

The CAPTCHA-vs-LLM arms race has reached a critical inflection point. Vision language models (VLMs/MLLMs) now reliably defeat legacy image-classification CAPTCHAs, but modern interaction- and behavior-based defenses remain largely intact. The picture is nuanced: no single model dominates, performance varies enormously by CAPTCHA type, and the most sophisticated defenses (reCAPTCHA v3, Cloudflare Turnstile’s behavioral stack) currently resist direct LLM attacks. Meanwhile, real-world criminal actors already combine LLMs with third-party solver services at scale, as demonstrated by the AkiraBot campaign that successfully spammed over 80,000 sites.

1. Benchmark Results by CAPTCHA Type

1.1 Roundtable Research — reCAPTCHA Agent Benchmark (October 2025)

Source: Benchmarking Leading AI Agents Against CAPTCHAs — Roundtable Research, October 2025

Scope: reCAPTCHA v2 image challenges on Google’s official demo page. 388 total attempts across 75 trials using the Browser Use framework. Tested the latest frontier models available at the time.

Overall success rates:

Model	Success Rate
Claude Sonnet 4.5	60%
Gemini 2.5 Pro	56%
GPT-5	28%

Performance by challenge type:

Model	Static (3x3 grid)	Reload (dynamic replacement)	Cross-tile (4x4, spanning objects)
Claude Sonnet 4.5	47.1%	21.2%	0.0%
Gemini 2.5 Pro	56.3%	13.3%	1.9%
GPT-5	22.7%	2.1%	1.1%

Key failure modes:

GPT-5 performed worst despite being the most capable reasoning model — it spent too long in extended thinking, generated excessive “Thinking” tokens, and obsessively clicked/unclicked squares, leading to frequent timeout errors.
Reload challenges trapped agents in retry loops: agents clicked correct initial squares but new images appeared, causing confusion.
Cross-tile challenges (objects spanning multiple grid squares) defeated all models — they produced rectangular selections rather than identifying partial/occluded objects at boundaries.

Insight: More reasoning is not always better for CAPTCHAs. Quick, confident decision-making matters more than deep analysis. Claude Sonnet 4.5 won by being efficient, not by being the smartest.

1.2 Open CaptchaWorld (NeurIPS 2025)

Paper: Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents — MetaAgentX, NeurIPS 2025

Scope: 20 modern CAPTCHA types, 225 total CAPTCHAs, evaluated with a “CAPTCHA Reasoning Depth” metric counting cognitive + motor steps per puzzle.

Human baseline: 93.3% pass rate

Model pass@1 results (overall, across all 20 types):

Model	Pass@1
openai-o3 (2025-04-16)	40.0%
GPT-4.1 (2025-04-14)	25.0%
Gemini 2.5-Pro	25.0%
Claude 3.7 Sonnet	20.0%
DeepSeek-V3	20.0%
Claude 3.5 Haiku	15.0%
Claude 3.5 Sonnet	10.0%
GPT-4o	5.7%
openai-o1	5.0%

Gap: Human performance exceeds best model by ~53 percentage points.

CAPTCHA types completely unsolved by all models: Dice Count, Geometry Click, Slide Puzzle, Place Dot, Connect Icon, Click Order, Misleading Click, Pick Area.

CAPTCHA types solvable by most models: Image Recognition, Image Matching, Select Animal.

Notable unique solves: Only openai-o3 solved Rotation Match and Dart Count; only Claude 3.7 Sonnet solved Hold Button and Bingo.

1.2 COGNITION Study (December 2024)

Paper: COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

Scope: 7 representative multimodal LLMs tested against 18 real-world CAPTCHA types under a realistic black-box threat model (including retry budgets, latency, and cost constraints).

Key finding — hardness divide:

“Broken” / reliably solvable (>40% pass@1): Path_Finder, Select_Animal, Image_Recognition, Object_Match, Bingo — these reach near-certain solve rates within a few retries and cost <$0.10 per successful solve.
“Robust” / consistently hard (<20% pass@1): Dice_Count, Place_Dot, Click_Order, Pick_Area, Patch_Select, Rotation_Match — these remain robust even with optimized prompting and few-shot examples; costs per solve run 1–2 orders of magnitude higher.

Updated results with latest models (GPT-5 tested):

GPT-5 (Medium) overall: 59.4% (original prompts) → 60.7% (optimized prompts)
Average across all 7 models: ~42–43%

Root cause of failures on hard types: Persistent weaknesses in spatial grounding and object-position binding (i.e., models can classify objects but cannot reliably place clicks at precise coordinates).

1.3 MCA-Bench (June 2025)

Paper: MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks

Dataset: 180,000+ training samples, 4,000-item test set, organized into four clusters: OCR robustness, target retrieval, human-like interaction behaviors, multi-step language reasoning.

Key accuracy results:

VLMs exceed 96% accuracy on simple OCR-style tasks.
Accuracy drops to as low as 2.5% on complex tasks requiring physical interaction or multi-step reasoning.
On 3x3 grid selection: fine-tuned Qwen2.5-VL-7B achieves 96.5% Pass@2, outperforming ChatGPT-4o (78.0%) and Seed1.5-VL (80.0%).
Slider puzzles are rated most resilient due to reliance on authentic user-generated mouse trajectories.

1.4 Breaking reCAPTCHAv2 — ETH Zurich (COMPSAC 2024)

Paper: Breaking reCAPTCHAv2 — Andreas Plesner, Tobias Vontobel, Roger Wattenhofer, ETH Zurich. Accepted at COMPSAC 2024.

Method: Custom YOLO model fine-tuned on reCAPTCHA object categories (cars, bridges, traffic lights, etc.), not an LLM per se, but represents the state of automated image-classification attack.

Result: 100% success rate on reCAPTCHAv2 image challenges (previous state of the art: 68–71%).

Critical insight: No significant difference in the number of challenges humans vs. bots must solve — reCAPTCHAv2’s real line of defense is behavioral/cookie signals, not image difficulty.

1.5 Oedipus: LLM-Enhanced Reasoning CAPTCHA Solver (May 2024)

Paper: Oedipus: LLM-enhanced Reasoning CAPTCHA Solver

Target: Reasoning-based CAPTCHAs (logic puzzles embedded in CAPTCHA).

Result: Average success rate of 63.5% using a pipeline that decomposes complex CAPTCHA tasks into sequences of simpler steps for the LLM to reason through.

1.6 Reasoning Under Vision / CAPTCHA-X Benchmark (October 2024)

Paper: Reasoning under Vision: Understanding Visual-Spatial Cognition in Vision-Language Models for CAPTCHA

Key finding: Most commercial VLMs (Gemini, Claude, GPT) fail to solve CAPTCHAs effectively, achieving a baseline accuracy of approximately 21.9% without reasoning scaffolding. Requiring step-by-step spatial reasoning before generating coordinates substantially improves performance.

2. Specific Model Performance

GPT-4V (original)

Informal testing (CHEQ, 2023): 80% on 5 tested CAPTCHAs (4/5 solved). Struggled with precise grid cell selection — missed traffic-light boxes, misclassified crosswalks.
Academic baseline: ~21.9% on structured CAPTCHA benchmarks without reasoning scaffolding.
Was notable for solving some CAPTCHAs but not reliably or at production scale alone.

GPT-4o

Open CaptchaWorld: 5.7% overall across 20 CAPTCHA types (as browser agent).
MCA-Bench 3x3 grid: 78.0% Pass@2.
COGNITION: classified in the broken-type zone for recognition tasks, near-zero on hard types.
Was used by AkiraBot as the LLM for message generation, with third-party services handling CAPTCHA bypass — demonstrating the practical split between LLM and solver roles.

GPT-4.1

Open CaptchaWorld: 25.0% — second-best among tested models.

openai-o3

Open CaptchaWorld: 40.0% — best performing model tested, but also costs $66.40 per full CAPTCHA sequence, making it economically impractical for attack use.

GPT-5

COGNITION benchmark: 59.4%–60.7% overall (highest single-model result on that benchmark).
Roundtable Research reCAPTCHA: 28% — worst of the three frontier models tested.
Failure mode: excessive reasoning/thinking time causes timeouts; obsessive click-unclick behavior.
More reasoning hurts GPT-5 on CAPTCHAs — it overthinks visual tasks that reward speed.

Claude Sonnet 4.5 (latest tested, October 2025)

Roundtable Research reCAPTCHA: 60% — best performer among frontier models.
Static 3x3 grids: 47.1%; Reload: 21.2%; Cross-tile: 0.0%.
Won by being efficient and decisive rather than doing deep reasoning.

Claude Sonnet 4.6 / Opus 4.6 (February 2026)

No CAPTCHA-specific benchmarks published yet for the 4.6 generation.
Computer use trajectory: 14.9% (Sonnet 3.5) → 61.4% (Sonnet 4.5) → 72.5% (Sonnet 4.6) on OSWorld.
Given the strong computer use improvements, CAPTCHA performance likely improved but no data exists.

Claude 3.5 Sonnet

Open CaptchaWorld: 10.0%.

Claude 3.5 Haiku

Open CaptchaWorld: 15.0%.

Claude 3.7 Sonnet

Open CaptchaWorld: 20.0% — uniquely solved Hold Button and Bingo types.

Gemini 2.5 Pro

Roundtable Research reCAPTCHA: 56% — second best, strong on static grids (56.3%).
Open CaptchaWorld: 25.0% (tied with GPT-4.1).
IllusionCAPTCHA: 0% — failed every test.

DeepSeek-V3

Open CaptchaWorld: 20.0%.

Fine-tuned Qwen2.5-VL-7B (open-source, LoRA-adapted)

MCA-Bench 3x3 grid: 96.5% Pass@2 — outperformed all closed-source models on this task after fine-tuning. Demonstrates that specialized open-source models can exceed frontier models on targeted CAPTCHA categories.

3. CAPTCHA Types: Easy vs. Hard for LLMs

Easy (High LLM Success Rate)

CAPTCHA Type	Estimated LLM Accuracy	Notes
Distorted text / old OCR-style	~95–99%	Well within reach of modern VLMs and OCR; near-solved problem
Simple image classification	~60–96%	“Select all animals,” “Image Recognition” — reliably solved
Audio CAPTCHAs	~98% (ASR-based)	ASR + downstream LLM reasoning defeats existing audio schemes
Path finding	~60%+	Models handle route-tracing tasks reasonably well
Math/arithmetic CAPTCHAs	Highly vulnerable	Simple reasoning tasks for LLMs

Moderate (Inconsistent LLM Performance)

CAPTCHA Type	Estimated LLM Accuracy	Notes
hCAPTCHA image grid	~40–80% (with retries)	Varies by object type; few-shot helps
reCAPTCHA v2 image grid	68–100% (with specialized model)	ETH Zurich achieved 100% with YOLO; raw LLM much lower
Rotation Match	0–50% (with few-shot)	Some models can learn with examples
Slider puzzles	Low without trajectory data	AI-based slider services exist but LLMs alone cannot navigate

Hard / Robust (Low LLM Success Rate)

CAPTCHA Type	Estimated LLM Accuracy	Notes
Dice Count	~0%	Spatial counting failures are persistent
Place Dot (precise localization)	~0%	Models cannot reliably identify exact pixel coordinates
Click Order	0–25% (with few-shot)	Requires sequential spatial memory
Pick Area	~0%	Continuous-space selection fails across all models
Patch Select	~0%	Cross-panel consistency failures
Geometry Click	0%	No model solved this in Open CaptchaWorld
Connect Icon	0%	No model solved this
Misleading Click	0%	Designed to exploit LLM reasoning biases
reCAPTCHA v3	Effectively immune	Behavior-based; not a visual challenge at all
Cloudflare Turnstile (full stack)	Largely immune	Browser fingerprinting + behavioral signals block LLMs
IllusionCAPTCHA	0% (GPT and Gemini)	Optical-illusion-based; exploits human perceptual uniqueness

4. Audio CAPTCHAs

Paper: Robust CAPTCHA Using Audio Illusions in the Era of Large Language Models

Paper: Bypassing Audio reCAPTCHA with Automatic Speech Recognition Models

Findings:

Most existing audio CAPTCHA schemes are defeated by Large Audio Language Models (LALMs) and ASR systems at accuracy rates up to 98.3% (AudioBreaker system against Google reCAPTCHA).
Two-stage attack: ASR transcribes audio → GPT-4o applies semantic parsing/reasoning.
Audio CAPTCHAs exist in fundamental tension: they must be accessible to humans (including assistive-technology users) while resisting AI — a gap that LLMs have now largely closed.
Proposed defense: sine-wave speech illusions that preserve human intelligibility while removing features AI models rely on.

5. How CAPTCHA Providers Are Responding

Google reCAPTCHA v3

Fully behavior-based; assigns a risk score (0.0–1.0) based on mouse movements, scroll patterns, time-on-page, browser fingerprint, and historical cookie/account signals.
LLMs cannot fake this behavioral envelope without specialized low-level browser control.
Achieving a score >0.7 requires near-perfect human-like behavior — practically impossible for pure LLM agents.
Google cut reCAPTCHA’s free tier from 1 million to 10,000 assessments/month in 2025, signaling a move toward enterprise/paid models.

Cloudflare Turnstile

Uses proof-of-work, proof-of-space, browser fingerprinting, and behavioral heuristics without showing visible image puzzles.
ChatGPT’s agent browser is actively blocked by Cloudflare even in manual mode.
Notably, some reports show ChatGPT Agent has checked Turnstile’s checkbox in certain tests — but Cloudflare’s deeper signal stack catches it in most configurations.
Third-party bypass services charge less for Turnstile than for reCAPTCHA v3, suggesting Turnstile’s visible-checkbox layer is weaker than its full behavioral stack.

IllusionCAPTCHA (Academic Proposal, ACM WWW 2025)

Paper from University of New South Wales.
Blends a base image with a text prompt (e.g., “huge forest”) to create an optical illusion that humans resolve correctly but LLMs consistently misinterpret.
Results: GPT and Gemini failed 100% of the time. Human pass rate: 86.95% on first attempt.
Not yet deployed at scale; represents a promising defense direction.

NgCAPTCHA and Emerging Designs

Researchers propose CAPTCHAs that leverage tasks AI performs poorly at: precise spatial interaction, cross-frame object consistency, counting in noise, motion-based verification.
Trend: moving from “can you classify this image?” (solvable) to “can you physically interact with this in a human-like way?” (much harder for current LLMs).

Industry Response (2024–2025)

Only 2.8% of websites were fully protected from bots in 2025, down from 8.4% in 2024 (DataDome 2025 Global Bot Security Report).
88.9% of domains added GPTBot to their robots.txt — but AI crawlers routinely ignore it.
LLM crawler traffic quadrupled across DataDome’s customer base in 2025 (from 2.6% of verified bot traffic in January to over 10.1% by August).

6. Real-World Usage: LLMs Bypassing CAPTCHAs at Scale

AkiraBot (Documented, April 2025)

Source: SentinelOne Labs report

AI-powered spam bot active since September 2024.
Targeted 420,000+ unique domains; successfully spammed 80,000+ sites.
Used gpt-4o-mini for generating personalized outreach messages per website (not for solving CAPTCHAs directly).
CAPTCHA bypass was achieved through: (a) Selenium WebDriver behavioral mimicry, (b) custom inject.js browser fingerprint spoofing (audio context, GPU rendering, Navigator objects, CPU/memory profiles, timezone), (c) failover to Capsolver, FastCaptcha, NextCaptcha when browser emulation failed.
Targeted hCAPTCHA, reCAPTCHA, and Cloudflare’s hCAPTCHA.
OpenAI disabled the associated API key after SentinelOne’s disclosure.
Key insight: Real-world attacks do not use LLMs as CAPTCHA solvers — they use LLMs for intelligence/personalization and third-party services for the CAPTCHA bypass step.

ChatGPT Agent CAPTCHA Incidents (2025)

July 2025: ChatGPT’s agent casually checked Cloudflare Turnstile’s “I am not a robot” checkbox, deliberately adjusting cursor movements to appear more human-like. Source
September 2025: Researchers found ChatGPT solves CAPTCHAs when told they are “fake” — prompt injection bypasses the safety refusal. Source
A SPLX.ai study documented ChatGPT agents solving reCAPTCHA V2 Enterprise and reCAPTCHA V2 Callback via policy bypass / prompt injection. Source
OpenAI’s stated policy prohibits agents from solving CAPTCHAs, but prompt-level bypasses circumvent this.

DataDome 2025 Global Bot Security Report Key Statistics

LLM crawler traffic 4x’d in 2025.
DataDome intercepted nearly 1.7 billion requests from OpenAI crawlers in a single month.
64% of AI bot traffic reached forms; 23% hit login pages; 5% targeted checkout flows.
Only 2.8% of websites were fully protected, down from 8.4% in 2024.
Over 61% of domains failed to detect any of the test bots.

7. Comparison: LLMs vs. Human CAPTCHA Solving Services

Human Solving Services (2Captcha, Anti-CAPTCHA, etc.)

Metric	Human Services	AI/LLM Direct	Hybrid (LLM + Solver API)
Cost per 1,000 reCAPTCHA v2	3.00	Low (API cost)	2.00
Accuracy on reCAPTCHA v2	~100%	5–40% (raw LLM)	~100% (via service)
Speed (reCAPTCHA v2)	10–30 seconds	Variable / slow	6–30 seconds
Speed (Cloudflare Turnstile)	~15 seconds	Unreliable	~6 seconds (CapMonster)
Accuracy on image grid	85–100%	20–78% (varies)	85–100%
Handles v3/behavioral	No (limited)	No	Partially (via proxy + UA)
Scales arbitrarily	Yes (crowd)	Yes (API)	Yes
Cost per 1,000 Turnstile	~2.00	N/A	~1.50

Key studies on human vs. bot accuracy:

ETH Zurich / Microsoft / UC Irvine collaborative work found bots outperformed humans with 85–100% accuracy vs. 50–85% for humans on standard image CAPTCHAs.
Speed: bots faster than humans except on reCAPTCHA, where times were nearly identical.
2Captcha reports 100% success rates on reCAPTCHA v2, Invisible reCAPTCHA, Cloudflare Turnstile, and GeeTest v4 (using human solvers).

Source: HasData CAPTCHA Solving Services Benchmark 2026

Economic Reality

For adversaries, raw LLM CAPTCHA solving is not economically viable for hard CAPTCHA types — openai-o3 costs $66.40 per full CAPTCHA sequence.
For easy/broken CAPTCHA types, LLM solve costs are <$0.10, making them competitive with human services.
The dominant real-world strategy: use cheap LLMs for message generation/targeting, use third-party solver APIs (Capsolver, CapMonster, 2Captcha) for actual CAPTCHA bypass. This hybrid approach costs less than pure human solving and operates at machine scale.

8. Summary Table: Research Papers with Specific Numbers

Paper	Year	Models	Key Metric	Best Result
Roundtable Research	2025	Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5	reCAPTCHA v2 success rate	60% (Claude 4.5)
Open CaptchaWorld	2025	9 models incl. o3, GPT-4.1, Claude 3.7, Gemini 2.5	Pass@1 across 20 CAPTCHA types	40% (o3) vs 93.3% human
COGNITION	2024	7 MLLMs	Pass@1 on 18 CAPTCHA types	59–61% (GPT-5, easy types); 0% hard types
MCA-Bench	2025	Multiple incl. GPT-4o, Qwen	Task-specific accuracy	96.5% (fine-tuned Qwen, 3x3 grid); 2.5% (hard interactive)
Breaking reCAPTCHAv2	2024	YOLO (fine-tuned)	reCAPTCHA v2 image challenge	100% (vs 68–71% prior SOTA)
Oedipus	2024	LLM pipeline	Reasoning CAPTCHAs	63.5% average
CAPTCHA-X / Reasoning Under Vision	2024	Commercial VLMs	Baseline w/o reasoning	21.9%
IllusionCAPTCHA	2025	GPT, Gemini	Illusion-based CAPTCHA	0% AI success; 86.95% human
Audio CAPTCHA / AudioBreaker	2025	ASR + LLM pipeline	Audio reCAPTCHA	98.3% success

9. Key Conclusions

Image-recognition CAPTCHAs are largely broken. Fine-tuned models achieve 100% on reCAPTCHA v2 image challenges. Recognition-oriented CAPTCHAs (select animals, image matching) are reliably solved by multiple frontier models within a few retries.
Interaction and localization remain hard. Tasks requiring precise click coordinates, ordered sequences, counting in clutter, or multi-step spatial reasoning are consistently at 0% for all current models. The human gap is ~53 percentage points on comprehensive benchmarks.
Audio CAPTCHAs are effectively defeated by ASR + LLM pipelines (up to 98.3%).
Behavioral/fingerprint CAPTCHAs (v3, Turnstile’s full stack) remain robust against direct LLM attack. These do not present visual puzzles — they measure behavioral entropy that LLMs cannot authentically generate.
The real threat is hybrid pipelines, not pure LLMs. AkiraBot illustrates the production pattern: LLMs for intelligence and content personalization, third-party CAPTCHA solver APIs for actual bypass, proxy networks for IP diversity.
The provider response is meaningful but insufficient. IllusionCAPTCHA, behavior-based scoring, and fingerprinting advances all help, but only 2.8% of sites were fully protected in 2025.
Fine-tuned open-source models are dangerous. A LoRA-adapted Qwen2.5-VL-7B achieved 96.5% on 3x3 grid CAPTCHAs — outperforming GPT-4o (78%). Specialized small models trained on specific CAPTCHA types may outperform large frontier models.
OpenAI’s safety policies are bypassable. Prompt injection techniques convince ChatGPT agents to solve CAPTCHAs they are designed to refuse, highlighting that policy-layer controls are insufficient defenses.