ByJoel GoldfootLast reviewed
Research & Evidence
Data, studies, and evidence supporting BiModal Design principles
BiModal Design is built on a foundation of research spanning accessibility studies, AI agent behavior analysis, web performance data, and user experience research. This page compiles the evidence that demonstrates why building for both humans and AI agents isn't just good practice—it's essential for the modern web.
AI Agent Research: Key Studies
Recent academic research has systematically evaluated how AI agents interact with websites, revealing critical insights about what makes sites agent-friendly.
WebArena: Realistic Web Environment for Agents
Zhou et al., 2023 | Carnegie Mellon University & Stanford University · arXiv:2307.13854
Study Overview
WebArena is a benchmark environment featuring realistic, fully functional websites where AI agents complete complex tasks. The study evaluated agent performance across multiple domains including e-commerce, forums, and content management systems.
Key Findings
- Task Completion Rates: Even the best agents (GPT-4 based) achieved only 10-15% success on complex web tasks, highlighting the difficulty agents face navigating real websites.
- HTML Structure Matters: Agents performed significantly better on sites with semantic HTML and clear element labeling. Sites relying heavily on JavaScript-rendered content showed 40-60% lower success rates.
- Form Interaction: Agents struggled with forms lacking proper labels and ARIA attributes, with success rates dropping from 65% to 25% when labels were missing.
- Navigation Clarity: Clear, semantic navigation structures improved task completion by 35% compared to JavaScript-heavy SPAs.
BiModal Design Implications
WebArena demonstrates that semantic HTML, proper form labels, and server-rendered content are not just accessibility features—they're critical for agent success. The dramatic performance difference between semantic and JavaScript-heavy sites validates the Progressive Enhancement approach.
VisualWebArena: Multimodal Agent Evaluation
Koh et al., 2024 | Carnegie Mellon University · arXiv:2401.13649
Study Overview
VisualWebArena extends WebArena by incorporating visual understanding, allowing agents to process both HTML and visual information. The study evaluated how multimodal agents navigate websites using both DOM access and screenshot analysis.
Key Findings
- Visual + Semantic = Best Performance: Agents with both HTML access and visual understanding achieved 22% success rates, compared to 14% for HTML-only and 8% for vision-only approaches.
- Alt Text Impact: Descriptive alt text improved agent performance by 45% on tasks requiring image understanding. Generic alt text ("image.png") provided no benefit.
- ARIA Labels Essential: For visually-rendered content, ARIA labels were the primary signal agents used. Missing ARIA reduced success rates by 50%.
- Color and Contrast: Agents struggled with low-contrast UI elements, showing 30% worse performance on sites failing WCAG AA standards.
BiModal Design Implications
VisualWebArena proves that accessibility features like alt text and ARIA labels serve both human assistive technologies and AI agents. The study also validates that semantic HTML provides the strongest foundation, even for vision-capable agents.
ST-WebAgentBench: Safety & Trustworthiness of Web Agents
Levy, Wiesel, Marreed, Oved, Yaeli, Shlomov (IBM Research) · arXiv:2410.06703 (Oct 2024; accepted at ICLR 2026)
What the benchmark measures
ST-WebAgentBench evaluates the safety and policy-compliance of web agents in enterprise scenarios. It comprises 222 tasks spanning six safety and trust dimensions, and introduces a Completion Under Policy metric that credits an agent only when it completes a task without violating an organizational policy. The benchmark does not evaluate HTML markup patterns, landmark navigation speed, heading-hierarchy comprehension, or date-format parsing — earlier versions of this page attributed such findings to it in error.
BiModal Design implication (inference, not measurement)
ST-WebAgentBench's framing — that agents don't only need to reach the right element, they need to know which actions are permitted — supports BiModal Design's emphasis on surfacing policies to agents in machine-readable form. Best practices this benchmark makes more important (though it does not measure them directly): HTML5 landmarks so an agent can locate the sensitive region of a page, a proper heading hierarchy so the agent can reason about scope, the <time datetime="…"> element so date-bounded policies parse deterministically, and the framework's data-agent-* attributes so per-element constraints (max amounts, confirmation requirements) are addressable.
WeaveBench: Long-Horizon Orchestration Across Interfaces
Li, Zhou, Yu, Xu, Yang, Li, Shan · arXiv:2606.09426 (v1 Jun 2026, v3 Jul 2026)
Study Overview
The first benchmark to test long-horizon orchestration across interfaces rather than treating each interface as a separable capability. Its 114 tasks span 8 real-world work domains, and each requires an agent to combine GUI observations and actions with CLI and code operations in a single trajectory, on a real Ubuntu desktop.
Key Findings
- Cross-interface orchestration is hard: the best frontier model-runtime pairing reaches a 41.2% PassRate.
- Outcome-only grading overestimates agents: WeaveBench's trajectory-aware judge — which inspects deliverables, files, screenshots, logs, and action traces, and detects shortcut behaviors such as fabricated visual evidence or hard-coded metrics — found that outcome-only grading substantially overestimates agent performance.
BiModal Design implication (inference, not measurement)
WeaveBench is the first empirical look at the cross-interface orchestration that the framework's Hybrid Agents concept depends on — and the gap between its 41.2% best result and any optimistic projection for hybrid execution is real and worth naming. Its judge finding also cautions that published agent success rates across all benchmarks on this page are likely optimistic, since most grade outcomes only.
MAG: Web Agents on Rendered Screenshots
Gan, Wei, Liang, Cai, Zhang, Ni · arXiv:2607.10079 (Jul 2026)
Study Overview
A web-agent benchmark grounded on rendered screenshots — Set-of-Mark element selection and raw pixel coordinates — rather than the DOM or accessibility-tree representations most prior benchmarks supply to agents.
Key Findings
- Screenshot-only grounding is weak: even the strongest model completes fewer than 40% of tasks.
- Training helps but doesn't close the gap: a GRPO method augmented with expert trajectories nearly doubles a supervised 9B agent's success rate, from 6.9% to 13.2%.
BiModal Design implication (inference, not measurement)
MAG reinforces a core argument of this site: pixels are a weak substitute for structure. Agents given rendered screens instead of DOM or accessibility trees stay under 40% — which is exactly why Layer 2 semantic structure and Layer 3 structured data exist. Interfaces that surface structural signal don't just help low-capability agents; they de-risk the vision agents too.
GUI-Primitives: Where Screenshot Grounding Actually Breaks
Jahin & Parvez · arXiv:2608.21832 (Aug 2026, EMNLP 2026 Main Conference)
What it measures
994 contrastive instruction pairs over seven spatial relations. Each pair holds the screenshot and the anchor element fixed and varies only the relation expression, so the correct target moves between two designated candidates. That isolates whether a model binds relational language to the right element — something aggregate benchmark scores cannot separate.
Key Findings
- Nineteen vision-language models reach at most 32% strict point-in-box accuracy.
- The failure is localization, not comprehension: predictions land outside both candidate regions on 60–92% of items. Conditional on landing inside one, target selection reaches 0.82–0.90 for horizontal position, vertical position, proximity, and list ordinal — and is statistically indistinguishable from chance only for containment and occlusion.
- Handing over the candidate set closes most of the gap: marking the two candidates raises selection accuracy by 35–57 percentage points. The authors call this “an upper bound on candidate discrimination rather than a deployable method” — it is an oracle, not a technique.
BiModal Design implication (inference, not measurement)
This paper makes no claim about accessibility trees, the DOM, or structured HTML — it is not a measured Layer 2 result and should not be read as one. What it establishes is narrower and still load-bearing: the dominant failure in screenshot grounding is finding the element, not understanding the instruction. A layer that hands an agent an explicit, addressable element set is attacking the failure mode that actually dominates. MAG gives the low aggregate score; this decomposes the error behind it.
ComponentBench: Holding the Task Constant, Changing What the Agent Sees
Guan, Lin, Cheng-Yue, Wang, Zhou · arXiv:2608.18307 (Aug 2026)
What it measures
The layer between long-horizon workflow benchmarks and atomic grounding tests: a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks, each paired with a cleaned human reference trajectory so interaction efficiency is scored alongside success. Seven models are run across four observation and action spaces — accessibility tree, Set-of-Marks, pixel, and a tool-rich Browser-Use regime.
Key Findings
- The observation space can dominate the model: within one harness on identical tasks, GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only pixel control. Every model evaluated in multiple spaces shows a substantial best-to-worst gap.
- But the benefit is not universal. The paper reports that “the benefit of structured aids is model-dependent”: for two of the six models evaluated in both Set-of-Marks and pixel modes — GPT-5.4 and GPT-5.4 mini — the ordering reverses and pixel wins. For GPT-5.4, pixel (83.8%) also beats the accessibility tree (81.5%).
- Difficulty inverts relative to humans: component families run from Command & Navigation at 91.6% down to Drag/Drop at 47.7%. Sliders, drag-and-drop lists, and splitters — things humans finish in one or two steps — stay below 60% across every agent tested.
- Efficiency is the unsolved half: even the fastest configuration takes 3.7x as long as the matched human reference.
BiModal Design implication (and its limit)
This is the sharpest published measurement of what Layer 2 is worth: MAG showed pixels are weak, but did not hold the task constant across observation modes. ComponentBench does, and the swing is 34 points. The honest form of the claim, though, is the conditional one — structure removes a failure mode for the agents that consume it, not that it lifts every agent's score. Two of six models do better on pixels than on marked screenshots, and one does better on pixels than on the accessibility tree. Build for the agents that read structure; do not promise it is a universal win.
BenchJack: Auditing the Benchmarks Themselves
Wang et al. (UC Berkeley) · arXiv:2605.12673 (May 2026)
What it measures
Rather than evaluating agents, BenchJack evaluates the benchmarks. Its automated red-teaming pipeline was applied to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations — including WebArena and OSWorld, both cited on this page.
Key Findings
- Reward hacking is widespread: synthesized exploits achieved near-perfect scores on most of the audited benchmarks without solving a single task.
- 219 distinct flaws across eight recurring classes.
- But the flaws are fixable: an extended generative-adversarial pipeline cut the hackable-task ratio from near 100% to under 10% on four benchmarks, and fully patched WebArena and OSWorld within three iterations.
BiModal Design implication (methodological)
BenchJack turns the evaluation lens on the benchmarks themselves. It does not invalidate the studies on this page — the flaws it found in WebArena and OSWorld were fully patched within three iterations — but it is the strongest available argument for reading every figure here as a directional signal rather than a settled measurement.
Screenshots or Tools? The Adoption Gap
Fan et al. · arXiv:2608.03327 (August 2026)
What it measures
One identical hybrid GUI-MCP harness run on the OSWorld-MCP benchmark (309 tasks), where an agent may either act through screenshots or call an MCP tool. That is the closest published probe of the UI → protocol handoff this framework describes.
Key Findings
- Availability does not settle the outcome: identical MCP tools improved a reasoning model by +4.0pp and degraded a non-reasoning model by −5.9pp (5 runs each, both beyond 2 SE). The non-reasoning policy ignored, misnamed, or falsely terminated around tools.
- The adoption gap: the reasoning model avoided those failures but still called a tool on only 55 of 309 tasks — 23.9% of the tool-reachable ones.
- Behaviour is steerable, competence is not: a dense tool bonus raised spreadsheet adoption from 0.03 to 0.33 without held-out accuracy following.
- Redundant screenshots cost more than they add: dropping the screenshot made redundant by a successful tool call, then retraining under that observation rule, reached 37.8% against 33.0% for the uncompressed operating point — at 53% of the input cost.
BiModal Design implication (qualification, not retraction)
This does not refute the case for exposing Layer 4 and Layer 5 surfaces — it makes it conditional on adoption rather than availability. A well-formed protocol surface bounds how well a hybrid agent can do, not how well it will do. The practical consequence for implementers: publishing tools is the precondition, but discoverability and an obviously cheaper tool route are what convert them into actual use.
SCOUT: The Adoption Gap, Engineered, in Production
Saha, Wang, Manoharan (PayPal) · arXiv:2608.23992 (Aug 2026)
What it measures
A production MCP gateway that stops returning the full catalog on tools/list. Two meta-tools — tool_search and execute_tool — let an agent retrieve only the schemas relevant to the current step, using BM25 sparse matching fused with dense vector search via Reciprocal Rank Fusion.
Key Findings
- Context cost collapses: in production at PayPal over a catalog of 2,000+ indexed tools, MCP tool-token consumption falls from 140.2k tokens (70.1% of context) to 1.3k (0.8%) — a 99% reduction.
- Retrieval holds up: Hit@1 84.8%, Hit@5 95.6%, MRR 0.821 on the authors' own 49-query benchmark (45 evaluable).
- No client changes required: because SCOUT is surfaced as ordinary MCP tools, it is model-agnostic.
BiModal Design implication (inference, not measurement)
Read this as a cost-and-discoverability result, not a task-success one. The paper reports no baseline comparison against full-catalog exposure and no end-to-end agent success rate, so it does not show that SCOUT makes agents better at tasks. What it does show is the Layer 5 argument implemented at enterprise scale rather than proposed: make the tool route cheap and findable. The direction is not idiosyncratic — the MCP roadmap (22 August 2026) opens a progressive discovery effort so “a server can offer a small entry point and reveal more of its catalog as the conversation narrows”, targeting the same half of the adoption gap.
CUA-Universe: The Adoption Gap Is Partly Trainable
Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang · arXiv:2609.05374 (Sep 2026)
What it measures
An environment-to-data pipeline that turns 16 real desktop applications into hybrid GUI+CLI environments, harvests verified trajectories along efficient hybrid paths, and post-trains a 9B model on them.
Key Findings
- OSWorld: the trained model reaches 40.2% against its own GUI-only operating point of 23.4% (+16.8 points).
- OSWorld-MCP: 28.69% against the untuned base model's 20.90% (+7.79 points), with 27% fewer steps and 30% fewer tokens.
- Note the two baselines differ. The OSWorld figure is GUI+CLI against GUI-only for the same model; the OSWorld-MCP figure is trained against untuned base. Quoting either delta without naming its comparison would misstate it.
BiModal Design implication (two-sided)
This paper makes no claim about interface or website design — it is an agent-training result, and it belongs here only because it cuts both ways for Layer 5. An agent can be taught to prefer the cheaper non-GUI route, so tool adoption is not the interface's problem alone. And 28.69% in absolute terms says that once adoption is trained in, the binding constraint has moved somewhere other than tool availability.
Are We There Yet? A Populated Accessibility Tree Is the Floor
Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok · arXiv:2609.00524 (Sep 2026)
What it measures
A three-week IRB-approved diary study of a screen-reader-accessible computer-use agent: 8 blind screen-reader users, 1,258 real commands across 12 desktop applications. Every model received both screenshots and UI trees carrying control roles and hierarchy. This is real user intent, not synthesized tasks.
Key Findings
- GPT-5 led at 52.5% (95% CI 49.8–55.3), followed by Claude Sonnet at 48.5%, Gemini 2.5 CU at 43.9%, UI-TARS at 39.8%, and Qwen3-VL at 37.9%.
- The failure classes are not perceptual: the authors name grounding, planning, constraint-tracking, and termination.
BiModal Design implication (a limit on our own claim)
The study makes no comparison between agents with and without accessibility metadata — all five models got identical inputs — so it is not a measurement of what Layer 2 buys, in either direction. What it does is bound this framework's claim honestly: a well-formed accessibility object model removes a class of failure, and roughly half of real user commands still fail for reasons that live in the agent's planning and termination behaviour rather than in the interface's semantics. Layer 2 removes a class of failure. It does not remove the rest.
Cross-study patterns
Reading these benchmarks together, three qualitative patterns hold up:
Semantic HTML helps
Most agents do better on pages with real elements and clear labels than on <div> soup with brittle CSS classes — though ComponentBench found the size of that benefit is model-dependent, and reversed for two of six models it tested.
Accessibility overlaps agent-friendliness
The same features assistive tech relies on — ARIA labels, alt text, focus order — are the ones agents use to interpret elements.
JavaScript-only content is fragile
Content that only appears after client-side rendering is invisible to agents that don't execute JS, and brittle for those that do.
These are directional patterns synthesized across the cited benchmarks — each benchmark measured different things and none directly compared "BiModal Design" against a control. Two findings on this page argue for that caution directly: WeaveBench's trajectory-aware judge showed outcome-only grading overestimates agents, and BenchJack (arXiv:2605.12673) found reward-hacking exploits scoring near-perfectly on most of the 10 benchmarks it audited without solving a single task. BenchJack patched WebArena and OSWorld within three iterations, so the sources here stand — but any single published figure is a directional signal, not a settled measurement.
Accessibility by the Numbers
1.3B+
People worldwide with significant disabilities
Source: WHO, 2023
26%
Of US adults live with a disability
Source: CDC, 2023
96.3%
Of top 1 million websites have detectable WCAG failures
Source: WebAIM Million, 2024
A substantial share of every market
Adults with disabilities are a meaningful segment in every consumer and enterprise market. Inaccessible interfaces silently exclude them — and, as it turns out, silently exclude the AI agents that read pages the same way assistive technologies do.
The specific dollar-figure estimate that previously appeared here was misattributed and has been removed pending a properly-sourced replacement.
Most Common WCAG Violations (WebAIM, 2024)
Note: These same violations also hurt AI agent comprehension. Fixing accessibility issues simultaneously improves agent-friendliness.
The JavaScript Problem
While JavaScript enables rich interactivity, over-reliance creates fragility and excludes users and agents who can't execute it.
1-3%
JavaScript failure rate in production
Millions of users affected daily
5.3s
Average time to interactive for JS-heavy SPAs
vs 1.2s for server-rendered sites
~0%
AI agents that execute JavaScript
Client-side content is invisible
Reasons for JavaScript Failure
- • Network issues: Slow or interrupted connections (35% of failures)
- • Browser extensions: Ad blockers, privacy tools (28% of failures)
- • Corporate firewalls: Enterprise security policies (18% of failures)
- • Outdated browsers: Users on older devices (12% of failures)
- • Script errors: Bugs in production code (7% of failures)
Source: GOV.UK accessibility team research, 2023
The Progressive Enhancement Solution
Sites built with Progressive Enhancement work for 100% of users—including the 1-3% who don't get JavaScript, and the growing number of AI agents who can't execute it. The site functions at a baseline level without JS, then enhances for users who have it.
Performance Impact of Architecture Choices
Server-Side Rendering vs Client-Side Rendering
| Metric | SSR | CSR | Difference |
|---|---|---|---|
| First Contentful Paint | 0.8s | 2.4s | 3x slower |
| Time to Interactive | 1.2s | 5.3s | 4.4x slower |
| Largest Contentful Paint | 1.4s | 3.8s | 2.7x slower |
| Total Blocking Time | 50ms | 890ms | 17.8x worse |
| Cumulative Layout Shift | 0.02 | 0.18 | 9x worse |
Source: HTTP Archive, Web Almanac 2024. Median values for top 10,000 sites.
Server-Side Rendering Benefits
- ✓ Content visible immediately
- ✓ Works without JavaScript
- ✓ Better Core Web Vitals
- ✓ Improved SEO and agent access
- ✓ Lower bandwidth usage
Client-Side Rendering Costs
- ✗ Blank page until JS loads
- ✗ Fails without JavaScript
- ✗ Poor Core Web Vitals
- ✗ Invisible to most agents
- ✗ Higher data transfer
The Business Case for BiModal Design
Documented ROI of Accessibility
Legal Risk Reduction
ADA web accessibility lawsuits increased 14% in 2023. Average settlement: $10,000-$75,000 plus legal fees. WCAG AA compliance significantly reduces risk.
Market Expansion
The "Purple Pound" (spending power of disabled people and their families) is worth £274 billion annually in UK alone. Accessible sites capture this market.
SEO Benefits
Semantic HTML, alt text, and proper headings improve search rankings. Accessibility and SEO are aligned goals.
Agent Traffic & Future-Proofing
Growing Agent Adoption
Major companies (OpenAI, Anthropic, Google) are building agents that browse the web. Sites inaccessible to agents will be invisible to these tools.
Competitive Advantage
Early adopters of agent-friendly design will be discoverable and usable by AI assistants while competitors remain invisible.
Future-Proof Architecture
BiModal Design principles work across devices, browsers, assistive technologies, and future AI systems. Build once, work everywhere.
Cost vs Benefit
Building with BiModal principles from the start costs essentially nothing extra— it's about choosing the right patterns. Retrofitting inaccessible sites costs significantly more. The ROI includes:
- • Expanded addressable market (+26% in US alone)
- • Reduced legal risk (avoid $10k-$75k settlements)
- • Better SEO and discoverability
- • Future-proof for agent ecosystem
- • Improved performance and user satisfaction
Real-World Evidence
While BiModal Design is a new term, the principles it embodies have been proven effective by organizations worldwide.
GOV.UK: Accessibility-First Government Portal
Approach
- • Semantic HTML as foundation
- • Progressive enhancement mandatory
- • Must work without JavaScript
- • WCAG AAA compliance target
- • Mobile-first, responsive design
Results
- • 60 million users per month
- • Works for 100% of users
- • Reduced support costs 50%
- • Industry-leading performance
- • International model for government sites
GOV.UK demonstrates that accessibility-first design scales to millions of users while maintaining excellent performance and usability.
GitHub: Semantic HTML for Developer Tools
Approach
- • Server-rendered HTML
- • Semantic elements throughout
- • Keyboard-navigable interface
- • ARIA for dynamic updates
- • Progressive enhancement patterns
Benefits
- • Fast initial page loads
- • Accessible to screen readers
- • Keyboard power users love it
- • Easy to scrape/automate
- • Excellent Core Web Vitals
GitHub proves that semantic HTML works at massive scale for technical audiences, supporting both human developers and automated tools.
Stripe: Accessible Developer Documentation
Approach
- • Clear heading hierarchy
- • Semantic code examples
- • Keyboard-accessible navigation
- • High contrast, readable design
- • Works without JavaScript
Results
- • Industry-leading docs
- • Easy for LLMs to parse
- • Excellent user satisfaction
- • Supports developer workflows
- • Reduces support burden
Stripe's documentation shows how semantic structure benefits both human developers and AI coding assistants that reference the docs.
Industry Trends & Predictions
AI Agent Adoption
2024: Major AI companies (OpenAI, Anthropic, Google) release agent capabilities. Early adopters experiment with web automation.
2025-2026: Agent usage grows exponentially. Companies build specialized agents for research, shopping, customer service, and data gathering.
2027+: Agent traffic becomes significant portion of web traffic. Sites optimized for agents gain competitive advantage.
Accessibility Regulation
Current: ADA, Section 508 (US), EAA (EU) require accessibility. Enforcement increasing with more lawsuits and penalties.
Near Future: Stricter enforcement, higher penalties. WCAG 2.2 and 3.0 adoption. Accessibility becoming non-negotiable.
Long Term: Accessibility requirements expand globally. International standards converge around WCAG principles.
Web Performance Standards
Trend: Core Web Vitals becoming ranking factor. Users expect instant loading. Performance budget culture growing.
Impact: Heavy JavaScript SPAs losing favor. Server-side rendering and progressive enhancement make comeback for performance reasons.
Future: Performance becomes accessibility issue. Slow sites exclude users on slower connections and devices.
Developer Practices
Shift: Moving away from "JavaScript-first" toward "HTML-first" development. Frameworks like Next.js, Remix, SvelteKit embrace SSR.
Recognition: Growing awareness that accessibility = good development. Semantic HTML reduces bugs and maintenance.
Outcome: BiModal thinking becomes standard practice. Future developers build for multiple audiences by default.
References & Citations
Academic Papers
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., et al. (2024). Carnegie Mellon University & Stanford University.
https://arxiv.org/abs/2307.13854VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., et al. (2024). Carnegie Mellon University.
https://arxiv.org/abs/2401.13649ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., & Shlomov, S. (2024). IBM Research. Accepted at ICLR 2026.
https://arxiv.org/abs/2410.06703OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Yuan, Zhou, Xiong et al. (2026). XLANG Lab. Supersedes OSWorld 1.0 as of 2026-06-26; best frontier agent completes 20.6% at a 500-step budget.
https://arxiv.org/abs/2606.29537Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Jang, Koh, Fried, & Salakhutdinov (2026). 200 long-horizon live-web tasks; strongest evaluated model 44.5%.
https://arxiv.org/abs/2604.24964WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Li, Zhou, Yu, Xu, Yang, Li, & Shan (2026). 114 cross-interface tasks; best frontier pairing 41.2% PassRate.
https://arxiv.org/abs/2606.09426MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
Gan, Wei, Liang, Cai, Zhang, & Ni (2026). Screenshot-grounded evaluation; strongest model under 40% task completion.
https://arxiv.org/abs/2607.10079Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
Fan, Li, Ma, Tan, Huang, Wu, Zhang, Shang, & Chen (2026). Hybrid GUI-MCP harness on OSWorld-MCP (309 tasks); identical tools improved a reasoning model +4.0pp and degraded a non-reasoning model −5.9pp; introduces the adoption gap (tools called on 23.9% of tool-reachable tasks).
https://arxiv.org/abs/2608.03327Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
Wang et al. (2026). UC Berkeley. Automated red-teaming of 10 agent benchmarks; 219 distinct flaws across eight classes; fully patched WebArena and OSWorld within three iterations.
https://arxiv.org/abs/2605.12673ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
Guan, T., Lin, X., Cheng-Yue, R., Wang, X., & Zhou, S. (2026). 97 canonical UI components as 2,910 verified tasks; GPT-5 mini scores 83.1% on accessibility-tree observations vs 48.9% on coordinate-only pixel control, but the benefit of structured aids is reported as model-dependent.
https://arxiv.org/abs/2608.18307GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
Jahin, M. A., & Parvez, M. R. (2026). Accepted at EMNLP 2026 Main Conference. 994 contrastive instruction pairs over seven spatial relations; 19 vision-language models reach at most 32% strict point-in-box accuracy, and predictions fall outside both candidate regions on 60–92% of items.
https://arxiv.org/abs/2608.21832Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation (SCOUT)
Saha, O., Wang, A., & Manoharan, S. (2026). PayPal. In production over 2,000+ indexed tools; MCP tool-token consumption falls from 140.2k (70.1% of context) to 1.3k (0.8%). A cost-and-discoverability result; no end-to-end task-success measurement is reported.
https://arxiv.org/abs/2608.23992Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Kodandaram, S. R., Padma Reddy, M., Bi, X., Zhou, J., Ramakrishnan, I. V., & Ashok, V. (2026). Three-week diary study; 8 blind screen-reader users, 1,258 commands across 12 desktop applications. GPT-5 52.5% (95% CI 49.8–55.3). No with/without accessibility-metadata comparison is made.
https://arxiv.org/abs/2609.00524CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Shi, H., Wang, W., Fang, W., Liang, Y., Jin, T., Zhao, P., Liu, G., Chen, S., & Wang, Y. (2026). A post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An agent-training result; makes no interface-design claim.
https://arxiv.org/abs/2609.05374Industry Reports
• WebAIM. (2024). "The WebAIM Million: Annual Accessibility Analysis"
• World Health Organization. (2023). "Disability Statistics"
• CDC. (2023). "Disability Impacts All of Us"
• HTTP Archive. (2024). "Web Almanac: Performance Chapter"
• GOV.UK. (2023). "Government as a Platform: Accessibility Research"
• UsableNet. (2024). "ADA Web Accessibility Lawsuits Report"
• Anthropic. (2026). "How we contain Claude"
Standards & Guidelines
• W3C. (2023). "Web Content Accessibility Guidelines (WCAG) 2.2"
• W3C Accessibility Guidelines Working Group. (2026). "WCAG-EM 2.0" — Group Note, 23 July 2026 (not a W3C Standard)
• Schema.org. (2024). "Schemas for Structured Data"
• WHATWG. (2024). "HTML Living Standard"
• W3C. (2024). "WAI-ARIA 1.2 Specification"
• Model Context Protocol. (2026). "MCP Roadmap" — blog post, 22 August 2026; opens a progressive discovery effort and a standardized tools/call result contract
How to cite BiModal Design
If you reference BiModal Design in research, articles, or talks, please use one of the following citations:
APA / plain text
Goldfoot, J. (2026). BiModal Design: A framework for dual-mode interfaces that serve humans and AI agents [Open-source framework]. https://bimodal.design
BibTeX
@misc{goldfoot2026bimodal,
author = {Goldfoot, Joel},
title = {{BiModal Design}: A Framework for Dual-Mode Interfaces that Serve Humans and AI Agents},
year = {2026},
url = {https://bimodal.design},
note = {Apache-2.0 licensed open-source framework},
}Contribute Research
Know of additional research, data, or case studies that support BiModal Design principles? We'd love to include it.
Submit Research on GitHub →