The Briefing Desk

Liquid AI released LFM2.5-24B-A2B, an open-weight reasoning model with 24 billion total parameters and about 2 billion active per token, together with a base model, on Hugging Face. It keeps the architecture of LFM2-24B-A2B (40 layers, 64 experts per block with top-4 routing), extends the context window from 32K to 131K tokens, and roughly doubles pre-training to about 33T tokens, followed by agentic reinforcement learning inside harnesses such as OpenClaw, Hermes Agent, Pi and OpenCode. Liquid reports 184 tokens per second on an Apple M5 Max and more than 21,000 output tokens per second at high concurrency on a single NVIDIA B300. It says the model leads or matches Qwen3.6-35B-A3B on five of eight tool-use benchmarks while trailing on knowledge and coding tests such as GPQA Diamond and SWE-Bench Verified; all the figures are Liquid’s own.

Why it matters. Liquid pitches the model at agents that run continuously on local hardware, where a cheap reasoning token matters more than a top benchmark score. Keep in mind that although only 2B parameters are active per token, the whole 24B-parameter model still has to fit in memory. Liquid itself says larger models keep an edge on knowledge-heavy and coding benchmarks, and suggests pairing it with retrieval or web search.

JetBrains released Mellum2.1, a 12B-parameter coding model with a sparse 2.5B-active architecture, under Apache 2.0. The company presents it as an open model aimed at code completion and broader agent tasks. The vendor description does not establish performance on your codebase.

Why it matters. JetBrains says Mellum2.1’s biggest improvement over Mellum2 is in agentic coding, after reinforcement learning across thousands of sandboxed environments, and positions it as a worker or sub-agent inside larger agent systems running on your own infrastructure. Whether a 2.5B-active model can change a real repository without breaking it is the thing to check; JetBrains’ comparisons are its own.

xAI released Grok Voice 3.0, a speech-to-speech model available in its Voice Agent API at $0.08 a minute ($4.80 an hour), the same price as Grok Voice 2.0; agents using grok-voice-latest were switched over automatically. xAI cites Artificial Analysis testing that ranks the model first on its Speech-to-Speech Index with a score of 86.9, and reports 97% on Full Duplex Bench and latency under one second. xAI’s API documentation describes real-time speech-to-speech agents over WebSockets with turn-taking and tool use. The company says the model handles millions of conversations a day in Tesla vehicles and answers about 20,000 Starlink support calls daily. The benchmark figures are as published by xAI; we did not reproduce them.

Why it matters. For voice products, a single speech-to-speech model competes against assembled speech recognition, language reasoning and synthesis pipelines. Developers must still measure interruptions, tool timing, recoveries and control over what is actually said.

OpenAI introduced an academic access plan for eligible researchers to use GPT-Rosalind and its life-sciences workbench. The offering describes tool orchestration across genomics, medicinal chemistry and research workflows with trusted-access approval. Access is restricted to qualifying programs; this is not a public self-service model release.

Why it matters. The story is less about one new score than the institutional packaging around a specialist model. Researchers need provenance, tool permissions and evaluation data before treating AI suggestions as scientific evidence, especially where experimental costs and safety controls are significant.

Google Research made Deep Review publicly available on October 14. The agentic system for checking long technical documents grew out of Google’s Paper Assistant Tool, which gave opt-in feedback to the authors of more than 16,000 manuscripts before submission to STOC, ICML, NeurIPS and ICLR. The pipeline splits a document into segments, gives more compute to dense sections such as proofs, reviews the segments in parallel with full-document context, and then runs a synthesis agent that checks consistency across sections and grounds claims with external tools. Every user gets two free reviews a week, with higher limits and API access through a Gemini API key, and PDFs and plain text are supported at launch. Google says uploaded documents aren’t used for training and are stored for one year only so they can be shown in the Deep Review interface.

Why it matters. Document review systems need more than summaries. The useful outcome is precise claims with page-level support, treatment of contradictory evidence and a clear indication when the document does not answer a question.

IBM released Granite-Switch-4.1-8B and -30B. Each one combines a Granite base model with a general-purpose Decision aLoRA, a safety-focused Guardian aLoRA and the embedded Granite-Speech-5.0-Turbo-CTC speech recognizer. The adapters are Activated LoRAs, so the KV cache from a base-model generation can be reused for later adapter calls, and IBM says the combined model needs about the same VRAM as the base model alone. The models run in vLLM through a plug-in and in llama.cpp, and they can answer Jev-style /v1/systemone decision calls alongside ordinary chat requests. IBM says its Guardian adapter beats Jev 1.13 and other decision models on average on safety-related decision tasks, and that a custom 3B Switch model made 60–100 decisions per second playing Doom against Jev’s reported 10; all of these are IBM’s own measurements.

Why it matters. IBM says composing the adapters onto one base model provides more capabilities with fewer GPUs, because chat, decisions and transcription share the base model’s memory footprint and KV cache. That is relevant to agent pipelines that mix free-text generation with many classification or routing decisions, but whether the savings hold depends on your workload, and IBM’s comparisons against Jev and other decision models are its own.

TypeSafe launched Jev on September 15 as a “System One” model: an application sends a state and typed Choice, Score or Noul (yes/no) questions, and Jev returns probabilities instead of generated text. The weights are closed, and TypeSafe charges $0.042 per million input tokens with output free, within limits of 64K tokens per request and 32K for the state plus any one question. Since Jev 1.14 arrived on October 14, TypeSafe’s docs also describe image input and a Box question type, both flagged as experimental. By October 16, OpenRouter listed 22 decision models from 13 makers.

Cloudflare’s Clef (27B, built on Qwen3.8-27B) and Clef-flash (9B) launched October 1 on Workers AI with Apache-2.0 weights on Hugging Face. Cloudflare calls the API fully Jev-compatible, and on Workers AI it accepts up to four images per request. In Cloudflare’s own tests, Clef scored 94.20 macro-F1 on BANKING77 against Jev’s 79.74, while Jev led on GPQA Diamond, 78.3 to 48.0. OpenAI’s Decisions API, in public beta since October 6, runs only on GPT-6 Luna, accepts text and images, costs $0.10 per million input tokens and uses OpenAI’s own schema, not Jev’s.

For self-hosting, Strands Labs’ 1.9B Strands Decider is Apache-2.0, publishes its training recipe and data inventory, and serves a local /v1/systemone endpoint; the team reports its v1 checkpoint solved 180 of 231 public tasks (77.9%) on the independent JevBench leaderboard. Fastino’s 340M, English-only GLiNER2.5-Decide combines decisions with span and relation extraction on CPU, and Mapika says its quantized Decider models run on CPU through llama.cpp. Most other published scores are vendor-run on different suites, and the widest cross-model comparison comes from Respan, which sells a competing model.

Why it matters. Decision models turn routing, policy checks and triage into typed answers code can threshold, instead of prose it has to parse. The competition gives teams options Jev doesn’t yet offer: open weights to self-host or fine-tune, and image input outside an experimental flag. But “Jev-compatible” describes the request shape, not the behaviour behind it: OpenRouter warns that probabilities differ between models on the same question.

Meta introduced Muse Wonder and Muse Wonder Full for personal and professional agent work. The launch centres on Active Memory, which carries important context across conversations, and Active Focus, which Meta describes as continuing to work toward a goal through blockers and new information. Muse Wonder is available through the Meta Model API and Muse Code, and Muse Wonder Full is available to Muse Code subscribers. The performance comparisons are Meta’s own.

Why it matters. Meta is pitching sustained execution rather than chat: it evaluates focus with FocusBench and long-running software work with SWE-Marathon. Those are the right properties for agent builders to test, but a benchmark chart is less persuasive than a trace showing an agent reaching an accepted outcome.

Anthropic made OpenTelemetry trace export available in public beta for Claude Managed Agents, the Messages API and Claude SDKs. Every agent turn, model call and tool use can be sent as a standard trace, with latency, token counts and IDs, to any tool that accepts an OTLP endpoint. The traces follow the OpenTelemetry GenAI semantic conventions; Managed Agents and the Messages API need a destination added in the Claude Console, while the SDKs export traces automatically.

Why it matters. A failed agent result can come from the model, an upstream tool or a lost retry. Anthropic says agent traces sit beside your logs and metrics in tools such as Grafana Cloud, Honeycomb, Sentry and Splunk, so you can see which service a slow or failed step came from. Our advice: before exporting prompt or response content, check what the traces contain and how long your observability vendor keeps it.

Anthropic says it averaged about 2,500 merged code changes a day in May 2026 and more than 30,000 a day by September, as Claude took on more of its coding. The main-branch history of its monorepo is almost 100 GB, and CI performs on the order of 25 million checkouts a day, which Anthropic says would mean more than 600 PB of git traffic daily if fetched naively; at its peak it ran several hundred internal git mirrors. It built Scion, an in-house version control system designed for always-online clients, in which the server handles history, blame, commits, merges and rebases while clients keep only a bounded local cache, and migrated the monorepo off git in five months. Every source-control query is also exposed through an API, so many CI jobs need no local checkout, and Anthropic credits Google’s Piper and Meta’s Sapling as prior art.

Why it matters. Even if a model can produce patches quickly, code review, dependency resolution and builds can become the bottleneck. Agentic engineering therefore changes the design of source-control infrastructure, not merely editor plugins.

NVIDIA released the complete Run:ai platform as open source on October 13 as part of its DSX AI Factory platform. It comes with four components that can be adopted independently: the KAI Scheduler for GPU-aware Kubernetes scheduling, a GPU Fractioning Operator for CUDA-native GPU sharing, KAI Resource Management for quotas and organizational hierarchy, and Workload Map. The code is on GitHub, and a Community Product with a Helm chart is on NGC; it needs no paid subscription but comes without NVIDIA enterprise support, and downloading it from NGC means accepting NVIDIA’s software license terms. Eight partners, including Crusoe, Mirantis and Nebius, are integrating it into their offerings.

Why it matters. The release moves part of the value from closed cluster consoles into inspectable scheduling primitives. Smaller teams may not need the whole platform, but fairness, quota enforcement and actual GPU occupancy can now be assessed on a controlled Kubernetes workload.

NVIDIA and Siemens describe evaluating an EDA agent on resulting Liberty files rather than loosely scored intermediate reasoning. They report improving a composite score from roughly 55 to 95 percent and reducing agent turns after changes to skills and MCP tool descriptions. Reported scores reflect the authors’ benchmark and specific engineering workload.

Why it matters. The broader lesson applies to any code-producing agent. A strict verifier gives the system an observable finish line and makes improvements traceable to concrete changes, rather than a growing pile of prompt instructions.

AWS’s Sales, Marketing and Global Services operations team has published the evaluation framework behind a multi-agent system, built on Amazon Bedrock AgentCore, that answers sellers’ business questions. Before launch, its Customer Insights Agent reported $410.08 of revenue as “$410.08 Million” and invented an annual recurring revenue estimate while monitoring stayed green. That led the team to define six failure types, including metric swaps, wrong time periods and unauthorized calculations, each with its own custom evaluator. The framework has nine stages: a 15-case sentinel set gates every staging deployment, a 200-question held-out set gates releases with tool calls replayed from frozen data, and batch evaluations score production every 30 minutes, paging on-call only after three consecutive breaches. The team reports that time to detect a quality regression fell from days or weeks to under four hours, and that pure hallucinations dropped from 138 to under 52 per 10,000 responses.

Why it matters. The post is unusually concrete about why uptime and error-rate dashboards miss agent failures: the agent keeps responding, just wrongly. Its patterns carry over to any stack: a small sentinel set on every deploy, frozen tool outputs replayed so scores reflect reasoning rather than tool variance, and alerting across several windows to avoid alarm fatigue. The results are AWS’s own figures for an internal system, and the post also serves as a showcase for AgentCore.

AWS has proposed “Trust in Depth”, an adaptation of defense in depth for autonomous agents, built around four questions. Scope asks what the agent may do before it runs, know asks which identity and authority it carries, bound asks whether its actions stay within limits while it runs, and observe asks whether each run can be reconstructed afterwards. The central argument is that limits should be enforced deterministically in the environment around the agent, not by the agent itself, so they hold even if the agent is buggy or prompt-injected. AWS maps the layers to its own tools: Policy in Bedrock AgentCore, which is written in plain language, checked by automated reasoning for gaps and contradictions, and enforced on tool calls at AgentCore Gateway; AWS Agent Registry and per-agent identities; the open-source Strands Box sandbox; and CloudWatch Omni for traces and evaluations. In a worked example, a help-desk agent offboards an employee using exactly six tools, and the gateway blocks a mailbox conversion until the employee’s sign-in is blocked, and a license change until a legal-hold check is done.

Why it matters. Having a human approve every action doesn’t scale, and the post makes a clear case for the alternative: precise, provable rules where they can be written, with human judgment reserved for the rest. Its line between rules that can be formally checked (“only a manager can approve access”) and rules that can’t (“keep a friendly tone”) is useful on any platform. This is a vendor position paper, not an independent evaluation of the products it names.

Anchor Browser, a cloud browser for AI agents, now works with Amazon Bedrock AgentCore payments, so agents can pay for resources protected by x402. x402 is an open protocol in which a server answers with HTTP 402 Payment Required plus the price, the accepted payment methods and a payment address. Setting session.x402 to true when creating an Anchor session links it to an AgentCore PaymentSession with a configured wallet and budget. When a page, fetch or XHR request returns 402, AgentCore checks the budget and signs the payment through Coinbase Developer Platform or Stripe Privy, and the browser retries the request once with the signed header. Private keys never enter the browser or agent runtime, AgentCore enforces the spending limits and logs payment attempts, denials and settlements to CloudWatch, and existing Playwright, Puppeteer or CDP automation needs no changes.

Why it matters. Pay-per-request data, APIs and paywalled content are a dead end for autonomous agents unless the agents can pay safely. Moving keys, budgets and signing out of the agent runtime deals with the obvious risks, such as a retry loop or a prompt injection draining a wallet. Turning the flag on authorizes real payments from the configured instrument, so the budget settings deserve the same review as any other spending control.

GitHub made agentic autofix for code scanning alerts generally available on October 15. According to GitHub's changelog, assigning an alert to Copilot starts an agent session that explores the relevant files, writes a fix, reruns CodeQL to confirm the alert is closed, iterates if it isn't, and then opens a draft pull request explaining the fix and its validation steps. Developers can ask for changes in PR comments or in the repository's Agents tab. Alerts can be assigned one at a time, up to 25 at once from the backlog or a security campaign (one session and one PR for the batch), or through the REST API by setting assignees to copilot-swe-agent[bot].

When enabled, it replaces the free Generate Fix button with Assign to Copilot. That older feature, Copilot Autofix, produced a single suggested change for one alert and doesn't consume AI credits. Each agentic session consumes Code Security AI Credits and GitHub Actions minutes, and the changelog says a Copilot billing method must be defined, although GitHub's docs say no Copilot license is needed. An enterprise administrator has to allow it through a new "Agentic autofix" policy before an organization or repository admin can turn it on.

GitHub's docs list the limits. It works only on alerts on the default branch, not alerts found only in PRs or other branches. Validation is best-effort, and Copilot can't confirm fixes for alerts from custom CodeQL queries, the security-extended suite or third-party scanning tools. It isn't available for Enterprise Managed Users repositories. GitHub doesn't publish fix-success rates, and the docs say to review and test every generated change before merging.

Why it matters. It moves security backlog work from "here is a suggested patch" to "here is a PR an agent has already checked against the scanner", which is the step most teams skip for lack of time. The trade-off is cost and scope: each fix now draws AI credits and Actions minutes, and the scanner check covers only the standard CodeQL suite.

Context Mode is an MCP server, with hooks for clients that support them, aimed at two causes of context loss in coding agents: raw tool output and conversation compaction. Its sandbox tools, such as ctx_execute and ctx_batch_execute, run code or commands and return only the result, and the README’s own example shrinks 315 KB of output to 5.4 KB. It also records file edits, git operations, tasks, errors and user decisions in a local SQLite database, indexes them with FTS5, and after compaction retrieves only the relevant events through BM25 search instead of re-injecting everything. In Claude Code it installs from the plugin marketplace and registers 11 MCP tools plus hooks; the README also documents setups for Gemini CLI, Codex CLI, Cursor, VS Code Copilot, OpenCode and other clients, and notes that clients without hooks rely on instruction files and follow the routing less reliably. It is released under the Elastic License 2.0, which makes the source available but bars offering it as a hosted service.

Why it matters. Long agent sessions often degrade because the context fills with logs, snapshots and file dumps, and compaction then throws away what the agent was doing. Treating the model as a code generator that returns summaries, backed by a searchable session memory, is a pattern any agent builder can borrow. The savings figures are the author’s examples and will vary with the workload.

K-Dense’s Scientific Agent Skills repository collects 177 skills covering bioinformatics, cheminformatics, proteomics, clinical research, microscopy, materials science, physics, geospatial analysis, lab automation and scientific writing, with guidance for more than 100 databases. Each skill documents one package, data source or workflow, including the scientific conventions and validation checks needed to use it correctly. The collection used to be called Claude Scientific Skills and now follows the open Agent Skills standard. It also ships as an Agent Plugins package, so it can be installed with npx skills add K-Dense-AI/scientific-agent-skills, with gh skill install, or through a plugin client such as Cursor or Codex. It is MIT-licensed and described in an arXiv paper (2609.00065).

Why it matters. Scientific analysis depends on domain conventions and validation checks that a general coding agent may not apply, and each skill bundles those for one tool or workflow. The project says this makes agents significantly stronger and more reliable; that is its claim rather than a measured result, so test it on your own analyses. The README presents the clinical and healthcare skills as research aids, not tools for patient-specific decisions.

HyperFrames, from HeyGen, is an Apache-2.0 framework that turns HTML, CSS, media and seekable animations into MP4 files. Its renderer seeks to each frame in headless Chrome and encodes with FFmpeg, so the same input always produces the same video, which the README pitches for CI and regression testing. A CLI handles scaffolding, preview with live reload, linting and rendering, and there are optional rendering paths through HeyGen’s hosted cloud and AWS Lambda; it requires Node.js 22+ and FFmpeg. For coding agents, the project ships 21 skills, including a router and workflows for product-launch videos, pull-request explainers, captions and Remotion ports, installable as a Claude Code plugin or with npx skills add heygen-com/hyperframes. The README says it was inspired by Remotion and differs mainly in using plain HTML instead of React components.

Why it matters. Video has been awkward for agents because most tools are GUI timelines, whereas HTML is something models already write well. Deterministic rendering also makes generated video testable, which matters if clips are produced automatically from changelogs or pull requests. The open-source core is separate from HeyGen’s hosted rendering option.

Superpowers, built by Jesse Vincent and Prime Radiant, is a set of composable skills plus bootstrap instructions that change how a coding agent approaches a task. Instead of writing code straight away, the agent runs a brainstorming skill to draw out requirements, presents a design for sign-off, creates an isolated git worktree, and writes a plan of 2–5 minute tasks with exact file paths and verification steps. Implementation then runs either through subagent-driven development, with a fresh subagent and a review for each task, or inline in the current session, with test-driven development enforced through a red-green-refactor cycle. The README lists installs for Claude Code (including Anthropic’s official plugin marketplace), Codex, Cursor, Gemini CLI, GitHub Copilot CLI, Grok Build CLI, OpenCode and several other harnesses. It is MIT-licensed; an optional visual companion loads a logo that reports the installed version, which you can turn off with SUPERPOWERS_DISABLE_TELEMETRY.

Why it matters. Many agent failures are misunderstandings rather than coding errors, and Superpowers front-loads the conversation that catches them. Its plans are written for a reader with no project context, which suits handing tasks to fresh subagents. The trade-off is a heavier process, and the README even includes a diagnosing skill for sessions that misfire or burn more tokens than expected.

Anthropic announced on October 15 that Code Review is out of research preview and generally available on Claude Team and Enterprise plans. According to Anthropic's announcement, most reviews now finish about three times faster than during the preview (6 to 10 minutes, down from about 20), and admins can set each repository to Quick, Standard (the default) or Deep. The speed and bug-finding claims are Anthropic's own; it published no benchmark.

According to the Claude Code documentation, a review runs on Anthropic's infrastructure in four stages: several agents read the diff and surrounding code in parallel, a verification step filters out false positives, the findings are deduplicated and ranked by severity, and the results are posted as inline comments plus a "Claude Code Review" check run. A review has read-only repository access, doesn't build or run the code, and never approves or blocks a pull request. The check run doesn't conclude as a failure. Reviews can run once per PR, after every push, or only when someone with write access comments @claude review, and they follow a REVIEW.md from the base branch, never from the PR itself.

The limits are specific. Code Review works only with GitHub (including Enterprise Server and Enterprise Cloud with data residency). It isn't available to organizations with Zero Data Retention, the HIPAA configuration or customer-managed encryption keys, and it isn't covered by Anthropic's BAA. Admins can't choose the model. Billing is by tokens through usage credits, with no flat per-review price, and the docs note that new repositories default to reviewing after every push, which multiplies cost. Anthropic is offering eligible Enterprise organizations at least $10,000 in review credits if claimed by January 15, 2027; the credits expire 30 days after claiming.

Why it matters. Agent-written code has moved the bottleneck to review, and this is Anthropic's managed answer to it, priced by tokens rather than seats. The docs are frank that a clean review doesn't mean the code is secure. Because a review can't approve, block or execute anything, it adds a reviewer, not a merge gate.

External memory systems depend on a model deciding what to store, how precisely to store it, and how new evidence should update what is already there; most frameworks give that job to a general-purpose LLM behind framework-specific prompts. Fei et al. present Hy-Memory-30B-A3B, which they describe as a compact long-term-memory writer trained across three frameworks: Mem0, MemOS and Hy-Memory. Training combines on-policy distillation of each framework’s write protocol with reinforcement learning that rewards downstream answerability, using per-chunk probes, rewards attributed across chunks, controls against over-storage and chunk-level GRPO. On fixed 20% subsamples of the benchmarks, the authors report that it ranks first on all four memory harnesses tested, including the held-out EverMemOS, with a 55.49% macro average against 54.62% for Claude Opus 5.

Why it matters. Memory quality is decided at write time: if the wrong facts get stored, retrieval can’t fix it later. The authors argue that putting a general-purpose LLM behind each framework raises serving cost, and offer a compact writer that works across frameworks, though the abstract reports no cost comparison. Its lead over Claude Opus 5 is under one point on subsampled benchmarks, so it shows parity more than a clear win.

Running a frontier model at every step of a long computer-use task is expensive, and small open-weight models often fail on their own. LEASH, from Wang et al., lets a weak agent act by default and hands control to a strong one based on latent uncertainty. It compares the weak model’s hidden states with those from its past successful and failed trajectories, reusing activations already computed during action generation, so it needs no extra forward pass and no trained monitor. It works at three levels: an initial uncertainty check for routing, step-level escalation during execution, and a shadow observer that keeps monitoring the black-box strong agent after handoff. On OSWorld across four model backbones, the authors report that LEASH beats random escalation by 6.55–11.94 points at matched escalation rates, and that with Qwen3.5-35B-A3B it reached 73.66% task success at $0.4117 per task, matching always-on Claude Opus 5 (72.90%) at 19.2% lower inference cost.

Why it matters. Routing between cheap and expensive models usually relies on heuristics or a separate classifier, and reusing signals the small model already produces is a low-overhead alternative. LEASH also covers a gap in most cascades: what happens once the expensive model takes over and starts to drift. The results come from a single benchmark, and the method needs access to the weak model’s activations, so it suits self-hosted open models rather than API-only setups.

FDE-Gym models the work of forward-deployed engineers: clarifying vague requirements, working in live systems and uncovering constraints nobody stated. It has 95 end-to-end tasks from 20 customer scenarios across 18 sectors, which industry experts adapted from real delivery projects, and runs them in simulated environments with scripted stakeholders and systems the agent can modify. A delivery’s score is the improvement in the customer’s own success metric, normalized between 0 (system unchanged) and 1 (reference solution), with penalties for messages, trial runs and waiting time. Across 13 configurations from seven providers, the best one reached a mean score of 0.56 in the realistic setting and resolved only 14 of the 95 tasks. Even on the 23 tasks where that configuration got every key fact from the stakeholders, it scored 0.8 or higher on just 10.

Why it matters. Most coding benchmarks hand the agent a well-specified ticket, which is the part of real delivery that’s already done. FDE-Gym isolates the harder step of finding out what the customer actually needs, and it shows performance falling sharply when agents have to discover constraints themselves. The authors also find that asking better questions alone doesn’t close the gap.

OpenGCUA is an ongoing project from Zhou et al. to build an open stack for generalist computer-use agents. The authors define these as systems in which the model chooses how to observe, act and organize its work across visual and programmatic interfaces. Code execution is central to the design, so intermediate results can drive later steps, and the authors propose moving selected observation and task-state decisions out of fixed harness rules and into the model. On OSWorld-v2, they report that their full configuration improved scores for all eight models tested, including GPT-6 Astra and Claude Opus 5, raising the mean partial score by 19.6 points and cutting reported steps per task by 53.0% on average compared with batched GUI-only execution. The release includes the implementation and prompts, evaluation trajectories from all eight models, and early versions of a real-device harness and an SDK that works across computers, phones and browsers.

Why it matters. Frontier computer-use systems are mostly closed, so an open harness with published trajectories gives builders something to inspect and reproduce. With the same GUI and shell access, the authors report that writing operations as code added 6.1 points and cut steps per task by 14.8% against a GUI+CLI baseline; the larger 53% cut comes from the full setting, which also moves decisions from the harness to the model. They also report that on four real macOS tasks, selected Qwen3.8-Max runs used 19.4 times as many reasoning tokens as GPT-6 Astra, a cost gap that benchmark scores alone hide.

Most LLM calls in an agent harness end in a multiple-choice decision, such as which tool to call next, and decision models like TypeSafe’s Jev answer those directly at a fraction of an LLM call’s time and price. Zhu et al. built AgentDecisionBench, which tests decision models on everything from isolated questions to controlling a running agent. They found that the model most accurate on isolated questions solved only 69.5% of agent tasks, and that delegating tool choices without a fitted confidence threshold cut success by 19.0 points. Their JevAgentHarness fits a threshold for each choice on the agent’s own trajectories and hands the decision back to the LLM below it, and they trained AgentDecision-4B, an open 4B model, on states agents reach after their own mistakes, using Jev’s answers as labels. On tasks unlike its training data, agents solved 91.5% with AgentDecision-4B against 82.0% with Jev, and on similar tasks the authors report that 4B models trained this way made agents 81.1–89.0% cheaper and up to 2.5 times faster while losing less than 3 points of success.

Why it matters. Decision models are multiplying fast (see this edition’s Models & Labs roundup), and most of their published scores are vendor-run. This paper tests what those benchmarks skip, whether a decision model can actually steer an agent, and its lesson is practical: accuracy on isolated questions doesn’t predict agent success, and a fitted fallback threshold is essential.

How well an agent performs depends heavily on the harness around the model, but existing benchmarks hold the harness fixed. Doc2Harness, from Ge et al., evaluates harness engineering directly in two stages. First, a build model working inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behaviour from twelve frontier harnesses, and constructs a complete harness in an isolated container. The frozen result is then driven end to end through each test case by a scripted mock model, and scored on four observable channels: the requests sent to the model, the files on disk, the process exit code and the command output, with positive controls to rule out empty passes. Across 19 frontier models, each building three times and tested on 328 cases, GPT-6 Astra had the highest pass rate at 53.05% and the weakest model 27.44%, with failures clustering around how cross-cutting functionality is split across modules.

Why it matters. As teams use agents to write more of their own tooling, building a correct, long-lived system from a specification matters more than fixing isolated issues. The authors say driving each harness with a scripted mock model makes the benchmark deterministic, which is rare for agent evaluations. With even the best model failing nearly half the cases, agent-built infrastructure needs testing as thorough as hand-written code gets.

AlphaDiana, from Zhou et al., is a measurement system for agents on verifiable reasoning tasks, with modular interfaces for datasets, agents, environments and scorers. It keeps each harness’s native control loop while controlling file reuse and memory retention separately, so researchers can study whole agents or change a single component, such as tools, skills or memory, under matched conditions. The authors tested three open-source models with three open-source harnesses on five answer-based benchmarks, and two of the models on two interactive benchmarks as well. Harnesses didn’t consistently improve accuracy: gains on questions the model couldn’t solve alone were sometimes outweighed by losses on questions it already answered correctly without tools, and neither skill guidance nor retained history reliably helped. The authors also warn that low token entropy can come with wrong answers, and that when accuracy falls, less output per run can still mean more output tokens per correct answer.

Why it matters. It’s common to assume that adding tools, skills or memory makes a model better, and this study suggests the effect is mixed and depends on the task. For builders, harness changes need before-and-after measurement on the questions the bare model already gets right, not only on the ones it fails. The results cover open models and harnesses on reasoning benchmarks, so they may not carry over directly to coding agents.