ASIL's 380-task result argues against making software agents click like humans
Watching an agent move a pointer and click through an application makes for a persuasive demo. It also asks software to work around the interface instead of using the state underneath it.
A new paper, ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions, tests another route. ASIL stands for Agent-Software Interaction Layer. It gives an agent structured JSON observations and semantic operations, then translates those operations through the deepest practical route each application offers: a file format, scripting interface, or API.
Closed models using ASIL scored above 80 across 300 single-application tasks and 80 multi-application workflows, averaging fewer than five executed actions per task. In a repaired screenshot-and-click comparison with a 50-step budget, GPT-5.4 scored 6.6 strict and sonnet4.6 scored 26.6.
Those numbers are not a universal verdict on computer use. They support a narrower rule for product teams: if an application can expose structured state, an agent should usually use it.
A screenshot leaves state behind
People need a rendered interface because eyes and hands are our general-purpose adapter. Software does not. A screenshot can omit a hidden panel, document property, off-screen row, or background job. A click records motor behavior without naming the intended state change.
ASIL makes three parts explicit: observe structured state, execute a schema-constrained action, and validate the result. Its adapters cover 15 applications, including Blender, GIMP, Inkscape, Kdenlive, LibreOffice, draw.io, JupyterLab, Gitea, Thunderbird, and Nautilus. Six are file-backed, four use native scripting, and five use a service or API.
A spreadsheet edit can therefore become one operation against the document instead of cell selection, keystrokes, scrolling, and visual checking. A shorter route is not automatically safer. It is easier to inspect, retry, and test.
The idea resembles a good tool call. Our guide to agent tool interfaces recommends meaningful operations, explicit inputs, and reviewable results. ASIL extends that pattern to applications originally designed around graphical controls.
Native does not always mean agent-friendly
The paper's most useful comparison is not ASIL versus screenshots. LibreOffice already has UNO, a mature automation surface. The LibreOffice SDK documentation describes UNO as the component model behind the office API, with bindings for Basic, Java, Python, and C++.
On 60 matched LibreOffice tasks, ASIL beat UNO by 28 to 38 strict points, depending on the model. The result suggests that a normalized observation and a small contract of actions can be easier for a model than a broad, low-level native API.
It does not follow that every application needs another universal wrapper. On 20 draw.io tasks, GPT-5.4 tied the draw.io MCP content contract at 55.0 strict. Sonnet4.6 scored 25.0 through ASIL and 55.0 through MCP. Draw.io's official MCP server already exposes focused operations, including diagram creation and native .drawio output. ASIL provided no advantage there.
Before writing an adapter, inspect the interface that already exists. Does it describe outcomes a user recognizes, or implementation machinery a programmer must assemble? A large native API may be too granular for a model. A focused domain tool may already have the right shape.
Keep the benchmark's asymmetries visible
The headline comparison comes with qualifications. The original ASIL prompts contained evaluator-derived success hints; the GUI prompts did not. The paper measured the effect in a balanced audit: 29 of 30 strict passes with hints versus 26 of 30 without them. Newer comparisons disable hints on both sides, but the authors say independent validation across all 380 tasks remains future work.
The benchmark also admits only applications with at least one open access path. Fully closed software without parseable files, scripting, or a service API falls outside the current method. That excludes a meaningful class of commercial desktop software.
Visual work exposes a different limit. On 24 difficult GIMP tasks involving image composition, GPT-5.4 averaged 14.7 and completed none. Adding screenshots did not fix the problem. Structured state works best when success can be represented and checked structurally. Taste, composition, and other perceptual goals still need better visual reasoning and richer action primitives.
The OSWorld comparison also requires restraint. ASIL describes one easier 60-task slice as “OSWorld-comparable,” not as an exact difficulty match. On that slice, GUI scores rose to 15.0 and 53.3. The full 380-task set includes harder workflows without staged intermediate progress.
The evidence is strongest in a defined region: applications with an open structural access path and outcomes that can be verified against resulting state. That is still a large and useful region.
Give the agent its own door
A product team can apply the pattern without adopting ASIL itself:
- Expose a compact observation with identifiers, versions, relevant hidden state, and the current artifact location.
- Name actions after outcomes such as
add_sheet,replace_range, orexport_diagram, not pointer movements. - Return updated state and a durable artifact after each mutation.
- Validate final application state independently of the agent's closing message.
- Keep screenshots for perceptual checks and human review, not as the only control channel.
- Add permissions, audit logs, and confirmation gates around destructive actions.
Independent validation deserves its own implementation path. If the same agent both makes a change and declares success, a fluent closing message can hide an incomplete artifact. Read the saved document, query the application state, or run a deterministic check after the mutation.
Permissions matter for the opposite reason. A more efficient interface can make mistakes more efficient too. Our least-privilege checklist applies whether actions arrive through MCP, a native API, or an ASIL-style adapter. Structured operations improve observability; they do not grant authorization.
The delivery path should fit the application. A remote tool can offer semantic actions through MCP, while an interactive result can use MCP Apps to keep a person in the loop. A local adapter can work on copied files or inside a sandbox, then return a preview before replacing the original.
“Our agent can use the interface like a person” sounds reassuringly general. Often it means the product withheld its real state and made a model reconstruct it from pixels. ASIL offers a better default: preserve the human interface, then add an agent-facing path that exposes the state transitions the software already understands.
