Oriel — AI & technology worth noticing, every day.

Agent Harness Awareness: Turning Verified Telemetry into Operational Control for Tool-Using Agents

Paper · https://arxiv.org/abs/2611.05391

arXiv:2611.05391v1 Announce Type: new Abstract: Tool descriptions tell an agent what an action can do, but they do not necessarily reveal whether that action is healthy or suitable at the moment of use. Execution harnesses often possess this information through monitors, traces, and recent failures while the model choosing the next action does not. We call this gap operational blindness. We introduce Agent Harness Awareness, a trust-separated design in which the host supplies a small, fresh, verified state snapshot before a consequential action while retaining measurement, enforcement, evaluation, and final dispatch authority. We test its tools-and-network instance with ToolRoute, a controlled isolation benchmark that holds the task and candidate semantics constant and varies the operational information available to the model. In a 216-decision ablation across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, verified tool-health state reduced mean observable policy regret by 96.9% against a same-shape neutral control (4,252 ms to 131 ms) and raised single-dispatch completion from 69.4% to 95.8%. In a separate 27-decision end-to-end replication with live local TCP/HTTP execution, OpenTelemetry client spans, and Toxiproxy-injected faults, verified state reduced mean regret from 8,956 ms to 0 ms (zero excess observable operational cost) and completed the live action in 9/9 model-by-fault cases versus 5/9 for the neutral control. These studies establish the tools-and-network instance of Agent Harness Awareness: selected host telemetry materially improves pre-dispatch operational choice. Runtime state, hardware pressure, economics, retry-aware recovery, and heterogeneous selection remain open empirical extensions of the same architecture.

First seen by Oriel on .

AI summary

Agent Harness Awareness gives tool-using agents fresh, verified host telemetry before consequential actions, reducing operational regret and improving completion in tools-and-network benchmarks.

  • The host retains authority over measurement, enforcement, evaluation, and final dispatch while supplying the model with a small state snapshot.
  • In a 216-decision benchmark across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, verified tool-health state reduced mean observable policy regret by 96.9%.
  • Single-dispatch completion rose from 69.4% to 95.8% against a same-shape neutral control.
  • In a 27-decision live replication with injected faults, verified state reduced mean regret from 8,956 ms to 0 ms and completed 9/9 cases, versus 5/9 for the control.
  • The study covers tools and network health; runtime state, hardware pressure, economics, retry-aware recovery, and heterogeneous selection remain open extensions.

#ai agents #tool use #telemetry #networking

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.05391v1https://arxiv.org/abs/2611.05391 Read discussion