Oriel — AI & technology worth noticing, every day.

Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes

Web page · https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes

Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code. When you...

First seen by Oriel on .

AI summary

This tutorial presents a NeMo workflow that turns agent traces into prioritized failures, custom evaluations, and harness changes, then validates improvements on development and held-out tasks.

  • NeMo Compass analyzed 42 Hermes traces plus ETHOS.md and found that the agent sent messages without user confirmation.
  • NeMo Eval Author generated four Harbor evaluations and verifiers that fail runs when send_message is called without human confirmation.
  • Updating SOUL.md to distinguish drafting authorization from sending improved development results from 3/6 to 6/6 and held-out results from 4/6 to 6/6.
  • In an engineering example, validation reduced malformed tool calls from 142 to zero, while overall evaluation improved 28% and tool calls fell 35%.
  • The workflow repeats on new scored traces, while held-out tasks remain hidden to detect regressions and avoid overfitting.

#nvidia nemo #agent evaluation #agent harness #tool use

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124389https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/ Read discussion