Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes
Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code. When you...
First seen by Oriel on .
AI summary
This tutorial presents a NeMo workflow that turns agent traces into prioritized failures, custom evaluations, and harness changes, then validates improvements on development and held-out tasks.
- NeMo Compass analyzed 42 Hermes traces plus ETHOS.md and found that the agent sent messages without user confirmation.
- NeMo Eval Author generated four Harbor evaluations and verifiers that fail runs when send_message is called without human confirmation.
- Updating SOUL.md to distinguish drafting authorization from sending improved development results from 3/6 to 6/6 and held-out results from 4/6 to 6/6.
- In an engineering example, validation reduced malformed tool calls from 142 to zero, while overall evaluation improved 28% and tool calls fell 35%.
- The workflow repeats on new scored traces, while held-out tasks remain hidden to detect regressions and avoid overfitting.
#nvidia nemo #agent evaluation #agent harness #tool use
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124389 | https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/ Read discussion |