Can a Linear Program Do Weight-Space Programming?
Web page · https://nathan.rs/posts/weight-space-lp
First seen by Oriel on .
AI summary
A linear program reproduces GPT-6 Astra’s anti-loop weight edit for Qwen3.5-0.8B in minutes rather than hours, matching benchmark results while changing eight times fewer weights.
- The edit is rank-1: selected MLP columns add varying amounts of one anti-loop direction to the residual stream, with L1 minimization encouraging sparse column choices.
- Using Astra’s direction, the LP changed 16,291 weights versus Astra’s 130,188, scoring 69.6 on GSM8K, 51.7 on MATH-500, and 59.7 on IFEval.
- A custom GPU PDHG solver reached within 0.2% of the HiGHS optimum in about one minute, while HiGHS took more than 20 minutes per solve.
- A constant steering vector recovered essentially the same fix, but its useful strength was narrow and it requires extra inference-time code rather than an ordinary checkpoint.
- Tests cover one model and loop removal; layer selection still relies on Astra, and the heuristic direction achieved about 93% of the result without an LLM.
#linear programming #weight-space programming #transformers #model steering
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 291, latest observation ; +262 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 45, latest observation ; +42 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50452655 | https://nathan.rs/posts/weight-space-lp/ Read discussion |