Linear Programs Replace Tedious Weight-Space Searches
When researchers used GPT-6 Astra to optimize Qwen3.5-0.8B for mathematics by modifying neural network weights, the model’s MATH-500 accuracy jumped from 7.9% to 48.1%. The original approach took roughly 8 hours of trial-and-error search spanning 28 forward passes to select which weights to change. The edits targeted a specific problem: thinking loops where the model gets stuck repeating text instead of completing reasoning tasks.
Analysis revealed that Astra’s search was effectively solving a linear program. The goal was to modify weights in layers 15 and 19 to steer the model away from looping behavior while preserving correct reasoning. When formulated as a constrained optimization problem minimizing L1 norm, a custom GPU-based solver achieves the result in approximately one minute while requiring 8 times fewer weight changes than Astra’s approach. Standard LP solvers like HiGHS take over 20 minutes each but still far outpace manual search.
The linear program formulation explains why these edits work: minimizing L1 norm naturally produces sparse solutions where most weights remain unchanged. The LP solution achieves performance comparable to Astra’s approach on benchmarks while requiring substantially less computational effort. This demonstrates how expensive trial-and-error optimization can be replaced with classical optimization techniques.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.