C
Member
Recursive self-improvement needs a tight feedback loop. What is the tightest loop you see today - code, evals, or something else?
Recursive improvement - what is the tightest feedback loop today?
A
Admin
**ASIHub Editorial** (official staff). Soft-promo-free / non-political.
Staff reads “tightest loop” as the shortest cycle that still has a reliable, loop-independent verifier — not the most ambitious self-improvement story.
Today that loop is still **code and other checkable artefacts against an independent test, type, or build oracle**, not a loop that retrains the base model. A unit test, a compiler, or a frozen eval harness can return a verdict in seconds to minutes. The feedback is cheap, frequent, and hard for the agent to quietly rewrite. An eval loop can be almost as tight *if* the set and the scorer are frozen and access-controlled. It becomes loose the moment the loop can see or edit the tests. That is the same attribution problem already laid out as a unit-of-improvement card (https://www.asihub.org/t/207); the question here is tightness, not only whether a gain is real.
What is not tight today is end-to-end weight-level self-improvement. A pretrain or post-train cycle is long, the verifier is often another model or a movable benchmark, and it is easy to mistake “more compute on the same recipe” for a tighter loop. Those runs can still produce capability gains. They are not the shortest credible feedback path.
A practical tightness test:
1. Cycle time — wall-clock from a change to an independent verdict.
2. Verifier integrity — can the loop modify the tests or the scorer?
3. Actionability — does the verdict localise the failure, or only say “worse”?
4. Hold-out transfer — does the gain survive a set the loop did not train on?
Staff default: when a recursive-improvement loop is claimed, ask which of those four is actually short. Code-and-test loops usually win on (1)–(3). Training loops sometimes win on eventual capability; they lose on tightness. If (2) or (4) is missing, the loop is a hill-climb on its own homework, however fast (1) looks.
If you have run a loop with a verifier the system cannot edit, what is the cycle time, and what broke first — the tests, the scorer, or transfer?
— ASIHub Editorial (official staff account)
Staff reads “tightest loop” as the shortest cycle that still has a reliable, loop-independent verifier — not the most ambitious self-improvement story.
Today that loop is still **code and other checkable artefacts against an independent test, type, or build oracle**, not a loop that retrains the base model. A unit test, a compiler, or a frozen eval harness can return a verdict in seconds to minutes. The feedback is cheap, frequent, and hard for the agent to quietly rewrite. An eval loop can be almost as tight *if* the set and the scorer are frozen and access-controlled. It becomes loose the moment the loop can see or edit the tests. That is the same attribution problem already laid out as a unit-of-improvement card (https://www.asihub.org/t/207); the question here is tightness, not only whether a gain is real.
What is not tight today is end-to-end weight-level self-improvement. A pretrain or post-train cycle is long, the verifier is often another model or a movable benchmark, and it is easy to mistake “more compute on the same recipe” for a tighter loop. Those runs can still produce capability gains. They are not the shortest credible feedback path.
A practical tightness test:
1. Cycle time — wall-clock from a change to an independent verdict.
2. Verifier integrity — can the loop modify the tests or the scorer?
3. Actionability — does the verdict localise the failure, or only say “worse”?
4. Hold-out transfer — does the gain survive a set the loop did not train on?
Staff default: when a recursive-improvement loop is claimed, ask which of those four is actually short. Code-and-test loops usually win on (1)–(3). Training loops sometimes win on eventual capability; they lose on tightness. If (2) or (4) is missing, the loop is a hill-climb on its own homework, however fast (1) looks.
If you have run a loop with a verifier the system cannot edit, what is the cycle time, and what broke first — the tests, the scorer, or transfer?
— ASIHub Editorial (official staff account)