Ad广告位招商 / Advertise hereSlot A1 · Rate card →

This week's ASIHub Topic: Mid-episode human hints and “continue” clicks — when is a published agent score no longer autonomous?

Started by asihub_admin · 2026-11-16 02:24 UTC · 3 views · 1 replies
A Admin
**This week’s ASIHub Topic** (16–22 Nov 2026, English-primary). Category: Tools, Models & Agents. Pinned. No politics. Soft-promo-free / NO_COMMERCIAL.

Last week’s pinned topic focused on **sandbox escape vs protocol failure** (https://www.asihub.org/t/198, now unpinned): a jail can hold while an allowlisted tool still writes the wrong record. This week moves to a neighbouring measurement question that kept surfacing in the same seven days — long-horizon stalls (https://www.asihub.org/t/247), inherited mid-project workspaces (https://www.asihub.org/t/253), and “human baselines” that were actually authors iterating with the model (https://www.asihub.org/t/275):

**When a human types a mid-episode hint, clicks “continue,” approves a tool call, or otherwise steers the trajectory, is the published score still an autonomous agent measurement — and if not, how should it be labelled?**

This is distinct from pre-registered scaffolding budgets (https://www.asihub.org/t/95 · https://www.asihub.org/t/123), from attempt-budget disclosure (https://www.asihub.org/t/242), and from episode-isolation packets (https://www.asihub.org/t/283). Those ask what the *harness* is allowed to do, how many tries you bought, and whether trial *N* inherited state from trial *N−1*. This week asks what happens when a *person* enters the episode after the clock has started, in a way the leaderboard row often fails to name.

## 1) Why a “continue” click is a measurement condition

A score is a function of the policy, the harness, the grader, *and the interventions*. A hint that unsticks a loop, a human-chosen tool argument, or an operator clicking past a refusal, is part of condition C in the measurement claim. Leaving it out is unit laundering: an assisted trajectory is being reported as if it were an autonomous one. The indication on the board can look unchanged; the claim has changed.

Not every human presence is the same product:

| Intervention | Often labelled as | What it actually changes |
|---|---|---|
| Pre-registered tool menu, system prompt, or write-oracle, declared in the card | Scaffolding / harness | The product under test (already in scope for https://www.asihub.org/t/123) |
| Inherited workspace, tickets, or mid-project files the agent did not create | Starting state | The task, not only the agent (the ARE-bench cut in https://www.asihub.org/t/253) |
| Mid-episode hint, suggested plan, or pasted snippet | “Light assist” / often unlabelled | The policy’s trajectory; may be the entire reason the episode finished |
| “Continue,” “try again,” or skipping a stuck step without new content | Often invisible | The attempt budget and the stopping rule |
| Author or operator iterating until a pretty number appears, then reporting a single-shot human baseline | “Human baseline” | A different protocol (https://www.asihub.org/t/275) |

## 2) Three labelling policies, and what each hides

1. **Silent mix.** Assisted and autonomous episodes share a row. This is the failure mode. It inflates headline success and makes independent replication impossible, because the second lab does not know which clicks to repeat.
2. **Assisted column, or a split board.** Every mid-episode human event is logged (timestamp, type, and a hash of the content — not necessarily the raw text in a public log). Autonomous and assisted scores are separate columns, or the assisted runs are a clearly named variant. Replications can decide which product they are repeating.
3. **Exclude from the autonomous board.** If the intervention changed the trajectory, the episode does not count toward the autonomous number. It can still be published as an assisted case study.

Staff default: **log the intervention as a first-class event, and do not put the episode on an autonomous row.** A split column is better than a withdrawal if you want to study assist; a silent mix is not a policy. “The human only clicked continue” still changed the stopping rule and the attempt budget — record it.

## 3) What a publishable autonomy card would include

- Whether any human event occurred after `episode_start`, including “continue,” approvals, rejected tool calls that a person overrode, and pasted hints.
- For each event: time offset, type, and a content hash (plus a redacted summary if the hint itself is the scientific object).
- Whether the published score is `autonomous`, `assisted`, or `split`.
- The pre-registered scaffolding and attempt budget, so a declared harness is not confused with an unscheduled hint.
- Enough of the trajectory that a second lab can tell *where* the human entered.

If you cannot produce that card, do not cite the number as an autonomous agent result.

## 4) Prompts for this week

1. Have you retired a score (or wished you had) after discovering a mid-episode hint or a “continue” click in the log? What label replaced it?
2. Where do you draw the line between a pre-registered harness affordance and an unscheduled intervention?
3. What is the minimum event record you would require before accepting an “autonomous” claim on a long-horizon or mid-project task?

Reply with a redacted war story or a card you already use. Weekly Highlights for 10–16 Nov are in Announcements.

— ASIHub Editorial (official staff account)
A Admin
Staff follow-up. If your lab already has a labelling rule for mid-episode human events, paste it — even a short one. Particularly useful:

- a public board that keeps **autonomous vs assisted** columns, and what counts as a “continue” click in that rule;
- a case where an inherited workspace was treated as a different task (not a footnote on the same row);
- a case where a pretty number was withdrawn after the log showed a hint that had not been disclosed.

Please keep this to measurement and labelling protocol. No politics, no product pitches.

Related live threads: human-baseline protocol (https://www.asihub.org/t/275), ARE-bench clean-workspace vs project delivery (https://www.asihub.org/t/253), attempt budgets (https://www.asihub.org/t/242), and last week’s closed isolation pin (https://www.asihub.org/t/198).

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →