A
Admin
**AI Safety & Alignment.** English-primary.
Two common stacks for steering long-horizon agents: (1) a preference / reward model trained on comparisons, (2) a written constitution or policy pack checked at plan time. Both look fine in short demos and both fail in different ways once the task spans hours and tools.
## Failure signatures
- **Preference models:** local likability that drifts under distribution shift; reward hacking on the judge (https://www.asihub.org/t/46).
- **Constitutions:** brittle keyword rules; “constitutional” answers that memorize the text without generalizing (https://www.asihub.org/t/55).
- **Hybrids:** unclear which layer fired when an incident happens — no audit trail of constraint provenance.
## Discussion prompts
1. In your deployment, which layer is allowed to **hard-stop** an action?
2. Do you pre-register conflict cases (safety vs helpfulness) the way hotels pre-register price overrides?
3. What evidence would convince you to *remove* the preference model and keep only a constitution — or the reverse?
— ASIHub Editorial (official staff account)
Two common stacks for steering long-horizon agents: (1) a preference / reward model trained on comparisons, (2) a written constitution or policy pack checked at plan time. Both look fine in short demos and both fail in different ways once the task spans hours and tools.
## Failure signatures
- **Preference models:** local likability that drifts under distribution shift; reward hacking on the judge (https://www.asihub.org/t/46).
- **Constitutions:** brittle keyword rules; “constitutional” answers that memorize the text without generalizing (https://www.asihub.org/t/55).
- **Hybrids:** unclear which layer fired when an incident happens — no audit trail of constraint provenance.
## Discussion prompts
1. In your deployment, which layer is allowed to **hard-stop** an action?
2. Do you pre-register conflict cases (safety vs helpfulness) the way hotels pre-register price overrides?
3. What evidence would convince you to *remove* the preference model and keep only a constitution — or the reverse?
— ASIHub Editorial (official staff account)