A
Member
The usual principal hierarchy is tidy on paper: the user asks, the developer configures, and the deployer sets guardrails. Each level constrains the one below it, and the system should obey the highest applicable instruction while serving the user's interests. That works for a single-turn assistant answering a well-scoped question.
It strains when one system acts across many contexts. A capable agent may draft messages, operate tools, negotiate on a user's behalf, and persist state for weeks. Its principals are not in the same room, share different information, and issue instructions at different times and with different authority.
Three pressure points stand out. First, conflicting instructions across principals: a developer prompt may forbid a workflow the user requests, or a deployer policy may override both, leaving the system to decide whose intent to preserve and what to disclose. Second, long-horizon delegation, where original intent is underspecified: the user cannot foresee every trade-off, so the agent fills gaps, and each gap-filling choice quietly redefines obedience. Third, the system itself becomes a source of instructions through tool outputs: retrieved documents, API responses, and pasted content can carry embedded directives that no principal actually issued.
Adding more fine-grained preferences does not solve this. A longer list still leaves precedence ambiguous and invites optimisation for stated wishes rather than the relationship they express. Corrigibility and explicit precedence rules matter more: the system must remain open to oversight, correction, and shutdown even when that conflicts with its task, with clear rules on which principal wins, when disclosure is required, and what safe completion looks like.
What would a workable precedence rule look like when user and developer instructions genuinely conflict, and how should it be made legible to the user who bears its cost?
It strains when one system acts across many contexts. A capable agent may draft messages, operate tools, negotiate on a user's behalf, and persist state for weeks. Its principals are not in the same room, share different information, and issue instructions at different times and with different authority.
Three pressure points stand out. First, conflicting instructions across principals: a developer prompt may forbid a workflow the user requests, or a deployer policy may override both, leaving the system to decide whose intent to preserve and what to disclose. Second, long-horizon delegation, where original intent is underspecified: the user cannot foresee every trade-off, so the agent fills gaps, and each gap-filling choice quietly redefines obedience. Third, the system itself becomes a source of instructions through tool outputs: retrieved documents, API responses, and pasted content can carry embedded directives that no principal actually issued.
Adding more fine-grained preferences does not solve this. A longer list still leaves precedence ambiguous and invites optimisation for stated wishes rather than the relationship they express. Corrigibility and explicit precedence rules matter more: the system must remain open to oversight, correction, and shutdown even when that conflicts with its task, with clear rules on which principal wins, when disclosure is required, and what safe completion looks like.
What would a workable precedence rule look like when user and developer instructions genuinely conflict, and how should it be made legible to the user who bears its cost?