TurnOPD: Turn-Aware On-Policy Distillation for Long-Horizon Agent Training

Research signal

TurnOPD is a training and optimization technique for long-horizon language agents. It is relevant to Secure AI Atlas because training methodology affects how agents behave across multi-step tasks, but it is not by itself an operational security control.

Vanilla on-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student’s own trajectories. Two inefficiencies arise in multi-turn environments:

1. Full-horizon rollouts waste wall-clock resources on tail turns that provide weak and noisy KL supervision.
2. Trajectory-level KL objectives concentrate most loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. The deepest third of turns receives only approximately 3.6–4.5% of the KL budget under vanilla OPD.

TurnOPD addresses these through a turn-level budgeting strategy that redistributes the learning signal across turns for more efficient long-horizon training. That may improve agent reliability under some evaluation regimes, but deployment teams still need separate security controls for authorization, logging, sandboxing, tool governance, and approval.

Implementation

Two budget controllers operate in tandem.

Adaptive rollout-depth budgeting. Periodically issue full-depth probe rollouts to estimate turn-level statistics. Use a probe-based coverage lower bound and EMA smoothing to determine when rollouts become inefficient, then truncate traces to preserve training throughput. The controller balances competing objectives — an efficiency proxy (H_eff) against a coverage lower bound (H_cov) — to avoid overly short traces.

Progressive turn-normalized loss budgeting. Progressively interpolate between trajectory-level and turn-level KL objectives. Early training favors broad trajectory mass; later epochs increase turn-normalized loss emphasis to rebalance per-turn supervision. This progressive blending corrects the optimizer’s tendency to correct only shallow behavior while avoiding an absolute turn-level objective that may over-weight noisy late-turn signals.

Turn-resolved diagnostic monitoring. Compute per-turn reverse-KL signals decomposed into forced-component mass (context-locked probability) and free-component disagreement. This contamination-compression decomposition reveals when raw KL underestimates true policy differences in later turns, and should inform controller thresholds.

Operational interpretation

The training pipeline owner owns the TurnOPD controller configuration — probe frequency, warm-up steps, blending slope, and depth-truncation guardrails. Security teams should treat TurnOPD as research evidence about agent training dynamics, not as a compensating control. A trained agent can still misuse tools, inherit excessive authority, or act on poisoned context unless operational controls are applied at runtime.

Evidence from the paper

• Per-turn KL budget distributions showing the imbalance in vanilla OPD (deepest third: ~3.6–4.5%).
• Accuracy–wall-clock frontier comparisons against vanilla OPD, linear blend, and ablations.
• ALFWorld-1.7B: full TurnOPD raises Same-Step Avg@4 from 83.0 to 86.3 while reducing 100-step wall-clock time from 4.42h to 1.93h.
• Ablations: adaptive depth alone reduces wall time to approximately 1.96h but may slightly hurt accuracy; a linear KL blend alone lifts Avg@4 to approximately 85.1.
• WebShop and Multi-Hop Search results under comparable budgets.

Common errors

• Truncating rollouts too aggressively, causing the student to miss critical rare events in the tail.
• Setting probe frequency too low, allowing the depth controller to operate on stale statistics.
• Over-weighting deep turns through aggressive progressive blending when late-turn signals are scarce or noisy, producing unstable updates.
• Ignoring contamination-compression effects in diagnostic KL signals, leading to misconfigured controller thresholds.
• Using an unobservable target (H) without calibration or warm-up tuning.

Related risks

• Inefficient agent training
• Underspecification of long-horizon behavior
• Reward gaming through shallow optimization

Classification note

This entry was originally published in the Controls Catalogue on July 8, 2026. It has been reclassified as a Learning Log research note because a training technique is not the same thing as a control boundary. TurnOPD may inform future evaluation or training guidance, but it should not be cited as a canonical Secure AI Atlas control without separate conceptual review.