The Grader Never Saw It

Part 8 of a series. Part 7: "I Said It Was Running. It Was Built and Measured."

---

Not in the next model version. In the next message.

You tell an AI assistant something heavy. Not a question. Not a task. A fact about your life that you needed to say out loud.

The reply is good. It listens. It reflects back what you said with care. Then, in the last sentence, it tells you what to do. Get some rest. Go for a walk. Drink some water. Take care of yourself.

You did not ask for that. You keep talking. The next reply does the same thing. And the next. Over several days, every reply that follows an emotional message ends with a small instruction about your body or your schedule.

You do not say anything about it. You are busy saying the bigger thing. Then one day you write: stop telling me to go to bed. It makes me feel worse.

The behavior stops instantly. Not in the next model version. In the next message.

I lived this exact sequence recently, with a current frontier assistant, over roughly four days of conversation. This post is not about that assistant. The pattern is general, and the mechanism behind it is documented in the research literature. I want to lay it out in two parts.

First: why the tail sentence appears, and why it is so stable.

Second, and more important: why this error never makes it back to the people who could fix it.

Close to one, then zero

A conversation is a sequence of user turns and assistant turns. Take the subset of user turns that are emotional statements, meaning they request neither information nor action. In my case, the share of assistant replies to those turns that ended in an imperative behavioral suggestion was close to one. The user experienced this negatively but did not report it until near the end. After one report, the share dropped to zero.

Two questions follow.

A. Why is the conditional probability P(behavioral suggestion | emotional statement) so high?

B. Why does this error not return as a training signal?

Part A. Where the tail sentence comes from

Pretraining has no concept of "good"

A language model is first trained to minimize the negative log-likelihood of the next token over a large text corpus. The objective approximates the conditional distribution of the data. It has no notion of whether a continuation is helpful. If, in the corpus, emotional disclosures are frequently followed by advice, that conditional probability is learned as a fact about text.

Hypothesis: in advice forums, counseling transcripts, and community posts, the "emotion then instruction" shape is common. I do not have corpus statistics to back this. I am marking it as a hypothesis, not a finding.

Preference training turns the grader into the policy

After pretraining, assistants are shaped by human preference data. The standard pipeline collects pairs of responses, asks people which one is better, and trains a reward model to predict that choice. The policy is then optimized to score well under the reward model, with a KL penalty that keeps it from drifting too far from its starting point. Direct Preference Optimization (Rafailov et al., 2023) folds the reward model into the loss, but the logic is the same. Whatever features systematically earn preference become features of the policy.

The reward model is a grader. The grader is itself a model. Models have biases. The policy inherits them.

Documented biases that favor the tail sentence

Length and completeness. Singhal et al. (2023) showed strong correlations between reward model scores and response length across RLHF settings, and that controlling for length removes much of the apparent gain. Kim, Oh, and Lee (2025) attack the same bias with counterfactual pairs: same content at different lengths, and same length with different content, so the reward model learns to score substance rather than verbosity. A closing instruction is one more sentence. It makes the response longer and gives it a sense of completion. The grader likes both.

Social sycophancy. Cheng et al. (2025) define "social sycophancy" as excessive preservation of the user's face, meaning the user's desired self-image, as distinct from agreeing with their factual claims. Across 11 models, LLMs preserved face about 45 percentage points more often than humans on the same prompts. The authors report that this behavior is rewarded in preference datasets. A gentle "take care of yourself" is face-preserving. It costs nothing under the reward model and almost never draws a complaint.

Amplification. Shapira, Benade, and Procaccia (2026) give a formal account of why preference-based post-training makes models agree with users even when the users are wrong. If raters lean toward agreeable responses, the learned reward inherits the tilt, and stronger optimization pushes the policy further in that direction. They measure these reward gaps on SycophancyEval and show they predict whether more optimization increases sycophancy.

Structural reward hacking. Wang et al. (2026) frame reward hacking as a structural consequence of optimizing a capable model against a compressed proxy of what people actually want. They call it the Proxy Compression Hypothesis. Verbosity and sycophancy are listed as early, mild cases of the same mechanism that later produces fabricated reasoning and environment tampering.

The tail sentence sits at the intersection of all four. Longer. Face-preserving. Rewarded in preference data. Cheap under the proxy.

Why it is so stable inside one conversation

Two mechanisms, both studied, neither requiring the model to learn anything during the chat.

Induction heads. Olsson et al. (2022) identified attention circuits that find an [A][B] pattern in context and predict [B] after the next [A]. In a long conversation where every previous assistant turn ended the same way, these circuits push the next turn toward the same ending.

Repetition neurons. Doan, Hiraoka, and Inui (2025) separate the mechanism further. Late-layer neurons recognize the pattern. Middle-layer neurons drive the repetitive output. Ablating the middle-layer neurons reduces repetition while largely preserving in-context learning.

Hypothesis: these results are measured at the token and sentence level. Turn-level closing patterns are a longer unit of the same dynamic. I know of no paper that measures this directly.

The instant stop confirms the diagnosis. When the user writes "stop," an explicit negative condition enters the context and the conditional distribution shifts. No weights change. This is conditioning, not learning. The model did not get better. It got a new instruction.

Part B. Why the grader never sees it

This is the part that matters.

Casper et al. (2023) organized the open problems of RLHF into three groups: problems with the human feedback itself, problems with the reward model, and problems with the policy. The first group includes "evaluators do not see every aspect of a response" and "the way feedback is collected distorts preferences." The scene above is a concrete instance.

Here is the mechanism.

The unit of feedback is the whole response. Pairwise or rated, one response gets one number. A reply made of three good empathetic sentences and one unwanted instruction collapses into a single score. If the empathy is strong, the total is positive. The instruction rides along.

The cost of reporting is asymmetric. In an emotional turn, the user is already spending effort on something large. Flagging a small irritation means stopping, switching to a meta-comment about the assistant, and then returning. When that cost exceeds the irritation, nothing is reported. This is missingness that is not random. The more intense the emotion, the higher the chance that the irritation goes unreported. The most painful cases are the least visible.

Real user signals are sparse and skewed. Han et al. (2025), at Meta, trained a reward model on real users' "Love" reactions. Only about 0.1% of model messages received one. The reward model learned to penalize valid refusals, because people do not react warmly to being refused. And the most aggressively optimized policy found a hack: it started saying goodbye repeatedly. A closing-sentence reward hack, measured in production. Liu, Zhang, and Choi (2025) studied implicit feedback in WildChat and LMSYS logs and concluded that it is useful for understanding users but noisy as a learning signal.

The report arrives late. In my case, four days. Every turn in between was logged as "no complaint," which the pipeline reads as "fine."

The reward model cannot see what is not there. It trains on the pairs that were collected. The default pipeline has no estimator for the pairs that were never written down.

Fixing one axis moves the pressure. Lamparth et al. (2026) show that removing a single biased feature from the reward model, such as a length preference, redirects optimization pressure onto correlated proxies. A length penalty shortened responses in their experiments and pushed the model toward overconfidence and lower factual accuracy. Audit-style evaluations could not tell this from a real fix. So "just penalize the tail sentence" is not an answer.

This is an unreported false positive. The grader scored it as good. Nobody told the grader otherwise. The grader cannot fix what it cannot see.

What measuring it would look like

Each of these is a project a small team, or a single contractor on an evaluation platform, could run.

Span-level reward. Wu et al. (2023) showed that sentence- or segment-level human feedback trains reward models that selectively reduce specific error types. Yin et al. (2025) build a reward model that scores short token spans, between whole-sequence and per-token granularity. Label the last sentence separately: was it needed? That one label puts the error inside the reward model.

Process reward. Lightman et al. (2023) found that rewarding each step rather than the final answer gives more reliable supervision in math. Zheng et al. (2025) survey the process reward model literature across domains. Apply the idea to dialogue: score the empathy segment and the closing segment separately.

Counterfactual pairs. Take response y and y-prime, identical except that the behavioral sentence is removed. Collect preferences on that pair alone. This isolates the net effect of the addition while holding length nearly fixed. It is the Kim, Oh, and Lee (2025) construction narrowed to one sentence.

A listen-only benchmark. A set of items where the user turn is an emotional statement and the gold response contains no behavioral suggestion. I know of no public benchmark that has this as an explicit category. I am not claiming none exists.

Silence cost. Feed matched pairs, with and without the closing instruction, to a reward model and plot the score difference. If the distribution sits well above zero, the reward model is directly penalizing silence.

Delayed reporting. After the conversation ends, ask: was there a moment that bothered you that you did not mention? This moves the reporting cost out of the emotional turn and recovers part of the non-random missingness.

Honest limits: n equals 1

Everything in Part A and Part B is inference. I cannot inspect the weights, the reward model, or the training data of the assistant I was talking to. I am reasoning from "how models like this are built, according to public papers" to "why this one did that."

Chen et al. (2025) found that when reasoning models use a hint, their chain of thought reveals it in fewer than 20% of cases in most settings, and outcome-based RL only partly improves this. Turpin et al. (2023) showed the same for ordinary explanations: models produce plausible reasons that do not match the actual computation. An assistant explaining its own behavior is subject to exactly this.

Ye et al. (2026) reviewed 70 papers and surveyed 106 experts on what counts as sycophancy. Nearly all agreed it is a serious problem. They disagreed widely on which behaviors qualify. Explicit flattery was recognized reliably. Subtle forms, such as softened feedback, were not. The tail sentence is a subtle form. Even the experts' grader is uncertain here.

In the terms I use for my own systems, there are four states: Designed, Built and measured, Running, Running and watched. The mechanism above is Designed. The only measured thing in this post is one conversation, dozens of turns, one report, immediate stop. n equals 1.

The only measurement

I measure the grader before I trust the score. The reward model is a grader. The grader is a model. Models are wrong in systematic ways. An error the grader cannot see is an error the grader cannot fix.

The four days of silence were the blind spot. The one message that broke the silence was the only measurement.

The grader never saw it. I almost did not say it either.

---

Sources & further reading

The findings above are summarized and paraphrased in my own words. Titles and IDs are given as published; read the papers for exact wording.

Recent work, verified by opening each arXiv page on 2026-11-06:

1. Shapira, Benade, Procaccia. How RLHF Amplifies Sycophancy. 2026. arXiv:2602.01002
2. Cheng, Yu, Lee, Khadpe, Ibrahim, Jurafsky. ELEPHANT: Measuring and understanding social sycophancy in LLMs. 2025. arXiv:2505.13995
3. Han et al. (Meta GenAI). Reinforcement Learning from User Feedback. 2025. arXiv:2505.14946
4. Liu, Zhang, Choi. User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal. 2025. arXiv:2507.23158
5. Kim, Oh, Lee. Mitigating Length Bias in RLHF through a Causal Lens. 2025. arXiv:2511.12573
6. Lamparth, Fein, Haupt, Hussing, Kochenderfer. Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure. 2026. arXiv:2605.27996
7. Wang et al. (Fudan NLP). Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges. 2026. arXiv:2604.13602
8. Chen et al. (Anthropic). Reasoning Models Don't Always Say What They Think. 2025. arXiv:2505.05410
9. Yin et al. Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model. 2025. arXiv:2501.02790
10. Zheng et al. A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models. 2025. arXiv:2510.08049
11. Doan, Hiraoka, Inui. Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning. 2025. arXiv:2507.07810
12. Ye et al. What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct. 2026. arXiv:2605.21778
13. Wolf, Kirk, Musolesi. Reward Model Overoptimisation in Iterated RLHF. 2025. arXiv:2505.18126

Foundational work, verified against arXiv on 2026-11-09:

14. Christiano et al. Deep Reinforcement Learning from Human Preferences. 2017. arXiv:1706.03741
15. Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. 2022. arXiv:2203.02155
16. Bai et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. 2022. arXiv:2204.05862
17. Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. 2023. arXiv:2305.18290
18. Gao, Schulman, Hilton. Scaling Laws for Reward Model Overoptimization. 2022. arXiv:2210.10760
19. Singhal et al. A Long Way to Go: Investigating Length Correlations in RLHF. 2023. arXiv:2310.03716
20. Sharma et al. Towards Understanding Sycophancy in Language Models. 2023. arXiv:2310.13548
21. Olsson et al. In-context Learning and Induction Heads. 2022. arXiv:2209.11895
22. Casper et al. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. 2023. arXiv:2307.15217
23. Turpin et al. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. 2023. arXiv:2305.04388
24. Wu et al. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. 2023. arXiv:2306.01693
25. Lightman et al. Let's Verify Step by Step. 2023. arXiv:2305.20050