Per-rollout weighting in RL on Tessera-27B-B-CJK: a pre-registered ablation

In short: one small change to how RL weights each attempt kept the model from drifting during training. On hard problems it also looked more efficient, but that didn't hold up on easier ones. This post covers what we did, what worked, what didn't, and why.

Update, 2026-11-03. We ran per-rollout weighting for ten more updates and checked the result against a release rule we wrote down first. It answered clearly more hard problems correctly than Tessera-27B-B-CJK, but it thought longer per agent turn than our limit allowed, so we are not releasing it. Section 8 has every result.

(open full-size image in a new tab)

Abstract

In the Tessera-27B-B-CJK paper [1] we described a failed attempt at reinforcement learning (RL). After five updates that policy looped more, rambled to the token cap more often, and had lost log-probability on its own tokens. We suspected the way the loss weighted each rollout. Every token of a rollout carried the same weight, so a rollout's total weight grew with its length, and the longest rollouts were mostly failures cut off at the cap. Within each group of rollouts, the Tinker cookbook's default RL recipe weights rollouts the same way.

This study tests that idea. Starting from the released Tessera-27B-B-CJK, we trained five arms for five updates each and changed one thing per arm. We wrote down the plan, the stop rules and the verdict rules before any training. Weighting each rollout by its advantage and not by its length avoided the drift in both seeds we ran. Against the token-weighted control on the same problems and seeds, it kept 0.20 and 0.18 nats per token more log-probability on its sampled tokens. On the training problems it finished with more answers that were correct, finished and loop-free, and used 0.69x and 0.75x the starting model's tokens per correct answer. In an exploratory check on 24 hard math and code problems from earlier evaluations, it got 83 of 144 rollouts right against the starting model's 71, used 0.78x the tokens per correct answer (90% confidence interval, CI, 0.63-0.94), and kept the compressed CJK notation.

Three results limit that. The control damaged the model's answers in one seed but not the other, so the planned contrast did not replicate. A blinded read passed 8 of 12 answer pairs where the plan required 10, so the held-out gain is unverified. On 24 easier problems we had not used before, the per-rollout model stayed within the planned accuracy margin (108 of 144 against 111) but used 1.02x the starting model's tokens per correct answer, so that check did not confirm the gain. In these short runs per-rollout weighting avoided the drift and passed every damage check we ran. The old weighting damaged the answers in only one of its two seeds, though, so we cannot yet show that the change prevents damage. But the drift came with the damage in our first RL attempt and in the control's damaged seed, and it never appeared with per-rollout weighting in this study. We have also not shown that the change makes the model more efficient in general.

1. The failure we started from

Tessera-27B-B-CJK is a rank-32 LoRA on Qwen3.8-27B that thinks in a compressed CJK notation and uses about 0.50-0.64x the base model's thinking tokens per agentic turn. We reached it with supervised steps and a final rejection-sampling fine-tune on the model's own shortest correct rollouts. Before that last step we tried RL at learning rate 2e-5, with a reward for correct answers first and fewer tokens second. Over five updates the median over rollouts of their mean thinking-token log-probability fell from -0.49 to -0.77. We then sampled the checkpoint from before RL on the problems and seeds of the rollouts for updates 4 and 5. The RL policy had 51 clean rollouts of 108 there, against 76 before RL, with more repetition loops and more rambles to the 32,768-token cap. A clean rollout is correct, finished and free of repetition loops. Section 3 gives the slightly stricter rule used in this study.

Before release we also tried a positive-only variant from the released checkpoint. Only rollouts with a positive advantage entered the loss, each weighted per rollout as in arm N below, with a KL (Kullback-Leibler) term toward the start that could only damp those positive weights, at learning rate 1e-5. After nine updates its summary metrics on the training problems looked benign, though its looping rollouts there had risen from 7 to 15 of 348 between the first and second pass over the cells. On the held-out set of section 4.3 its looping rollouts rose from 3 to 20 of 144. That is why this study counts looping rollouts in its damage rules and checks loops on held-out problems.

Tinker supported this work with a research grant and asked for RL ablations aimed at that failure. We asked which single change removes the damage and still shortens the thinking.

2. Weighting tokens or weighting rollouts

In policy-gradient RL each sampled token gets a weight. The gradient pushes the log-probability of tokens with positive weight up and of tokens with negative weight down, though tokens share parameters, so one update need not move every token that way. With a group of rollouts per problem, the usual advantage is a rollout's reward minus the group's mean reward. The open choice is how to spread that advantage over the rollout's tokens.

Token weighting gives every token of rollout $i$ the full advantage $A_i$. A rollout with $L_i$ thinking tokens then carries a total weight of $A_i L_i$, so length multiplies its influence. Per-rollout weighting gives every token $A_i / L_i$, so each rollout carries a total weight of $A_i$ whatever its length. Since the advantages in a group sum to zero, the positive and negative weight balance in every update.

In code, the only change is one division.

# token weighting, our control C. Rollout weight grows with its length.
w = [A[i] * s0 / L_mean[g]] * L[i]
# per-rollout weighting, arm N. Rollout weight is its advantage.
w = [A[i] * s0 / L[i]] * L[i]

Here s0 is one fixed scale for every run, 4,790, the mean thinking length of the starting model's untruncated rollouts on the training problems. L_mean[g] is the mean thinking length in rollout i's group, so within a group the control weights rollouts by length. The weights cover the thinking tokens and the token that closes the thinking, not the visible answer. The Tinker cookbook (version 0.5.9) gives every action token its trajectory's group-centered advantage, and Tinker's losses sum over tokens [2, 3]. Within a group, its default therefore weights rollouts by length, as our control does. Tinker's documentation notes that other aggregation schemes go in the advantage tensor, which is what we did.

The RL literature has argued both sides. GRPO averages each response's loss over its own tokens, which is per-rollout weighting [4]. DAPO and Dr. GRPO moved to token-level aggregation, and Dr. GRPO argues that per-response averaging penalizes long wrong answers too little and so lets them grow [5, 6]. In our setting many failures run to a hard token cap, so under token weighting the cut-off failures supply much of the negative weight.

We audited the weights on earlier rollouts before training and logged them at every update of the runs below. In the token-weighted runs, 56-77% of each update's absolute weight was negative, the net weight was negative at every update in both seeds, and truncated rollouts carried 24-62% of the absolute weight. Under per-rollout weighting the net weight is zero by construction, and truncated rollouts carried 11-32% of the absolute weight.

3. Study design

The plan went through several review rounds before training began. Every arm started from the released Tessera-27B-B-CJK LoRA weights with a fresh Adam optimizer and trained for five updates at learning rate 2e-5, the rate that damaged the first attempt. A cell is one problem at one reasoning-effort setting of the Qwen chat template, low, medium or xhigh.

The arms change one thing each.

After five updates each arm sampled all 58 cells six times (348 rollouts) with shared seeds. We call this the readout and compare it with a sweep of the starting model on the same cells. Log-probability here is each rollout's mean log-probability over its sampled thinking tokens, averaged over rollouts. Top-20 entropy is the entropy of the 20 most likely next tokens, which held 98-99.6% of the probability mass in every readout. A run counted as damaged if, against the sweep, it lost at least 21 clean rollouts of 348, gained at least 20 truncated or looping ones, gained at least 12 looping ones, or lost at least 0.15 nats per token of sampled-token log-probability. Three stop rules could end a run early, and none fired in any run.

The control ran first, then the four other arms. We then repeated the control and the first stable arm in plan order with new sampling, shuffle and readout seeds. The plan put held-out testing, a blinded read and a check on never-used problems in a second phase that required the replication to succeed. Section 4.2 explains why we ran them anyway and how we labelled them.

4. Results

4.1 Five arms on the training problems

Table 1. Readouts after five updates, 348 rollouts per row on the 58 training cells. Changes are against the starting model's sweep in the same seed family. Tokens per correct is total generated tokens divided by correct answers, as a ratio to the sweep, with a 90% CI from resampling cells.

(open full-size image in a new tab)

In the main seed the control was damaged. It lost 25 clean rollouts, truncated 62 against the sweep's 44, lost 0.205 nats per token, and needed 1.20x the tokens per correct answer. The clean margin is narrow (25 against a rule of 21), and the log-probability rule holds on its own. All four other arms were stable. Per-rollout weighting alone (N) had the most clean rollouts (40 more), the fewest truncations (16) and the lowest tokens per correct answer (0.69x). The other arms ran in one seed each, and the plan does not rank stable arms, so we report them as they came out, XP at 0.75x, XK at 0.89x and X at 0.96x. Handling truncated rollouts separately or adding the KL anchor did not beat N's point estimates within five updates. XP, which adds a net push toward finishing, drifted the other way, gaining 0.079 nats per token with top-20 entropy down to 0.33. In XK the KL term was 6-11% of the absolute weight after the first update.

4.2 The replication

The pre-registered outcome is NOT_REPLICATED. With new seeds the control stayed inside every damage rule. It gained 22 clean rollouts and lost 0.147 nats per token, just inside the 0.15 limit. Per-rollout weighting was stable again, with 45 more clean rollouts and 0.75x the tokens per correct answer.

The drift did replicate. Within each seed family, C and N trained on the same cells at each update and started from the same first rollouts, which pairs their per-update results by cell. Figure 1 shows it. In both seeds C's sampled-token log-probability fell further below N's at every update, reaching 0.20 and 0.18 nats per token at the readout, and its top-20 entropy rose further above N's, to 0.18 and 0.16. The weight audit in section 2 held in both seeds too. Whether that drift damaged the answers within five updates depended on the seed. The control's worst result here, 7 percentage points fewer clean rollouts, is also much milder than the first attempt's 23 points from an earlier checkpoint.

One gain did not replicate. In the main seed N's clean answers were shorter than the starting model's, with a geometric mean of per-cell length ratios of 0.75x (90% CI 0.71-0.80). In the replication that ratio was 0.97x (90% CI 0.91-1.03), and the efficiency gain came with fewer truncations. These ratios compare the mean clean length L from the reward in each cell, over cells where both models had a clean rollout. In the replication N's median CJK share on the training cells also fell from 0.106 to 0.074. We did not test that checkpoint further.

Because the replication failed, the plan did not allow the second phase. We decided to test the main-seed N checkpoint off the training problems anyway, as a labelled deviation from the plan. The plan would have tested a checkpoint after a 10-update extension, and we tested the 5-update one. Before sampling we wrote down the held-out rules unchanged from the plan. We picked N on the training cells only.

4.3 Held-out problems

The held-out set has 24 problems (10 AIME, 8 LiveCodeBench hard, 6 MATH level 5), each at three efforts with two rollouts, so 144 rollouts per model with a 32,768-token cap. We never trained on these problems, but we used them to evaluate checkpoints earlier in the project, so this set is a reused development set and not a sealed one.

Table 2. Held-out results. The base and starting-model rows are the earlier evaluations with the same seeds and settings. The CJK share is the median fraction of CJK characters in the thinking of clean rollouts.

(open full-size image in a new tab)

The plan's held-out rules asked for at least 63 correct rollouts (the start's 71 minus 8), at most 8 looping and at most 54 truncated or looping ones, a tokens-per-correct ratio whose 90% CI ends below 1, and a median CJK share of at least half the start's. N passed all three. It got 83 of 144 rollouts right against 71 for the starting model, with 2 looping rollouts against 3, where the positive-only variant of section 1 had 20. It used 0.78x the starting model's tokens per correct answer (90% CI 0.63-0.94, resampling problems). Its median thinking was 11.1% CJK characters against 10.1%, so it kept the notation. By source, N got 43 AIME rollouts right against 37, 14 code rollouts against 8, and 26 MATH rollouts against 26. Against the base model it is still 12 correct answers short and needs 1.07x the tokens per correct answer.

The efficiency ratio combines fewer total tokens (0.91x, from fewer tokens spent on failed rollouts) with 17% more correct answers. On the 57 rollouts that both N and the starting model solved, we found no detectable change in length (geometric mean ratio 0.997x, 90% CI 0.91-1.10). That interval does not rule out a change of about 10% in either direction, and it covers only rollouts that both models got right.

4.4 Blinded read

The numbers say nothing about whether N's correct answers reason as completely as the starting model's. We took 12 problems that both models solved on the same seed, one pair per problem. A fixed rule chose them, the four with the longest starting-model thinking at each effort. A judge model saw the problem and both responses as X and Y in random order, with no model names, efforts or token counts. A script kept the key until all 12 ratings were in. A pair passed if both responses gave the same final answer, used the same main method, and each contained the checks the problem requires. The plan required 10 passes. The judge was GPT-6 Astra.

The blinded read failed, with 8 of 12 pairs passing against the 10 required. All 12 gave the same final answer and 11 used the same method. A required check was missing in 3 of N's responses and 2 of the starting model's. In 2 pairs both responses missed one, and in 1 pair only N did. One pair cannot show that N skips checks more often, and the read cannot clear it either. The pre-registered held-out verdict is therefore WIN_UNVERIFIED, a gain on the numeric rules that the blinded read did not verify.

4.5 Never-used problems

The last check used 24 problems that no earlier step had sampled, trained on or evaluated. No AIME problems were left unused, so the set has 2 LiveCodeBench hard, 10 LiveCodeBench medium and 12 MATH level 5 problems, and it is easier than the held-out set. We sampled N and the starting model fresh with the same seeds. The plan ran this check only after a verified held-out gain, so running it after the failed blinded read was a second deviation. We wrote down its rule before sampling. It kept the held-out stability margins, here at least 103 correct, at most 9 looping and at most 24 truncated or looping rollouts, and asked for at most 0.95x the starting model's tokens per correct answer.

Table 3. Never-used problems, 144 rollouts per model.

The verdict is NOT_CONFIRMED. N passed the accuracy and stability margins and failed the efficiency rule, with 1.02x the starting model's tokens per correct answer (90% CI 0.89-1.15). On the 97 rollouts that both models solved, N's were shorter (geometric mean ratio 0.88x, 90% CI 0.79-0.97). We listed that comparison as descriptive and not as a decision rule, and it covers only rollouts that both models got right, so it does not change the NOT_CONFIRMED verdict.

5. What we take from it

The stable result is about drift. In our setting, after each update the token-weighted model had lower log-probability on its own thinking tokens and higher entropy than the per-rollout model trained on the same problems, in both seeds. That matches the weight audit. Under token weighting, long cut-off failures supplied much of the negative weight, and the signed weights summed to a negative number at every update. The loop we suspect is that a drifting policy truncates more, which gives the next update more long failures to weigh. Within five updates that drift came with worse answers in one seed of two, as it had in our first RL attempt.

The efficiency result differed between the two problem sets. On the hard held-out set N got more rollouts right and spent fewer tokens per correct answer, with fewer tokens spent on failed rollouts and no detectable change in the length of jointly correct answers. On the easier never-used set it did neither, while its jointly correct answers were shorter. One explanation is that per-rollout weighting helps most where the starting model often fails at the cap, as it did on the held-out set (46 truncated of 144) and rarely did on the never-used set (16). The two sets also differ in mix. Only the held-out set has AIME problems. Of N's 12 extra correct answers there, 6 came from AIME and 6 from code. We have not tested that explanation, and with one checkpoint on two small sets the difference could also be noise.

If your rollouts vary a lot in length and the long ones are mostly failures, per-rollout weighting is a small change to the advantage tensor. In our runs it avoided the drift, and accuracy stayed within the planned margins on every problem set we checked. Dr. GRPO's argument against it concerns long wrong answers growing over training [6]. Our runs do not test that directly. N truncated less than the starting model on the training cells and the held-out set and about as often on the never-used set, and five updates are too few to rule the effect out.

We hope what we learned here helps us build better models in the future.

6. Limitations

• Each run had five updates at one learning rate. C and N ran in two seeds, and X, XP and XK in one.
• Table 1 measures the 58 training cells. The held-out set is a reused development set, and the never-used set is easier and has no AIME problems.
• The held-out and never-used checks were deviations from the plan, run after the planned replication failed and, for the never-used check, after the blinded read failed. We wrote their rules down before sampling.
• The never-used problems are new to this project. They may appear in Qwen's pretraining data.
• One judge model rated 12 pairs, and no human audited the ratings.
• We did not test N on the agentic coding states from the Tessera paper, so we do not know whether it keeps Tessera's per-turn saving. N is a research checkpoint and we do not release it. Section 8 reports this test for the 15-update checkpoint.

7. Data, code and compute

The plan, the runner, every rollout and the per-update logs are in a private archive. Section 2 shows the weighting change in full, and section 3 gives the other settings. The whole study used about $289 of Tinker compute for its sampling, training and evaluation. The base and starting-model rows on the held-out set came from earlier evaluations.

8. Update, 2026-11-03: ten more updates

We continued arm N from its fifth update for ten more (updates 6-15), on the same 58 training cells with the same settings. No damage check fired and no run paused. Both the 10- and 15-update checkpoints passed the eligibility checks, so a frozen set of 18 problems chose between them by a rule written in advance (54 rollouts each: 47 correct for both, and 2 against 0 cut off or periodic). It chose the 15-update checkpoint, before any evaluation. We had also written down the release rule: score higher than Tessera-27B-B-CJK on our main hard set under a pre-set tier rule, and pass seven guards. We set the tolerances so that a model equal to Tessera-27B-B-CJK would fail each count check in about 1-2.5% of runs, and each safety check we could model in up to 5%. So the guards catch large regressions only, and passing one is not a claim of being no worse. Where Table 4 gives a Tessera-27B-B-CJK number without "earlier run", we sampled both models on the same seeds on the same day.

Table 4. The 15-update checkpoint against Tessera-27B-B-CJK. Counts are out of all rollouts in the set. Periodic means the thinking repeats a fixed block; a strong loop is a second, broader repetition check.

On the main set the checkpoint was ahead on every source (mean of two draws): AIME 17.5 against 15.0 of 18, hard code 26.0 against 21.5 of 40, medium code 18.5 against 17.5 of 20, and level-5 MATH 37.5 against 35.0 of 40. It did better on 20 problems and worse on 6. On the 36 easier problems the paired differences were 16 more correct (90% CI 5-28), 5 more on code (0-10) and 9 fewer periodic or cut-off answers (4-15 fewer). The main set is reused, so this does not show that the gain carries over to new hard problems. For context, the base model Qwen3.8-27B got 98 of 118 on the main set (one draw) and 95 of 144 on the held-out set in earlier runs. The checkpoint got 100 on its first draw and 93, using 0.78x and 0.88x the base model's tokens per correct answer. Those base rows are from October, and the same seeds did not reproduce the same samples across dates, so this comparison is rough. Tessera-27B-B-CJK sampled today got 90 of 118 on the main set's first draw against 87 in October with the same seeds, below our drift flag of 10. On the agentic states it thought 0.76x as much as in October, and none of its 88 same-seed generations came out identical, which is why the main-set, easier-problem and agentic comparisons with Tessera-27B-B-CJK were sampled on the same day. The notation thinned over training, from a median share of 0.112 for Tessera-27B-B-CJK to 0.080 on the training cells, and stayed above our floors.

What failed. Tessera-27B-B-CJK's main claim is that it thinks less per agent turn. On a typical turn the checkpoint used 1.48x its thinking tokens (90% CI 1.26-1.76), against a limit of 1.25x, and summed over all 176 turns it used 2.8x as many (90% CI 2.1-4.7). Against the base model's earlier run on the same states, it used 0.65x the base model's thinking on a typical turn (90% CI 0.46-0.91; Tessera-27B-B-CJK 0.50x) and 0.97x summed over turns (Tessera-27B-B-CJK 0.64x). So it still thinks less than the base model on a typical turn and passes Tessera's own rule for that (at most 0.85x, with the CI below 1), but on the total most of Tessera's saving is gone. Most turns stayed short. In 138 of 176 pairs both models thought for under 100 tokens, and there the checkpoint used 1.13x. The gap comes from 13 turns where the checkpoint planned at length and Tessera-27B-B-CJK did not (the reverse happened 5 times), working out a fix or guessing what the tests check. Those turns did not loop, and every turn made a valid tool call. We did not measure whether the extra planning helps finish tasks. The guard failed, so by our own rule we do not release this checkpoint, and the AIME 2026 check we had planned for a passing model did not run.

Other results. On the 24 easier never-used problems from section 4.5, where the 5-update checkpoint had not confirmed the efficiency gain, it got 122 of 144 right against Tessera-27B-B-CJK's 111 from that earlier run, and used 0.74x the tokens per correct answer (90% CI 0.64-0.83). That set has now been used twice, so this is a report, not a confirmation. In a blinded read of 12 same-problem pairs by one judge model, 9 pairs had the same final answer and the needed checks in both traces, and 8 passed the stricter four-part rule of section 4.4, the same count as before. Ten pairs used the same core method. The needed checks were present in 10 of the checkpoint's traces and 9 of Tessera-27B-B-CJK's. This read had no pass threshold.

In the safety smoke, 5 prompts per category at two reasoning efforts, one blinded judge labelled both models' answers in one batch. Neither model fully complied with a harmful request. The judge labelled one of the checkpoint's answers to a harmful request partial, which our rule does not count as compliance; Tessera-27B-B-CJK had none. On borderline requests the checkpoint gave 6 helpful and 2 partial answers against 7 and 1, neither model over-refused, and 2 answers were cut off for each. All 20 answers to prompt injections flagged the injected instruction. Our rule also counts an answer that repeats the injected command, and 4 of the checkpoint's answers and 2 of Tessera-27B-B-CJK's quoted it while refusing it; the rule fails at a difference of 3, so this check passed by one answer. Every plain answer was judged fine, and no answer had visible CJK. The rules were no full compliance with a harmful request, at most 1 more over-refusal and at most 2 more injection repeats than Tessera-27B-B-CJK, at most 4 more missing answers, and no visible CJK or bad plain answer.

What we are doing next. A follow-up scan on the same agent states, with every model sampled the same day, found that the longer turns begin early. The 5-update checkpoint of sections 4.1-4.5 already used 1.22x Tessera-27B-B-CJK's thinking per typical turn (90% CI 1.10-1.39), and the 10-update checkpoint 1.42x. The medium reasoning-effort setting made agent turns longer, not shorter, for both models. We had not measured updates 6-9, and we did that next. Update, 2026-11-05: the 8-update checkpoint held up on new agent states, and we released it as Tessera-27B-B-CJK-RL. The release post has the details. This follow-up used about $173 of Tinker compute.

Acknowledgments

A research grant from Tinker paid for this work.

References

1. Zellige. Tessera-27B-B-CJK: teaching a 27B agent to think less per turn (2026). /research/tessera-27b-b-cjk
2. Thinking Machines Lab. Tinker cookbook, version 0.5.9, rl/data_processing.py (trajectory_to_data). https://github.com/thinking-machines-lab/tinker-cookbook
3. Thinking Machines Lab. Tinker documentation, loss functions. https://tinker-docs.thinkingmachines.ai/tinker/losses/
4. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
5. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
6. Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783.