> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/huggingface/trl/llms.txt
> Use this file to discover all available pages before exploring further.

# Reward Trainer

> Outcome-supervised Reward Modeling (ORM) trainer for training reward models used in RLHF pipelines.

## Overview

Reward models (RMs) are trained to assign scalar scores to model outputs, reflecting how well a response aligns with human preferences. A well-trained reward model can then be used to guide online RL methods (such as PPO, GRPO, or RLOO) or to evaluate model outputs at inference time.

`RewardTrainer` trains a `AutoModelForSequenceClassification` model (with `num_labels=1`) on a preference dataset using a Bradley-Terry pairwise ranking objective. The model learns to assign higher scores to preferred responses than to rejected ones.

## Quick start

```python theme={null}
from trl import RewardTrainer
from datasets import load_dataset

trainer = RewardTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()
```

## Dataset format

`RewardTrainer` requires a preference dataset with `chosen` and `rejected` fields. An optional `prompt` field is supported. Both standard and conversational formats are accepted.

```python theme={null}
# Standard preference — implicit prompt
{"chosen": "The sky is blue.",
 "rejected": "The sky is green."}

# Conversational preference — implicit prompt
{"chosen": [{"role": "user", "content": "What color is the sky?"},
            {"role": "assistant", "content": "It is blue."}],
 "rejected": [{"role": "user", "content": "What color is the sky?"},
              {"role": "assistant", "content": "It is green."}]}

# Standard preference — explicit prompt
{"prompt": "The sky is",
 "chosen": " blue.",
 "rejected": " green."}

# Conversational preference — explicit prompt
{"prompt": [{"role": "user", "content": "What color is the sky?"}],
 "chosen": [{"role": "assistant", "content": "It is blue."}],
 "rejected": [{"role": "assistant", "content": "It is green."}]}
```

To convert from a different format:

```python theme={null}
from datasets import load_dataset
import json

dataset = load_dataset("lmarena-ai/arena-human-preference-55k")

# Filter out ties
dataset = dataset.filter(lambda example: example["winner_tie"] == 0)

def make_conversation(example):
    prompt = json.loads(example["prompt"])[0]
    chosen = json.loads(example["chosen"])[0]
    rejected = json.loads(example["rejected"])[0]
    return {
        "chosen": [{"role": "user", "content": prompt}, {"role": "assistant", "content": chosen}],
        "rejected": [{"role": "user", "content": prompt}, {"role": "assistant", "content": rejected}],
    }

dataset = dataset.map(make_conversation)
dataset = dataset.select_columns(["chosen", "rejected"])
```

## How reward modeling works

Under the Bradley-Terry model, the probability that response `y⁺` is preferred over `y⁻` is:

```
p(y⁺ ≻ y⁻ | x) = σ(r(x, y⁺) − r(x, y⁻))
```

The reward model is trained with the negative log-likelihood of observed preferences:

```
L(θ) = -E[ log σ(r_θ(x, y⁺) − r_θ(x, y⁻)) ]
```

<Tip>
  The Bradley-Terry model is underdetermined — adding a constant to all rewards does not change preference probabilities. Use `center_rewards_coefficient` to encourage mean-zero rewards, which helps with reward hacking.
</Tip>

## Key configuration parameters

<AccordionGroup>
  <Accordion title="Training">
    <ParamField path="center_rewards_coefficient" type="float | None">
      Coefficient for an auxiliary loss term that encourages the model to output mean-zero rewards. Recommended value: `0.01`. Addresses the underdetermination of the Bradley-Terry model.
    </ParamField>

    <ParamField path="activation_offloading" type="bool" default="false">
      Offload activations to CPU to reduce GPU memory usage.
    </ParamField>

    <ParamField path="disable_dropout" type="bool" default="true">
      Disable dropout in the model during training. Recommended to improve consistency of reward estimates.
    </ParamField>
  </Accordion>

  <Accordion title="Data preprocessing">
    <ParamField path="max_length" type="int | None" default="1024">
      Maximum tokenized sequence length. Samples where either `chosen` or `rejected` exceeds this length are filtered out.
    </ParamField>

    <ParamField path="eos_token" type="str | None">
      Token used to indicate end of sequence. Defaults to the tokenizer's `eos_token`.
    </ParamField>

    <ParamField path="pad_token" type="str | None">
      Token used for padding. Defaults to `processing_class.pad_token`, falling back to `eos_token`.
    </ParamField>
  </Accordion>

  <Accordion title="Model initialization">
    <ParamField path="model_init_kwargs" type="dict | None">
      Keyword arguments forwarded to `AutoModelForSequenceClassification.from_pretrained` when the `model` argument is a string. Note: `num_labels` is always set to 1 automatically and cannot be overridden here.
    </ParamField>

    <ParamField path="chat_template_path" type="str | None">
      Path to a tokenizer or Jinja template file to set as the model's chat template.
    </ParamField>
  </Accordion>
</AccordionGroup>

<Note>
  `RewardConfig` overrides some `TrainingArguments` defaults: `logging_steps=10`, `gradient_checkpointing=True`, `bf16=True`, and `learning_rate=1e-4`.
</Note>

## Training with PEFT/LoRA

When fine-tuning a base causal LM as a reward model using LoRA, include the classification head (`score`) in `modules_to_save`:

```python theme={null}
from datasets import load_dataset
from trl import RewardTrainer
from peft import LoraConfig

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = RewardTrainer(
    "Qwen/Qwen3-4B",
    train_dataset=dataset,
    peft_config=LoraConfig(modules_to_save=["score"]),
)
trainer.train()
```

<Warning>
  When training a reward model adapter on a base causal LM (not a sequence classification model), you must include `"score"` in `modules_to_save`. This ensures the classification head is trained and saved alongside the adapter.
</Warning>

To continue training an existing PEFT reward model:

```python theme={null}
from datasets import load_dataset
from trl import RewardTrainer
from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained("trl-lib/Qwen3-4B-Reward-LoRA", is_trainable=True)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = RewardTrainer(
    model=model,
    train_dataset=dataset,
)
trainer.train()
```

<Tip>
  When training reward model adapters, use a higher learning rate (around `1e-3`) since only new parameters are being learned.
</Tip>

## Using the reward model in an RLHF pipeline

After training, use the reward model as the `reward_funcs` argument in online RL trainers:

```python theme={null}
from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs="path/to/reward-model",  # HuggingFace Hub ID or local path
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
trainer.train()
```

You can also combine a trained reward model with custom reward functions to create a hybrid reward signal.

## Logged metrics

| Metric        | Description                                                    |
| ------------- | -------------------------------------------------------------- |
| `accuracy`    | Proportion of examples where chosen reward > rejected reward   |
| `loss`        | Average Bradley-Terry loss                                     |
| `margin`      | Average reward margin (chosen minus rejected)                  |
| `mean_reward` | Average reward score across both chosen and rejected responses |
| `min_reward`  | Minimum reward score (averaged over the logging interval)      |
| `max_reward`  | Maximum reward score (averaged over the logging interval)      |