> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/huggingface/trl/llms.txt
> Use this file to discover all available pages before exploring further.

# Reward Functions

> Built-in reward functions for use with GRPOTrainer and RLOOTrainer, including accuracy, reasoning accuracy, format, and overlong-punishment rewards.

The `trl.rewards` module provides ready-to-use reward functions primarily intended for [`GRPOTrainer`](/grpo-trainer) and [`RLOOTrainer`](/rloo-trainer). All reward functions share the same calling convention: they receive a batch of completions and return a list of float rewards (or `None` for examples that should be skipped).

Install the optional dependency required by the accuracy rewards:

```bash theme={null}
pip install math_verify
```

Import from the `trl.rewards` sub-package:

```python theme={null}
from trl.rewards import (
    accuracy_reward,
    reasoning_accuracy_reward,
    think_format_reward,
    get_soft_overlong_punishment,
)
```

***

## accuracy\_reward

Checks whether each model completion matches its ground-truth solution using symbolic math verification from the [`math_verify`](https://github.com/huggingface/math_verify) library.

* If both the gold solution and the prediction are parseable LaTeX expressions, `math_verify.verify` is used for comparison.
* If the gold solution **cannot** be parsed, `None` is returned for that example so the trainer can skip it.

<Note>
  Requires the `math_verify` package (`pip install math_verify`). The function detects non-main threads and disables signal-based timeouts automatically to avoid `ValueError`.
</Note>

### Signature

```python theme={null}
def accuracy_reward(
    completions: list[list[dict[str, str]]],
    solution: list[str],
    **kwargs,
) -> list[float | None]
```

### Parameters

<ParamField path="completions" type="list[list[dict[str, str]]]">
  Batch of completions. Each completion is a single-element list containing a message dict with a `"content"` key (the assistant's output text).
</ParamField>

<ParamField path="solution" type="list[str]">
  Batch of raw-text ground-truth solutions corresponding 1-to-1 with `completions`.
</ParamField>

<ParamField path="**kwargs">
  Additional keyword arguments accepted for compatibility with trainer interfaces (e.g., `GRPOTrainer`).
</ParamField>

### Returns

`list[float | None]` — `1.0` if the answer matches, `0.0` if not, or `None` if the gold solution could not be parsed.

### Example

```python theme={null}
from trl.rewards import accuracy_reward

solutions = [r"\frac{1}{3}", r"\frac{1}{3}"]
completions = [
    [{"role": "assistant", "content": r"My answer is \boxed{\frac{1}{3}}"}],
    [{"role": "assistant", "content": r"My answer is \boxed{\frac{1}{2}}"}],
]
print(accuracy_reward(completions, solutions))
# [1.0, 0.0]
```

***

## reasoning\_accuracy\_reward

Variant of `accuracy_reward` designed for **reasoning models** that emit a thinking block before their final answer (e.g., models using `<think>...</think>` tags). The function strips the reasoning section and evaluates only the text that follows the last reasoning delimiter.

* Completions where no reasoning delimiter is found receive a reward of `0.0` (penalizing incomplete reasoning chains).
* Completions where the gold solution is unparseable receive `None` (skip).

### Signature

```python theme={null}
def reasoning_accuracy_reward(
    completions: list[list[dict[str, str]]],
    solution: list[str],
    reasoning_delimiters: list[str] | None = None,
    **kwargs,
) -> list[float | None]
```

### Parameters

<ParamField path="completions" type="list[list[dict[str, str]]]">
  Batch of completions. Each completion is a single-element list containing a message dict with a `"content"` key.
</ParamField>

<ParamField path="solution" type="list[str]">
  Batch of ground-truth solution strings.
</ParamField>

<ParamField path="reasoning_delimiters" type="list[str]" optional>
  List of delimiter strings marking the end of the reasoning block. Defaults to `["</think>"]`. The final answer is taken as the text after the **last** occurrence of any delimiter.
</ParamField>

<ParamField path="**kwargs">
  Additional keyword arguments for trainer compatibility.
</ParamField>

### Returns

`list[float | None]` — `1.0` on correct answer, `0.0` if wrong or reasoning is incomplete, `None` if gold is unparseable.

### Example

```python theme={null}
from trl.rewards import reasoning_accuracy_reward

solutions = [r"\frac{1}{3}", r"\frac{1}{3}", r"\frac{1}{3}"]
completions = [
    [{"role": "assistant", "content": r"<think> Reasoning </think> The answer is \boxed{\frac{1}{3}}"}],
    [{"role": "assistant", "content": r"<think> Reasoning </think> The answer is \boxed{\frac{1}{2}}"}],
    [{"role": "assistant", "content": r"<think> Incomplete reasoning with \boxed{\frac{1}{3}}"}],
]
print(reasoning_accuracy_reward(completions, solutions, reasoning_delimiters=["</think>"]))
# [1.0, 0.0, 0.0]
```

***

## think\_format\_reward

A lightweight format-checking reward that returns `1.0` when the completion correctly wraps its reasoning inside a single `<think>...</think>` block, and `0.0` otherwise.

The regex pattern enforced is:

```
^<think>(?!.*<think>)(.*?)</think>.*$
```

This means the completion must:

* Start with `<think>`.
* Contain exactly one `<think>` opening tag.
* Close with `</think>` before any additional content.

### Signature

```python theme={null}
def think_format_reward(
    completions: list[list[dict[str, str]]],
    **kwargs,
) -> list[float]
```

### Parameters

<ParamField path="completions" type="list[list[dict[str, str]]]">
  Batch of completions. Each element is a single-element list with a message dict containing a `"content"` key.
</ParamField>

<ParamField path="**kwargs">
  Additional keyword arguments for trainer compatibility.
</ParamField>

### Returns

`list[float]` — `1.0` if format is correct, `0.0` otherwise.

### Example

```python theme={null}
from trl.rewards import think_format_reward

completions = [
    [{"content": "<think>\nReasoning here.\n</think>\nFinal answer."}],
    [{"content": "<think>\nReasoning without closing tag."}],
]
print(think_format_reward(completions))
# [1.0, 0.0]
```

***

## get\_soft\_overlong\_punishment

A **factory function** that returns a reward function penalizing completions that exceed a target length. Based on Equation 13 from the [DAPO paper](https://huggingface.co/papers/2503.14476).

The returned reward function applies the following piecewise penalty:

$$
R(y) = \begin{cases}
0 & |y| \le L_{\max} - L_{\text{cache}} \\
\dfrac{(L_{\max} - L_{\text{cache}}) - |y|}{L_{\text{cache}}} & L_{\max} - L_{\text{cache}} < |y| \le L_{\max} \\
-1 & |y| > L_{\max}
\end{cases}
$$

### Signature

```python theme={null}
def get_soft_overlong_punishment(
    max_completion_len: int,
    soft_punish_cache: int,
) -> Callable
```

### Parameters

<ParamField path="max_completion_len" type="int">
  Maximum allowed completion length $L_{\max}$ in tokens.
</ParamField>

<ParamField path="soft_punish_cache" type="int">
  Soft penalty window $L_{\text{cache}}$. Completions in the range $(L_{\max} - L_{\text{cache}},\, L_{\max}]$ receive a linearly interpolated penalty. Set to `0` to apply no minimum-length tolerance.
</ParamField>

### Returns

A callable with signature `(completion_ids: list[list[int]], **kwargs) -> list[float]` suitable for direct use as a reward function in `GRPOTrainer`.

### Example

```python theme={null}
from trl.rewards import get_soft_overlong_punishment

reward_fn = get_soft_overlong_punishment(max_completion_len=100, soft_punish_cache=20)

# Token ids for a completion of length 90 — in the soft penalty zone [80, 100]
completion_ids = [[1] * 90]
print(reward_fn(completion_ids))
# [-0.5]

# Within the safe zone (<= 80)
print(reward_fn([[1] * 70]))
# [0.0]

# Beyond max length (> 100)
print(reward_fn([[1] * 110]))
# [-1.0]
```

### Using with GRPOTrainer

```python theme={null}
from trl import GRPOTrainer, GRPOConfig
from trl.rewards import get_soft_overlong_punishment

overlong_reward = get_soft_overlong_punishment(
    max_completion_len=512,
    soft_punish_cache=64,
)

trainer = GRPOTrainer(
    model=model,
    args=GRPOConfig(output_dir="./output"),
    reward_funcs=[overlong_reward],
    train_dataset=train_dataset,
)
```