> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/huggingface/trl/llms.txt
> Use this file to discover all available pages before exploring further.

# Reducing memory usage

> Techniques to reduce GPU memory consumption during TRL training, including truncation, packing, PEFT, Liger Kernel, gradient checkpointing, and more.

Training workflows can often be optimized to reduce memory consumption. TRL provides several built-in features to help achieve this.

Experiment with different combinations to find the configuration that works best for your setup. Each method below includes examples for the supported trainers.

## Truncation

Sequences in a dataset can vary widely in length. When batched, all sequences are padded to the longest one in the batch, which can cause high memory usage even when most sequences are short.

To reduce this, truncate sequences to a reasonable length using the `max_length` parameter.

**DPO**

DPO truncation is controlled via `max_length`, which truncates the combined prompt+completion sequence.

```python theme={null}
from trl import DPOConfig

training_args = DPOConfig(..., max_length=512)
```

> [!WARNING]
>
> The legacy `max_prompt_length` and `max_completion_length` parameters have been removed. Filter or pre-truncate overlong prompts and completions in your dataset before training.

**SFT**

SFT truncation is applied to the full input sequence via the `max_length` parameter.

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(..., max_length=512)
```

### Choosing the right `max_length`

* Too small: important tokens at the end of sequences are discarded, reducing training quality.
* Too large: memory spikes and potential out-of-memory (OOM) errors. Without packing or padding-free batching, many tokens will also be padding.

Visualize the sequence length distribution in your dataset to pick an appropriate value before training.

## Packing

> [!TIP]
>
> Packing is available only for **SFT** training and setups that use **FlashAttention** or its variants.

Packing mitigates truncation drawbacks by grouping multiple sequences into the same training row, filling each row up to `max_length`. TRL uses Best-Fit Decreasing (BFD) bin packing to group sequences efficiently.

Three strategies are supported:

> **bfd (default)**
>
> Uses Best-Fit Decreasing packing. If a sequence exceeds `max_length`, the overflow tokens are discarded.

> **bfd\_split**
>
> Uses Best-Fit Decreasing packing, but long sequences are split into chunks of at most `max_length` before packing. This preserves all tokens.

> **wrapped**
>
> All tokens are concatenated into a stream and split into fixed-length blocks. Minimizes padding but may mix unrelated examples, which can hurt performance.

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(
    ...,
    packing=True,
    packing_strategy="bfd",
    max_length=512,
)
```

> [!NOTE]
>
> If all sequences are shorter than `max_length`, `bfd` and `bfd_split` behave identically since no truncation or splitting is required.

## PEFT for parameter-efficient fine-tuning

Methods like LoRA train only a small number of adapter parameters instead of all model weights, significantly reducing memory requirements and enabling fine-tuning of larger models on limited hardware.

```python theme={null}
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train")

peft_config = LoraConfig()

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
    peft_config=peft_config,
)
```

PEFT can be combined with quantization (4-bit or 8-bit) for even greater memory savings. See the [PEFT Integration](/peft-integration) guide for quantization examples.

## Liger Kernel

[Liger Kernel](https://github.com/linkedin/Liger-Kernel) is a collection of Triton kernels for LLM training that can increase multi-GPU throughput by 20% and reduce memory usage by 60%.

**SFT**

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(..., use_liger_kernel=True)
```

**DPO**

```python theme={null}
from trl import DPOConfig

training_args = DPOConfig(..., use_liger_kernel=True)
```

**GRPO**

```python theme={null}
from trl import GRPOConfig

training_args = GRPOConfig(..., use_liger_kernel=True)
```

**KTO**

```python theme={null}
from trl.experimental.kto import KTOConfig

training_args = KTOConfig(..., use_liger_kernel=True)
```

**GKD**

```python theme={null}
from trl.experimental.gkd import GKDConfig

training_args = GKDConfig(..., use_liger_kernel=True)
```

For more details, see the [Liger Kernel Integration](/liger-kernel-integration) guide.

## Padding-free batching

Padding-free batching flattens a batch into a single sequence, avoiding padding entirely. Unlike packing, it keeps all sequences complete and intact.

> [!WARNING]
>
> Use padding-free batching with **FlashAttention 2** or **FlashAttention 3**. Without it, you may encounter batch contamination.

**DPO**

```python theme={null}
from trl import DPOConfig

training_args = DPOConfig(
    ...,
    padding_free=True,
    model_init_kwargs={"attn_implementation": "kernels-community/flash-attn2"},
)
```

**SFT**

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(
    ...,
    padding_free=True,
    model_init_kwargs={"attn_implementation": "kernels-community/flash-attn2"},
)
```

## Activation offloading

Activation offloading reduces GPU VRAM usage by moving activation tensors to CPU RAM during the forward pass and bringing them back during the backward pass. It reduces peak memory at the cost of slightly increased training time.

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(..., activation_offloading=True)
```

Under the hood, TRL uses PyTorch's `saved_tensors_hooks` to intercept activations during the forward pass. CUDA streams are used by default to overlap computation with CPU-GPU transfers.

## Gradient checkpointing

Gradient checkpointing trades compute for memory by recomputing intermediate activations during the backward pass instead of storing them.

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(..., gradient_checkpointing=True)
```

> [!NOTE]
>
> Gradient checkpointing is enabled by default in all TRL trainers. Disable it by setting `gradient_checkpointing=False` if needed.

For more memory optimization techniques, see the [Transformers performance guide](https://huggingface.co/docs/transformers/perf_train_gpu_one#gradient-checkpointing).

## Disabling model gathering for generation (DeepSpeed ZeRO-3)

When using DeepSpeed ZeRO-3, model weights are sharded across GPUs. Online methods gather weights onto a single GPU for generation, which can cause OOM errors on large models.

Disable gathering with:

**GRPO**

```python theme={null}
from trl import GRPOConfig

training_args = GRPOConfig(..., ds3_gather_for_generation=False)
```

**RLOO**

```python theme={null}
from trl import RLOOConfig

training_args = RLOOConfig(..., ds3_gather_for_generation=False)
```

**Online DPO**

```python theme={null}
from trl.experimental.online_dpo import OnlineDPOConfig

training_args = OnlineDPOConfig(..., ds3_gather_for_generation=False)
```

**PPO**

```python theme={null}
from trl.experimental.ppo import PPOConfig

training_args = PPOConfig(..., ds3_gather_for_generation=False)
```

This avoids OOM errors but may result in slower generation speeds.

## vLLM sleep mode

When using vLLM as the generation backend for online training, enable sleep mode to offload vLLM parameters and cache to CPU RAM during the optimization step and reload them when needed.

**GRPO**

```python theme={null}
from trl import GRPOConfig

training_args = GRPOConfig(..., vllm_enable_sleep_mode=True)
```

**RLOO**

```python theme={null}
from trl import RLOOConfig

training_args = RLOOConfig(..., vllm_enable_sleep_mode=True)
```

Offloading keeps GPU memory usage low, particularly useful when training large models or using limited GPU resources. Waking the vLLM engine from sleep mode introduces some host–device transfer latency.

## Padding sequences to a multiple

> [!TIP]
>
> This technique is currently supported for **SFT** and **Reward** trainers.

Pad all sequences to a multiple of a given value to improve computational efficiency on hardware that benefits from aligned sequence lengths.

**SFT**

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(..., pad_to_multiple_of=2048)
```

**Reward**

```python theme={null}
from trl import RewardConfig

training_args = RewardConfig(..., pad_to_multiple_of=2048)
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.