> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/huggingface/trl/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Get started with TRL in minutes. Train your first model using SFTTrainer, GRPOTrainer, DPOTrainer, or RewardTrainer.

TRL provides dedicated trainer classes for every stage of the post-training pipeline. Each trainer is a lightweight wrapper around the Hugging Face `Trainer` and supports distributed training out of the box.

## Install TRL

```bash theme={null}
pip install trl
```

## Trainers

<Steps>
  <Step title="Supervised Fine-Tuning with SFTTrainer">
    `SFTTrainer` is the starting point for most post-training workflows. It fine-tunes a model on a dataset of demonstrations.

    ```python theme={null}
    from trl import SFTTrainer
    from datasets import load_dataset

    dataset = load_dataset("trl-lib/Capybara", split="train")

    trainer = SFTTrainer(
        model="Qwen/Qwen2.5-0.5B",
        train_dataset=dataset,
    )
    trainer.train()
    ```

    See the [SFT Trainer](/sft-trainer) docs for options like dataset packing, chat templates, and LoRA.
  </Step>

  <Step title="Reinforcement learning with GRPOTrainer">
    `GRPOTrainer` implements [Group Relative Policy Optimization (GRPO)](https://huggingface.co/papers/2402.03300) — a memory-efficient RL algorithm used to train DeepSeek-R1. It generates groups of completions and optimizes them against a reward function.

    ```python theme={null}
    from datasets import load_dataset
    from trl import GRPOTrainer
    from trl.rewards import accuracy_reward

    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    trainer = GRPOTrainer(
        model="Qwen/Qwen2.5-0.5B-Instruct",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
    )
    trainer.train()
    ```

    <Note>
      For reasoning models, use the `reasoning_accuracy_reward()` function for better results.
    </Note>

    See the [GRPO Trainer](/grpo-trainer) docs for reward function configuration and vLLM integration.
  </Step>

  <Step title="Preference alignment with DPOTrainer">
    `DPOTrainer` implements [Direct Preference Optimization (DPO)](https://huggingface.co/papers/2305.18290), which trains the model directly on preference pairs without a separate reward model. DPO was used to post-train Llama 3 and many other models.

    ```python theme={null}
    from datasets import load_dataset
    from trl import DPOTrainer

    dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

    trainer = DPOTrainer(
        model="Qwen/Qwen2.5-0.5B-Instruct",
        train_dataset=dataset,
    )
    trainer.train()
    ```

    See the [DPO Trainer](/dpo-trainer) docs for reference model configuration and loss variants.
  </Step>

  <Step title="Reward modeling with RewardTrainer">
    `RewardTrainer` trains a scalar reward model on preference data. Reward models are used as the reward signal for online RL methods like GRPO and RLOO.

    ```python theme={null}
    from trl import RewardTrainer
    from datasets import load_dataset

    dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

    trainer = RewardTrainer(
        model="Qwen/Qwen2.5-0.5B-Instruct",
        train_dataset=dataset,
    )
    trainer.train()
    ```

    See the [Reward Trainer](/reward-trainer) docs for dataset format and evaluation.
  </Step>
</Steps>

## Command Line Interface

The `trl` CLI lets you run fine-tuning jobs directly from your terminal without writing any Python code.

**SFT — supervised fine-tuning:**

```bash theme={null}
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
    --dataset_name trl-lib/Capybara \
    --output_dir Qwen2.5-0.5B-SFT
```

**DPO — preference alignment:**

```bash theme={null}
trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO
```

**Reward modeling:**

```bash theme={null}
trl reward --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name trl-lib/ultrafeedback_binarized
```

Run `trl --help` or any subcommand with `--help` to see all available options. See the [CLI docs](/clis) for the full reference.

## Troubleshooting

### Out of memory

Reduce batch size and accumulate gradients to maintain an effective batch size:

<CodeGroup>
  ```python SFT theme={null}
  from trl import SFTConfig

  training_args = SFTConfig(
      per_device_train_batch_size=1,
      gradient_accumulation_steps=8,
  )
  ```

  ```python DPO theme={null}
  from trl import DPOConfig

  training_args = DPOConfig(
      per_device_train_batch_size=1,
      gradient_accumulation_steps=8,
  )
  ```
</CodeGroup>

For more aggressive memory reduction, install PEFT and enable LoRA:

```bash theme={null}
pip install "trl[peft,quantization]"
```

See the [memory optimization guide](/reducing-memory) and [PEFT integration](/peft-integration) for details.

### Loss not decreasing

A learning rate that is too high or too low is a common cause. A good starting point for fine-tuning:

```python theme={null}
from trl import SFTConfig

training_args = SFTConfig(learning_rate=2e-5)
```

For more help, open an [issue on GitHub](https://github.com/huggingface/trl/issues).

## Next steps

<CardGroup cols={2}>
  <Card title="SFT Trainer" icon="graduation-cap" href="/sft-trainer">
    Full guide to supervised fine-tuning: packing, chat templates, and LoRA
  </Card>

  <Card title="GRPO Trainer" icon="trophy" href="/grpo-trainer">
    Group Relative Policy Optimization for reasoning and RL alignment
  </Card>

  <Card title="Distributed training" icon="server" href="/distributing-training">
    Scale to multi-GPU and multi-node with DeepSpeed and FSDP
  </Card>

  <Card title="PEFT integration" icon="memory" href="/peft-integration">
    Train large models on consumer hardware with LoRA and QLoRA
  </Card>
</CardGroup>