# TRL - Transformers Reinforcement Learning

## Docs

- [Introduction](https://mintlify.wiki/huggingface/trl/introduction.md): TRL is a comprehensive library for post-training foundation models using supervised fine-tuning, preference optimization, and reinforcement learning techniques.
- [Installation](https://mintlify.wiki/huggingface/trl/installation.md): Install TRL from PyPI or from source. Requires Python 3.10 or later.
- [Quickstart](https://mintlify.wiki/huggingface/trl/quickstart.md): Get started with TRL in minutes. Train your first model using SFTTrainer, GRPOTrainer, DPOTrainer, or RewardTrainer.
- [Dataset formats](https://mintlify.wiki/huggingface/trl/dataset-formats.md): An overview of the dataset formats and types supported by TRL trainers, and utilities for converting between them.
- [Training methods](https://mintlify.wiki/huggingface/trl/training-methods.md): An overview of the training methods and trainers available in TRL, organized by method type.
- [SFT Trainer](https://mintlify.wiki/huggingface/trl/sft-trainer.md): Supervised Fine-Tuning (SFT) trainer for adapting language models to follow instructions and complete text tasks.
- [GRPO Trainer](https://mintlify.wiki/huggingface/trl/grpo-trainer.md): Group Relative Policy Optimization (GRPO) trainer for reinforcement learning from reward functions, including training reasoning models.
- [DPO Trainer](https://mintlify.wiki/huggingface/trl/dpo-trainer.md): Direct Preference Optimization (DPO) trainer for aligning language models to human preferences without a separate reward model.
- [Reward Trainer](https://mintlify.wiki/huggingface/trl/reward-trainer.md): Outcome-supervised Reward Modeling (ORM) trainer for training reward models used in RLHF pipelines.
- [RLOO Trainer](https://mintlify.wiki/huggingface/trl/rloo-trainer.md): REINFORCE Leave-One-Out (RLOO) trainer for efficient online RL alignment, offering a simpler alternative to PPO.
- [Training customization](https://mintlify.wiki/huggingface/trl/customization.md): Customize TRL training with optimizers, schedulers, callbacks, evaluation metrics, mixed precision, and gradient accumulation.
- [Reducing memory usage](https://mintlify.wiki/huggingface/trl/reducing-memory.md): Techniques to reduce GPU memory consumption during TRL training, including truncation, packing, PEFT, Liger Kernel, gradient checkpointing, and more.
- [Speeding up training](https://mintlify.wiki/huggingface/trl/speeding-up-training.md): Accelerate TRL training with vLLM generation, optimized attention kernels, Liger Kernel, and mixed precision.
- [Distributing training](https://mintlify.wiki/huggingface/trl/distributing-training.md): Scale TRL training across multiple GPUs and nodes using Accelerate, DeepSpeed ZeRO, FSDP, sequence parallelism, and multi-node setups.
- [Use a model after training](https://mintlify.wiki/huggingface/trl/use-model.md): Load a fine-tuned TRL model for inference, merge LoRA adapters, push to the Hugging Face Hub, and run text generation.
- [Command line interface](https://mintlify.wiki/huggingface/trl/clis.md): Launch TRL training jobs from the command line using trl sft, trl dpo, trl grpo, and other commands with inline flags or YAML config files.
- [PEFT Integration](https://mintlify.wiki/huggingface/trl/peft-integration.md): Use Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and QLoRA with TRL trainers to reduce memory requirements and training costs.
- [vLLM Integration](https://mintlify.wiki/huggingface/trl/vllm-integration.md): Accelerate online RL training by offloading generation to a vLLM server, eliminating the slow generation bottleneck in methods like GRPO and RLOO.
- [DeepSpeed Integration](https://mintlify.wiki/huggingface/trl/deepspeed-integration.md): Scale TRL training across multiple GPUs and nodes using DeepSpeed ZeRO optimizer state, gradient, and parameter partitioning.
- [Liger Kernel Integration](https://mintlify.wiki/huggingface/trl/liger-kernel-integration.md): Speed up LLM training and reduce GPU memory usage with Liger Kernel's optimized Triton kernels, compatible with FlashAttention, FSDP, and DeepSpeed.
- [Callbacks](https://mintlify.wiki/huggingface/trl/api/callbacks.md): Trainer callbacks for TRL, including BEMA, reference model sync, completion logging, rich progress display, and Weave integration.
- [Reward Functions](https://mintlify.wiki/huggingface/trl/api/rewards.md): Built-in reward functions for use with GRPOTrainer and RLOOTrainer, including accuracy, reasoning accuracy, format, and overlong-punishment rewards.
- [Data Utilities](https://mintlify.wiki/huggingface/trl/api/data-utils.md): Functions for preparing, converting, packing, and unpacking datasets for TRL trainers.
- [Script Utilities](https://mintlify.wiki/huggingface/trl/api/script-utils.md): Argument dataclasses, parser, dataset loading, and logging helpers for TRL training scripts.
- [Chat Template Utilities](https://mintlify.wiki/huggingface/trl/api/chat-template-utils.md): Utilities for cloning chat templates between models, adding response schemas, and obtaining training-compatible prefix-preserving templates.

## OpenAPI Specs

- [openapi](https://mintlify.wiki/huggingface/trl/api-reference/openapi.json)