Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.
Fetches grpo-rl-training from davila7/claude-code-templates and configures it for Cursor.
3
Select Cursor when prompted
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
◆ Which agents do you want to install to?
│
│ ── Universal (.agents/skills) ────────────────
│ · Cline · Codex · Goose · Windsurf
│ ●Cursor(selected)
│ · Cursor · Aider · Continue
4
Verify installation
Confirm successful installation by checking the skill directory location:
.cursor/skills/grpo-rl-training
Restart Cursor to activate grpo-rl-training. Access via /grpo-rl-training in your agent's command palette.
⚠
Security Notice
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.
When to Use This Skill
Use GRPO training when you need to:
Enforce specific output formats (e.g., XML tags, JSON, structured reasoning)
Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking)
Improve reasoning capabilities by rewarding chain-of-thought patterns
Align models to domain-specific behaviors without labeled preference data
Optimize for multiple objectives simultaneously (format + correctness + style)
When you already have high-quality preference pairs (use DPO/PPO instead)
Core Concepts
1. GRPO Algorithm Fundamentals
Key Mechanism:
Generates multiple completions for each prompt (group size: 4-16)
Compares completions within each group using reward functions
Updates policy to favor higher-rewarded responses relative to the group
Critical Difference from PPO:
No separate reward model needed
More sample-efficient (learns from within-group comparisons)
Simpler to implement and debug
Mathematical Intuition:
For each prompt p:
1. Generate N completions: {c₁, c₂, ..., cₙ}
2. Compute rewards: {r₁, r₂, ..., rₙ}
3. Learn to increase probability of high-reward completions
relative to low-reward ones in the same group
2. Reward Function Design Philosophy
Golden Rules:
Compose multiple reward functions - Each handles one aspect (format, correctness, style)
Scale rewards appropriately - Higher weight = stronger signal
Use incremental rewards - Partial credit for partial compliance
Test rewards independently - Debug each reward function in isolation
Reward Function Types:
Type
Use Case
Example Weight
Correctness
Verifiable tasks (math, code)
2.0 (highest)
Format
Strict structure enforcement
0.5-1.0
Length
Encourage verbosity/conciseness
0.1-0.5
Style
Penalize unwanted patterns
-0.5 to 0.5
Implementation Workflow
Step 1: Dataset Preparation
Critical Requirements:
Prompts in chat format (list of dicts with 'role' and 'content')
Include system prompts to set expectations
For verifiable tasks, include ground truth answers as additional columns
Example Structure:
from datasets import load_dataset, Dataset
SYSTEM_PROMPT ="""
Respond in the following format:
<reasoning>
[Your step-by-step thinking]
</reasoning>
<answer>
[Final answer]
</answer>
"""defprepare_dataset(raw_data):"""
Transform raw data into GRPO-compatible format.
Returns: Dataset with columns:
- 'prompt': List[Dict] with role/content (system + user messages)
- 'answer': str (ground truth, optional but recommended)
"""return raw_data.map(lambda x:{'prompt':[{'role':'system','content': SYSTEM_PROMPT},{'role':'user','content': x['question']}],'answer': extract_answer(x['raw_answer'])})
Pro Tips:
Use one-shot or few-shot examples in system prompt for complex formats
Validate data quality before training (garbage in = garbage out)
Step 2: Reward Function Implementation
Template Structure:
defreward_function_name( prompts,# List[List[Dict]]: Original prompts completions,# List[List[Dict]]: Model generations answer=None,# Optional: Ground truth from dataset**kwargs # Additional dataset columns)->list[float]:"""
Evaluate completions and return rewards.
Returns: List of floats (one per completion)
"""# Extract completion text responses =[comp[0]['content']for comp in completions]# Compute rewards rewards =[]for response in responses: score = compute_score(response) rewards.append(score)return rewards
Example 1: Correctness Reward (Math/Coding)
defcorrectness_reward(prompts, completions, answer,**kwargs):"""Reward correct answers with high score.""" responses =[comp[0]['content']for comp in completions] extracted =[extract_final_answer(r)for r in responses]return[2.0if ans == gt else0.0for ans, gt inzip(extracted, answer)]
Example 2: Format Reward (Structured Output)
import re
defformat_reward(completions,**kwargs):"""Reward XML-like structured format.""" pattern =r'<reasoning>.*?</reasoning>\s*<answer>.*?</answer>' responses =[comp[0]['content']for comp in completions]return[1.0if re.search(pattern, r, re.DOTALL)else0.0for r in responses]
Example 3: Incremental Format Reward (Partial Credit)
defincremental_format_reward(completions,**kwargs):"""Award partial credit for format compliance.""" responses =[comp[0]['content']for comp in completions] rewards =[]for r in responses: score =0.0if'<reasoning>'in r: score +=0.25if'</reasoning>'in r: score +=0.25if'<answer>'in r: score +=0.25if'</answer>'in r: score +=0.25# Penalize extra text after closing tagif r.count('</answer>')==1: extra_text = r.split('</answer>')[-1].strip() score -=len(extra_text)*0.001 rewards.append(score)return rewards
Critical Insight:
Combine 3-5 reward functions for robust training. Order matters less than diversity of signals.
›Claude Desktop or compatible AI client with skill support
›Clear understanding of task or problem to solve
›Willingness to iterate and refine outputs
Time Estimate
15-45 minutes depending on use case complexity
Steps
1Install skill using provided installation command
2Test with simple use case relevant to your work
3Evaluate output quality and relevance
4Iterate on prompts to improve results
5Integrate into regular workflow if valuable
Common Pitfalls
⚠Expecting perfect results without iteration
⚠Not providing enough context in prompts
⚠Using skill for tasks outside its intended scope
⚠Accepting outputs without review and validation
Best Practices
✓ Do
+Start with clear, specific prompts
+Provide relevant context and constraints
+Review and refine all outputs before using
+Iterate to improve output quality
+Document successful prompt patterns
✗ Don't
−Don't use without understanding skill limitations
−Don't skip validation of outputs
−Don't share sensitive information in prompts
−Don't expect skill to replace human judgment
💡 Pro Tips
★Be specific about desired format and style
★Ask for multiple options to choose from
★Request explanations to understand reasoning
★Combine AI efficiency with human expertise
When to Use This
✓ Use when
Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.
✗ Avoid when
Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.
Learning Path
1Familiarize yourself with skill capabilities and limitations
2Start with low-risk, non-critical tasks
3Progress to more complex and valuable use cases
4Build expertise through regular use and experimentation
