Systematic evaluation of LLM applications using automated metrics, human feedback, and statistical testing.
Works with
Covers three evaluation approaches: automated metrics (BLEU, ROUGE, BERTScore, accuracy, precision/recall), human evaluation across dimensions like accuracy and coherence, and LLM-as-Judge for pointwise, pairwise, and reference-based scoring
Includes implementations for text generation, classification, and retrieval (RAG) evaluation with ready-to-use metric functions and custom me
AI-first code editor with Composer
Before installing skills in Cursor, ensure your development environment meets these requirements:
node --versionllm-evaluationExecute the skills CLI command in your project's root directory to begin installation:
Fetches llm-evaluation from wshobson/agents and configures it for Cursor.
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Confirm successful installation by checking the skill directory location:
Restart Cursor to activate llm-evaluation. Access via /llm-evaluation in your agent's command palette.
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Submit your Claude Code skill and start earning
Automate repetitive workflows and reduce manual effort
Example
Generate reports, summarize documents, draft communications
Save 3-5 hours per week on routine tasks
Learn new skills, understand complex topics, get expert guidance
Example
Explain concepts, provide examples, suggest learning resources
Accelerate learning and skill development by 2x
Enhance output quality through reviews, suggestions, and refinements
Example
Review drafts, suggest improvements, catch errors
Improve work quality by 30-40% with less effort
0
total installs
0
this week
33.1K
GitHub stars
0
upvotes
Run in your terminal
0
installs
0
this week
33.1K
stars
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
Fast, repeatable, scalable evaluation using computed scores.
Text Generation:
Classification:
Retrieval (RAG):
Manual assessment for quality aspects difficult to automate.
Dimensions:
Use stronger LLMs to evaluate weaker model outputs.
Approaches:
from dataclasses import dataclass
from typing import Callable
import numpy as np
@dataclass
class Metric:
name: str
fn: Callable
@staticmethod
def accuracy():
return Metric("accuracy", calculate_accuracy)
@staticmethod
def bleu():
return Metric("bleu", calculate_bleu)
@staticmethod
def bertscore():
return Metric("bertscore", calculate_bertscore)
@staticmethod
def custom(name: str, fn: Callable):
return Metric(name, fn)
class EvaluationSuite:
def __init__(self, metrics: list[Metric]):
self.metrics = metrics
async def evaluate(self, model, test_cases: list[dict]) -> dict:
results = {m.name: [] for m in self.metrics}
for test in test_cases:
prediction = await model.predict(test["input"])
for metric in self.metrics:
score = metric.fn(
prediction=prediction,
reference=test.get("expected"),
context=test.get("context")
)
results[metric.name].append(score)
return {
"metrics": {k: np.mean(v) for k, v in results.items()},
"raw_scores": results
}
# Usage
suite = EvaluationSuite([
Metric.accuracy(),
Metric.bleu(),
Metric.bertscore(),
Metric.custom("groundedness", check_groundedness)
])
test_cases = [
{
"input": "What is the capital of France?",
"expected": "Paris",
"context": "France is a country in Europe. Paris is its capital."
},
]
results = await suite.evaluate(model=your_model, test_cases=test_cases)
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
def calculate_bleu(reference: str, hypothesis: str, **kwargs) -> float:
"""Calculate BLEU score between reference and hypothesis."""
smoothie = SmoothingFunction().method4
return sentence_bleu(
[reference.split()],
hypothesis.split(),
smoothing_function=smoothie
)
from rouge_score import rouge_scorer
def calculate_rouge(reference: str, hypothesis: str, **kwargs) -> dict:
"""Calculate ROUGE scores."""
scorer = rouge_scorer.RougeScorer(
['rouge1', 'rouge2', 'rougeL'],
use_stemmer=True
)
scores = scorer.score(reference, hypothesis)
return {
'rouge1': scores['rouge1'].fmeasure,
'rouge2': scores['rouge2'].fmeasure,
'rougeL': scores['rougeL'].fmeasure
}
from bert_score import score
def calculate_bertscore(
references: list[str],
hypotheses: list[str],
**kwargs
) -> dict:
"""Calculate BERTScore using pre-trained model."""
P, R, F1 = score(
hypotheses,
references,
lang='en',
model_type='microsoft/deberta-xlarge-mnli'
)
return {
'precision': P.mean().item(),
'recall': R.mean().item(),
'f1': F1.mean().item()
}
def calculate_groundedness(response: str, context: str, Prerequisites
Time Estimate
15-45 minutes depending on use case complexity
Steps
Common Pitfalls
✓ Do
✗ Don't
💡 Pro Tips
✓ Use when
Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.
✗ Avoid when
Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.
wshobson/agents
wshobson/agents
wshobson/agents
am-will/codex-skills
davila7/claude-code-templates
intellectronica/agent-skills
Registry listing for llm-evaluation matched our evaluation — installs cleanly and behaves as described in the markdown.
Solid pick for teams standardizing on skills: llm-evaluation is focused, and the summary matches what you get after install.
We added llm-evaluation from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
llm-evaluation is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
llm-evaluation reduced setup friction for our internal harness; good balance of opinion and flexibility.
llm-evaluation has been reliable in day-to-day use. Documentation quality is above average for community skills.
Useful defaults in llm-evaluation — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
llm-evaluation fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
I recommend llm-evaluation for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.
Useful defaults in llm-evaluation — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
showing 1-10 of 36