hamelsmu/evals-skills▌

Calibrate an LLM judge against human judgment.

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.

Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.

Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.