Productivity

langsmith-evaluator

langchain-ai/langsmith-skills · updated Apr 8, 2026

$npx skills add https://github.com/langchain-ai/langsmith-skills --skill langsmith-evaluator
summary

Build evaluation pipelines for LangSmith with LLM-as-Judge and custom code evaluators.

  • Three core components: creating evaluators (LLM-as-Judge or custom code), defining run functions to capture agent outputs and trajectories, and running evaluations locally or auto-running via uploaded evaluators
  • Supports both offline evaluators (comparing run outputs to dataset examples) and online evaluators (real-time quality checks on production runs)
  • Requires LangSmith API key and project confi
skill.md

langsmith-evaluator

No content available.