langsmith▌

Development platform for debugging, evaluating, and monitoring language models and AI applications.

Keyword: langsmith · llm tracing · llm evaluation · @traceable · langsmith evaluate

Create, manage, and upload evaluation datasets to LangSmith for testing and validation. \n \n Supports four dataset types: final_response (full conversations), single_step (individual node behavior), trajectory (tool call sequences), and RAG (question/chunks/answer/citations) \n CLI commands for dataset lifecycle management: create, list, get, delete, export, and upload from local JSON files \n SDK-based dataset creation in Python and JavaScript with programmatic example addition \n Example mana

Build evaluation pipelines for LangSmith with LLM-as-Judge and custom code evaluators. \n \n Three core components: creating evaluators (LLM-as-Judge or custom code), defining run functions to capture agent outputs and trajectories, and running evaluations locally or auto-running via uploaded evaluators \n Supports both offline evaluators (comparing run outputs to dataset examples) and online evaluators (real-time quality checks on production runs) \n Requires LangSmith API key and project confi

Add tracing to LangChain/LangGraph apps and query trace data via CLI for debugging and dataset generation. \n \n Automatic tracing for LangChain/LangGraph apps with environment variables; manual tracing via @traceable decorator and wrap_openai() for other frameworks in Python and TypeScript \n Query traces (complete execution trees) or runs (individual nodes) using langsmith CLI with filters for time, latency, errors, tags, and custom metadata \n Export traces to JSONL files preserving hierarchy

Fetch and analyze LangChain and LangGraph execution traces from LangSmith Studio for agent debugging. \n \n Retrieves recent traces and performs root-cause analysis on agent failures, tool calls, memory operations, and performance issues \n Supports multiple output formats (pretty, JSON, raw) and time-based filtering to isolate specific execution windows \n Includes four core workflows: quick debug of recent activity, deep-dive analysis of specific traces, session export with metadata, and error