GenAI PMLangsmith
LangChain’s evaluation and observability product, referenced in the context of trajectory labeling. Useful for PMs instrumenting agent traces and evals.
Key Highlights
- Langsmith is positioned as LangChain’s evaluation and observability product for agent and LLM workflows.
- It is repeatedly referenced for trace review, trajectory labeling, messages replay, and nuanced eval workflows.
- Newsletter mentions tie Langsmith to practical PM concerns including cost tracking, harness optimization, and managed deep-agent infrastructure.
- The tooling appears useful not just for engineers but for cross-functional teams that need product-facing visibility into agent behavior.
- Benchmarks cited in the newsletter suggest Langsmith can catch obvious failures, while still requiring human product judgment for harder edge cases.
At a glance
- Category
- Evaluation/Observability
Langsmith
Overview
Langsmith, also referred to as LangSmith, is LangChain’s evaluation and observability product for tracing, reviewing, and improving agent and LLM workflows. In the supplied references, it appears most often in the context of trajectory labeling, trace inspection, eval design, and measurement for coding agents and deep-agent systems. For AI Product Managers, Langsmith matters because it turns raw agent behavior into inspectable artifacts such as traces, threads, messages, and trajectories that teams can use to debug failures, understand cost drivers, and improve product quality.The newsletter mentions also position Langsmith as more than a logging tool. It is described as part of a broader improvement loop: tracking usage and cost, powering evaluation workflows, enabling custom trace-review interfaces, and serving as managed infrastructure for deep agents. For PMs working with agentic products, this makes Langsmith relevant for instrumentation, cross-functional review, and setting up repeatable evals tied to real product behavior.
Key Developments
- 2026-07-12: Harrison Chase launched LangSmith, describing it as offering cloud-based sandboxes and deployments, deep-agent orchestration, and observability tracing. It was said to integrate with hundreds of LangChain models and support recursive improvement via the LangSmith engine.
- 2026-07-30: Harrison Chase demoed openwiki, a memory-powered wiki that parses LangSmith traces from coding agents to automatically update codebase documentation.
- 2026-08-09: Harrison Chase said Managed Deep Agents had launched and combined a deep agents harness with managed LangSmith infrastructure.
- 2026-08-15: Harrison Chase shared new LangSmith documentation covering traces, threads, and trajectories, and noted that observability data can support memory and learning.
- 2026-08-24: Hamel’s benchmark grouped LangSmith with Braintrust Loop and Arize Alex as automated-eval tools that caught many obvious failures but still missed some product-judgment failures; stated best-case precision was 80% to 90%.
- 2026-09-03: Harrison Chase shared that LangSmith’s Messages View helps teams debug agents by replaying conversations and tool calls as the agent experienced them.
- 2026-09-10: Harrison Chase said a loop was being built around deepagents for orchestration logic and LangSmith for measurement, reinforcing its role in iterative agent improvement.
- 2026-10-03: Harrison Chase said agent teams want their own trace review UI, and that LangSmith Custom Apps lets users build one on their own data and ship it directly into the workspace.
- 2026-10-05: Harrison Chase shared that coding-agent costs fell for a second straight month after tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness.
- 2026-10-10: In LangSmith evals, LangChain’s Jev assessed difficulty and correctness with separate typed answers in one pass on every trace; Harrison Chase framed trajectory labeling as involving multiple questions instead of a single pass/fail judgment.
Relevance to AI PMs
1. Instrument agent behavior in a product-relevant way. Langsmith gives PMs a structured way to inspect traces, threads, messages, and trajectories so they can move from anecdotal bug reports to concrete failure patterns in real workflows.2. Build evals that reflect product quality, not just model quality. The mentions around trajectory labeling, typed judgments, and one-pass trace assessment suggest Langsmith can support more nuanced evaluation criteria such as correctness, difficulty, and failure mode analysis.
3. Manage cost and improvement loops across teams. Langsmith shows up in examples of usage tracking, cost reduction, harness optimization, and managed deep-agent infrastructure, making it useful for PMs coordinating engineering, applied AI, and operations around measurable iteration.
Related
- LangChain: Langsmith is described as LangChain’s evaluation and observability product and integrates with LangChain models and workflows.
- Harrison Chase: Frequently cited as the source of product updates, launch notes, and positioning around LangSmith.
- deepagents / managed-deep-agents: Langsmith is paired with deepagents for measurement and appears as managed infrastructure in the Managed Deep Agents launch.
- agent-harnesses / OpenSWE / coding-agents: Langsmith is used to track usage, inspect traces, and support optimization of coding-agent harnesses.
- traces, threads, trajectories: These are core conceptual objects in LangSmith’s observability model and are especially relevant for PMs designing review and evaluation workflows.
- llm-as-a-judge / Jev / openai-evals / promptlayer-evaluations: Langsmith sits in the broader eval tooling landscape and is referenced specifically in connection with typed judgments on traces.
- Braintrust Loop / Arize Alex / Hamel: These provide useful comparison points in automated evaluation; the mentions suggest LangSmith shares strengths in catching obvious failures but still requires product judgment.
- openwiki: Demonstrates a downstream use case where LangSmith traces power automated documentation updates.
- agent-observability / ai-agents / agent-and-llm-pipelines: Langsmith is directly relevant anywhere teams need visibility into agent execution and iterative improvement.
Frequently asked questions
What is Langsmith?
LangChain’s evaluation and observability product, referenced in the context of trajectory labeling. Useful for PMs instrumenting agent traces and evals.
What should AI product managers know about Langsmith?
Langsmith is positioned as LangChain’s evaluation and observability product for agent and LLM workflows. It is repeatedly referenced for trace review, trajectory labeling, messages replay, and nuanced eval workflows. Newsletter mentions tie Langsmith to practical PM concerns including cost tracking, harness optimization, and managed deep-agent infrastructure.
Explore this topic
Newsletter Mentions (22)
“In LangSmith evals, LangChain’s Jev assesses difficulty and correctness with separate typed answers in one pass on every trace.”
Harrison Chase commented that trajectory labeling involves several questions rather than a single pass/fail judgment. In LangSmith evals, LangChain’s Jev assesses difficulty and correctness with separate typed answers in one pass on every trace.
“Harrison Chase shared that coding-agent costs fell significantly for a second month in a row after three steps: tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness with techniques such as model routing.”
#6 𝕏 Harrison Chase shared that coding-agent costs fell significantly for a second month in a row after three steps: tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness with techniques such as model routing.
“Harrison Chase commented that agent teams want their own trace review UI, and that LangChain’s LangSmith Custom Apps lets users build one on their own data and ship it directly into the workspace.”
#6 𝕏 Harrison Chase commented that agent teams want their own trace review UI, and that LangChain’s LangSmith Custom Apps lets users build one on their own data and ship it directly into the workspace.
“He said a loop is being built around deepagents for orchestration logic and LangSmith for measurement.”
#8 𝕏 Harrison Chase endorsed the view that agent improvement is harness improvement, with key interventions at the tool boundary: context selection, tool availability, failure recovery, and measurement. He said a loop is being built around deepagents for orchestration logic and LangSmith for measurement.
“Harrison Chase shared that LangSmith’s Messages View helps builders debug agents by replaying conversations and tool calls as the agent experienced them, making traces useful beyond infrastructure teams.”
Harrison Chase shared that LangSmith’s Messages View helps builders debug agents by replaying conversations and tool calls as the agent experienced them, making traces useful beyond infrastructure teams.
“Hamel’s benchmark found automated-eval tools such as Braintrust Loop, Arize Alex, and LangSmith recovered many obvious failures but missed product-judgment failures such as unhandled sales objections; their best-case precision was stated as 80% to 90%, meaning 10% to 20% of flagged errors were not actual errors.”
#1 ▶️ How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel Peter Yang Shreya uses the free Error Discovery Skill in Claude Code with Opus 4.8 to turn human feedback on AI-generated writing into annotated failure modes, an evaluation rubric, and reusable LLM-judge criteria. The Error Discovery Skill performs five steps: identifies the dataset’s semantic type, designs visual encoding, builds an HTML/Python review app, clusters data and selects diverse samples, then uses an interactive feedback loop to propose new samples and rubric criteria. In the live run, Claude took about 15 minutes to build a three-tab review interface with article-by-article, map/clustering, and progress views; after feedback was supplied, it generated 361 suggested annotations, including 249 instances of the “less than four words staccato” rule. Hamel’s benchmark found automated-eval tools such as Braintrust Loop, Arize Alex, and LangSmith recovered many obvious failures but missed product-judgment failures such as unhandled sales objections; their best-case precision was stated as 80% to 90%, meaning 10% to 20% of flagged errors were not actual errors.
“Harrison Chase shared new LangSmith documentation covering traces, threads, and trajectories—a new concept—and said observability data can support memory and learning, making a clear mental model especially helpful.”
#13 𝕏 Harrison Chase shared new LangSmith documentation covering traces, threads, and trajectories—a new concept—and said observability data can support memory and learning, making a clear mental model especially helpful.
“#3 𝕏 Harrison Chase shared a 20-minute explanation of Managed Deep Agents, which he said had launched the previous day and combines a deep agents harness with managed LangSmith infrastructure.”
GenAI PM Daily August 09, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 10 insights for PM Builders. Claude Code sessions can now message each other #3 𝕏 Harrison Chase shared a 20-minute explanation of Managed Deep Agents, which he said had launched the previous day and combines a deep agents harness with managed LangSmith infrastructure.
“Harrison Chase demos openwiki, a “dreaming” memory–powered wiki that runs scheduled background jobs to parse LangSmith traces of coding agents and automatically update your codebase documentation.”
#10 𝕏 Harrison Chase demos openwiki, a “dreaming” memory–powered wiki that runs scheduled background jobs to parse LangSmith traces of coding agents and automatically update your codebase documentation. #11 𝕏 Guillermo Rauch built a harness that adds vision capabilities to every AI model, enabling them to process and interpret images seamlessly.
“Harrison Chase launched LangSmith, offering cloud-based sandboxes & deployments, deep‐agent orchestration, and observability tracing. It integrates with hundreds of LangChain models and powers recursive improvement via the LangSmith engine.”
#7 𝕏 Harrison Chase launched LangSmith, offering cloud-based sandboxes & deployments, deep‐agent orchestration, and observability tracing. It integrates with hundreds of LangChain models and powers recursive improvement via the LangSmith engine.
Related
Co-founder and CEO of LangChain, cited discussing trajectory labeling and evaluation design. Relevant to PMs working on evals and agent traces.
The company behind LangSmith, mentioned in connection with trajectory labeling and evaluation workflows. Relevant to AI PMs building agent monitoring stacks.
Autonomous or semi-autonomous workflows that perform tasks, coordinate tools, and execute actions on behalf of users.
An agent framework that now supports dynamically loading tools when a skill is loaded. This pattern helps keep toolsets modular and can preserve prompt cache efficiency.
Agents used to write, review, and iterate on code as part of software development workflows. The newsletter frames them as shifting developers toward specification, architecture, and evaluation work.
Amazon’s cloud platform, referenced in a story about a training pipeline running on a 4-GPU instance. The anecdote highlights GPU utilization monitoring and infrastructure waste.
Stay updated on Langsmith
Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.
Subscribe Free