GenAI PMLangsmith
A developer/evaluation tool cited in benchmark testing of automated eval systems. The newsletter uses it as part of a comparison against harder-to-detect product-judgment failures.
Key Highlights
- LangSmith is positioned as a platform for tracing, evaluating, and operating AI agents and LLM applications.
- Newsletter coverage ties LangSmith closely to LangChain, Harrison Chase, and deep-agent infrastructure workflows.
- Its observability model emphasizes traces, threads, and trajectories to support debugging, memory, and iterative improvement.
- Benchmarks cited in the newsletter suggest LangSmith can catch many obvious failures but may miss nuanced product-judgment errors.
- For AI PMs, LangSmith is most useful when combined with human review, strong rubrics, and cross-functional evaluation processes.
LangSmith
Overview
LangSmith is a tool and platform used for building, tracing, evaluating, and operating AI agent and LLM applications. In the newsletter, it appears both as a developer workflow product—covering observability, sandboxed prototyping, deployments, and managed infrastructure—and as an evaluation tool used in benchmark comparisons against other automated eval systems. It is closely associated with Harrison Chase and the LangChain ecosystem.For AI Product Managers, LangSmith matters because it sits at the intersection of agent development, evaluation, and operational feedback loops. The mentions portray it not just as a debugging layer for traces, but as an org-wide system for collaboration on AI products: aligning engineers, PMs, evaluators, and domain experts around shared telemetry, test harnesses, and improvement cycles. At the same time, benchmark commentary in the newsletter highlights an important caveat: automated eval platforms like LangSmith can catch many obvious failures, but may still miss nuanced product-judgment failures unless paired with stronger rubrics and human review.
Key Developments
- 2026-05-10: Harrison Chase described LangSmith as an org-wide platform for building AI agents, emphasizing faster cross-functional collaboration and tighter feedback loops.
- 2026-05-24: LangSmith was cited as one of the tools already using LLM-as-a-judge patterns to automate evaluation and speed prompt iteration, alongside OpenAI Evals and PromptLayer Evaluations.
- 2026-06-01: Harrison Chase shared how to evaluate DeepAgents at scale on AWS with LangSmith, including datapoint and evaluator design for longer-horizon agents.
- 2026-06-04: LangSmith was presented with three core components: Sandbox for isolated prototyping, LLM Gateway for unified model access, and Observability for end-to-end monitoring.
- 2026-06-29: Chase announced Harbor, a LangChain/LangSmith integration for running sandboxed evaluations, with self-hosted sandboxes planned.
- 2026-07-12: LangSmith was described as offering cloud-based sandboxes and deployments, deep-agent orchestration, and observability tracing; it was also framed as integrating with hundreds of LangChain models and powering recursive improvement workflows.
- 2026-07-30: In a demo of openwiki, Harrison Chase showed LangSmith traces being parsed from coding agents to automatically update codebase documentation through scheduled background jobs.
- 2026-08-09: Chase explained Managed Deep Agents, which combined a deep agents harness with managed LangSmith infrastructure.
- 2026-08-15: New documentation introduced a clearer mental model for traces, threads, and trajectories, with trajectories presented as a new concept. Chase noted that observability data can support memory and learning.
- 2026-08-24: Hamel’s benchmark compared automated eval tools including LangSmith, Braintrust Loop, and Arize Alex. The finding: these systems recovered many obvious failures but often missed harder product-judgment failures, such as unhandled sales objections, and showed best-case precision in the 80%–90% range.
Relevance to AI PMs
1. Operationalize evaluation beyond demos. LangSmith is relevant if your team needs a structured way to trace agent behavior, run evaluations, and compare versions over time. PMs can use it to turn vague quality concerns into testable failure modes and measurable regressions.2. Improve cross-functional feedback loops. The newsletter repeatedly frames LangSmith as a collaboration layer, not just a developer utility. PMs can use traces and observability artifacts to align engineering, design, support, and subject-matter reviewers on what actually happened in an agent run and where the product broke down.
3. Manage the limits of automated evals. The benchmark mention is especially practical for PMs: LangSmith may catch obvious errors at scale, but nuanced product failures can still slip through. Tactically, that means PMs should combine platform-generated eval signals with human review, sharper rubrics, and business-specific judgment criteria.
Related
- LangChain: LangSmith is tightly linked to the LangChain ecosystem and was described as integrating with hundreds of LangChain models and workflows.
- Harrison Chase: The primary individual associated with LangSmith across the newsletter mentions; he shared launches, documentation, and demos.
- DeepAgents / Managed Deep Agents / agent-harnesses: LangSmith appears as infrastructure and evaluation support for longer-horizon, multi-step agent systems.
- agent-observability / traces / threads / trajectories: These concepts are central to LangSmith’s product framing, especially for debugging, memory, and learning loops.
- OpenAI Evals / PromptLayer Evaluations / llm-as-a-judge: These are adjacent evaluation approaches and tools used for comparison in discussions of automated AI evaluation.
- Harbor: A LangChain/LangSmith integration focused on sandboxed evaluations.
- openwiki: A demo project that used LangSmith traces from coding agents to automatically maintain documentation.
- AWS: Mentioned in the context of evaluating DeepAgents at scale with LangSmith.
- Braintrust Loop / Arize Alex / Hamel: Relevant comparators and evaluators in benchmark discussions that tested how well automated eval systems detect real product failures.
Newsletter Mentions (17)
“Hamel’s benchmark found automated-eval tools such as Braintrust Loop, Arize Alex, and LangSmith recovered many obvious failures but missed product-judgment failures such as unhandled sales objections; their best-case precision was stated as 80% to 90%, meaning 10% to 20% of flagged errors were not actual errors.”
#1 ▶️ How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel Peter Yang Shreya uses the free Error Discovery Skill in Claude Code with Opus 4.8 to turn human feedback on AI-generated writing into annotated failure modes, an evaluation rubric, and reusable LLM-judge criteria. The Error Discovery Skill performs five steps: identifies the dataset’s semantic type, designs visual encoding, builds an HTML/Python review app, clusters data and selects diverse samples, then uses an interactive feedback loop to propose new samples and rubric criteria. In the live run, Claude took about 15 minutes to build a three-tab review interface with article-by-article, map/clustering, and progress views; after feedback was supplied, it generated 361 suggested annotations, including 249 instances of the “less than four words staccato” rule. Hamel’s benchmark found automated-eval tools such as Braintrust Loop, Arize Alex, and LangSmith recovered many obvious failures but missed product-judgment failures such as unhandled sales objections; their best-case precision was stated as 80% to 90%, meaning 10% to 20% of flagged errors were not actual errors.
“Harrison Chase shared new LangSmith documentation covering traces, threads, and trajectories—a new concept—and said observability data can support memory and learning, making a clear mental model especially helpful.”
#13 𝕏 Harrison Chase shared new LangSmith documentation covering traces, threads, and trajectories—a new concept—and said observability data can support memory and learning, making a clear mental model especially helpful.
“#3 𝕏 Harrison Chase shared a 20-minute explanation of Managed Deep Agents, which he said had launched the previous day and combines a deep agents harness with managed LangSmith infrastructure.”
GenAI PM Daily August 09, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 10 insights for PM Builders. Claude Code sessions can now message each other #3 𝕏 Harrison Chase shared a 20-minute explanation of Managed Deep Agents, which he said had launched the previous day and combines a deep agents harness with managed LangSmith infrastructure.
“Harrison Chase demos openwiki, a “dreaming” memory–powered wiki that runs scheduled background jobs to parse LangSmith traces of coding agents and automatically update your codebase documentation.”
#10 𝕏 Harrison Chase demos openwiki, a “dreaming” memory–powered wiki that runs scheduled background jobs to parse LangSmith traces of coding agents and automatically update your codebase documentation. #11 𝕏 Guillermo Rauch built a harness that adds vision capabilities to every AI model, enabling them to process and interpret images seamlessly.
“Harrison Chase launched LangSmith, offering cloud-based sandboxes & deployments, deep‐agent orchestration, and observability tracing. It integrates with hundreds of LangChain models and powers recursive improvement via the LangSmith engine.”
#7 𝕏 Harrison Chase launched LangSmith, offering cloud-based sandboxes & deployments, deep‐agent orchestration, and observability tracing. It integrates with hundreds of LangChain models and powers recursive improvement via the LangSmith engine.
“#4 𝕏 Harrison Chase announces Harbor, a LangChain/LangSmith integration for running sandboxed evaluations, with self-hosted sandboxes coming soon.”
The newsletter mentions Harbor as a LangChain/LangSmith integration aimed at sandboxed evaluations.
“#19 𝕏 Harrison Chase unveiled LangSmith with three core components—Sandbox for isolated prototyping, LLM Gateway for unified model access, and Observability tools for end-to-end monitoring.”
GenAI PM Daily June 04, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 25 insights for PM Builders, ranked by relevance from Blogs, X, YouTube, and LinkedIn. Google launches Gemma 4 12B for local multi-step reasoning #19 𝕏 Harrison Chase unveiled LangSmith with three core components—Sandbox for isolated prototyping, LLM Gateway for unified model access, and Observability tools for end-to-end monitoring.
“Harrison Chase breaks down how to evaluate DeepAgents at scale on AWS with LangSmith, covering concrete datapoint and evaluator design methods for longer-horizon agents.”
#2 𝕏 Harrison Chase breaks down how to evaluate DeepAgents at scale on AWS with LangSmith, covering concrete datapoint and evaluator design methods for longer-horizon agents.
“Using an LLM to evaluate another (LLM-as-a-judge) lets teams automate large-scale evaluation and speed up prompt iteration from days to minutes, and is already used in tools like OpenAI Evals, LangSmith, and PromptLayer Evaluations.”
#7 📝 PromptLayer Blog LLM as a Judge: How Do You Know If Your AI Is Actually Good? - Using an LLM to evaluate another (LLM-as-a-judge) lets teams automate large-scale evaluation and speed up prompt iteration from days to minutes, and is already used in tools like OpenAI Evals, LangSmith, and PromptLayer Evaluations. However, judges inherit model biases—preferring longer answers, producing inconsistent or phrasing-sensitive scores—so reliable evaluation needs detailed rubrics and mixed signals (heuristics, human review, structured checks), which PromptLayer offers as a first-class feature.
“#7 𝕏 Harrison Chase calls LangSmith an org-wide platform for building AI agents that speeds up cross-functional collaboration and tightens feedback loops.”
GenAI PM Daily May 10, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 11 insights for PM Builders, ranked by relevance from X, Blogs, and LinkedIn. PromptLayer’s multi-step agent evaluation framework #1 𝕏 Jason Zhou launched `/goal` support in CodeX and Hermes agents for one-step autonomous coding, advising use of interview mode, clear stop conditions, and a goal-buddy to manage state and goal files. #2 📝 PromptLayer Blog What Is Agent Evaluation? A Practical Guide for AI Teams - Agent evaluation tests whether an AI agent reliably completes tasks across real inputs, edge cases, and new versions by scoring not just final outputs but multi-step behavior via black-box, trajectory, and component-level evaluations, using metrics like task completion rate, tool selection accuracy, unsupported-claim rate, latency/cost per step, and regression pass rate. PromptLayer offers tracing with span-level context, reusable datasets, batch evaluations, backtesting, regression testing, automated evaluation triggers on new prompt versions, and flexible pipelines including code execution, human input, conversation simulation, regex checks, and LLM assertions. #3 in Udi Menkes built his new product’s entire data flow in a single interactive HTML file—complete with diagrams, in-page navigation, and color-coded complexity—letting his team understand it in minutes instead of hours. #4 𝕏 Garry Tan suggests diagramming your AI agent codebases and architecture in plain ASCII, then relentlessly questioning each component to clarify design and accelerate product development. #5 𝕏 Boris Cherny says Claude Code’s switch to a native installer means npm-only stats undercount its real usage. On Thursday it hit its second-highest signup day ever with 15× growth since Jan 1—now you can ask Claude to debug your SQL. #6 𝕏 Boris Cherny is enhancing Claude Code’s UX for snappier performance and adding debug logs so users can self-serve hang diagnostics. #7 𝕏 Harrison Chase calls LangSmith an org-wide platform for building AI agents that speeds up cross-functional collaboration and tightens feedback loops. #8 𝕏 Santiago showcases a step-by-step guide for constructing Python-powered multi-agent systems from scratch, leveraging MCP and A2A patterns to incrementally add complexity and enable collaborative AI agents. #9 𝕏 Garry Tan spends $2K/mo on Openclaw AI tokens to turbocharge product development and startup insights. He’s “tokenmaxxing” now with a goal to make these capabilities affordable for everyone in 18 months. #10 𝕏 Harrison Chase argues that treating AI agents as systems to measure and iteratively improve isn’t just a technical challenge—it demands intentional human collaboration and team processes. #11 in Peter Yang warns that unedited AI-generated markdown can compound small errors over time—what starts as 5% “slop” quickly balloons into an overwhelming pile of confusing, unverified content. Found this valuable? Share it with another PM - they can subscribe at genaipm.com Unsubscribe • Switch to Weekly
Related
Founder and builder in the AI agents ecosystem, associated here with explaining deepagents’ architecture and its use of LangGraph. Relevant to PMs for understanding agent-loop/backend separation and deployment patterns.
A framework for building LLM applications and agents. In this newsletter it appears in the story about the founders’ attempt to automate dropshipping.
Autonomous or semi-autonomous AI systems that use tools, manage context, and complete tasks on behalf of users. The newsletter discusses common blockers such as tool quality, context overload, and system verification.
An agent framework/architecture that separates the agent loop from backend operations like filesystem access and optional sandboxed code execution. Useful for building agents with flexible local/cloud deployment and multiple interfaces.
Amazon’s cloud platform, referenced in a story about a training pipeline running on a 4-GPU instance. The anecdote highlights GPU utilization monitoring and infrastructure waste.
Autonomous software agents that write, maintain, and redesign code systems. For PMs, they represent a shift in how engineering and research work gets allocated.
Stay updated on Langsmith
Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.
Subscribe Free