GenAI PMHarrison Chase
Co-founder and CEO of LangChain, cited discussing trajectory labeling and evaluation design. Relevant to PMs working on evals and agent traces.
Key Highlights
- Harrison Chase is most relevant to AI PMs as a builder and commentator on agent evals, traces, routing, and harness design.
- He framed trajectory labeling as a multi-question evaluation problem rather than a single pass/fail judgment.
- He shared a practical model-routing workflow focused on task understanding, model understanding, harness implementation, and outcome tracking.
- He linked coding-agent cost reductions to observability in LangSmith, user-level caps via an LLM gateway, and harness optimization.
- He emphasized that agent teams often need custom trace review interfaces and harness-integrated memory and tooling.
At a glance
- Role
- Co-founder and CEO
- Company
- LangChain
Harrison Chase
Overview
Harrison Chase is the co-founder and CEO of LangChain and a frequently cited voice on agent evaluation, harness design, routing, observability, and trace review workflows. In the supplied mentions, he appears most often in connection with practical questions AI teams face when operating agents in production: how to evaluate full trajectories, how to control cost, how to route across models, and how to build tooling around traces rather than only final outputs.For AI Product Managers, Harrison Chase matters because his comments and launches consistently point toward an operational view of agent products. Instead of treating evals as a single pass/fail score, he emphasizes typed judgments across traces, harness-level routing, custom review interfaces, and integrated memory and tool loading. That perspective is especially relevant for PMs responsible for agent reliability, observability, and cost-performance tradeoffs.
Key Developments
- 2026-09-13: Shared LangChain’s "How to Build a Custom Agent Harness" by Sydney Runkle, highlighting the importance of domain-specific harnesses for agent systems.
- 2026-09-16: Commented that agent memory has seen strong interest for about two years without fully sticking, arguing that memory must be tightly integrated with the agent harness and that deciding what to remember is often application-specific.
- 2026-09-21: Described "JEV as a judge" as a cheap, fast semantic verifier useful for evals, especially online evaluations that need to grade many traces.
- 2026-09-22: Announced that SemIf, an open-source alternative to the decision model Jev based on qwen3.5, was being hosted free through LangSmith Gateway for a limited period.
- 2026-10-02: Shared a four-step approach to model routing: understand the tasks, understand the models, build the router in the harness, and track outcomes, with the goal of reducing cost without hurting performance.
- 2026-10-03: Commented that agent teams want their own trace review UI, and pointed to LangSmith Custom Apps as a way to build and ship that directly into the workspace on top of team data.
- 2026-10-04: Shared ModelRouterMiddleware, a LangChain project used in a DSPy and Jev model-router example where model selection can happen at the start of a run and optionally be reselected after tool results.
- 2026-10-05: Shared that coding-agent costs declined significantly for a second straight month after tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness with model routing techniques.
- 2026-10-08: Announced that deepagents now supports dynamically loading tools when a skill is loaded, allowing skill-specific tools without making them always available; with OpenAI or Anthropic models, this can be done without breaking prompt cache behavior.
- 2026-10-10: Commented that trajectory labeling involves several questions rather than a single pass/fail judgment; in LangSmith evals, this showed up as separate typed answers for difficulty and correctness on every trace.
Relevance to AI PMs
1. He offers a practical blueprint for eval design. Harrison Chase’s framing of trajectory labeling as multiple typed questions instead of one binary score is useful for PMs designing agent evals. It suggests breaking evaluation into dimensions like correctness, difficulty, routing quality, and tool use rather than compressing everything into one metric.2. He connects cost control to product instrumentation. The mentions tie lower coding-agent costs to a repeatable operating loop: track usage, enforce cost caps, then optimize routing in the harness. PMs can use this as a tactical model for reducing spend without blindly degrading quality.
3. He treats harnesses, traces, and UI as product surfaces. His comments on custom agent harnesses, custom trace review UIs, dynamic tool loading, and integrated memory all point to a broader lesson: agent product quality depends on infrastructure choices that PMs should actively shape, not just model selection.
Related
- LangChain: The company Harrison Chase co-founded and leads; central to many mentions involving agents, harnesses, routing, and developer tooling.
- LangSmith: Closely connected through discussions of trace review, evals, usage tracking, and custom apps for reviewing agent behavior.
- deepagents / deep-agents: Referenced in connection with dynamic tool loading and skills, relevant to modular agent design.
- Jev / SemIf / trace-judge: Related to evaluation and semantic verification, especially for grading large numbers of traces efficiently.
- ModelRouterMiddleware / model routing / LLM gateway: Connected to cost-performance optimization, user-level spend controls, and harness-level routing strategy.
- Sydney Runkle: Mentioned via the custom agent harness post shared by Harrison Chase.
- OpenSWE / coding agents: Tied to his comments on reducing coding-agent costs through observability and routing optimization.
- Agent memory / traces / observability: Recurring adjacent topics that show his relevance to PMs building reliable long-running agent systems.
Frequently asked questions
What is Harrison Chase?
Co-founder and CEO of LangChain, cited discussing trajectory labeling and evaluation design. Relevant to PMs working on evals and agent traces.
What should AI product managers know about Harrison Chase?
Harrison Chase is most relevant to AI PMs as a builder and commentator on agent evals, traces, routing, and harness design. He framed trajectory labeling as a multi-question evaluation problem rather than a single pass/fail judgment. He shared a practical model-routing workflow focused on task understanding, model understanding, harness implementation, and outcome tracking.
Explore this topic
Newsletter Mentions (92)
“Harrison Chase commented that trajectory labeling involves several questions rather than a single pass/fail judgment.”
Harrison Chase commented that trajectory labeling involves several questions rather than a single pass/fail judgment. In LangSmith evals, LangChain’s Jev assesses difficulty and correctness with separate typed answers in one pass on every trace.
“#6 𝕏 Harrison Chase announced that deepagents now supports dynamically loading tools when a skill is loaded, pairing skill-specific tools without making them always available.”
#6 𝕏 Harrison Chase announced that deepagents now supports dynamically loading tools when a skill is loaded, pairing skill-specific tools without making them always available. With OpenAI or Anthropic models, this can be done without breaking the prompt cache.
“Harrison Chase shared that coding-agent costs fell significantly for a second month in a row after three steps: tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness with techniques such as model routing.”
#6 𝕏 Harrison Chase shared that coding-agent costs fell significantly for a second month in a row after three steps: tracking usage in LangSmith, setting user-level cost caps through an LLM gateway, and optimizing the OpenSWE harness with techniques such as model routing.
“Harrison Chase shared ModelRouterMiddleware, a LangChain project by @dbreunig, in the context of a DSPy and Jev model-router example.”
GenAI PM Daily - Perplexity Computer now generates interactive 3D learning visualizations October 04, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 12 insights for PM Builders from X and LinkedIn. Perplexity Computer now generates interactive 3D learning visualizations #3 𝕏 Harrison Chase shared ModelRouterMiddleware, a LangChain project by @dbreunig, in the context of a DSPy and Jev model-router example. Jev reads the first message to select a model for the entire run, with a custom hook allowing reselection after each tool result.
“Harrison Chase commented that agent teams want their own trace review UI, and that LangChain’s LangSmith Custom Apps lets users build one on their own data and ship it directly into the workspace.”
#6 𝕏 Harrison Chase commented that agent teams want their own trace review UI, and that LangChain’s LangSmith Custom Apps lets users build one on their own data and ship it directly into the workspace. #11 𝕏 Harrison Chase commented on a Google Research multi-agent paper’s pattern of giving verifiers clean context, preventing explorers’ reasoning from swaying them toward bad proofs while still allowing unconventional ideas.
“Harrison Chase shared a four-step approach to model routing: understand the tasks, understand the models, build the router in the harness, and track outcomes.”
#13 𝕏 Harrison Chase shared a four-step approach to model routing: understand the tasks, understand the models, build the router in the harness, and track outcomes. The goal is to lower costs without a performance hit.
“Harrison Chase announced that SemIf, an open-source alternative to the “decision model” jev, is based on qwen3.5 and is being hosted free through LangSmith Gateway for the next week.”
#17 𝕏 Harrison Chase announced that SemIf, an open-source alternative to the “decision model” jev, is based on qwen3.5 and is being hosted free through LangSmith Gateway for the next week.
“Harrison Chase described “JEV as a judge” as a cheap, fast semantic verifier useful for evals, especially online evaluations that grade many traces.”
Harrison Chase described “JEV as a judge” as a cheap, fast semantic verifier useful for evals, especially online evaluations that grade many traces.
“Harrison Chase commented that agent memory has drawn strong interest for roughly two years without sticking. He said updating and using memory must be tightly integrated with the agent harness, while deciding what to remember is often application-specific and prompt-based, making standalone products difficult to build.”
#15 𝕏 Harrison Chase commented that agent memory has drawn strong interest for roughly two years without sticking. He said updating and using memory must be tightly integrated with the agent harness, while deciding what to remember is often application-specific and prompt-based, making standalone products difficult to build. #16 ▶️ How to run your first AI UGC campaign (step-by-step guide) AI Jason
“Harrison Chase shared LangChain’s “How to Build a Custom Agent Harness,” describing it as a great blog by Sydney Runkle.”
#4 𝕏 Harrison Chase shared LangChain’s “How to Build a Custom Agent Harness,” describing it as a great blog by Sydney Runkle. The post quoted Garry Tan on systems of record versus domain-specific harnesses.
Related
An AI company mentioned for releasing a report on model behaviors and for internal tool use in later context. Relevant for PMs tracking safety reporting and frontier model operations.
Anthropic’s coding agent/environment used here to control physical tools in a long-running home experiment.
OpenAI is an AI company building foundation models and consumer products like ChatGPT. In this newsletter it is discussed in connection with GPT-6, Dots, ChatGPT for Teens, and College Planner.
An AI coding tool shown here supporting follow-up questions and a /visualize command that generates a new chart in-chat.
An AI data framework company that demonstrated LlamaParse on a complex earnings deck. Relevant for PMs working with document ingestion and structured extraction.
An OpenAI coding model/tool used here as part of coding harnesses turned into reinforcement-learning environments.
An investor/operator commenting on agent harnesses and token use. In this issue he contrasts lab-developed harnesses with startup utility.
A partner company listed in the Sign in with ChatGPT launch set. The newsletter provides no additional context beyond the integration mention.
MCP is a protocol for connecting AI systems to external tools and data sources. Here it is described as the way Granola can use captured context through internal tooling.
AI search company behind Perplexity Decider.
The company behind LangSmith, mentioned in connection with trajectory labeling and evaluation workflows. Relevant to AI PMs building agent monitoring stacks.
A Claude-based workflow or assistant used for structured journaling and planning. In this newsletter it transforms captured notes into organized outputs and follow-up actions.
LangChain’s evaluation and observability product, referenced in the context of trajectory labeling. Useful for PMs instrumenting agent traces and evals.
Slack is a workplace messaging platform often connected to AI agents and internal tools. In this context it is one of the data sources powering an incident-command overview.
Autonomous or semi-autonomous workflows that perform tasks, coordinate tools, and execute actions on behalf of users.
An agent framework that now supports dynamically loading tools when a skill is loaded. This pattern helps keep toolsets modular and can preserve prompt cache efficiency.
The code-hosting and collaboration platform used for pull requests and repository workflows. Here it is part of Warp’s automated engineering loop.
Anthropic's SDK for building Claude-powered agents and workflows. In the newsletter it is used in a formal verification example with Lean and TLA+.
A protocol or capability layer mentioned as part of an open, composable extension philosophy for AI tooling. It is grouped with MCP and Plugins.
A 2.8T-parameter open-weight model described as frontier-level by the speaker in the newsletter. It is notable for strong quality and deployment on Nebius Token Factory.
Agents used to write, review, and iterate on code as part of software development workflows. The newsletter frames them as shifting developers toward specification, architecture, and evaluation work.
Amazon’s cloud platform, referenced in a story about a training pipeline running on a 4-GPU instance. The anecdote highlights GPU utilization monitoring and infrastructure waste.
Vercel’s SDK for integrating AI features into apps. The newsletter highlights token savings from a single line of code in DeepSeek-powered workflows.
An agent framework or approach that uses summarization middleware to manage context while preserving work history. It emphasizes offloading tool results and older messages into external storage.
A framework for internal agents that emphasizes instructions, skills, channels, and connectors. It is presented as a default choice for building internal agent systems.
A convention for storing agent instructions in a repository. Here it is cited as one place where preferences can be set for fx.sh.
A tool that provides coding agents with real-time API documentation so they can produce more accurate code. It targets agent-assisted development workflows.
A company focused on AI development workflows and agent harnesses. It is mentioned for its Missions framework and multi-step orchestration.
A standard endpoint/protocol for agent-to-agent or agent interoperability, mentioned here alongside MCP as a supported interface. It matters to PMs as part of agent connectivity and integration strategy.
LangChain’s deployment offering for launching agents securely and at scale. It is important for PMs evaluating production readiness, observability, and managed infrastructure for agents.
A modular layer that adds tools, guardrails, and custom instructions to AI agents. It is described as a composable harness for production agent systems.
Community middleware example for customizing agent behavior and steering tasks in agent frameworks.
Stay updated on Harrison Chase
Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.
Subscribe Free