GenAI PMPostHog
An analytics platform used for tracking LLM events, product outcomes, and evaluation signals.
Key Highlights
- PostHog is positioned here as a practical analytics platform for tracking LLM events, outcomes, cost, latency, and evaluation signals.
- A recommended PostHog event schema includes request started, completed, failed, output rated, and task completed events with standardized properties.
- The guidance emphasizes backend event logging and avoiding raw prompts or outputs in analytics payloads for safer observability.
- PostHog was also mentioned in connection with LlamaIndex-powered LLM Analytics and Nebula's broader agent automation stack.
PostHog
Overview
PostHog is a product analytics company and platform that appears in this context as a practical destination for LLM analytics. For AI Product Managers, it matters because it can be used to track model requests, task outcomes, user feedback, latency, token usage, cost, and evaluation signals in the same analytics workflow used for broader product measurement. That makes it useful for connecting AI system behavior to business and user outcomes rather than treating model telemetry as a separate silo.Across the mentions here, PostHog shows up in three especially relevant ways: as an analytics layer for structured LLM event logging, as part of an LLM Analytics workflow integrated with LlamaIndex, and as a service that can plug into agent-driven automation systems like Nebula. Together, those examples position PostHog as part of the operational stack AI PMs can use to measure, debug, and improve AI features in production.
Key Developments
- 2026-01-27 — Nebula was described as integrating with PostHog alongside services like GitHub and Notion, enabling specialized agents and automated workflows for solo operators and small teams.
- 2026-02-14 — PostHog was noted as having integrated LlamaIndex into its LLM Analytics, with a demo agent workflow using LlamaIndex, LlamaParse, and OpenAI to match product specs to user needs.
- 2026-06-08 — A detailed workflow for tracking LLM analytics in PostHog highlighted a recommended event schema: `llm_request_started`, `llm_request_completed`, `llm_request_failed`, `llm_output_rated`, and `llm_task_completed`, with properties such as `trace_id`, `request_id`, `prompt_version_id`, `model`, `provider`, `environment`, `latency_ms`, `input_tokens`, `output_tokens`, `estimated_cost_usd`, and `status`. The guidance emphasized sending events from the backend and avoiding raw prompts or outputs in analytics payloads.
Relevance to AI PMs
1. Create a standard LLM measurement layer. PostHog can serve as the system where AI PMs define a consistent event taxonomy for model requests, completions, failures, ratings, and task outcomes. This makes it easier to compare prompts, models, and versions over time.2. Tie AI performance to product outcomes. Because the recommended schema includes both technical signals and product or evaluation fields, AI PMs can measure more than latency and cost. They can analyze whether an AI feature actually improves task success, user satisfaction, or funnel conversion.
3. Support safer observability practices. The guidance to avoid logging raw prompts and outputs, and instead use safe references like `prompt_version_id`, `prompt_hash`, and linked traces, is tactically useful for teams balancing debugging needs with privacy and governance requirements.
Related
- LlamaIndex — Connected through PostHog's LLM Analytics integration, showing how orchestration and retrieval workflows can feed analytics.
- LlamaParse — Part of the demo workflow mentioned with LlamaIndex and OpenAI for matching product specs to user needs.
- OpenAI — Referenced as the model provider used in the integrated demo workflow.
- Nebula — An agent platform that integrates with PostHog, suggesting PostHog's role in broader automated operating stacks.
- GitHub and Notion — Mentioned as adjacent integrations in the Nebula workflow context, indicating the kinds of tools PostHog may sit beside in operational systems.
- PromptLayer — The source of the June 2026 guidance on how to structure LLM analytics in PostHog.
Newsletter Mentions (3)
“How to track LLM analytics in PostHog - Log a small, consistent set of LLM events in PostHog (llm_request_started, llm_request_completed, llm_request_failed, llm_output_rated, llm_task_completed) with core properties like trace_id, request_id, prompt_version_id, model, provider, environment, latency_ms, input_tokens, output_tokens, estimated_cost_usd, and status plus product/outcome/eval fields, send events from your backend, and never include raw prompts/outputs—use safe references (prompt_version_id, prompt_hash, document_type) and link to traces for debugging.”
#4 📝 PromptLayer Blog How to track LLM analytics in PostHog - Log a small, consistent set of LLM events in PostHog (llm_request_started, llm_request_completed, llm_request_failed, llm_output_rated, llm_task_completed) with core properties like trace_id, request_id, prompt_version_id, model, provider, environment, latency_ms, input_tokens, output_tokens, estimated_cost_usd, and status plus product/outcome/eval fields, send events from your backend, and never include raw prompts/outputs—use safe references (prompt_version_id, prompt_hash, document_type) and link to traces for debugging. Example payload in the article shows model gpt-4.1-mini with latency_ms 1840, input_tokens 1284, output_tokens 312, estimated_cost_usd 0.0048, prompt_version_id pv_2026_06_04_003, and trace_id trace_01J7ZP8E9K4VQ2.
“PostHog has integrated LlamaIndex into its LLM Analytics, demoing an agent workflow that uses LlamaIndex, LlamaParse, and OpenAI to match product specs to user needs.”
#8 𝕏 LlamaIndex 🦙 PostHog has integrated LlamaIndex into its LLM Analytics, demoing an agent workflow that uses LlamaIndex, LlamaParse, and OpenAI to match product specs to user needs.
“Nebula integrates with cloud services like Google Slides, Ghost, PostHog, GitHub, Notion and allows spawning specialized agents (blog worker, analytics worker, lead-gen worker) to automate end-to-end workflows for solo entrepreneurs.”
From YouTube • Video Content Inside $180B Co-Founder's AI Agent System Greg Isenberg • January 26, 2026 Greg Isenberg sits down with Furqan Rydhan to demo Nebula, a Slack-inspired AI agent platform where each channel hosts an agent that writes and executes code to build Google Slides decks, generate images, publish blog posts on Ghost, integrate with services like PostHog, and schedule autonomous workflows for one-person businesses. Key Takeaways: Nebula agents operate in Slack-style channels and can write and run Python code to call APIs—e.g., creating and updating Google Slides presentations, generating AI images, and retrying failed tasks until completion. Users can automate recurring tasks via cron-style triggers, such as adding two new slides per day to reach a 15-slide deck in a week or publishing three blog posts daily on a Ghost blog with built-in web search. Nebula integrates with cloud services like Google Slides, Ghost, PostHog, GitHub, Notion and allows spawning specialized agents (blog worker, analytics worker, lead-gen worker) to automate end-to-end workflows for solo entrepreneurs.
Related
An AI company building frontier models, ChatGPT, and custom inference hardware. Here it is discussed for Jalapeño and ChatGPT Business Premium Seats.
An AI infrastructure company and community that recapped a founder dinner in San Francisco. The discussion focused on vertical agents, moats, and go-to-market implications.
A prompt management and AI workflow company. The newsletter cites its blog post arguing that fine-tuning is often the wrong default compared with RAG and other methods.
A document parsing tool from LlamaIndex. Here it is notable for extracting form fields into structured JSON without an additional schema or API call.
A software development platform used here as the source and sync target for repositories. It is central to AI coding workflows, plugin distribution, and agent automation.
A workspace and note-taking tool used here to store research outputs as cards. In this workflow it supports agent-generated content operations.
A Slack-inspired AI agent platform for autonomous workflows. It lets each channel host an agent that writes code, calls APIs, and automates tasks across multiple services.
Stay updated on PostHog
Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.
Subscribe Free