OpenAI Updates SWE-bench Verified Metrics

#1 📝 OpenAI News

Why SWE-bench Verified no longer measures frontier coding capabilities - OpenAI explains why the SWE-bench Verified benchmark is no longer used to measure frontier coding capabilities, outlining limitations of the metric and reasons it can misrepresent real-world model performance. The piece describes the rationale for retiring or deprioritizing the benchmark and points toward alternative evaluation approaches for assessing coding ability.

Also covered by: @Sebastian Raschka

#2 📝 Simon Willison

Ladybird adopts Rust, with help from AI - Andreas Kling describes using coding agents (Claude Code and Codex) to port Ladybird's LibJS JavaScript engine from C++ to Rust, producing byte-for-byte identical output and completing ~25,000 lines of Rust in about two weeks. The availability of a thorough conformance test-suite like test262 and the ability to compare outputs with a trusted implementation made the agent-assisted port feasible and low-risk.

#3 ▶️

“I haven’t written a single line of front-end code in 3 months”: Notion’s prototype playground

How I AI Podcast

#4 𝕏

Philipp Schmid rolled out a QoL update to the Gemini Interactions API, adding include_input=True so you can fetch past interactions with their inputs (TTL: 1 day free tier, 55 days paid).

#5 𝕏

Sebastian Raschka flags SWE-Bench Verified as misleading after OpenAI’s audit found 59.4% flawed tests among 27.6% of frequently failed tasks. He also points out data leakage from open-source repos inflating frontier model scores.

Also covered by: @Sebastian Raschka

#6 📝 PromptLayer Blog

Why LLM Evaluation Results Aren't Reproducible — And What to Do About It - Explains the reproducibility problem when running the same model multiple times yields different outputs and outlines why consistent results are essential for research and production. Offers guidance on approaches to improve reproducibility in LLM evaluations.

#7 📝 PromptLayer Blog

Get Out of the Model's Way - Argues that adding more constraints and tooling around LLMs can be counterproductive; engineers often over-engineer guardrails for models that are now more capable than the constraints. The post advocates rethinking how we structure systems around LLMs.

#8 📝 Simon Willison

Red/green TDD - A brief guide recommending 'red/green TDD' as an effective practice when working with coding agents: write tests first, confirm they fail, then implement until they pass. This test-driven approach helps produce reliable, verifiable code when using agentic development tools.

#9 𝕏

Jason Zhou discovered that Codex CLI now supports multi-agent mode—just add `[features] multi_agent = true` to your `config.toml` and run `/experimental`. It unlocks three built-in agents: explorer, worker, and general helper.

#10 𝕏

Cognition launched a one-click inline PR fix feature in Devin Review that auto-suggests and applies code changes. Just swap “github” for “devinreview” in any PR link to try it—no account needed.

#11 𝕏

Santiago helps companies set up LLM-as-judge evaluations, recommending they run high-quality models like GPT-4/5 on a sampled subset of traffic to balance accuracy and cost.

#12 𝕏

LlamaIndex 🦙 launched file uploads in LlamaAgents Builder, so you can feed sample docs as context into its natural-language interface. This lets the agent craft more accurate, tailored workflows for your real-world use cases.

#13 in

Guillermo Rauch just added video support to Vercel AI Gateway and released the QeXexeAI SDK API—try it out in the configurable AI Gateway playground at vercel.fyi/video.

#14 ▶️

How I Use Obsidian + Claude Code to Run My Life

Greg Isenberg

#15 𝕏

Philipp Schmid unveiled a chip that etches Llama 3.1 8B model parameters directly into its transistors—merging storage and compute to hit 18,000 tokens/sec. He notes that even a “dumb” 8B model is incredibly useful at that speed.

#16 𝕏

Anthropic warns that distillation attacks on AI models are growing in sophistication and intensity, and calls for rapid, coordinated action among industry players, policymakers, and the AI community, offering detailed detection and prevention strategies in their new report.

#17 𝕏

Anthropic launched the AI Fluency Index, analyzing 11 user behaviors across thousands of Claude conversations—such as iteration and refinement—to quantify how fluently people collaborate with AI.

#18 in

Carl Vellotti highlights Anthropic’s AI Fluency Index, which tracked 24 behaviors across nearly 10,000 Claude conversations and found that users who iterate prompts unlock 2.67× more fluency behaviors, are 5.6× likelier to question reasoning, and 4× more apt to spot gaps.

#19 𝕏

Jeff Dean highlights AI’s educational potential and Google’s launch of Gemini training for all 6 million U.S. K–12 and higher-ed teachers, featuring concise, flexible modules with real-world examples and badges to certify AI literacy.

#20 𝕏

Santiago explains that most AI products spike at launch and flatline two weeks later—not because of the model or UX but due to a critical retention factor that teams almost never consider.

#21 in

Udi Menkes built a Claude Code skill called /one-step-better that pulls the latest GenAI PM briefs and tells you the single actionable insight to apply to your current project.

#22 in

Peter Yang shares Nat Eliason’s blueprint for a $4K/week OpenClaw bot—build a 3-layer memory system, give the bot its own accounts, relentlessly remove bottlenecks, and coordinate via Telegram.

#23 in

Claire Vo unveils Notion’s Prototype Playground—a deployed app where anyone can spin up high-quality, AI-driven prototypes directly in Notion by combining Monologue, Claude Code and a custom icon-hallucination skill.