Langfuse

@Langfuse list all datasets in our project and show me the item count for each one

Found 4 datasets in your Langfuse project: 'customer-support-qa' (127 items), 'product-classification' (89 items), 'sentiment-analysis-v2' (203 items), and 'summarization-benchmark' (45 items). Each dataset includes ground-truth examples you've curated for evaluation. The sentiment-analysis-v2 dataset was last modified 3 days ago and appears to be your most actively maintained collection.

This read-only query demonstrates how Langfuse organizes evaluation datasets. Useful for auditing what test sets exist before running experiments. Requires API key with read access to datasets; no side effects.

@Langfuse create a score for trace abc123 with name 'accuracy' value 0.87 and comment 'correct entity extraction'

Score created successfully. Attached 'accuracy' score (0.87) to trace abc123 with your comment. This score is now visible in the Langfuse UI under that trace's evaluation tab and will be included in aggregate metrics for any experiment or dataset run referencing this trace. The timestamp reflects when this score was recorded, not when the original trace executed.

This write operation permanently adds evaluation metadata to a trace. Useful for programmatic scoring after manual review or automated eval runs. Cannot be undone through the MCP—you'd delete via Langfuse UI. Ensure trace ID exists before scoring.

@Langfuse get the latest production prompt named 'system-instruction' and compare it to version 3 to see what changed

Retrieved two versions of 'system-instruction': production deployment (v5, updated 2 days ago) and your requested v3 (from 11 days ago). The production version added a new constraint about citation formatting and removed the temperature suggestion that was in v3. Both are chat-style prompts with identical variable placeholders ({{user_query}}, {{context}}). The v5 system message is 47 tokens longer, primarily due to the expanded citation rules.

Demonstrates prompt versioning and dependency resolution. Langfuse tracks every prompt edit, so you can diff changes or roll back. The MCP returns full prompt text—useful for auditing what's live in production versus what your code expects.

When Langfuse beats spreadsheets for prompt iteration

A three-person AI team ships a customer-facing chatbot and needs to track which prompt versions actually reduce hallucination rates. Langfuse wins here because the Create Score and Get Prompt tools let you version prompts, run evals against dataset items, and attach scores to specific traces—all without leaving your CI pipeline. The annotation queue tools mean your PM can flag bad outputs for review without opening a Jupyter notebook. The threshold: if you're only running evals once a week in a notebook, a spreadsheet is simpler. But if you're iterating on prompts daily and need to compare scores across versions, Langfuse keeps the feedback loop tight. Set up API key auth, point your eval script at List Dataset Run Items, and you're tracking regressions by lunch.

How annotation queues turn support tickets into training data

A five-person support team fields complaints about an AI agent giving wrong answers. They need to triage which failures are prompt bugs versus missing knowledge. Langfuse's Get Annotation Queue and List Annotation Queues tools let support staff flag traces for engineering review without touching code. Each flagged trace becomes a dataset item that feeds back into your eval pipeline. The trade-off: if your agent only handles 20 conversations a day, manual review in Slack is faster. But once you cross 100 daily sessions, the annotation queue keeps support and engineering aligned on what "bad" looks like. Your support lead reviews queues in the morning, engineering pulls flagged traces into datasets by afternoon, and you're running experiments on fixes by end of week.

When experiments beat gut-feel for feature launches

A four-person product team wants to ship a new summarization feature but can't tell if GPT-4 or Claude performs better for their use case. Langfuse's List Experiments and List Dataset Run Items tools let you run side-by-side comparisons on real user data, then attach scores to each model's output. The experiments view shows which variant wins on latency, cost, and quality without writing custom analytics. The boundary: if you're only testing one feature once a quarter, a manual CSV export works fine. But if you're shipping LLM features every sprint and need to justify model choices to stakeholders, Langfuse turns gut-feel into data. Run your A/B test, pull experiment results, and you've got a slide deck ready for the retro.