Humanloop

@Humanloop list all active experiments in project pr_abc123 and summarize their current status

Found 3 active experiments in project pr_abc123: 1. **Prompt Variant A/B Test** (exp_xyz789) — Testing two system prompts for customer support responses. Running since Jan 15, 47 sessions logged. 2. **Temperature Optimization** (exp_def456) — Comparing temperature settings 0.7 vs 1.0 for creative writing tasks. 23 sessions, early results favor 0.7. 3. **Model Comparison: GPT-4 vs Claude** (exp_ghi012) — Head-to-head evaluation on summarization quality. 61 sessions, no clear winner yet. All experiments are collecting data; none have reached statistical significance for a conclusive result.

This example shows Humanloop's experiment-tracking strength — useful when your team runs multiple prompt or model tests in parallel. Requires a valid project ID (format: pr_*). The MCP returns raw experiment metadata; the AI synthesizes it into a readable status report. Note that experiment results depend on your evaluation criteria being defined in Humanloop's UI first.

@Humanloop create a new project called 'Q1 Marketing Copy Generator' with description 'A/B testing GPT-4 prompts for ad headlines'

Project created successfully: **Q1 Marketing Copy Generator** (pr_new987) Description: A/B testing GPT-4 prompts for ad headlines Organization: org_default456 Created: 2025-01-28 14:32 UTC You can now log sessions to this project or set up experiments in the Humanloop dashboard. The project ID is `pr_new987` — use this when calling other Humanloop tools or logging data via API.

Demonstrates project creation, the starting point for organizing prompt experiments in Humanloop. The MCP returns a project ID immediately, but you'll still need to configure prompts, models, and evaluation criteria in Humanloop's web UI before running experiments. This is a write action — it creates a permanent resource in your Humanloop account until explicitly deleted.

@Humanloop pull the last 20 sessions from project pr_abc123 and identify any patterns in user feedback scores

Retrieved 20 most recent sessions from pr_abc123 (Jan 26–28): **Feedback score breakdown:** - 5 stars: 12 sessions (60%) - 4 stars: 5 sessions (25%) - 3 stars or below: 3 sessions (15%) **Emerging pattern:** Sessions with feedback ≤3 stars all occurred during evening hours (6–9 PM UTC) and involved multi-turn conversations exceeding 8 messages. Lower scores correlate with longer context windows, suggesting the model may lose coherence in extended dialogues. Consider testing a context-pruning strategy or switching to a model with larger context capacity for evening traffic.

Shows how Humanloop's session logs pair with AI analysis to surface insights your team might miss in raw data. The MCP fetches session metadata (timestamps, feedback, message counts); the AI connects the dots. Requires that your application is already logging sessions to Humanloop with feedback scores attached. Pagination defaults to 20 results — adjust the 'size' parameter for larger datasets.

When Humanloop fits a 3-person AI feature team

A 3-person team shipping an AI-powered search feature needs to track which prompt variants perform best across user cohorts. Humanloop is the right call here: the Create Project and List Experiments tools let the team spin up a project per feature, run A/B tests on prompts, and pull experiment results into Switchy for standup review. The List Sessions tool surfaces individual user interactions when something breaks. The trade-off: if your team isn't iterating on prompts weekly, you're paying for infrastructure you don't use. This works when prompt experimentation is a weekly ritual, not a one-time setup task.

Using Humanloop to diagnose support chatbot failures

A 5-person customer success team runs a support chatbot and needs to understand why certain queries return unhelpful answers. The List Sessions tool in Humanloop pulls session logs filtered by project, so the team can review failed interactions in Switchy without leaving their workspace. The Delete Project tool cleans up test environments after debugging sprints. This scenario works if your chatbot logs to Humanloop already—if you're using a different LLM provider's logging, adding Humanloop just for session review is overkill. The buying call: if you're already paying Humanloop for prompt management, surfacing those sessions in Switchy saves 10 minutes per support escalation.

When a solo builder should skip Humanloop in Switchy

A solo founder building an AI writing assistant wants to review how users interact with different prompt templates. Humanloop's 4-tool MCP gives access to projects and experiments, but the overhead of API key setup and project creation is high for one person who could just check the Humanloop dashboard directly. The List Experiments and List Sessions tools shine when a team needs shared visibility in Switchy, not when one person is the only stakeholder. The threshold: if you're coordinating with a co-founder or contractor on prompt performance, Humanloop in Switchy pays off. If you're solo, the native Humanloop UI is faster.