Braintrust
Braintrust is an AI engineering platform for managing prompts, datasets, experiments, evaluations, observability, and production AI applications.
Verdict
Common use cases
- Version prompts before deploying to production
- Compare experiment results across model runs
- Log eval scores after each sprint
- Retrieve dataset events for debugging
- Archive old experiments when cleaning up
Integration
- Vendor
- Braintrust
- Category
- other
- Auth
- API_KEY
- Tools
- 17
- Composio slug
braintrust
Tools
- Create Dataset
Create a named dataset under a Braintrust project, or return the existing dataset unchanged when its name already exists in that project.
- Create Experiment
Create an experiment under a Braintrust project. When its name already exists, Braintrust returns the existing experiment unchanged unless ensure_new is true, which creates an experiment with a unique name.
- Create Project
Create a Braintrust project, or return the existing project unchanged when its name already exists.
- Create Prompt
Upsert a versioned prompt configuration under a Braintrust project without invoking it. The same project and slug reuse the prompt ID but create a new version that replaces the stored prompt configuration with the submitted configuration. M
- Delete Experiment, Dataset, or Promptdestructive
Delete one Braintrust experiment, dataset, or prompt by UUID for explicit lifecycle cleanup.
- Delete Projectdestructive
Soft-delete a Braintrust project and return its deletion state.
- Fetch Events
Fetch one newest-to-oldest page of project log, experiment, or dataset events.
- Get Braintrust Resource
Get one project, experiment, dataset, prompt, or function by UUID.
- Insert Dataset Events
Upsert a batch of examples into an existing Braintrust dataset. New or omitted IDs insert events; an existing supplied ID replaces that event's current values.
- Insert Experiment Events
Upsert a batch of result events into an existing Braintrust experiment. New or omitted IDs insert events; an existing supplied ID replaces that event's current values.
- List Braintrust Resources
List and filter one page of projects, experiments, datasets, prompts, or functions visible to the connected Braintrust API key.
- Query Braintrust Data
Run one bounded synchronous read-only BTQL query over Braintrust project logs, an experiment, or a dataset. The first-page query must end with a positive LIMIT; continue later pages by passing the returned next_cursor unchanged.
- Summarize Experiment or Dataset
Return metadata and optional aggregate summaries for one Braintrust experiment or dataset.
- Update Dataset
Update the name, description, tags, or metadata of a Braintrust dataset. At least one non-null mutable field is required.
- Update Experiment
Update selected metadata and linkage fields on a Braintrust experiment. At least one non-null mutable field is required.
- Update Project
Update selected fields of a Braintrust project. At least one non-null mutable field is required: name, description, or settings.
- Update Prompt
Update a Braintrust prompt's metadata or configuration without invoking it. At least one non-null mutable field is required.
Setup
Setup guide
- 11. Open your Switchy workspace settings and navigate to the Integrations tab. 2. Click 'Add MCP' and select Braintrust from the catalog. 3. In a separate browser tab, log into your Braintrust account at braintrust.dev and go to Settings → API Keys. 4. Generate a new API key with read and write permissions, then copy it. 5. Paste the key into the Switchy connection dialog and click 'Connect'. 6. Switchy will verify the key and confirm the connection. 7. Open any Project in Switchy and type '@Braintrust create a project called test-workspace' to confirm the MCP responds. 8. If the command succeeds, you'll see a project UUID in the reply and the integration is live.
What teammates see: by default, memories from Braintrust are scoped to the Project (PROJECT visibility) - you can mark any memory PRIVATE or share it ORG-wide.
Works well with
Top models
Compatibility data appears once enough Projects have used this MCP together with a given model.
How Switchy teams use it
Starter prompts
Create Experiment
@Braintrust create an experiment called 'gpt4-vs-claude' under project UUID abc-123. Set the baseline model to gpt-4.
Version a Prompt
@Braintrust create a prompt in project 'chatbot-prod' with slug 'greeting-v2' and this template: 'You are a helpful assistant. User question: {{question}}'Fetch Recent Events
@Braintrust fetch the 20 most recent events from experiment UUID def-456 so I can see the last eval scores.
Retrieve Dataset
@Braintrust get the dataset with UUID ghi-789 and show me its name, description, and row count.
Clean Up Project
@Braintrust delete the project with UUID jkl-012 and confirm it's been removed from my workspace.
Example outputs
Illustrative - representative of the model's voice and quality, not literal recordings.
@braintrust fetch the last 20 events from our 'customer-support-classifier' experiment and summarize which prompt versions performed best
Fetched 20 events from experiment 'customer-support-classifier'. The data shows prompt version 3 (deployed 2 days ago) achieved 94% accuracy across 847 inferences, outperforming version 2's 89%. Version 3 uses a more structured system message with explicit category definitions. The experiment also logged 12 feedback corrections where the model initially misclassified urgent tickets as low-priority—these corrections fed back into your dataset for retraining.
This example demonstrates Braintrust's core observability workflow: fetching logged inference events and using the AI to interpret performance trends. The MCP returns raw event data; the AI synthesizes it into actionable insight. Requires read access to the specified project and experiment UUID.
@braintrust create a new dataset called 'support-tickets-march' under the 'customer-support' project and confirm it's ready for upload
Created dataset 'support-tickets-march' under project 'customer-support' (UUID: ds_a3f9c2e1). The dataset is now active and ready to receive records via the Braintrust SDK or API. If a dataset with this name already existed, Braintrust returned the existing resource unchanged—check the UUID to confirm whether this is a new dataset or a reference to an existing one.
Braintrust's create-dataset tool is idempotent: re-running the same name returns the existing dataset rather than erroring. This is useful for setup scripts but means you should verify the UUID if you need certainty about whether a dataset was just created. Requires API key with write permissions on the target project.
@braintrust list all prompts in the 'email-generation' project, then draft a changelog summarizing version differences for the 'welcome-email' prompt slug
Found 3 prompt versions under slug 'welcome-email' in project 'email-generation'. Version 1 (Jan 15) used a formal tone with product feature bullets. Version 2 (Feb 3) shifted to conversational language and added a personalized greeting variable. Version 3 (current, Mar 1) introduced dynamic content blocks based on user signup source. Changelog: v1→v2 improved engagement by 18% per A/B test logs; v2→v3 added conditional logic for trial vs. paid signups, reducing unsubscribe rate by 9%.
This synthesis example pairs Braintrust's prompt-versioning metadata with the AI's ability to compare configurations and infer intent. The MCP retrieves stored prompt objects; the AI interprets their evolution. Useful for teams managing dozens of prompt variants across experiments. Requires read access to the project's prompt registry.
Use-case deep-dives
When Braintrust wins for structured prompt regression testing
A 6-person ML team ships a customer-facing chatbot and runs weekly eval sprints to catch prompt regressions before deploy. Braintrust's dataset and experiment tools let the team version prompts, log eval runs, and compare outputs across iterations without leaving the AI workspace. The Create Experiment and Fetch Events tools surface which prompt version broke the edge case. This works when your eval corpus fits in a few hundred rows and you run evals at least weekly. If you're running one-off prompt tests or your team doesn't have a formal eval cadence, the 17-tool surface is overkill—stick with a shared doc and manual spot-checks. For teams shipping LLM features on a sprint rhythm, Braintrust keeps the eval loop tight and the history auditable.
When versioned prompts save a production fire
A 10-person startup's support bot starts hallucinating after a prompt tweak goes live Friday afternoon. The on-call engineer uses Braintrust's Get Resource and Create Prompt tools to pull the last-known-good prompt version and redeploy it in under five minutes. The MCP's prompt versioning and UUID-based retrieval turn what could be a multi-hour rollback into a single command. This scenario assumes you're already logging prompt changes to Braintrust—if you're editing prompts in a text file or Notion, the MCP can't help you rewind. The trade-off: you need API key auth and the discipline to version every production prompt. For teams running LLM features in prod with no formal version control, Braintrust is the safety net that pays for itself the first time something breaks.
When experiment tracking beats spreadsheet chaos
A 4-person research lab benchmarks three foundation models every quarter against a fixed eval set. They use Braintrust's Create Dataset and Fetch Events tools to snapshot the eval corpus, run experiments for each model, and pull comparison metrics into a shared report. The MCP keeps the benchmark reproducible—next quarter's run references the same dataset UUID and compares apples to apples. This works when your benchmark cadence is monthly or slower and you care more about trend lines than real-time dashboards. If you're running daily model evals or need live alerting, Braintrust's fetch-and-compare flow is too manual—look for a dedicated observability stack. For low-frequency, high-rigor benchmarks where reproducibility matters more than speed, Braintrust turns ad-hoc spreadsheets into a durable artifact.
Frequently asked
What does the Braintrust MCP do in Switchy?
It lets your AI agents create and manage Braintrust projects, experiments, datasets, and prompts without leaving the conversation. Agents can log evaluation runs, version prompt configs, fetch experiment events, and delete resources by UUID. Think of it as programmatic access to Braintrust's core workflow primitives — project setup, experiment tracking, dataset management — all callable from chat.
Do I need a Braintrust API key to connect this MCP?
Yes. The MCP authenticates with a Braintrust API key, which you generate in your Braintrust account settings. Paste it into Switchy's connection flow once. The key grants read-write access to all projects in your Braintrust workspace, so treat it like a password and only share it with teammates who need full experiment and dataset control.
Can the Braintrust MCP run evaluations or score outputs?
No. It creates the scaffolding — projects, experiments, datasets, prompts — but doesn't execute evals or compute scores. You still run evaluations in your own code or Braintrust's SDK, then use this MCP to fetch the logged events or create new experiment containers. It's infrastructure automation, not an eval runner.
Why use this MCP instead of calling Braintrust's API directly?
The MCP wraps Braintrust's REST API in natural-language tools, so your agents can say "create an experiment called Q1-baseline" instead of crafting JSON payloads. It's faster for ad-hoc workflows — spinning up test projects, versioning prompts mid-conversation — but offers no advantage over the SDK for programmatic pipelines you'd write in Python or TypeScript.
Who on the team should connect the Braintrust MCP?
Whoever owns your Braintrust workspace and generates API keys. Since the key grants full project and dataset write access, limit it to ML engineers or team leads who already manage experiments. Don't hand it to every analyst — they can view results in Braintrust's UI without needing MCP access in Switchy.