Openlayer
Openlayer is an AI quality platform for managing projects, monitoring production data, evaluating tests, and inspecting model and application performance.
Verdict
Common use cases
- Check which model tests failed overnight
- Compare pass rates across recent commits
- Audit data-source drift before a release
- Surface workspace usage against plan limits
- Triage monitoring alerts during standup
Integration
- Vendor
- Openlayer
- Category
- other
- Auth
- API_KEY
- Tools
- 10
- Composio slug
openlayer
Tools
- Get Data Source
Return one Openlayer monitoring data source, including its backend, monitoring status, record and test counts, and optional project or workspace details.
- Get Project Version
Return one Openlayer project commit or version with its processing status, commit metadata, storage URI, and passing, failing, and total test counts.
- Get Workspace
Return a workspace's identity, status, connected-account membership, usage counts, and plan limits. Obtain workspace_id from List Projects first.
- List Data Sources
Return one page of monitoring data sources for an Openlayer project, optionally filtered by name.
- List Data Source Test Results
Return one page of the latest monitoring test results for an Openlayer data source, optionally filtered by test category or execution status.
- List Projects
Return one page of projects visible to the connected Openlayer workspace, optionally filtered by name or task type.
- List Project Versions
Return one page of commits and versions for an Openlayer project, including processing status, commit metadata, and passing, failing, and total test counts.
- List Project Version Test Results
Return one page of test results for a project version, optionally filtered by test category, archive state, or result status.
- List Test Results
Return one page of results for a test, with optional project-version, data-source, and status filters and optional generated insights.
- List Tests
Return one page of tests configured for an Openlayer project, with filters for category, archive state, origin version, production-data use, and suggested status.
Setup
Setup guide
- 11. In Switchy, open your workspace settings and navigate to the Integrations tab. 2. Click 'Add Integration', search for Openlayer, and select it. 3. Log in to your Openlayer dashboard, go to Settings → API Keys, and generate a new key with read permissions. 4. Paste the key into Switchy's auth prompt and click Connect. 5. Return to any Project, type '@Openlayer list projects' in the chat, and confirm you see your project list — that confirms the connection works. 6. To invoke a specific tool, @mention Openlayer followed by the action (e.g. '@Openlayer get project version <id>') and the assistant will fetch live data.
What teammates see: by default, memories from Openlayer are scoped to the Project (PROJECT visibility) - you can mark any memory PRIVATE or share it ORG-wide.
Works well with
Top models
Compatibility data appears once enough Projects have used this MCP together with a given model.
How Switchy teams use it
Starter prompts
Overnight Test Summary
@Openlayer list the latest test results for data source <id>, filtered by failing status, and summarize which categories are broken.
Version Comparison
@Openlayer list project versions for <project_id>, show the three most recent, and highlight any drop in passing test count.
Data Drift Check
@Openlayer list data sources for project <id> and flag any with monitoring status indicating drift or anomalies.
Workspace Health
@Openlayer get workspace details for <workspace_id> and report current usage against plan limits.
Commit Test Audit
@Openlayer get project version <version_id> and list all test results, grouping by category and result status.
Example outputs
Illustrative - representative of the model's voice and quality, not literal recordings.
@Openlayer list all projects in my workspace and show me which ones have failing tests in their latest versions
Found 4 projects in your workspace. Two have failing tests: 'customer-churn-model' (v2.3.1, 3 of 12 tests failing—data drift and performance degradation detected) and 'fraud-detection-api' (v1.8.0, 1 of 8 tests failing—latency threshold exceeded). The other projects ('sentiment-classifier' and 'recommendation-engine') show all tests passing in their most recent commits. Would you like details on any specific failure?
This example combines List Projects with List Project Versions to surface test health across your ML portfolio. Requires read access to the workspace. Note that test counts reflect the latest processed version only—older commits may have different results.
@Openlayer get the latest monitoring test results for the 'fraud-detection-api' data source and tell me what's breaking
Retrieved 6 test results from the 'fraud-detection-api' monitoring data source (last updated 2 hours ago). One test is failing: 'p95_latency_threshold' shows 450ms response time against a 300ms limit. Five tests passed: data schema validation, prediction distribution stability, feature null-rate checks, and two custom business rules. The latency spike correlates with increased traffic volume in the past 6 hours.
Demonstrates real-time monitoring insights by fetching List Data Source Test Results. Useful for on-call debugging or daily standup prep. The MCP returns test metadata and status but not raw prediction logs—you'll need the Openlayer UI for trace-level investigation.
@Openlayer compare test results between version 2.2.0 and 2.3.1 of the customer-churn-model project and summarize what regressed
Version 2.2.0 had 12 tests with 1 failure (accuracy below 0.85 threshold). Version 2.3.1 shows 12 tests with 3 failures: the original accuracy issue persists, plus two new failures—'feature_importance_stability' (top features shifted unexpectedly) and 'data_drift_detection' (input distribution diverged 18% from training baseline). This suggests the retrain introduced instability despite attempting to fix accuracy. Recommend rolling back or investigating the new training data.
Showcases cross-version analysis by calling List Project Version Test Results twice and synthesizing differences. Requires knowing version identifiers upfront (get them via List Project Versions first). The AI interprets test names and counts but can't access raw model artifacts or training configs.
Use-case deep-dives
When Openlayer MCP catches model drift before production push
A 5-person ML team ships a recommendation model every two weeks. The data scientist runs List Project Versions at sprint close to compare test pass rates across the last three commits, then pulls List Project Version Test Results for any version showing new failures. If a commit drops below 85% passing tests, the team gates the deploy and investigates. This MCP works when your model pipeline already writes to Openlayer and you need quick CLI or agent access to test summaries without opening the dashboard. The threshold: if you're triaging more than 20 test failures per version, the paginated results get tedious—at that scale, you want the full Openlayer UI. For teams shipping 1-3 models per sprint with under 50 tests each, this MCP turns version comparison into a 30-second Slack check-in instead of a 10-minute dashboard hunt.
Using monitoring data sources to answer 'why did the model fail'
A 3-person support team at a fintech startup fields tickets when customers see unexpected loan denials. The support lead uses Get Data Source and List Data Source Test Results to pull the latest monitoring stats for the underwriting model's production data source. If test results show a category failure (say, fairness or data quality), the agent surfaces the exact test name and timestamp to engineering. This scenario wins when your models are already instrumented with Openlayer monitoring and support needs read-only proof that a known issue exists. The boundary: if your monitoring generates more than 100 test results per day, pagination makes this slow—you'll want a dedicated monitoring dashboard or alerting instead. For low-to-mid-volume production models (under 10k predictions daily), this MCP gives support a factual handoff without granting full platform access.
When List Projects and Get Workspace map your ML estate
A compliance officer at a 40-person AI company runs a quarterly audit to document which models are live, which workspaces own them, and whether test coverage meets internal policy. She uses List Projects filtered by task type to enumerate all classification and regression projects, then Get Workspace to confirm each workspace's plan limits and connected accounts. The MCP returns project counts, version counts, and workspace membership in a single pass, which feeds a compliance spreadsheet. This works when you have 3-10 Openlayer workspaces and need a lightweight inventory snapshot without custom API scripting. The trade-off: if you're managing 50+ projects or need historical version metadata beyond the last page of results, you'll hit pagination limits and want a dedicated data export. For mid-sized teams with centralized ML governance, this MCP turns a half-day audit into a 20-minute agent query.
Frequently asked
What does the Openlayer MCP do in Switchy?
It pulls monitoring data, test results, and project versions from your Openlayer ML observability workspace. You can query data sources, check which tests are passing or failing, and inspect commit-level metrics without leaving Switchy. Useful when you want AI agents to reason over model performance or flag regressions during a conversation.
Do I need admin access to connect Openlayer?
You need an Openlayer API key with read permissions on the workspace you want to query. Openlayer uses API_KEY authentication, so whoever generates the key controls which projects and data sources Switchy can see. Check your workspace settings to confirm the key has access to the projects your team cares about.
Can the Openlayer MCP trigger new test runs or upload datasets?
No. The MCP is read-only—it fetches existing data sources, versions, and test results. If you need to push new data or kick off monitoring jobs, use Openlayer's SDK or API directly. This integration is for inspection and analysis, not orchestration.
Why use this instead of logging into Openlayer's dashboard?
The MCP lets you ask natural-language questions about test failures, compare versions, or pull metrics into a broader workflow without context-switching. If you're already in Switchy debugging a model issue, querying Openlayer inline is faster than opening another tab and clicking through the UI.
Who on the team should connect the Openlayer workspace?
Whoever owns the API key. Typically an ML engineer or platform lead who already has Openlayer access. Once connected, anyone in your Switchy workspace can query the data, but the connection inherits the permissions of that original key—so choose someone with visibility into the projects you want to monitor.