Arize AX

@arize-ax retrieve the full dataset and all versions for dataset ID ds_prod_evals_2024 so we can review what examples we're testing against

Retrieved dataset 'Production Evals Q1 2024' (ds_prod_evals_2024) with 3 versions. Version v1.0 contains 847 examples with fields: user_query, expected_response, context_docs. Version v1.1 added 203 examples on March 15. Version v2.0 (current) restructured schema to include sentiment_label and dropped context_docs. Total examples across versions: 1,050 in v1.0, 1,253 in v1.1, 1,189 in v2.0. This is an illustrative summary; actual field names and counts will match your workspace state.

Demonstrates read access to Arize's dataset versioning system. Useful when you need to audit what test data backs your experiments. Requires API key with read scope on the target space. Dataset IDs are workspace-specific, so prompts referencing them won't transfer across Arize accounts.

@arize-ax create a new experiment called 'GPT-4o mini routing test' in space sp_customer_support, link it to dataset ds_support_golden, and log the first 5 runs with latency and cost metrics

Created experiment 'GPT-4o mini routing test' (exp_8x3k9m) in space sp_customer_support, associated with dataset ds_support_golden. Logged 5 initial runs: run_1 (latency: 1.2s, cost: $0.003), run_2 (latency: 0.9s, cost: $0.002), run_3 (latency: 1.5s, cost: $0.004), run_4 (latency: 1.1s, cost: $0.003), run_5 (latency: 1.0s, cost: $0.002). Experiment is now live and ready for additional runs. This example illustrates the creation flow; your actual run IDs and metric values will differ.

Showcases Arize's experiment tracking for prompt evaluation workflows. Side-effect: creates a persistent experiment in your workspace that will appear in the Arize UI. You'll need write permissions on the target space and a valid dataset ID. Useful for A/B testing prompt variants or model configurations.

@arize-ax compare the last 10 runs from experiment exp_routing_v2 and summarize which prompt version had lower average latency and cost, then suggest whether we should promote it to production

Analyzed 10 most recent runs from exp_routing_v2. Prompt version 'v3_concise' averaged 0.8s latency and $0.0021 per run across 6 runs. Prompt version 'v2_detailed' averaged 1.3s latency and $0.0035 per run across 4 runs. Version v3_concise shows 38% faster response time and 40% lower cost. Recommendation: promote v3_concise to production if accuracy metrics (not shown here) remain acceptable. Verify with a larger sample size before rollout. This synthesis pairs Arize experiment data with reasoning; actual recommendation depends on your accuracy thresholds.

Combines Arize's experiment retrieval with AI reasoning to guide deployment decisions. The MCP fetches run metadata; the AI interprets trends. Requires read access to the experiment. Note that this example assumes runs include latency and cost fields—your experiment schema may differ. Always validate statistical significance before acting on small sample sizes.

When you need version control for prompts that actually ship

A 6-person support team is tuning a GPT-4 bot that answers tier-1 questions. They test three prompt variants every week, track which version cuts ticket volume, and roll back when a new prompt hallucinates. Arize AX is the right call here: the Create Prompt Version tool locks each iteration, the experiment runs tie performance metrics to specific prompts, and the dataset tools let you replay real customer questions against every variant. The auth is straightforward API key, so your ops lead can wire it into CI without waiting on IT. If you're only tweaking prompts once a quarter or your team is under 3 people, a shared Google Doc is faster. But once you're shipping prompt changes weekly and need to prove which version reduced escalations, Arize AX pays for itself in the first sprint.

How a 10-person ML team keeps retrieval quality from drifting

A product analytics startup runs a RAG pipeline over user event logs. Every two weeks, the ML team adds 20 new edge-case queries to their eval dataset—questions where the retriever failed or returned stale chunks. Arize AX fits this workflow: Add Dataset Examples appends the new cases without versioning churn, Get Dataset pulls the full test set into your CI pipeline, and the experiment tools let you compare retriever configs side-by-side. The 24-tool surface is overkill if you're just logging inference traces, but once you're curating hundreds of examples and running A/B tests on embedding models, the dataset versioning becomes load-bearing. If your eval set is under 50 examples or you only re-run evals manually, start with a CSV in S3. Beyond that threshold, Arize AX keeps your test data from becoming a versioning nightmare.

When a 3-person AI team needs to compare 12 model runs fast

A fintech's AI team is prototyping a fraud classifier. Over two weeks they test four base models, three prompt strategies, and two retrieval configs—12 runs total. They need to see which combo hits 95% precision without manually diffing spreadsheets. Arize AX works here: Create Experiment sets up the test matrix, Add Experiment Runs logs each trial with arbitrary metrics, and Get Experiment pulls the leaderboard into Slack at end-of-day. The API key auth means the intern can start logging runs in 10 minutes. The downside is the 24-tool count—if you only run experiments once a month, the onboarding overhead outweighs the benefit. But in a sprint where you're iterating daily and need a single source of truth for which config won, Arize AX is faster than stitching together Weights & Biases and Notion.