The problem with eval data in a silo​

DeepEval is great at running evaluations. But by default its results go to Confident AI — a different platform from where your QA and engineering teams already track quality. That split means two dashboards, two notions of "is this passing," and no shared history between your functional tests and your eval suite.

We kept hearing the same ask: just put the evals where the rest of our tests are. So we built a bridge.

How it works​

testrelic-deepeval reads DeepEval's in-memory TestRun and uploads it to TestRelic's native evals API. No monkey-patching, no HTTP redirect — it just reads the state DeepEval already produced. There are two ways to use it.

As a pytest plugin (zero code changes):

pip install testrelic-deepeval
testrelic login
deepeval test run tests/

The plugin captures the run at session finish and uploads it. If no credentials are present it quietly no-ops, and it never fails your tests — upload errors are swallowed.

As a programmatic wrapper:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from testrelic import evaluate

results = evaluate(
    test_cases=[LLMTestCase(input="Hi", actual_output="Hello")],
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
)
# results is whatever deepeval.evaluate() returns; upload is automatic

The evaluate() wrapper is a drop-in for deepeval.evaluate() — run your eval the way you already do, and the upload happens for you. Full command reference is on the CLI Reference page.

The eval workspace​

Once a run uploads, it appears in the new LLM Evaluations workspace. Eval runs render through the same Run Detail and Session Workspace as your functional tests — there's no bespoke eval-only UI to learn. Each repository gets an Evaluations tab next to Test Cases and Test Runs, and you can opt eval rows into the global Test Runs feed.

The metric we think you'll watch most is Eval Stability — how consistently your eval cases pass across runs over time. A metric that flips between pass and fail run-to-run is telling you something about either your prompt or your judge, and stability surfaces it the same way flakiness surfaces an unreliable test.

Evals that never block CI​

LLM evals call out to model providers, and provider hiccups shouldn't fail your pipeline's reporting. testrelic-deepeval writes failed uploads to a local offline queue (~/.testrelic/queue/) and replays them on demand:

testrelic drain

It also auto-detects CI context — branch, commit, and CI URL — so every eval run is anchored to the change that produced it.

Migrating from Confident AI​

Already sending evals to Confident AI? The package ships a helper that prints the migration steps:

testrelic migrate-from-confident

In most cases it's just installing the package, running testrelic login, and pointing your existing deepeval test run at TestRelic — your eval code doesn't change.

Get started​

Install testrelic-deepeval, run testrelic login, and run your existing eval suite. The results show up in your LLM Evaluations workspace immediately. This pairs with the broader Python SDK launch — evals are one of four Python packages we shipped today.

Related documentation​

• TestRelic for DeepEval — overview
• Get Started with DeepEval
• DeepEval CLI Reference
• LLM Evaluations workspace
• Repositories — Evaluations tab

Why Python, and why now​

Half the test suites we get asked about are Python. API teams live in pytest + httpx. Browser teams have moved to pytest-playwright. The LLM teams run deepeval. Mobile teams drive Appium from the Python client. Until now, all of them had to choose between rich local reports and TestRelic's cloud — because the cloud only understood our Node SDKs.

So we ported the whole thing. Not a thin shim — a field-for-field port of the analytics model, so a Python run opens the exact same Session Workspace, AI Insights, and history views as a JavaScript run. Pick the package that matches your stack and you're done. The new Python SDK section of the docs walks through each one.

pytest: zero-config analytics for any suite​

testrelic-pytest is the generic reporter. It works with any pytest project — unit, integration, or API — and the headline is that it needs no configuration at all:

pip install testrelic-pytest
export TESTRELIC_API_KEY=tr_live_...
pytest

That's the whole setup. The plugin auto-registers, captures every test's outcome, markers, captured output, and CI/git metadata, and uploads the run.

The part we're most excited about: automatic HTTP API detection. If a test makes httpx or requests calls, TestRelic classifies it as a rest API test and opens it in the API Session Workspace instead of the web view — capturing request and response headers and bodies, with sensitive headers masked. Real assert statements are recorded too, with the source expression and the evaluated values, so a failing check reads like this in the Assertions tab:

response.status_code == 201
→ 200 == 201   (failed)

No fixtures, no decorators. Plain pytest tests just become observable. There's more in HTTP API Auto-Capture.

Playwright, ported to Python​

testrelic-playwright is a full port of @testrelic/playwright-analytics (schema 1.3.0 — field-for-field parity). It's a pytest plugin for pytest-playwright that captures navigation timelines, network stats, action steps, console logs, expect() assertions, screenshots, and video — then renders a single-file interactive HTML report and uploads to the cloud.

pip install pytest-playwright testrelic-playwright
playwright install
pytest

Output lands in test-results/analytics-timeline.json and .html. For large suites it streams to disk automatically (configurable via reportMode and streamingThreshold), and the bundled testrelic-playwright serve command hosts the report locally. BrowserStack and LambdaTest runs are detected automatically. Full reference: Configuration and Get Started.

LLM evaluations with DeepEval​

testrelic-deepeval captures DeepEval evaluation results and uploads them to your TestRelic org instead of Confident AI. The pytest plugin captures DeepEval's in-memory run at session finish — no code changes — and there's a drop-in evaluate() wrapper for programmatic runs:

pip install testrelic-deepeval
testrelic login
deepeval test run tests/

Eval runs land in the new LLM Evaluations workspace, render through the same Run Detail and Session Workspace as everything else, and track an Eval Stability metric over time. We wrote a whole post on this — see Own your LLM evals.

Mobile testing with Appium​

testrelic-appium (Beta) brings analytics to Appium + pytest. It auto-wraps any Appium WebDriver your test creates and records a full command timeline, Android logcat / iOS syslog, CDP network capture, screenshots, and video — with a named-assertion helper:

from testrelic_appium import assert_that

with assert_that("toHaveText", expected="Welcome"):
    assert el.text == "Welcome"

Pure-Selenium tests are never touched. See Get Started with Appium (Python).

One cloud, every language​

All four share the same conventions: set TESTRELIC_API_KEY, runs go to https://platform.testrelic.ai, failed uploads queue locally and replay with a drain command, and CI metadata is auto-detected. Once a run uploads it shows up in the Test Runs dashboard alongside your JavaScript runs.

Get started​

Pick your package on the Python SDK overview, pip install, set your API key, and run your tests. If you're already a TestRelic user, your existing org, repos, and AI features work with Python runs immediately.

Related documentation​

• Python SDK overview — choose your package
• pytest — HTTP API Auto-Capture
• Playwright (Python) — Get Started
• DeepEval — Get Started
• Appium (Python) — Get Started
• Cloud Platform overview

Five new capability sets​

MCP capabilities are opt-in via --caps (or the TESTRELIC_MCP_CAPS env var); core is always on. On top of the existing coverage, creation, healing, impact, triage, signals, and devtools, 3.1 adds five:

• ai — the Ask AI surface: tool catalog, a single-turn agent, conversation management, and artifact generators (dashboards, reports, test plans, presentations, navigation paths).
• marketplace — first-class testing integrations (Jira, GitHub Actions, BrowserStack, LambdaTest, Sentry, Amplitude, Grafana Loki): catalog, connect / OAuth, and a unified invoke.
• apps — a generic action runner for connected apps (Slack, Notion, Linear, HubSpot, Google Calendar, and more): list apps, list actions, connect, and a universal execute.
• artifacts — list, get, export (PNG, PDF), and save-to-file for artifacts the Ask AI agent produces.
• sessions — cloud test session workspaces (BrowserStack / LambdaTest): search, video, screenshots, logs, and a multi-pane workspace render.

Mix and match them for the agent you're building. The full list is on the Capabilities page.

One-click install in Cursor​

The fastest way to try it is the Cursor Marketplace:

1. Open Cursor and search Settings → Plugins for testrelic-mcp.
2. Click Install. Cursor wires up the MCP server automatically.
3. Open the agent and ask something like "list my TestRelic projects".

The marketplace default boots in mock mode — no account, no token, no network calls — so you can explore every tool safely before connecting anything. When you're ready for real data, create a personal access token (tr_mcp_*) at https://platform.testrelic.ai/settings/mcp-tokens, run npx @testrelic/mcp login, and drop --mock-mode. The Quickstart has the details.

A cleaner tool namespace​

Every tool now uses a tr_* prefix — tr_list_repos, tr_coverage_report, tr_heal_run, tr_analyze_diff, and so on. The old testrelic_* names still work as deprecated aliases, so existing integrations don't break; new work should use tr_*. The full mapping is in the Tools Reference.

We also corrected the default cloud URL to platform.testrelic.ai across the board, so a fresh install points at the right place with no configuration.

Get started​

If you use Cursor, install the plugin and start in mock mode — it's the lowest-friction way to see what the agent can do. For Claude Desktop, VS Code, or any other MCP client, the Overview covers local setup. This release is part of a broader day of launches — see the Python SDK announcement for the rest.

Related documentation​

• TestRelic MCP overview
• Capabilities
• Tools Reference
• Quickstart

Parity with the Slack app​

If you've used the TestRelic Slack app, you already know the Teams app. It talks to the same Ask AI runtime, produces the same artifacts, and shares the same memory and privacy posture. What you get in Teams:

• Ask AI in a channel — address the TestRelic bot with a prompt and the answer threads under your message, with charts and deep links.
• Scheduled QA digests — any prompt can post on a schedule to a channel (a Monday flaky digest, a nightly regression alert that only posts on failure).
• Share-back from the web app — push any Ask AI artifact to a Teams channel as a rich card or uploaded file.
• Failure alerts on quality gates — a gate posts the moment it fails, with the failing case and an Open in TestRelic button.
• Threaded follow-ups — replies continue the same Ask AI conversation, so context carries through.

Conversations are unified across web, Slack, and Teams — a Teams-originated chat shows up in the Ask AI sidebar with a Teams badge. The app only acts on prompts addressed to it; it does not read general channel traffic.

Installing is two steps (this part differs from Slack)​

Teams is stricter than Slack about how bots reach a workspace, so installation has two parts — and it's worth being explicit about it:

1. Connect in TestRelic — go to Settings → Integrations → Microsoft Teams → Connect. This uses Microsoft Entra ID (Azure AD) OAuth and binds your Teams tenant to your TestRelic org.
2. Add the Teams app package — the bot only appears in Teams once the app package is added: sideloaded by you, or uploaded by a Teams admin to your org's app catalog.

If you Connect and the bot doesn't show up, step 2 hasn't happened yet — that's the most common support question and it's covered in the Microsoft Teams integration guide.

Share any AI artifact with a link​

The other half of today's release works everywhere Ask AI does. Every artifact in the Ask AI panel — a dashboard, report, test plan, presentation, table, or chart — can now be shared with a public or private link:

• Public links render a read-only view of the artifact, filters and all, to anyone with the URL.
• Private links require org sign-in.

Shared artifacts collect in an artifact library you can browse and re-share. It's the fastest way to get a generated report in front of a stakeholder who doesn't live in TestRelic. Details on the AI Insights & Artifacts page.

Plan notes​

The Teams app talks to Ask AI, which is part of the Growth plan (Plans & Billing). The integration installs on any plan; prompts on Starter orgs return an upgrade message.

Related documentation​

• Microsoft Teams integration — full reference + two-step install
• Slack integration — the sibling app
• Ask AI — runtime, artifacts, memory
• AI Insights & Artifacts — artifact types and share links
• Plans & Billing

Why a Slack app​

Most teams already have a Slack channel where QA, engineering, and on-call meet. Until today, getting a test answer into that channel meant: open the web app, navigate, screenshot, paste. Three steps too many.

We watched our own team and a handful of design partners do this dozens of times a week. The right answer wasn't another dashboard — it was making Ask AI reachable from inside the conversation. So we built it.

The Slack App is the inverse of what we already shipped as Ask AI Apps: Ask AI Apps let the AI call out to Slack/Jira/Gmail from the web composer. The Slack App lets you call in to Ask AI from Slack. Both work together; you can install both and they don't overlap.

What you can do today​

/testrelic slash commands​

/testrelic <prompt> runs the prompt through the same Ask AI runtime as the web composer — same tool calls, same context, same artifact types. Responses thread under your message with charts and deep links:

/testrelic which tests broke after yesterday's deploy?

/testrelic show me the flakiest tests in the checkout suite over the last 7 days

Threaded follow-ups continue the same Ask AI conversation, so context like @repo or @test_run set on the first prompt carries through. You can chain prompts the way you'd chain DM replies.

Scheduled QA digests​

Any Ask AI prompt can be scheduled as a recurring digest to a Slack channel. We use these internally:

• Weekly flaky digest to #qa-leads every Monday 9am — three stat tiles (pass rate, flaky %, MTTR) plus the top five offenders with retry counts.
• Nightly regression alert to #oncall after the prod E2E run — only posts if something failed.
• Monthly compliance evidence to a private channel — a generated PDF report uploaded as a Slack file for audit retention.

Create one by running the prompt once, then clicking Share → Schedule to Slack on the artifact. Or skip the web app entirely:

/testrelic schedule a weekly flaky digest to this channel every Monday 9am

Manage all schedules at Settings → Integrations → Slack → Schedules.

Share-back from the web app​

Every artifact in the Ask AI panel has a Share → Slack action. Pick a channel, add a message, and the artifact posts as a rich Slack card with metric snapshots and a deep link. Presentations and reports upload as Slack files so they stay searchable.

The artifact types that round-trip cleanly today:

Failure alerts on quality gates​

Connect a monitoring quality gate to a Slack channel and the gate posts the moment it fails — with the failing test case, the threshold that was crossed, and an Open in TestRelic button. Reply in the thread with /testrelic was this a flake or a real regression? and the AI inspects retry patterns and CI metadata before answering. We use this to triage on-call without leaving Slack.

Sample use cases​

A few patterns from our design partners and our own team:

Daily standup helper​

Run during standup to anchor the conversation in real numbers:

/testrelic what's the test pass rate trend for the last 5 days,
and any new flaky tests this morning?

The card surfaces three stat tiles plus a list of overnight regressions, so engineers can divide work before the meeting ends.

Sprint review one-liner​

The day before sprint review, a scheduled digest posts to #engineering-leads:

/testrelic generate a 4-slide sprint review covering test coverage,
flakiest tests, new regressions, and MTTR for the last 14 days

The deck uploads as a Slack file. Stakeholders read it in Slack; engineers open the artifact in TestRelic for drill-down.

Cross-team coverage check​

A PM in #growth asks a coverage question without leaving Slack:

/testrelic map critical test paths from the checkout journey to confirmation
and flag any P0 gaps for @repo:web-app

The reply renders a navigation-path map and a coverage-gap table. PM and QA align in-channel; no dashboard handoff.

Compliance evidence​

A monthly digest posts to a private compliance channel:

/testrelic generate a monthly QA evidence report covering all P0 paths,
pass-rate trends, and unresolved escapes from the @prod environment

Uploaded as a PDF, linked back to the underlying TestRelic runs.

Plan and privacy notes​

The Slack App talks to Ask AI, which is part of the Growth plan (Plans & Billing). On Starter orgs, /testrelic responds with an upgrade prompt.

A few privacy details worth calling out:

• Conversations are unified. Slack-originated conversations show up in the Ask AI Chat Sidebar in the web app, tagged with a Slack badge. Memory is scoped to your TestRelic user, so triggering /testrelic from Slack and continuing from the web app under the same account gives you continuous memory.
• Channel messages are not ingested. The app only acts on prompts directly addressed to it (slash commands, scheduled digests, share-back). It does not read general channel traffic.
• Same training policy as the rest of Ask AI. Prompts and responses are stored in your governed conversation history and are never used to train shared models.

Full scope list, install steps, and troubleshooting live on the Slack integration page.

Installing​

The fastest path:

1. Open Settings → Integrations in the TestRelic web app.
2. Click Connect on the Slack card.
3. Pick the workspace and channel; the app installs in well under a minute.

Or install directly from the Slack App Directory. The first slash command from any user prompts them to link their Slack identity to a TestRelic account — one-time, per user.

What's next​

A few things on the near-term roadmap based on design-partner feedback:

• @slack-thread context — attach a Slack thread as context to an Ask AI prompt, so the AI can pick up the conversation where the channel left off.
• Per-channel default scopes — pin @repo or @environment to a channel so /testrelic prompts there inherit it automatically.
• Workflow Builder steps — drop TestRelic actions into Slack Workflows for no-code triage flows.
• Bidirectional Jira — file a Jira ticket from a Slack alert thread with one click, pre-filled with the failure context.

If any of these are blocking for your team, ping us in our public Slack — the design partners we work with shape what ships next.

Related documentation​

• Slack Integration — full reference, scopes, troubleshooting
• Ask AI — runtime, artifacts, memory, Ask AI Apps
• AI Insights & Artifacts — what each artifact type looks like
• Plans & Billing — Growth plan limits and AI token quotas
• Jira Integration — for /testrelic open a ticket follow-ups

Access, URLs, and plan gating​

Per the Ask AI doc:

• Open Ask AI from the left sidebar in the product.
• Routes follow /ai for a new conversation and /ai/:conversationId when resuming history.

Plan requirement: Ask AI requires the Growth plan. Pricing and limits are documented on Plans & Billing. The Introduction includes a high-level plan comparison table (Starter vs Growth) for SDK, cloud, AI, storage, and retention—use that table when stakeholders ask what unlocks Ask AI versus reporter-only usage.

Starting conversations and example prompts​

Ask AI streams token-by-token responses after you submit a prompt. The documentation ships copy-ready examples:

1. Analyze recent failures — e.g. common failure patterns over seven days scoped to a repository.
2. Generate a test plan — structured scenarios such as checkout happy path, payment failure, inventory edge cases.
3. Summarize a specific run — distinguish infrastructure noise from product defects using @ context (below).

Teams onboarding many engineers should paste these patterns into internal playbooks so prompts stay consistent and auditable.

@ mentions: precision without leaking unrelated data​

The @ context picker (documented in a table on the Ask AI page) attaches typed entities so the model grounds answers in your org:

Type @ in the composer to search and attach. This pattern is what makes Ask AI materially different from a generic LLM chat: the platform can invoke tools against those attachments (see “Streaming and tool use” below).

Conversation history, privacy, and feedback​

• History: Conversations save automatically. The Chat Sidebar lists prior threads; you can resume, rename, or delete them. Conversations are private to your user within the organization—as documented on the Ask AI page.
• Feedback: Thumbs up / down on each assistant message feeds product improvement loops.

Document these policies in your internal AI governance memo so security reviewers understand data residency at the conversation level.

Structured artifacts beyond plain text​

Ask AI can emit artifacts rendered beside the chat. Supported types (from the official table) include:

Deeper styling and behavior notes live under AI Insights & Artifacts, which also covers session insights (automatic analysis in the Session Workspace Insights tab on Growth—see that page for defect, user-impact, Jira linkage, and MTTR panels).

Streaming, tool calling, and resilience​

Ask AI’s doc explains the runtime model:

• The assistant issues tool calls to fetch repository, run, and test-case data; the UI surfaces an indicator while tools run, then merges results into the streamed answer.
• Streaming uses Server-Sent Events (SSE); the platform retries and resumes if the connection drops mid-generation.

For reliability engineers, that means timeouts should be tuned at the proxy layer with SSE-friendly settings—not by disabling streaming outright without product guidance.

Platform context: where Ask AI sits​

The Cloud Platform Overview lists AI & Intelligence as a first-class pillar alongside runs, repositories, test navigation, monitoring, integrations, and administration. Practically, Ask AI consumes the same org-wide ingestion path described there: SDK reporters produce JSON; the cloud stores and indexes it; Ask AI and other AI surfaces read it through governed tools.

For run-level drill-down before or after an Ask AI session, send engineers to the Dashboard & Test Runs and Session Workspace documentation.

Ask AI vs TestRelic MCP​

Both can leverage the same underlying TestRelic data; they are complementary, not interchangeable.

Related documentation​

• Ask AI
• Plans & Billing
• Introduction (plan table + onboarding links)
• Cloud Platform Overview
• AI Insights & Artifacts
• Session Workspace
• Dashboard & Test Runs
• TestRelic MCP overview

Architecture at a glance​

The overview’s sequence diagram summarizes the trust boundary: your MCP host exchanges JSON-RPC with the MCP server over stdio (or streamable HTTP for hosted mode); the server then calls TestRelic cloud APIs with your credential. Integration backends (Jira, Amplitude, Grafana Loki, GitHub, TestMu AI, BrowserStack) are not duplicated inside MCP config—they are resolved from your organization in the cloud after the server authenticates, as stated in the overview and Authentication pages.

Production connection paths​

The MCP quickstart is the step-by-step source for:

Sandbox vs production: mock mode and local clones are documented as evaluation-only. For live org data, remove --mock-mode and supply a real token (quickstart + authentication).

Authentication: one token, clear storage rules​

MCP authentication documents the full lifecycle:

1. Create an MCP personal access token in the web app (tr_mcp_… prefix).
2. Store it either with npx -y @testrelic/mcp@3.0.0 login (writes the default file) or via TESTRELIC_MCP_TOKEN / your client’s mcp.json env block—never commit secrets.
3. Align API base using TESTRELIC_CLOUD_URL or --cloud-url if your reporters already use a non-default endpoint; match the same API base the SDK uses (Configuration, Appium configuration).
4. Optionally set TESTRELIC_DEFAULT_REPO_ID or --default-repo-id when tools omit a project identifier.
5. Disable mock mode when you are ready for outbound cloud calls.

That page reiterates: Jira, Amplitude, Loki, GitHub, TestMu AI, and BrowserStack credentials flow from cloud org settings, not from extra MCP secrets.

Capability bundles (--caps) for lean tool schemas​

The Capabilities reference defines optional tool groups beyond always-on core:

Suggested bundles on that page cover day-to-day authoring, on-call / failures, and repo-wide exploration. Passing only the caps you need keeps the MCP tool schema smaller—important for models with finite context windows.

Operational verification and next workflows​

After wiring credentials, the quickstart recommends tr_health to confirm the server reaches TestRelic successfully. From there, Tools reference enumerates callable tools for your enabled caps, and Testing workflow explains how teams chain MCP-assisted steps with their existing TestRelic data habits.

How MCP complements Ask AI and the web UI​

MCP is IDE- and agent-centric tool access. Ask AI (/ai/ask-ai) is a full-page product experience in the cloud with its own plan rules. Many organizations use both: humans triage in the UI while agents refactor tests with MCP-fed run context. Compare goals before standardizing on one surface.

Related documentation​

• TestRelic MCP overview
• MCP quickstart
• Authentication
• Capabilities
• Tools reference
• Testing workflow
• Ask AI (product UI contrast)
• Playwright Configuration (cloud URL alignment)

Why Playwright teams adopt execution-level observability​

Playwright excels at deterministic automation, but flaky timing, third-party dependencies, and routing bugs often show up only as a single failed assertion. The Introduction describes how TestRelic enriches runs with navigation timelines, network stats, console output, failure diagnostics, and CI metadata so reviewers can answer: what did the browser actually do before the expect() failed? That is distinct from screenshot-on-failure or trace zip alone—it is a structured analytics model aligned to navigations and tests, which you can diff across runs once data reaches the cloud.

Requirements (from the intro): Node.js ≥ 18 and Playwright ≥ 1.40.

End-to-end browser testing: automatic navigation tracking​

The E2E Testing (Browser) page is the canonical reference for browser suites. When tests import the page fixture from @testrelic/playwright-analytics/fixture, the reporter tracks:

• Navigation timeline — timing around page load, DOM content loaded, network idle-style signals where applicable.
• Network statistics — request counts, byte transfers, and response-time style signals per navigation.
• Navigation types — goto, link clicks, back/forward, SPA route changes, and hash changes.

That combination helps teams distinguish “the app never loaded” from “the app loaded but the wrong route rendered” without re-running the suite locally.

API-only tests, unified runs, and combined JSON​

Not every suite drives a browser for every test. The Get Started documentation includes an API testing setup path: configure the reporter with options such as trackApiCalls, captureRequestBody, and captureResponseBody, and compose tests with testRelicApiFixture from @testrelic/playwright-analytics/api-fixture as documented there.

When you mix browser and API coverage, the Unified Test Reports (Browser + API) guide explains how a single JSON report can contain:

• Navigation timeline entries (URLs, navigation types, durations, network statistics).
• Pass/fail/flaky/skipped results for every test, including those that used only request.
• Failure diagnostics and CI metadata in the same schema family as other TestRelic reports.

For SEO and internal discovery, bookmark that page whenever your squad searches for “combined E2E and API Playwright analytics.”

Install paths: local-only vs cloud-backed​

The Introduction outlines two onboarding tracks:

1. Local reporting — install the package, add the reporter to playwright.config.ts, run tests; artifacts land under paths like ./test-results/ as described in the installation guide.
2. With the cloud platform — sign up at testrelic.ai, complete onboarding, add your API key to reporter configuration, and uploads happen automatically on subsequent runs (see intro for the ordered checklist).

The Cloud Platform Quickstart expands the cloud path for teams that want dashboards, AI features, and integrations—not just local files.

Configuration as the single source of truth​

Every tuning knob for the Playwright reporter lives on the Configuration page: output location, stack traces, code snippets around failures, network capture toggles, streaming vs batched reporting, redaction rules, custom testRunId, custom metadata, and more. The page groups minimal, development, CI, performance-optimized, and redaction patterns so platform engineers can copy a baseline and tighten it for regulated environments.

Common questions—changing outputPath, disabling network tracking, filtering navigation types, attaching metadata—are answered in the “What does each option do?” section on that same page, plus a Troubleshooting section for when reports look empty or oversized.

From JSON to dashboards and AI​

Once runs reach the cloud, the Cloud Platform Overview describes how the same underlying data powers the test runs dashboard, session workspace (timeline, console, network, video where available), AI & intelligence features, test navigation maps, monitoring (smoke, regression, nightly), and integrations (GitHub Actions, Jira, Grafana Loki, Amplitude, TestMu AI, BrowserStack). Your Playwright reporter remains the producer; the platform is the consumer for org-wide analytics.

Related documentation​

• Introduction
• Get Started
• Configuration
• E2E Testing (Browser)
• API Testing
• Unified Test Reports (Browser + API)
• Browser report and API report where you split concerns per report type
• Cloud Platform Quickstart
• Cloud Platform Overview
• Merging shard reports for parallel CI shards

This hub page orients new readers: what TestRelic is, which packages to install, how the cloud fits in, and where to read next—including longer blog articles that tie multiple doc sections together for SEO-friendly learning paths.

What is TestRelic?​

TestRelic is a Testing Observability Platform that gives engineering teams deep visibility into their automated test suites — whether they run in a browser or on a mobile device. Unlike traditional test reporters that only show pass/fail results, TestRelic captures rich execution data and surfaces it through AI-powered insights, monitoring dashboards, and deep session inspection.

TestRelic has two layers:

• SDK Reporters — lightweight packages that run alongside your tests and capture structured data
• Cloud Platform — a web application at testrelic.ai where that data is visualized, analyzed, and acted on

SDK Reporters​

@testrelic/playwright-analytics — Browser Testing​

The Playwright reporter captures a rich timeline of your browser test execution including:

• Every page navigation during your tests
• Network request statistics for each page
• Detailed failure diagnostics with code snippets
• CI/CD integration metadata

@testrelic/appium-analytics — Mobile Testing (Android / iOS)​

The Appium reporter captures a comprehensive trace of your mobile test execution including:

• Every Appium / WebDriver command with timing and arguments
• Android logcat and iOS syslog device logs per test
• HTTP network traffic captured via CDP, safariNetwork, or proxy fallback
• Device screenshots and screen video recordings
• CI/CD integration metadata

Why TestRelic?​

Modern applications run in browsers and on mobile devices. Understanding test failures in either environment requires more context than just an error message. TestRelic provides that context by:

1. Execution Timeline — See exactly which pages your browser tests visited, or which Appium commands your mobile tests issued, and in what order
2. Network Insights — Understand HTTP traffic for both browser and mobile sessions
3. Device Observability — Stream Android logcat and iOS syslog logs aligned to individual test cases
4. Failure Context — Get code snippets and screenshots showing exactly where tests failed
5. CI Integration — Automatically capture CI metadata for correlation with builds

Key Features​

Automatic Navigation Tracking (Playwright)​

Simply use the TestRelic fixture in your Playwright tests and navigation is tracked automatically:

import { test, expect } from '@testrelic/playwright-analytics/fixture';

test('user journey', async ({ page }) => {
  await page.goto('https://example.com');
  await page.click('a[href="/products"]');
  await page.click('.product:first-child');
  // All navigations tracked automatically!
});

Mobile Test Observability (Appium)​

Add TestRelicService and TestRelicReporter to your WDIO config and every Appium command, device log, and network request is captured automatically:

import { TestRelicService } from '@testrelic/appium-analytics/service';
import { TestRelicReporter } from '@testrelic/appium-analytics';

// In your wdio.conf.ts
services: [[TestRelicService, { includeDeviceLogs: true, includeNetworkLogs: true }]],
reporters: [[TestRelicReporter, { htmlReportPath: './test-results/report.html' }]],

Interactive HTML Reports​

Both reporters generate self-contained interactive HTML reports with a timeline view for commands, logs, network requests, screenshots, and video — no server required.

Cloud Platform​

Upload results to the TestRelic cloud platform to unlock:

• Org-wide test run history and health metrics
• AI-powered session insights and Ask AI chat
• Smoke, regression, and nightly observatory monitoring
• Integrations with GitHub Actions, Jira, Grafana Loki, and Amplitude

CI/CD Ready​

TestRelic automatically detects your CI environment and captures:

• Build ID and run number
• Commit SHA and branch name
• CI provider information

Supports GitHub Actions, GitLab CI, Jenkins, CircleCI, and more.

Mobile and Maestro beyond Playwright​

Browser coverage is only part of modern quality engineering. For Appium on Android and iOS, the @testrelic/appium-analytics package pairs with WebdriverIO as documented under Appium Introduction and Appium Installation. For Maestro mobile flows, see Maestro Introduction and Maestro Installation. Each stack produces timelines, logs, and media aligned to how that framework actually executes commands—so mobile engineers get parity with the Playwright observability story.

Plans: when cloud AI features unlock​

The Introduction compares Starter and Growth plans at a glance (monthly prices, storage, retention, and which tier includes AI insights and Ask AI). Use that table alongside Plans & Billing when procurement asks what changes if you upgrade for Ask AI, session insights, or higher retention limits.

(Figures from the Introduction page; see Plans & Billing for the authoritative, up-to-date commercial terms.)

Deeper reads on this blog​

For narrative guides that span multiple top-level doc areas:

• Playwright testing observability — timelines, unified reports, configuration, and cloud paths.
• TestRelic MCP for AI assistants — hosted vs stdio MCP, tokens, caps, verification.
• Ask AI and org test context — @ mentions, artifacts, streaming, and how Ask AI differs from MCP.

Getting Started​

Ready to add TestRelic to your project? It only takes a few minutes.

For Playwright browser testing:

npm install @testrelic/playwright-analytics

Configure the reporter in playwright.config.ts and run npx playwright test.

For Appium mobile testing:

npm install @testrelic/appium-analytics
npm install --save-dev webdriverio @wdio/reporter

Configure TestRelicService and TestRelicReporter in your wdio.conf.ts and run npx wdio run wdio.conf.ts.

Check out the Getting Started guide for complete setup instructions for both frameworks.

What's in the Documentation?​

Our documentation covers everything you need:

• Getting Started — Quick installation and setup guide for Playwright and Appium
• Packages — Detailed reference for @testrelic/playwright-analytics and @testrelic/appium-analytics
• Configuration — All available options and how to use them
• Advanced — Merging shard reports and advanced use cases
• Examples — Real-world examples for browser and mobile testing
• Cloud Platform — Full reference for the TestRelic cloud application

Free to Use​

TestRelic is free to use and published under the MIT license. Get started today and enhance your test observability!

• npm (Playwright): @testrelic/playwright-analytics
• npm (Appium): @testrelic/appium-analytics
• Support: Contact us for questions or feature requests

What's Next?​

• Browse the documentation to learn more
• Try the getting started guide to set up your framework
• Explore the configuration options for the Playwright reporter
• See the Appium package reference for mobile testing
• Check out examples to see TestRelic in action
• Connect your IDE assistant with the MCP overview and quickstart
• Open Ask AI on a Growth plan for conversational suite analysis
• Skim the blog index for opinionated learning paths across the doc set

We're excited to see what you build with TestRelic!