ROI & Impact Analysis#

Open Analytics → ROI & Impact Analysis to understand how AI usage and spend relate to engineering and business outcomes.

ROI & Impact Analysis combines gateway usage and cost data with data from connected source control, CI, issue-tracking, and other systems. It uses this data to calculate metrics that help you measure AI adoption, analyze AI spend and unit costs, and examine engineering delivery and quality indicators. Use default or custom metrics to analyze the indicators relevant to your organization and review the results in dashboards and reports.

Use ROI & Impact Analysis when you need to:

Relate AI spend to engineering delivery, for example by tracking cost per feature or cost per merged pull request.
Track AI adoption, such as active users or the AI-assisted share of merged pull requests.
Monitor metrics and identify when they cross configured thresholds.
Prepare recurring reports on AI spend, adoption, and engineering indicators for finance or engineering leadership.

Unlike Usage and Cost and Optimizations, ROI & Impact Analysis connects AI usage and spend with engineering and business indicators derived from data outside the gateway.

Engineering indicators show observed relationships and trends. They do not establish that AI caused a particular engineering outcome.

At a high level, the workflow is:

AI usage and cost data + Connected business and engineering data
                              ↓
                           Metrics
                              ↓
                    Dashboards and reports

Page overview#

The page contains four tabs:

Overview — Open dashboards and review metric results. See Overview tab.
Reports — Group metrics into reports for specific audiences and review their latest values. See Reports tab.
Metrics — Define what to measure and review current values. See Metrics tab.
Schedules — Recompute active metrics automatically and optionally send a summary after each run. See Schedules tab.

Open Configuration from the page header to configure models, computation defaults, and data sources used by the analysis. See Configuration reference.

Select Refresh to reload the summary cards and the contents of the current tab.

Next: If you are setting up ROI & Impact Analysis for the first time, follow First steps: configure and analyze your first dashboard. If it is already configured, start with the Overview tab.

First steps: configure and analyze your first dashboard#

Use these steps to create a dashboard, activate its metrics, compute their first values, and review the results.

Before you start:

An organization is selected in the header.
At least one provider is Active with a model enabled.
Gateway traffic is flowing. Check Gateway ingest and Last ingest on the summary cards.
Configure the minimum. Open Configuration, select the required models, and connect the data source required by the dashboard you plan to create. See Configuration.
Create a dashboard from a preset. On the Overview tab, choose a persona preset. See Create a dashboard from a preset.
Activate the metrics. Activate the Draft metrics that should participate in scheduled computation. See Activate a metric.
Compute the metrics. Recompute the metrics manually, or add them to a schedule and run the schedule. See How computation runs.
Analyze the result. Return to Overview, open the dashboard, and review its widgets after the metrics have computed. See Analyze the result.

Configuration#

Open Configuration from the page header and complete the minimum setup required by the metrics in your first dashboard: the models and the data sources those metrics read. Field-level settings are in Configuration reference.

Analyzer traffic is billable gateway traffic

LLM calls made by ROI & Impact Analysis route through your gateway and appear in gateway usage and cost data.

Next: Create a dashboard from a preset.

Create a dashboard from a preset#

After completing the required configuration, create a dashboard from a persona preset.

A preset creates the dashboard and creates any missing metrics required by its widgets as Draft. Existing metrics are reused without changing their status. If a required metric template is unavailable, the corresponding widget is skipped.

1. On the Overview tab, click Create dashboard.

2. Click Browse persona presets.

3. Choose the preset that matches the audience:

CEO — adoption, cost per delivery unit, and quality direction.
CFO — spend dynamics, delivery cost, innovation-versus-KTLO split, and waste.
CTO — quality, adoption, review-time proxies, and model mix.
Engineering Manager — team-level delivery units and threshold status.

4. Click Create dashboard on the selected preset.

The dashboard appears under Your dashboards on the Overview tab. Its widgets can appear before their metrics have values.

Next: Activate the Draft metrics that should be recomputed on a schedule, then compute them to populate the dashboard.

Activate a metric#

Metrics created by a persona preset are created as Draft.

Activate a metric when you want to include it in scheduled recomputation. Activation changes the metric status but does not compute a value.

To activate a metric:

Open Analytics → ROI & Impact Analysis → Metrics.
Find the Draft metric.
In the Actions column, click Activate.
Confirm that Status changes to Active.

After activation, compute the metric manually or include it in a schedule. See How computation runs.

Analyze the result#

On the Overview tab, open the dashboard you created from the preset.

After its metrics have computed, the dashboard widgets show their latest results. Depending on the widget, you can review a current value or a trend over time.

Widgets can include information such as:

The metric name and latest value.
Widget type — for example, Time series or Single stat.
Type — whether the metric is deterministic or model-scored.
Sample size — how many records contributed to the result.
Data freshness — when the value was last computed, when freshness stamps are enabled.

Use Single stat widgets to review current values and Time series widgets to examine changes over time. Grouped time series can be used to compare categories such as teams. Groups smaller than Minimum group size are hidden; see Minimum group size.

If a widget has no value, see Troubleshooting.

Overview tab#

Use the Overview tab to open dashboards and review metric results.

The System Dashboards list provides predefined views of AI spend, adoption, and engineering indicators. Your dashboards contain dashboards created by users, either manually or from a persona preset.

Open a dashboard to review its widgets. Depending on the dashboard configuration, you can add, remove, or rearrange widgets and include metrics that you created separately.

Create a dashboard without a preset when you already know which metrics and widgets you need. Use a persona preset when you want a predefined starting point.

Metrics tab#

Metrics are the core unit of ROI & Impact Analysis. Each metric defines an indicator computed from gateway data, connected-system data, or both.

Use the Metrics tab to define what to measure and review metric values, thresholds, computation information, and status.

Each row can include:

Domain — The reporting area the metric belongs to, such as Finance, Engineering, or Custom.
Type — How the metric is computed.
Current value — The latest computed result. Sample size — The number of records that contributed to the result.
Threshold — Whether the latest result is within the configured bounds.
Last computed — When the metric was last computed.
Status — Active (included in scheduled recomputation) or Draft (saved, not scheduled).
Actions — Controls available for the metric, such as Activate/Pause, Edit, and Delete. The available actions depend on the metric status.

Metric types#

Deterministic metrics compute results directly from source data using their configured computation.

Model-scored metrics use model-scored source data, such as delivery-unit scoring. Model-scored delivery units are intended for tracking trends within your organization and should not be used as a cross-company benchmark.

The classifier model configured in General is not the model that calculates a model-scored metric value.

Create a metric#

Open Analytics → ROI & Impact Analysis → Metrics.
Click + CREATE METRIC.
Choose a predefined template or define a custom metric.
Configure its computation, thresholds, and cadence.
Save the metric as Draft or activate it.

How computation runs#

Computation is configured in three places:

Configuration → General → Computation defaults → Computation cadence — Sets the default cadence assigned to newly created metrics. Changing this default does not update existing metrics or trigger computation. See Computation defaults.
Metrics → Create metric (Schedule and Activation section) or Edit metric → Computation cadence — Sets the cadence stored on an individual metric. This value does not create a recurring task by itself.
Schedules tab — Creates the recurring task that automatically recomputes selected Active metrics. See Schedules tab.

To recompute a metric without a recurring task, use its manual recompute action on the Metrics tab. Activation changes the metric status but does not trigger computation.

Verify a metric#

After creating or changing a metric:

Review its status and definition.
Open Configuration → Data Sources and confirm that each connector required by the metric shows a connected state.
Compute the metric manually or run a schedule that includes it.
Confirm that Current value and Last computed are populated.
Review Sample size to see how many records contributed to the result.
Open the metric detail page to review its trend, validation status, and diagnostics.

Metric detail page#

Open a metric to review its trend, validation status, and computation diagnostics.

Use diagnostics when a value is missing or unexpected. Validation findings can identify conditions such as an empty or very small sample or records excluded by configured rules.

Edit a metric#

On the Metrics tab, locate the metric.
In the Actions column, click Edit.
Update the fields available for editing and Save.
Recompute the metric to produce a value using the updated settings.

Delete a metric#

Before deleting a metric, check whether it is referenced by a dashboard, report, or schedule.

To delete it:

Open Analytics → ROI & Impact Analysis → Metrics.
Click Delete in the metric row.
Confirm the deletion.

To keep the definition but exclude it from scheduled recomputation, leave it as Draft.

Schedules tab#

Use the Schedules tab to recompute selected Active metrics automatically.

Each schedule defines when its metrics run. You can also configure an optional email or Slack summary after each run. Threshold status is included in the run result; a threshold breach does not create a separate schedule notification.

To add a schedule:

Click + ADD SCHEDULE.
Enter a name and configure when the schedule should run.
Select the Active metrics to recompute.
Optionally configure an email or Slack notification for completed runs.
Save the schedule.

In the schedule row, click Run now in the Actions column to trigger the task without waiting for its next scheduled run.

The Slack destination uses the organization Slack webhook configuration. Email recipients are configured for the task.

Reports tab#

Use the Reports tab to save a set of metrics for a particular audience and review their latest values.

Each report includes:

Name
Audience
Computation cadence
Metrics
Written commentary

Audience and Computation cadence describe the report. They do not schedule metric computation.

To recompute metrics automatically, use the Schedules tab.

To create a report:

Click + ADD.
Enter a name.
Set the audience and cadence labels.
Select the metrics to include.
Optionally enable Written commentary.
Save the report.

Written commentary uses the configured reasoning model.

Group metrics according to the decision or audience the report supports—for example, finance metrics for a finance report or engineering delivery and quality indicators for an engineering report.

Configuration reference#

Use Configuration to set organization-level defaults and connect the data required by ROI & Impact Analysis.

The page contains:

General — Models, computation defaults, organization-wide signal selection, alerts, the SCM watcher, and reporting controls.
Data Sources — Connectors that metrics read.
Webhooks — Organization inbound endpoint.

General#

Use General to configure models, computation defaults, organization-wide signal selection, notification destinations, SCM settings, and reporting controls.

Models#

Reasoning model — Used for features such as report narratives and other reasoning-assisted operations.
Classifier model — Used for supported classification operations during analysis.

Only available models from configured providers can be selected.

Computation defaults#

Computation cadence — Default cadence assigned to newly created metrics. See How computation runs.
Join confidence floor — Minimum confidence accepted when joining gateway activity with delivery records. Raising the value accepts fewer, higher-confidence joins; lowering it can include more lower-confidence joins.
Show data freshness stamps — Controls whether metric and dashboard views display when values were last computed.

Data sources#

Use Data sources on the General tab to choose which signals each connected source contributes to ROI & Impact Analysis. These selections apply to the whole organization and are available to every metric.

Each source shows its connection state and the number of selected signals out of all signals it provides, for example 4 of 4 signals. Expand a source to review its signal selection.

This section controls analysis inputs. To configure a connector and review its metric usage and connection details, use the separate Data Sources tab.

Alert channels#

Configure organization-level destinations used by supported notification features.

Email — Organization email destinations available to notification features.
Slack webhook URL — Slack destination used by scheduled run notifications.

Email recipients for a scheduled run are configured on the schedule itself.

SCM watcher#

The SCM watcher collects supported pull-request data from configured SCM integrations.

Pull requests can be detected through supported SCM webhooks. A polling path is also available for supported GitHub/MCP configurations.

Pull requests tracked — Number of pull requests currently tracked from supported ingestion paths.
Detected by polling — Number of tracked pull requests whose detection source is polling.

Minimum group size#

Minimum group size — The minimum group size required to display aggregated data. Aggregates smaller than this value are hidden to prevent individuals from being identified from small groups.

The value is set by the platform and cannot be changed here.

Data Sources#

Use Data Sources to review the external data sources used by metrics, such as SCM, CI, and issue-tracking integrations.

Each connector shows its type, metric usage, and connection state.

This tab manages connector-specific setup and shows which metrics use each connector. Organization-wide selection of the signals those connectors contribute is under General → Data sources.

Connector-specific webhook integrations can have their own payload URL and secret. MCP-based sources are configured through MCP Servers.

For CI events, only finished builds are materialized as build records. Other accepted events can still be retained as signals.

A connector with no metrics using it can remain connected but show a metric count of 0.

Webhooks#

Use Webhooks to configure the organization endpoint for supported inbound events.

Accepted events that pass verification can be stored as signals and used by metrics. Requests that are rejected, invalid, duplicated, or used only as ping events are not stored as normal signals.

The page includes:

Payload URL and Secret for the organization endpoint.
What has arrived — Event types, delivery counts, and the most recent arrival time.
Diagnostics log — Troubleshooting information about recent processing activity.

Signal history and the Diagnostics log serve different purposes. Stored signals provide source data for analysis; diagnostics help investigate processing problems.

Connector-specific SCM and CI webhook endpoints can have behavior different from the organization endpoint. Use Data Sources to configure those connectors.

Monitor health#

Summary cards at the top of the page show the current state of the analysis pipeline. Select Refresh to update the cards and the current tab. A pointer icon means the card is a shortcut.

Active metrics / Threshold alerts — How many metrics are active, and how many of those breach a threshold. Opens the Metrics tab.
Active schedules — Schedules currently enabled. Opens the Schedules tab.
Data feeds — Connected data sources out of the configured total. Opens Configuration → Data Sources.
Gateway ingest / Last ingest — Gateway event volume and the most recent ingest. Opens AI spend overview in System Dashboards.
SCM ingest / Last PR detected — SCM activity and the most recently detected pull request. Opens Configuration → General (SCM watcher).

Use these indicators when a dashboard value is missing or outdated.

Troubleshooting#

A metric or dashboard widget has no value

Confirm that the required data sources are connected.
Compute the metric, or confirm it is Active if a schedule should run it.
Review the metric detail page for validation findings and diagnostics.
A dashboard can show a Draft metric or a metric that has not yet produced a value.

Expected records are missing

Review the configured Join confidence floor.
Lowering the floor can include more records but also accepts lower-confidence joins.
Review the metric diagnostics to determine whether records were excluded.

Values are outdated

Check Last ingest and the metric's last computation time.
Select Refresh to reload the page.
Recompute the metric or use Run now on its schedule.

A schedule does not run

Confirm that the schedule is enabled.
Confirm that the metrics included in it are Active.
Use Run now to test the task immediately.

A team or group is missing from a dashboard

Groups smaller than Minimum group size are hidden. See Minimum group size.