FindingsReportsConnections
Northstar Labs
Reports

Report · 31111111

AI cost evidence report

Jun 28, 2026 – Jul 28, 2026 · Published Jul 28, 2026

completed

Summary

Independent synthesis for this run

The strongest opportunities are in model routing and repeated prompt context. Provider usage and runtime traces agree that a large low-complexity request segment uses the most capable model, while repository evidence shows a stable instruction prefix repeated across calls. Retry behavior is worth tightening, but its measured impact is less certain.

Ranked findings

3 recommendations

01
AIO-MODEL-001Model selection

Route low-complexity extraction traffic to a smaller model

open

Structured extraction accounts for 38% of calls and has narrow outputs, stable latency, and low observed retry rates.

Recommendation

Add a task classifier for the extraction path and canary a smaller model against the existing evaluation set before routing production traffic.

Why it ranks here

Ranked first because usage, traces, and code independently identify the same bounded request path and the change has a reversible rollout.

high confidencemedium quality riskmedium effort

Three independent sources support the task boundary, traffic share, and current model configuration.

Evidence

SourceMetricValueObserved
OpenAI productionextraction request share38 %Jul 28, 2026
Datadog LLM Observabilityextraction success rate99.2 %Jul 28, 2026

Evidence gaps

· No smaller-model quality evaluation has been run on the current extraction dataset.

Validation plan

Replay 1,000 representative extraction requests, require schema accuracy and task quality to remain within the current threshold, then canary 5% of traffic with automatic rollback.

02
AIO-CACHE-001Caching

Cache the stable policy and schema prefix

implemented

A large prompt prefix is identical across extraction calls and is assembled before request-specific context.

Recommendation

Normalize the stable prefix and enable provider prompt caching on the extraction path with tenant isolation in the cache key.

Why it ranks here

Ranked second because evidence is direct and implementation is small, but realized benefit depends on provider cache eligibility.

high confidencelow quality risklow effort

Provider token aggregates and repository code both show the same stable prefix.

Evidence

SourceMetricValueObserved
GitHub organizationstable prefix tokens2840 tokensJul 28, 2026

Evidence gaps

· Provider cache hit rate is not instrumented yet.

Validation plan

Instrument cache eligibility and hit rate, compare output equality on a replay set, and enable for one tenant before wider rollout.

03
AIO-RETRY-001Reliability

Bound duplicate retries after client timeouts

open

A small request cohort is retried after client timeouts even when the original inference later completes.

Recommendation

Add request idempotency, classify retryable failures, and cap retries for the affected path.

Why it ranks here

Ranked third because the behavior is real but trace coverage leaves its total impact uncertain.

medium confidencelow quality riskmedium effort

Runtime traces show duplicate request signatures, but not every provider call has a correlated application request ID.

Evidence

SourceMetricValueObserved
Datadog LLM Observabilityprobable duplicate calls1842 callsJul 28, 2026

Evidence gaps

· 9% of provider calls lack a correlated application request ID.

Validation plan

Add correlation IDs, observe duplicate completion for one week, then enable idempotency on the highest-confidence path.

Other opportunities

Not included in the ranked set

AIO-CONTEXT-002Context efficiency

Tighten retrieval count for support answers

Candidate is plausible, but repository and runtime signals do not yet identify a safe context threshold.