CLike Harper Telemetry Portal
Overview
The CLike Harper Telemetry Portal is a lightweight operational dashboard for inspecting Harper pipeline execution data across IDEA, SPEC, PLAN, KIT, EVAL, GATE, and FINALIZE phases.
It provides engineering teams with a centralized view of model usage, run duration, token consumption, generated artifacts, evaluation outcomes, and gate decisions. The portal supports the CLike objective of making AI-native development observable, auditable, and continuously improvable.
Default local URL:
http://localhost:8000/v1/metrics/harper/ui
Access. The page itself is public on loopback, but the telemetry API requires the CLike service token. On first load the portal asks for
CLIKE_API_TOKENand exchanges it for anHttpOnly,SameSite=Strictcookie scoped to/v1/metrics(8 hours).POST /v1/metrics/logoutclears it.
The portal is served by the Gateway telemetry routes under:
/v1/metrics/harper
Functional Purpose
The Telemetry Portal helps product engineers and platform owners answer practical questions such as:
- Which Harper phases were executed for a project?
- Which model/provider was used for each run?
- How many tokens were consumed?
- How long did each phase take?
- Which REQs passed or failed evaluation?
- Which REQs were promoted or blocked by the gate?
- Which runs are expensive, slow, unstable, or repeatedly failing?
- How does Agent execution compare with Cloud execution?
The portal is not only a reporting UI. It is part of the Eval-Driven Development feedback loop: telemetry data helps identify bottlenecks, model quality issues, prompt regressions, and pipeline defects.
Local-agent runs
Harper phases run by Claude Code or Codex do not call a provider through the gateway: the
orchestrator reports them with POST /v1/harper/telemetry (from /local-agent/complete). Their
rows have execution: "local_agent", provider/executor = claude_code | gpt_codex, the
model reported by the agent, usage, duration_ms and pricing.total_cost as the
API-equivalent cost (unit: usd_api_equivalent; subscription runs are not billed per token).
Claude Code reports that cost itself. Codex does not: the portal estimates it from the model’s
price in configs/models.yaml (input_per_1k, cached_input_per_1k, output_per_1k; Codex
counts cache reads inside input_tokens), marked pricing.estimated: true, also for rows written
before. A model without a price (or a wrong model name) stays at zero.
Prompt cache. Costs include the provider prompt cache, priced from configs/models.yaml
(OpenAI and Anthropic list prices): OpenAI counts cached input inside input_tokens
(input_tokens_details.cached_tokens) and it is charged at cached_input_per_1k; Anthropic
reports cache reads (cache_read_input_tokens, cached_input_per_1k) and writes
(cache_creation_input_tokens, cache_write_per_1k for the 5-minute TTL, cache_write_1h_per_1k
for 1 hour) apart from input_tokens. Each row’s pricing has cached_tokens, cached_cost,
cache_write_tokens, cache_write_cost; the aggregate has tokens_cached per phase and day and
total_tokens_cached, and the raw table a Cached column. Rows recorded before this (OpenAI
cache charged as normal input, Anthropic cache not charged) are recomputed when shown
(pricing.cache_recomputed).
Requirements, drill-down, KPIs and estimate
Four views over the selected project or file:
- Overview → click a phase in Cost by Phase: the requirements behind that phase (runs, time, cost, tokens), ranked by cost; a row opens the requirement.
- Requirements: the ranking of every REQ — cost, work time (agent/model time and eval sandbox time), elapsed time (start of its first run to end of its last), KIT runs and repairs, evals and failed evals, the first eval’s outcome, the last gate, input tokens — sortable by any column, with the top 15 by a chosen metric. A row opens its breakdown per phase (time, cost, tokens, cache).
- KPIs & estimate: over the REQs of the selection — cost and work time per REQ (median, p90, mean), elapsed per REQ, repairs per REQ, eval first-pass rate and evals per REQ, gate pass rate, prompt cache hit rate, agent share of time, and the cost of the phases without a REQ (idea, spec, plan, extend, finalize). Estimate: for N more requirements of similar size, cost and work time from median (typical) to p90 (prudent) of this project’s history.
Endpoint: GET /v1/metrics/harper/reqs?project_id=…&estimate_reqs=N (reqs_file?relpath=… for one file).
Rows of an eval or gate (execution: eval_sandbox) count as eval time; the others as agent/model time.
Technical Architecture
The telemetry UI is exposed by the Gateway service.
VS Code Extension
-> Orchestrator
-> Gateway
-> /v1/metrics/harper/ui
-> /v1/metrics/harper/*
Telemetry files are read from the configured Harper telemetry directory:
HARPER_TELEMETRY_DIR
Default directory:
/workspace/telemetry
Each Harper phase can emit telemetry containing:
- project ID
- phase name
- run ID
- provider
- model
- context window
- prompt/input tokens
- completion/output tokens
- duration
- generated or changed files
- test counts
- eval status
- gate decisions
- error and warning metadata
Main Endpoints
Telemetry UI
GET /v1/metrics/harper/ui
Browser-based dashboard for interactive telemetry inspection.
Projects
GET /v1/metrics/harper/projects
Lists projects with available telemetry data.
Files
GET /v1/metrics/harper/files
Lists raw telemetry files available to the Gateway.
Aggregate Metrics
GET /v1/metrics/harper/aggregate?project_id=<PROJECT_ID>
Returns aggregated metrics for a project.
Typical use cases:
- total runs by phase
- total token usage
- total duration
- pass/fail counts
- model/provider distribution
Time Series
GET /v1/metrics/harper/series?project_id=<PROJECT_ID>
Returns time-series telemetry for trend analysis.
Typical use cases:
- token usage over time
- phase duration trend
- evaluation stability
- gate pass/fail trend
Top Runs
GET /v1/metrics/harper/top?project_id=<PROJECT_ID>&limit=10
Returns top runs by cost, duration, token usage, or other ranking criteria exposed by the Gateway.
Raw Telemetry
GET /v1/metrics/harper/raw?project_id=<PROJECT_ID>
Returns raw telemetry records for debugging or external analysis.
Why It Matters
CLike is designed as an AI-native software engineering pipeline where human intent is translated into verifiable software through short, governed iterations.
Telemetry is essential because AI-generated software must not be treated as a black box. Every run should be observable and explainable.
The portal supports:
- governance
- auditability
- reproducibility
- cost control
- model comparison
- eval-driven improvement
- agent vs cloud execution analysis
- detection of recurring prompt or pipeline failures
Example Workflow
A typical usage flow is:
/spec
/plan
/kit REQ-003
/eval REQ-003
/gate REQ-003
Then open:
http://localhost:8000/v1/metrics/harper/ui
Use the portal to inspect:
- The model used for each phase.
- Duration and token usage.
- Whether
/evalpassed or failed. - Which test or lint case failed.
- Whether
/gatepromoted or blocked the REQ. - Whether the failure came from code, runtime setup, missing dependencies, or structural gate blockers.
Operational Value
The Telemetry Portal gives CLike a measurable feedback loop.
Without telemetry, AI-native development risks becoming opaque: models generate code, agents execute actions, and developers only see the final output.
With telemetry, every phase becomes measurable:
intent -> generation -> evaluation -> gate -> promotion
This allows the team to improve:
- prompt quality
- model routing
- agent package design
- eval contracts
- gate policies
- dependency handling
- cloud vs local agent performance
Recommended Slide Summary
The CLike Harper Telemetry Portal provides a real-time operational view of the AI-native development pipeline. It captures model, token, duration, artifact, eval, and gate data for each Harper phase, enabling teams to compare Cloud and Agent executions, detect failures, control costs, and continuously improve prompt, eval, and gate quality. It turns vibe coding into an auditable, eval-driven engineering workflow.
Recommended Checks
Before using the portal, verify:
echo "$HARPER_TELEMETRY_DIR"
Start the Gateway and open:
http://localhost:8000/v1/metrics/harper/ui
If the portal shows no data:
- Confirm that Harper commands generated telemetry files.
- Confirm the Gateway can read
HARPER_TELEMETRY_DIR. - Confirm the project ID matches the telemetry records.
- Check Gateway logs for file parsing or path errors.
Future Improvements
Recommended next improvements:
- Add Cloud vs Agent comparison panels.
- Add cost estimation per provider/model.
- Add failure taxonomy: code failure, runtime failure, dependency failure, gate failure.
- Add per-REQ trend analysis.
- Add downloadable CSV/JSON exports.
- Add model quality scorecards.
- Add gate blocker visualization.
- Add eval case drill-down with raw logs.