Building an Honest AI Model Dashboard

The AI benchmark space has a credibility problem. Numbers get copy-pasted, contexts get stripped, and you end up with "intelligence scores" that are optimistic marketing dressed as engineering.

The dashboard on this site works differently. Here are the rules it follows.

Rule 1: Never fabricate a number

If a data point doesn't exist, we render N/A. OpenRouter's /models endpoint is excellent for pricing but does not expose per-model throughput. So the Speed column is permanently N/A:

export interface ModelRow {
  // ...
  speed: null; // Always null — OpenRouter doesn't expose this reliably
}

Rather than pulling a number out of thin air, the UI renders the column with an honest label. Future me might wire in a benchmarking API; until then, the field stays null.

Rule 2: Separate what you curate from what you fetch

The dashboard has two data sources, and they're kept completely separate:

| Source | What it provides | How it's accessed | |---|---|---| | OpenRouter /api/v1/models | Pricing ($/1M tokens) | Server Component, revalidate: 3600 | | /data/model-scores.json | Intelligence scores | Local file read |

The getModelRows() function in src/lib/models.ts merges them. Only models that appear in the curated JSON file are shown — that's what keeps 400+ OpenRouter models from flooding the chart.

Rule 3: Disclose the cost-per-task formula

"Cost per task" is a derived metric. The formula assumes a configurable token count:

export const TOKEN_ASSUMPTION = {
  inputTokens: 1_000,
  outputTokens: 500,
  label: "1 K input + 500 output tokens",
};
 
// costPerTask = (promptPrice * INPUT + completionPrice * OUTPUT) / 1_000_000

The assumption is surfaced in the UI so the number is interpretable. If your tasks are long-context reasoning calls, tune TOKEN_ASSUMPTION in src/lib/models.ts and every figure in every chart re-derives from it automatically.

The caching model

const res = await fetch("https://openrouter.ai/api/v1/models", {
  next: { revalidate: 3600 }, // Next.js fetch cache: 1-hour TTL
});

This runs in a Server Component, so the API key stays server-side only. If the fetch fails (network error, missing key, rate limit), getModelRows() returns null from the live path and the UI falls back to hardcoded fallback prices with a visible amber warning badge. The dashboard never goes blank.

What's next

Phase 3 ships the node-based workflow visualizer — an interactive canvas of the real agent architectures I build for clients, built with @xyflow/react. Coming soon.