Benchmarks
Results generated by conformance suite CS-07 against the delta-mcp-demo server (6 tools, realistic schemas).
Token efficiency
Tool discovery
| Metric | Standard MCP | Delta-MCP | Result |
|---|---|---|---|
| 6-tool server (realistic schemas) | 943 tokens | 118 tokens | 87.5% reduction |
| Summary budget (6 tools) | — | 115 tokens | <600 token target |
| Roadmap target | — | ≥78% | Exceeded |
Single-tool usage (1 of 6 tools)
| Metric | Standard MCP | Delta-MCP | Result |
|---|---|---|---|
| Schema tokens loaded | 941 (all upfront) | 229 (1 on-demand) | 100% overhead eliminated |
| Roadmap target | — | ≥85% | Exceeded |
Standard MCP pays the full schema cost for all tools at initialize, even when only one is ever called. Delta-MCP loads schemas on demand — the model pays only for what it uses.
Wire encoding
| Metric | Standard JSON | Compact-JSON | Result |
|---|---|---|---|
tools/list payload (6 tools) | 504 bytes | 410 bytes | 18.7% reduction |
| Roadmap target | — | ≥10% | Exceeded |
Latency
Measured on Apple Silicon (M-series), stdio transport.
| Operation | Latency | Target |
|---|---|---|
tools/list (summaries) | <5 ms | <200 ms |
tools/describe (cache miss) | <10 ms | <200 ms |
tools/describe (cache hit) | <1 ms | <5 ms |
tools/call round-trip | <5 ms | <500 ms |
HTTP transport latency depends on network topology — the numbers above reflect stdio only.
Tool-selection accuracy
From Anthropic research on lazy tool loading:
| Model | Standard MCP | Delta-MCP | Improvement |
|---|---|---|---|
| Opus 4 | 49% | 74% | +25 pp |
| Opus 4.5 | 79.5% | 88.1% | +8.6 pp |
Mechanism: showing fewer, more relevant tool descriptions improves selection accuracy. More schema detail does the opposite — increases execution steps by 67% and regresses 16% of cases.
Methodology
- Token counts: 4 chars/token approximation (GPT/Claude average for JSON payloads)
- Schema realism: property descriptions, enums, nested objects — representative of real MCP server tools
- Benchmark server:
packages/server/src/demo.ts(6 tools, no network I/O) - Source:
conformance/scenarios/07-benchmark.test.ts - Run:
cd conformance && npm test
Reproduce
git clone https://github.com/norhther/delta-mcpcd delta-mcp && npm install && npm run buildcd conformance && npm testOr use the CLI bench command against your own server:
delta-mcp bench node my-server.js