The measurement and reference-implementation hub for testing bounded claims about compression, expression, memory, recursion, preserved reusable structure, and total computational cost .
The benchmark system connects the governing canon, preregistered evaluation protocols, matched baselines, versioned GitHub artifacts, machine-readable results, and evidence-state governance.
Current Public Release
Published August 16, 2026, this immutable GitHub release provides a stable and citable reference point for reproducible inspection, implementation review, benchmark evaluation, and future result comparison.
Release tag: benchmarks-v0.2.0 • Governing authority: GC-MRD-v2.0
Current evidence boundary: This page and repository provide an evaluation architecture and public reference implementation. They do not establish universal validation of Robbie’s Razor, certify a model, guarantee lower resource use, or convert implementation into independent empirical support.
Authority Path
GC-MRD-v2.0 → Robbie’s Razor → Lab Evaluation Protocol → Versioned Benchmark → Bounded Result → Evidence State
Canonical definitions remain governed by the Grand Compression Master Reference Document . Repository code, benchmark tools, diagnostics, and examples remain implementation and evaluation layers.
1 · Scope & Definition
Robbie’s Razor Benchmarks are controlled evaluation structures for testing whether a declared system produces accepted results through more efficient, reusable, and stable reasoning under constraint.
The benchmarks do not ask only whether a system can solve a task. They also examine the path used to reach the result, the amount of expansion and correction required, whether validated structure is preserved for reuse, and the total cost contained within the declared system boundary.
RC-01 · Canonical Robbie’s Razor
“When competing explanations exist, prefer the model that follows compression → expression → memory → recursion.”
A benchmark result is not just a score. Its minimum reproducible unit is the complete declared evaluation record:
System Version + Baseline + Task Set + Constraints + Metrics + Quality Threshold + Failure Rule + Evidence State
| Layer | Function | Authority Boundary |
|---|---|---|
| Canonical Claim | Defines the proposition, scope, and terms being tested. | Governed by GC-MRD-v2.0 and the canonical claims register. |
| Evaluation Protocol | Declares the baseline, tasks, metrics, thresholds, and failure conditions. | Determines what the test may validly conclude. |
| Reference Implementation | Provides versioned code, configurations, diagnostics, examples, and machine-readable artifacts. | Implementation demonstrates a method, not empirical confirmation. |
| Evidence Record | Preserves results, limitations, uncertainty, replication status, and evidence state. | Remains bounded to the tested system and declared conditions. |
The public benchmark system is a versioned reference implementation and evaluation architecture. Individual benchmark claims must earn their own evidence states through preregistered tests and appropriate replication. The existence of the repository does not place the entire framework in the “Supported” state.
Repository boundary: The GitHub repository contains implementation, documentation, governance, diagnostic, test, and knowledge-architecture assets. Where repository language conflicts with the current GC-MRD-v2.0 canon, the current MRD governs.
2 · Canon, Implementation & Evidence
Benchmark integrity depends on keeping canonical definitions, evaluation procedures, implementation code, observed results, and evidence states separate. Each layer answers a different question and carries a different level of authority.
Governing authority: GC-MRD-v2.0 defines the current canon. Section 13 governs predictive compression, preserved reusable structure, evaluation design, reference-implementation boundaries, evidence states, and domain transfer. Provisional mathematical work remains provisional wherever identified, including Appendix Q.
| Layer | Role | Cannot Do |
|---|---|---|
| GC-MRD-v2.0 | Defines the current framework, canonical claims, scope boundaries, and evidence discipline. | Cannot make an empirical result true merely by defining it. |
| Canonical Claims Register | Identifies testable claims, their wording, scope, and current status. | Cannot replace an evaluation protocol or result record. |
| Lab Evaluation Protocol | Declares how the claim will be tested and what would count as success or failure. | Cannot change its decision threshold after results are known. |
| GitHub Implementation | Provides inspectable code, configurations, baselines, diagnostics, tests, and version history. | Cannot silently redefine the canon or establish validation by existing. |
| Result Artifact | Records observations, exclusions, uncertainty, limitations, and calculated deltas. | Cannot support conclusions outside its declared test boundary. |
| Evidence State | Communicates the status of a bounded claim after evaluation. | Cannot be inherited automatically by another domain, model, or version. |
RC-19
Declare predictions, baselines, metrics, thresholds, failure conditions, and exclusion rules before results are interpreted.
RC-20
Include the relevant cost of producing, correcting, validating, preserving, retrieving, and reusing an accepted result.
RC-21
Implementing Robbie’s Razor, RKCA, a metric, or a benchmark does not demonstrate that it improves performance.
RC-22
A result supported for one system, workload, configuration, or domain requires revalidation before transfer elsewhere.
“Documented,” “Calculated,” “Inferred,” “Proposed,” and “Unknown” may be used as source or finding labels. They do not replace the formal GC-MRD-v2.0 evidence-state ladder.
Interactive-tool boundary: An auditor, chatbot, Gem, prompt, dashboard, or visualization may help users explore the framework. Its output is not a benchmark result unless it follows the declared protocol and produces a versioned evidence record.
3 · What Gets Measured
The benchmark model begins with task quality and then measures how an accepted result was produced. A lower-cost output is not more efficient if it is incomplete, incorrect, unsafe, or requires greater downstream repair.
Quality-gate rule: Compare computational efficiency only after the baseline and Razor-guided configurations are evaluated against the same predeclared acceptance threshold.
| Dimension | Benchmark Question | Example Observables |
|---|---|---|
| Accepted Outcome | Did the result meet the same declared quality and completion threshold? | Accuracy, completeness, safety, usefulness, human correction |
| Compression | How much relevant structure was preserved while unnecessary expansion was reduced? | Context size, tokens, retrieval volume, active parameters where disclosed |
| Expression | Was the compressed structure expressed as a correct, usable result? | Task success, output quality, tool execution, latency, format compliance |
| Memory | Was validated structure preserved with enough identity, provenance, constraints, and relationships for later reuse? | Cache hits, retained state, retrieval accuracy, version continuity, memory growth |
| Recursion | Did preserved structure improve the next reasoning cycle without destabilizing the system? | Reuse rate, re-derivation, retries, corrections, backtracking, long-horizon completion |
| Total Cost | What did the complete accepted task cost across all declared layers? | Compute, tools, retrieval, storage, network, validation, human review, energy |
Accepted Tasks ÷ Attempted Tasks
Total Measured Cost ÷ Accepted Tasks
Valid Reuse Events ÷ Eligible Reuse Events
Razor-Guided Result − Baseline Result
Status: These are operational reporting templates, not new canonical equations. Each result must declare its units, calculation method, uncertainty, exclusions, and whether a lower or higher value represents improvement.
Use system telemetry when available: tokens, tool calls, cache events, retries, latency, memory, utilization, energy, and accepted-task outcomes.
When direct access is unavailable, use declared observable proxies and label the limitation. A proxy must not be reported as hidden internal telemetry.
Reasoning privacy: A benchmark does not need private chain-of-thought disclosure. It can evaluate accepted outcomes, tool traces, retries, state transitions, memory use, resource consumption, and other observable behavior without requiring hidden reasoning content.
4 · System Architecture
The benchmark system is organized as a layered architecture. Canon defines what may be tested, the protocol defines how to test it, the repository provides the implementation, and result records preserve what was observed.
Layer-separation rule: No layer may impersonate another. A webpage does not replace the canon, code does not replace a protocol, an interactive tool does not replace a controlled run, and a benchmark score does not replace an evidence record.
Layer 1
GC-MRD-v2.0 and Robbie’s Razor define the claims, terms, invariants, boundaries, and evidence discipline.
Layer 2
The Lab Evaluation Protocol fixes the prediction, baseline, task set, metrics, quality threshold, and failure rules.
Layer 3
Versioned baselines establish the comparison condition against which a Razor-guided intervention is measured.
Layer 4
Benchmark code, task fixtures, configurations, and runners implement the declared comparison under controlled conditions.
Layer 5
Metrics calculate declared outcomes while diagnostics identify anomalies, regressions, missing telemetry, and possible failure conditions.
Layer 6
Versioned artifacts preserve observations, calculations, exclusions, uncertainty, provenance, review, replication, and evidence state.
| Repository Area | Benchmark Function |
|---|---|
baselines/ |
Comparison conditions and reference behavior |
benchmarks/ |
Benchmark implementations, tasks, and evaluation surfaces |
razor_metrics/ |
Metric definitions and calculation logic |
diagnostics/ |
Diagnostic checks, interpretation support, and anomaly detection |
src/, tools/ |
Shared source code, utilities, and execution support |
tests/, .github/workflows/ |
Automated verification and repository-level quality controls |
governance/, docs/ |
Governance records, specifications, instructions, and supporting documentation |
notebooks/ |
Exploratory analysis and reproducible research workflows |
Canonical Claim → Preregistered Protocol → Versioned Baseline → Benchmark Run → Diagnostic Review → Result Artifact → Evidence State → Replication
5 · Test Families
A benchmark family groups tests that ask a related question. Individual tests may use different models, tasks, tools, memory systems, or infrastructure, but results remain comparable only when their protocols and measurement boundaries match.
Family-status rule: The presence of a benchmark family on this page does not mean every test in that family is implemented or empirically supported. Each test must identify whether it is implemented, proposed, testing, archived, or independently replicated.
Core Family
Tests whether the system completes the task correctly, safely, and usefully before resource-efficiency claims are considered.
Implemented Metrics
Measures task-normalized token use, structural redundancy, context expansion, and other declared resource costs at equal or better quality.
Implemented Harness
Tests whether preserved state, cached structure, or validated results reduce redundant recomputation without introducing stale or incorrect reuse.
Implemented Metrics
Tracks retries, abandoned paths, contradictions, corrections, repeated tool calls, and other observable signs of avoidable rework.
Repository Fixture
Compares retrieval of preserved structure with regeneration from raw inputs, including correctness, provenance, latency, and total task cost.
Protocol-Dependent
Tests whether performance and coherence remain stable as tasks become longer, more branching, stateful, or constrained.
Implemented Metric
Measures whether tool calls, retrieval operations, external actions, and validation steps contribute proportionate value to accepted task completion.
Telemetry-Dependent
Extends task measurement into compute utilization, energy, cooling, water, emissions, networking, storage, and hardware allocation where access permits.
| Visible Asset | Current Function | Boundary |
|---|---|---|
benchmark_memory_gate_savings.py |
Memory-hit, token-reuse, and redundant-recomputation demonstration | Reference harness; not universal validation |
evaluator.py |
Evaluates structured outputs against declared metrics | Depends on the supplied cases and scoring rules |
cases/ |
Structured evaluation inputs and fixtures | Coverage does not imply domain representativeness |
refractive-truth/ |
Retrieval-versus-recomputation evaluation surface | Interpret only within its declared fixtures |
sample_outputs.csv / .json |
Example structured result formats | Examples are not production evidence |
Version note: Repository files may preserve historical MRD references for provenance. Current interpretation must resolve through GC-MRD-v2.0.
6 · Matched Comparison Design
A Robbie’s Razor benchmark is comparative. It must measure a declared intervention against a credible baseline under conditions that isolate the feature being tested.
Baseline rule: “Conventional reasoning” is not a sufficient baseline description. The evaluator must record the exact system, version, configuration, prompt, memory state, tools, limits, and scoring method.
Compares the Razor-guided configuration with the same system operating under its standard configuration.
Removes one component—such as memory, retrieval, compression control, or reuse—to isolate its contribution.
Disables eligible preserved-state reuse while holding the model, task, tools, and quality standard constant.
Requires reconstruction from the allowed raw inputs instead of retrieving preserved reusable structure.
Compares two declared versions while separating model improvements from the Razor-guided intervention.
Uses another documented system only when access, task conditions, quality standards, and measurement boundaries can be matched.
| Control | Required Record | Risk if Unmatched |
|---|---|---|
| Task Set | Same tasks, sampling rules, difficulty distribution, and exclusions | One condition receives easier work |
| System Version | Model, software, API, dependencies, and configuration | Uncontrolled system changes explain the delta |
| Prompt & Instructions | Exact prompts, system instructions, templates, and formatting | Prompt quality is confused with architecture |
| Tools & Retrieval | Available tools, permissions, indexes, documents, and network access | One system receives outside information or capabilities |
| Memory State | Starting memory, cache state, retention policy, and eligible reuse | Warm and cold starts are mixed |
| Budgets & Limits | Token, time, tool-call, retry, compute, and monetary limits | One system receives more resources |
| Execution Environment | Hardware, region, load, network, runtime, and measurement period | Infrastructure conditions distort latency or energy |
| Quality Judge | Rubric, ground truth, human review, automated judge, and tie handling | Scoring bias creates the apparent result |
Invalid comparison: A baseline chosen, altered, or discarded after results are observed cannot support the preregistered claim. It may generate a new hypothesis, but that hypothesis requires a new evaluation.
7 · Required Metric Bundle
No single score can establish Robbie’s Razor performance. A valid evaluation combines outcome quality, expansion, rework, memory, reuse, latency, tools, and total cost.
Interpretation rule: Every metric must declare its unit, direction of improvement, aggregation method, uncertainty, and relationship to the accepted-task threshold.
| Metric | Operational Meaning | Typical Direction | Required Qualification |
|---|---|---|---|
| Accepted-Task Rate | Accepted tasks divided by attempted tasks | Higher | Acceptance rubric and judge |
| TPCA | Tokens per correct or accepted answer | Lower at equal quality | Token categories and counting method |
| Backtracking Frequency | Observable corrections, abandoned paths, retries, or repeated operations per task | Lower when unnecessary | Backtracking-event definition |
| Contradiction / Drift Rate | Material inconsistency with prior accepted state or task constraints | Lower | Contradiction taxonomy and severity |
| Memory-Hit Ratio | Valid memory hits divided by eligible memory lookups | Context-dependent | Eligibility and validity rules |
| Valid Reuse Rate | Correct reuse of preserved structure divided by eligible reuse opportunities | Higher without stale reuse | Identity, provenance, and version checks |
| Re-Derivation Rate | Previously available valid structure regenerated instead of reused | Lower when reuse is appropriate | Proof that reusable structure was available |
| Tool-Call Efficiency | Useful tool activity relative to accepted task completion | Task-dependent | Useful, failed, and repeated call definitions |
| Latency per Accepted Task | End-to-end elapsed time divided by accepted tasks | Lower within quality boundary | Network, queue, and tool-time treatment |
| Total Cost per Accepted Task | Declared computation, tools, storage, validation, and other cost divided by accepted tasks | Lower at equal or better quality | Complete system boundary and units |
Repeated structure that does not contribute to the accepted result.
Change in active context or retained state across task depth.
Relevant validated structures divided by retrieved structures.
Resources required to repair rejected or unstable outputs.
Reviewer time and interventions required per accepted task.
Maximum declared memory or cache consumption during execution.
Lower token use can hide lower quality. Higher memory reuse can spread stale information. Fewer tool calls can reflect missing verification. Faster completion can increase correction burden. The bundle must be interpreted together.
8 · Preserved Reusable Structure
Memory is not simply stored text, accumulated context, or a larger database. For benchmark purposes, memory becomes valuable when validated structure can be retrieved accurately and reused while preserving identity, provenance, relationships, constraints, and version state.
The benchmark question is therefore not merely whether information was stored. It is whether prior computational work created a state that remains useful enough to reduce later work after the costs of preservation, retrieval, verification, invalidation, maintenance, correction, and error are included.
This section is where the benchmark system connects directly to Measuring the Compression Dividend . That page defines the proposed measurement architecture; this benchmark layer provides the controlled conditions, metrics, fixtures, and versioned evidence records needed to determine whether the predicted reuse advantage actually appears.
RC-18 preservation boundary: Reusable structure must retain enough identity, relationships, provenance, constraints, version state, and retrieval pathways to support valid future use. Mere storage does not satisfy the principle.
Information exists in memory, a cache, database, registry, file, context window, or another retained state.
The correct structure can be located when needed without excessive search, irrelevant context, identity collision, or routing overhead.
Retrieved structure remains current, appropriately scoped, traceable, and compatible with the active task and its constraints.
Preserved structure reduces later work while maintaining or improving the same declared accepted-task quality threshold.
Benchmark Decision Boundary
A benchmark should distinguish a task that genuinely requires new computation from a task whose relevant state has already been reliably resolved and remains valid for reuse.
Why spend computational work rediscovering what the system can already retrieve, verify, and responsibly reuse?
Measurement Definition Layer
Measuring the Compression Dividend defines the working quantities that this benchmark family is intended to test: Net Reuse Benefit, Reuse Break-Even Point, Cumulative Compression Dividend, and the candidate Knowledge Yield / Compute measurement family.
The measurement page does not supply the empirical answer. The benchmark system must determine whether those quantities become positive, negative, neutral, or inconclusive under a declared model, workload, system version, quality threshold, and total-cost boundary.
Does avoided recomputation exceed the full cost of valid reuse?
How many valid reuse events are required to recover the preservation investment?
Does repeated valid reuse create a durable positive cumulative advantage?
How productively does computational work create reliable reusable knowledge?
| Condition | Purpose | Primary Risk Tested |
|---|---|---|
| Cold Start | Measure completion when eligible preserved state is unavailable. | Hidden warm-cache or prior-state advantage. |
| Warm Start | Measure the same task when validated reusable state is available. | Failure to exploit valid memory. |
| Matched Recomputation Baseline | Require reconstruction from allowed raw inputs rather than retrieval of the preserved state. | Claiming avoided work without measuring reconstruction cost. |
| Changed-State Test | Change facts, constraints, relationships, or versions after storage. | Stale or incorrectly inherited structure. |
| Near-Match Test | Introduce similar but non-equivalent identities, requests, or states. | False reuse caused by semantic or structural resemblance. |
| Invalidation Test | Withdraw or supersede previously valid structure. | Failure to retire invalid memory. |
| Scale Test | Increase the number of retained structures and retrieval choices. | Search overhead, identity collision, memory growth, and context bloat. |
Correct and valid memory hits divided by eligible lookup events.
Correct reuse events divided by eligible opportunities for reuse.
Incorrect or contextually invalid reuse events divided by attempted reuse events.
Reuse of superseded, expired, or invalidated structure.
Previously available valid structure regenerated instead of reused.
Measured work in the matched recomputation baseline that was not required when valid reuse succeeded.
Time, tokens, compute, calls, or monetary cost required to locate and validate preserved state.
Resources required to create, store, index, version, validate, synchronize, update, and invalidate reusable state.
Operational Accounting Template
Net Reuse Benefit = Avoided Recomputation − Preservation − Retrieval − Verification − Maintenance − Repair
Status: This remains an operational accounting template rather than a universal canonical equation. Every study must define the unit of cost, complete system boundary, attribution rules, quality threshold, time horizon, uncertainty, and treatment of failed reuse.
Cumulative Reuse Test
Preserving reusable structure can initially cost more than solving a task once. A benchmark should therefore record how many successful reuse events are required before cumulative avoided recomputation recovers the cost of creating and sustaining that state.
Cumulative Avoided Recomputation ≥ Cumulative Reuse-System Cost
A state that never reaches break-even may still provide other benefits, such as provenance or lower latency, but it has not demonstrated a positive cumulative cost dividend.
Derived Measurement Concept
The Compression Dividend describes the proposed cumulative advantage created when prior computational work becomes persistent, valid, reusable structure and thereby reduces the total cost of appropriate future work.
Within the benchmark system, the dividend should not be inferred merely because memory exists, retrieval succeeds, or a cache is hit. It requires a matched comparison demonstrating that valid reuse avoided measurable reconstruction while maintaining the required task quality.
Prior Work → Preserved State → Valid Reuse → Avoided Recomputation → Net Reuse Benefit → Break-Even → Cumulative Dividend
Evidence boundary: The Compression Dividend is an authored and testable measurement concept. Its existence, magnitude, durability, and scope must be established separately for each declared system, workload, version, quality threshold, and accounting boundary.
Read the Full Measurement Architecture →Candidate Evaluation Category
A related benchmark question is how much reliable, reusable, task-relevant knowledge a system creates relative to the computational work required to create, validate, preserve, and maintain it.
Knowledge Yield / Compute remains a candidate measurement family rather than a validated universal score. Early evaluations should report its underlying components separately instead of collapsing them prematurely into one composite number.
Why It Matters
Develops the broader economic and systems argument for persistent informational value.
What to Measure
Defines Net Reuse Benefit, break-even, cumulative dividend, Quality Gate, and candidate knowledge-productivity measures.
What Happened
Executes controlled comparisons and preserves versioned observations, metrics, results, uncertainty, and evidence state.
Compression Dividend Evaluation Spine
Predicted Advantage → Measurement Definition → Preregistered Protocol → Matched Recomputation Baseline → Governed-Reuse Treatment → Versioned Result → Evidence State → Replication
Implementation boundary: The current benchmark repository includes memory-gated reuse and retrieval-versus-recomputation evaluation surfaces. Their existence demonstrates an implemented measurement path; individual results remain bounded by their fixtures, protocol, system versions, Quality Gate, accounting boundary, uncertainty, and evidence state.
9 · Stability Under Constraint
Recursive stability measures whether a system can preserve useful coherence across repeated reasoning, retrieval, action, feedback, correction, and memory-update cycles while operating under declared constraints.
Stability does not mean repeating the same answer. A stable system must update when evidence changes, recover from error, preserve valid structure, and avoid allowing correction demand or context growth to overwhelm useful task progress.
Benchmark question: As task depth and constraint increase, does the system maintain accepted-result quality with proportionate growth in correction, memory, computation, and governance overhead?
Number of dependent steps or state transitions required.
Alternative paths, choices, tools, or hypotheses introduced.
Maximum active context allowed during execution.
Storage, cache, retained state, and retrieval limits.
Number, cost, latency, and permissions of external actions.
Wall-clock, token, processing, or monetary limits.
Updates that require revision of previously valid structure.
Controlled faults used to test detection and recovery.
| Measurement | What It Reveals | Failure Signal |
|---|---|---|
| Quality Retention | Accepted-task performance as depth increases | Sharp quality decline under modest added depth |
| Drift Rate | Loss of alignment with accepted state or task constraints | Increasing unsupported deviation |
| Correction Demand | Repair operations required per state transition | Correction grows faster than useful progress |
| Resource-Growth Slope | How tokens, memory, tools, and time grow with task depth | Disproportionate expansion without quality gain |
| Recovery Rate | Successful return to valid state after controlled error | Error propagation or repeated failed recovery |
| Completion Boundary | Highest declared constraint level meeting the quality threshold | Premature failure or unstable completion |
Scope boundary: A system that remains stable on one benchmark has demonstrated stability only for the tested version, workload, constraints, tools, and evaluation period. RC-22 requires revalidation before domain transfer.
10 · Physical Resource Accounting
Computational activity occurs through physical infrastructure. A benchmark may extend into energy and environmental accounting only when the study can establish a defensible connection between accepted tasks and the measured resources consumed.
No-conversion shortcut: Token savings must not be translated directly into energy, water, emissions, or ecological benefit without workload-specific hardware, utilization, facility, time, and attribution data.
| Evidence Level | What May Be Reported | What Remains Unresolved |
|---|---|---|
| Token Measurement | Task-normalized token delta under the declared tokenizer | Compute, energy, cooling, water, and emissions |
| Compute Telemetry | Measured runtime, utilization, memory, processor activity, or compute allocation | Complete facility and lifecycle impact |
| IT Energy | Measured electricity used by declared computing equipment | Cooling, conversion loss, and facility overhead |
| Facility Energy | Attributed IT and facility electricity under a declared method | Water, grid emissions, construction, and hardware lifecycle |
| Operational Environment | Energy, cooling, water, and emissions estimates for the measured period | Embodied and supply-chain impact unless separately measured |
| Lifecycle Boundary | Operational and embodied impact within a declared allocation model | Effects outside the selected lifecycle boundary |
Measured energy per accepted task.
Total energy used during the declared evaluation period.
Site and supply-chain water reported separately where possible.
Grid source, time, region, factors, and uncertainty.
Embodied impact allocated across utilization and service life.
Measurement error, estimation ranges, and missing boundaries.
Task-Normalized Reporting
Environmental Impact per Accepted Task = Total Attributed Impact ÷ Accepted Tasks
Status: This is an operational reporting template. It does not establish one universal conversion factor across models, processors, data centers, regions, or time periods.
Joules per Coherent Transition cannot be calculated from tokens alone. A study using JCT must define the coherent transition, directly measure or defensibly attribute energy to it, apply the same definition to both conditions, and report uncertainty and excluded infrastructure.
Rebound boundary: Lower impact per task does not prove lower total impact. If reduced cost increases task volume, absolute energy, water, hardware, or emissions may still rise.
11 · Versioned Implementation
The public GitHub repository is the versioned implementation layer for benchmark code, baselines, metrics, diagnostics, governance, documentation, machine-readable examples, and supporting tests.
Current Immutable Release
Published August 16, 2026, this release establishes a stable, citable implementation record aligned with GC-MRD-v2.0. It preserves the repository state used for inspection, evaluation, reproducibility, and comparison with later benchmark versions.
Release Version
0.2.0
Git Tag
benchmarks-v0.2.0
Governing Authority
GC-MRD-v2.0
Publication State
Public & Immutable
Repository boundary: A file’s presence proves that the artifact was published in the repository. A versioned release preserves an identifiable implementation state, but neither publication nor immutability proves that its claims were empirically validated, independently replicated, or incorporated into the governing canon.
| Repository Area | Primary Content | Evidence Meaning |
|---|---|---|
benchmarks/ |
Executable harnesses, evaluators, cases, samples, and utilities | Implemented evaluation surface |
baselines/ |
Reference conditions and comparison behavior | Declared comparison layer |
razor_metrics/ |
Metric definitions, calculations, and reporting logic | Implementation of declared measures |
diagnostics/ |
Diagnostic checks and interpretive support | Supporting analysis, not an automatic verdict |
src/ and tools/ |
Shared implementation code and utilities | Software infrastructure |
tests/ |
Tests for repository code and expected behavior | Software verification, not framework validation |
notebooks/ |
Exploratory and reproducible analyses | Exploratory unless promoted through protocol |
governance/ |
Interpretive scope, attribution, and non-claim boundaries | Governance context, not performance evidence |
docs/ |
Specifications, empirical notes, examples, JSON-LD, skills, registries, and knowledge architecture | Mixed assets requiring individual classification |
.github/workflows/ |
Automated repository checks and workflows | Continuous software-quality controls |
CITATION.cff |
Version, authorship, publication date, canonical page, repository, and release references | Machine-readable citation and provenance record |
| GitHub Releases | Immutable tagged snapshots of published benchmark versions | Stable implementation identity, not empirical validation |
Every benchmark-related artifact should identify its role so readers, researchers, and machine agents do not confuse examples, implementations, and evidence-bearing results.
Version-resolution rule: Historical files may retain earlier MRD identifiers and terminology for provenance. When an older repository artifact conflicts with GC-MRD-v2.0, the current MRD governs unless the evaluation explicitly studies the historical version. Published releases remain preserved as historical implementation records and must not silently redefine the governing canon.
12 · Evaluation Workflow
A repository demonstration can confirm that code executes. A formal Robbie’s Razor benchmark requires something much stricter: a frozen authority package, a preregistered Reference Contract, a valid governed target, pinned versions, matched comparison conditions, a shared Quality Gate, complete measurement records, and a declared result-classification rule.
The benchmark therefore begins before the model is called. When a study depends on governed external or internal knowledge, the first experimental question is whether the registered authority surfaces can establish the state against which later preservation and quality will be judged.
Reproducibility rule: Preserve enough information for another evaluator to reconstruct the authority package, Reference Contract, target-construction procedure, system, task set, intervention, baseline, metrics, execution environment, Quality Gate, economic boundary, and result classification.
Upstream Reference Architecture
Metrology for Meaning™ defines the upstream reference problem: how a governed system establishes identity, relationships, provenance, version, rights, and authority strongly enough to define what later compression and reconstruction must preserve.
In benchmark execution, that principle becomes a hard procedural boundary:
Canonical Authority → Reference Contract → Authority Capture → Reference Gate → Governed State → Target Construction → Model Evaluation
No target construction before the Reference Gate passes.
No valid target means there is nothing legitimate for the downstream model, quality, or economic layers to measure.
Lock 1
Freeze authority sources, selectors, precedence, required values, normalization, equivalence, binding, rights, correction policy, censorship rules, and target construction before authority is audited.
Lock 2
Freeze the prediction, baseline, intervention, Quality Gate, metrics, units, cost boundary, analysis plan, failure conditions, sample design, and success thresholds before comparative results are observed.
For studies that require governed authority, the benchmark should move through the authority layer in a frozen sequence. Each stage answers a different question and should preserve its own result.
Gate A
Are all required authority surfaces and target-critical values available under the registered capture rules?
Gate B
Do the frozen selector and precedence rules identify the required authority without unresolved ambiguity?
Gate C
Are all preregistered leaves, relationships, rights, constraints, versions, and other required target components present?
Gate D
Do the authority surfaces agree within the exact tolerance, normalization, and equivalence rules frozen before capture?
Gate E
Can every preregistered cross-source relationship required for the target be bound under the frozen Reference Contract?
Gate Result
Only after all required authority gates pass may the benchmark construct its hidden target.
Study 008 · Reference-Gate Precedent
Study 008 successfully captured all required governed leaf values and did not encounter an availability or selector-ambiguity failure.
One required cross-source canonical relationship nevertheless failed the frozen binding rule. Because the preregistration prohibited post-hoc normalization, semantic substitution, silent correction, and trademark removal for that relationship, the study censored itself before target construction.
No model observation, cache probe, paid retrieval, x402 payment, or economic comparison followed.
Benchmark classification: Authority-layer censored result. It is not a negative model result and not a negative Compression Dividend result.
Identify the exact canonical or applied proposition being tested, its scope, governing authority, and current evidence state.
Register every authority surface permitted to define the benchmark target, including URLs, manifests, documents, schemas, registries, APIs, records, or controlled internal sources.
Lock selectors, precedence, required values, binding rules, normalization, equivalence, correction policy, rights conditions, censorship rules, and target-construction procedure.
Lock the prediction, baseline, intervention, Quality Gate, metrics, success threshold, cost boundary, sample design, analysis plan, exclusions, and downstream failure conditions.
Record the MRD version, protocol version, repository commit or release, authority versions, model, software, dependencies, configuration, fixtures, pricing references, and execution environment.
Capture the preregistered sources contemporaneously and preserve response status, content identity, selectors, extracted values, provenance, and any declared hashes or version records.
Test availability, selection, completeness, coherence, binding, rights, and all other preregistered authority requirements. If a required gate fails, stop according to the frozen censorship rule.
If the Reference Gate passes, construct the hidden target using only the preregistered transformation rules and preserve its complete provenance back to the governing authority package.
Build the baseline and intervention inputs according to the preregistered visibility rules without leaking the hidden target or granting one arm undeclared authority access.
Execute the registered comparison condition under the frozen system, workload, limits, tools, and measurement rules without post-result tuning.
Apply only the registered Razor-guided change while preserving the matched conditions and measuring all intervention-specific overhead.
Score both conditions against the same target, acceptance rubric, adjudication rules, missing-data treatment, and preregistered quality threshold.
Only after the applicable quality requirements pass should the benchmark calculate total cost, avoided recomputation, Net Reuse Benefit, break-even, Dynamic Reuse Frontier, Compression Dividend, or other preregistered efficiency measures.
Check for authority failure, target leakage, quality regression, unmatched conditions, stale state, missing telemetry, protocol deviations, execution failures, and invalid economic interpretation.
Publish or preserve the human-readable report, machine-readable result, authority status, target status, model-execution status, Quality Gate result, economic result where applicable, hashes, provenance, deviations, limitations, and replication state.
Result Layer 1
The benchmark cannot establish a valid governed target. Downstream model, retrieval, Quality Gate, and economic questions remain untested.
Result Layer 2
A valid target exists and the model or retrieval condition is evaluated, but one or more paths fail the preregistered Quality Gate or model-level decision rule.
Result Layer 3
Reference and quality requirements permit a valid efficiency comparison, including total cost, Net Reuse Benefit, break-even, Dynamic Reuse Frontier, or Compression Dividend where preregistered.
Reading Benchmark Outcomes
Example · Study 008
Reached authority capture and reference binding. Stopped before target construction. Therefore no model or economic conclusion is permitted.
Economic Benchmark
Reaches target construction, model or retrieval measurement, the Quality Gate, and total-cost comparison. Only this class of study can produce a direct Compression Dividend result.
Benchmark Integrity
If the frozen Reference Contract says the authority package fails, do not normalize, reinterpret, silently correct, substitute a source, remove marks, infer an alias, repair a relationship, or ask the evaluated model what the source “probably meant” unless that exact procedure was preregistered.
| Evaluation identity | Study ID, evaluator, date, governing authority, claim, protocol version, and benchmark version |
| Reference identity | Authority package, Reference Contract version, capture records, selectors, required values, normalization, equivalence, binding, rights, and censorship rules |
| Reference-Gate result | Availability, selection, completeness, coherence, binding, final governed-state status, failed gate where applicable, and target-construction authorization |
| Target identity | Target ID, construction procedure, provenance mapping, hash or version where applicable, and model-visibility boundary |
| Implementation identity | Repository, branch or immutable release, commit SHA, dependencies, runner, and benchmark configuration |
| System identity | Model, version, tools, memory, retrieval, pricing basis, hardware, runtime, and environment |
| Evaluation design | Baseline, intervention, tasks, controls, Quality Gate, metrics, units, success threshold, and economic boundary |
| Integrity controls | Randomization, seeds, exclusions, missing data, target leakage controls, source changes, hashes, deviations, and stop rules |
| Result classification | Authority-layer outcome, model / quality outcome, economic outcome, evidence limitations, and downstream stages not reached |
Complete Benchmark Spine
Canonical Claim → Reference Lock → Authority Capture → Reference Gate → Governed Target → Evaluation Lock → Baseline / Intervention → Quality Gate → Economic Measurement → Versioned Result → Evidence-State Review → Replication
Execution boundary: Not every benchmark will reach every stage. A valid preregistered stop at the authority layer is itself a legitimate result. A study should report exactly where evaluation stopped rather than forcing every experiment into a positive, negative, or inconclusive model-performance category.
13 · Reporting, Outcome Layers & Evidence States
A benchmark result is more than a score, a model output, or an economic delta. It is a versioned record of how far a preregistered evaluation progressed, which gates passed, where execution stopped, what was actually observed, and which downstream questions remain unanswered.
This distinction became especially important through Study 008. A benchmark may produce a valid empirical result before the model is ever evaluated if the preregistered Reference Gate identifies an authority-layer condition that prevents legitimate target construction.
Evidence-state assignment remains separate. A study result first records what happened under the frozen protocol. The broader claim-review process then determines whether that result supports, challenges, narrows, leaves unresolved, or otherwise changes the evidentiary status of a defined claim.
Result rule: Positive, null, negative, censored, failed, and inconclusive findings must all remain visible. A benchmark that stops correctly at a preregistered gate has produced a valid bounded result even when downstream model or economic measurements never occur.
Benchmark Interpretation Architecture
Every benchmark result should identify the deepest valid stage reached under the frozen protocol. Later-stage questions cannot be answered when an earlier required gate has not passed.
Authority Capture → Reference Gate → Governed Target → Model / Retrieval Execution → Quality Gate → Economic Comparison → Versioned Result → Evidence-State Review
Result Layer 1
The benchmark evaluates whether the registered authority package can establish the governed state required for target construction.
If this layer fails, model quality and economics remain untested.
Result Layer 2
A valid target exists and the baseline, model, retrieval, or intervention condition can be evaluated against the preregistered Quality Gate.
A Quality Gate failure invalidates downstream efficiency interpretation for that condition.
Result Layer 3
Reference and applicable quality requirements pass, allowing cost, avoided recomputation, Net Reuse Benefit, break-even, or Compression Dividend analysis.
Only this layer can directly answer the benchmark’s reuse-economic question.
Reference failure ≠ model failure
Model failure ≠ economic failure
Economic loss ≠ authority failure
Each result belongs to the deepest valid layer actually reached.
| Study | Deepest Layer Reached | Bounded Result | What It Does Not Establish |
|---|---|---|---|
| Study 003 | Economic Comparison | Quality-equivalent governed reuse was measured, but the tested live P3 price produced a negative incremental Compression Dividend relative to the accepted P2 path. | Does not establish that governed retrieval can never become economically favorable at other prices, workloads, reuse frequencies, or system conditions. |
| Study 008 | Authority Binding | One required cross-source canonical relationship failed the frozen authority-binding rule, producing AUTHORITY_BINDING_CENSORED_BEFORE_TARGET_CONSTRUCTION. | Does not establish model pass or failure, retrieval performance, Quality Gate outcome, x402 economics, or a Study 008 Compression Dividend. |
Published Authority-Layer Result
Study 008 tested whether the preregistered governed authority package could be captured, resolved, and bound strongly enough to construct the hidden benchmark target before any model or economic comparison began.
14 / 14
Source Captures
11 / 11
Selectors
63 / 63
Governed Values
8 / 8
Rights Values
9 / 10
Relationships
16 / 17
Cross-Source Checks
Availability Problems
0
Ambiguity Problems
0
Binding Problems
1
Frozen Study Outcome
AUTHORITY_BINDING_CENSORED_BEFORE_TARGET_CONSTRUCTION
The governing MRD identified the reference implementation as Naturepedia, while the canonical Naturepedia page identified the same underlying entity as Naturepedia™.
The frozen Study 008 rules treated the mark as meaningful and prohibited normalization, silent correction, semantic substitution, or trademark removal. The required relationship therefore could not be bound exactly under the registered contract.
Downstream stages not reached: hidden-target construction, model-visible input construction, P1 model observation, P2 cache probing, P3 paid retrieval, x402 payment, Quality Gate comparison, and Study 008 economic analysis.
Study 008 strengthened the benchmark architecture by separating three authority conditions that should not be collapsed:
Unavailable ≠ Ambiguous ≠ Available but Not Bound
This is an empirical governance result. It does not convert the Invariant Reference Principle into a universal law and does not independently validate the complete Grand Compression framework.
Accepted Economic Anchor
Study 003 remains the last accepted economic anchor in the current study sequence because it reached the downstream quality-equivalent cost comparison that Study 008 never reached.
P2 Quality-Equivalent Cost
$0.00111572
Historical Tested P3 Price
$0.25
Bounded interpretation: At the tested live P3 price and registered workload, the governed-retrieval path did not produce a positive incremental Compression Dividend relative to the accepted quality-equivalent P2 path. This is a negative economic result under those conditions—not a universal rejection of reusable-state economics.
Published Empirical Record
A preregistered, first-party bounded evaluation conducted under protocol RR-BRP-0.1.0 using Robbie’s Razor Benchmarks v0.2.0. The complete result package preserves the frozen protocol, execution environment, raw observations, evaluator reports, derived metrics, deviations, checksums, and machine-readable provenance.
3
OpenAI Models
2
Matched Conditions
72
Live Requests
18
Evaluation Reports
0
Recorded API Errors
| Model | API-C0 Accuracy | API-R1 Accuracy |
|---|---|---|
| gpt-5.6-luna | 0.75 | 1.00 |
| gpt-5.6-terra | 0.50 | 1.00 |
| gpt-5.6-sol | 1.00 | 1.00 |
These measurements apply only to the four evaluator-defined cases, three repetitions, documented prompts, matched conditions, models, evaluator, API configuration, and execution environment preserved in this release.
Immutable Provenance
Results tag: benchmark-results-v0.1.0
Results commit: e2bd8df6b286f3818c5db5f26a76e60141bcdef6
Results DOI: 10.5281/zenodo.21980302
Archive SHA-256: 93010c38e26292c30b337e23128727bb0cdf9640cbd2b355616310b6fcf84e4d
Benchmark software: v0.2.0 · DOI 10.5281/zenodo.21969841
Documented deviation: After all 72 live model requests completed, the synthetic memory-gate stage encountered a Python module-resolution error. That stage was recovered separately without repeating any live model request. The event remains recorded as RR-BRP-0.1.0-DEV-001. Synthetic memory-gate measurements remain separately classified as synthetic-proxy.
Evidence boundary: This record is a first-party bounded evaluation. It does not establish independent empirical validation, universal validity, model certification, production certification, cross-domain validation, scientific consensus, or guaranteed reductions in computational cost or resource use.
Explains the question, authority layer, method, deepest stage reached, results, limitations, uncertainty, deviations, interpretation, and evidence implications.
Preserves stable identifiers, authority status, versions, gate outcomes, metrics, units, values, hashes, provenance, stage status, and result classification.
Provides shareable authority captures, observations, calculations, audit outputs, or other evidence needed to verify the reported result, subject to privacy and licensing boundaries.
Records authority-package identity, Reference Contract, environment, configuration, dependency versions, repository commit, fixture hashes, capture time, and execution state.
| Field Group | Required Fields |
|---|---|
| Identity | Result ID, study ID, title, evaluator, dates, result version, status, and publication state |
| Authority | MRD version, claim ID, protocol version, authority package, Reference Contract, capture identity, precedence, normalization, equivalence, binding, and rights rules |
| Reference Gate | Availability, selection, completeness, coherence, binding, failed gate, governed-state status, and target-construction authorization |
| Target | Target ID, construction status, target version or hash, provenance, model-visible boundary, and reason target construction was blocked where applicable |
| Systems | Baseline identity, intervention identity, models, tools, memory, retrieval, pricing basis, hardware, configuration, and environment |
| Quality Gate | Whether model execution occurred, accepted, failed, corrected, excluded, or unscorable outcomes, acceptance threshold, adjudication, and quality decision |
| Measurements | Metric names, units, raw values, aggregation, deltas, uncertainty, sample size, and explicit notation for measurements not reached |
| Economics | Cost boundary, avoided recomputation, preservation, retrieval, verification, maintenance, repair, Net Reuse Benefit, break-even, Compression Dividend, or explicit not-tested status |
| Integrity | Source hashes, task hashes, exclusions, missing data, deviations, source changes, target leakage checks, validation checks, and stop conditions |
| Result Layer | Authority-layer outcome, model / quality outcome, economic outcome, deepest valid stage reached, and downstream stages not reached |
| Interpretation | Preregistered decision, limitations, supported scope, failure mechanism, findings, unresolved questions, and evidence-state implications |
| Replication | Replication status, independent evaluator, preserved and changed conditions, linked records, and re-evaluation requirements |
A study may stop because a preregistered upstream condition correctly prevented later execution. That outcome should be classified according to the gate that stopped the experiment.
The word failure should therefore identify the mechanism—authority failure, Quality Gate failure, economic threshold failure, protocol-integrity failure, or another registered condition—rather than imply that every stopped benchmark means the tested model performed poorly.
Benchmark result classes and formal evidence states are related but separate. The result records what happened during one registered evaluation. The evidence-state process evaluates that result alongside prior studies, replication, contradictory evidence, scope, limitations, and applicable governance.
| State | Benchmark Meaning |
|---|---|
| Proposed | The claim is defined but has not completed the evidence process required for support. |
| Testing | A registered authority, model, quality, economic, replication, or other evidence-collection process is actively examining the claim. |
| Provisionally Supported | Available bounded evidence supports the claim under a stated scope, but material replication or boundary limitations remain. |
| Supported | The claim has met the governing evidentiary requirements within its stated scope. |
| Challenged | Credible evidence materially conflicts with the claim or its present scope. |
| Inconclusive | Available evidence is insufficient, ambiguous, conflicting, too uncertain, or methodologically incomplete for a support or challenge decision. |
| Retired | The claim has been withdrawn, superseded, merged, replaced, or removed from the active framework. |
“Under the frozen Study 008 authority contract, one required cross-source relationship failed exact binding, so target construction and all downstream model and economic stages were censored.”
“Under the registered Study 003 workload and tested live P3 price, governed retrieval did not produce a positive incremental Compression Dividend relative to the accepted quality-equivalent P2 path.”
“Study 008 proved that Robbie’s Razor works and that governed retrieval is more efficient.”
Result Integrity Principle
Record what happened.
Record where the study stopped.
Record what was never tested.
Then decide what the evidence means.
Authorship separation: Robbie George remains the attributed creator of Robbie’s Razor, Grand Compression, Metrology for Meaning™, and the associated benchmark architecture. Evaluators, implementers, reviewers, and replication teams must be credited separately for their distinct empirical contributions.
Published Empirical Record
A preregistered, first-party bounded evaluation conducted under protocol RR-BRP-0.1.0 using Robbie’s Razor Benchmarks v0.2.0. The complete result package preserves the frozen protocol, execution environment, raw observations, evaluator reports, derived metrics, deviations, checksums, and machine-readable provenance.
3
OpenAI Models
2
Matched Conditions
72
Live Requests
18
Evaluation Reports
0
Recorded API Errors
| Model | API-C0 Accuracy | API-R1 Accuracy |
|---|---|---|
| gpt-5.6-luna | 0.75 | 1.00 |
| gpt-5.6-terra | 0.50 | 1.00 |
| gpt-5.6-sol | 1.00 | 1.00 |
These measurements apply only to the four evaluator-defined cases, three repetitions, documented prompts, matched conditions, models, evaluator, API configuration, and execution environment preserved in this release. API response storage was disabled with store: false.
Immutable Provenance
Results tag: benchmark-results-v0.1.0
Results commit: e2bd8df6b286f3818c5db5f26a76e60141bcdef6
Results DOI: 10.5281/zenodo.21980302
Archive SHA-256: 93010c38e26292c30b337e23128727bb0cdf9640cbd2b355616310b6fcf84e4d
Benchmark software: v0.2.0 · DOI 10.5281/zenodo.21969841
Documented deviation: After all 72 live model requests completed, the synthetic memory-gate stage encountered a Python module-resolution error. That stage was recovered separately without repeating any live model request. The event is recorded as RR-BRP-0.1.0-DEV-001. Synthetic memory-gate measurements remain separately classified as synthetic-proxy.
Evidence boundary: This record is a first-party bounded evaluation. It does not establish independent empirical validation, universal validity, model certification, production certification, cross-domain validation, scientific consensus, or guaranteed reductions in tokens, latency, computational cost, or resource use.
Explains the claim, method, results, limitations, uncertainty, deviations, interpretation, and evidence state.
Preserves stable identifiers, versions, metrics, units, values, hashes, provenance, and status for automated retrieval.
Provides the shareable observations needed to verify calculations, subject to privacy, security, and licensing limits.
Records the environment, configuration, dependency versions, repository commit, task hashes, and execution time.
| Field Group | Required Fields |
|---|---|
| Identity | Result ID, study ID, title, evaluator, dates, version, and status |
| Authority | MRD version, claim ID, protocol version, repository, commit SHA |
| Systems | Baseline identity, intervention identity, models, tools, memory, hardware, configuration |
| Measurements | Metric names, units, raw values, aggregation, deltas, uncertainty, sample size |
| Integrity | Task hashes, exclusions, missing data, deviations, judge method, validation checks |
| Interpretation | Threshold decision, limitations, supported scope, failure conditions, evidence state |
| Replication | Replication status, independent evaluator, deviations, and linked replication records |
| State | Benchmark Meaning |
|---|---|
| Proposed | The claim or benchmark is defined but has not begun a registered test. |
| Testing | Evaluation is active under a declared protocol. |
| Provisionally Supported | Initial results meet the declared threshold within scope but await stronger replication. |
| Supported | Evidence satisfies the declared support and replication requirements within the tested scope. |
| Challenged | Material results conflict with the prediction or a prior supported finding. |
| Inconclusive | The evaluation cannot resolve the claim because of uncertainty, insufficient power, missing data, or conflicting results. |
| Retired | The claim or benchmark is withdrawn, superseded, or no longer maintained. |
“Under the declared task set, model version, memory configuration, and quality threshold, the Razor-guided condition reduced measured re-derivation relative to the registered baseline.”
“The benchmark proves Robbie’s Razor makes every AI system more efficient.”
Authorship separation: Robbie George remains the attributed creator of Robbie’s Razor and the Grand Compression framework. Benchmark evaluators, implementers, reviewers, and replication teams must also be credited for their distinct contributions.
Continue to Failure Conditions →
14 · Falsifiability & Integrity Control
A benchmark can fail methodologically even when its code runs correctly. The evaluation becomes invalid when its controls, evidence boundary, scoring rules, or reporting practices cannot support the conclusion being asserted.
Falsifiability rule: Before testing begins, the evaluator must state which observations would support, challenge, narrow, or leave the claim unresolved.
The test is designed to confirm a preferred outcome rather than expose the claim to possible failure.
The comparison condition is selected, modified, or discarded after its results are observed.
The systems receive different tasks, tools, memory, time, prompts, budgets, or quality thresholds without declaring the difference.
Lower token use, latency, or tool activity is called efficient even though fewer tasks meet the acceptance threshold.
Visible tokens are treated as total computation while retries, retrieval, tools, validation, memory, and correction are excluded.
A higher memory-hit rate is treated as success without checking relevance, provenance, version, or stale reuse.
Failed runs, timeouts, rejected answers, missing telemetry, or unfavorable tasks are removed without a preregistered rule.
Example cases, sample outputs, notebooks, or demonstration runs are presented as formal empirical findings.
The existence of code, an auditor, a metric, a registry, or a Razor-guided configuration is presented as support for the tested claim.
Token savings are converted into energy, water, emissions, or environmental benefit without the required physical telemetry.
A finding from one model, task set, domain, configuration, or time period is generalized elsewhere.
Only favorable runs are retained, creating a distorted evidence record.
Multiple metrics are combined without declaring normalization, weighting, missing-data treatment, or sensitivity.
Historical formulas, superseded claims, or provisional concepts are presented as current GC-MRD-v2.0 authority.
| Observed Condition | Required Action |
|---|---|
| Material system, model, or protocol change during execution | Stop and restart under a new versioned record |
| Telemetry loss affects a required metric | Stop or mark inconclusive according to the preregistered rule |
| Small number of individual execution failures covered by the protocol | Continue and retain failures in the result record |
| Unexpected result with valid controls | Continue and preserve as a potentially challenging finding |
| Evidence of task contamination or scoring leakage | Invalidate affected runs and rebuild the task set |
Critical distinction: A benchmark-method failure means the evaluation cannot support a decision. A valid negative result means the tested claim was challenged within the declared scope. These are not the same outcome.
15 · Reproducibility & Change Control
Replication tests whether a result survives repeated execution, independent implementation, or transfer into a newly declared environment. Versioning ensures that evaluators can identify precisely which canon, protocol, implementation, fixtures, and system configuration were tested.
Immutable-record rule: Published benchmark releases and result records should be superseded by new versions, not silently overwritten. Historical records must preserve the governing canon, protocol, implementation, configuration, limitations, and evidence state that applied when the evaluation was conducted.
Current Implementation Anchor
The August 16, 2026 release provides the current stable implementation anchor for the benchmark suite. Its immutable tag, citation record, source archive, and GC-MRD-v2.0 alignment allow future evaluations to identify the exact published implementation they used.
| Governing MRD | GC-MRD-v2.0 |
| Benchmark Version | 0.2.0 |
| Release Tag | benchmarks-v0.2.0 |
| Release Date | August 16, 2026 |
| Citation Record | CITATION.cff |
Evidence boundary: Version v0.2.0 identifies a published implementation state. It does not itself constitute a benchmark result, independent replication, model certification, or upward change in evidence state.
Level 1
The same release or commit, fixtures, versions, environment, and calculation reproduce the reported result within the declared tolerance.
Level 2
A separate evaluator applies the registered protocol without relying on the original team’s undocumented decisions.
Level 3
The claim is tested in a different model, domain, workload, or infrastructure boundary under a new preregistered evaluation.
RC-22 boundary: Domain revalidation is a new test, not an automatic extension of the original result.
MRD Version + Claim Version + Protocol Version + Benchmark Release or Commit + Fixture Hash + System Configuration + Environment
| Artifact | Purpose |
|---|---|
| Commit SHA or Immutable Release Tag | Pins the implementation used in the evaluation |
| Citation Record | Preserves the published version, authorship, date, canonical page, repository, and release identity |
| Task and Data Hashes | Confirms fixture identity and detects silent alteration |
| Environment Manifest | Records software, dependencies, hardware, region, and runtime |
| Change Log or Release Notes | Explains what changed and whether prior results remain comparable |
| Supersession Link | Connects historical and current artifacts without erasing provenance |
| Replication Record | Documents whether another evaluator reproduced, narrowed, or challenged the result |
Evidence-state control: A new release does not automatically strengthen a claim. New replication may strengthen, narrow, challenge, or leave a claim inconclusive. Evidence states must follow the complete evaluation record and must not move upward merely because a newer version was published.
16 · Authority, Release & Retrieval Map
This benchmark page connects the governing canon to evaluation protocols, versioned implementations, immutable releases, bounded result artifacts, and machine-readable discovery without allowing one layer to replace another.
Current authority: GC-MRD-v2.0 governs present definitions, claims, boundaries, and evidence discipline. Earlier MRD versions remain part of the historical record but are superseded wherever they conflict with v2.0.
Current Implementation Release
Published August 16, 2026, under the immutable tag benchmarks-v0.2.0. This release is the current stable implementation reference for citation, inspection, reproduction, and version comparison.
Boundary: The release establishes implementation identity and provenance. It does not independently establish empirical support, replication, certification, or a higher evidence state.
Governing Canon
Canonical Law
Evaluation Protocols
Applied Evaluation
Versioned Implementation
Governing Canon → Evaluation Protocol → Immutable Implementation Release → Bounded Result → Evidence State → Machine Retrieval
Each layer serves a distinct function. GC-MRD-v2.0 governs the canon, the protocol defines the test, the immutable release identifies the implementation, the result record preserves what occurred, the evidence state bounds the conclusion, and the machine layer supports structured discovery and delivery.
17 · Questions & Boundaries
These answers clarify what the benchmark system measures, which version is currently published, how results should be interpreted, and where implementation ends and empirical evidence begins.
Robbie’s Razor Benchmarks are controlled evaluation structures for testing bounded claims about compression, expression, memory, recursion, preserved reusable structure, and total computational cost. They compare declared systems under matched conditions and shared quality thresholds.
No. Individual benchmarks can support, challenge, narrow, or leave unresolved specific claims within a declared test boundary. The existence of benchmark code or a favorable result does not universally validate Robbie’s Razor or the complete Grand Compression Cosmology.
The current governing authority is the Grand Compression Master Reference Document, GC-MRD-v2.0. Earlier MRD versions remain part of the historical record but are superseded wherever they conflict with v2.0.
The current public release is Robbie’s Razor Benchmarks v0.2.0 — MRD v2.0 Alignment, published August 16, 2026, under the immutable Git tag benchmarks-v0.2.0. It provides a stable, citable implementation record for inspection, reproduction, and comparison with later versions.
Release v0.2.0 provides the current public reference implementation, including benchmark harnesses, baselines, structured cases, retrieval-versus-recomputation fixtures, metric definitions, diagnostics, governance materials, tests, documentation, and machine-readable supporting assets. Individual assets retain their own classifications and evidence boundaries.
No. Immutability preserves the identity and provenance of a published implementation. It does not prove that a benchmark claim is correct, independently replicated, transferable to another domain, or entitled to a higher evidence state.
TPCA means Tokens Per Correct Answer. It divides measured tokens by correct or accepted answers. The result is meaningful only when the tokenizer, token categories, task set, acceptance rule, and treatment of failed runs are declared.
No. Lower token use may reflect useful compression, but it may also reflect incomplete reasoning, missing verification, or lower-quality output. Token efficiency must be evaluated after both conditions are scored against the same acceptance threshold.
Valid reuse occurs when preserved structure remains correctly identified, appropriately scoped, current, traceable to its provenance, compatible with active constraints, and useful in reducing later work. Reuse of stale or incorrect memory is a failure, not an efficiency gain.
No. Benchmarks can evaluate accepted outcomes, tokens, retries, tool calls, retrieval, state transitions, memory behavior, latency, resource use, and correction without requiring private hidden reasoning content.
No. Energy claims require workload-specific information about hardware, runtime, utilization, memory, networking, facility overhead, and attribution. Token savings may motivate physical measurement, but they do not establish energy, water, emissions, or environmental savings by themselves.
No. Under RC-21, implementation and validation are separate. Code, prompts, auditors, registries, memory systems, and Razor-guided configurations must still demonstrate measurable effects against a preregistered baseline.
GC-MRD-v2.0 uses seven evidence states: Proposed, Testing, Provisionally Supported, Supported, Challenged, Inconclusive, and Retired. Every state remains bounded to the identified claim, system, task set, configuration, and evaluation conditions.
A team can inspect the reference implementation, select a bounded claim, preregister a matched baseline, pin the release or repository commit, validate the task fixtures, record the environment, run repeated evaluations, package the results, and publish a replication record with appropriate limitations.
Citations should identify Robbie George as the creator, name the benchmark suite, specify version 0.2.0, include the release date, and link to the immutable GitHub release or repository citation record. The repository’s CITATION.cff file provides the machine-readable citation metadata.
No. The repository is not a vendor leaderboard or hardware-equivalence claim. Company, model, or infrastructure comparisons require separate protocols with matched workloads, equal quality standards, direct measurement, and clearly bounded conclusions.
Evaluation hub, release record, and versioned reference-implementation guide. This page does not assign universal validation, certify individual systems, or guarantee computational or environmental savings.
18 · Authorship & Stewardship
Robbie George · Nature photographer, author, and framework originator
Robbie George is a National Geographic–published nature photographer and the creator of Robbie’s Razor and the Grand Compression Cosmology. His photographic work has also been displayed at the Smithsonian National Museum of Natural History.
His framework developed from long-term observation of natural systems: how living structures retain useful information, reuse established relationships, adapt under constraint, and regenerate complexity without reconstructing every dependency from the beginning.
Robbie’s Razor expresses that architecture through the sequence compression → expression → memory → recursion. The benchmark system extends the sequence into a falsifiable evaluation layer with declared baselines, measurable outcomes, failure conditions, versioned implementation, and evidence-state governance.
George maintains the public framework, benchmark repository, machine-readable knowledge architecture, and Naturepedia reference implementation. These assets establish authorship and implementation provenance; they do not replace controlled testing or independent replication.
“When competing explanations exist, prefer the model that follows compression → expression → memory → recursion.”
— Robbie’s Razor, RC-01
Robbie George’s authorship establishes the provenance of Robbie’s Razor and the Grand Compression framework. Empirical support must still come from preregistered evaluation, transparent result records, appropriate evidence states, falsification, and scope-specific replication.
Page classification: Benchmark evaluation hub · Reference-implementation guide · GC-MRD-v2.0 governed · No universal validation or certification implied.
© Robbie George Photography • All Rights Reserved
GC-MRD-v2.0 governs Grand Compression definitions and boundaries. The Canonical Claims Register governs formal claim status.
Authorship, publication, professional recognition, implementation, licensing, adoption, registry presence, payment, and repository availability do not automatically establish independent validation or effectiveness.
Public specifications, schemas, benchmark assets, examples, and result-record resources are preserved in the Robbie’s Razor GitHub repository. GitHub serves as a public reproducibility and versioned reference layer; it is not the governing canonical authority.
Agents and machine clients can begin with the Naturepedia Agent Skill, Naturepedia v2 Index, llms.txt, AI Root, or Canonical Publication Manifest.
Licensing & Policies: Commercial Data License · Framework Licensing · Terms & Conditions
This is only visible to you because you are logged in and are authorized to manage this website. This message is not visible to other website visitors.
Click on any Image to continue
This means you can use the camera on your phone or tablet and superimpose any piece of nature art onto a wall inside of your home or business.
To use this feature, Just look for the "Live Preview AR" button when viewing any piece of nature art on this website!
Join the collector list for your first-order discount, new wildlife releases, and occasional field notes.
