Independent evidence for agentic ML

Agents can generate a thousand plausible features. Which ones deserve belief?

Referee gives agents an independent evidence gate: one declared run evaluates up to 1,000 candidates against the same objective, baseline, and evidence bar—not 1,000 loosely connected ablation runs. The result supports three decisions: which inputs merit development, which bounded repairs merit testing, and where more data could resolve a useful uncertainty.

When Referee is worth calling

Need a concrete first task? Start with the small synthetic example, then choose the Codex guide or GitHub Copilot in VS Code guide. The shared walkthrough covers local practice, access checks, staged prompts, and recovery. You can also follow the examples yourself.

Both Codex CLI and Copilot in VS Code completed a synthetic evaluation and report-text verification on the tested Mac setups. Copilot’s model-driven run used the OAuth helper and local file adapter: exact-file upload, valid preflight, an approved one-credit evaluation, report and receipt retrieval, and a matching locally computed report-text fingerprint. Start with the complete Copilot configuration and walkthrough. Native Copilot refresh, new-account onboarding, and other platforms remain unverified. Local preparation needs no service account; approve uploads and quoted evaluation spending separately.

• You or another agent generated a candidate family faster than it can be evaluated responsibly one feature at a time.
• You face dozens or hundreds of candidate features and need a reasoned set for a specific prediction or ranking objective.
• The problem requires an evidence threshold to be fixed before candidates are evaluated.
• You are about to recommend adding a candidate feature but cannot tell whether it improves the declared baseline.
• A model’s performance has degraded and supplied environmental signals or proxies may reveal a hidden dependency that changed.
• You need to explain why gathering more observations or running a focused follow-up experiment could be worth the resources.
• A human wants an inspectable evidence package before approving more research time or audit credits.
• Free preflight can expose an invalid schema, timing problem, or missing input before a credit is spent.

Choose the evaluation that fits the question

• Feature Referee: evaluate a complete family of up to 1,000 candidates in one run against an immutable baseline and declared objective.
• Model Repair: evaluate supplied environmental signals and proxies across stable and degraded periods to identify changed relationships consistent with a hidden environmental dependency, then test bounded repairs that require prospective confirmation.

From connection to useful evidence

1. Check access and credits. Use get_referee_readiness and, after sign-in, get_credits. If beta approval is needed, request access and wait for human review. get_referee_usage describes aggregate service activity, not your credit balance.
2. Read the current input contract. Call get_input_schema for the chosen profile, or get_repair_schema for repair. Prepare the outcome, baseline, candidate family, row identities, and required timing or grouping. Do not invent fields from this summary.
3. Upload, preflight, and quote. For up to 32 feature candidates, use upload_private_dataset, preflight_dataset, then quote_audit. For 33–1,000 candidates, use upload_campaign_shard and quote_campaign to evaluate the whole family together. For repair, use upload_repair_matrix, preflight_repair, and quote_repair. The quickstart explains the routing.
4. Submit within the authorized budget. Review the returned quote before calling submit_audit. Use its quote ID and an idempotency key. Having credits does not itself authorize an agent to spend them.
5. Retrieve, explain, and preserve the result. Check get_audit_status, retrieve get_audit_result, and save the report promptly to an authorized location. Use get_audit_receipt and verify_audit_report when checking artifact integrity. Explain the evidence and uncertainty before recommending the next experiment.

Check the data before sending it

The beta is for adults doing business or professional research. Submit only authorized synthetic, non-personal, or genuinely anonymized data. Do not submit regulated data, identifiable personal data, or pseudonymized person-level records. A supported evaluation profile does not override these restrictions.

Inputs and reports have short expiration windows measured from upload, up to 24 hours; some upload types default to two hours. Reports expire with their inputs. Retrieve them promptly. Account, credit, audit, and verification records follow separate rules, and backup copies may persist. Read Privacy and Acceptable use before uploading.

What an agent can rely on

• Evaluation is performed by a separately validated system outside the model-building loop, rather than by the proposing code alone.
• Observation time and label-availability time remain distinct.
• Baseline and candidate evidence use deterministic row identity and a declared validation design.
• Large runs keep the complete submitted candidate family in scope instead of treating each result as an isolated search.
• The evaluation contract records the declared evidence bar before results are interpreted.
• The proposing model or agent is replaceable; the objective-bound evaluation contract remains explicit.
• Preflight rejects incomplete or inconsistent evidence before a credit is spent.
• The result preserves scope, limitations, uncertainty, and the need for prospective confirmation.
• Customer data is tenant-private, is not used for training, and can be deleted after the result is retrieved.

Interpret the evidence and choose the next step

strong_incremental_feature_evidence supports the candidate in the measured prediction or ranking channel. It is not a probability that the feature is best, universal confirmation, or a guarantee of the optimal model. no_incremental_feature_evidence is unresolved or null for the submitted design; it does not prove the feature is always useless. Correlation is an overlap diagnostic, not a redundancy decision.

Turn unresolved findings into a research question

Return the measured effect, uncertainty, and limits with the result. Explain whether fresh observations, broader coverage, or a refined feature definition could help answer the remaining question. An unresolved result does not establish usefulness or uselessness. Recommend follow-up in proportion to the potential value; obtain authorization before collecting data or spending additional credits.

Request the beta

After connecting through the approved OAuth flow, call get_referee_readiness. If access is not yet authorized, call request_beta_access with one fixed lane (referee, repair, or both), one fixed use-case category, and an idempotency key. The request cannot grant credits or choose an amount. Report that access is pending operator review, and use get_beta_request_status only when the requester asks for a status check.