Identify which candidate inputs add evidence beyond the existing baseline.
Statistical feature evaluation for ML
Feature selection for machine learning. One run, up to 1,000 candidates.
Your agents can fill a feature backlog overnight. The harder question is what deserves the next training run. Referee evaluates candidates against your baseline under one statistical design, helping you select inputs, assess bounded repairs, and focus further data collection.
01 / Where it fits
An independent evaluation stage, not another model-building framework.
For ML practitioners, this is statistical feature selection. Use Referee after candidate features or repairs have been proposed and before spending another development cycle building around them.
For model performance optimization, narrow the inputs worth developing. For model repair, investigate a decline. For research prioritization, use unresolved results to decide where better data could help. Final training, prospective validation, deployment review, and monitoring remain with your team.
The workflow: generate candidates → evaluate with Referee → construct or repair → validate on fresh data. Referee is separate from the proposing system; its assessment stays tied to your declared problem.
02 / Why use Referee
Replace serial feature hunting with a declared, family-wide analysis.
Focus the next training run on inputs with evidence, rather than tuning around an unexamined feature set.
Account for the fact that many candidates were searched, reducing false discoveries under the declared assumptions.
Evaluate supplied signals from stable and degraded periods to focus bounded repair work.
03 / The evaluation contract
Make the prediction question and evidence standard explicit.
Outcome and baseline
Declare the target, row semantics, current benchmark, and practical direction of improvement.
Choose baseline-feature refitting or supplied out-of-fold predictions. The walkthrough below explains which inputs to prepare.
Historical validation data
Provide aligned labels, predictions, features, timing, and groups needed for a defensible design.
Referee checks declared timing and can flag some forms of lookahead bias and data leakage. Features must still use only information available at prediction time; these checks do not rule out every form of leakage.
Full candidate family
Submit up to 1,000 proposed inputs together so the analysis reflects the breadth of the search.
Evidence bar and limits
Set the declared standard and document what the result can and cannot establish.
04 / Your first evaluation
A data table. A connection. A report you can act on.
You can run this yourself, through an authorized coding agent, or as part of a team. Keep your existing training code. Referee is a remote evaluation service; the website is the guide, not a spreadsheet editor or model-training workspace.
1. Turn your prediction problem into explicit column roles.
For a demand forecast, one row could represent a product-week. Separate the observed outcome, the baseline inputs, and the proposed additions. These synthetic rows illustrate the distinction:
| Product-week | Actual sales Outcome | Recent sales average Baseline input | Promotion planned Candidate input | Holiday week Candidate input |
|---|---|---|---|---|
| A / week 1 | 120 | 105 | Yes | No |
| A / week 2 | 98 | 108 | No | No |
| B / week 1 | 210 | 180 | Yes | Yes |
The outcome is known when evaluating past examples, but must not be an input to the prediction. Features must be knowable at prediction time. Follow the live schema for identifiers, timestamps, label-availability times, grouping, types, and validation requirements; do not upload this illustration as a ready-made template.
Choose your comparison: in default refit mode, Referee fits an evaluation model from your baseline features and compares candidate additions—it does not run your production model. In supplied-prediction mode, it evaluates aligned out-of-fold baseline and candidate predictions you provide. Preserve the complete proposed family and the same observations. You need enough eligible history for the selected validation design, not just a few example rows.
2. Connect your client and discover the current schema.
Account authorization uses OAuth through Auth0. Sign in to Referee in your browser and authorize your client to access your account. The client supplies an access token with protected MCP calls; Referee checks the token, account access, and credits. Separate registrations serve ChatGPT, hosted Claude, Codex, and Copilot in VS Code. You do not need to configure Auth0 yourself. See the authorization guide and client test status.
Configure the remote MCP endpoint https://api.mlreferee.com/mcp in a supported client using the approved OAuth connection flow. Client compatibility matters; entering a URL alone does not guarantee a working connection. No passwords or tokens should be pasted into agent instructions.
Call describe_referee, list_verticals, and get_referee_readiness. Choose the appropriate profile and request get_input_schema; for model repair, request get_repair_schema. Use MCP tool discovery for current call arguments. This is a tool sequence, not a list of REST paths to construct.
Sign in, request beta access if needed with request_beta_access, and wait for human approval. Use get_beta_request_status to check a request and get_credits to check the account balance. Approval and credits are separate from connectivity; signup does not grant credits automatically.
3. Prepare locally, then upload and obtain a quote.
Export eligible data as CSV or JSON following the returned schema. Confirm permission before sending it. The beta permits synthetic, non-personal, or genuinely anonymized data—not regulated data or identifiable or pseudonymized person-level records.
- 1–32 candidate features:
upload_private_dataset→preflight_dataset→quote_audit. Resolve input errors before quoting. - 33–1,000 candidate features: use
upload_campaign_shardfor aligned shards of at most 32 candidates, thenquote_campaignfor the complete family. The quote checks the combined inputs. Do not treat separate small evaluations as one family-wide result. - Model repair:
upload_repair_matrix→preflight_repair→quote_repair, using one complete matrix and the required stable/degraded-period information.
Keep the returned upload identifiers. The quote returns the credit requirement and execution estimate. Hosted preflight happens after upload. Local preparation and validation can catch errors before you send the file; hosted preflight is an input check, not a local-only privacy gate or proof that the data or proposed model is valid in every respect. Eligibility checks do not automatically detect every prohibited personal-data submission.
Where your data goes: Referee runs numerical evaluations without sending evaluation datasets to an LLM, and does not reuse them to train shared AI models. Statistical fitting for the requested comparison is part of the evaluation. Your own AI client may still process uploads, tool arguments, and results under its provider’s policies. AI-assisted support review of evaluation data requires separate, explicit permission.
Inputs and reports use encrypted, short-lived service storage—not a zero-storage or instant-purge architecture. Account, credit, verification, and operational records follow separate rules; backup copies may outlast live deletion. Read the LLM-use policy and retention and deletion details before uploading.
4. Authorize the job and retrieve its evidence.
Review the quote against your available credits and approved budget. Call submit_audit with its quote_id and an idempotency key (a unique request key to prevent duplicate jobs); reuse that key only when retrying the same submission. Keep the returned audit identifier, check get_audit_status at bounded intervals, and retrieve the completed output with get_audit_result.
You receive structured results and report text describing candidate evidence, uncertainty, and limits. Save them promptly: reports expire with their associated inputs, up to 24 hours from upload, and some uploads default to two hours. Use the result to plan the next training run, repair test, or data collection—not as automatic permission to deploy.
05 / The deliverable
A report you can inspect, not just a winning score.
Which inputs earned consideration?
Each candidate receives a bounded result for the declared outcome, baseline, data, and validation design.
Was the complete search considered?
Findings are interpreted across the submitted family instead of as isolated discoveries.
What does the result establish?
The report separates the predictive assessment from operational or deployment decisions.
Where could more work add value?
Unresolved results focus the next data collection, feature refinement, or prospective experiment.
Downstream compatibility: Use the findings to narrow inputs for XGBoost, linear models, neural networks, or another appropriate framework. The assessment is not a guarantee that a feature will improve every architecture; confirm the final model in its intended setup.
06 / Evidence in practice
A supported improvement. An unresolved incremental claim.
Research case study
Build better safety models. Know what the evidence supports—and where to work next.
Referee supported the augmented model’s predictive improvement over the default baseline. Separately, the demonstration measured 64 additional unsafe prompts detected and 11 additional false alerts at independently calibrated, frozen thresholds. The added representation did not establish incremental benefit over the matched control—a focused question for further validation.
Read the AI-safety case study →07 / Complete the cycle
Give the next research allocation a reason.
Turn a feature backlog into disciplined evidence.
Bring the outcome, baseline, validation data, and complete candidate family. Referee returns a bounded feature-quality opinion and a focused next-research agenda.
Request beta access