Feature selection for machine learning

The hardest part of machine learning isn’t making a prediction. It’s knowing which evidence to trust.

Machine learning makes predictions: how much inventory will sell, which transaction may be fraudulent, which machine may fail. Better predictions can mean less waste, fewer losses, and better decisions.

People and AI agents can propose a thousand possible improvements. ML Referee evaluates up to 1,000 candidate features—the clues a model uses—in one run. Its independent statistical assessment helps you choose inputs, investigate a model’s decline, and see where more data could be worth the investment.

Request beta access

01 / Where Referee enters

Ideas come first. Evidence guides which ones to keep.

Referee fits at the evaluation stage: your team has a starting model and ideas for improving it. Before building around those ideas, test what they add.

The result supports a practical decision: what to build, what to repair, and what to investigate next.

01 / PredictBuild a starting model
Learn from historical examples to predict a measurable outcome.
02 / ProposeGenerate possible improvements
People or agents suggest new clues—features—that might improve the prediction.
03 / ML RefereeEvaluate the evidence
Test the full candidate family against the same baseline and evidence standard.
04 / DecideBuild, investigate, or stop
Advance supported ideas, focus the next experiment, and deprioritize weak leads.

What would I actually send?

Start with something that looks like a spreadsheet: historical examples in rows, the answer you wanted to predict in one column, and possible clues—features—in the others. You or your developer identify the starting benchmark and the proposed improvements.

Your developer or coding agent connects to Referee, prepares those inputs in its required format, checks them, and reviews a credit quote before submitting. Referee returns a report to guide the next model change or research investment. A solo developer can do this; you do not need a separate ML team.

Before evaluating: you need historical outcomes, a baseline, candidate improvements, eligible non-personal data, and approved beta access with credits.

Want to see the steps before bringing your own data? Start with a practice example, on your own or with Codex →

02 / What Referee does

Three questions. One better-informed research plan.

A / Feature Referee

Feature selection for machine learning.

What should enter the model? Compare candidate inputs with the same starting model, or baseline. See which earned statistical support before investing in the next build.

Bring: An outcome, a baseline, historical data, and candidate features
Receive: Evidence for selecting inputs and validating the next model

B / Model Repair

Model repair when performance degrades.

What changed when it stopped working? Compare stable and degraded periods using signals you supply. Investigate changed relationships and bounded repairs worth testing on new data.

Bring: Evidence from stable and degraded periods, with possible explanations
Receive: Repair evidence and a plan to confirm it on fresh data

C / Research priorities

Know where more data could help.

Which uncertainty is worth resolving? An unresolved feature is not automatically a dead end. Use its evidence and limitations to focus data collection, refine a proposal, or plan the next experiment.

Bring: The same evaluation, plus the question your next investment must answer
Receive: A basis for deciding where further research could add value

A thousand ideas. Which deserve the next experiment?

You remove one input—a clue the model uses. The predictions get slightly worse. You restore it and remove another. A small improvement. By Friday, you have spent a great deal of computing time and still cannot answer the question that started the exercise: which changes should you believe?

A small difference might matter. It might be luck. Test enough combinations and chance gets plenty of opportunities to look clever.

Clues also work together. One that looks useless alone may become valuable alongside another. Removing them one at a time tells you only part of that story.

Referee began with this ambiguity. Did a proposed combination genuinely improve the prediction—or merely produce a more flattering score?

Referee evaluates up to 1,000 proposed inputs in one run. It considers how they work together and how many possibilities were tested, assessing how much confidence the evidence deserves—not just whether a score moved.

The result helps you decide which ideas to keep, which to set aside, and where more data could settle a question worth answering.

03 / The hidden trap

When ideas multiply, the luckiest result can look like the best one.

One baseline for the run

Every submitted candidate faces the same declared baseline, observations, and validation design. No candidate gets an easier test.

Control the multiple-testing problem

Testing hundreds of features means some can look useful by chance. Referee evaluates the complete declared feature family together and applies family-wide error controls, reducing false discoveries under the stated assumptions.

Keep strength visible

Each candidate keeps its measured effect, statistical strength, uncertainty, limitations, and next confirmation step.

Agents propose. Referee evaluates. Humans set the boundaries.

The evidence bar is chosen before the results arrive. Referee then keeps supported, provisional, and unresolved findings distinct, showing both what earned belief now and where further research could resolve a useful uncertainty.

04 / One fair review

Set the question. Run the evaluation. Inspect the evidence.

Define the run

Agree on the outcome, starting model, candidate inputs, and required strength of evidence. Specify which information was available when each prediction was made.

Preflight the family

Check that required columns are present, required values are valid, and labels are used only after they become available—before the run consumes credits.

Evaluate hundreds at once

Compare up to 1,000 candidates with the same baseline and evaluation design, without coordinating a separate ablation job for each feature.

Review the qualified set

See which features cleared the declared bar, which remain provisional, and which were unresolved. Prospective confirmation remains part of the work.

05 / Evidence in practice

A better safety model—and a clearer next experiment.

Referee supported an augmented model’s predictive improvement over the default baseline. Separately, the demonstration measured 64 additional unsafe prompts detected and 11 additional false alerts at independently calibrated, frozen thresholds. An added feature’s incremental value remained unresolved: a question for further research, not an automatic rejection.

Read the case study and limitations →

0.03959held-out log-loss gain

1,915held-out prompts

+64unsafe prompts detected

+11additional false alerts

06 / The decision

Know what to build. Know what to investigate next.

The report connects evidence to the next investment. Your team can advance supported inputs, test a repair, or gather data to resolve an important uncertainty. It supports your decision; final model training and validation on new data come next.

Candidate evidenceStrong, provisional, unresolved, or no incremental evidence for the submitted design.
Selection contextThe objective, baseline, evidence bar, and complete submitted family behind the result.
ScopeThe exact prediction or ranking channel the result supports.
Next testWhat to verify prospectively before drawing a broader conclusion.

07 / Where it fits

Feature selection for the predictions your organization relies on.

A strong fit when

You need feature selection for a machine-learning model.

You have a measurable outcome, a starting model, historical examples with known answers, and candidate inputs to test. This includes forecasting demand, detecting fraud, ranking search results, and predicting customer churn.

Not ready yet when

You do not yet have a testable ML problem.

If those pieces are missing, begin by defining the outcome, assembling suitable data, and establishing a baseline. The practical introduction explains how they fit together.

08 / How to use it

Agents can move the work forward. Humans keep authority.

Called by agents

Research and model-building agents can discover Referee, check readiness, preflight evidence, obtain a quote, submit an evaluation, and return structured results for review. The proposing model can change; the evaluation contract remains explicit.

MCP connection and agent quickstart →

Available to developers

Teams that prefer direct integration can use the same authenticated service through its HTTP API or thin clients. The public website is informational and does not accept datasets.

Budgeted autonomy

Today, an agent can quote and submit calls within an authorized beta-credit pool. Planned paid access will let a human approve an allowance and boundaries so the agent can purchase credits and proceed without approval before every call.

Give the agent a budget—not a blank check.

The human-defined spending policy remains the authority. Planned x402 support will provide the programmatic payment rail; calls outside the approved allowance, per-run ceiling, or expiration will stop for human review.

Give the next research decision a firmer foundation.

Bring one prediction problem to the free, invite-only beta. Tell us what you want to improve and how evidence would change your next step. People and connected agents can request access; approval is manual.

Request beta access