Can AI Science Agents Design Their Own Experiments? What a 67% Success Rate Really Means

\n

If AI Chooses the Next Experiment, What Will Scientists Do?

The era of AI summarizing papers and proposing research ideas has already begun. But now a more important question is emerging:

What if AI directly decided “what to experiment on next” within a limited budget and number of experiments?

The Autonomous Science Agent, drawing attention in 2026, is not simply a system that explains scientific knowledge in words. It chooses which experiments to test first from a range of candidates, reads the results, and revises its next experimental plan based on what went wrong. In other words, AI is beginning to take part in one of the most costly decision-making processes in a scientist’s day-to-day work.

What AI Is Changing About Science: The Next Action, Not the Right Answer

Traditional AI tools for science have helped researchers by answering their questions—searching papers, analyzing data, predicting molecular performance, and writing code. Autonomous science agents, by contrast, do not stop at answering questions.

They work through a process like this:

Set a research goal
→ Generate candidate experiments
→ Prioritize them within a limited budget
→ Run an experiment or call a measurement service
→ Analyze the results
→ Update hypotheses and predictive models
→ Choose the next experiment

Consider the search for a new battery material. There may be thousands or even tens of thousands of possible compositions, but it is impossible to test them all. In this situation, rather than repeatedly choosing only candidates that already look promising, AI must consider both their potential to improve performance and the value of the information they could provide.

The key skill, then, is not predicting, “This material looks promising.” It is deciding, “With a limited budget, this is the candidate we should test first.”

Discovery and Learning Are Not the Same

New ways of evaluating autonomous science agents make this distinction clear. One example is Autonomous Science Bench, which assesses whether agents have both of the following abilities:

• Discovery: Did the agent find an outstanding material or design among the candidates it actually tested?
• Learning: Could it accurately predict the performance rankings of candidates it had not yet tested?

The two may sound similar, but they are not the same ability at all.

It may be possible to stumble upon one good candidate by chance. But understanding patterns across the entire candidate space from only a small number of experimental results—and accurately predicting candidates that have not yet been tested—is much harder. What research needs is not a one-time stroke of luck, but the ability to learn in ways that continue to pay off in the next experiment.

In a published evaluation, eight agent–model combinations completed 144 trials in total, but only 31 trials passed both the discovery and learning criteria. The top-performing combination succeeded in 12 out of 18 trials, while on the most difficult task, agents succeeded just 3 times out of 24.

These figures show that AI has demonstrated its potential. They also show that it is still a long way from becoming a fully autonomous scientist.

So, Will Scientists Disappear?

The short answer is no—not in the foreseeable future. Their roles are more likely to change.

If AI narrows down candidates and recommends priorities for repetitive experiments, scientists can focus on higher-level judgment:

• Deciding whether a research goal is scientifically and socially meaningful
• Setting safety and ethical standards
• Verifying that AI hypotheses are grounded in actual causal relationships
• Finding new research questions in unexpected results
• Replicating experimental results and reviewing them independently

AI is particularly good at producing plausible explanations, but there is no guarantee that those explanations reveal the true cause. A researcher’s role may therefore expand from simply conducting experiments to overseeing AI’s exploration and taking responsibility for verifying the meaning of its discoveries.

The Real Test for AI Science Is the Lab

The value of autonomous science agents cannot be judged by their language skills or the quality of their reports alone. What matters is whether they can reduce the cost of real experiments, find better candidates faster, and learn from failure to shape their next strategy.

In the future, scientific competitiveness will depend on more than one larger model. It will also require systems that safely connect AI with laboratory equipment, report uncertainty honestly, track experimental histories for auditing, and support collaboration with human researchers.

When AI chooses the next experiment, scientists do not become less important. On the contrary, their role in deciding what is worth discovering and turning those discoveries into trustworthy knowledge becomes even more important.

Why AI Must Distinguish Between “Discovery” and “Prediction”

If AI finds one promising candidate, can we say it understands the scientific principles of that field?

Not necessarily. It may have simply stumbled upon a high-performing material or design among countless candidates. Conversely, even if it can predict the performance ranking of all candidates fairly accurately, it may still fail to identify the most promising one within a limited experimental budget.

To make this distinction clear, Autonomous Science Bench does not collapse an AI agent’s performance into a single score. Instead, it evaluates Discovery and Learning & Prediction as separate gates.

Did it find a genuinely good candidate? The Discovery gate

The first gate is straightforward: Does any candidate the AI selected and tested perform well enough?

It answers the question:

Did the AI discover a truly valuable candidate within a limited number of experiments?

Imagine searching for battery materials. With thousands of candidate materials, it is impossible to test every combination. The AI must therefore decide, within a limited budget, “Which materials should we make and measure first?”

Passing the Discovery gate means that among the experiments the AI chose to run, it found a promising material that actually met the performance threshold. That is an important result in real-world research. The sooner researchers find a good candidate, the more they can reduce development time and experimental costs.

But this alone is not enough. It is difficult to tell whether a successful choice came from scientific understanding or simply from luck during the search.

Did it understand even the candidates it had not seen? The Learning gate

The second gate is more demanding. The AI must predict the performance of candidates it has not tested—and in particular, it is evaluated on how accurately it ranks their performance relative to one another.

This gate asks:

Can the AI generalize from observed results to areas it has not yet explored?

A good scientific model does more than explain data that has already been measured. It should also make reasonably accurate predictions under new conditions, for new materials, and for new designs.

Suppose the AI tests only 10 out of 100 catalyst candidates. If it merely picks the best-performing candidate among those 10, it may succeed at discovery. But if it can also reasonably predict the performance ranking of the remaining 90, that suggests it may have learned the structure of the candidate space.

That is why the Learning gate matters. When designing the next experiment, researchers need more than a summary of past results. They need a reliable judgment about which untested candidates should be prioritized for validation.

An AI that is good at discovery is not the same as one that is good at prediction

Separating the two gates reveals distinct ways an AI agent can fail.

| Type | Meaning | Limitations in research | |---|---|---| | Discovery succeeds, prediction fails | Finds a good candidate but does not sufficiently learn patterns across the candidate space | Its performance may be difficult to reproduce in the next project or under new conditions | | Prediction succeeds, discovery fails | Understands relative trends among candidates but does not use its experimental budget effectively to find good ones | It may take longer to deliver tangible research results | | Both gates succeed | Finds promising candidates and makes predictions that generalize to untested ones | The most desirable form of an autonomous research system | | Both gates fail | Both the search strategy and the modeling are unreliable | Strong oversight from human researchers is needed |

This distinction means more than simply designing a better evaluation. It shows that plausible explanations or high model scores alone are not enough to determine whether AI can truly improve research productivity.

The key to an AI science agent is not “getting the answer right,” but “choosing the next experiment”

An autonomous science agent is not a model that predicts the right answer once. It is a decision-making system that runs experiments within limited time and cost, then adjusts its strategy in response to the results.

A good AI agent must therefore do two things:

• Exploration: Investigate areas that are not yet well understood to uncover new possibilities
• Exploitation: Use the information gathered so far to focus on the most promising candidates

If it leans too heavily toward exploration, it may exhaust its experimental budget without producing results. If it focuses only on exploitation, it may keep revisiting familiar areas and miss opportunities for greater discoveries.

That is exactly why Autonomous Science Bench evaluates discovery and prediction separately. Saying that AI is good at science means more than generating a good answer or correctly predicting the outcome of a single experiment. It must also be able to discover promising candidates, learn beyond what it has observed, and use that knowledge to choose the next experiment.

AI Autonomous Science Agents: The Closed-Loop Technology That Runs the Lab

AI has received the results of its first experiment. The important question is not simply, “What have we learned?” The more important question is, “What should we measure next to learn the most?”

The essence of an autonomous science agent is not answering this question just once. It lies in repeatedly running a closed loop that interprets results and changes what it chooses next—all within a limited experimental budget and timeframe.

Set research goal
   ↓
Generate experiment candidates
   ↓
Prioritize and select an experiment
   ↓
Run it using a lab, robot, or measurement service
   ↓
Collect results and check their quality
   ↓
Update the AI model
   ↓
Analyze uncertainty and information value
   ↓
Choose the next experiment
   ↺ Repeat

An AI That Decides What to Do Next, Not Just One That Reads Results

General-purpose AI tools excel at summarizing experimental data, explaining the causes of results, or predicting performance when given experimental data. But an autonomous science agent does not stop there.

Suppose there are 10,000 candidate materials, but only enough time and resources to run 50 experiments. If not every candidate can be tested, the AI must decide each round:

• Which candidate currently looks most promising?
• Which candidates have uncertain predictions?
• Which experiment could confirm or challenge an existing hypothesis?
• Which candidate could provide the most information in a single experiment?
• Given the remaining budget and time, is the balance between exploration and validation appropriate?

In other words, an autonomous science agent is not merely an analysis system; it is a decision-making system that allocates experimental resources.

Balancing Exploration and Exploitation Is Key

The hardest problem in a closed loop is deciding whether to keep testing candidates that already look promising or explore areas that are still poorly understood.

This is commonly called balancing exploitation and exploration.

| Strategy | Meaning | Advantage | Risk | |---|---|---|---| | Exploitation | Run more tests on the candidates that currently look best | Can quickly identify high-performing candidates | May become trapped by existing assumptions | | Exploration | Test candidates with high uncertainty or explore new areas | Increases the chance of unexpected discoveries | May use a limited budget inefficiently |

For example, if the AI predicts that a particular catalyst composition will perform well, an exploitation strategy focuses experiments on nearby compositions. An exploration strategy, by contrast, selects a very different composition range to investigate possibilities the current model may have overlooked.

A good agent does not commit to just one of these approaches. It changes strategy over time: exploring broadly at first to map the candidate space, then examining promising regions in greater detail.

AI Must Calculate the “Value of Information”

When an autonomous science agent chooses its next experiment, the deciding factor is not simply whether a candidate is expected to perform well. The key is how much decision-making value the experiment will provide.

Consider two candidates:

• Candidate A: likely to perform well, but the AI is already highly confident in its prediction
• Candidate B: performance is uncertain, but its result could substantially change the current hypothesis

If the goal is short-term performance, choosing A is the safer option. But from the perspective of scientific learning, B may be more valuable. If B produces an unexpected result, the AI may need to rethink how it views the entire candidate space.

Some of the main methods used for this include:

• Active learning: Selecting experiments that provide data the model can learn from most effectively
• Bayesian optimization: An optimization approach that considers both the goal of finding high performance and the goal of reducing uncertainty
• Uncertainty estimation: Calculating not only a prediction, but also the confidence level associated with it
• Acquisition function: A function that combines factors such as expected performance, uncertainty, and experimental cost to prioritize the next candidate

Ultimately, saying that AI can create a good experimental plan does not simply mean that it can predict the right answer. It means having the ability to turn questions that reduce uncertainty and increase the chances of discovery into experiments.

Failed Experiments, Too, Inform the Next Choice

In science, failure is not the same as having no data. Unexpected results, results that cannot be reproduced, and underperforming candidates can all provide important information for revising the next hypothesis.

An autonomous science agent should handle failures by:

1. Checking whether the result was caused by a simple measurement error.
2. Inspecting whether there was a problem with the experimental conditions or equipment.
3. Analyzing why the original prediction was wrong.
4. Estimating whether similar failures are likely to recur.
5. Reassessing which variables need to be tested in the next experiment.

For example, if a material is theoretically expected to have high conductivity but performs poorly in actual measurements, the AI should not simply discard that candidate. It should look for variables missing from the prediction model, such as the manufacturing process, impurities, surface structure, or measurement conditions.

The better this process works, the more the AI evolves beyond “a tool for selecting promising candidates” into a system that updates its research strategy by learning from failure.

How the Closed Loop Connects to a Real Laboratory

In real-world autonomous research environments, AI does not do everything on its own. Tasks are generally divided as follows:

• Researchers: Set research goals, safety standards, budgets, and constraints
• AI agent: Generate candidates, prioritize experiments, analyze results, and propose the next round
• Automation equipment or experimental services: Prepare samples and conduct measurements and analyses
• Validation stage: Check outliers, run replication experiments, and obtain approval from human researchers

In this structure, speed is not the only priority. Auditability is essential: records must show why an experiment was chosen, what data led to a change in the hypothesis, and how much uncertainty remains. Scientific discovery requires more than results that look promising; other researchers must be able to review and reproduce them.

The future of autonomous science agents does not lie in AI that generates more text. It lies in closed loops that choose better questions from limited experimental opportunities, learn even from failure, and accelerate discovery in partnership with human researchers.

Autonomous AI Science Agents: The 21.5% Reality Behind 67% Success

The most eye-catching number is clear. The top-performing combination, Opus 5.5 + Claude Code, succeeded in 12 out of 18 trials, achieving a 67% success rate. It is certainly impressive that AI could select candidates, interpret results, and design the next experiment within a limited experimental budget—and achieve this level of performance.

But judging the current state of the technology by this number alone means overlooking an important reality.

Across the 144 trials conducted by all eight agent–model combinations, only 31 met both of the two key criteria. That puts the overall success rate at about 21.5%.

| Metric | Result | |---|---:| | Total number of trials | 144 | | Passed both Discovery and Learning | 31 | | Overall success rate | About 21.5% | | Best-performing combination | 12/18 trials | | Best combination’s success rate | 67% | | Results on the most difficult campaign | 3/24 trials |

This gap is not simply a statistical fluctuation. It means that today’s autonomous AI science agents can be excellent experimental partners for some problems, but are still a long way from being general-purpose researchers that work reliably across all scientific problems.

The best score and the average score tell different stories

The 67% result is the top score for a particular model, agent environment, and problem combination. It reflects not only the model’s reasoning ability, but also the code execution environment, tool-calling methods, prompt design, and iterative strategy.

So it should not be interpreted as “this AI solves 67% of scientific problems.” A more accurate interpretation is:

Well-designed AI agents can perform strongly on some structured scientific exploration problems. But across the full set of conditions, success is still the exception.

In this benchmark, success is not determined by a single score. The agent must not only find good candidates, but also predict with reasonable accuracy how well candidates it has not yet tested will perform.

“Finding a good candidate” and “truly learning” are different things

Autonomous Science Bench divides success into two gates:

• Discovery gate: Did the AI find a sufficiently good design or material among the candidates it actually measured?
• Learning gate: Did it correctly predict the performance ranking of candidates it did not measure?

Passing only the first gate still leaves open the possibility that the agent selected a good candidate by chance. Passing only the second could mean it identified a trend in the data but failed to secure a high-performing candidate within its limited experimental opportunities.

That is why only the 31 trials that passed both gates count as complete successes. The criterion is meant to determine whether AI can do more than generate plausible interpretations—whether it can combine discovery with generalization.

Performance drops sharply on difficult tasks

The overall average of 21.5% matters, but the variation across tasks deserves even closer attention. On the most difficult campaign, only 3 out of 24 trials succeeded—a success rate of about 12.5%.

This result shows how strongly today’s AI agents are affected by the structure of a problem. They can perform well when the candidate space is relatively clear, measurement patterns are consistent, and tool-use procedures are well-defined. But when interactions between variables are complex or experimental results are highly uncertain, their exploration strategies can easily falter.

Common failure modes include:

• Narrowing the search too early by relying too heavily on initial results
• Repeatedly selecting only candidates that appear to perform well in the short term
• Failing to explore areas with high uncertainty sufficiently
• Misinterpreting correlations in the results as causal relationships
• Finding a good candidate but failing to generalize to unmeasured areas

Where AI stands today: an advanced experimental assistant, not an independent researcher

The result—31/144—is not disappointing; it accurately shows where the technology stands today. AI has moved beyond summarizing papers or writing experimental code and is beginning to demonstrate the ability to choose the order of experiments within a limited budget.

However, it is not yet ready to run a laboratory without supervision from human researchers. A realistic approach would look like this:

1. The researcher sets the goal, budget, and safety criteria.
2. AI proposes candidate experiments and priorities.
3. The researcher reviews key decisions and potential risks.
4. Automated equipment or a laboratory service conducts the measurements.
5. AI incorporates the results and designs the next round of experiments.

In other words, 67% is a signal of what may be possible, while 21.5% is a reminder of the reality that still needs to be tested. The competitiveness of autonomous science agents will depend less on increasingly impressive language generation and more on their ability to manage uncertainty, learn from failure, and allocate experimental resources.

AI-Driven Autonomous Science: A Verifiable Research Partner, Not an Independent Researcher

AI has moved beyond summarizing papers and explaining experimental data. It is now beginning to suggest “what to experiment on next.” But that does not mean AI has become an independent researcher capable of solving every scientific problem.

As the results of the Autonomous Science Bench show, even the best-performing systems did not solve every task reliably. Of 144 trials in total, only 31—about 21.5%—passed both the discovery and learning gates. In some tasks, systems found promising candidates but failed to accurately predict the performance of untested ones. In others, they made good predictions but failed to discover truly strong candidates.

This distinction matters. In science, it is not enough to come up with a plausible explanation. We need to be able to answer:

“Why should we run this experiment now, and if the result is wrong, what should we change next?”

What AI Does Well: Quickly Narrowing the Experimental Search Space

Research into drugs, batteries, catalysts, and new materials can involve thousands or even millions of candidates. Testing every candidate directly is impossible in terms of both time and cost. In this context, AI’s most practical role is to help researchers use their experimental budgets more efficiently.

AI agents can be particularly effective at tasks such as:

• Prioritizing promising candidates based on results so far
• Considering both the likelihood of strong performance and the value of the information a candidate could provide
• Identifying patterns in failed experiments and adjusting the next set of conditions
• Predicting the performance and uncertainty of untested candidates
• Automating tasks such as repeated experiments, data organization, and code execution
• Structuring hypotheses and supporting evidence for researchers to review

The key is not for AI to declare the answer, but to help researchers find the most valuable next question faster.

The Roles Human Researchers Still Need to Play

Even as autonomous science systems advance, human researchers will not disappear. In fact, the most important judgments will remain in human hands.

Researchers must first define the goal. The direction AI should explore will be entirely different depending on whether the aim is to find “the highest-performing material” or “a material that is low in toxicity and can be mass-produced.” People must also set safety and ethical standards, cost limits, and reproducibility requirements.

Explanations provided by AI must also be verified. Language models can interpret results fluently and persuasively, but fluent explanations do not guarantee accurate causal reasoning. That is why research settings need safeguards such as:

• Records of why each experiment was selected and which model version was used
• Explicit statements of uncertainty, alongside predictions
• Independent replication of experiments
• Predefined criteria for success and failure
• Auditability of data and experimental procedures
• Blocking proposals that fall outside safety or ethical standards

A good AI science agent should not declare conclusions on a researcher’s behalf. It should be a system that proposes experiments in ways that can be challenged and reproduced.

The Most Realistic Future Is a “Human–AI Closed Loop”

The labs of the near future are more likely to follow a collaborative model like this than to become fully unmanned:

1. Researchers set the research goal, safety boundaries, and success criteria.
2. AI designs candidate experiments based on the literature, existing data, and experimental results.
3. Automated equipment or the research team conducts the experiments.
4. AI analyzes the results and their uncertainty, then proposes the next experiments.
5. Researchers assess the scientific validity and risks of those proposals.
6. Only validated experiments proceed to the next round.

In this model, AI is more than a simple analysis tool: it becomes a research partner that accelerates iterative exploration. Humans, meanwhile, define the meaning of the problem, interpret unexpected results, and decide which directions of research are socially acceptable.

Conclusion: AI’s Value Lies in “Verifiable Acceleration,” Not “Replacement”

Today’s autonomous science technologies represent a significant turning point. They make it possible to connect experimental design, result interpretation, and the selection of next-step candidates in a single closed loop. But the wide variation in benchmark performance and the limited overall success rate also show that it is still too early to call AI an independent scientist.

The most promising future, then, is not one in which AI pushes researchers aside. It is one in which AI helps researchers ask better questions, with less time and money, and reach verifiable discoveries faster.

The next competitive edge in science will depend not on “who can propose the most plausible hypothesis,” but on who can choose the next experiment in a more transparent and reproducible way.