Does RFT Require Less Data? Our Experimental Deep-Dive

Does RFT Require Less Data? Our Experimental Deep-Dive




We tested Amazon Bedrock Reinforcement Fine-Tuning to find out when it makes sense to improve models with reinforcement learning

This article is based on a probe conducted by Genie, Emumba’s dedicated GenAI R&D group.
Reinforcement fine-tuning has attracted attention as a way to improve language models without preparing a large set of manually written target responses.
This creates an appealing possibility. If a task has a clear way to verify whether an answer is correct, could a model learn from a relatively small set of prompts and an automated reward signal?
We tested this idea using Amazon Bedrock Reinforcement Fine-Tuning. Our goal was to answer two questions:
  1. Can RFT improve a model using a small dataset?
  2. What are the practical considerations of using RFT as a managed service within AWS Bedrock?

Why RFT made a buzz

Supervised fine-tuning is commonly used to improve a language model on a particular task. It trains the model on examples that contain both an input and the desired response.
For many specialized tasks, preparing these datasets can require thousands or tens of thousands of high-quality examples. Each response may need to be written, reviewed, or corrected by a person with relevant domain knowledge. This makes dataset preparation expensive and time-consuming.
RFT uses a different learning signal.
Instead of showing the model an ideal response for every prompt, the system generates candidate responses and scores them using a reward function. The model is then updated to favour responses that receive higher rewards.
For tasks with verifiable outcomes, the reward may come from:
  • Executing generated code
  • Checking a mathematical answer
  • Validating structured output
  • Comparing a result against a known rule
  • Using an automated evaluator
This can reduce dependence on manually labeled target responses. However, reducing the labeling burden is not the same as reducing the total number of training examples.
That distinction became the central question in our experiment.

How we configured the Bedrock RFT experiment

At the time of the experiment, Amazon Nova Lite was the model supported for RFT within Bedrock.
We selected difficult mathematical reasoning problems because they provided two useful characteristics:
  • The base model had room to improve.
  • Final answers could be checked programmatically.
The experiment used the following configuration:
Component
▲▼
Configuration
▲▼
Model
Amazon Nova Lite
Task
Difficult mathematical reasoning problems
Test set
100 held-out examples
Training runs
125 and 200 examples
Reward signal
Correct or incorrect final answer
For each problem, the model generated multiple possible solutions. A reward function evaluated the answers, and Bedrock used those scores to update the model.
We implemented the evaluator as a deterministic code-based judge. It normalized mathematical expressions so that equivalent answers could be recognized correctly. For example, 1/2, 0.5, and 50% should receive the same correctness score.
This approach was faster and more predictable than using another LLM to grade every generated answer.

The small-data results were surprising

We compared the fine-tuned models with the original Nova Lite model on the held-out test set.
Model configuration
▲▼
Answer accuracy
▲▼
Base model
0.63
RFT after 125 training examples
0.54
RFT after 200 training examples
0.60
Neither RFT run outperformed the base model.
Accuracy first fell from 0.63 to 0.54. Increasing the training data from 125 to 200 examples raised it to 0.60, but the result remained below the original model.
The improvement between the two RFT runs suggested that additional data was helping. However, with only a few hundred examples, training had not produced a stable improvement over the baseline.
This does not establish that RFT can never work with a small dataset. Results depend on the model, task, reward function, training parameters, and quality of the examples. It does show that RFT should not be treated as a reliable shortcut to better performance from a very small amount of data.

So, does more data help?

We also evaluated reinforcement fine-tuning outside Bedrock across multiple models and tasks.
In most of those experiments, additional data produced substantial gains. Accuracy often increased from below 30% to approximately 80–90%.
However, those improvements required thousands of examples rather than a few hundred.
Our answer to the first question was therefore:
RFT can significantly improve models, but it does not always remove the need for a substantial amount of training data.
Its primary data advantage may be a reduction in the amount of human labeling required, rather than a guaranteed reduction in the number of examples.
A task may still need thousands of prompts. The difference is that those prompts can sometimes be scored automatically instead of requiring a person to write the ideal response for each one.

Practical lessons from using Bedrock RFT

Overall, Bedrock made the RFT workflow accessible for a team already familiar with AWS. It managed the training infrastructure and allowed a custom reward function to be connected through AWS Lambda.
However, several practical considerations affected the experiment.

Reward design affects behaviour and cost

The reward function does more than determine whether the model is learning. It also shapes how the model tries to earn a higher score.
In one configuration, we rewarded both answer correctness and compliance with a particular response format. The model began producing unnecessarily long responses, reaching approximately 4,000 tokens in some cases.
In the second configuration, we simplified the reward to binary answer correctness. Responses became much shorter, at approximately 1,200 tokens.
Across the two configurations, total training cost fell from approximately $360 to $52. This reduction also reflected changes to the dataset size and number of generated completions, but it demonstrated how strongly reward design and training parameters can affect resource usage.
A poorly designed reward can encourage behaviour that technically earns points without improving the outcome that matters.

Deterministic judges work well for objective tasks

For mathematics, coding, and other objectively verifiable tasks, a code-based judge has several advantages:
  • Low evaluation latency
  • Predictable scoring
  • Lower inference cost
  • No dependence on another model’s judgment
  • Fewer API throttling and timeout issues
An LLM-based judge may still be needed for subjective tasks, but it introduces additional cost and variability.

Managed does not mean configuration-free

Bedrock manages the underlying training infrastructure, but the surrounding AWS setup still requires care.
The workflow may involve:
  • Bedrock service roles
  • Lambda execution and resource policies
  • S3 access
  • Logging permissions
  • Training parameters that must be compatible with the managed infrastructure
These requirements are manageable for AWS-savvy teams, but they should be included in implementation planning.

Cost can change quickly

RFT cost is affected by more than the number of examples. It can also change with:
  • The number of completions generated per prompt
  • Response length
  • Batch size
  • Training epochs
  • Reward-function behaviour
  • Small hyperparameter changes
Teams should therefore begin with controlled pilot runs and monitor both learning metrics and generation behaviour before scaling the dataset.

Deployment remains within Bedrock

Models fine-tuned through Bedrock RFT are deployed within the Bedrock platform.
This simplifies integration for systems already built on AWS, but it also creates a platform constraint. Teams should consider where the model must eventually be hosted before committing to the managed fine-tuning workflow.

Key takeaway

Labeling thousands of examples for language-model fine-tuning is difficult and expensive.
RFT can reduce this burden by replacing manually written target responses with automated reward signals. This is especially valuable for tasks where correctness can be verified through code, rules, tests, or other deterministic checks.
However, reducing reliance on labeled responses does not necessarily mean requiring fewer training examples.
In our Bedrock experiment, 125 and 200 examples were not enough to improve Nova Lite beyond its baseline accuracy. In our broader experiments, strong gains were possible, but often only after training on thousands of examples.
RFT should therefore be viewed as a way to change how training data is evaluated, not as a guarantee that very little data will be required.
Its success still depends on the task, the quality and quantity of examples, the reward function, the stability of training, and the cost of experimentation.