AI
SageMaker RLVR's Cached Score Does Not Prove Custom-Reward Gains
A pinned SageMaker audit traces a precomputed tuned-score path, unestablished custom-reward checkpoint lineage, and the limits of a reasoning-presence heuristic.
Which training run produced that score?
If a workshop teaches a custom reward function, what evidence would convince me that the custom reward improved its model?
The October 12 Daily Dose of DS issue points to a SageMaker AI RLVR workshop. The useful discovery is not that a Python function can return a reward. It is that the workshop’s custom-reward evaluation notebook defaults to a precomputed tuned result under a lab3a prefix, while an optional import helper targets a lab3a-named repository and registers its checkpoint in a custom-reward-labeled group.
Those paths make a workshop easier to explore. They do not establish the benefit of the custom scorer in Lab3b.
Source Audit scope: this post is dated to the newsletter’s Korea-time issue. I inspected workshop commit 4d56206fc9f10014e81f2640c830565238f127fa, committed October 12 at 22:21 KST, on October 14. Live AWS documentation is explicitly supplemental. I recalculated saved metric aggregates locally, but did not train a model, invoke the scorer, launch an evaluation, deploy an endpoint, or obtain an AWS bill.
Follow the result, not the lab label
The three evaluation paths are different
The pinned evaluation notebook sets RUN_EVALUATIONS = False. Its default result-download branch uses a lab3a/eval_results precomputed prefix, downloads only tuned_benchmark_results.json, and passes base=None to the comparison display. The default therefore displays a tuned-only result, not a paired comparison.
If the live branch is enabled, it launches Benchmark.MATH for the tuned model with evaluate_base_model=False. That is a fresh tuned-model evaluation path, not a newly paired base-versus-custom-reward comparison. It also does not evaluate the GSM8K test split prepared earlier merely because that split was registered.
The optional import helper names nicolasjjj/sagemaker-serverless-customization-lab3a as its Hugging Face repository. It resolves files from unpinned main, so the repository name and a group ending in custom-rw-rlvr do not establish the checkpoint’s actual training lineage.
| Inspected path | What it establishes | What it does not establish |
|---|---|---|
| Default precomputed download | Code to display a stored tuned-only result | A paired baseline, or a newly completed Lab3b evaluation |
| Live tuned-only MATH evaluation | The notebook can request that evaluation | A fresh paired baseline, or evaluation of the prepared GSM8K test split |
| Optional import from a Lab3a-named repository | Code to register a checkpoint in a custom-labeled group | Its actual training lineage or use of Lab3b’s scorer |
Original diagram. Solid arrows describe the inspected default path. Dashed arrows are a recommended provenance chain, not a workflow executed in this audit.
The six inspected Lab3b notebooks have no saved code-cell outputs and no non-null execution counts. That is a limitation of the checked-in evidence, not proof that their authors never ran them.
The recorded headline is a category macro mean
The saved Lab3a result files contain seven MATH subject scores. Their all exact-match value is the unweighted mean of those subject scores. I independently recalculated that aggregate and a category-count-weighted mean from the saved JSON.
| Saved Lab3a results | Base | Tuned | Interpretation |
|---|---|---|---|
| Seven-category macro mean | 0.406449 | 0.507121 | Each subject has equal weight |
| Category-count-weighted mean | 0.444600 | 0.541000 | Weights use the recorded counts, totaling 5,000 examples |
The macro difference is about 10.07 percentage points. It is a stored Lab3a comparison, not a measured custom-scorer improvement. The weighted figure is arithmetic over category scores and counts, not a new evaluation or a reconstruction of per-example predictions.
The JSON records do not freeze the dataset revision: the task hf_revision values are null. Endpoint example settings elsewhere in the workshop are not evidence of the benchmark’s decoding configuration. I did not reproduce repeated seeds, paired significance testing, or training-job-to-result linkage.
This is the same baseline discipline discussed in Before You Fine-Tune a Jev-Style Model, Write Down the Baseline: identify the experimental unit before interpreting its improvement.
The reward is more precise than “correct reasoning”
The custom scorer combines four components:
1
2
3
4
reward = 0.6 * numeric_correctness
+ 0.2 * parseability
+ 0.1 * delimiter_format
+ 0.1 * reasoning_presence
Here, reasoning_presence means at least eight whitespace-separated words before the first ####. It does not verify whether those words form valid mathematical reasoning.
The parser takes the first delimited numeric match, or otherwise the last numeric substring, and compares Decimal values. That is a concrete numeric-answer verifier, not a general symbolic-mathematics verifier.
By inspection and arithmetic, a wrong answer can receive 0.4 when it is parseable, formatted, and wordy enough. A short correct answer without the delimiter can receive 0.8 through correctness and parseability alone. These are static consequences of the scorer, not executed scorer-test results.
That does not make reward shaping inherently wrong. It tells us what behavior is being purchased with each part of the reward. A format reward can train a useful output convention; an eight-word check cannot justify a claim that reasoning quality was verified.
“180 examples” is the input pool, not the final training split
The data-preparation notebook takes the first 180 streamed GSM8K training records. It does not randomly sample the entire training set.
A 20% validation split, followed by a 10% test split of the remainder, both use random_state=42. Under ordinary scikit-learn rounding, that implies 129 training, 36 validation, and 15 test records. This is a static size derivation; I did not reconstruct the sample membership.
The training notebook does pass a custom reward-function object to RLVRTrainer. That verifies the integration path in source. The configured API call and the completed training run remain different claims.
Make the scorer-to-score chain inspectable
For a follow-up experiment, I would require a compact lineage record before comparing results:
- Data: immutable dataset revision, split IDs, and a hash of the evaluated prompts.
- Verifier: scorer source hash plus tests for conflicting delimiters, absent answers, numeric formats, and intentionally wrong but well-formatted responses.
- Training: base model revision, reward weights, job identifier, and the exact output checkpoint.
- Evaluation: that checkpoint and the base model on the same held-out prompts, with recorded decode settings and an explicitly named aggregation rule.
- Operations: the bill and a resource inventory, rather than an assumption that “serverless” means no persistent cost.
This is a proposed evidence contract, not an experiment completed here. It connects to RLM-FORGE’s evidence-gated recursion: a machine-readable result becomes useful evidence only when its inputs, scope, and provenance remain attached.
Budget and cleanup need their own audit
As a live supplemental observation on October 14, the workshop’s own-account setup table estimates Qwen3 0.6B RLVR at $80/hour in US East (N. Virginia). This is not a measured bill, an immutable October 12 price, or a price for the workshop’s default us-west-2 region.
The observed cleanup instructions cover inference-component, model, endpoint, and endpoint-configuration deletion, plus optional evaluator removal. They are not a verified account-wide cleanup inventory. Dataset artifacts, S3 objects, scorer Lambda/IAM resources, Studio compute, and logs need separate checks where applicable. The scorer’s debug logging includes reference answers and response previews, so retention also belongs in that inventory.
The pinned SageMaker plugin deployment guide separately warns that deployed endpoints keep billing while idle. That guide is procedural context, not proof that this workshop’s different deployment route was tested or cleaned up.
Key Takeaways
- The custom-reward lab’s default path points to a precomputed tuned-only result under a Lab3a prefix. It does not display a paired baseline.
- A custom label on an imported checkpoint is not custom-training lineage.
- Numeric correctness, parseability, format, and an eight-word heuristic are distinct reward components.
- The saved macro scores are not sample-weighted accuracy, and neither aggregate measures a new Lab3b run.
- The useful next step is a versioned, paired experiment with a budget and cleanup ledger, not a larger claim from the existing demonstration.
New Questions
Would the custom scorer outperform a correctness-only scorer when both are evaluated on the same held-out prompts? Which reward components improve answer accuracy rather than formatting? How often does the numeric parser accept a response whose final intended answer is different?
Those are experiments to run, not results to borrow from a neighboring lab.
References
Pinned code and results
Supplemental documentation, observed October 14
Research and discovery context
- DeepSeekMath, the GRPO paper, background on the optimization method, not evidence of this workshop’s gain.
- The October 12 Daily Dose of DS email supplied the discovery links. Its promotional courses are classified in the accompanying research inventory, not used as implementation evidence.
Editorial method: claims in this article point to primary documentation, pinned source code, runnable checks, or dated measurements, with AI-assisted research under the site's evidence-gated policy. How this site verifies claims and discloses AI assistance · Report a correction.
Working on something like this?
I take a small number of paid, scoped reviews: AI agent/RAG architecture diagnosis, Unity CI & build-automation audits, and multimodal QA design review. Each one ends in a written findings document.
Work with me