AWS Unveils Fine-Tuning Approach for GPT-OSS-20B Multilingual Reasoning: Combining SFT and GRPO for Enhanced Performance

Summary:AWS researchers demonstrated a fine-tuning approach for GPT-OSS-20B on multilingual reasoning tasks using Amazon SageMaker HyperPod. By combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) reinforcement learning, the model achieved significant performance improvements. SFT boosted reasoning-language accuracy from 22% to 100%, while GRPO further refined answer-language accuracy from 62% to 70%. This approach offers a novel technical pathway for optimizing multiling

---

Background and Challenge

GPT-OSS-20B is an open-source reasoning model that primarily reasons in English. When prompted to reason in French, German, Spanish, or Italian, the model acknowledges the instruction but produces its reasoning chain in English anyway. On a 50-question multilingual reasoning test, the model reasoned correctly in the target language only 22% of the time, with almost all correct responses coming from English examples (English 100%, French 0%, German 10%, Spanish 0%, Italian 0%). For teams deploying reasoning models for non-English users, this gap is critical.

Technical Solution

To address this issue, AWS researchers fine-tuned GPT-OSS-20B on Amazon SageMaker HyperPod, focusing on the combination of Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) reinforcement learning.

Dataset and Task

The multilingual reasoning dataset used is HuggingFaceH4/Multilingual-Thinking, which includes chain-of-thought reasoning examples translated into English, French, German, Spanish, and Italian, with approximately 949 training and 50 validation examples. The model's task is to reason and answer in the target language according to the system prompt instruction.

Combining SFT and GRPO

• SFT's Role: SFT teaches the model a behavior it doesn't have. In this case, SFT boosted reasoning-language accuracy from 22% to 100%.
• GRPO's Role: GRPO optimizes a behavior the model can already sometimes produce. In this case, GRPO refined answer-language accuracy from 62% to 70%.

The combination of SFT and GRPO demonstrates that SFT lays the foundation, and GRPO refines when there is room for improvement.

Training Architecture

Training was conducted on Amazon SageMaker HyperPod with Amazon EKS orchestration. The initial SFT and GRPO runs used the PyTorch TRL library, while the Qwen control experiment switched to veRL as the trainer, with Ray and KubeRay for distributed orchestration and vLLM for fast rollout generation. All fine-tuning used Low-Rank Adaptation (LoRA), which is parameter-efficient and well-suited to both SFT and GRPO.

Experimental Results

• SFT Results: After SFT, the model reasoned correctly in the target language 100% of the time across all five languages.
• GRPO Results: GRPO further improved answer-language accuracy from 62% to 70%, while maintaining 100% reasoning-language accuracy.

Best Practices

1. Start with SFT: GRPO cannot teach a behavior with zero base probability. It can only reinforce responses the model already sometimes produces.
2. Use LoRA for both SFT and GRPO: LoRA r=8/α=16 is parameter-efficient and performed well in the Qwen control experiment.
3. Match the technique to the gap: Use SFT to fill in missing behaviors, and use GRPO to refine behaviors that already exist but are not fully correct.
4. Check for reward headroom before spending GPU-hours: If most groups are all-correct, GRPO will not receive a gradient.

Conclusion

SFT did most of the work in the multilingual reasoning task, while GRPO provided significant refinement where there was room for improvement. This approach offers a new technical pathway for optimizing multilingual AI models.

---

Source: AWS Machine Learning Blog (2026-10-20)

— END —

Tags: #AWS Follow #GPT-OSS-20B Follow #Multilingual Reasoning Follow #Fine-Tuning Follow #Reinforcement Learning Follow