The University of Southampton
University of Southampton Institutional Repository

Transformers with RL or SFT provably learn sparse Boolean functions, but differently

Transformers with RL or SFT provably learn sparse Boolean functions, but differently
Transformers with RL or SFT provably learn sparse Boolean functions, but differently
Transformers can acquire Chain-of-Thought (CoT) capabilities to solve complex reasoning tasks through fine-tuning. Reinforcement learning (RL) and supervised fine-tuning (SFT) are two primary approaches to this end. In this work, we specifically examine RL with process rewards and SFT for learning $k$-sparse Boolean functions with a one-layer transformer through intermediate reasoning steps akin to CoT. In particular, we consider $k$-sparse Boolean functions that can be recursively decomposed into fixed 2-sparse Boolean functions. We first analyze the learning dynamics of RL fine-tuning with process reward and SFT in a unified way. This allows us to identify sufficient conditions under which the transformer provably learns these sparse Boolean functions. We then verify that these conditions hold for three basic examples, including $k$-PARITY, $k$-AND, and $k$-OR, thus demonstrating their learnability via both RL and SFT. Notably, we reveal that RL and SFT exhibit distinct learning behaviors: RL learns the whole CoT chain simultaneously, whereas SFT naturally learns the CoT chain step by step. Overall, our findings provide insights on the mechanisms underlying RL and SFT and how they differ in triggering the CoT capabilities of transformers, and suggest that the comparison between RL and SFT may need to consider the reward design and the use of teacher forcing.
cs.LG, stat.ML
PMLR
Lyu, Bochen
fb1af04c-d0d6-490b-a238-73c1dd1aed2a
Jia, Yiyang
b50e71bb-01b7-45d2-9cf0-5cdfc128390b
Cai, Xiaohao
de483445-45e9-4b21-a4e8-b0427fc72cee
Zhu, Zhanxing
e55e7385-8ba2-4a85-8bae-e00defb7d7f0
Lyu, Bochen
fb1af04c-d0d6-490b-a238-73c1dd1aed2a
Jia, Yiyang
b50e71bb-01b7-45d2-9cf0-5cdfc128390b
Cai, Xiaohao
de483445-45e9-4b21-a4e8-b0427fc72cee
Zhu, Zhanxing
e55e7385-8ba2-4a85-8bae-e00defb7d7f0

Lyu, Bochen, Jia, Yiyang, Cai, Xiaohao and Zhu, Zhanxing (2026) Transformers with RL or SFT provably learn sparse Boolean functions, but differently. In International Conference on Machine Learning (ICML) 2026. PMLR. 50 pp . (doi:10.48550/arXiv.2511.17852).

Record type: Conference or Workshop Item (Paper)

Abstract

Transformers can acquire Chain-of-Thought (CoT) capabilities to solve complex reasoning tasks through fine-tuning. Reinforcement learning (RL) and supervised fine-tuning (SFT) are two primary approaches to this end. In this work, we specifically examine RL with process rewards and SFT for learning $k$-sparse Boolean functions with a one-layer transformer through intermediate reasoning steps akin to CoT. In particular, we consider $k$-sparse Boolean functions that can be recursively decomposed into fixed 2-sparse Boolean functions. We first analyze the learning dynamics of RL fine-tuning with process reward and SFT in a unified way. This allows us to identify sufficient conditions under which the transformer provably learns these sparse Boolean functions. We then verify that these conditions hold for three basic examples, including $k$-PARITY, $k$-AND, and $k$-OR, thus demonstrating their learnability via both RL and SFT. Notably, we reveal that RL and SFT exhibit distinct learning behaviors: RL learns the whole CoT chain simultaneously, whereas SFT naturally learns the CoT chain step by step. Overall, our findings provide insights on the mechanisms underlying RL and SFT and how they differ in triggering the CoT capabilities of transformers, and suggest that the comparison between RL and SFT may need to consider the reward design and the use of teacher forcing.

Text
RL:SFT_ICML'26 - Version of Record
Download (3MB)

More information

Published date: 2026
Keywords: cs.LG, stat.ML

Identifiers

Local EPrints ID: 513893
URI: https://eprints.soton.ac.uk/id/eprint/513893
PURE UUID: e5297ec6-f3c9-4654-9408-12334112f27d
ORCID for Xiaohao Cai: ORCID iD orcid.org/0000-0003-0924-2834
ORCID for Zhanxing Zhu: ORCID iD orcid.org/0000-0002-2141-6553

Catalogue record

Date deposited: 01 Sep 2026 16:41
Last modified: 13 Sep 2026 15:51

Export record

Altmetrics

Contributors

Author: Bochen Lyu
Author: Yiyang Jia
Author: Xiaohao Cai ORCID iD
Author: Zhanxing Zhu ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×