Large Reasoning Model


原 PPT 独立图片资源
Natural Language
Understanding

Large Reasoning Model
Mingming Sun
BIMSA

Part III: Course Map


原 PPT 独立图片资源
Part III: Course Map
3.1 Reasoning Processes and Search

3.2 Learning and Scaling Reasoning

3.3 Logic and Uncertainty

3.1 Reasoning Processes and Search 3.1.1 Explicit Reasoning and Decomposition



原 PPT 独立图片资源

3.1 Reasoning Processes and Search 3.1.1 Explicit Reasoning and Decomposition

原 PPT 独立图片资源

When Direct Answers Fail

An answer without the steps

A cafeteria starts with 23 apples, uses 20 for lunch, then buys 6 more. How many remain?

Standard-prompting output in the paper:
“The answer is 27.”

Which operation went wrong?

Example from Wei et al., Figure 1; not a new model run.

Does a larger model solve it?

GSM8K · standard few-shot prompting

Model Accuracy
PaLM 540B 17.9%

The prompt supplies question–answer examples, but no intermediate reasoning.

Wei et al., Table 2. Historical benchmark setting.

What if the examples showed how to reach the answer?

Chain of Thought: Building the Prompt

Example question: Roger has 5 tennis balls and buys 2 cans with 3 balls each. How many balls does he have?

Answer-only demonstration

Answer: 11.

Question → final answer

Add the intermediate steps

Two cans add 2\times3=6 balls.
Roger now has 5+6=11 balls.
Answer: 11.

Question → reasoning steps → answer

Apply the pattern to the apples: 23-20=3;\quad3+6=9, replacing the paper’s direct answer 27.

Few-shot CoT: change the demonstrations, keep the model weights fixed. Adapted from Wei et al., Figure 1.

Chain of Thought: Worked Example


原 PPT 独立图片资源
Chain of Thought: Worked Example
An interesting phenomenon in few shot learning on Pretrained LLM
原 PPT 独立图片资源
https://arxiv.org/pdf/2201.11903
原 PPT 独立图片资源

Chain of Thought: Results


原 PPT 独立图片资源
Chain of Thought: Results
https://arxiv.org/pdf/2201.11903
原 PPT 独立图片资源
原 PPT 独立图片资源

Zero-shot CoT: Method


原 PPT 独立图片资源
Zero-shot CoT: Method
https://arxiv.org/pdf/2205.11916
原 PPT 独立图片资源

Zero-shot CoT: Results


原 PPT 独立图片资源
Zero-shot CoT: Results
https://arxiv.org/pdf/2205.11916
原 PPT 独立图片资源
原 PPT 独立图片资源

Least-to-Most Prompting

Plan subproblems; reuse their answers

  1. Decompose: identify the smaller questions needed to solve the original task.
  2. Solve in order: include earlier question–answer pairs in each later prompt.
  3. Compose: use those intermediate answers to solve the original question.
Amy’s slide
One trip: 4+1=5 minutes.
15\,\text{minutes}\div5=3 complete trips.

A missing dependency or wrong sub-answer can propagate. No interpreter is required.

Least-to-Most Figure 1: decompose Amy’s water-slide question, then append the first answer to solve the second question.
Zhou et al. · ICLR 2023 · Fig. 1, p. 2
Original figure; click to enlarge. Few-shot demonstrations are omitted in the paper figure.
Longer action instructions (SCAN)
“run and look twice” → RUN LOOK LOOK
CoT 16.2% → Least-to-Most 99.7%
Length split · code-davinci-002 · Table 8.

PAL: Program-aided Language Models

Generate a program; execute the calculation

Language model: map the story to variables, operations and executable Python.

Interpreter: run the generated program and return its computed answer.

baked = 200
sold = 93 + 39
returned = 6
answer = baked - sold + returned  # 74

“Returned” means +6 here. An interpreter also executes the wrong translation (-6) faithfully.

PAL Figure 1 lower-right panel: generated bread-inventory program, interpreter execution, and result 74.
Gao et al. · ICML 2023 · Fig. 1, p. 2
Paper screenshot, lower-right panel. Click for the full CoT / PAL comparison.
When the arithmetic gets harder
GSM-Hard: CoT 23.1% → PAL 61.2%
Word-problem quantities replaced by large numbers.
code-davinci-002 · Table 1; three prompt orders.

PoT: Program of Thoughts

Prompt supplied to the model

Write Python. Store the result in ans.
# Example: 5 balls + 2 cans, 3 per can.
balls = 5
new_balls = 2 * 3
ans = balls + new_balls

# New question: 23 apples; use 20
# for lunch, then buy 6. How many remain?
# Python:

Few-shot: question–program examples, then a new question with its program left blank.

Zero-shot: instruction + new question, without the worked examples.

Model continuation → execution

apples = 23
remaining = apples - 20
ans = remaining + 6

Python executes the code: ans = 9.

Illustrative prompt and continuation; one demonstration shown.

PoT Figure 2 upper panel: few-shot Codex results, CoT in blue and PoT in red, across seven numerical reasoning datasets.
Chen et al. · TMLR 2023 · Fig. 2, p. 3
Upper panel: few-shot Codex. GSM8K: CoT 63.1% → PoT 71.6%. Historical results under the paper’s evaluation protocol.

3.1 Reasoning Processes and Search 3.1.2 Test-Time Scaling I: Search and Feedback



原 PPT 独立图片资源

3.1 Reasoning Processes and Search 3.1.2 Test-Time Scaling I: Search and Feedback

原 PPT 独立图片资源

Reasoning Paths as a Search Space


原 PPT 独立图片资源
Reasoning Paths as a Search Space
The decoding space is huge!
It may contain the correct answer for complex problems.
The problem is: How to find them in the huge decoding space?
We’ve learned to activate the COT using carefully designed prompt;
Can we do better without controlling prompt?
Explore in the decoding phase to find logical path to the correct answer ! – Testing Time Scaling

Allocating Test-Time Compute

Breadth

How many candidate paths or branches should we explore?

Continuation
& Revision

How long should we continue a path, and how many times should we revise it?

Selection
& Stopping

Which candidates should we keep, and when should we stop?

CoT-decoding: Candidate Generation


原 PPT 独立图片资源
CoT-decoding: Candidate Generation








Pre-trained LLMs have had responses with step-by-step reasoning among the generations started with the top-k tokens;
But how can we extract the correct reasoning answers?
原 PPT 独立图片资源
Image Source: Xuezhi Wang, et. al, Chain-of-Thought Reasoning Without Prompting, arXiv, 2024

CoT-decoding: Answer Selection


原 PPT 独立图片资源
CoT-decoding: Answer Selection
When a step-by-step reasoning path is present, the final answer will have higher confidence;
Extracting the final answer and then compare the confidence, we get COT sequence with good performance.
原 PPT 独立图片资源
Image Source: Xuezhi Wang, et. al, Chain-of-Thought Reasoning Without Prompting, arXiv, 2024
原 PPT 独立图片资源

CoT-decoding: Results


原 PPT 独立图片资源
CoT-decoding: Results
When a step-by-step reasoning path is present, the final answer will have higher confidence;
Extracting the final answer and then compare the confidence, we get COT sequence with good performance.
Image Source: Xuezhi Wang, et. al, Chain-of-Thought Reasoning Without Prompting, arXiv, 2024
原 PPT 独立图片资源
原 PPT 独立图片资源

Self-Consistency: Answer Marginalization


原 PPT 独立图片资源
Self-Consistency: Answer Marginalization
With COT, LLM does following in decoding:
arg max 𝑃(𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔 𝑝𝑎𝑡h, 𝑓𝑖𝑛𝑎𝑙 𝑎𝑛𝑠𝑤𝑒𝑟|𝑝𝑟𝑜𝑏𝑒𝑙𝑚)  
But we want:
arg max 𝑃(𝑓𝑖𝑛𝑎𝑙 𝑎𝑛𝑠𝑤𝑒𝑟|𝑝𝑟𝑜𝑏𝑒𝑙𝑚)  
We can do this by sampling:
𝑃(𝑓𝑖𝑛𝑎𝑙 𝑎𝑛𝑠𝑤𝑒𝑟|𝑝𝑟𝑜𝑏𝑒𝑙𝑚)=∑{𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔 𝑝𝑎𝑡h}𝑃(𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔 𝑝𝑎𝑡h, 𝑓𝑖𝑛𝑎𝑙 𝑎𝑛𝑠𝑤𝑒𝑟|𝑝𝑟𝑜𝑏𝑙𝑒𝑚)
Xuezhi Wang, et, al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.

Self-Consistency: Sampling and Aggregation


原 PPT 独立图片资源
Self-Consistency: Sampling and Aggregation
Xuezhi Wang, et, al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
原 PPT 独立图片资源

Self-Consistency: Results and Agreement


原 PPT 独立图片资源
Self-Consistency: Results and Agreement
Xuezhi Wang, et, al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
原 PPT 独立图片资源
原 PPT 独立图片资源

Tree of Thoughts: Search and Pruning


原 PPT 独立图片资源
Tree of Thoughts: Search and Pruning
In each state, generate several next thought, then evaluate and prune.
Just like tree search.
原 PPT 独立图片资源
Shunyu Yao, et, al. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Arxiv, 2023

Tree of Thoughts: Game of 24


原 PPT 独立图片资源
Tree of Thoughts: Game of 24
Shunyu Yao, et, al. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Arxiv, 2023
原 PPT 独立图片资源

Self-Refine: Feedback and Revision


原 PPT 独立图片资源
Self-Refine: Feedback and Revision

原 PPT 独立图片资源
Image Source: https://selfrefine.info/

Self-Refine: A Revision Example


原 PPT 独立图片资源
Self-Refine: A Revision Example
原 PPT 独立图片资源
Image Source: https://selfrefine.info/

Self-Correction: Limits without External Feedback


原 PPT 独立图片资源
Self-Correction: Limits without External Feedback
Jie Huang, et.al, Large language models cannot self-correct reasoning yet, ICLR 2024
“Large language models cannot self-correct reasoning yet”
Question: If an LLM possesses the ability to self-correct, why doesn’t it simply offer the correct answer in its initial attempt?
The promising results of previous research is misleading;
LLMs cannot properly judge the correctness of their reasoning -- the feedback is misleading.
原 PPT 独立图片资源

Reason, Act, Reflect

ALFWorld: use the observed state to complete a household task.

ReAct: clean and place a knife

Act fails: it has the knife, but tries to clean it before reaching the sink; commands repeat.

ReAct succeeds: track the completed subgoal; go to the sink, clean the knife, then place it.

Act 45% → ReAct 71%

Success rate · PaLM-540B · 134 tasks.
Best of 6 prompt sets for each method; Table 3.

Reflexion: inspect a mug by a lamp

First trial fails: the lamp was on desk 1, but the agent moved to desk 2 and tried to use it.

Next trial succeeds: remember the location; return to desk 1, take the mug, and use the lamp.

130 / 134 tasks solved

ReAct + Reflexion · GPT-3 · Figure 3.
Cumulative across up to 12 trials, not one-shot.

ReAct updates the current plan; Reflexion reuses a lesson on the next attempt. Neither updates model weights.

Paper trajectories, paraphrased. The two result panels use different evaluation protocols.

Learned Feedback: Critic GPT


原 PPT 独立图片资源
Learned Feedback: Critic GPT
Nat McAleese, et.al, LLM Critics Help Catch LLM Bugs, arxiv 2024
Useful critiques still need independent checks.
原 PPT 独立图片资源

Test-Time Compute: When Is It More Efficient?


原 PPT 独立图片资源
Test-Time Compute: When Is It More Efficient?
Snell et al., Figure 1: left, inference allocation; right, FLOPs-matched pretraining versus inference, split by difficulty and total inference load.
MATH · same total FLOPs
A: pretrain a ~14× larger model;
use greedy decoding.
B: keep PaLM 2-S*;
spend more on revisions / search.
Same pretraining token count in both options.
R = total inference tokens / pretraining tokens
Medium questions · revisions
R = 0.16+27.8%R = 22−24.3%
Relative accuracy of B vs A, not percentage points.
Positive: B wins. Negative: A wins.

Test-Time Compute: A Unified Framework

Apple case: 23-20+6. A first answer says 27; the correct answer is 9.

compute overviewAt fixed model weights, a controller spends a question budget on generation, continuation and evaluation. Feedback returns to the controller; only stopping produces the final answer. This loop contains no parameter update.
Question x
Budget B
Allocate computationBranch · continue · stop
Generate with \theta
Weights stay fixed
Evaluate / selectVote · score · testfeedback or selected state → continue / stop → answer
Control What changes in this case What it buys
Candidate breadth Sample answers: 27, 9, 9 Expose alternatives
Sequential work Recheck 23-20=3; then +6 Repair the arithmetic
Selection & stopping Vote for 9; or execute the expression Return a checked answer

Learning: checked experience → update \theta → freeze \theta^{\prime} → use the same controls again.

Constructed paths, not a benchmark. A majority can still be wrong. Count all generation and checking costs.