4.7 Learning and Evaluation 4.7.1 Feedback and System Optimization



原 PPT 独立图片资源

4.7 Learning and Evaluation 4.7.1 Feedback and System Optimization

原 PPT 独立图片资源

Search, Learn, Evaluate: A Conditional Cycle

improvement cyclePolicy Propose candidates → Search + check Improve / filter outputs → Training Update parameters → Held-out evaluation Measure gain and cost PolicyPropose candidatesSearch + checkImprove / filter outputsTrainingUpdate parametersHeld-out evaluationMeasure gain and cost

Better search outputs may teach a better policy—if supervision is useful and transferable.

Possible failure: biased checks, copied errors, no useful reward, distribution shift.

Stop or revise: no held-out gain, worse reliability, excessive cost, exhausted evidence.

An iterative procedure is not evidence of unlimited improvement or inevitable superintelligence.

Case Study: Alpha Evolve


原 PPT 独立图片资源
Case Study: Alpha Evolve
https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
原 PPT 独立图片资源

Agent Paradigm


原 PPT 独立图片资源
Agent Paradigm
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

Agent Flow


原 PPT 独立图片资源
Agent Flow
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

Agent Flow


原 PPT 独立图片资源
Agent Flow
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

4.7 Learning and Evaluation 4.7.2 Distillation and Transfer

Distillation: Can Slow Work Improve Fast Responses?

system2 distillationExtra computation Generate stronger outputs → Select targets Consistency / filtering → Answer-only SFT Do not retain rationale → Direct response Evaluate on held-out tasks Extra computationGenerate stronger outputsSelect targetsConsistency / filteringAnswer-only SFTDo not retain rationaleDirect responseEvaluate on held-out tasks

Yu et al. (2024): several methods transfer useful behavior; transfer is task-dependent.

Self-consistency filtering is not formal verification.
This experiment removes intermediate reasoning from the training target; other distillation designs can retain it.

Distillation Can Fail: Keep the Negative Result

Llama-2-70B-chat · GSM8K · k = 1 · original Table 4

Condition Accuracy Generated tokens
Direct few-shot (no CoT) 7.58% 57
Zero-shot CoT teacher 52.77% 270
Answer-only CoT distillation 7.13% 18

Fewer output tokens did not preserve the teacher’s math accuracy.

This is one base model and one self-supervised answer-only recipe. It does not show that all reasoning distillation is impossible.

S1-Bench: Test the Easy End Too

IJCAI 2026: evaluate simple tasks, response cost and consistency across repeated samples.

Table 3 pass@1 acc@5 (all 5 correct) Mean tokens
DS-R1-1.5B 81.47% 54.50% 489.54
DS-R1 (671B) 100.00% 100.00% 646.40

Good hard-task scores do not settle simple-task reliability or efficiency.

T=0.6; top-p=0.95; k=5; max 10K. Paper-defined metrics use formatting checks and GPT-4o judging. “System 1” is a benchmark label, not a diagnosis of a mental mechanism.

4.7 Learning and Evaluation 4.7.3 Evaluation and Integration

Evaluate the Whole Controller

Compare on the same tasks Report together
Always direct / always deliberate Correctness, hard-constraint violations
Fixed budget / learned allocation Tokens, actual latency, tail latency
Reason only / retrieve evidence Tool cost, evidence gained, retries
Before / after training Held-out transfer, training cost, regressions

Count routing, failed attempts and tool calls. Preserve the same checking rules.
A policy that refuses every task is cheap; report coverage as well as accepted-case accuracy.

The Agent System as a Whole

agent course handoffControl policy What should happen next? → Tool calling Execute a typed action → Memory + knowledge Retain / retrieve evidence → Feedback + evaluation Update state; check goals Control policyWhat should happen next?Tool callingExecute a typed actionMemory + knowledgeRetain / retrieve evidenceFeedback + evaluationUpdate state; check goals

Tools, code, memory and retrieval provide operations for the controller.
Next: divide work across agents and check the joint result.

A successful local step is not enough: evaluate the complete task, including handoffs and failed attempts.

Thank You!



原 PPT 独立图片资源
Thank You!
原 PPT 独立图片资源