
4.7 Learning and Evaluation 4.7.1 Feedback and System Optimization

Better search outputs may teach a better policy—if supervision is useful and transferable.
Possible failure: biased checks, copied errors, no useful reward, distribution shift.
Stop or revise: no held-out gain, worse reliability, excessive cost, exhausted evidence.
An iterative procedure is not evidence of unlimited improvement or inevitable superintelligence.








Yu et al. (2024): several methods transfer useful behavior; transfer is task-dependent.
Self-consistency filtering is not formal verification.
This experiment removes intermediate reasoning from the training target; other distillation designs can retain it.
Llama-2-70B-chat · GSM8K · k = 1 · original Table 4
| Condition | Accuracy | Generated tokens |
|---|---|---|
| Direct few-shot (no CoT) | 7.58% | 57 |
| Zero-shot CoT teacher | 52.77% | 270 |
| Answer-only CoT distillation | 7.13% | 18 |
Fewer output tokens did not preserve the teacher’s math accuracy.
This is one base model and one self-supervised answer-only recipe. It does not show that all reasoning distillation is impossible.
IJCAI 2026: evaluate simple tasks, response cost and consistency across repeated samples.
| Table 3 | pass@1 | acc@5 (all 5 correct) | Mean tokens |
|---|---|---|---|
| DS-R1-1.5B | 81.47% | 54.50% | 489.54 |
| DS-R1 (671B) | 100.00% | 100.00% | 646.40 |
Good hard-task scores do not settle simple-task reliability or efficiency.
T=0.6; top-p=0.95; k=5; max 10K. Paper-defined metrics use formatting checks and GPT-4o judging. “System 1” is a benchmark label, not a diagnosis of a mental mechanism.
| Compare on the same tasks | Report together |
|---|---|
| Always direct / always deliberate | Correctness, hard-constraint violations |
| Fixed budget / learned allocation | Tokens, actual latency, tail latency |
| Reason only / retrieve evidence | Tool cost, evidence gained, retries |
| Before / after training | Held-out transfer, training cost, regressions |
Count routing, failed attempts and tool calls. Preserve the same checking rules.
A policy that refuses every task is cheap; report coverage as well as accepted-case accuracy.
Tools, code, memory and retrieval provide operations for the controller.
Next: divide work across agents and check the joint result.
A successful local step is not enough: evaluate the complete task, including handoffs and failed attempts.

