Without reproduction people cannot form a stable mental model; the learning curve flattens

What it is

Learning an interface is being able to run “I do this, it comes back like that” again. Generation turns the come-back into a distribution; run it again and another face arrives; the causal sentence will not finish. People stay in probing and never reach fluency. Non-reproducibility flattens the learning curve: time increases, the model stays put.

A lucky hit makes people overestimate ability — that is attribution. This is skill acquisition itself missing a repeatable trial.

Why it happens

Skill comes from repetition with feedback. If the same act yields different results, the variance of the reinforcing signal swamps the mean and policy updates approach random. People either give up induction (“it’s just mysticism”) or induce superstition (a fixed incantation). Neither is the expert user a product wanted.

Instructional design depends on exercises that can be redone. In a generate tutorial, “do it like this” walks out a different result next time, and the exercise cannot be homework. On a plot the learning curve becomes a horizontal line: trial n is no more skilled than trial 1.

Studying it

Practise the same task several times; compare a determinate tool with a generate tool on strategy convergence (can they state a rule, does time on equivalent tasks fall). Independent variables: whether a successful sample can be pinned, whether “this is another draw” is shown. Dependent variables: stateability of a rule, slope of the time curve, appearance of superstitious prompts.

A horizontal line must be split from a ceiling: an already-perfect task is also flat, and that is mastery. Here the flat still has a high error rate.

Where it stops holding

Exploratory making treats non-reproducibility as material; the learning goal is not “make that picture again,” and the curve was never meant to be falling time. Professionals who work on distributions (look at a batch, then tune) are learning tuning, not a one-shot mapping, and the curve can fall on another layer. One-off chat does not need a skill to form. This entry covers scenes where people think they are learning a repeatable operation.

Applying it

• Let at least one successful result be pinned and replayed (a snapshot, not a seed redraw), as the worked answer for homework.
• Tutorial steps should not say “you will see this sentence.” Say “you will see a class of results; here are three.”
• Split a practice mode: lower temperature or show samples side by side, so people see the range of the mapping rather than a point.
• Check: have a new user do the same class of task five times, and see whether trial 5 has less probing and a clearer “this is how I should write it” than trial 1. If not, the curve is flat. Then listen for an incantation being recited — a fake skill squeezed out by non-reproducibility.

Related

• Same group: L1.11.1 Stochastic defects will not replay · L1.11.2 Seeds do not survive version changes · L1.11.4 Trace needs input, version, and output together · L1.11.5 One draw cannot rank two systems
• Nearby: L1.01 Mismatch between stochastic output and deterministic UI · L2.03 Examples and template guidance · L2.07 Prompt history and reuse
• Search terms: flattened learning curve · unstable mental model · non-reproducible practice