The fifth honesty check is whether the conclusion survives being redrawn a different, reasonable way

What it is

The fifth honesty check is rendering robustness: under another equally reasonable rendering (a different window start, different grouping, different chart type, different axis scale), does the same conclusion still hold? This check directly tests the hidden misleading where "the conclusion depends on presentation choices"—if a conclusion holds only under one particular rendering, it is more likely a product of the rendering than a property of the data.

Why it happens

The rendering-sensitivity check consolidates robustness ideas scattered across earlier knowledge (window-shift tests, binning sensitivity, aspect-ratio tests) into one general action: identify the presentation choices the conclusion depends on (origin, binning, ratio, chart type), substitute each with an equally reasonable alternative, and verify whether the conclusion's direction stays stable. Its power lies in separating two degrees of freedom: the data's ability to determine the conclusion versus the chart maker's choices' ability to determine it—the former is the signal's robustness, the latter is room for manipulation. When the conclusion is stable across reasonable renderings, the manipulation room is proven near zero, and the chart earns credibility in its strongest form; when the conclusion flips with the rendering, the flip itself becomes information that must be disclosed ("the conclusion depends on a window starting in March"), or the conclusion must be downgraded. This check is also the meta-check of the previous three: charts can pass the axis, scope, and annotation checks yet still have conclusions that depend on rendering—only this test catches that.

Where it stops holding

Defining "reasonable rendering" is the check's gray zone: substitutes must be unbiased, conventional options (another standard bin width, a standard window length), not renderings specially constructed to flip the conclusion—flipping via an unreasonable rendering does not indict the original conclusion. The check's cost should also scale with the conclusion's importance: core business conclusions warrant the full substitution matrix (origin × binning × chart type); routine charts need one or two key substitutions. When the conclusion flips under some substitutes, the treatment is not simple abandonment but reporting the flip's boundary conditions ("the 3-month window shows a rise; the 3-year window shows flat"), presenting the instability as part of the conclusion.

Applying it

• Check action: list the 2-3 presentation choices the conclusion depends on, substitute each with a reasonable alternative, redraw, and verify direction stability; record any flip as "conclusion depends on rendering" and disclose it on the chart.
• Run the full substitution matrix for seasonal reports on core metrics; routine charts need at least a window-shift test.
• Rewrite flipped conclusions in boundary-condition form ("short-window and long-window conclusions differ"), never hidden.
• Verification: check whether the charts from the last important presentation carry a rendering-sensitivity record; no record means this check was not run.

Related

• Same group: U10.04.1 Check whether axes start at zero or are truncated or broken · U10.04.2 Check whether data range, filter conditions, and exclusions are stated · U10.04.3 Check whether title and annotation claims are supported by the graphic itself · U10.04.4 Check whether uncertainty, missing proportion, and sample size are labeled · U10.04.6 The checklist runs before publication, not after being challenged
• Nearby: U10.02.1 The chosen time window decides the trend's direction · U8.06.2 The choice of grouping can flip a conclusion's direction
• Search terms: robustness check · sensitivity analysis · alternative rendering