When Evaluation Goes Blind

Can a learned evaluator miss a structured artifact?

Automated vision‑language evaluators rely on image encoders such as vision transformers (ViTs). We test whether the patch grid of a ViT reduces sensitivity to particular image changes. We overlay checkerboards with varying cell size, alignment, and preprocessing at controlled perturbation magnitude. Across seven ViTs, checkerboards at certain cell sizes cause smaller changes in encoder outputs than checkerboards at nearby sizes. Two convolutional controls show much weaker effects, while one aesthetic‑scoring model shows the same sensitivity difference. These findings reveal a reproducible evaluation blind spot: checkerboard perturbations applied at the same strength can produce sharply different responses depending on their scale and alignment relative to the model's patch grid.

Try the probe yourself

Add a checkerboard δ(u,v) = a·(−1)^{⌊(u+φx)/c⌋+⌊(v+φy)/c⌋} to the paper's example image (DIV2K #0898) and see how it sits on the encoder's patch grid. Wherever the paper measured that exact setting, the panel shows the measured embedding change for this image and the median over all 100 images.

The patch‑grid rule

A ViT cuts its input into non‑overlapping P×P patches. Let c_enc be the checker cell width at the encoder input. At zero offset, we predict reduced sensitivity in three cases, and use sub‑odd patterns as a control. Numbers below: how much less a predicted‑blind cell size moves the embedding than a neighbouring size (median over 7 ViTs, all 100 images, a = 0.025; sub‑odd: the 3 encoders with P = 14).

Every cell width from 1 to 112 pixels

Median relative drift across the 100 images when the checkerboard is added after resizing at zero offset. Filled blue points satisfy the patch‑grid rule; the drops line up with them. Hover for values, click a point to load it into the playground above.

What we found

Figures

Click to enlarge.

Probe your own evaluator

The probe itself is a few lines of code: add the checkerboard after the encoder's resize/crop and before channel normalization, then compare embedding drift for a predicted‑blind width against its neighbours.

Copy
import numpy as np

def checkerboard(h, w, c, a, phi=(0, 0)):
    """delta(u,v) = a * (-1)^(floor((u+phi_x)/c) + floor((v+phi_y)/c)), same for all RGB channels."""
    u = np.arange(w)[None, :] + phi[0]
    v = np.arange(h)[:, None] + phi[1]
    return a * (-1.0) ** ((u // c) + (v // c))

def perturb(x, c, a, phi=(0, 0)):
    """x: H x W x 3 image in [0,1] at the encoder input (after resize/crop, before normalization)."""
    d = checkerboard(x.shape[0], x.shape[1], c, a, phi)[..., None]
    return np.clip(x + d, 0.0, 1.0)

def rel_drift(z, z_pert, D_f):
    """z, z_pert: unit-L2 embeddings; D_f: mean cosine distance between clean images."""
    return (1.0 - (z * z_pert).sum(-1)) / D_f

def blindness_gap(d_neighbour, d_blind):
    """Positive = smaller response to the predicted-blind width; 1.0 = tenfold lower drift."""
    return np.log10(d_neighbour) - np.log10(d_blind)

# e.g. a ViT with P = 14: compare c = 14 (exact) with c = 13 and c = 15 at a = 0.025

BibTeX

Copy
@inproceedings{jung2026blind,
  title     = {When Evaluation Goes Blind: Analyzing Patch-Grid Failure Modes in Vision Transformers},
  author    = {Jung, Steffen and Keuper, Margret},
  booktitle = {NeurIPS 2026 Workshop on Trust-AI-Eval (TAE): Can We Trust AI Evaluation?},
  year      = {2026},
  url       = {https://openreview.net/forum?id=sZ1Du9f3GA}
}