Reproducing a neuro-symbolic model for Raven's Progressive Matrices

This is starter content. Replace it with your own write-up — the front-matter above is the only part the site cares about.

Raven’s Progressive Matrices are the classic test of abstract visual reasoning: you get eight cells of a 3×3 grid, each containing shapes that vary along some hidden rule, and you have to pick the ninth. Humans are good at it. Neural networks are good at it too, right up until you ask them why.

That gap is what Zhao et al. (2023) set out to close, with a three-stage pipeline that keeps an interpretable representation in the middle. Our team project in Cognitive Systems at Universität Bamberg spent March to August trying to reproduce it from the paper.

The pipeline

Three stages, each one trained separately:

1. A CNN that turns each panel into a dense feature vector.
2. A supervised VAE that compresses those features into named semantic attributes — shape type, size, colour, number — rather than an anonymous latent.
3. A cognitive map that reads the attribute grid and performs relational reasoning over it symbolically.

The second stage is the whole idea. Because the latent dimensions are supervised against known attributes, you can read the model’s intermediate state as a sentence: row 2 holds triangles, increasing in size, constant in colour.

class SemanticVAE(nn.Module):
    def __init__(self, feature_dim: int, attr_dims: dict[str, int]):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(feature_dim, 512), nn.ReLU(),
            nn.Linear(512, 256), nn.ReLU(),
        )
        self.mu = nn.Linear(256, sum(attr_dims.values()))
        self.logvar = nn.Linear(256, sum(attr_dims.values()))
        # Each attribute gets its own slice of the latent, and its own loss term.
        self.attr_dims = attr_dims

    def forward(self, x):
        h = self.encoder(x)
        mu, logvar = self.mu(h), self.logvar(h)
        z = mu + torch.randn_like(mu) * (0.5 * logvar).exp()
        return z, mu, logvar

Rebuilding the datasets first

Before any of that, we had to reproduce two dataset generators: RAVEN and RAVEN-FAIR. RAVEN has a well-documented flaw — the answer can be recovered from the candidate set alone, without looking at the matrix at all, because the correct answer is the modal value of the distractors. RAVEN-FAIR fixes the sampling.

Generating both was not optional. A number reported on RAVEN and a number reported on RAVEN-FAIR are not comparable, and papers are not always loud about which one they used.

Where the accuracy came from

We reached 94% prediction accuracy, which lands in the neighbourhood of the published figure. The honest caveats:

• Most of the lift came from the attribute supervision in stage two, not from the symbolic reasoner. An ablation with the reasoner replaced by a small MLP lost far less than we expected.
• The CNN’s hyperparameters were not fully specified in the paper. We swept them, which means our stage one is a stage one, not the stage one.
• Training three stages separately means three chances to leak information between splits. We rebuilt the split boundaries twice before we trusted the number.

What reproduction actually teaches

The paper is not wrong. It is underspecified, which is a different and much more common problem. Roughly a third of our six months went into decisions the authors presumably made in an afternoon and did not think worth writing down: normalisation constants, the exact ordering of attribute losses, whether the KL term is annealed.

If you are writing one of these papers: publish the generator seeds.