It turns out that the Pansy has an enemy. Mine were starting to look a bit pale. On closer inspection, their leaves and flowers had small holes in them. On even closer inspection, spiky brown caterpillars were gnawing at said holes. And so, I googled these pests. Turns out they're called "Pansy Caterpillars". Funny, that's what I would have named them.