The short answer is both, and which one dominates depends on the layer, the training data and how the router was regularised — which is why the results look messy.
Early layers tend to specialise in shallow, surface features: particular token types, punctuation, common phrases or a language. That is specialisation, but not the kind of clean semantic specialisation the name suggests. Later layers show more task-like grouping, for example experts that activate more often on code, on arithmetic or on a particular domain, but even there the boundaries are fuzzy. An expert that looks like a code expert will still fire on ordinary text that shares the same surface patterns, so counting activations alone can overstate how specialised it is.
Memorisation is a real risk, especially when the router is trained without a load-balancing loss or with too many experts for the data. In that case a few experts can become specialists in frequent training examples rather than in a general skill, and they generalise poorly when the distribution shifts. A useful diagnostic is to test routing stability: feed paraphrases of the same input, or the same task with different surface wording, and see whether the same experts are chosen. If the routing changes a lot while the meaning stays the same, the router is keying on surface form, not on the task.
For design, I would treat specialisation as something to measure, not assume. Report per-expert activation by task and by paraphrase, check what happens when an expert is disabled, and use a load-balancing loss that keeps experts used without forcing them to be identical. The practical payoff is not a tidy taxonomy of experts, but a model whose routing is stable enough that removing or replacing one expert has a predictable effect.