We show that Bayesian and MDL inference can be statistically inconsistent under misspecification: for any $a > 0$, there exists a distribution $P$, a set of distributions (model) $M$, and a 'reasonable' prior on $M$ such that
(a) $P$ is not in $M$ (the model is wrong)
(b) There is a distribution $P' \in M$ with KL-divergence $D(P \| P') =a$,
yet, if data are i.i.d. according to $P$, then the Bayesian posterior concentrates on an (ever-changing) set of distributions that all have KL-divergence to $P$ much larger than $a$. If the posterior is used for classification purposes, it can even perform worse than random guessing.

The result is fundamentally different from existing Bayesian inconsistency results due to Diaconis, Freedman and Barron, in that we can choose the model $M$ to be only countably large; if $M$ were well-specified (`true'), then by Doob's theorem this would immediately imply consistency.

Joint work with John Langford of the Toyota Technological Institute, Chicago.