On reflection, it is quite easy to see how the kind of people that would spend money on meteorite insurance would also be so willing to believe that Dümbar had arcane powers of foresight; and when his prophetic vision of meteoric annihilation was realised they soon sought him out, eager to learn of new and more improbable ways that they might all be meeting their terrible doom in the near future.
Like all good con artists, Dümbar tended to prey upon the more mentally infirm; and so it is no coincidence that the majority of the "Brethren of FaDoom" are somewhat elderly and, more often than not, raving mad. Most are former militia, veterans of countless grisly wars against the greenskins and beasts of Chaos in the vast, dense forests of the Empire. For some the war never ended, and many kept a few weapons after they were retired from frontline service just in case society crumbled.
Which is quite convenient.