About the lyrium/red lyrium:
"blue-green crystal [...] Once refined - that is, dissolved in a liquid or heated until it crumbles into a fine powder - lyrium is safe for all to use."
"If living beings are exposed to lyrium in its raw form, it will slowly drive them mad. It can also cause physical calamities, such as deafness, memory loss, and nausea."
Chalk or marble consist mainly of calcium carbonate, which can be dissolved in acid or (acidulated) water.
CaCO3 + 2 HCl
CaCl2 + H2O + CO2
So the result is water, calcium chloride, and carbon dioxide. The latter will vanish into the air, heat the water and it will evaporate, and all you have left is the salt. of course, this example is bad because it's not toxic and all, but just to point out that heat or dissolving can decompose a mineral and if the toxic parts of it are "lost" (gas) or disposed of, the rest might be safe to consume.
Of course I cannot name a real mineral to compare the effects (mental sickness etc.). I assume there is nothing quite like this in our world.
Radiation is a compelling idea but the effects are different from those of lyrium. The effects mentioned sound rather like brain cell decay due to intoxication. Hence, maybe lyrium ejects something toxic... gas? Like a constant... very slow... transformation of its aggregate state due to... air pressure, like sedimentation, only stronger. I have no idea.