If you're a rival elf Inquisitor and you ask to join him you'' say (might not be exact) "I'd give my life to restore the elven people." then Solas replies with "Would you give the lives of everyone you've ever known?"
Yes, he does say that, but that doesn't contradict my point. He doesn't use the word "kill" or say outright "people will die", that line could also be read as "you say you would give everything to restore the elves, would you do it on the expense of the lives of others?". It's not even clear how much of it is related directly to his goal and how much is to put off the Inquisitor by serving as an example.
I'm trying to say that I think the removal of the Veil isn't like pressing an insta-kill button, If dropping the Veil means chaos (mags, titans, spirits, etc.) then a lot would die, but not as a direct consequence of the removal itself.
When asked "why does this world has to die for the elves to return?" he answers:
"A good question, but not one I will answer."
Low-approval: “You will survive this day, Inquisitor, and though I owed you an explanation, I will not give you tools to use against me.
High-approval: "You have always shown a thoughtfulness I respected. It would be too easy to tell you too much."
I don't think he is speaking about stopping the removal of the Veil, it's obvious the Inquisitor can't, Solas is a demi-god in his powers and the Inquisitor cannot oppose him directly.
It's not clear if "you will survive this day" speaks about the current day or some day in the future in which Solas sunders the Veil, but if it's the latter - it's even more telling.
If the Inquisitor knew why his plan would result in chaos they could take countermeasures to ensure the survival of most people, so he can't tell them that. I imagine that if most of the Thedosians will live through the merging of the worlds then the ancient elves would be outnumbered and wiped out in some Exalted March, thus ruining Solas' plans of restoration of his people.