The only minor gripe I have with it is how you define "privilege".
You seem to equate it with comfort: it is not.
The latin roots of the word suggest the correct meaning which is: an advantage over others that is unearned, unjustified, and obtained in virtue of one's birth or position. Privilegium, in Roman law, was a special provision by which the Roman emperor conferred on some individual more rights than other citizens.
It is indeed - in my opinion - not only "a bad word", as you call it. It is everything that is wrong with our societies...