I hate that word now. 'privilege'. Why can't a person be judged as a person and not by categorizing them into groups like gender, race, sexual orientation, or religion?
Privilege isn't about judgement. Privilege is about categorisation, and what happens because of it.
It's about recognising that the groups we fall into, including the characteristics we have no choice about, may give us advantages or disadvantages in any given situation. It doesn't make us better or worse people--if someone throws the word 'privilege' at you like an accusation, they are doing it wrong--but it might mean that the world treats us differently, and either gives us things or takes away things based on a bunch of stuff none of us can control, or that not everyone can have.
When you recognise that being Caucasian looking in Australia means that the police will treat you differently, you understand more about the world, and you understand that you can't really know what it's like to be, say, African looking and have to deal with the police. All you can do is listen to people who are African (we have a lot of Sudanese refugees where I live, and there's a growing body of evidence that the police are discriminating against them) when they talk about their experiences, and act as an ally to them.
It's an ongoing revelation; once you start to think about how being your particular ethnicity, your gender, having your level of education, your status as able-bodied (or not) and so on gives you advantages (once, in other words, you start to 'check' your privilege), you start to see all sorts of new ways to understand justice and injustice, how you react to people, how they react to you, where the gaps in your knowledge are. It's useful.