Physical damage is one of the weaker types of damage in the game because it's mitigated by armour, which most enemies have to a greater or lesser extent. As a mage I treat any staff with physical damage as loot. There's almost always a better alternative. On hard setting or lower I usually carry a fire staff, a frost staff and Malcolm's Honor (spirit). On nightmare I usually add a nature staff to the mix. There are too many good elemental staffs to bother with one that does physical damage.
The situation is different for accessories with +%physical damage. For all classes it will affect the damage of any spells and talents that do physical damage. For a mage that would things like Stonefist and Haemorrhage. Decide what spells you use most often and get accessories which boost the damage they do. It tells you in the description of the talent or spell. Bear in mind that some talents ignore enemy armour, such as haemorrhage so a +%physical damage item is even more useful.
Most warrior and rogue talents do physical damage, so those accessories are good for them.
Given the wide spread of types of damages that spells do, I usually opt for accessories which boost as many elemental damage types as possible. e.g. Freedom's Promise staff.
If you want to tell us more details about the build you're working on and if you're using DLC items we can give you more specific advice.