I would hope that stealing someone's currently equipped weapon would be extremely difficult. You should fail most times you try it.
Would you consider a sheathed weapon still equipped?
And I'm just visualizing this... would pulling a sword out of a city guard's backpack (or wherever NPCs store items) be done any more easily than stealing it out of their sheath?
I've often disliked pickpocketing in games, where it turns an NPC into a mobile chest. I always thought there should be two separate skills - appraisal (whereby you watch and evaluate a target, using a skill roll to determine what, if anything, of value they might be carrying) and pickpocketing (where you basically reach into someone's pocket and pull out something - random in nature in most cases or, if you are looking for something particular, a much harder "called" skill roll that would need to be used with the appraisal skill beforehand to target what you want).
Of course, such a system would be unwieldy for a party based game, or a game where combat is easy, which would make all the sneaking and avoidance merely tedious for the majority of players.