I don't mind 'em, per se. All I'd say is that they should be unobtrusive, completely and totally optional, and give a reward that is useful, but not a must-have.
Story-wise, these sorts of quests can be the result of something the inquisitor takes note of but doesn't actually do anything about until such time as the player decides to act. Overhearing a conversation, etc. There would be no dialogue and nothing else to actually prompt the inquisitor to act. This way, if the player chooses to pass these quests by they'll be nothing more than something heard or witnessed in passing, but if the player decides to complete the objective there will be a reward.
Simple and out of the way, yet still present.