How do you say, "No, thanks!"?
While trying to be an "unobtrusive and anonymous witness", as Jorge Torralba has put it, you sometimes get seen. No matter how small and "non-professional" your camera looks, that it is not shiny and new, that you only hold it to your eye at the last moment matters little. Sometimes someone sees you and decides to pose. I'm sure there is a genre here that could yield wonderful photos, but it is not what you want. As someone trying to engage in street photography, you want to say, "No, thanks!", but then you look again and take the shot. It is still a moment, it is just not the moment you wanted.