Mike, I am very much like you in this regard. I've known multiple occasions where someone was panhandling for money (with the cardboard sign which said "Hungry. God Bless.") but when offered a sandwich from Subway they either refused or threw it in the trash.
But occasionally there is a special connection when you know the need is real. That's when I help out.
I've thought about what I would do if I were homeless ... it's a dismal thought of how to get back onto one's feet.
I guess we all have to use our better judgment when faced with this kind of dilemma....Not common where I live now, but when I make trips to the city, I see a lot of people under the interstate underpasses and I want to help so badly....especially when I see a disabled veteran on crutches/in a wheelchair, etc. Is it legit? Who knows....but I have to make a decision whether I want to give and risk my gift being for naught, or to not give and let it hang on my conscience....