Detective investigator for NYC here. I am sorry that man gave you the willies but it's always better to be safe than sorry and always follow your instincts. I wouldn't lose sleep over him but I'd like to point out some general tips for everyone here.
1. Never go anywhere without telling anyone where you are going. The more who know the better.
2. If at all possible try to use the buddy system. More is better than one.
3. Always keep your doors and windows locked, even if you live in a "safe" neighborhood. It is always better to be safe than sorry. Remember it only takes one time for something to go wrong.
4. Also, as a general side note try not to get picked up in a car if you think you are being watched. I'm not sure where you are from, but here in NY anyone with a plate number and $50 can get anyone's car registration information (name, address and date of birth) from the DMV.
5. If at any time you feel unsafe get to a well populated area and call whoever you know that is able to come and meet you right away.
If you have any questions feel free to ask.