How do i get my horse to trust.?

I have a 3 year old paint gelding. I bought him after his owner died, but his owner had abused him when he was younger. Now he doesn't trust people. When i got him we had to fool him on the trailer with food, he didn't have a halter or anything on him. It took me about a week just to get a halter on him after we put him in a small pasture, even them the only way i got it on him was using food. Now he trusts me but no one else i can do just about anything with him. But i want to be able to ride him but he is so scared of everyone else that no one can help me break him. He is epically scared of men, im guessing bc a man abused him. How do i get him to trust guys??? A man cant even walk by him in the pasture without him stiffening up and stuff. And if i have him tied up and a man walks up he starts shaking. If anyone has any ideas of how to get him to trust guys please tell (: thanks