Am i right to stay with him despite tainted rumors of him cheating on me?

I have been dating him for a year and a half. We are so opposite but work awesome together. First time i really feel love run through me. His good friend is so into me and is telling me that people close to my man are saying stuff. Like he is getting it on with his ex. His friend even says my man tells him stuff. That i should leave him. I've always had trust issues but that's just because of past relationships. I really want an honest relationship. I've asked him, and i feel like i hurt him every time. He honestly has said no every time. I know he hangs with her and his friends, but that's just because they share the same circle of friends. We have taken a little break but still talk about our future. We always come full circle back together. I know that the friend most defiantly could be lying to me just to get with me. And since my break he is all over me. I love my man so much. Should i just ignore the crap his friend says and give my guy the chance he deserves? I know doubt is an indicator. But that was put in my head.