My children are fearful of visiting with their dad

My ex husband was awarded visitation with our two boys every other weekend, however, he has yet to even pick them up himself. He always has an excuse as to why he has his mother do it. First he didn't even really have one and now his excuse is work (sometimes he does work over but not every weekend...and I know this because he works at the same factory my mother does). Just yesterday (Friday), I was told by my ex on Thurs. that his 70-something year old grandmother would be picking up the boys since his mother was in the Bahamas on a cruise. I didn't like that idea because his grandmother is NOT in good health and has a history of passing out a lot (she lives alone). Since the boys returned home from their last visit with their biological dad, they have been BEGGING me not to make them go back over there. I've tried talking to them about it but at ages 3 and 5, I haven't really gotten too much of an answer other than them saying "I'm scared of my dad" etc. I ask they why they are scared of him but they act like they are going to be hurt or something for telling me. This has had me concerned so I tried talking with my ex about it and of course, he blew up about it and turned the conversation around to cussing me out, calling me names, saying I was trying to just stir things up, I was lying, etc. I go through this every time I ask him even simple questions such as "Would you like to come to see the boys in their play at church?". My 3 year old screamed, kicked, and fought getting out of my car when I dropped them off yesterday. He was screaming at my ex's grandmother that he didn't want to go. And when my dad's wife got out of her vehicle and walked over, it got worse. My son ended up throwing up all over me from being so upset. My ex's grandmother said "Just throw him in my car and he will shut up". I have not taken my children around my dad or his wife for over 2 years now because I caught my dad smoking pot in front of my boys. Now, my dad and his wife have become friends with my ex husband (who they hated so much that they let me live with them when my ex first kicked us out of our home). Anyway, I knew that my dad's wife was going to end up keeping them and not my ex's grandmother so I drove around the block and then waited a few minutes to drive by the grandmother's home and sure enough, there was my dad's wife leaving with my boys. My ex had to work today as well so he has not seen the kids as of yet this weekend. This is typically how it goes when his mother picks them up. She has admitted to me that he does not come see them except an hour or two on Sunday. My divorce papers say that the drop off is between ME and their father. Not me and his mom, me and his girlfriend, etc. What are my options here? I desperately want to find out why my children are fearful of seeing their dad. They are okay as long as his mom is picking them up but literally freak out (in a bad way) when they see him.