Confused.... What should I do???

so this is the story.... at the end of the last year my mother had a accident and was very ill. I quit my job and focused on her and getting her to and from rehab and making sure that she was ok. She was in the hospital for 2 months and in rehab for 3. During the months that my mother was in the hospital my husband was not there for me at all. I mean he showed up maybe six times. I would leave my house in the morning and return at night from taking care of my mother. He even had the nerve to call me several times while I was at the hospital to ask me if I was coming home early because if i wasn't he was going to hang out with his friends. I feel that I should mention that up until my mothers accident we were perfect, in  the 11 years that we have been together (4 married and no children) we have had maybe 4 arguments. We did everything together.

I left him in May and I thought that we were taking the time to work on each other and our marriage, but it turns out that two months into our separation he had an affair with a coworker. I found out because I stopped by our home and found all of our pictures facing down and a card that she had sent him telling him how much she LOVED him and was glad that he was in her life and talked about she couldn't wait to be together for ever.

That has been about a month ago and we talked about it and he swears that he has ended things with her. I honestly believe that he has. We have started to date again and I thought that things were going great, but this weekend he called me and told me that we needed to talk. I had a bad feeling about things and I told him that that would see him at home like we had agree and we could talk then. He told me that it was easier if we did it over the phone. I told him no, that if it was going to be bad news the least he could do was tell me to my face and then I hung up. He text message me within a couple of minute saying..."I don't want to work on it anymore. Don't come by. I am sorry."   Can you guys believe it?  The jerk wanted to end things over a txtmsg. I ended up showing up to our home, but he was not there. I called him and he would not answer my phone calls . We finally agreed to meet the following day (through text msg.) We did and he told me the same thing that he has been telling me, that he is afraid that in a year or so i will change my mind and leave him or that I will not be able to forgive him and it will not work out. He says that he loves me and I love him and I am willing to work through it and do what ever it takes, I don't know how to help him let go of the guilt. I feel that he should be the one trying to make things better for me and not the other way around. I am so confused because just last weekend we had lunch with his family and things seemed so much better, almost normal and then he does this. We agreed to a month off and reevaluate things at the end of the month and see where we are. Should I just give up or should I give him the time he needs. Everything inside of me tells me not to give up on him and I know that we can work it out even his family tells me that he loves me and not to give up on him. What should I do? Has anyone dealt with a partner that keeps changing their mind?