Single mom needs advice :(?

I am a single mom of a 1st grade boy. His father and I divorced when he was a year old and have had quite a tumultouse relationship since. Tried to repair it several times but to no evail. My son has spent a little more time with me but spends quite a bit of time with his father as well. I have over compensated (as Im sure most of us do) in the parenting department for my son since day 1 of the seperation, I work only the hours that he is in school so Im able to be there before and after for him. I have lived consistantly in the same home (just him and I) for 6 years and recently moved us into a larger more suitable one, that he loves. However, in the last year or so he has slowly been clinging to dad more and more. Tonight I went to pick him up from my ex husbands to help him out in a jam and when my son saw me he rudely said "What are you doing here?" So I just gave him a hug and a kiss and left. I know Im the adult here but I am crushed. Any advice?