If you feel that your husband and his mother do not love your child, then it is advisable to make a decision that is in the best interest of the child.
Ask yourself these questions, what will happen to my child if I make this decision? Will my decision have a negative or positive effect on my child's life and future? Will my child stand to lose or gain if I make this decision?
If this is your first child, disclose to your partner how you feel about the way he treats your child. Maybe he is not aware that her words are causing problems.