Is There Any Law Against Verbal Bullying

My neighbour is a bad guy. He is 33 years old and much stronger than me. In front whole neighbourhood he will say that he and his friends will have sex with my wife. Even though I am not married, and he might be joking and won't do it in reality, but I don't like such statements and want to beat him, but as he is stronger than me I can't. He also has the support of some violent men from slums. This is mentally affecting me that I am not able to take any action for insulting me in front of everybody. I don't have financial problems. I don't think the police will take any action for this as compared to other cases this is minor. Instead, I will become a laughing stock, and this will further affect my mental distress. He tells other boys that "He can't produce a biological child, and we must fuck his wife if he gets married and produces babies" If I say anything then his friends will threaten me that Ïf you abuse our friend we will beat you with sticks". They want me to tolerate his abusive words.