Legal Action Against Informal Debt

My father has an informal(taken from some villagers without any legal procedure) debt of about 1 crore and my family did not know about this before and also we do not know about what my father did with that much amount. And my father is saying that this all happened due to wrong calculation with many peoples in past. And after knowing about the debt We transferred the property of my father to us(me and my brother). We decided to pay this all debt after some years by selling some property and by job income (I am MBBS final year student and my brother aspiring for a Police job), as that property is our only source of income for now. but that villagers are pressurizing us now to sell the property and pay their debt. They are regularly harassing us, defaming us by spreading any type of rumors against us. Now sir can we take any type of legal action against them. and if yes then what will be the procedure.