Threatening To Demolish Our Property, Which In Our Possession Since 50 Yrs

There was a plot of 22 decimal, in which my father got ownership of 20 decimal. 2 Dec. plot couldn't be written to anyone(might by mistake). But still in our possession since then. In land survey of Bihar, (approx in 1973), a separate plot of 18 decimal carved out from 20 decimal-(say plot -A(our possesion)). Remaining 2 decimal (out of 20) and other 2 decimal (that couldn't be written) got merged to a different plot (say it plot B). Plot B doesn't even belong to any of my ancestor even. It belongs to a person of different caste. Plot B is of 10 decimal total. 4 decimal in our possesion and 6 decimal in owner of B possession. Since death of my father in 2010, he started claiming this 4 decimal land, and now in 2021 he wants to demolish. My Question is - 1. Is it possible to recover all 4 decimal that has been merged to plot B 2.Can he demolish it legally? What action should I take on owner of land B, if he demolishes my structures in 4 decimals that is in our possession since 50 years?