What does this imply: 'He that in a neat house will dwell, must priest and pigeon thence expel'?

If you want a house that is 'neat' (OE, orderly, precise, exact in form and manner) then you need to remove religion (the priest) and the *cough* lower orders (the pidgeons) from it. The word 'house' is context means not the physical house that you live in but the 'house' that represents your life and companions. In other words, remove religion and the lower classes from your life in order to live in a comfortable and hassle free manner.