'We had to decide whether to go there or not considering the fact it was late already'
Is it grammatically correct?