Casually wear my softball pants to school?

That depends on the school and if your school has a strict uniform policy and such, if not, I don't see why not. Softball pants seem casual enough.