I understand the concept of subtraction not existing, but doesn't that work the other way too?

You are babbling. For most cases a number is defined as the length of a line, positive to the left and negative to the right. To add numbers you place the foot of the second line to the point of the first and then measure from the foot of the first to the point of the second. To subtract you do exactly the same except you reverse the direction of the indicated line. So I guess you could say that both operations are actually adding, but that is not a very useful observation.