Why does the brake pedal move side to side?

Answer 1

The reason your bake pedal may be moving side to side is because they are not properly aligned. Do not drive a vehicle with this problem, as your brakes may give out.

+++

Either the pedal-arm has a lot of free-play axially on the shaft on which it pivots, or the pivot (s) is (are) worn. The latter would make the pedal move in an arc.

It won't make the brakes "give out" unless the pedal components are worn to the point of risking snapping under hard braking, or if the pedal is so closely linked to the master-cylinder that the play puts an unfair sideways strain on its piston-rod..