I work in construction/with my hands, so I already use my muscles enough during the day at work. Do I have to do these physical exercises?

Yes. Quite often, in such cases, only the same few muscles are being used (e.g., a bricklayer most likely uses the same arm to lift the bricks). We want to avoid repetition and give our bodies more varied movement.