I actually agree, but with three little kids and a shit work schedule (maybe I'll follow your lead eventually), it makes it nearly impossible to keep up with.
I use to be the same way, having 3 girls of my own.
I can't always do it all at once. A hour here and then a finish the rest later. All together including the field, it'll take just under 3 hours, so it's a lot to do.
Just any time away to myself is greatly appreciated, plus I get a little pleasure making the yard look nice.
Also, gardening is very relaxing. We've got a little strawberry patch we planted last year that is doing well that is nice to go back and pick. We've also planted tomatoes, cucumbers and rhubarb. When we first moved out here we did a big 30×30 space. That was super nice to go out in the morning and water and see what was coming up.
I was a full blown city boy turned country bumpkin now.