We got our boat when my 3 kids were age's 4-6. Kept it for almost 8 years and re-sold it for $1k less than we we paid for it. Used it at least every other weekend and often times both Saturday and Sunday...was also always fun to take a random "Tuesday" off and head to the otherwise deserted lake and play. Great memories that you couldn't pay me enough to give up. Sold the boat once the kids were heavier into travel sports (I hate those) and everyone was going different directions. I'm having a pool put into our new house as I type this (literally looking at a $50K hole in my back yard) and hoping that it gives us a lot of those same memories as our kids get older and possibly even when/if grandchildren come along.
A few things that worked for us on the boat front.
1. Buy a model (pay attention to the motor) that has a history of reliability. I did this and had a friend that knew boats well really go over it and we never had maintenance issues. Our line of motor had also been around for a while and was proven...which also would have made most repairs very cheap.
2. Make sure you've got a relatively close/convenient place to get to. It was nice just loading up on a whim and being on the water in 20 minutes. Short pull back home and good to go.
3. Know what you want to do with it and get the right type of boat. If you really want to fish...get a fishing boat. Don't think a random open bow will be fine, you'll end up regretting it. While I've never had one, don't totally put off a pontoon/tritoon. Those have gotten really nice and are versatile.
4. Unless you're really well off, I would skip a new boat...to say they're marked up would be an understatement (I once was offered a new cabin cruiser that had been priced at $90k for $55k out the door). With that said, just used a buddies $150k+ new Nautique a couple of weeks ago and it was amazing.
5. When in doubt, buy bigger. I could be wrong on this one, but ours was a 21' boat and I thought would be more than enough boat and it was the first couple of years. Once kids get bigger, more people liked going with us etc, I was always starting to look at bigger and bigger boats.
Have fun with whatever you choose...and post pics of some of the lake "scenery" you come across. That was also a very pleasant bonus to our weekends out!