$25 for dinner?!? You were living high on the hog!
I don't know if I would consider swimming more difficult than other sports because you're competing against your own team (or yourself) - it's just different. You would be competing against teammates to some extent, as no one is going to back off just to let a teammate win. Typically, NCAAs was the ultimate goal, though only a few had a realistic shot at making an NCAA qualifying time on our team back then. Olympic Trials was actually easier to qualify for (and still is) as compared to NCAAs. But that's because the Olympic Trials might have 20 heats of an event, while NCAA typically only has ~5. (In 2016, 3,000 swimmers competed at the US Olympic Trials!)
Also, a lot of the time you knew going in whether you could realistically beat the people you were racing against. Same with the other team.
For example, if we were swimming IU or Michigan in a dual meet, we generally knew they were going to beat us. Same thing in reverse when we swam a MAC school, or some other schools. The only time that wasn't necessarily true is if the other team was rested and shaved just to try to beat us in their own pool. It was crazy that teams would do that mid-season, but some did. (We never rested significantly mid-season just for one meet, though we might have a couple light days before a specific meet just to try to put up some decent times - often as a mental reward for all of the work we hade done.)
I'll never forget swimming at Illinois State one year. We had a brutal week of practice leading up to it, and drove to ISU that morning only to find a team that had rested and shaved down just for us. I was supposed to swim a race I did not normally swim in a dual meet, but coach came up and told me he had to move me back to my usual to make sure we did not somehow lose to ISU. I got up on the block for my event (200 fly) and they introduce the guy next to me as the national champion and Olympian from Ecuador in that event. I looked at the guy and then at my coach who just shrugged his shoulders and mouthed "kick his ass." I think I beat him by half a pool length.