The 17th Amendment enshrined the concept of popular election of Senators in the Constitution. Sure, the Senate probably has the right to expel him for whatever reasons they desire*, but if those reasons are things that the voters were already aware of, they would ultimately simply be overturning an election because they disagree with what the voters chose. It doesn't violate the text of the 17th, but it certainly violates the spirit of it.Why? I don't think it's ridiculous. Just because voters may look past illegality -- not to mention moral turpitude of some of the worst kind -- doesn't bind the Senate or House from doing so.
It would be different, obviously, if he were elected, and then all this stuff came out after the fact. Then the Senate could rest on the argument that the voters were essentially defrauded. But as it stands now, any argument they have for expelling him essentially boils down to "We know better than the voters do." That's pretty bad politics.
*I only say "probably" to account for insane hypotheticals in which the Senate doesn't act in good faith. For example, if a supermajority ever expelled all opposition for the express purpose of creating a one-party chamber, I have no doubt the courts would find some way to say that violates the Constitution. Obviously, the insane hypotheticals I'm thinking of are things that would never happen in real life.