My problem with mail in votes is ballot harvesting. That kind of stuff has the potential to create a new age of political machines. If you have a person who has laid the groundwork of ensuring a large group of people are getting ballots, are helping them fill them out, and are returning the ballots, you have the makings of a political boss.
When voting in person, no one is allowed in the booth with you. There's nothing to prevent a political agent from going into homes and ensuring people fill out mail in ballots how they're supposed to.
If you're going to have large scale mail in voting, you need to have strict laws about filling ballots out and delivering them and organizations that try to get people to vote by mail need to be strictly monitored to ensure they're not violating any laws.