Absolutely. I wont say that Rivera didn't deserve to be in the Hall. He's probably the best closer in the history of the game. When you're the best ever at your position you should be in the Hall. But the only reason he was unanimous is because of where he played. If he'd been the Braves closer, he'd have gone into the Hall but it wouldn't be unanimous.
To illustrate, Greg Maddux had a career WAR nearly 3 times that of Rivera, had absolutely zero suspicion of steroid use, won 4 Cy Youngs, and is regarded by many as the best pure pitcher in the history of the game. He was not unanimous.
Honestly, if it wasn't for racism you'd probably have already had a unanimous selection in Aaron or Mays.