How is the arm rated? (Real question.) And the other question, the one I asked thewupk - how does this measure/reward/not a fielder who is superior at positioning himself? I'm thinking about Cal Ripken on that comment, as it was always said that his range was a little short but because he knew what he was doing, he was always in the right place.
Good questions.
Also, how often does the player in question throw to the right man (cut off as opposed to trying to gun a guy down at home to show off his arm). Is his arm accurate, not just strong? How fast does he get the ball out on it's way to hopefully the right man (ie. does he need to "load up" to flash his arm)?
The positioning question is also hard to quantify. Does the guy play short and run back or stay back and come forward - would have a significant effect on his momentum and acceleration going to a ball. Even positioning of the umpires could have an effect on limited plays.
As is tendencies of a pitching staff or particular pitcher (Greg Maddux for instance almost never had balls hit down the line, either side, so OF would "cheat" or lean to the inside providing increased momentum on going after a play that way (if they didn't get lazy because of Andruw). Also, the number of chances would vary which shouldn't theoretically effect an individual outfield performance but would due to boredom and decreased attention over the progress of a game which at minimum would slow reaction time and decrease range and possibly catch efficiency.
Also, park effects like size and configuration (Tal's hill), number of day games played, ability to pick a ball up quickly coming from home plate, air quality (Candlestick Park, Coors, etc), quality of the outfield surface, grass height, grass or turf composition.
And then you have the fan effect. Batteries being thrown, trash, insults and beer, souvenir hunters fighting for a ball.
Lots and lots to consider.