I agree 100%.
I look at it like this: Defensive value has many external factors that play into the valuation that aren't necessarily accounted for. For example, defensive value certainly is affected by the play of other players on the field. Take Greg Maddux for example - heavy ground ball pitcher (rare rf relevance), significant K pitcher (no rf significance), got tons of ball in play back up the middle (no rf relevance).
Also, the other team has an effect on defensive relevance for a player. What if the other team is full of RH pull bats? The rf ends up standing around.
Also, the defensive coaches play a part. What if Heyward is positioned wrong where he has to run, successfully, to make a play? That makes the play better than it should have been. What if the coaches always have the rf positioned right where he should be? Then his opportunities to provide above average impact are diminished.
Also, how does defense translate over time? Andruw Jones was great early but all the belly flops (sometimes unnecessarily) over time eroded his ability (and affected his bat).
The value of defense is relative. It's good to have it but it doesn't always have the opportunity to be used. Whereas hitting ability is always used, every at bat.
But most of this could also be used to describe offense. You're arguing that on a game-by-game basis, defensive value can be negated somewhat. Well, if a really good hitter who is merely decent against lefties faces a LHP, then his offensive value is diminished. Teams can shift defensively to negate a pull hitter's value, they can pitch around them, the manager may call for a bunt or hit and run, etc. All these same arguments can go the other way, too.
I would argue defensive value is actually less volatile than offensive value for most players, and offense declines with age just as defense does.
The advanced defensive metrics try to measure the value the player is actually providing, given all the variables you mentioned, not just a theoretical value under certain circumstances.
Also, most metrics are based on certain fielding zones, so positioning matters, but in the opposite way you think. A player positioning himself well will be valued higher because he will get to more balls; a player who consistently positions himself poorly will be hurt because he won't. Running further because you're out of position doesn't help you in these metrics.