I agree. But certain XMen were part of the Avengers at various times, Wolverine, Storm, Rogue, Havoc, Sabertooth, Deadpool, Cable, and a few others I'm forgetting. Of course those characters ties to the Avengers are not even remotely as strong as Quicksilver and Scarlet Witch, I'm not gonna pretend that's the case. I'm just pointing out they likely had an option to go with them and took it, but I think they killed Quicksilver to play nice. FWIW I think that Aaron Johnson's Quicksilver is the much cooler one.
Unrelated, I think that FOx only holds Fantastic 4 because they don't want Marvel to get it and do something cool with it. While I think there's limitations to the characters which has been a struggle, I think beyond that they don't want to see Galactus and/or Doctor Doom as the next big Marvel baddie. They're sitting on a Gold Mine of villains.
Fun fact. Fox doesn't hold the movie rights to F4. A company called Constantin Films do out of Germany. They have partnered with Fox to allow them to produce the movie while Constantin distributes it. Now there are rumors of Constantin killing their films division and concentrating on other things. This was 6 months ago and if that were to happen I'm not sure if the rights would go back to Marvel or what.