I think that sympathy for white supremacism and general white grievance provides a substantial base of support for Trump, and Trumpism. You're right that the raw numbers of outright white nationalists are small, but in the Trump era they have been given a platform and encouraged to a degree that I have never personally observed before in national politics. Charlottesville was an instructive example of this, and it's rather cowardly IMO to avoid this.
You are very dismissive of the Breitbart/white supremacist connection, but that BuzzFeed piece laid it out in pretty stark detail. The same crew that ran the Trump campaign and got a couple of seats on the National Security Council was consulting with Nazis on how to mainstream their ideology. That's worth worrying about it, isn't it?