I think you could argue the semantics around if your first example is 'racism' or not- I believe the dictionary definition of racism includes a line about racism typically having to involve discrimination against minority or marginalized populations, which would be the major difference between your example and this club.
Where systemic racism comes in is recognizing that discrimination has existed in the justice system, healthcare, housing, employment, political power, education, etc. for a very long time. You can't just say 'we aren't racist anymore' and fix those things. as we've seen time and time again in this country. You have to do something to even out the pipeline if you ever want to actually solve the problem. You see this in places like NFL head coaching jobs, Fortune 500 CEO roles, Supreme Court justices, etc. where there are very few minorities in high-ranking roles in part because there are very few minorities in the pipeline, and its tough to fix one of those problems without fixing the other. (I would suggest this largely applies to both women and racial minorities, btw, with women being a couple of steps ahead as you would expect based on historical timelines)
A real-life example for you: My mother-in-law was one of the first female neurosurgeons in the country. She was the only female neurosurgeon in her state, and she was routinely harassed in the workplace by her male counterparts- kicked under the table if she tried to voice an opinion, locked out of the room where most of her coworkers ate lunch, etc. She often had women who would consider a career in neurosurgery only to choose another path, and the most common reason that they gave was that they felt the field was not welcoming to women- they rarely if ever saw women in high ranking roles, often saw issues of harassment, and felt they wouldn't have the same opportunities for mentorship and growth that they might in another field. Just saying 'women can be neurosurgeons' isn't enough to overcome the realities of the old boys club- another variation of discrimination that is systemically ingrained- you have to actually do the work to make them feel welcome.
In this particular case, I can imagine that it would be incredibly uncomfortable for a black person to join a club full of white people that until recently all agreed to explicitly prohibit their membership. However, I would imagine that if that club, for example, waived the membership fees for a group of black members, welcomed them into the club, and brought in some new speakers based on their feedback and interests, you might over time start to see additional black members feel comfortable joining at full price. For Whitehouse to just shrug it off as 'this is the way things have always been so it's fine' would suggest to me that he is either ignorant, doesn't believe it's a problem, or both.