You make my point for me with your very first sentence...
Just because you don't find it offensive, doesn't mean that's the same for everybody.
I'm fat and I don't mind fat jokes, I'm one of the worst culprits for them, does that mean we can all go around body shaming people?
Stereotyping of any sort is generally best avoided these days for this very reason. Do I agree with that? Not always, but that's the world we find ourselves in, and it's probably a good idea we adapt whether we agree fully or not.
There's a whole deeper discussion to be had on cancel culture and wokism in general, but a rollercoaster forum is not the place for it for one thing, and for another, that discussion can be far too divisive, so I'm keeping my opinions on it to myself for the most part.
My original point was simple, but I'll reiterate it anyway. With cancel culture being what it is, and the controversies he's already (rightly or wrongly) been linked with, I doubt very much that him saying a few things to a colleague is going to make any difference at all.