I can totally agree with Stephen, it might feel like that sometime.
Having networked extensively (in person) across many industries and countries worldwide, IMHO, my observation is this:
*** We cannot be friends with nor agree with everyone! ***
That is why not it is the perception that I network a lot, but in fact, I try my best NOT to meet people or put it another way, ONLY meet with quality people when I can.
As it is NOT pleasant to say no or not to follow up.
It is therefore, my rule to try to understand who someone else is before I approach/get introduced to them and try to see if we connect/understand each other.
However, being a celebrity is a totally different beast, plus having a 'direct' connection/contact with everyone is sadly not physically possible, especially if like Stephens you have close to million followers!
Please do not forget, you also have some normal, respectful and friendly folks (like me with our limited no. of DMs between us.)
Of course it is your decision, but my suggestion is maybe you adopt the strategy of the 'good guy', namely treat people good when they are good to you, and ignore those bad guys until they apologise.. some people might want in their little lives their 'purpose', which might just be trying to be the one to put off a celebrity from using twitter.
I have observed that in various circles..
Our lives' decision should not be decided by a small/minority.. Twitter's transparency does not help. but that is the reason why my personal tweets are private. ;-)
Stephen is very brave.. it is your decision to decide whether to invest further for the rest of the almost million followers of yours (me included!) whatever your choice, we will support you!