When you click it, collaborator icons there are still visible. The button then says “Show” instead of “Hide”, but nothing has actually changed. Bug.
Agreed, this is really annoying. I don’t want to see any activity of my coworkers unless they’re working on the same page as I am. Please fix this bug.