When accidentally add a reviewer to someone else's change there is no way to remove him.

In general I think "everybody" should be allowed to remove reviewers to keep noise down. E.g. if I see someone adds a "wrong" reviewer to a change in "my" component I should be able to remove him. Likewise for cases where people added half a dozen reviewers "just in case they might be interested".

The danger of mis-use should be rather limited, as the added-and-removed reviewers get notification emails anyway.