You need to first select the member from the list (which will only appear when the repository is selected), and then you can remove the member of the currently selected repository. Because the repository list can only be refreshed when you supply the Personal Access Token, obviously this means that only owner of the repository can remove such a member.