What if the kick ability is only available if you run out of time?
Or if the vote will be only about you ending your turn? And if you run out of times too many times, the kick vote will be available.
Anything can and will be abused. Even the current kicktimer is used to annoy other players. Kick vote would at least allow you to kick the players who are being annoying. And the kick votes can also be used against the ones who voted unfairly - if the admin sees that you were already ahead and were kicked, the other players would be punished.