A standard response to term-limits advocates in America is that the ballot box already provides the means to remove elected officials. If you dislike your congressman, then you do have a say. But what about the other 434? You have no say in their districts. Dislike Nancy Pelosi or Eric Cantor? Too bad, you don't get to vote against them even though make policy that directly affects you. What if 434 members decided to gang up on the 1 other member and split the pork? Or 433:2, or 432:3, or 218:217? So much for the ballot box defense.
When it comes to the presidency or the governorship, we do all get to have our say. The ballot box would seem to be a viable option for removing these types of office-holder. Strangely though, these offices do have term limits.
Why is it seemingly backwards? There's probably a Federalist Paper on it.