Going from your cable/DSL ISP to your house and then back to the ISP adds quite a bit of lag as well, compared to a server sitting in a DC somewhere.

I've made a beefy VM or even used a spare server at the office on a 100/10mb pipe and a $5 VPS is much more responsive. Especially noticeable with things like an IRC shell.