HP only means how much torque you're making at the upper part of your power band.

Sounds vague, eh? That's because it is.

What makes a car fast is how broad the torque curve is, especially on the top end.
Most cars, especially stock-ish domestic engines, drop off sharply on the upper end.