There is this quaint town called Columbus Texas - located right on the Colorado River.

They even have horse and buggy rides, walking tours, and lots of registered historical sites.