The Esterly House, located on 604 W. Main Street, was built by George Esterly in the mid-1800's. Esterly was known for his farming inventions, reknown at the Chicago World Fair for his contributions to the agriculture community. When Esterly first moved to Whitewater he owned over 1100 acres of land, primarily used for farming. In home directly in back of 604 W. Main St. was the Esterly's servant's house.

The Esterly House is now the home to Delta Zeta sorority, formerly Delta Sigma Epsilon, where it was purchased from the Phi Sigma Kappa fraternity in the 1970's.