One of the entrepreneurs who fashioned a fortune from milling, banking and speculative ventures in Rochester was James Livingston, a descendant of an old Hudson River family. In 1827, Livingston built one of the first grand mansions in Rochester's Third Ward, soon to be full of other columned monuments to their newly wealthy owners. In 1835, the house was sold to businessman Joseph Strong, who, three years later, sold it for $10,000 to Dr. Frederick Backus, a prominent figure in civic and cultural affairs and an elected official when the City of Rochester was formed in 1834.

Backus made substantial structural alterations to the house, employing Greek Revival elements and detailing. A one-story ell attached to the main block permitted the doubling of the parlor, while the entrance hall and stairway were shifted from the front to the side. Stylish decorative alterations were made on the interior. These changes, some very subtle, record an important phase in the history of the structure and were retained in its restoration.

Following the Civil War, the house again underwent change. The kitchen wing was replaced by a large brick addition as the building was converted into a fashionable girls' school. For three-quarters of a century, Livingston Park Seminary attracted genteel young ladies from the eastern United States.

When this important structure was threatened with destruction in the 1950s, a prominent Rochester benefactor acquired it and had it carefully dismantled and stored until an appropriate location and use could be found. In 1970, the benefactor found a location that satisfied his requirements. He presented the house to the museum and shared in its reconstruction cost.