The Priory of the Augustinian Order of the Holy Sepulcher of Jerusalem was established by Henry de Newburgh, first earl of Warwick, in 1109. The building which housed the Order was completed sometime around 1119. Warwick was the seat of all the Holy Sepulcher houses in Britain and enjoyed a golden age for close to a century.

When Henry VIII broke from the Catholic Church in 1536, he confiscated the Priory at Warwick. The hundreds of monasteries and nunneries that were closed throughout Britain in the sixteenth century were sold to the king's favorites. The first owner of the Priory was Thomas Hawkins alias Fisher, a fishmonger, who acquired the building and lands during the short reign of Edward VI. Fisher later purchased much of the land surrounding the Priory and eventually dismantled most of the monastic buildings and built a house he called "Hawk's Nest," set in a landscaped park. It was at his Tudor manor house that Fisher entertained the newly crowned Elizabeth I. The curvilinear Dutch gables were added to the front façade around 1620, during the reign of James I. The fortunes of the Priory at Warwick rose and fell with its subsequent owners among whom were Henry Wise, Royal Gardener to Queen Anne, who acquired the house in 1709. The Lloyd banking family bought the Priory in the mid-nineteenth century but were forced to sell it in the early part of the twentieth century.

In 1925, Alexander and Virginia Weddell bought the Priory at a demolition sale. The Weddells had the house dismantled and rebuilt part of it in Richmond, Virginia, where they hoped the west wing would serve as a museum for the Virginia Historical Society. They planned that the remainder of the house would one day serve as the Society's headquarters. With these plans in mind, in 1929 the Weddells deeded Virginia House to the Historical Society and maintained a lifetime tenancy for themselves.