During the 19th century the castle mound and eight acres of surrounding land was owned by the Taylor family. In 1907 they decided to sell-up and this open space was acquired for the town.
The creation of Castle Gardens took place in 1907/08 when it was laid out with trees, flower beds, footpaths, fencing, bandstand and a shelter converted from an old cow shed. And as with all public spaces, bylaws were put in place: forbidding the public to beat their carpets or drive cattle through the gardens, or park perambulators on the flower beds.
While he held this town the Conqueror built a small castle, or keep, on a high artificial mound of earth, .which must have existed in Saxon times, in the meadows lying between the town and the village of Hockerill on the E., called Waytemore Castle. This castle he gave to the Bishops of London, from whence the place had its surname.
King John, in resentment against Bishop William de Maria, who had been instrumental in the laying of England under Papal interdict, destroyed the castle, but he was afterwards obliged to make compensation to the bishop, and gave him his manor of Stoke near Guildford in substitution.
The castle contained a chapel dedicated to St. Paul. There was a dungeon, called “the Bishops Hole,” used as an ecclesiastical prison, last used by Bishop Bonner, and also called the “Convicts’ Prison,” out of which, in Queen Mary’s time, one of the prisoners was taken and burnt on a green called “Goose-meat,” near the road from Stortford to Hockerill.
There remain now only some flint-built walls upon the mound, all besides has disappeared. (Castles Of England, Sir James D. Mackenzie, 1896)