After the death of John in around 1321, Dudley Castle became the property of his sister Margaret who married into the de Sutton family. At this time large amounts of building work took place at the castle, including a barbican to the outer gatehouse and chapel.
In the 1530s the castle's residential block was rebuilt in the latest style by John Dudley. But he was arrested and executed by Queen Mary for his involvement in the attempt to place his daughter-in-law, Lady Jane Grey, crowned as the Queen of England. The castle then went to Edward Sutton who in 1575 invited Queen Elizabeth I to visit.
The last time Dudley Castel took part in historical events was during the English Civil War. The castle was held by Royalisty troops and sieges were held. Eventually the castle was surrendered and in 1647 the castle's defenses were dismantled so that it could not be used as a stronghold again. The remains became the property of the Ward family who used the domestic range for some time.