udley Castle was originally a Norman castle and was built in the Midlands by William Fitz-Ansculf just after the Norman Conquest in 1071. The land had been held by Edwin, Earl of Mercia, in the time of Edward the Confessor and he was allowed to retain the lands after the Conquest. Edwin was later killed after rebelling against William the Conqueror. During the civil war between Matilda and King Stephen the castle was held by a supporter of Matilda until 1138 when Stephen attacked and destroyed the castle and the surrounding area. In 1165 Gervase, a son of the Castle's owner, founded a priory at Dudley, but in 1175 he rebelled against King Henry II. The King in return attacked and destroyed much of the castle. On the death of Gervase the castle became the property of the de Somery family through marriage to Gervase's sister. In 1264 Roger de Somery was granted a license to refortify the castle. Building work continued during the ownership of John de Somery who had a bad reputation for murder and theft to fund the construction of the new castle.

After the death of John in around 1321, Dudley Castle became the property of his sister Margaret who married into the de Sutton family. At this time large amounts of building work took place at the castle, including a barbican to the outer gatehouse and chapel.

In the 1530s the castle's residential block was rebuilt in the latest style by John Dudley. But he was arrested and executed by Queen Mary for his involvement in the attempt to place his daughter-in-law, Lady Jane Grey, crowned as the Queen of England. The castle then went to Edward Sutton who in 1575 invited Queen Elizabeth I to visit.

The last time Dudley Castel took part in historical events was during the English Civil War. The castle was held by Royalisty troops and sieges were held. Eventually the castle was surrendered and in 1647 the castle's defenses were dismantled so that it could not be used as a stronghold again. The remains became the property of the Ward family who used the domestic range for some time.