During the Sengoku period, Kazutoyo Yamauchi was the lord of the castle for 10 years. Kazutoyo, who was in the prime of his career, carried out large-scale castle repairs, constructing the castle tower and Otemon gate, as well as concentrating on the maintenance of the castle town and flood control work on the Oigawa River.

In the Meiji period, the castle was abandoned, and the stone walls and castle tower were demolished.

The castle tower has now been restored. Afterwards, the Otemon Gate and Ninomaru Palace were restored, and in 2002, the castle tower and palace of Kakegawa Castle was designated as a national important cultural property.