It is located on a hill that juts out into Hirado Seto overlooking Hirado Port and the mainland of Kyushu on the opposite bank, and is surrounded by the sea on three sides and has a natural moat.

It is a Hirayama castle built in the echelon style, with the main enclosure built at the top of the hill, the Ninomaru on the south side, and the Sannomaru on the east side.

It became an abandoned castle due to the Meiji Castle Abandonment Ordinance.
The castle tower is a mock tower made of reinforced concrete.