The misfits - ne me quitte pas
Once you’ve placed the terrain, use that to inform the locations of the main buildings. Here the castle goes on the highest promontory, with a commanding view of the sea and the land around. The cliffs on the promontory provide natural defences. Any land based threat must come down the peninsula, and the town will want to defend the harbour, so it’s natural for there to be a wall across the end of the peninsula.