Here are a few books I've found to be stimulating:
- The castellated and domestic architecture of Scotland / David MacGibbon, Thomas Ross
- The ecclesiastical architecture of Scotland from the earliest Christian times to the 17th century / David MacGibbon, Thomas Ross
- The castles of England: their story and structure / Sir James Dixon Mackenzie
- The castles and abbeys of England / William Beattie
- Some Account of Domestic Architecture in England: From the conquest to the end of the 13th century / Thomas Hudson Turner, John Henry Parker
The books are thoroughly illustrated with drawings of structures, architectural details, and site plans. They are in the public domain (so they're typically around a hundred years old). When it comes to this kind of thing, however, age can be an asset (especially when there are engravings/drawings of sites that have been destroyed or significantly changed in the last hundred years).*
*An interesting example of this is the Stokesay Castle, where the only image of the castle with a wet moat is from an 18th century engraving. Compare the pic below with modern photos on the CastleWales.com site:

Another great place to look for public domain works on Castles, etc. is http://www.archive.org.