I love these type of threads as I learn a lot about others passion for their castle sets. Thanks to those who have elaborated a bit on their collecting...
The first castle set I bought was 6054. Still my favorite mid sized set.

My oldest castle set is a second hand 375 purchased in 2007.

I started collecting in 1988 when I was first accepted into post graduate school. Luckily for me I was old enough and poor enough (i.e. college student) that I took great care of those early sets I bought. I have every piece, instruction manual, and box from every set I have ever bought. From 1988 I became a collector of castle and bought atleast one of every castle set and multiples of many. In 1997 after 10 happy years, the fright knights drove me into a dark age. I was still buying the sets, but I couldn't even bring myself to build them. I thought they were that bad.

For 3 years I purchased sets that sat new in the boxes. After 2000 when lego gave castle a big break, I took one too. I went into a true dark age with no purchasing at all.
I thought lego was doomed and had no future on the path they were heading and didn't buy anything castle for 7 years. That was until 2007 when the fantasy collection came out! Holy moly!

These new sets were well designed and looked awesome. I was reborn and excited about lego again. I decided to go back and collect all of the old sets I missed out on. I got back just in time. I was able to obtain all of the Knights Kingdom II and Viking sets I had missed out on for less than their original cost. These things were still in stock on Amazon and other online retailers 3 years after their first release.
Then I went back and built every old classic set I had. I even opened all of the fright knight and kk1 sets from their sealed boxes and built them too. Then I built all of the kk2 and viking sets. Finally I built the new fantasy sets. My garage was bursting with lego sets on tables. It was really cool to build everything in chronological order to see where lego went through the years. It was the most fun I have ever had with lego. Has anyone else ever done this?
Since my emergence from my dark age in 2007, I have gone back and obtained all but about 10 castle sets ever made. The ones I am missing are small sets. (less than 100 pieces) I plan on completing my collection over the next couple of years. After buying some beat up sets on ebay, I have learned to be careful in buying classic castle sets. I now use bricklink only for obtaining them.
Funny enough I do not own a single set from the Ninja line nor do I desire to. I didn't think of them as castle. Yet I like the viking and fantasy lines. Am I the only one that feels this way?