Watch Doug Cutting on search with Lucene and Nutch
A lot of developers don't realize the kind of genius and time it takes to build a tool like Lucene, I think we are very lucky to have someone of Doug Cutting's talents spending all of his time working on this stuff, how many people could do this:
Question (from the interview): How do you make some thing like Lucene as fast as it is?
Answer: I do it a few times. I have written a few search engines and done a lot of benchmarking and looked where they spent their time and then rethought it and I think it helped a lot that it wasn’t the first search engine I had written. I think at Xerox I did a few iteration of very different architectures, then did so again at Apple and then again at Excite and so I have been through it a few times and knew what needed to be quick and what did not.