Free Search

On the first of this year, after four years as an independent contractor, I accepted a full-time job with Yahoo!. This isn’t as big of a change as it sounds. For much of the past four years my work on Nutch had been in-part funded by Yahoo! (and Overture before they were acquired by Yahoo!). I’m still primarily working from home, and, so far, entirely working on open-source stuff: Lucene, Hadoop and Nutch. The biggest change is that I don’t have to draft contracts, submit invoices, etc. I can now instead better focus on the technology and the open-source process.

We’ve split the distributed computing parts of Nutch into a new project named Hadoop. This includes a filesystem modelled after GFS and a distributed computing system modelled after Google’s MapReduce. So far a few folks are using Hadoop on tens of machines, and we’re testing it on clusters with hundreds of machines. Next stop, thousands!

Someone suggested that I should be shot.