Hadoop Summit

Last week I attended the Hadoop Summit. It was sold out ten days prior! That doesn't happen very often these days, in an environment where corporate travel is severely cut back. I even saw sizable contingents from Japan and Brazil. This is a very exciting time to be involved in the field of search and data processing. The science track was quite interesting, as was the talk by facebook about how they are crunching something like 90 TB per week, and are siting on 85 Peta Bytes. They are using hadoop mapreduce in near real-time. Processing latencies are getting close to 1 minute. That's nearly real-time availability of all the information you post on facebook, to everyone else (and every moving part) in the facebook system. Oh, and the lunch was great.