Dare has posted some excellent summaries of Google's Scalability conference. (I can't wait till the videos are online!)
I am always entertained by counterintuitive results, and the solution to stragglers in MapReduce is no exception.
Google had an issue with "stragglers" performing tasks very slowly in the MapReduce infrastructure. The solution: duplicate the same tasks on multiple machines and throw away the redundant results. Go faster by doing more work! ;)
Related idea: set-based design. Optimize the throughput of the entire system, instead of each part.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment