What do you do to process a million records in 30 seconds?

  • Answer?
    • You divide and conquer!
  • How?
    • You divide a million records as 1000 jobs with 1000 records each and run them in parallel.
  • So whats the big deal?
    • Well! You need 1000 CPUs for just 30 secs. After that you dont need them ;-)
  • What does this mean?
    • Forget the number of servers you have in your data center. Servers are not the constraint anymore. Use cloud! You have can suddenly acquire 1000 server run the job and release them back and pay only for the usage. Of course, Google App Engine has 30 second limit to respond back to the user, because if computing resource is not a constraint, then user experience & response times are! In other words, the constraints are reversed.