Batch Processing: From Unix Tools to Distributed Systems

Much of the traditional software operations we deal with are online, we click a button, wait for a moment, and the transaction or operation is completed. But there is a big area that deals with software operations that require offline processing. For example, background processing of jobs, e.g., OpenAI training/improving its existing GPT models behind the scenes using the data it gathers from its users. Batch Processing Whenever such an offline system runs a job that typically generates output from a batch of inputs, we call that batch processing. Inputs here are immutable, which avoids side effects. Benefits of batch processing: You can time travel. In case of any failure or unintentional outputs, you can jump to the last input checkpoint before a batch processing job. This handling is often referred to as human fault tolerance. Using batch processing and offline systems, compute usage efficiency can be improved. For example, whenever a heavy computation needs to be done, it's better to do it in bulk on maybe a GPU compute rather than crashing the CPU host where the server is online. Though the boundary between online and batch processing is not always clear. For example, a long-running database query could also be categorised as batch processing. Another alternative to batch processing is stream processing, which we will understand in the next article. MapReduce MapReduce is a batch processing algorithm that is utilized by Hadoop, CouchDB, and MongoDB as well. It is a balanced approach that is less extreme than completely parallelizing the jobs. There are several other frameworks like this that are now replacing MapReduce. For example, DataFrames APIs, query languages, etc. We will see MapReduce in detail sometime later. Simulating Batch Processing with Unix Tools (Single Host) If you are a Linux user, this simulation could be very easy for you to grasp. If not, just put it in ChatGPT or any AI tool to understand the command in detail if interested. A typical Ngin