Apache Crunch

A pipeline is a concatenation of operations to perform a specific job, arranged so that the output of each element is the input of the next.

Apache Crunch provides an easier method of dealing with Apache Hadoop MapReduce pipelines.

Crunch simplifies this process by providing a large number of ready-made methods and functions which can be used to assemble and manipulate MapReduce pipelines in various forms.

The project includes a native Java API, along with a Scala one (named Scrunch).

Support is additionally included for handling Avro records and HBase rows and columns.