Pipeline Architecture AKA Pipes and Filters
- sometimes is used as a pattern in solutions
- the data flow in pipes between filters
- he data flows unidirectionally
- the paradigm is identical to the piping in Linux “ls | grep data
- there are 4 types of filters (ways of handling data)
- producer – the origin of data (source)
- transformer – receives data and can transform it (map)
- tester – tests the data and outputs some of it (reduce)
- consumer – the data flow stops (destination)
- examples: ETL tools, orchestrators, document transformations, operating systems
- pros: Simple, testable, Modular
- cons: scalability, fault tolerance, elasticity, deployability, performance
