Skip to content
José da Cruz

Enterprise Architect, and Life Thinker

José da Cruz

Enterprise Architect, and Life Thinker

Pipeline Architecture AKA Pipes and Filters

josedacruz, June 22, 2023

  • sometimes is used as a pattern in solutions
  • the data flow in pipes between filters
  • he data flows unidirectionally
  • the paradigm is identical to the piping in Linux “ls | grep data
  • there are 4 types of filters (ways of handling data)
  • producer – the origin of data (source)
  • transformer – receives data and can transform it (map)
  • tester – tests the data and outputs some of it (reduce)
  • consumer – the data flow stops (destination)
  • examples: ETL tools, orchestrators, document transformations, operating systems
  • pros: Simple, testable, Modular
  • cons: scalability, fault tolerance, elasticity, deployability, performance

Related

architecture architecturefilterpipelinepipes

Post navigation

Previous post
Next post

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • architecture (167)
  • artificial-intelligence (6)
  • books (2)
  • courses (4)
  • decision-frameworks (4)
  • finances (1)
  • java (8)
  • life-improvement (20)
  • metrics (3)
  • observability (2)
  • puzzles and challenges (3)
  • reviews (1)
  • security (8)
  • Soft Skills (1)
  • spring-boot (36)
  • Systems Thinking (3)
  • GitHub Weekly: AI Agent Tools Trending Right Now (November 2 – November 8, 2026)
  • Designing a Distributed Session Store: Why Sticky Sessions Break Under Load
  • Essential Reading: Release It! — Why Production Is a Different Game Than Development
  • GitHub Weekly: Top 10 Trending Repos (October 26 – November 1, 2026)
  • GitHub Weekly: Architecture & Infra Tools Worth a Look (October 19 – October 25, 2026)
©2026 José da Cruz | WordPress Theme by SuperbThemes