MapReduce word count

Fan-out to mappers, barrier to a shuffler, fan-out to reducers, barrier to completion, coordinated through shared state

MapReduce word count Fan-out to mappers, barrier to a shuffler, fan-out to reducers, barrier to completion, coordinated through shared state 01 / Input 02 / Map 03 / Shuffle 04 / Reduce 05 / Output Text source · split into chunks · 01 / Input Text source split into chunks Mapper 1 · emit (word, 1) · 02 / Map · parallel Mapper 1 emit (word, 1) parallel Mapper 2 · emit (word, 1) · 02 / Map · parallel Mapper 2 emit (word, 1) parallel Shuffler · group by word · 03 / Shuffle · barrier Shuffler group by word barrier Shared state · barrier + counts · 03 / Shuffle · coordination Shared state barrier + counts coordination Reducer 1 · sum counts · 04 / Reduce · parallel Reducer 1 sum counts parallel Reducer 2 · sum counts · 04 / Reduce · parallel Reducer 2 sum counts parallel Word counts · final result · 05 / Output · complete Word counts final result complete chunk chunk (word, 1) (word, 1) grouped grouped partial sum partial sum map barrier sync reduce barrier sync Legend primary data policy / PII async batch data store

Map phase

  • • Input is split into independent chunks
  • • Mappers run in parallel with no shared memory
  • • Each mapper emits (word, 1) pairs

Shuffle barrier

  • • All mappers must finish before shuffle starts
  • • Pairs are grouped by key across the set
  • • Groups fan out to parallel reducers

Shared state

  • • Coordinates both barriers between phases
  • • Tracks per-stage completion counts
  • • Final barrier releases the completed result

Data-flow diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • T theme • E export