S1E5: Mastering the Concurrency — Fan Out Design Pattern

In this series we have already learned: (Concurrency Design Patterns — YouTube)

1. S1E1: Concurrency In Go | Goroutine | Channels | Waitgroup | Buffered Channel | by Arshlan | Jul, 2023 | Medium
2. S1E2: Concurrency Boring Desing Pattern in Go | by Arshlan | Jul, 2023 | Medium
3. S1E3: Mastering Concurrency with Worker Pool in GoLang: A Scalable Solution for Efficient Task Processing | by Arshlan | Aug, 2023 | Medium
4. S1E4: Mastering Concurrency Fan-In Design Pattern | by Arshlan | Aug, 2023 | Medium
5. S1E5: Mastering the Concurrency in Go with Fan Out Design Pattern | by Arshlan | Aug, 2023 | Medium

Video Link: S1E5: FAN-OUT Concurrency Design Pattern | Concurrency Design Pattern Playlist — YouTube

https://youtu.be/ZExWgjTOIzU

The Fan-Out pattern is a concurrency gem that allows us to efficiently distribute tasks among multiple worker goroutines for blazing-fast parallel processing. Whether you’re handling data, web scraping, load balancing, or real-time event processing, the Fan-Out pattern has your back.

Photo from the YouTube video:

1. The “Source” generates tasks that need to be processed concurrently.
2. The “Task Queue” acts as a buffer, holding the tasks before they are picked up by worker goroutines.
3. The “Worker Goroutines” pick tasks from the queue, process them independently, and put results (if any) in the “Result Queue.”
4. The “Result Queue” holds the processed results from the worker goroutines.
5. The Fan-Out pattern involves distributing tasks from the source to multiple worker goroutines for parallel processing.

NOTE: Keep in mind that the number of workers goroutines and the concurrency level can be adjusted based on your application’s requirements and available resources. The Fan-Out pattern allows you to efficiently utilize available CPU cores for parallel task processing.

Working Code as Per the Diagram: (Explained in the CodePiper YouTube Channel)

package main

import (
 "fmt"
 "sync"
 "time"
)

func source(tasks chan<- int, numTasks int) {
 for i := 1; i <= numTasks; i++ {
  tasks <- i
 }
 close(tasks)
}

func worker(id int, taskQueue <-chan int, results chan<- int, wg *sync.WaitGroup) {
 defer wg.Done()
 for task := range taskQueue {
  // Simulating task processing
  fmt.Printf("Worker %d processing task %d\n", id, task)
  time.Sleep(time.Millisecond * time.Duration(task))
  results <- task * 2 // task 1 * 2 | task 2 * 2
 }
}

func main() {
 const numWorkers = 3
 const numTasks = 10

 taskQueue := make(chan int, numTasks)
 resultQueue := make(chan int, numTasks)
 var wg sync.WaitGroup

 // Start the source goroutine to generate tasks
 go source(taskQueue, numTasks)

 // Create worker goroutines
 for i := 1; i <= numWorkers; i++ {
  wg.Add(1)
  go worker(i, taskQueue, resultQueue, &wg)
 }

 // Wait for all workers to finish
 go func() {
  wg.Wait()
  close(resultQueue)
 }()

 // Process results from the result queue
 for result := range resultQueue {
  fmt.Printf("Result received: %d\n", result)
 }

 fmt.Println("All tasks completed.")
}

Use Case of Fan-Out Design Pattern:

1. Data Processing Pipelines: In data processing scenarios, such as ETL (Extract, Transform, Load) pipelines, you can use the Fan-Out pattern to distribute data processing tasks across multiple workers. This is useful for transforming and enriching data before loading it into a database or data warehouse.
2. Web Scraping: When scraping data from multiple websites concurrently, each worker can handle the scraping of a specific website. This allows you to efficiently retrieve data from multiple sources in parallel.
3. Load Balancing: In server applications, you can use the Fan-Out pattern to distribute incoming client requests to a pool of worker goroutines. This enables efficient load balancing and ensures that the workload is evenly distributed.
4. Real-Time Data Processing: In applications that require real-time data processing, such as monitoring systems or event-driven architectures, the Fan-Out pattern can be employed to process incoming events or messages concurrently.
5. Parallel File Processing: When dealing with large files or a batch of files, you can use the Fan-Out pattern to concurrently process different segments of the files, enabling faster file processing.
6. Image or Video Processing: In multimedia applications, the Fan-Out pattern can be used to parallelize the processing of images or frames in videos, resulting in quicker analysis, transformation, or editing.
7. Distributed Search: When performing distributed search or indexing operations, you can distribute search queries to multiple worker goroutines to simultaneously search through different data partitions or indices.
8. Scalable Networking: In network-related tasks, like handling incoming network connections, the Fan-Out pattern can be used to process each connection concurrently, enabling efficient use of resources.
9. Distributed Computing: In distributed systems, tasks can be distributed across worker nodes to leverage the computational power of multiple machines.
10. Parallel Algorithm Execution: The Fan-Out pattern can be applied to parallelize the execution of certain algorithms, such as sorting or data aggregation, by distributing subtasks to different worker goroutines.

• Youtube Channel: https://www.youtube.com/@codepiper
• Github Link: CodePiper/GO/concurrency-patterns/Basics at main · arshad404/CodePiper (github.com)