S1E4: Mastering Concurrency Fan-In Design Pattern

In this series we have already learned: (Concurrency Design Patterns — YouTube)

1. S1E1: Concurrency In Go | Goroutine | Channels | Waitgroup | Buffered Channel | by Arshlan | Jul, 2023 | Medium
2. S1E2: Concurrency Boring Desing Pattern in Go | by Arshlan | Jul, 2023 | Medium
3. S1E3: Mastering Concurrency with Worker Pool in GoLang: A Scalable Solution for Efficient Task Processing | by Arshlan | Aug, 2023 | Medium
4. S1E4: Mastering Concurrency Fan-In Design Pattern | by Arshlan | Aug, 2023 | Medium
5. S1E5: Mastering the Concurrency in Go with Fan Out Design Pattern | by Arshlan | Aug, 2023 | Medium

Youtube Video Link on Fan In Concurrency Design Pattern with full code and diagram explaination: S1E4: FAN-IN Concurrency Design Pattern | Concurrency Design Pattern Playlist — YouTube

The “Fan-In” pattern is used in concurrent programming to consolidate and merge data from multiple input sources (channels) into a single output channel. This pattern has several use cases where it can simplify code, improve performance, and enhance data aggregation in concurrent applications:

1. Aggregating Data from Multiple Sources:
   — Real-time Data Collection: When you need to collect data from multiple sources, such as sensors, devices, or data streams, the Fan-In pattern helps you consolidate data efficiently.
2. Handling Concurrent Producers:
   — Multiple Goroutines Producing Data: If you have multiple goroutines that produce data concurrently, the Fan-In pattern provides a structured approach to handle their output without the complexity of managing individual channels for each goroutine.
3. Simplifying Code:
   — Centralized Logic: Instead of handling data processing and aggregation logic separately in each goroutine, the Fan-In pattern centralizes the logic in a single place (the merging function), making your code more organized and maintainable.
4. Performance Optimization:
   — Reduced Goroutine Overhead: When the number of goroutines sending data is significant, managing all those channels can lead to increased overhead. The Fan-In pattern minimizes the number of channels and simplifies synchronization, resulting in improved performance.
5. Efficient Resource Utilization:
   — Limited Resource Consumption: If resources like network connections or API requests are involved, using the Fan-In pattern prevents excessive resource consumption by limiting the number of concurrent operations.
6. Combining Results from Concurrent Operations:
   — Parallel Processing: For scenarios involving parallel processing of tasks on different datasets, the Fan-In pattern helps merge the results efficiently, allowing for better utilization of available CPU cores.
7. Composite Data Creation:
   — Complex Data Composition: When data needs to be composed or assembled from multiple sources before being used, the Fan-In pattern facilitates the aggregation and composition process.

Real-World Use Cases:
— Financial Data Aggregation: Combining data from various financial markets or stock exchanges for analysis.
— Web Scraping: Collecting data from multiple websites or APIs concurrently and merging the results.
— Distributed Systems Monitoring: Gathering data from different nodes in a distributed system and centralizing it for monitoring and analysis.

In summary, the Fan-In pattern is a versatile tool that simplifies the management of data from multiple sources in concurrent applications. It enhances code organization, improves performance, and streamlines the aggregation process. By understanding the use cases where the Fan-In pattern is beneficial, you can leverage it effectively to create more efficient and scalable concurrent applications.

Note: Code explained in the Youtube Video

package main

import (
 "fmt"
 "sync"
)

func producer(id int) <-chan int {
 ch := make(chan int)
 go func() {
  defer close(ch)
  for i := 0; i < 3; i++ {
   ch <- id*10 + i
  }
 }()
 return ch
}

func fanIn(inputs ...<-chan int) <-chan int {
 var wg sync.WaitGroup
 output := make(chan int)

 // Function to copy values from a channel to the output channel
 copy := func(c <-chan int) {
  defer wg.Done()
  for val := range c {
   output <- val
  }
 }

 wg.Add(len(inputs))
 for _, ch := range inputs {
  go copy(ch)
 }

 go func() {
  wg.Wait()
  close(output)
 }()

 return output
}

func main() {
 // Step-1 Producer
 ch1 := producer(1)
 ch2 := producer(20)

 // Step-1 FanIn
 mergedOutputChannel := fanIn(ch1, ch2)

 // Step-3 Consumer
 for val := range mergedOutputChannel {
  fmt.Println(val)
 }
}

• Youtube Channel: https://www.youtube.com/@codepiper
• Github Link: CodePiper/GO/concurrency-patterns/Basics at main · arshad404/CodePiper (github.com)