Diagram source: aws

Introduction

As the name suggests, "pub-sub" its an architectural pattern where a publisher sends out messages, and subscribers receives them. The cool part is that the publisher doesn't need to know who the subscribers are or how many of them are receiving the message. There are some terms to understand in pub sub:

Publisher: The entity that publishes message.

Message: Data published (sent) by publisher.

Topic: A central hub to manage similar queues. Subscribers of topic will receive messages related to this topic.

Subscribers: The entity that consumes and process message.

Publishers send messages to a central hub (topic), without any knowledge of who will receive the message. Any services interested in this publisher will subscribe to the topic where publisher publishes message. Subscribers express interest in one or more topics and receive messages related to those topics.

Implementation

There are several ways to implement the pub-sub pattern in a distributed system. Popular options include AWS services like SQS and SNS, which are commonly used for decoupling services and handling asynchronous messaging. SNS (Simple Notification Service) broadcasts messages to multiple subscribers, while SQS (Simple Queue Service) queues messages for processing. Another option is Apache Kafka, a distributed streaming platform renowned for its durability and fault tolerance, making it ideal for high-throughput scenarios. Kafka is well-documented and widely used in large-scale data heavy systems. NATS is another lightweight and high-performance messaging system that is well-suited for cloud-native applications. It supports the pub-sub pattern with low latency and minimal overhead, making it an excellent choice for distributed system where fast, reliable messaging is crucial. Azure offers similar solutions, such as Azure Service Bus for reliable messaging between services and Azure Event Grid for event-based pub-sub communication across various Azure services.

Implementing Pub-sub with websockets

Here, I will be using SignalR to implement pub-sub pattern. SignalR simplifies real-time communication between the server and clients, allowing you to send messages to multiple connected clients simultaneously. About system:

1. publisher.api service acts as a publisher, broadcasting weather data to its subscribers.

2. weathersub is a service that consumes the weather data to perform specific processing tasks.

3. weathersub01 is another service that also consumes the weather data for its own processing needs.

When i publish a weather message:

Subscribers receive messages:

Source code: Github

You might notice that weathersub01 receives the message twice. This happens because whenever a new subscriber connects, it triggers the publisher.api to send out the weather data to all connected clients. As a result, every time a subscriber connects, weathersub01 and other subscribers receive a copy of the broadcasted message.

Using SignalR to introduce pub-sub in distributed system is very straightforward and fast. It simplifies real-time communication, making it easy to broadcast messages to multiple connected clients simultaneously. SignalR can be effective for certain use cases in a distributed system, but its suitability depends on specific requirements. If your system is just adapting pub-sub, then SignalR is a good choice due to its ease of integration and real-time messaging capabilities. It’s particularly useful for scenarios where real-time updates are needed and message persistence is not critical.

Disadvantages of implementing pub-sub using SignalR

One disadvantage of using SignalR for pub-sub is the potential for message duplication. When a new connection is established, it can trigger the publisher to resend data, causing multiple copies of the same message to be received by subscribers.

Another drawback is that if a subscriber is offline or unavailable when a message is sent, it will miss that message entirely. Unlike some other pub-sub systems that provide message persistence or retry mechanisms, SignalR doesn’t inherently store messages for offline subscribers, so any missed messages are not retried or delivered once the subscriber reconnects.

Handling disadvantages

To handle this, you can add retry mechanism and idempotency to every messages. This approach involves ensuring that every message is processed only once, even if it is received multiple times, and implementing strategies to retry messages if a subscriber is temporarily offline. However, implementing these solutions from scratch can be complex and time-consuming.

Instead of reinventing the wheel, it’s often more practical to use existing tools and systems designed to handle these challenges efficiently. For larger or more complex distributed systems where message durability, high throughput, and guaranteed delivery are crucial, specialized tools like Apache Kafka, NATS, AWS SQS and SNS, or Azure Service Bus and many more such tools might be more appropriate. These tools offer robust solutions for handling large volumes of messages, ensuring message delivery (retry mechanism), and providing fault tolerance.

Summary

This article explored the pub-sub (publish-subscribe) pattern, which allows a publisher to send messages to multiple subscribers without needing to know who they are. I explained various tools for implementing pub-sub, such as AWS SNS for broadcasting messages, Apache Kafka for durable, high-throughput streaming, NATS for lightweight, high-performance messaging, and Azure services like Service Bus and Event Grid for reliable messaging and event-based communication.

In the implementation part, i used SignalR to implement pub-sub with WebSockets, where publisher.api broadcasts weather data to multiple services like weathersub and weathersub01. A noted disadvantage of SignalR is the potential for message duplication and the risk of missed messages if a subscriber is offline. To address these issues, you could implement custom retry mechanisms and idempotency to handle duplicates and missed messages. However, using specialized tools and systems that already handle these challenges efficiently is often more practical and less complex.

References

1. Aws article on pub-sub

2. NATS example

3. Kafka docs

4. Getting started with Pub-sub by Nick Chapsas

5. Real-time applications with Azure Web PubSub - Poornima Nayar - NDC Oslo 2023

Introduction

Messaging in a broader way is a form of communication between some parties. In distributed system, it refers to the process of sending and receiving messages(data) between some services to provide communication.

Asynchronous Messaging

Asynchronous messaging in distributed system is a pattern where sender and the receiver of a message do not need to communicate directly with each other. The sender sends the message but do not wait for the receiver's response. The sender just continues with its operation without waiting for the receiver's response.

For example, if Service A is a sender and Service B is a receiver, in asynchronous messaging, Service A sends a message to an intermediary system (like a message queue), and Service B retrieves and processes the message from that intermediary. Even though Service A and Service B do not interact directly, the message is still delivered and handled through this intermediary system. Using queuing technologies for messaging in a distributed system effectively manages this asynchronous communication.

Queuing technologies

Queuing technologies are just some set of tools and patterns designed to facilitate asynchronous messaging through message queues and message buses. A message queue serves as an intermediary storage system, holding messages until they are processed. Let’s first explore the message queue and then the message bus, while they share similarities, the message bus offers additional features. There are three general components in a queuing system:

1. Producer: Who produces message.

2. Consumer: Who consumes message.

3. Queue: Place where messages are stored from producer. It store the messages somewhere on the disk until consumer consumes it.

It can be visualized as:

How does a message queue works?

You might have used socials platform like Facebook or Instagram, where user can configure a settings to receive notifications via in-app push notification, email, or SMS when certain events occur such as receiving a comment on your post.

I'm not sure how Facebook have designed it but if I had to design this feature using queuing technology, I would do so: when a user receives a new comment notification, that notification service creates a message (data) and place that in a message queue. Instead of directly sending notifications through email and SMS services in synchronous way, these services would consume messages from the queue. Each service, such as the email service or SMS service, would pull messages from the queue and process them according to its delivery method. This setup decouples the notification generation from the delivery process, allowing for more flexibility and scalability.

So, message queue operates as a middleware between services. There is a message publisher who sends message, a queue where message is stored until they are processed or queue is full, the consumer who consumes and uses the message accordingly.

What problem does Message queue solves?

In previous article, I mentioned a lot of What ifs in synchronous communication like: when calling to some remote service and it is unresponsive, scalability issues where we might increase afferent and efferent coupling between services, etc. These things tend to severely impact overall system's performance and reliability. Message queue addresses these challenges and provides a robust and reliable asynchronous mechanism.

1. Reliability & Inconsistent data: When Service A directly calls Service B and Service B becomes unresponsive, it can lead to failures in both services. This issue is compounded when multiple services are involved; for instance, in a system with 20 or more services, a single unresponsive service can disrupt the entire process and result in inconsistent data.

   Solution: Messaging pattern. Instead of directly calling services, we can rely on some sorta buffer i.e. a message queue. The message will be stored in a queue in a physical queue, until it is processed by some consumer. This buffering mechanism ensures that even if a service is temporarily unavailable, messages are preserved and processed once the service is back online, thereby enhancing reliability and consistency across the system.

2. Reduced in coupling: Message queuing decouples the calling (consumer) and the responder (publisher) service. The consumer processes messages from the queue asynchronously, allowing the services to operate independently of each other. This decoupling improves system flexibility and scalability, as changes to one service (such as updates or failures) have minimal impact on the other.

3. Improved fault tolerance: Message queues enhance fault tolerance by cutting-off failures. If a service encounters issues, messages remain in the queue and can be processed later once the service recovers. Additionally, message queues often include features like retry mechanisms and dead-letter queues to handle persistent issues, ensuring that the system remains robust and messages are eventually processed.

4. Proper error handling (using dead letter queue or poison queue): Most of the message queuing technologies support dead letter queue or poison queue. Imagine a scenario where a publisher sends a message that is faulty or causes an error during processing. Instead of having the system fail or lose the message, it is redirected to a dead letter queue. This queue is specifically designed to capture messages that cannot be processed successfully after several attempts.

Types of Message Queue

There are various types of message queue. Here are some common ones:

1. Point-to-point queue: In a point-to-point queue system, a producer sends a message to a specific queue intended for a single consumer. Once the message is received by the consumer, it processes the message and sends a response back to the producer. If the response indicates success, the message is removed from the queue, confirming that it has been successfully delivered and processed by the consumer. This mechanism ensures that each message is handled by only one consumer, providing reliable message delivery and acknowledgment.

2. Publish-subscribe queue: Here, queue are called "topic". Publisher publishes message to this topic and any number of subscribers subscribed to this topic receives a copy of message. This process is called fanout. Pub-sub queue is ideal when you need to send message to multiple services.

3. In-memory queue: Here, queue stores messages on memory rather than on disk or "database". They are performant but not ideal if you want your data to be persisted and durable.

4. Priority queue: Typically, a message queue operates on a FIFO (first-in, first-out) basis, meaning messages are processed in the order they are received. However, a priority queue allows messages to be stored and processed according to their priority level. In a priority queue, messages with higher priority are dequeued and processed before those with lower priority, regardless of the order in which they were added. This ensures that critical messages are handled quickly, enhancing the responsiveness of the system to high-priority tasks.

Messaging Protocols

Every queuing technology follow a set of rules on how messages are serialized or formatted, how they communicate, and how they are transmitted. Messaging protocols ensures to enable standard among queuing technology. Here are some commonly used messaging protocols:

1. AMQP (Advanced Message Queuing Protocol): AMQP is a widely used robust, language-agnostic, wire-level messaging protocol that facilitates message oriented middleware. Wire-level means it specifies exactly how data should be structured and transmitted over the network. This allows different services to understand each other, no matter what language or platform they are using. This makes AMQP reliable and useful for complex systems that need to exchange messages consistently and securely.

2. MQTT (Message Queuing Telemetry Transport): MQTT is a light weight pub-sub messaging protocol designed for simple and efficient communication, especially over networks that are unreliable. It’s often used in situations where devices have limited processing power or bandwidth, like in IoT applications. MQTT works using a pub-sub pattern, where devices can send messages (publish) to a central broker, and other devices can receive (subscribe to) those messages if they’re interested. This makes it easy to send data from sensors, for example, to a server or other devices, without needing a direct connection between them.

3. STOMP (Simple Text Oriented Messaging Protocol): STOMP is a straightforward messaging protocol that's is easy to understand and work with because it is text-based messages. In other word, they are human readable, making it simple to debug and develop with. STOMP is often used to send messages between different services, like sending updates from a server to a web application. It works by having clients connect to a message broker, where they can send (publish) messages or receive (subscribe to) messages. The text-based nature of STOMP makes it accessible and flexible, allowing it to be used in a variety of applications where simplicity and ease of integration are important.

4. XMPP (Extensible Messaging Presence Protocol): Mainly used for real-time communication, XMPP is a protocol based on XML. XMPP is extensible, meaning you can add custom features to it, like file sharing or voice calls. This flexibility makes it easy to build more complex applications on top of the basic messaging functions. For example, a simple chat app using XMPP can be extended to include video calls, group chats, or even real-time collaboration tools, all while using the same underlying protocol. XMPP supports real-time updates, so you can see if someone is online, away, or busy. This protocol is widely used in chat applications, collaboration tools, and even IoTs, where quick, real-time communication is important.

Understanding some popular queuing choices

There are many popular message queuing technologies out there with its own features and tradeoffs. Here are some popular choices:

1. RabbitMQ: Opensource queuing technology that supports various types of messaging protocols including: AMQP, STOMP, MQTT, HTTP, Web-sockets, etc. It is known for its robustness, and flexibility. RabbitMQ comes with visual interface where you can see and manage queues and exchanges. It is ideal for systems where there are complex routing patterns (meaning there are many rules for deciding where messages should go based on their content), requires data and delivery reliability, and systems using multiple messaging protocols.

2. AWS Simple Queue Service (SQS): A fully managed cloud-based queuing system provided by Amazon Web Services (AWS). Fully managed means AWS takes care of servers and infrastructures. SQS are very scalable. There are two types of queues provided by SQS:

   1. Standard Queue => High throughput and at-least one delivery but doesn't guarantee the order of messages.

   2. FIFO Queue => Messages are processed in order they were sent and only once.

SQS provides options for encrypting messages and controlling access to the queue making it secure and ensuring that only authorized users and services can access or manage the messages. SQS is very ideal for serverless systems.

3. Apache ActiveMQ: Apache ActiveMQ, an open-source message broker that facilitates communication between different systems or components by handling messages supports various messaging protocols like: AMQP, STOMP, and MQTT, making it versatile. Apache ActiveMQ is ideal for JVM applications where versatility, and reliability are must.

Message Bus

A message bus is a more complete messaging framework that serve as a centralized channel for communication between various services in a system. A message bus typically includes all the core features of a message queue, such as reliable message delivery and message storage, and adds additional functionalities for routing, processing, and transforming messages. Like a message queue, it stores messages until they are processed. Additional features like:

1. Advanced routing: The message bus can direct messages to different services based on their content or other factors. For example, it might send customer orders to an order processing service and customer feedback to a support team.

2. Message processing: It can handle and modify messages before sending them out. For example, it might add extra information to a message or filter out unnecessary details.

3. Message Transformation: The message bus can change messages into different formats or structures to fit the needs of different systems. For example, it might convert a message from XML to JSON.

4. Pub-Sub Pattern: Instead of sending messages to one specific place, a message bus can send the same message to multiple destinations. For example, a news update could be sent to both a website and a mobile app (both different services).

5. Orchestration: The message bus manages how messages flow between different services. It ensures that the right messages go to the right places in the correct order. For example, it might ensure that a customer’s order goes through several steps, like payment, shipping, and confirmation (in order).

How does a Message bus works?

Message bus can be described as message queue on roids. While a message queue focuses on storing and delivering messages in a straightforward, point-to-point manner, a message bus takes it up a notch with additional features (mentioned above). Message infrastructure can be a bit complex to understand, but here is a detail look at it:

1. Producer: A service that generate messages. They send data or events to the message bus. For example, a service might generate a message when a user submits a form.

2. Consumer: These are the services that receive and process messages. Consumers subscribe to the message bus to get the messages they are interested in.

3. Message Broker: A component within the message bus that routes messages from producers to the appropriate consumers. It may handle tasks like message transformation, and routing based on various criteria.

4. Message Topics or Channels: These are logical channels or categories through which messages are classified. Producers and consumers can publish or subscribe to specific topics to filter the messages they handle.

5. Message Store: A persistent storage (physical disk) that keeps messages until they are successfully processed. This ensures that messages are not lost in case of failures.

To sum up, a producer generates a message and sends it to the message bus, specifying the intended topic or channel. The message bus, often through its broker, routes the message to the appropriate queue(s) based on the topic or channel specified. The routing may involve applying filters or transformations. The message is temporarily stored in a queue if it cannot be immediately delivered. This ensures that messages are not lost and can be processed later. Consumers subscribe to topics or channels and receive messages from the message bus. The delivery may be direct or involve additional processing based on the bus’s configuration. After processing, the consumer acknowledges receipt of the message. If the processing fails, the message bus might retry delivery or move the message to a dead-letter queue/poison queue for further investigation.

Difference between Message queue and Message bus

What problem does a Message bus solves?

Similar to a message queue, a message bus solves problems related to complex communication patterns, scalability, and system integration. It enhances reliability and fault tolerance by managing message routing, integration across diverse systems, and providing robust monitoring tools. By decoupling services and supporting advanced features like message persistence and transformation, a message bus ensures smooth and scalable communication in complex, distributed environments.

So Message queue or Message bus?

Well, choosing any type of solution really depends upon the requirements.

1. Message Queue: Ideal for simpler use cases where you need reliable, asynchronous communication between components. It focuses on storing and delivering messages between producers and consumers (services), handling scenarios where a producer needs to communicate with a specific consumer.

2. Message Bus: Suited for more complex scenarios involving multiple services and diverse communication patterns. It offers advanced features like message routing, transformation, and integration, making it effective for managing communication in distributed systems with diverse interactions.

In summary, use a message queue for straight forward asynchronous communication, or opt for a message bus when you need a more comprehensive solution with enhanced routing, integration, and scalability capabilities.

Implementation

Now that you’re familiar with messaging and queuing technologies, let’s put them into practice. Consider this scenario: a payment system where a user can send and receive money. When payment is completed, both sender and receiver will receive a promotional email and SMS notification. So there are 3 services, payment.API,promotional.Email and promotional.SMS.

I'll use .NET 8, RabbitMQ, and MassTransit to build this system.

MassTransit is an open-source distributed application framework for .NET that provides a consistent abstraction on top of the supported message transports. The interfaces provided by MassTransit reduce message-based application complexity and allow developers to focus their effort on adding business value.

Basically, masstransit is a wrapper messaging technologies (most of the messaging frameworks and technologies) providing high level abstraction to use any types of messaging or queuing technologies with added functionalities like sagas, retries, transactional, and more to build resilient and robust distributed system.

Here, message publisher is payment.API whereas message receivers are promotional.Email and promotional.SMS.

An exchange is a virtual entity within RabbitMQ that receives messages from producers and routes them to queues based on certain rules. It acts as a central hub for message distribution. There are several types of exchanges in RabbitMQ, but the most common ones are:

1. Direct Exchange: This is the simplest type. Messages are routed to queues based on an exact match between the routing key specified by the producer and the binding key of the queue.

2. Fanout Exchange: This exchange broadcasts messages to all queues.

3. Topic Exchange: This exchange allows for flexible routing based on patterns in the routing key.

4. Headers Exchange: This exchange routes messages based on message headers, not the routing key.

Routing determines how messages are delivered from an exchange to specific queues. Routing Key => A string set by the producer when publishing a message. It's used to determine the destination queue. Binding Key => A pattern associated with a queue. It defines which messages the queue will receive.

By default, MassTransit creates an exchange called amq.default.This is a direct exchange, meaning messages are routed based on an exact match with the routing key.

So, wherever sender transfer fund to a receiver, both of them will get an email and SMS.

payment.API's swagger UI.

1. Seed payment accounts. It will generate random users and other information.

2. View all account.

3. Copy account no and transfer fund.

4. You can check both account in GET request.

Payment transferred from payment.API like so:

This message will be published and processed by other services.

SMS service:

Email service:

This solution offers fault tolerance and resiliency. If either the SMS or email service experiences downtime, messages will be queued and processed when the service becomes available again.

Full source code: Github

Messaging is the solution?

No. It depends. A fundamental principle in distributed system design is to minimize direct communication between services. However, preferring asynchronous communication over sync introduces a tradeoff: request-response.

The decision to implement messaging should be deliberate and based on a careful evaluation of project requirement and trade-offs.

Summary

This article explores asynchronous messaging in distributed systems, focusing on how messaging facilitates communication between services without requiring direct interaction and key concepts like message queues and message buses, explaining their roles in decoupling services, improving reliability, and enhancing fault tolerance. Using RabbitMQ, MassTransit, and .NET 8, I demonstrate building a payment system that sends email and SMS notifications highlighting the benefits of asynchronous messaging, exploring concepts like message queues, message buses, and various protocols (AMQP, MQTT, STOMP). I will try to publish more articles on building data consistency over the system, resiliency, and security next.

References

Messaging: NDC presentation by Chris Patterson

RabbitMQ: docs

MassTransit: Good intro to MassTransit by Milan Jovanović, Nick Chapsas Tutorial

More: About Saga Pattern, Messaging Protocols, Distributed system design fundamental course by Udi Dahan, Async communication in Distributed System

Diagram: draw.io

2 words "distributed" and "system" meaning a system which is built up of multiple components that work together as-a-whole. For instance, consider an online platform like Amazon. It is made up of various services to handle inventory management, authentication, order processing, and payments and many more. These services must communicate effectively to ensure that users experience a seamless shopping experience. For example, when a customer places an order, the system needs to update the inventory, process the payment, and prepare the shipment. These components may be separated physically across different networks or located on a same network.

System separated Physically vs. Local

Firstly, two aspects to understand: components separated physically and components on the same network.

Building a bad distributed system is straightforward, just call an RPC on another service and ignore resiliency, data consistency, fault tolerance, security, and other crucial aspects. However, building a robust, efficient, and reliable distributed system is challenging and complex.

Now its clear about "being distributed", questions like how to establish communication between these components, have data consistency all over the component, handle failures, and many more arises.

Communication

There are several ways to establish communication between services. Effective communication is fundamental to ensuring that the system operates as a cohesive unit. In general, communication can be categorized into two types: synchronous and asynchronous.

Synchronous communication

In synchronous communication, a service makes a request and waits for a response before proceeding. Also know as the request-response pattern. Basically some service A request to service B and waits for service B to process the request. Example using synchronous communication:

private List<string> _database = new List<string>();
public async Task<bool> CreateNewCustomer(string customer)
{
    _database.Add(customer);
    // also save to service B
    using HttpClient client = new HttpClient();
    HttpResponseMessage response = await client.PostAsync("https://api.serviceb.com/v1/welcome/sendEmail", new StringContent(customer));
    response.EnsureSuccessStatusCode();
    return response.IsSuccessStatusCode;
}

In this example, Service A first adds the customer to the local _database with _database.Add(customer); and then makes a POST request to Service B to send a welcome email. This is a common practice where, upon a new user's registration, a welcome email is sent to enhance user engagement.

Although this function is async, the interaction between Service A and Service B is synchronous in practice. When await is used to POST request service B, it pauses the execution of the CreateNewCustomer() (but doesn't block thread) until the awaited task completes (more on async await). During this pause, control returns to the calling method, and the thread is free to perform other operations. This means Service A waits for Service B's response before moving forward. The function does not complete until the response from Service B is received, effectively making the interaction synchronous.

What ifs ...

1. Service B is slow or down? Service B might be down for maintenance.

2. Service B throws an error or Service B throttles due to high number of users registration. Explanation: If a large number of users register, Service A processes the registrations, but when it tries to send welcome emails, Service B might throttle the requests due to high volume. A common approach to manage email sending is through queuing. However, queues have limits and can only handle a certain number of requests at a time. If the queue is full, additional requests may be delayed or rejected, leading to throttling issues.

3. After sending welcome email i would also like to include them in some sort of marketing camp. Adding another call to Service C?

   1. What if Service A (creating customer), and Service B(sending welcome email) succeed but service C (marketing camp) failed?

Asynchronous communication

In asynchronous communication, a service makes a request but doesn't wait for the response. It just moves forward. Basically some Service A requests Service B without waiting for Service B response.

Some ways to achieve asynchronous communication are:

Messaging

Basically sending data in a non-blocking way using some sort of queuing system. A message queue can be defined as a process where messages (data) are stored in a queue and then some other services processes it. Working mechanism: Some Service A sends data (a producer) to a message queue, then some service B processes (a consumer) the queue accordingly. The queue acts as a buffer which holds messages until they are processed by a consumer (or delete after some time). Messaging solves a crucial issue we had in synchronous communication, i.e. what if some n number of services are added to increase user enhancement:

3. After sending welcome email i would also like to include them in some sort of marketing camp. Adding another call to Service C?

Using a message queue, any number of services can consume the messages from the queue. Queues can further be categorized into topics. Topic helps to categorize messages(data). Messaging can be visualized like so:

In summary, messaging with queuing technology provides a robust and resilience way to achieve asynchronous communication.

Pub-Sub pattern

The publish-subscribe (pub-sub) pattern is another asynchronous way to commute between services. Simply understanding pub-sub, a service (publisher) publishes a message and other services (subscribers) interested in that publisher receives that message. This is very similar to YouTube channel where user can subscribe to their favorite creator. User will be notified when their fav creator publish a video.

Publisher: Service which sends a message to a message broker. Publisher sends message without knowing who will receive them.

Message broker: Sorta middleman whose job is to distribute message from Service A to Service B or from Service A to multiple services. It ensures messages are delivered to all interested subscribers.

Subscriber: Service which receives message through a broker.

Pub-sub pattern can be visualized as below:

Pub-sub using Web sockets

One way to implement pub/sub pattern between services is to use web sockets. Web socket provides a real time, point to point communication channel in a long lived connection between services. Simply understanding, using web sockets allows a service to send messages to other services instantly and continuously without reconnecting every single time. There are 2 key components in web sockets: publisher and subscriber (same as above). The publisher broadcasts(send) messages to all subscribers connected to the Web Socket, allowing for immediate and continuous communication.

Pub-sub pattern using web socket can be easily implemented in .NET applications using SignalR. (I'll go more into this later). SignalR simplifies working with web socket and provides high level abstraction to jump right into logic without worrying about low-level details.

Pub-sub using web socket can be visualized as below:

So basically, Pub-sub using web socket (SignalR) enables direct and close to real-time communication between services (publisher and subscribers).

Event-driven approach

Event-driven approach (or Event-driven Architecture) (EDA) is an pattern where systems operate based on some events. EDA can be explained using a simple example of vending machine. Lets say you want to get a drink (Coke) from a vending machine:

1. You select your choice of drink (i.e. Coke) (ItemSelected)

2. Then you make a payment using some sort of online payment vendors or cash PaymentCompleted

3. The vending machine verifies the payment (after PaymentCompleted event).

4. If payment was successful, the vending machine will dispense the drink. (DrinkDispensed)

Each action generates an event. Generated events trigger more events. Events drive the flow of the system.

A key pattern to observe in above bullet points, all events are past tensed. This is because an event represent an action or changes that has already occurred in the system. Event cannot be ItemSelected or CompletePayment, or DispenseADrink, etc.

In EDA, a system is designed to respond to actions that have already occurred and perform subsequent operations based on these actions. The ordering of events is critical, as the sequence of actions determines the flow of the system. Additionally, ensuring transactional integrity is essential to maintain consistency and reliability within the system. System can be "transactional" by using Saga pattern, and event sourcing pattern (More on this later).

If i had to design a software system for vending machine:

Here,

• Event(Service) Bus : Routes events between different services. It ensures events are properly delivered to handle them. This can be a Service Z.

• Service A : Can be a frontend where user will select their choice of drink. This will generate ItemSelected event.

• Service B : User makes a payment for their choice of drink. This will generate PaymentCompletedevent.

• Service C : User will receive their choice of drink. This will generate DrinkDispensed event.

That's how a typical EDA can be explained but this is not perfect for real world scenario.

What ifs..

• In Service A, there is no coke (selected drink)?

• Payment failed?

• DrinkDispensed occured before PaymentCompleted or inappropriate ordering of events? (more about Event Sourcing in later articles)

• Some services are unresponsive.

• Duplicate events. DrinkDispensed occured twice (well, it can be. What if the user paid for 2 drinks? But what about if PaymentCompleted occured twice?).

• Loss of a event. User paid for a drink but no response. Why? Services being unresponsive, loss of a event in-between?

In Event-Driven Approach (EDA), it is crucial to conduct thorough requirements gathering to identify all possible actions and events. Implementing robust error handling, ensuring transactional integrity, and incorporating fault tolerance strategies are essential for maintaining system reliability and efficiency in real-world scenarios. These measures ensure that the system can handle unexpected situations, such as errors, service unresponsiveness, and event inconsistencies, effectively.

Summary

In this article, I covered the basics of distributed systems, including the differences between geographically and locally distributed systems. Effective communication between services can be synchronous or asynchronous, with methods like messaging and pub-sub patterns enhancing decoupling and resilience. The Event-Driven Approach (EDA) was explored as a method for handling operations based on events, emphasizing the need for thorough event handling, error management, and maintaining transactional integrity. Proper implementation of these concepts is essential for building reliable and efficient distributed systems. Stay tuned for more detailed discussions on each topic.

Introduction

Categorizing and processing data based on some context is called pattern matching. By context I mean specific characteristics, properties, or structure of the data that we are analyzing. This context provides the criteria or patterns against which we match and classify the data, allowing us to make informed decisions and take appropriate actions. Pattern matching can help make codebase look much more concise. In this article, I will introduced you to various types of pattern matching.

Basic Patterns

Basic pattern are simple building blocks of pattern matching which is used to compare an expression's value against simple criteria.

Here are some basic pattern:

Constant pattern

It allows you to match a specific constant value against the value being evaluated. Constant pattern makes conditional logic concise and introduce better readability.

Eg: Using is operator.

string message = "Welcome back!";
if(message is null){
   Console.WriteLine("Message is null");
}
// Output: Since, condition is not fullfiled, it won't print message.

You can use is operator to check for specific match. C# version 9.0 introduced not operator. Now, you check of specific match like:

string message = "Welcome back!";
if (message is not null){
   Console.Writeline($"Message is not null. {message}");
}
// Output: Message is not null. Welcome back!

Constant patterns can be used with various other expressions like variables, literals, properties, method calls, and more.

Relational Pattern

Similar to constant pattern which checks for a constant specific value, relational pattern checks for a constant specific value and apply relational operators. Imaging if you want to write a logic which checks if the message is not null and message's length is greater than 20.

string message = "Welcome back!"; // length = 13
if(message is not null && message.Length is not < 20)
    Console.WriteLine(message);
else Console.WriteLine("Welcome back to FC 24!");
// output: Welcome back to FC 24!

Relational pattern is same as constant pattern but you can use relational operators.

Type Pattern

As the name says, type pattern checks if the value is of some specific type. Imagine a scenario where you have an object which may be of type string or int. If it is string, its a name else its an age. You can perform this conditional logic by:

void isNameOrAge(object data)
{
    if (data is string name)
        Console.WriteLine($"{name} is a name.");
    else if (data is int age)
        Console.WriteLine($"{age} ia an age.");
    else Console.WriteLine("Invalid");
}
isNameOrAge("ram bahadur"); // prints: ram bahadur is a name.
isNameOrAge(21); // prints: 21 ia an age.

You can read this: if (data is string name) ... like if data's type is string, assign that value to name, else if (data is int age) ... else if it is int, assign that value to age.

Declaration Pattern

This is similar to type pattern but declares a new value if an condition is fulfilled. Imagine a scenario where you need to write a logic to check if a person's age is greater than 21. But age can be of type string or int.

void isGreaterThan21(object age)
{
    if (age is int ageIntValue || (age is string ageStringValue && int.TryParse(ageStringValue, out ageIntValue) && ageIntValue is < 21))
        Console.WriteLine($"Age is {ageIntValue}. You cannot play this game."); 
    else
        Console.WriteLine($"You can play this game.");
}
isGreaterThan21("42");
isGreaterThan21(15);
/*
Output: 
       You can play this game.
       Age is 15. You cannot play this game.
*/

Let me break down this code: if (age is int ageIntValue || (age is string ageStringValue && int.TryParse(ageStringValue, out ageIntValue) && ageIntValue is < 21))

In my first or condition, I checked if age is int. If it is int then, create a new variable of type int called ageIntValue and assign object age's value to it. It won't check for another condition. After that it will move on to another line which prints out some message.

If my first or condition fails, it will check second condition. In second condition, I've checked if it is string. If it is an string, then create a new variable of type string called ageStringValue and assign object age value to it. After that I have an and operator which parse ageStringValue into int and assign it to ageIntValue. Finally, another and operator which checks if ageIntValue is greater than 21 (using relational pattern).

Switch Expression

Switch expression were introduced in C# version 8.0 to write more concise switch "statement". Instead of chaining if() else(), switch statement is much more versatile and concise.

Switch expression is a evolution of switch statements with added benefit of chaining pattern matching.

Here is simple switch statement which prints about fruit:

public void GetFruitInfo_SwitchStatement(string fruitName)
{
    switch (fruitName.ToLower())
    {
        case "apple":
            Console.WriteLine("An apple a day keeps the doctor away!");
            break;
        case "banana":
            Console.WriteLine("Banana is good for potassium.");
            break;
        case "orange":
            Console.WriteLine("Oranges are a great source of Vitamin C.");
            break;
        case "mango":
            Console.WriteLine("Mangoes are delicious and refreshing.");
            break;
        case "grapes":
            Console.WriteLine("Grapes come in many varieties, red, green, and purple!");
            break;
        case "strawberry":
            Console.WriteLine("Strawberries are perfect for adding a sweet touch to desserts.");
            break;
        default:
            Console.WriteLine("Not a fruit.");
            break;
    }
}
// calling code
GetFruitInfo_SwitchStatement("grapes");
/*
Output: Grapes come in many varieties, red, green, and purple!
*/

Now, I will change the same switch statement to switch expression:. Before that I would like to let you know that void is not supported for switch expression. You can return Action for similar (void) result.

public string GetFruitInfo_SwitchExpression(string fruitName) => fruitName.ToLower() switch
{
    "apple" => "An apple a day keeps the doctor away!",
    "banana" => "Banana is good for potassium.",
    "orange" => "Oranges are a great source of Vitamin C.",
    "mango" => "Mangoes are delicious and refreshing.",
    "grapes" => "Grapes come in many varieties, red, green, and purple!",
    "strawberry" => "Strawberries are perfect for adding a sweet touch to desserts.",
    _ => "Not a fruit."
};
// calling code
Console.WriteLine(GetFruitInfo_SwitchExpression("strawberry")); 
/*
Output: Strawberries are perfect for adding a sweet touch to desserts.
*/

This is much more concise. But conciseness isn't the only advantage of switch expressions. It's the combination of switch expressions with pattern matching that makes them truly versatile and powerful. To demonstrate this, lets modify our parameter fruitName from string to an object type. I will create an immutable data which consist detail of the fruit.

public record Fruit(string name, decimal cost, string producedCountry);

public string GetFruitInfo_SwitchExpression(Fruit fruit) => fruit switch
{
    { name: "apple", costInUSD: < 0.5m, producedIn: "Nepal"} => "An apple from Nepal which is cost efficient.",
    { name: "apple", costInUSD: > 2, producedIn: not "Nepal"} => "An apple which is not from Nepal and is expensive.",
    { name: "apple", costInUSD: < 1,  producedIn: _ } => "A juicy apple.",
    { name: "apple", costInUSD: _ , producedIn: _ } => "An apple.",
    _ => "Not an apple."
};
// calling code
Fruit fruit = new Fruit(name: "apple", costInUSD: 0.75m, producedIn: "Nepal");
Fruit fruit1 = new Fruit(name: "apple", costInUSD: 4.5m, null);

Console.WriteLine(GetFruitInfo_SwitchExpression(fruit)); 
Console.WriteLine(GetFruitInfo_SwitchExpression(fruit1)); 
/*
Output:  A juicy apple.
         An apple which is not from Nepal and is expensive.
*/

This switch expression leverages pattern matching (Constant Pattern using not, and Relational Pattern using < > ) to compare the properties of the fruit object against different cases. Each case is enclosed in curly braces {}.

Each case in the expression doesn't explicitly use a return statement. This is because of lambda. You can find more about lambda and closed variables here. Basically result of this lambda function is the return type/value of GetFruitInfo_SwitchExpression() function. _ is known as discard. I will explain this next, for now think this as default of switch statement.

Discard Pattern

Discard is denoted by _ (underscore). Discard pattern is used when you don't want a value or you don't care about something. Discard pattern automatically infer type, meaning _'s type will be automatically inferred based on the context(value assigned to it). Discarded value will also not be stored in a allocated memory, but stored in a temporary variable which will be collected by garbage collector.

Examples:

string someVal = "44";
if (int.TryParse(someVal, out _ )) // discard pattern
    Console.WriteLine("someVal is a int.");
// output: someVal is a int.

public string GetFruitInfo_SwitchExpression(Fruit fruit) => fruit switch
{
    { name: "apple", costInUSD: < 0.5m, producedIn: "Nepal"} => "An apple from Nepal which is cost efficient.",
    { name: "apple", costInUSD: > 2, producedIn: not "Nepal"} => "An apple which is not from Nepal and is expensive.",
    _ => "Not an apple." // any Fruit except apple will reach here.
};
GetFruitInfo_SwitchExpression("orange");
// output: Not an apple.

Deconstruction Patterns

Deconstruction pattern are simply deconstructing an object(extracting information) and applying your conditional logic. Before moving on to some deconstruction pattern, let's look at deconstruct in C#.

Deconstruct

Deconstructing was introduced in C# version 7.0. Main motive of deconstruct is to pull out data from an user defined type.

Deconstructing is done by adding a method of type void with name Deconstruct. This method's parameter are those which you want the user of this object to pull out. Deconstruct method takes only out parameter. Why? Because Deconstruct method needs to return back value to the caller which is not done by return statement but out parameter. Why? Deconstruct method extract values outside the method. So when you pass a parameter to Deconstruct, the method assigns the extracted values directly to those parameter.

Here's a simple example:

public struct LaptopRecommendation(decimal _budget, string _brand)
{
    public decimal budget = _budget;
    public string brand = _brand;

    public void Deconstruct(out decimal LaptopBudget, out string LaptopBrand) => (LaptopBudget, LaptopBrand) = (budget, brand);
}

Now, when any caller use LaptopRecommendation object, they can leverage Deconstruct method. Basically the caller can easily unpack LaptopRecommendation object's property (budget and brand) into a separate variables so, the caller can efficiently interact with the LaptopRecommendation object's data.

For example, lets say I have a list of LaptopRecommendation objects. I want to store any LaptopRecommendation object whose brand is Acer and laptop whose cost is less than 1300.

// list of LaptopRecommendation object
var recommendations = new List<LaptopRecommendation>();
recommendations.Add(new LaptopRecommendation(1200, "acer"));
recommendations.Add(new LaptopRecommendation(600, "dell"));
// list of laptop whose brand is acer and cost is less than 1300
var acerRecommendations = recommendations
                        .Where(recommendation =>
                        {
                            (decimal budget, string brand) = recommendation;
                            return brand is "acer" && budget < 1300;
                        })
                        .ToList();
Console.WriteLine(string.Join(", ", acerRecommendations.Select(x => x.brand)));
// output: acer

In acerRecommendations's where filter, I've create two variables budget and brand of type decimal and string respectively and assigned them to recommendation (single object of List<LaptopRecommendation>). recommendation deconstructs and assign value in these newly created variables. LaptopRecommendation's Deconstruct method contains two out params, they will be assigned to our two newly created variables(budget and brand).

Here are some deconstruction pattern:

Property pattern

Property pattern is just a trick to deconstruct an object and match specific property within the object. What I mean by that is, it allows you to check for a specific condition within the property or field within that object. Eg: Check if an Person object's property Age is greater than 18. Here, we deconstruct Person object (extracting properties' values) and check the property Age if its greater than 18.

Syntax: expression is {property1: someValue1, property2: someValue2, ...}

Eg: Using property pattern

// Person object.
public class Person(int _age, string _name, DateTime _dob)
{
    public int Age { get; set; } = _age;
    public string Name { get; set; } = _name;
    public DateTime DateofBirth { get; set; } = _dob;

    public override string ToString()
    {
        return $"Name: {Name}, Age: {Age}, Date of birth: {DateofBirth}";
    }

    public void Deconstruct(out int personAge, out string personName, out DateTime personDateofBirth) => (personAge, personName, personDateofBirth) = (Age, Name, DateofBirth);
}

// initialize person object
var person1 = new Person(17, "ram bahadur", DateTime.Parse("2004-01-01"));
Console.WriteLine(person1.ToString()); // prints person object info
if(person1 is { Age: < 21 })
    Console.WriteLine($"{person1.Name} is not eligible to drink beer. Try milk.");
else Console.WriteLine("Here is your beer. Drink responsibly.");
/* 
Output: 
Name: ram bahadur, Age: 17, Date of birth: 1/1/2004 12:00:00 AM
ram bahadur is not eligible to drink beer. Try milk.
*/

Using property pattern when working with DateTime is very useful. Imagine a scenario where you need to check if date is between some particular time. Eg: I would like to know if a person is born in March.

bool isBornInMarch(DateTime dob)
{
    if (dob is { Month: 3 })
        return true;
    else return false;
}
Console.WriteLine($"{person1.Name} is born is march? {isBornInMarch(person1.DateofBirth)}."); 
/*
Output: ram bahadur is born is march? False.
*/

Similarly, you can deconstruct an object and apply your conditional logic using property matching.

Positional Pattern

Similar to property pattern which checks deconstruct an object and applies condition to property, positional pattern deconstruct object and checks for defined condition.

Imagine a scenario where you need to write a function which takes budget and brand as a parameter and return appropriate laptop.

Firstly, I will create an data structure to hold both of these value. This can be tuple as well but in this example I will create an struct to hold values. Here, object is of struct type but you can create of reference type as well.

public struct LaptopRecommendation(decimal _budget, string _brand)
{
    public decimal budget = _budget;
    public string brand = _brand;

    public void Deconstruct(out decimal LaptopBudget, out string LaptopBrand) => (LaptopBudget, LaptopBrand) = (budget, brand);
}

Main function to check the conditional logic:

string getLaptopDetail(LaptopRecommendation info) => info switch
{
    ( <= 1000 and > 500, "acer") => "You should get a Acer Aspire 5.",
    ( <= 1400 and > 1000, "acer") => "You should get a Acer Nitro.",
    ( <= 1500 and > 1000, "acer") => "You should get a Acer Predator.",
    ( <= 1000 and > 500, "dell") => "You should get a Dell Inspiron.",
    ( <= 1400 and > 1000, "dell") => "You should get a Dell G series PC.",
    ( <= 1500 and > 1000, "dell") => "You should get a Dell Alienware.",
    ( >= 3000, object) => "You should either get a Mac Pro M2 Series or a beefy razer series laptop. (I would get a beefy razer laptop)",
    _ => "I recommend you get any laptop from Lenovo Legion series.",
};

More

Before looking at some special pattern like List Pattern, I would like to make sure you understand ternary and null coalescing operator.

Null coalescing

Null coalescing was introduced in early version of C# (2.0). Null coalescing or ?? operator, is a way to handle null values. ?? is a binary operator, meaning it operates in two operands (left and right).

string someData = null;
string someDataBak = "N/A";
string returnData = someData ?? someDataBak;
Console.WriteLine(returnData);

You can read someData ?? someDataBak; like, if someData not null, use it else use someDataBak.

Ternary Operator

Conditional operator or ternary operators is a way to perform conditional operation in a concise way. Syntax: condition ? expression1 : expression2 or you can read this as: if condition is true, compute expression1 else expression2.

Simple example to check user's age to vote using ternary:

int age = 17;
string consoleMessage = age > 18 ? "Eliglble to vote." : "Not eligible to vote";
Console.WriteLine(consoleMessage); 
/*
Output: Not eligible to vote
*/

Special Pattern

Var pattern

Var pattern was introduced in C# version 7.1. As the name say "var", var pattern is used when you want something to be stored in var and utilize it for some expression. Var pattern is very similar to type pattern. Instead of explicitly specifying type, you use var.

Example: Imagine you have a list of random objects. You want to know if this random list contains more than two animals.

var listOfAnimal = new List<string>() { "dog", "cat", "rhino"};
var listOfRandomItems = new List<object>() { "Ram", "hari", "John", 421, "bijay", "macafee", "apple", "dog", "cat" };
bool containsAnimal(List<object> randomStuff)
{
    return randomStuff.Intersect(listOfAnimal) is var animalCount && animalCount.Count() > 2;
}
Console.WriteLine($"listOfRandomItems contains animal? : {containsAnimal(listOfRandomItems)}");
/*
Output: listOfRandomItems contains animal? : False
*/

Instead of explicitly defining type like List<object> or IEnumerable<object>, you can simply use var pattern.

List pattern

Introduced in C# version 11.0, list pattern is a very concise way to condition a list. Meaning list pattern helps you to filter and match certain condition in a list without having to write lengthy code.

For example, I have a list and I would like to check if the first value of that list is Blueskie, check third value name is Kumar, and check if last value of that list is Bret.

First case: Check if the first value is "Blueskie"

var simpleListOfName = new List<string> { "Blueskie", "Jon", "Jonsky", "Angel", "Bret" };
if(simpleListOfName is ["Blueskie", _, _, _, _])
    Console.WriteLine("First name is Blueskie.");
/*
Output: First name is Blueskie.
*/

Here in the first case, I only care about 1st value of the list. So I discarded other values (_).

Second case: Check if the third value is "Kumar"

if(simpleListOfName is [_, _, "Kumar", _, _])
    Console.WriteLine("Third name is Kumar.");
else Console.WriteLine("Third name is not kumar.");
/*
Ouput: Third name is not kumar.
*/

Here in the second case, I only care about 3rd value of the list. So I discarded other values (_).

Third case: Check if the last value is "Bret"

if (simpleListOfName is [.., "Bret"])
    Console.WriteLine("Last name is Bret.");

Here in the third case, I only care about last value of the list. I don't really care about values till last. So, I can use slice sub-pattern ( .. or two dots ). Slice pattern can be used only in list pattern. Also, C# compiler prohibits using more than one slice pattern.

Another example using list pattern with switch expression. Imagine a scenario where you want print first and fourth value from a list.

var listOfNumbers = new List<int> {765, 2, 34, 5, 2, 3, 33, 66 };
var firstAndFourth = listOfNumbers is [int firstVal, _, _, int fourthVal, ..] ? $"First num is:{firstVal}, fourth num is: {fourthVal}" : "N/A";
Console.WriteLine(firstAndFourth);
/*
Output: First num is:765, fourth num is: 5
*/

Another example, print second and last value from a list using switch expression.

var listOfNumbers = new List<int> { 765, 2, 34, 5, 2, 3, 33, 66 };
var numResult = listOfNumbers switch
{
    [_, int second, ..] => $"Second num is: {second}",
    [.., int last] => $"Last num is: {last}",
    _ => "I dont care",
};
Console.WriteLine(numResult);
/*
Ouput: Second num is: 2
*/

In switch expression using list pattern when condition is matched, only that matched condition is executed. Therefore, above example only prints Second num is: 2.

Another example of list pattern is using it in CSV file. I have a CSV data of employees.

dawg@example.com,1,Tom,Thomson,founder
laura@example.com,4,Laura,Grey,manager
craig@example.com,8,Craig,Johnson,deputymanager
mary@example.com,77,Mary,Jenkins,engineer
jamie@example.com,101,Jamie,Smith,intern
mikey@gmail.com,00,Mike,Paul,clown

I would like to provide some information based on the employee. Like, greet them if they are manager, and let them know how early they joined the company (based on their id which is sequential).

using var emp = new StreamReader("../../../email.csv");
while (!emp.EndOfStream)
{
    string[]? empInfo = emp.ReadLine()?.Split(',');
    var greetAccordingly = empInfo switch
    {
        [_, _, _, string lastName, "manager"] => $"Good Evening, Manager {lastName}.",
        [_, string id, _, ..] => int.TryParse(id, out int empId) && (empId > 2 && empId < 10) ? "You are top 10 employee to join da Company." : "You joined the company after 10 people joined.",
        [_, string id, ..] => int.TryParse(id, out int empId) && empId is 1 ? "Hello Mr CEO." : "Well, Hello there :-) ",
        _ => "Nothing matched."
    };
    Console.WriteLine(greetAccordingly);
}
/*
Output: 
        You joined the company after 10 people joined.
        Good Evening, Manager Grey.
        You are top 10 employee to join da Company.
        You joined the company after 10 people joined.
        You joined the company after 10 people joined.
        You joined the company after 10 people joined.
*/

Another example, check employee email for verification (on same above CSV file). Basically, email with the extension @example.com is verified.

using var emp = new StreamReader("../../../email.csv");
while (!emp.EndOfStream)
{
    string[]? empInfo = emp.ReadLine()?.Split(',');
    var checkVerifiedEmployee = empInfo switch
    {
        [string email, ..] => email.Split("@").Contains("example.com") ? $"{email} is Verified Example worker." : $"{email} is Unverified user.",
        _ => "Nothing matched."
    };
    Console.WriteLine(checkVerifiedEmployee);
}
/* Output:
           dawg@example.com is Verified Example worker.
           laura@example.com is Verified Example worker.
           craig@example.com is Verified Example worker.
           mary@example.com is Verified Example worker.
           jamie@example.com is Verified Example worker.
           mikey@gmail.com is Unverified user.
*/

You can learn more about List pattern :

1. Microsoft docs

2. endjin's blog

Summary

• Pattern matching offers a versatile way to check data against various condition, including types, values, relational operators, and object properties.

• Basic Patterns: Describes constant, relational, type, and declaration patterns with examples.

• Switch Expression: Switch expressions introduced in C# 8.0 allow chaining of pattern matching conditions, which results to more concise and readable code compared to traditional if-else statements.

• Deconstruct method is used to extract specific properties from objects for easier manipulation.

• You can combine multiple relational operators with logical operators (&& - and, || - or) to create complex conditions within a single pattern.

• Discard Pattern: Use the _ pattern to ignore irrelevant values during matching.

• Deconstruction Patterns: Describes deconstruction patterns and their use in extracting information from objects. Pattern matching facilitates deconstructing objects to extract specific properties, simplifying data manipulation.

• Null Coalescing and Ternary Operators: Briefly discusses null coalescing and ternary operators for handling null values and conditional operations.

• Special Patterns: Covers var pattern and list pattern, explaining their syntax and applications with examples.

• Using clear and descriptive patterns, code becomes easier to understand and scalable.

Sourcecode.

Introduction

SOLID is an abbreviation which translates to some set of principles created by Robert Cecil Martin a.k.a Uncle Bob. SOLID principle states that the object must have single responsibility (S), it must be open for extensions but closed for modification (O), child class must be able to replace parent class (L), unwanted behavior must be excluded from the object (I), use abstractions to ensure modules depends on interfaces rather than concrete implementation (D).

Single Responsibility

First principle of SOLID stands for Single Responsibility. It means that any unit or block of code, whether its a class, or method, or a function must have single responsibility. Each unit should be responsible for only one functionality. This makes the software more easy to debug. Therefore, it states that your unit of code must be specific.

Demonstration

Here's an scenario demonstrating good design implementing single responsibility and bad design not implementing it.

Imagine a auth module in an web application where user needs to register themself. If user inputs correct information, they will receive an email confirming their identity for this web application.

First approach. A more naive (Not implementing Single Responsibility).

/* Data model used for demonstration */
public class UserInputs
{
    public required string UserName { get; set; }
    public required string Email { get; set; }
    public required string Password { get; set; }
}

public class Auth(UserInputs inputs)
{
    private UserInputs _inputs = inputs;
    private string[] Users = ["ram", "sam", "john", "tom", "hari"];
    private string[] Emails = ["ram@mail.com", "sam@mail.com", "john@mail.com", "tom@mail.com", "hari@mail.com"];

    public bool VerifyUser()
    {
        if (string.IsNullOrWhiteSpace(_inputs.UserName) || string.IsNullOrWhiteSpace(_inputs.Email) || string.IsNullOrWhiteSpace(_inputs.Password))
        {
            Console.WriteLine("Username, Email and Password are required field.");
            return false;
        }
        if (Emails.Contains(_inputs.Email))
        {
            Console.WriteLine("Email already exists. Please sign in.");
            return false;
        }
        if (Users.Contains(_inputs.UserName))
        {
            Console.WriteLine("User name already exists.");
            return false;
        }
        if (_inputs.Password.Length < 8)
        {
            Console.WriteLine("Password length must be atleast 8 characters.");
            return false;
        }
        Console.WriteLine($"New user {_inputs.UserName} has been created.");
        // user's inputs seems fine, now send them email confirming their identity.
        Console.WriteLine("--- New email from Auth Module ---");
        Console.WriteLine($"Account has been created successfully. \n" +
            $"Username: {_inputs.UserName} \n" +
            $"Email: {_inputs.Email} \n" +
            $"Password: {_inputs.Password} \n");
        return true;
    }
}

Above code violates the Single Responsibility Principle as the method should have a singular focus. VerifyUser() has multiple responsibility like validating user, creating user, and sending confirmation email. If I had to modify my validation logic, I need to modify VerifyUser(), which consists of multiple responsibility. Therefore, this codebase will become hard to test and maintain.

// calling code and output when single responsibility principle is applied
var user1 = new UserInputs()
{
    UserName = "sushant",
    Email = "sushant@mail.com",
    Password = "password"
};

badExampleCode::Auth badExample = new badExampleCode::Auth(user1);
badExample.VerifyUser();
/*
Output: New user sushant has been created.
        --- New email from Auth Module ---
        Account has been created successfully.
        Username: sushant
        Email: sushant@mail.com
        Password: password
*/

Before moving on, you can try refactoring it to enhance the design.

Here's another approach which is much more better than the first approach and follows single responsibility principle.

In the above code, we need to separate 3 things to achieve single responsibility: Validation, Registration, and Confirmation email. So I will add: ValidateUser() for validation, RegisterUser() for registration and SendEmail() for email confirmation. Sending email is useful in other modules as well. So I will create a separate class for it.

public class Auth(UserInputs inputs)
{
    private UserInputs _inputs = inputs;
    private string[] Users = ["ram", "sam", "john", "tom", "hari"];
    private string[] Emails = ["ram@mail.com", "sam@mail.com", "john@mail.com", "tom@mail.com", "hari@mail.com"];

    public bool VerifyUser()
    {
        if (ValidateUser() && RegisterUser())
        {
            /* when everthing goes fine */
            string toEmail = _inputs.Email;
            string subject = "New email from Auth Module";
            string body = $"Account has been created successfully. \n" +
                            $"Username: {_inputs.UserName} \n" +
                            $"Email: {_inputs.Email} \n" +
                            $"Password: {_inputs.Password} \n";
            EmailHelper email = new EmailHelper();
            email.SendEmail(toEmail, subject, body);
            return true;
        }
        /* when something goes wrong */
        return false;
    }

    public bool ValidateUser()
    {
        if (string.IsNullOrWhiteSpace(_inputs.UserName) || string.IsNullOrWhiteSpace(_inputs.Email) || string.IsNullOrWhiteSpace(_inputs.Password))
        {
            Console.WriteLine("Username, Email and Password are required field.");
            return false;
        }
        if (Emails.Contains(_inputs.Email))
        {
            Console.WriteLine("Email already exists. Please sign in.");
            return false;
        }
        if (Users.Contains(_inputs.UserName))
        {
            Console.WriteLine("User name already exists.");
            return false;
        }
        if (_inputs.Password.Length < 8)
        {
            Console.WriteLine("Password length must be atleast 8 characters.");
            return false;
        }
        return true;
    }

    public bool RegisterUser()
    {
        Console.WriteLine($"New user {_inputs.UserName} has been created.");
        return true;
    }
}

public class EmailHelper
{
    public void SendEmail(string to, string subject, string body)
    {
        Console.WriteLine($"Email to: {to}");
        Console.WriteLine($"Subject: {subject}");
        Console.WriteLine($"Body: {body}");
    }
}

This is much better design and follows single responsibility principle. This codebase promotes clearer separation of concerns which is organized, has better readability, flexible, and is easy to maintain. This results in overall betterment of software design.

At the end, building robust, flexible, and scalable software is the end goal of an ever evolving software requirements which is emphasized by following the single responsibility principle.

// calling code and output when single responsibility principle is applied
var user2 = new UserInputs()
{
    UserName = "jon",
    Email = "bones@mail.com",
    Password = "jonjones11"
};
goodExampleCode::Auth goodExample = new goodExampleCode::Auth(user2);
goodExample.VerifyUser();

/*
Output: New user jon has been created.
        Email to: bones@mail.com
        Subject: New email from Auth Module
        Body: Account has been created successfully.
        Username: jon
        Email: bones@mail.com
        Password: jonjones11
*/

Takeaway from Single Responsibility Principle

A whole single component may consist many pieces of components. A better software design would be to have each component have a single responsibility. It doesn't matter whether we consider the component as a whole or as individual pieces doing their part. Each component must have a specific responsibility.

In the first approach, VerifyUser() component had multiple responsibility. We decompose that and created multiple separate component. Finally, we compose all the component as whole.

Open/Close Principle

Second principle of SOLID stands for Open/Close Principle. It means that an software entity must be open for extension and close for modification. Software entities can be class, module, method, or a function. Therefore, it states that your software entities should allow new functionalities without hampering existing behavior. To achieve Open/Close principle, we mostly use inheritance and interfaces.

Demonstration

To demonstrate Open/Close principle, I will use the same example I used above (Auth module in an web application).

Open/Close principle state an entity should be open for extension and close for modification. I will create an abstract class which consist authentication logic.

public abstract class Authenticate
{
    private string[] Users = ["ram", "sam", "john", "tom", "hari"];
    private string[] Emails = ["ram@mail.com", "sam@mail.com", "john@mail.com", "tom@mail.com", "hari@mail.com"];

    public bool VerifyUser(UserInputs _inputs)
    {
        /* when everthing goes fine */
        if (ValidateUser(_inputs) && RegisterUser(_inputs))
        {
            string toEmail = _inputs.Email;
            string subject = "New email from Auth Module";
            string body = $"Account has been created successfully. \n" +
                            $"Username: {_inputs.UserName} \n" +
                            $"Email: {_inputs.Email} \n" +
                            $"Password: {_inputs.Password} \n";

            EmailHelper email = new EmailHelper();
            email.SendEmail(toEmail, subject, body);
            return true;
        }

        /* when something goes wrong */
        return false;
    }

    private bool ValidateUser(UserInputs _inputs)
    {
        if (string.IsNullOrWhiteSpace(_inputs.UserName) || string.IsNullOrWhiteSpace(_inputs.Email) || string.IsNullOrWhiteSpace(_inputs.Password))
        {
            Console.WriteLine("Username, Email and Password are required field.");
            return false;
        }
        if (Emails.Contains(_inputs.Email))
        {
            Console.WriteLine("Email already exists. Please sign in.");
            return false;
        }
        if (Users.Contains(_inputs.UserName))
        {
            Console.WriteLine("User name already exists.");
            return false;
        }
        if (_inputs.Password.Length < 8)
        {
            Console.WriteLine("Password length must be atleast 8 characters.");
            return false;
        }
        return true;
    }

    private bool RegisterUser(UserInputs _inputs)
    {
        Console.WriteLine($"New user {_inputs.UserName} has been created.");
        return true;
    }
}

When I implement this abstract class, authentication logic will be close for modification but open for extension.

public class Auth : Authenticate
{
    public bool IsPastUser(UserInputs _inputs)
    {
        var pastUserList = new List<string>()
        {
            "volk@mail.com", "chael@mail.com", "topuria@mail.com", "islam@mail.com", "sushantpant@mail.com"
        };

        if (pastUserList.Contains(_inputs.Email))
            return true;

        return false;
    }
}

Here, Auth implements Authenticate abstract class. Authenticate is close for modification. It is not possible to override VerifyUser() because it is not marked to be overridden (virtual). I added a new method IsPastUser() to check if the user had already registered before. I can extends this class and add more features as I want. This makes Auth class do exactly what it mean to do that is to authenticate (Close for modification) and add more features (Open for extension).

// calling code and output
var user3 = new UserInputs()
{
    UserName = "sushantpant",
    Email = "sushantpant@mail.com",
    Password = "p@ssWo3d"
};

openclosePrinciple::Auth openCloseP = new openclosePrinciple::Auth();
openCloseP.VerifyUser(user3);
if (openCloseP.IsPastUser(user3))
    Console.WriteLine($"Welcome back {user3.UserName}.");

/*
Output: New user sushantpant has been created.
        Email to: sushantpant@mail.com
        Subject: New email from Auth Module
        Body: Account has been created successfully.
        Username: sushantpant
        Email: sushantpant@mail.com
        Password: p@ssWo3d

        Welcome back sushantpant.
*/

Takeaway from Open/Close Principle

The Open/Closed Principle emphasizes software entities to be open for extension but closed for modification. This makes adding new functionalities and behavior to a software entity easy without hampering the default behavior of other components.

Liskov Substitution Principle

Third principle of SOLID stands for Liskov substitution Principle. It means that a child class should be able to replace a parent class, inheriting all the behaviors of the parent class. Therefore, according to the Liskov Substitution Principle, objects of the parent class can be replaced with objects of the child class without affecting the correctness of the program. Applying Liskov substitution principle is possible by using inheritance. If you need a new object with new functionalities but majority of it being same as some other object, you just extend that object and add your new functionalities and behavior.

Demonstration

Imagine a new requirement in our previous example of Auth Module.

For example: Send a discount coupon to a new user.

Firstly, we don't want to do any changes in the original Auth object. Just add a new feature to send discount coupon. To do this, I will create a new class called AuthWithDiscount which will be derived from Auth.

public class AuthWithDiscount : Auth
{
    public void SendDiscountCoupon(string email)
    {
        Random rand = new Random();
        int discount = rand.Next(0, 30);

        var arr = new string[] { "A", "B", "C", "Z", "X" };
        var discountCoupon = new StringBuilder();
        for(int i = 0; i < 3; i++)
        {
            discountCoupon.Append(arr[rand.Next(i)]);
            discountCoupon.Append(rand.Next(i+i));
        }

        string subject = "Discount for new user.";
        string body = $"Thanks for joining. To celebrate this, here is a discount of: {discount}%. " +
            $"Discount coupon: {discountCoupon}";
        EmailHelper emailHelper = new EmailHelper();
        emailHelper.SendEmail(email, subject, body);
    }
}

Here, AuthWithDiscount (child class) will have all behavior of Auth (parent class), with additional behavior. Now back to definition of Liskov substitution principle, child class should be able to replace parent class. In our case, AuthWithDiscount will replace Auth class and still maintain the same functionality of authenticating.

// calling code and output
var user4 = new UserInputs()
{
    UserName = "dawg",
    Email = "dawg@mail.com",
    Password = "handshakeleema"
};

AuthWithDiscount authWithDiscount = new AuthWithDiscount();
if(authWithDiscount.VerifyUser(user4))
    authWithDiscount.SendDiscountCoupon(user4.Email);

/*
Output: New user dawg has been created.
        Email to: dawg@mail.com
        Subject: New email from Auth Module
        Body: Account has been created successfully.
        Username: dawg
        Email: dawg@mail.com
        Password: handshakeleema

        Email to: dawg@mail.com
        Subject: Discount for new user.
        Body: Thanks for joining. To celebrate this, here is a discount of: 16%. Discount coupon: A0A0A1
*/

Takeaway from Liskov Substitution Principle

The Liskov substitution Principle (derived class should be able to replace base class) promotes code reusability, maintainability, scalability, and extensibility of an software object. This results in a more flexible software design.

Interface Segregation

Fourth principle of SOLID stands for Interface Segregation. Interfaces are use to define the behavior that a class has to provide. The Interface segregation principle suggests that interface must be tailored to object. It must not contain irrelevant functionalities. If a class/object only needs a subset of functionalities from an interface, it should not be burdened with unnecessary methods or properties. By defining interfaces that are object-specific, classes can implement only the interfaces that are relevant to their functionality. This promotes a cleaner and more modular design approach.

Demonstration

I will continue with my previous example of Auth module.

In Auth module, we can add a lot of feature. Let us consider 3 ways a user can register: social media account, email account, or phone number. For user from North America, all these 3 options are available. For APAC user, only social media and email account is available.

There can be multiple type of authentication. Like: auth with discount, auth for APAC user, auth for North America user, and more. Specifically auth has specific behavior. APAC has some other specific behavior. Therefore, I will create set of behavior for auth with email, auth with mobile no, and auth with social media.

public interface IEmailAuth
{
    public bool VerifyUserWithEmail(string email, string password);
    public bool IsEmailConfirmed(string email);
}

public interface IMobileAuth
{
    public bool VerifyUserWithMobile(string mobileNo, string password);
    public bool IsMobileConfirmed(string mobileNo);
}

public interface ISocialMediaAuth
{
    public bool VerifyUserWithSocialMedia(int socialMediaId, string userName, string password);
    public bool IsSocialMediaConfirmed(string userName);
}

APAC user can register themself by using email address or social media. Therefore, auth for APAC user will implement IEmailAuth, and ISocialMediaAuth.

public class AuthForAPAC : IEmailAuth, ISocialMediaAuth
{
    public bool IsEmailConfirmed(string email)
    {
        if (email is not null && email.Contains('@'))
        {
            Console.WriteLine("Email is verified.");
            return true;
        }
        return false;
    }

    public bool IsSocialMediaConfirmed(string userName)
    {
        var listOfSocialMediaUsername = new string[] { "jon", "dawg", "sushant" };
        if (userName is not null && listOfSocialMediaUsername.Contains(userName))
        {
            Console.WriteLine("Social media is verified.");
            return true;
        }
        return false;
    }

    public bool VerifyUserWithEmail(string email, string password)
    {
        if (email.Contains('@') && email.Contains('.') && password.Length > 8)
            return true;

        return false;
    }

    public bool VerifyUserWithSocialMedia(int socialMediaId, string userName, string password)
    {
        var listOfSocialMediaUsername = new string[] { "jon", "dawg", "sushant" };
        if (socialMediaId > 1 && userName is not null && listOfSocialMediaUsername.Contains(userName) && password.Length > 8)
            return true;

        return false;
    }
}

In the above code, only functionalities that is relevant to auth for APAC user is added.

Now for american user, they will have options to register with email, mobile or social media handle. Therefore, it will implement IEmailAuth, IMobileAuth, and ISocialMediaAuth.

public class AuthForAmerica : IEmailAuth, ISocialMediaAuth, IMobileAuth
{
    public bool IsEmailConfirmed(string email)
    {
        if(email is not null && email.Contains('@'))
        {
            Console.WriteLine("Email is verified.");
            return true;
        }
        return false;
    }

    public bool IsMobileConfirmed(string mobileNo)
    {
        if (mobileNo is not null && mobileNo.Contains("+123") && mobileNo.Length is 10)
        {
            Console.WriteLine("Mobile is verified.");
            return true;
        }
        return false;
    }

    public bool IsSocialMediaConfirmed(string userName)
    {
        var listOfSocialMediaUsername = new string[] { "jon", "dawg", "sushant" };
        if (userName is not null && listOfSocialMediaUsername.Contains(userName))
        {
            Console.WriteLine("Social media is verified.");
            return true;
        }
        return false;
    }

    public bool VerifyUserWithEmail(string email, string password)
    {
        if(email.Contains('@') && email.Contains('.') && password.Length > 8)
            return true;

        return false;
    }

    public bool VerifyUserWithMobile(string mobileNo, string password)
    {
        if (mobileNo is not null && mobileNo.Contains("+123") && mobileNo.Length is 10 && password.Length > 8)
            return true;

        return false;
    }

    public bool VerifyUserWithSocialMedia(int socialMediaId, string userName, string password)
    {
        var listOfSocialMediaUsername = new string[] { "jon", "dawg", "sushant" };
        if (socialMediaId > 1 && userName is not null && listOfSocialMediaUsername.Contains(userName) && password.Length > 8)
            return true;

        return false;
    }
}

Here, only functionalities that is relevant to auth for american user is added.

// calling code and output
var user5 = new UserInputs()
{
    UserName = "rambdr",
    Email = "rambdr@mail.com",
    Password = "mteverest8848"
};
AuthForAPAC authForAPAC = new AuthForAPAC();
var verify = authForAPAC.VerifyUserWithEmail(user5.Email, user5.Password); // register with email
if (verify)
    Console.WriteLine($"New account registered with email: {user5.Email}");
/*
Output: New account registered with email: rambdr@mail.com
*/

Finally, interface segregation suggests breaking down complex entities into small relevant entities (decomposition). This promotes flexibility and loose coupling (less attachment with other object) in software design.

Takeaway from Interface Segregation Principle

The Interface segregation principle suggest breaking down relevant functionalities into interface and only implement it to class where it is required. This results in a more flexible software design which is modular, flexible, and maintainable.

Dependency Inversion

Fifth and final principle of SOLID stands for Dependency inversion principle. Dependency inversion principle states multiple rules.

• Firstly, it state that a higher level object should never depend on a concrete lower level object. Meaning, high-level modules should not directly rely on the specific implementations of low-level modules. Instead, both high-level and low-level modules should depend on abstractions, such as interfaces or abstract classes.

• This bring up to the second rule, abstractions should not depend upon on details. Abstraction should not be coupled to the specific implementation details of lower-level modules.

• Finally, the third rule states that details should depend on abstractions.

Covering up all, dependency inversion state that there must be an abstraction hiding implementation (use interface) and specify implementation to that interface (dependency injection). (Higher level module is an interface and lower level module is the implementation of that interface)

Demonstration

To demonstrate dependency inversion principle, I will work on same above example of auth module. Basically, there will be 2 auth option: auth with facebook and auth with email. So I will create 2 separate interface:

public interface IAuthWithFacebook
{
    public bool VerifyFacebookAcc(string facebookId, string email, string password);
}

public interface IAuthWithEmail
{
    public bool VerifyAcc(string email, string password);
}

Both of these auth option: auth with facebook IAuthWithFacebook and auth with email IAuthWithEmail will have a single concrete implementation. Therefore, I will create 2 classes AuthWithFacebook and AuthWithEmail to implement these interfaces.

public class AuthWithFacebook : IAuthWithFacebook
{
    public bool VerifyFacebookAcc(string facebookId, string email, string password)
    {
        var listOfSocialMediaUsername = new string[] { "jon", "dawg", "sushant", "joerogan" };
        if (facebookId is not null && listOfSocialMediaUsername.Contains(facebookId) && password.Length > 8)
            return true;

        return false;
    }
}

public class AuthWithEmail : IAuthWithEmail
{
    public bool VerifyAcc(string email, string password)
    {
        var listOfEmail = new string[] { "jon@mail.com", "dawg@mail.com", "sushant@mail.com", "joerogan@mail.com" };
        if (email is not null && listOfEmail.Contains(email) && password.Length > 8)
            return true;

        return false;
    }
}

The implementations (AuthWithFacebook and AuthWithEmail) depend on these interfaces, which is in line with the Dependency Inversion Principle. This allows the high-level modules (implementation) to remain decoupled from the specifics of how authentication is performed. This ensures that high-level modules depend on abstractions (interfaces), while the details of implementations are encapsulated within the low-level modules.

Then I will create another class called CallingCode to demonstrate dependency inversion.

public class CallingCode(IAuthWithFacebook _authWithFacebook)
{
    private readonly IAuthWithFacebook authWithFacebook = _authWithFacebook;

    public bool VerifyAuth(string userName, string email, string password)
    {
        return _authWithFacebook.VerifyFacebookAcc(userName, email, password);
    }
}

Here, CallingCode takes IAuthWithFacebook as argument by using latest C# 12 feature: primary constructor. VerifyAuth() does nothing but redirects to IAuthWithFacebook's method.

// main calling code
var user6 = new UserInputs()
{
    UserName = "joerogan",
    Email = "joerogan@mail.com",
    Password = "joeroganexperience"
};
var authWithFacebookObj = new AuthWithFacebook();
CallingCode callingCode = new CallingCode(authWithFacebookObj);
var facebookAuth = callingCode.VerifyAuth(user6.UserName, user6.Email, user6.Password);

if (facebookAuth)
    Console.WriteLine($"New account registered with facebook: {user6.UserName}.");

/*
Output: New account registered with facebook: joerogan.
*/

The CallingCode class depends on the abstraction provided by the interface (IAuthWithFacebook). It doesn't have direct knowledge of the concrete implementation (AuthWithFacebook), promoting decoupling between components. Hence, I will create a variable authWithFacebookObj assigning it as concrete implementation (higher module). This is know as dependency injection (you have a class that take interface as an parameter and you provide its concrete implementation at runtime or compile time).

Takeaway from Dependency Inversion Principle

The Dependency Inversion Principle states decoupling high-level modules (interfaces) from low-level modules (implementations) by introducing abstractions (such as interfaces or abstract classes) that define the interactions between them. By depending on abstractions rather than concrete implementations, software design become more flexible, extensible, and easier to maintain.

Summary

1. Single Responsibility Principle:

   • Each unit of code (class, method, function) should have a single responsibility.

   • Promotes easier debugging and maintainability.

   • Example: Separate validation, registration, and email confirmation in an authentication module.

2. Open/Closed Principle:

   • Software entities should be open for extension but closed for modification.

   • Encourages adding new functionalities without altering existing code.

   • Example: Extending authentication logic without modifying the original code.

3. Liskov Substitution Principle:

   • Child classes should be exchangeable for their base (parent) classes.

   • Objects of a subclass should be able to replace objects of the superclass without affecting program correctness.

   • Encourages code reusability and extensibility.

   • Example: Extending authentication classes without changing their behavior.

4. Interface Segregation Principle:

   • Interfaces should be tailored to specific objects and should not contain irrelevant functionalities.

   • Classes should only implement interfaces that are relevant to their functionality.

   • Promotes cleaner and more modular design.

   • Example: Defining separate interfaces for different authentication methods.

5. Dependency Inversion Principle:

   • High-level modules should not depend on low-level modules. Both should depend on abstractions.

   • Abstractions should not depend on details; details should depend on abstractions.

   • Promotes decoupling, flexibility, and maintainability.

   • Example: Using interfaces to define interactions between modules and injecting concrete implementations at runtime or compile time.

Sourcecode

Introduction

In TAP(Task-based Asynchronous Programming), async await keyword and Task class does everything to achieve asynchronous behavior. Simply understanding, the async is used to say "this is my asynchronous method/function". But just highlighting a function with async keyword doesn't make it asynchronous. If the asynchronous operation which is being awaited by using await keyword has not yet completed, it will return immediately. Task is a common type that can be awaited. Synchronous flow like not executing next operation is still maintained in asynchronous operation but without blocking or pausing the thread. If the task is already in completed state, code flow will execute as usual.

When long running operation is being awaited, the compiler sets a continuation to be executed after the operation has completed. Meaning, execution will start from where it left after the awaited operation has completed. Continuation is a callback (delegate) that will be executed as soon as the awaited operation has completed. Simpler terms, continuation is just a piece of code to be executed when the awaited operation has completed. Continuations are Action delegate that receives result of the asynchronous operation (Action<Task<TResult>> or Action<Task>). The type of the awaited asynchronous operation is usually Task or Task<TResult>. You can obviously have your own custom awaitable types but .NET provide ValueTask<TResult> which is better than having your own custom type. Custom types for awaiting is not preferred due to potential compatibility and tooling issues.

Task has a method called ContinueWith to attach continuations. Continuations are heavily used in event-based pattern (EAP). EAP pattern involves the use of events and callback methods to notify the completion of asynchronous task.

TAP (Task-based Asynchronous Programming) doesn't rely heavily on continuations like EAP. TAP uses await keyword which provides a way to pause or wait for the asynchronous task to complete without blocking the thread. You might be wondering "what exactly does it mean to block the thread". It means the situation where the execution of code is blocked. It cannot execute further. Basically, the thread where this execution is happening becomes unable to proceed with other tasks.

Task-based asynchronous pattern

In TAP, asynchronous operation doesn't wait for the task to complete. It starts and returns immediately. But what does it return?. It returns a sort of token. After initiating the asynchronous operation and obtaining the "token" (usually a Task or Task<TResult>), the execution of the calling code continues immediately. The "token" (Task or Task<TResult>) is used for coordination. What I mean by that is when you await a task, further execution is paused which allows asynchronous operation to finish before moving on to next step of code. You can await the task, register continuations, or perform any other operations based on the completion status of the asynchronous operation. Here's a simple example to visualize an async method:

public async Task<string> PrintSomethingAsync()
{
    Console.WriteLine("Before awaiting.");
    await Task.Delay(2000); // simulating time consuming operation. 
    Console.WriteLine("After awaiting");
    return "returned val from async method.";
}

Until the code reaches await keyword, it runs synchronously. After that, it starts an asynchronous operation which delays for 2 seconds simulating some computation. After waiting for 2 seconds, it prints and finally returns to the caller.

// calling code
string ret = await eg.PrintSomethingAsync();
Console.WriteLine($"Returned from async method: {ret}");
Console.WriteLine("All done.");

Output:

Before awaiting.
After awaiting
Returned from async method: returned val from async method.
All done.

When you await an asynchronous operation (represented by a Task or Task<TResult>), it pauses the execution of the method at that point. However, it doesn't block the calling thread. Instead, it allows the calling thread to do other work while waiting for the asynchronous operation to complete. But if the await waits for the operation to complete and also immediately returns to the caller, how does it know where to start when the task is completed? This is achieved through the continuation. The compiler generates code to capture the current context and create a continuation that gets executed when the awaited task completes.

Task

Simply put, Task class represents an asynchronous operation. Task encapsulates the ongoing or completed asynchronous operation. Task class provides varieties of methods to work with asynchronous operation. Some common methods are:

Task has a Status property (TaskStatus enum) which holds the current state of the operation. Task.Status can be Running, WaitingToRun, WaitingForActivation, RanToCompletion, Faulted, WaitingForChildrenToComplete and Canceled.

One more thing, Task doesn't need to be explicitly disposed. The management of Task instances is handled by the runtime and the GC (garbage collector).

When you await a Task, the compiler performs an unwrapping operation. This operation handles exceptions if Task is faulted, else extracts the result. The result type of the await expression is the type of the awaited task's result, which will be TResult for Task<TResult> or void for a plain Task. The result is not the original Task itself. Example:

Task<string> task = SomeHeavyTaskAsync();
string result = await task; // unwraping operation converts Task<string> to string

Here, I have a variable called task of type Task<string>. But when I await, it will unwrap from Task<string> to string.

Exceptions in Task and why you should await a Task

I previously said that awaiting a Task handles exceptions if faulted, else extracts the result. However, what if the task is not awaited? When the Task is not awaited, any exceptions that may have occurred during its execution may not be observed immediately. There may be multiple exceptions during this execution period. If not awaited, they will be stored but not thrown. This "stored" exception is called AggregateException. AggregateException is a wrapper that aggregates multiple exception. The main reason to not throw exception like synchronous flow is to prevent unhandled exceptions causing termination of the application. Also you don't want to hamper other Task when there is exception in some Task which is executing in parallel. Here is an example to demonstrate it.

public async Task<string> DemonstrateExceptionAsync()
{
    await Task.Delay(1000); // simulating some operation
    Console.WriteLine("after some operation.");
    MethodWithExceptionAsync(); // not awaiting
    Console.WriteLine("after exception in other task.");
    throw new Exception(message:"exception inside the method.");
    return "retured from DemonstrateExceptionAsync";
}

public async Task MethodWithExceptionAsync()
{
    await Task.Delay(500); // simulating some operation
    Console.WriteLine("inside MethodWithExceptionAsync method.");
    throw new Exception(message:"Exception outside the method.");
}

Here, when MethodWithExceptionAsync() is initiated, it won't throw exception. Instead this exception is stored in AggregateException. Execution continues and reach another exception which will be stored in AggregateException too.

// calling code
var exceptionMethod = eg.DemonstrateExceptionAsync();
await Task.Delay(1500);
Console.WriteLine("after initiating method with exception.");
await exceptionMethod;
Console.WriteLine("after awaiting faulted task");
/*
Output:
    after some operation.
    after exception in other task.
    after initiating method with exception.
    inside MethodWithExceptionAsync method.
-- Finally, throws an exception --
System.Exception: 'exception inside the method.'
*/

In calling code, I didn't await the anonymous type variable exceptionMethod which is assigned to an async method DemonstrateExceptionAsync(). All exceptions that occurred within this method is stored in AggregateException. Unless I await an async method or if I try to access the result of an async method (Task.Result), there won't be any exception. So if I didn't do await exceptionMethod;, it won't throw an error.

It's important to understand that using Task.Result to get the result of an asynchronous operation can lead to deadlocks. Imagine a scene where you're waiting for a result using Task.Result, and the asynchronous operation needs a specific context, like the UI thread. If that context is blocked, a deadlock may occur, which hampers the operation. Output will be the following:

after some operation.
after exception in other task.
after initiating method with exception.
after awaiting faulted task

When I finally await this async method (await exceptionMethod;), the most recent exception of AggregateException is thrown which is System.Exception: 'exception inside the method.'.

Now I'm sure you know why awaiting a task is crucial. You don't want exception to be silent. Also awaiting a task ensures that the code after the await statement is executed only after the task has completed. This flow of execution is important to consider in asynchronous programming, preventing race conditions and ensuring expected order of execution.

race condition: A race condition occurs when two or more threads can access shared data and they try to change it at the same time.

Cancelling an asynchronous operation

Cancellation in Task is supported via CancellationToken. This allows an asynchronous operation to cancel its operation. You can think of a real scenario like visiting a website. If that website is taking too long to load, you'll close that tab and reenter the url in new tab or probably reload the page. In such cases(reload, closing tab), browser(client) will send a cancellation token to the server to cancel the operation. This helps in resource utilization.

When an asynchronous operation is cancelled, its state becomes Canceled. To use cancellation in Task, you can pass CancellationToken which will be checked periodically. Example:

// asynchronous method which can be cancelled
public async Task DoSomethingWithCancellationAsync(CancellationToken cts)
{
    try
    {
        Console.WriteLine("Inside DoSomethingWithCancellationAsync method.");
        await Task.Delay(5000, cts);
        Console.WriteLine("Still inside DoSomethingWithCancellationAsync method, after long running operation.");
    }
    catch(OperationCanceledException oce)
    {
        Console.WriteLine(oce.Message);
    }
}

Here, Task.Delay method has a parameter that accepts cancellation token (cts) as an argument. This means that if the cancellation token is signaled before the delay completes, the operation will be canceled, and an OperationCanceledException will be thrown.

// calling code
using (CancellationTokenSource cts = new CancellationTokenSource())
{
    Task someTask = eg.DoSomethingWithCancellationAsync(cts.Token);  // start the task
    try
    {
        await Task.Delay(2000); // delay further execution for 2 seconds
        cts.Cancel(); // cancel the operation after 2 seconds.
        await someTask;        
    }
    catch (OperationCanceledException oce)
    {
        Console.WriteLine($"Operation cancelled: {oce.Message}");
    }
}
/*
Output:
    Inside DoSomethingWithCancellationAsync method.
    A task was canceled.
*/

Here, someTask has already started the task. someTask will take 5 seconds to complete its operation. But I will cancel this operation in 2 seconds.

Now, what if we pass cancellation token after the operation has already completed? Well, cancellation token will have no effect. The asynchronous operation, once completed, cannot be canceled. Example:

// calling code
using (CancellationTokenSource cts = new CancellationTokenSource())
{
    Task someTask = eg.DoSomethingWithCancellationAsync(cts.Token);  // start the task
    try
    {
        await Task.Delay(6000); // delay further execution for 6 seconds
        cts.Cancel(); // cancel the operation after 6 seconds.
        await someTask;        
    }
    catch (OperationCanceledException oce)
    {
        Console.WriteLine($"Operation cancelled: {oce.Message}");
    }
}
/*
Output:
    Inside DoSomethingWithCancellationAsync method.
    Still inside DoSomethingWithCancellationAsync method, after long running operation
*/

Here, someTask has already started the task. someTask will take 5 seconds to complete its operation. If I cancel this operation after 6 seconds, it won't cancel someTask because it has already completed having its state as RanToCompletion.

Creating an asynchronous method

Asynchronous methods can be generic, static, and can have any access modifiers. Creating an asynchronous method involves using the async keyword in the method signature, and the method should return either Task for void methods or Task<TResult> for methods with a result or ValueTask<TResult> or custom awaitable types. Parameters for async methods are similar to regular methods, but ref and out are disallowed.

ref and out are disallowed due to the complications of managing the state of these parameters across various asynchronous operation. Imagine a scenario where some async method is computing progress value (percentage). If ref and out were allowed, it would really be complex to work around. What if some referenced variable's value is modified in race condition?

Asynchronous anonymous function

Async anonymous function are just unnamed lambda expression but are asynchronous. It allows for concise representation of asynchronous operations without the need to define a named method. To declare it, you use the async keyword just before a lambda expression.

Asynchronous lambda expression syntax: async (parameter-list) => (expression-body)

This results in a concise way to represent a function with asynchronous operations. Example:

public async Task DoSomethingWithAnonFuncAsync()
{
    Func<Task<string>, Task<int>> getStringLengthDelegateAsync = async (str) =>
    {
        string task = await str;  // await parameter which is of type task
        return task.Length;
    };
    async Task<string> someStringLocalFuncAsync() // local function can be asynchronous as well
    {
        await Task.Delay(2000); // simulating some operation
        return "something";
    }
    Task<int> getLengthTask = getStringLengthDelegateAsync(someStringLocalFuncAsync());
    int len = await getLengthTask; // unwrapping operation by await. Task<int> --> int
    Console.WriteLine($"Length of string is: {len}");
}

This code might look a lot but let me break it down. Firstly, I created a delegate of type Func<Task<string>, Task<int>>. For implementation of this delegate, I created an anonymous async function which returns the length of the string passed(str). I awaited the passed string of getStringLengthDelegateAsync because it is asynchronous of Task type. await unwraps the result of Task<string> to string. Then return the length of that string. After that, you can see I created a local async function. Yes, local function can be asynchronous too. This local function named someStringLocalFuncAsync returns Task<string> after a 2 seconds delay.

Next, I created a variable of type Task<int> and assign it to the delegate. I passed a local function as an argument for this delegate because you can see that the return type of the local function someStringLocalFuncAsync and the parameter for the delegate Func<Task<string>, Task<int>> getStringLengthDelegateAsync are the same, i.e., Task<string>. After this, I await the Task<int> which unwraps the result as int and assign that to a integer variable. Finally, print that int.

// calling code
await eg.DoSomethingWithAnonFuncAsync();
/*
Output:
    Length of string is: 9
*/

Success or failure of async execution

Awaited asynchronous have two possibilities. Either it has completed without any exception (RanToCompletion) or it has failed due to some exception (Faulted). When an async method waits for an asynchronous operation to complete, it really means the method is doing nothing. A continuation is attached to the operation and the method returns immediately. SynchronizationContext makes sure that the continuation executes in the right thread in GUI application.

From stackoverflow: Simply put, SynchronizationContext represents a location "where" code might be executed. Delegates that are passed to its Send or Post method will then be invoked in that location. (Post is the non-blocking / asynchronous version of Send)

In Asp.Net Core, there is no dedicated UI thread. It makes sense to eliminate the dedicated UI thread in web APIs and web server scenarios. Asp.Net Core utilizes (TAP) task-based asynchronous pattern's async-await pattern to achieve asynchrony. Console application and background services utilizes thread pool. When an asynchronous operation completes in console app or background services, the continuation may run on a different thread from the one that initiated the operation. The thread pool efficiently manages these threads, preventing the application from being blocked.

await does it all

When you await an asynchronous operation, the compiler pause the further execution until the operation completes. The compiler generates continuation to execute as soon as the operation completes and returns. If the awaited operation has not completed, it will return immediately. The calling code can continue with its execution, ensuring non-blocking behavior. If the environment is GUI, there is a SynchronizationContext which ensures that the continuation runs on the same UI thread. In context of Asp.Net core, where UI thread is not needed, continuations may run on a different thread (from ThreadPool).

If the awaited task completes with an exception, await handles it. If the task is faulted, it throws an exception. If it's canceled, it throws an OperationCanceledException. If the task completes successfully, it continues with the next line of code.

What can be awaited

Any type that has a suitable methods or extension method can be awaited. Most common type that can be awaited is Task, Task<TResult>, ValueTask<TResult>, IAsyncEnumerable<T>, or any custom type which fulfills Task-Based Asynchronous Pattern can be awaited. Awaitable type must implement System.Runtime.CompilerServices.INotifyCompletion interface.

If we look at Task<TResult>, C# compiler looks for the following detail to be awaitable:

1. GetAwaiter method: The Task<TResult> type should have a parameterless method named GetAwaiter that only returns an awaiter object and cannot be void. For eg: Task<string> has such method like: public TaskAwaiter<string> GetAwaiter();. The return type of GetAwaiter method is called the awaiter type.

2. Awaitable types must implement System.Runtime.CompilerServices.INotifyCompletion interface. For eg: The Task<string> has such struct as public struct TaskAwaiter<string> : INotifyCompletion.

3. This awaiter type must have following property and method, respectfully:

   • bool IsCompleted

   • void OnCompleted(Action<TResult>)

4. IsCompleted indicates whether the asynchronous operation has completed or not.

5. OnCompleted receives Action<TResult> that will be executed once the awaited operation finishes or once IsCompleted is true.

6. Awaitable type must have GetResult method. GetResult is a non-generic parameterless method of the awaiter type (e.g., TaskAwaiter<TResult>). It retrieves the result of the awaited asynchronous operation. It is called when IsCompleted is true. Eg: public TResult GetResult(){ ... } for TResult or public void GetResult() for Task or void.

7. GetResult method is also responsible for handling the result of the asynchronous operation. When the asynchronous operation has an exception, it will throw that exception to the caller. This is to make sure that any exceptions that occurred during the asynchronous operation are appropriately shown to the point where the asynchronous operation was awaited.

Full codebase goes into async mode

When there is an async method which has an asynchronous operation calling another async method and every operation is awaited, this creates a chain of asynchronous operations which ensures that the application remains responsive and efficient, handling tasks without blocking threads. Or in other word "the full codebase goes in async mode". That's how you achieve asynchrony.

public async Task A()
{
    await B();
}

private async Task B()
{
    await C();
}

private async Task C()
{
    await D();
}

private async Task D()
{
    await E();
}

private async Task E()
{
    await F();
}

private async Task F()
{
    await Task.Delay(1000);
}

Here, you can see a chain of asynchronous tasks where each task awaits the completion of the next one.

// calling code
var someAsyncCodeTask = asyncMode.A();
await someAsyncCodeTask;
Console.WriteLine($"All async mode task completed. Status: {someAsyncCodeTask.Status}");
/*
Output:
    All async mode task completed. Status: RanToCompletion
*/

Here, call to A method would initiate a chain of an asynchronous operation. A is like the entry point. When you call A, it kicks off a sequence of tasks. A immediately says, "I'm going to wait for B to finish before I'm done." B is the second task. When A is waiting for B, it's like putting B on the to-do list and moving on to other things. B also says, "I'm going to wait for C to finish." C joins the chain. A is waiting for B, and B is waiting for C. C says, "I'll wait for D to finish." Same goes for Task D and E.

Finally, F is the last in line. A, B, C, D, and E are all waiting for F.

In summary, transitioning your codebase into async mode not only improves the efficiency and responsiveness of your application but also sets the foundation for scalable and performant software, especially in scenarios where parallel execution of tasks can significantly enhance overall system.

Summary

1. TAP uses async await keyword and the Task class to achieve asynchrony.

2. Task class represents an asynchronous operations.

3. Async methods don't wait for completion; they start and return immediately.

4. Awaiting tasks handles exceptions immediately.

5. Avoid using Task.Result to prevent deadlocks.

6. CancellationToken allows canceling asynchronous operations.

7. SynchronizationContext manages the execution context, mostly in GUI applications where there is UI thread.

8. Asp.Net Core utilizes TAP to achieve asynchrony without a dedicated UI thread.

9. Async methods can be generic, static, and can have various access modifiers.

10. ref and out parameters are disallowed for simplicity in state management.

11. Awaiting tasks can result in completion (RanToCompletion) or failure (Faulted).

12. Proper awaiting ensures expected execution order and prevents race conditions.

13. Common awaitable types include Task, Task<TResult>, ValueTask, and IAsyncEnumerable.

14. Any type with suitable methods or extension methods can be awaited.

15. Chaining async methods (asynchronous programming) creates a responsive and efficient application.

Sourcecode

More on async await can be read at:

• C# in depth Fourth edition by Jon Skeet

• David Fowler's tips on best asynchronous programming practices

• NDC conference: Efficient Async and Await by Filip Ekberg

One final misconception I would like to mention is "Asynchrony means parallel execution". This is false. Parallelism means executing multiple tasks at the same time. While asynchrony does allow parallelism, not all asynchronous operations are parallel. Asynchronous operations allow tasks to proceed independently without waiting, it doesn't necessarily be simultaneous execution.

Introduction

Synchronous operation, or simply the normal flow of code, pauses the thread when the task or any computation is not yet completed. Computation can be any sort of thing like performing CRUD operation, or anything that requires some resource. During the pause session, the thread is waiting for the computation to finish and it cannot perform other tasks. Executing tasks without blocking the thread for the responsiveness of the application or APIs is said to be asynchrony. The difference between synchronous and asynchronous is that the asynchronous code waits for the operation to complete without blocking the current execution thread. Normal synchronous flow of code blocks the thread when there is some operation going, and it does this until the operation has completed.

A bit of history to achieve Asynchrony in .NET

Early version of .NET introduced Asynchronous Programming Model (APM) to achieve asynchrony. APM used IAsyncResult to achieve asynchronous operation, it has all the functionality to achieve asynchronous operation. IAsyncResult provides properties to get information about the operation whether it has completed or not, the user state associated with the operation, and an asynchronous wait handle. To implement APM pattern, you would create two methods: Begin and End. Begin was used to initiate asynchronous operation with parameters such as callback delegate, the user state, any other required information. Begin returns IAsyncResult object. End was used as callback to retrieve the result of the operation. End takes IAsyncResult as a parameter which is returned by the Begin method.

As an evolution to APM pattern, .NET introduced Event-based Asynchronous pattern (EAP). Event-based Asynchronous pattern (EAP) is to notify when an operation completes by utilizing events and delegates. In EAP, the initiation of an asynchronous operation is performed by invoking a method suffixed with 'Async' such as DoSomethingAsync. Upon initiating the operation, an associated event, often named with a 'Completed' suffix, like DoSomethingCompleted, is triggered upon completion of DoSomethingAsync, signaling that the asynchronous task has finished. This event-driven approach allows to subscribe and handle the completion of events.

.NET 4.5 introduced Task-Based Asynchronous pattern (TAP) by utilizing Task Parallel Library (TPL). TAP has become the standard pattern to achieve asynchrony in C#. async await keyword and Task<TResult> is used to achieve TAP. The Task<TResult> class is particularly powerful in TAP, allowing to perform asynchronous operations that produce a result. This result can be awaited using the await keyword, providing a clean way to handle asynchronous responses. async keyword is used to denote that the method is asynchronous.

Showcasing asynchrony

Here is an simple example to demonstrate the importance of asynchrony. I will create an desktop application using Windows Forms and add two buttons: Non-Async Button and Async Button. Non-Async Button's click event will have a synchronous flow whereas Async Button will have an asynchronous flow. This example is based on TAP pattern. Here is the code:

// Non-async button event handler
private void button1_Click(object sender, EventArgs e)
{
    textBox1.Text = "Non-async button clicked. Pausing thread for 5 seconds...";
    Thread.Sleep(5000); // replicating time consuming operation.
    textBox1.Text = "Non-async operation completed.";
}

Before Thread.Sleep(5000);, the textBox1 text field will also not be printed. It is due to the fact that the UI is not being updated during the Thread.Sleep(5000) operation because this operation is blocking the UI thread.

// async button event handler
private async void button2_Click(object sender, EventArgs e)
{
    textBox2.Text = "Async button clicked. delaying task for 5 seconds without pausing thread.";
    await Task.Delay(5000); // replicating time consuming operation.
    textBox2.Text = "Async operation completed.";
}

Output:

You can see that I can write text before pressing Non-Async Button. But when I clicked Non-Async Button, whole application freezes. I cannot write or highlight over the textfield. The Thread.Sleep(5000) in the button1_Click (Non-async button) method is being executed on the UI thread, causing the entire UI to freeze during that period (5 seconds). In GUI application like Windows forms, the UI thread is responsible for handling inputs, updating UI and responding to events. When a time-consuming operation is performed on the UI thread, such as using Thread.Sleep, it freezes the entire UI because the thread is occupied with that operation and cannot perform its other duties.

But when I used async/await pattern, there is no pause and thread blockage. The await Task.Delay(5000) line represents an asynchronous delay of 5 seconds. During this period, the UI thread is not paused, and it can handle other tasks, respond to the user input, and update the UI as needed. Actually the code runs synchronously till await. As soon as it reaches await, it doesn't matter if the operation has completed or not, it will return. This is the power of the async/await pattern. When the code reaches the await keyword, it essentially says, "I'm going to pause here, but don't wait for me. Continue with other tasks you have to do, and when I'm ready, come back to this point." But how did the code ensure to update appropriate UI when something is awaiting? It does so by using SynchronizationContext class.

A continuation is created when the code reaches await. Continuation is basically a callback delegate that will be executed as soon as the awaited operation has completed. I will provide a detailed explanations on this topic in my upcoming blog post. Asynchronous operation is very crucial when responsiveness and non-blocking execution is your priorities.

Conclusion

In this post, we get a touch of asynchronous and some history of asynchrony in .NET. Next post, I will cover the async/await pattern, the significance of the await keyword, and insights into writing effective asynchronous code. Stay tuned.

Sourcecode

Stephen Cleary's blog post on asynchronous operation has no thread.

variable : a symbolic name associated with a value and whose associated value may be changed - Wikipedia

Variables are like placeholders tied to values. They are entities that can be captured, moved around, and tweaked. Taking about captured variable, they refer to the instances where a variable defined in one scope is accessed and utilized in a different scope, typically in a lambda expression. Before digging deep into captured variables, let's understand lambda expression.

Lambda Expression

Lambda expression were introduced in C# version 3.0. Lambda expression allows a compact way to quickly create a function without following the standard way of a full function declaration. Lambda expression are frequently called as anonymous function due to their ability to define inline and unnamed functions.

Lambda expression syntax: (parameter-list) => (expression-body)

You can read this as a function that takes the parameter-list as a parameter and returns the result of the expression-body. parameter-list is similar to regular parameter of a function/method. It can be any data type or object. The expression-body is the code that gets executed when the lambda expression is called. It is the logic body of function similar to method body. The difference is that the expression-body is inline. Lambda expressions are useful in scenarios where a short, scoped (one-time-use) function is needed. Here is a simple lambda expression which returns the sum of 2 numbers:

public Func<int, int, int> sumOf2Nums = (x, y) => x + y;

If we recall from my previous article on delegates, first two type parameters of above delegate Func<> sumOf2Nums takes 2 input parameter types(int, int), and the last int represents the return type. Then, we have a lambda expression (x, y) => x + y; which takes 2 int parameters which returns its sum. We can call this delegate like any other method/function:

sumOf2Nums(4,8);

When sumOf2Nums(4,8) is called, it invokes the anonymous function(lambda expression) with x = 4 and y = 8 which returns an int 12 which is the sum of x and y.

Now, let's write the similar method in standard way:

public int sumOf2NumsRegularWay(int x, int y)
{
    return x + y;
}

Both of our method (delegate and the standard way) returns same output but using lambda expression is slightly more concise. Lambda expression shines when you use extension method particularly working with sequences using LINQ. Here is an example:

// A delegate with lambda expression to get the total sum of a list.
public Func<List<int>, int> sumOfList = (listOfNums) => listOfNums.Sum();

List<int> someNumbers = new List<int> { 1, 5, 6, 9, 4, 2, 7, 11, 3, 16 };
int sum = eg.sumOfList(someNumbers); // invoke the delegate.
Console.WriteLine(sum);
// prints the sum of the list which is: 64

We can do a lot of computation in the expression body of lambda expression. What if we want to get the sum of odd numbers that are less than 6? Here's how we can utilize other extension method:

// A delegate with lambda expression to get the total sum of odd numbers that are less than 6 a list.
public Func<List<int>, int> sumOfOddNums = (listOfNums) => listOfNums.Where(x => x < 6 && x % 2 != 0).Sum();

sumOfOddNums expanded the usage of lambda expressions (x => x < 6 && x % 2 != 0) by implementing the LINQ extension Where method to filter odd numbers less than 6 before calculating their sum.

List<int> someNumbers = new List<int> { 1, 5, 6, 9, 4, 2, 7, 11, 3, 16 };
int sumOfOddNums = eg.sumOfOddNums(someNumbers); // invoke the delegate.
Console.WriteLine(sumOfOddNums);    
// prints the total sum of odd numbers which are less than 6 which is: {1,5,3} = 9

It demonstrate how lambda expressions make it easy to express complex conditions in a clean and compact manner. Now rolling back to captured variables.

Captured variables

When a variable defined in one scope is accessed and utilized in a different scope often within a lambda expression, it said to be captured variables. In simpler terms, any kind of variable that is declared outside of the lambda expression, but used within the lambda expression is said to be captured variable. It's important to note that the parameter of the lambda expression and variables declared within the lambda expression are not captured variables. Here's an example to demonstrate it:

public Action<int> IncrementCounterAction(int incrementBy)
{
    int defaultNumber = 2;
    Action<int> action = input =>
    {
        Console.WriteLine($"Default counter value: {defaultNumber}");
        int result = input + defaultNumber + incrementBy;
        Console.WriteLine($"Increment action by (method argument): {incrementBy}");
        Console.WriteLine($"Sum of method argument, lambda argument, and default local method value: {result}");
    };
    defaultNumber = 10; // modify local variable
    return action;
}

In above code, defaultNumber and incrementBy is captured variable. Both of them is beyond the scope of action() delegate. To call this method:

// eg is the instance of a class
Action<int> action = eg.IncrementCounterAction(5);
action(2);

What do you think the output would be?

Output:

Default counter value: 10
Increment action by (method argument): 5
Sum of method argument, lambda argument, and default local method value: 17

This might be surprising. The reason behind this is defaultNumber being captured by reference. Lambda expression captures variable by their references. When you use a variable within a lambda expression, it captures the reference to that variable rather than its value. This means any changes made to the captured variable outside the lambda expression will be reflected inside the lambda, and vice versa. Capturing variables using reference allows lambda expression to have access to current state of variable in the surrounding scope and avoids creating unwanted copies of variables especially when lambdas are passed as parameter or used in asynchronous tasks.

Being unaware of this behavior can lead to unexpected results when variables are modified after the creation of lambda expression.

Closures

Closure is a mechanism that allows a lambda expression to capture and remember the variables and the environment in its lexical scope { }, even if the lambda expression is executed outside that scope. Simply put, closures encapsulates variables and environment in which it was created, preserving both the variable and environment context. By environment, it is referring to the set of variables that are in scope and accessible at the time the closure is created. It ensures that the lambda expression keep a hold on its original surroundings, no matter when and where it is executed.

In our previous example, when the variable defaultNumber was captured in lambda expression, it formed a closure. This closure essentially locked onto the defaultNumber and its environment. So, when defaultNumber was modified outside the lambda expression, that change was also noticed and applied inside the lambda expression. Closures ensure that the lambda expressions have access to the current state of the variables in their surrounding scope.

Example:

Below, I'm creating a CreateCounterAction() method which returns an Action which essentially is a closure.

public Action CreateCounterAction()
{
    int incrementBy = 5;
    // here, action take empty parameter list.
    Action action = () => 
    {
        incrementBy++;
        Console.WriteLine($"Counter at: {incrementBy}");
    };
    return action;
}

The closure in action captures the variable incrementBy which is outside of action lambda expression. In the CreateCounterAction method, the environment consists of the variable incrementBy. When the closure is formed, it encapsulates not only the variable itself but also the entire environment in which it exists meaning it holds onto the context in which it was created.

// calling code
Action closureAction = eg.CreateCounterAction();
closureAction();
closureAction();
closureAction();
closureAction();

When the closureAction is invoked, it increments the captured incrementBy and prints the value.

Output:

Counter at: 6
Counter at: 7
Counter at: 8
Counter at: 9

The closure here is like cache, it holds the state of the incrementBy. Each time you call closureAction, it recalls the previous value and continue.

Closures might get confusing due to the mechanism of how it capture variables. The confusion arises from the fact that closures capture variables by reference, not by value. In a loop, when a closure captures a variable, it doesn't store the current value of that variable at the time of creation. Instead it maintains a reference to the variable. Simply put, when the closure is executed, it looks up the current value of the variable in its original scope.

Example:

Below, I created a method that returns an array of Action. Firstly, I created an array of four actions. Each action is a closure capturing the loop variable i. When invoked, all closures print the final value of i after the loop. This happens because all the closures captured the variable i by reference, and when the closure(lambda expression) is actually invoked and not when they are created, they see the final value of i after the loop has completed resulting in an output of "4" for each invocation.

public Action[] ClosureActionInLoop()
{
    Action[] actions = new Action[4];
    for(int i = 0; i < 4; i++)
    {
        actions[i] = () => Console.WriteLine(i);
    }
    return actions;
}

// calling code
Action[] closureLoopActions = eg.ClosureActionInLoop();
foreach(var loopAction in closureLoopActions)
{
    loopAction();
}

Here, they all reference the same variable i, which has been updated to 4 by the end of the loop. Consequently, they all print the current value of this shared reference, resulting in "4" for each invocation.

Output:

4
4
4
4

To fix this, we can introduce local variable inside the loop. Fixed code:

public Action[] FixedClosureActionInLoop()
{
    Action[] actions = new Action[4];
    for(int i = 0; i < 4; i++)
    {
        int localVar = i; // introduce local variable to let closure remember
        actions[i] = () => Console.WriteLine(localVar);
    }
    return actions;
}

By introducing localVar, each closure captures it own value, addressing the common issue of closures in loop.

// calling code
Action[] fixedClosureLoopActions = eg.FixedClosureActionInLoop();
foreach(var loopAction in fixedClosureLoopActions)
{
    loopAction();
}

Output:

0
1
2
3

Each closures assigned to actions[i] captures this local variable localVar. Unlike the previous example where the closure captured the loop variable i by reference, now it captures the localVar by value at the time of its creation.

Outside of a loop, variables are often captured by reference, while inside a loop, introducing a local variable allows for capturing by value, ensuring each closure remembers its value.

Closures impact on memory

It is crucial to understand the potential impact on memory while using closures. Yes, closures offer flexibility but when there is a long-lived object, you must consider its impact on memory. For instance, if a closure captures a reference to a large dataset, it keeps the data in memory even when it's no longer needed. This results in unnecessary memory consumption impacting the performance.

Some tips on overcoming this issue is to limit the scope of captured variable ensuring they only encapsulate necessary resource.

Conclusion

Captured variables are entities shared across scopes. Lambda expression or anonymous function serve as a concise function to create instant short-lived function. These lambda expressions often form closures, encapsulating both variables and their lexical environment { }(set of variables in a scope). Closures ensure that the lambda expression retain access to the external variable's state allowing flexibility and preservation.

Summary

1. Lambda Expressions: Short, inline functions; e.g., (x, y) => x + y sums two numbers.

2. Captured Variables: Extend scope by accessing external variables, crucial in lambda expressions for dynamic behavior. Illustrated in IncrementCounterAction method, showcasing capture of variables in a closure.

3. Closures: Preserve variable states and environment, ensuring access to original surroundings even outside the lexical scope { }. Shown in CreateCounterAction method, capturing variables to maintain context outside the method.

4. Memory Impact: Closures, while flexible, can lead to unnecessary memory consumption; mitigate by limiting the scope of captured variables. Addressed with examples, warning about unnecessary memory use in closures and recommending limited scope for efficient resource use.

Sourcecode.

More on closures:

• Martin Fowler's Closure article

• Jon Skeet's C# in depth

Delegates in C# are like the instructions you give to different pieces of code, telling them what to do and how to work together. Delegates were introduced in C# to provide a flexible and powerful mechanism for implementing callback functions, event handling, functional programming and for asynchronous programming.

Use Cases

• Callback function: Invoke one piece of code after an event or after execution of some code.

  Eg: Imagine a weather app where you need to fetch real time weather data from 3rd party sources. Instead of freezing the entire application while waiting for data, initiating an asynchronous data retrieval process can be optimal. You provide a callback function (delegate) that processes the weather once it is fetched. This ensures that your application remains responsive, and the weather information is seamlessly integrated when available.

• Event handling: Communicate and respond to certain occurrences, specially in event driven architecture.

  Eg: You have two buttons in your weather app interface: one for Celsius and another for Fahrenheit. Each button is associated with a click event. When a user clicks the Celsius button, the Celsius button click event is triggered which shows the data in °C.

• Functional programming: Delegates allow passing of function or methods as parameter which is the main idea behind functional programming. Think of it like legos. Just like Lego pieces that can be interchanged and combined to create various structures, delegates enable the passing of functions or methods as parameters to other functions. This paradigm is the core principle of functional programming.

• Asynchronous programming: Delegates allow you to define callback methods that get invoked when an asynchronous task completes.

  Eg: Imagine you're uploading photos to the cloud. Without asynchronous help, you'd have to wait for each photo to be uploaded before doing anything else. Asynchronous operation helps your app to start sending photos and not wait around. So, you're free to use the app while all your pictures get uploaded to the cloud in the background.

Example

By using delegate keyword with return type, delegate name, and parameters, you declare a delegate. A simple example portraying use of delegate:

delegate int Addition(int a, int b);

// method that takes two numbers (a and b) and returns their sum.
private int AddNumbers(int a, int b)
{
    return a + b;
}

public void OperateAddNumbers()
{
    // create an instance of the Addition delegate and point it to the private AddNumbers method.
    Addition addition = new Addition(AddNumbers);
    // invokes the delegate, calling the AddNumbers method with following arguments.
    int ret = addition(1,2);
    Console.WriteLine($"Result of addition: {ret}");
}

Delegates are like messengers that deliver your instructions to someone who knows how to perform a specific task. In this example, we declare a delegate named Addition that represents the idea of adding two numbers. The private method AddNumbers is the actual source which computes in this task. Now, when the public method OperateAddNumbers wants to add numbers, it creates a helpful messenger (delegate) named addition and tells it, 'Go to AddNumbers, and ask it to add 1 and 2.' The messenger delivers the request, and we get the result.

So, delegates simplify our code by acting as messengers between different parts of our program, making sure the right source handle specific tasks.

Another scenario, imagine you have a notification service that can send messages, and you have users who want to receive those messages. A messenger is the one who delivers news to everyone.

We will build a NotificationService class that consists of the delegate NotificationHandler, messenger to notify of type NotificationHandler and the message sender SendNotification.

public class NotificationService
{
    // Delegate handling notification
    public delegate void NotificationHandler(string message);

    // "messenger" to notify
    public NotificationHandler Notify;

    // send notification 
    public void SendNotification(string message)
    {
        // ensure that there are subscribers to send notification
        if(Notify != null)
        {
            Notify(message);
        }
    }
}

Above, we declare a special kind of helper named NotificationHandler (a delegate). It's like saying, "Hey, I'm creating a messenger that can carry messages." Notify is our messenger. It's the one who will deliver the messages. When the SendNotification method is called, it checks if there is a messenger (Notify). If yes, it tells the messenger to deliver the message. User receives the notification through the messenger who ensures that the message is sent to every subscribed user.

// Class portraying subscriber.
public class Subscriber
{
    public string Username { get; set; }

    // Imagine this as a smartphone, showing user notification.
    public void ReceiveNotification(string message)
    {
        Console.WriteLine($"New notification for you {Username}: {message}");
    }
}

Each Subscriber has a method ReceiveNotification, which is what happens when the messenger delivers a message. In this case, it shows the message on the console.

// create notification service
NotificationService notificationService = new NotificationService();
// create subscribers using collection initializer
List<Subscriber> listOfSubscriber = new List<Subscriber>()
{
    new Subscriber(){ Username = "Priti"},
    new Subscriber(){ Username = "Jon"},
    new Subscriber(){ Username = "Ram"}
};

// Subscribe Users to the Notification Service 
foreach (Subscriber subscriber in listOfSubscriber)
{
    notificationService.Notify += subscriber.ReceiveNotification;
}

// random amount of voucher 
Random randAMount = new Random();
int discountAmount = randAMount.Next(10, 251);

// send notification to the subscribers who are subscribed. 
notificationService.SendNotification($"Here's a voucher of Rs. {discountAmount}! Enjoyyyyyy.");

Now, each time the SendNotification method is called, it generates a random discount amount and includes it in the notification message. Subscribers will receive notifications with different voucher amounts every single time. Each user in the listOfSubscribers is subscribed to the Notify event of the NotificationService , its like saying, "Hey, when there's a message, let me know, and here's what I'll do with it". When the SendNotification method is called, it triggers the Notify event, and each subscribed user's ReceiveNotification method is invoked, displaying the received message.

Output:

In simple terms, the delegate (NotificationHandler) acts as a messenger between the NotificationService and the subscribers (Subscriber). It allows the service to notify all subscribers when there's something exciting happening, like sending a voucher. So, the delegate facilitates communication and ensures everyone who's interested gets the news.

Delegates are extensively useful when implementing Observer Design Pattern and in Event-Driven architecture where an object (in our example, NotificationService) needs to notify other objects (such as Subscriber) about changes or events.

Built-in delegates

C# introduced built in delegates types Func<>, Predicate<>, and Action<> in version 2.0. Instead of writing delegates yourself, these built in delegate types can be more expressive way to work with delegates. Func<>, Predicate<>, and Action<> are of generic delegate type meaning they can be used with different types of parameter and return types. This makes them highly reusable in various scenarios reducing the need for custom delegate. It also seamlessly integrate with LINQ expressions, making it more expressive on collections.

I often skip writing my own custom delegates. Func<>, Predicate<>, and Action<> are my preferences due to the versatility and readability.

The versatile: Func<>

The versatility of Func<> allows it to produce result based on the input parameter. It's a generic delegate type that can represent any method with parameters and a return type.

Where to use: you want to get result after doing something.

Example: You want to calculate the total price of items in a shopping cart.

public Func<List<int>, int> calculateTotalPriceInTheCart = items => items.Sum();

List<int> itemInCartPrices = new List<int> { 210, 450, 1500, 2600 };
int totalPrice = eg.calculateTotalPriceInTheCart(itemInCartPrices);
Console.WriteLine(totalPrice);
// Prints: 4760

Conditional crafter: Predicate<>

Predicate<> is perfect for scenarios where you need to evaluate a condition and return a boolean. Predicate<> can be used in filtering collections, conditional validation, removing elements, custom conditions in LINQ, event handling, conditional logic in algorithms, and testing conditions in unit testing, leveraging its true or false outcomes to express a wide range of conditions in a single line of code.

Where to use: you want to check some condition.

Example: You want to check if a customer is eligible for a discount based on their purchase amount. If the total amount exceed 1000, customer receives 10% discount.

public Predicate<int> isEligibleForDiscount = purchaseAmount => purchaseAmount > 1000;

bool eligible = eg.isEligibleForDiscount(totalPrice); // Result: True
Console.WriteLine($"Is eligible for discount: {eligible}");
if (eligible)
{
    Console.WriteLine($"Total price before discount: {totalPrice}");
    totalPrice = totalPrice - (int)(0.1 * totalPrice); // explicit casting. Implicit casting is not allowed from decimal to int.
    Console.WriteLine($"Total price after 10% discount: { totalPrice}");
}
/*
Prints:
Is eligible for discount: True
Total price before discount: 4760
Total price after 10% discount: 4284
*/

The silent executor: Action<>

Action<> is a go-to delegate when you need to represent a method that takes parameters but doesn't return anything (its a void). It focus on action rather than returning the values. Action<> can be used where the focus is on execution rather than computation.

Where to use: focus is on some action rather than computation.

Example: You want to log a confirmation message when a customer makes a purchase.

public Action<string> logPurchaseConfirmationMessage = message => Console.WriteLine($"Log: {message}");

eg.logPurchaseConfirmationMessage("Purchase confirmed for username12345.");
// Action: it prints the log message. 
// Log: Purchase confirmed for username12345.

Multicast delegate

Up to this point, we understood that a delegate references to a method allowing for a level of abstraction and flexibility in function invocation.

Moving on, the concept of multicast delegate is its ability to point to and invoke multiple methods concurrently. Multicast delegate can maintain a list of methods and invoke them concurrently. Its declaration is similar to regular delegate. The difference is we keep adding methods to the delegate. Basically, multicast delegate is a collection with multiple references to methods allowing it to invoke all of them when triggered at the same time.

In our previous example, Subscriber had only one method to receive message ReceiveNotification(string message). We will add another method here ReceiveAnotherNotification(string message). Think of this like apps in your smartphone. You can have bunch of them installed. You can receive messages/notifications from all of them. In our case, we can have as much notification as our subscribers want but we will invoke both of them using multicast delegate. Here is the added method:

public void ReceiveAnotherNotification(string message)
{
    Console.WriteLine($"New notification for you from other service {Username}: {message}");
}

Subscriber class now look like this:

public class Subscriber
{
    public string Username { get; set; }

    public void ReceiveNotification(string message)
    {
        Console.WriteLine($"New notification for you {Username}: {message}");
    }

    // newly added method to demonstrate multicast delegate
    public void ReceiveAnotherNotification(string message)
    {
        Console.WriteLine($"New notification for you from other service {Username}: {message}");
    }
}

Now, we will add ReceiveAnotherNotification(string message) to be included in our notify. += is used to add event handlers (methods or delegates) to an event (notify), allowing multiple subscribers to respond to an event when it occurs.

// create notification service
NotificationService notificationService = new NotificationService();
// create subscribers
List<Subscriber> listOfSubscriber = new List<Subscriber>()
{
    new Subscriber(){ Username = "Priti"},
    new Subscriber(){ Username = "Jon"},
    new Subscriber(){ Username = "Ram"}
};

// Subscribe Users to the Notification Service 
foreach (Subscriber subscriber in listOfSubscriber)
{
    notificationService.Notify += subscriber.ReceiveNotification;
    // add reference of another method to our Notify delegate.
    notificationService.Notify += subscriber.ReceiveAnotherNotification;
    // this is multicast delegate. 
}

// random amount of voucher 
Random randAMount = new Random();
int discountAmount = randAMount.Next(10, 251);

// send notification to the subscribers who are subscribed. 
notificationService.SendNotification($"Here's a voucher of Rs. {discountAmount}! Enjoyyyyyy.");

Everything looks similar to above delegate example except in the foreach loop where we added reference of ReceiveAnotherNotification in our Notify a "messenger". Now, when the SendNotification method is called, the multicast delegate Notify invokes both the ReceiveNotification and ReceiveAnotherNotification methods for each subscriber. It's like receiving notifications from multiple services or apps on your smartphone. The power of multicast delegates shines in scenarios where you want to notify multiple methods or subscribers about an event concurrently, enhancing the flexibility and extensibility of your event-driven architecture.

Summary

1. Delegate:

   • Delegates in C# act as messengers, providing a flexible and powerful mechanism for callback functions, event handling, functional programming, and asynchronous programming.

2. Use Cases:

   • Callback Function: Invoke code after an event or execution.

   • Event Handling: Communicate and respond to occurrences in event-driven architecture.

   • Functional Programming: Pass functions or methods as parameters.

   • Asynchronous Programming: Define callback methods for asynchronous tasks.

3. Example:

   • NotificationService class and Subscriber class demonstrate delegates in action, facilitating communication between a service and subscribers.

4. Built-in Delegate Types:

   • Func<>: Represents a method with parameters and a return type.

   • Predicate<>: Represents a method that evaluates a condition (always returns boolean).

   • Action<>: Represents a method with parameters and no return (its a void).

5. Func<> :

   • Used when you want to produce a result based on input parameters.

   • Example: Calculating the total price of items in a shopping cart.

6. Predicate<> :

   • Perfect for checking conditions and returning a boolean result.

   • Example: Checking if a customer is eligible for a discount based on purchase amount.

7. Action<> :

   • Used when the focus is on executing actions without returning values.

   • Example: Logging a confirmation message when a customer makes a purchase.

8. Multicast Delegates:

   • Extension of regular delegates, pointing to and invoking multiple methods concurrently.

   • Maintain a list of methods for simultaneous invocation.

   • Example: Sending notifications to subscribers using a multicast delegate in NotificationService.

9. More :

   • Delegates simplify code by acting as messengers between different parts of the program.

   • Func<>, Predicate<>, and Action<> are built-in delegate types for added expressiveness and versatility.

   • Multicast delegates enhance flexibility and extensibility in event-driven architectures.

   • Multicast delegate can maintain a list of methods and invoke them concurrently.

Sourcecode.

Now, consider an alternative scenario, instead of looking for a book yourself, you could've asked the librarian. The librarian would hand you the exact book you're looking for. Iterators function much like this librarian. Instead of traversing through the entire collection of book, they promptly provide you with the exact book you seek. The librarian and iterator share a common trait, they give you something only when you ask for it. In above example, it was a book. You asked the librarian for that specific book, you got the book.

An iterator is a method that provides sequence or source for an enumeration using the yield break; or yield return; statement. And yes, it is necessary to use yield keyword to declare an iterator block. yield return statement produces a value and pauses the execution whereas yield break statement terminates the execution of the iterator method. Iterators can return one of the following types:

• IEnumerable

• IEnumerable<T>

• IEnumerator

• IEnumerator<T>

where T can be any type parameter or regular type

If the return type of iterator method is non-generic, the yield type is object whereas for generic, the yield type is the provided type parameter. Eg:

• Non-Generic Iterator method (return type is object) :

public IEnumerable NonGenericIteratorBlock() {
    yield return "hello from non-generic iterator block.";
}

• Generic iterator method (return type is of type parameter, i.e. int) :

public IEnumerable<int> GenericIteratorBlock() {
    yield return 42;
    yield return 100;
}

Iterators stops the execution in the following cases:

• Exception occurs within the iterator block

• Reaches yield return statement which says it is ready to produce the value

• Hits the yield break statement which signals end of the sequence, hence MoveNext() returns false

• End of the method or the iterator block (MoveNext() returns false)

Here is another simple iterator method that prints even numbers. It uses the yield keyword to generate even numbers on demand :

public IEnumerable<int> GenerateEvenNumbers()
{
    for (int i = 0; ; i++)
    {
        yield return i * 2;
    }
}

The absence of condition in the for loop makes it an infinite. To get first 10 even numbers :

Example example = new Example(); // Initialize class which contain iterator method
IEnumerable<int> evenNumbers = example.GenerateEvenNumbers();// iterator method
// invoke iterator by taking first 10 
Console.WriteLine("Iterator block:");
foreach (int evenNumber in evenNumbers.Take(10))
{
    Console.WriteLine(evenNumber);
}

The loop stops after processing 10 elements, demonstrating the lazy evaluation of the iterator. Output of this snippet:

It is totally possible to create a method with similar result without using yield or iterator block.

public List<int> GenerateEvenNumbersNonIterator(int totalCount)
{
    List<int> evenNumbers = new List<int>();
    int i = 0;
    while (evenNumbers.Count < totalCount)
    {
        evenNumbers.Add(i);
        i += 2;
    }
    return evenNumbers;
}

Console.WriteLine("Non-iterator block:");
foreach (int evenNumber in example.GenerateEvenNumbersNonIterator(10))
{
    Console.WriteLine(evenNumber);
}

Output:

Though the end results may seem similar, the main difference is lazy evaluation. Before digging into lazy evaluation, let's look at IEnumerable and IEnumerator.

IEnumerable

IEnumerable interface from System.Collections namespace enables a collection to be enumerated or sequenced over. In simpler terms, implementing IEnumerable allows you to iterate through the collection using a loop. IEnumerable is the base class for all non-generic collections that can be enumerated. It has a single method GetEnumerator() which returns an IEnumerator.

IEnumerator

IEnumerator interface from System.Collections namespace is a standard mechanism used for iterating over a collection. It has methods and properties like Current (to get the current item or element), MoveNext() (advances to the next element), Reset() (resets the iterator block or sets enumerator to its initial value). Enumerator is used to read underlying data in the collection and is not designed for mutable(changing or modifying) operations.

IEnumerable/IEnumerator relationship

Think of an IEnumerable as a playlist of music (collection of music) and IEnumerator as a music player. IEnumerable represents a collection of playlist that you can iterate over. IEnumerator as a music player, it keeps track of the current song in the playlist (collection). The music player is responsible for playing songs one at a time. IEnumerator keeps track of the current song in the playlist (collection) as you iterate over it. You can play or skip each song using a loop and the MoveNext() method respectively. It doesn't change the playlist at all. It only change the music player's current state.

You can have multiple playlists (collections of music). You can also have multiple music players (instances of IEnumerator), each playing its playlist independently. Moving the music player to the next song doesn't affect other music players or the playlists itself.

When you want to start playing through songs, you ask the playlist (IEnumerable) for a new music player (IEnumerator) with IEnumerable.GetEnumerator(). This new music player is set to play through the songs in the playlist. Now, you're in control, able to pause, skip, and enjoy the music. What's interesting is that these actions, like moving to the next song using MoveNext(), only impact the current state of your music player, not the original playlist.

Lazy evaluation

In context of iterator block, lazy evaluation is a mechanism where a sequence is delayed or paused until its value is actually needed. It avoids unnecessary computation until the result is required. With the yield keyword, the iterator provides values on demand and one at a time. So the entire collection is not generated and stored in memory upfront. Iterators can have the following pros :

• Efficient memory usage: As I mentioned, the entire collection is not stored in memory upfront. Hence, storing only when it is required.

• Infinite sequence: Since the values are generated as needed, working with "infinite" sequence is feasibly practical.

• Performant: Iterator avoids unnecessary computation, resulting in improved performance. The difference can be measured when working with resource intensive computation or calculations.

Here is a snippet to compute infinite even numbers:

// example is a instance of a class that has the iterator GenerateEvenNumbers()
IEnumerable<int> enumerable = example.GenerateEvenNumbers();
IEnumerator<int> enumerator = enumerable.GetEnumerator();
while (enumerator.MoveNext())
{
    int val = enumerator.Current;
    Console.WriteLine(val);
}
 // Ctrl + C to stop the execution.

Here, we first created an enumerable of type int (IEnumerable<int>), serving as an iterator for generating even numbers. Then, we instantiated an enumerator for that enumerable using GetEnumerator(), which returns an enumerator of type int. This sets up the collection to be iterated over one element at a time, initializing the enumerator to its initial element. Following that, a while loop checks enumerator.MoveNext(). The MoveNext() method moves the enumerator to the next element in the sequence. If an element exists, it returns true; otherwise, it returns false. The loop continues as long as MoveNext() returns true. enumerator.Current is a property that retrieves the current element in the sequence indicated by the enumerator.

Let's recall again to execute this "infinite" iterator 10 times :

IEnumerable<int> enumerable = example.GenerateEvenNumbers();
foreach(int val in enumerable.Take(10))
{
    Console.WriteLine(val);
}

enumerator.MoveNext() returns true for the first 10 iterations of the loop. The Take(10) method ensures that the loop iterates only 10 times by limiting the number of elements taken from the infinite sequence.

Working mechanism of lazy evaluation

In the above example, we have a iterator GenerateEvenNumbers(); which is a IEnumerable<int>. Nothing happens right away when we call it. Execution happens when we call MoveNext() of GenerateEvenNumbers()'s IEnumerator. The cool part? It doesn't bother generating all the numbers at once. It only works when prompted. Call MoveNext(), get a number. Don't call it, and the iterator stays in sleep mode, not bothering with the rest of the sequence. When MoveNext() is called again, it continues from where it left. Enumerator keeps track of the execution state allowing it to seamlessly pick up from where it left off. It's like bookmark in the book.

Lazy evaluation is like asking for even numbers one at a time using GenerateEvenNumbers(). It doesn't bother showing all the numbers at once; only when you say "Show me more" with MoveNext(). It remembers where it left off, acting like a bookmark in a book, making it easy to continue when you want.

Being lazy enhances efficiency and resources utilization. Unlike other means, where all computations are performed upfront, lazy evaluation delays computation until the result is actually needed. In GenerateEvenNumbers(), it generates even numbers on-demand using yield return. Without lazy evaluation, attempting to generate an infinite sequence would result in a program trying to calculate an infinite number of values, which is not practical or feasible. Nature of lazy enables you to work with potentially infinite data structures in a way that is both efficient and effective.

Using LINQ in Iterators

LINQ operates seamlessly with iterators allowing you to manipulate and be expressive with large set data. Here is a simple example to print even numbers greater than 10 and less than 100.

var conditionalEvenNumbers = example.GenerateEvenNumbers().Where(x => x > 10 & x < 100);
foreach(int val in conditionalEvenNumbers)
{
    Console.WriteLine(val);
}

This snippet prints out even numbers greater than 10 and less than 100 but there's a catch. GenerateEvenNumbers() will never terminate because the sequence is infinite. MoveNext() is not called after 98 (which is the last element from the where clause). The IEnumerator keeps track of its internal state, including the position in the sequence, allowing you to seamlessly continue from where you left off. So, when you conditioned the sequence, the state of the collection is still preserved by IEnumerator. This state-preserving behavior is a fundamental characteristic of iterators and lazy evaluation.

var conditionalEvenNumbers = example.GenerateEvenNumbers().Where(x => x > 10 & x < 100);
foreach(int val in conditionalEvenNumbers.Take(5))
{
    Console.WriteLine(val);
}

Here, we take first 5 elements. This will only take 5 elements that are greater than 10 and less than 100. This ensures that the iterator terminates and doesn't run indefinitely.

When working with iterators, its crucial to handle errors to ensure robustness. Use yield break; to exit iterators. If you want to end the sequence when some condition is met, use yield break;

Summary

1. Iterator:

   • Method that provides sequence or source for an enumeration.

   • Implemented using yield return and yield break statements.

   • Requires the yield keyword in iterator blocks for lazy execution.

2. Return Types of Iterators:

   • Can return types such as IEnumerable, IEnumerable<T>, IEnumerator, or IEnumerator<T>.

   • The return type of non-generic iterators is object, while generic iterators return the specified type parameter.

3. Halting Execution:

   • Iterators pause execution in cases of exceptions, encountering yield return, reaching yield break, or reaching the end of the iterator block.

4. Lazy Evaluation:

   • Lazy evaluation is a key feature of iterators.

   • Mechanism where a sequence is delayed or paused until its value is actually needed.

   • Demonstrated by generating even numbers on demand with the yield keyword.

   • Non-iterator methods compute results upfront, in contrast to the lazy evaluation of iterators.

5. IEnumerable and IEnumerator Relationship:

   • IEnumerable represents a collection, and IEnumerator moves through elements one at a time.

6. Efficient Resource Utilization:

   • Lazy evaluation enhances efficiency by avoiding unnecessary computation until the result is needed.

   • Enables working with potentially infinite data structures.

7. LINQ with Iterators:

   • LINQ operates seamlessly with iterators, allowing expressive manipulation of data.

8. Handling Errors in Iterators:

   • Using yield break to exit iterators when necessary.

   • Using try-catch properly ensures smooth runtime execution.

Sourcecode.

public class List<T> : IList<T>, IList, IReadOnlyList<T>
{
    private const int DefaultCapacity = 4;

    internal T[] _items; 
    internal int _size; 
    internal int _version; 

    private static readonly T[] s_emptyArray = new T[0];

    // Constructs a List. The list is initially empty and has a capacity
    // of zero. 
    public List()
    {
        _items = s_emptyArray;
    }

    // Constructs a List with a given initial capacity. The list is
    // initially empty, but will have room for the given number of elements
    // before any reallocations are required.
    public List(int capacity)
    {
        if (capacity < 0)
            ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument.capacity, ExceptionResource.ArgumentOutOfRange_NeedNonNegNum);

        if (capacity == 0)
            _items = s_emptyArray;
        else
            _items = new T[capacity];
    }

    // Constructs a List, copying the contents of the given collection. The
    // size and capacity of the new list will both be equal to the size of the
    // given collection.
    //
    public List(IEnumerable<T> collection)
    {
        if (collection == null)
            ThrowHelper.ThrowArgumentNullException(ExceptionArgument.collection);

        if (collection is ICollection<T> c)
        {
            int count = c.Count;
            if (count == 0)
            {
                _items = s_emptyArray;
            }
            else
            {
                _items = new T[count];
                c.CopyTo(_items, 0);
                _size = count;
            }
        }
        else
        {
            _items = s_emptyArray;
            using (IEnumerator<T> en = collection!.GetEnumerator())
            {
                while (en.MoveNext())
                {
                    Add(en.Current);
                }
            }
        }
    }

    // methods like: Add(T item), AddRange(IEnumerable<T> collection), 
    // Insert(int index, T item), Remove(T item), RemoveAt(int index), 
    // RemoveAll(Predicate<T> match), Count(), Contains(T item), ...

}

So when a List<int> is initialized, the compiler is replacing all of T in List<T> class including all of its method with int of List<int>. The compiler automatically determines the type of the generic type parameter T in List<T>, a process often referred to as type inference. Type inference is done without explicit casting and providing type safety.

Now, it's time to craft our very own generic class. Imagine we have a ware house where we can store clothes and shoes. To start, we'll define a Cloth class.

public class Cloth
{
    private string _type { get; set; }
    private int ? _price { get; set; }

    public Cloth(string type, int price)
    {
       _type = type;
       _price = price;
    }
}

Following that, we'll define a Shoe class.

public class Cloth : IProductService
{
    private string _type { get; set; }
    private int? _price { get; set; }

    public Cloth(string type, int price)
    {
        _type = type;
        _price = price;
    }
}

Then, we'll create a versatile warehouse class capable of offering flexibility for various storage needs either for cloth or shoe.

public class Warehouse<T>
{
    private List<T> _items;

    public Warehouse()
    {
        _items = new List<T>();
    }

    public void AddItem(T item)
    {
        _items.Add(item);
        Console.WriteLine("Item added successfully.");
    }

    public void RemoveItem(T item)
    {
        _items.Remove(item);
        Console.WriteLine("Item removed.");
    }

    public void PrintTotalItems()
    {
        int itemsCount = _items.Count;
        Console.WriteLine($"Total items in the ware house: {itemsCount}");
    }
}

Let's now get to the business. I would like to create a Warehouse to store Clothes. Before storing cloth in the warehouse, I need some clothes.

Cloth shirt = new("denim jeans", 600);
Cloth jacket = new("leather", 1200);
Cloth sweater = new("woolen", 1800);

Now, a warehouse to store cloth.

Warehouse<Cloth> clothesWareHouse = new();

Store clothes in the warehouse.

clothesWarehouse.AddItem(shirt);
clothesWarehouse.AddItem(jacket);
clothesWarehouse.AddItem(sweater);

Check status of the clothe warehouse. Then, remove an item from the warehouse.

clothesWarehouse.PrintTotalItems();
Console.WriteLine("-- Remove an item from the warehouse. --");
clothesWarehouse.RemoveItem(sweater);
clothesWarehouse.PrintTotalItems();

Output:

Now, discover the true impact of Generics in action. Let's build a warehouse to store Shoes. Before storing, I need some shoes.

Shoe highTop = new("converse", 42, 900);
Shoe boot = new("gucci", 43, 3400);
Shoe lowTop = new("air max 90", 42, 4500);

A warehouse to store them.

Warehouse<Shoe> shoeWarehouse = new();

Store shoes in the warehouse.

shoeWarehouse.AddItem(highTop);
shoeWarehouse.AddItem(boot);
shoeWarehouse.AddItem(lowTop);

Check status of the shoe warehouse. Then, remove 2 items from the warehouse.

shoeWarehouse.PrintTotalItems();
Console.WriteLine("-- Remove 2 items from the warehouse. --");
shoeWarehouse.RemoveItem(highTop);
shoeWarehouse.RemoveItem(boot);
shoeWarehouse.PrintTotalItems();

Output:

We used the same Warehouse to store clothes and shoes. What did we achieve? Well, we didn't create a separate class for ware house. By using generics, we've created a versatile Warehouse class that can adapt to different types of items without the need for separate implementations for clothes and shoes.

We achieved:

• Code re-usability : The Warehouse adapts to various item types, providing the freedom to store clothes, shoes, or any other compatible items without rewriting the code.

• Flexibility : The Warehouse can store different types of items like clothes, shoes, and any other compatible items without needing to change the code.

• Reduced redundancy : Generics cut out the need for repeating code across different classes, eliminating the necessity for duplicate code in separate classes for different types.

• Maintainability : Centralizing logic improves maintainability, enhancing the overall manageability of our code.

Type Constraints

Here are some problems with our plain, non-restricted generic class. What if someone wants to store string in a warehouse like Warehouse<string> or any other type. Warehouse is specifically for a "Product" which might not make sense for a warehouse to store type other than "Product". By introducing type constraints to ensure that only items of the specified type ("Product" in our scenario) can be added to the warehouse. Type constraint enhance type safety and prevent such potential runtime errors. We will introduce an interface IProductService, which will be implemented by any "Product". Let's construct IProductService and add a method that display product details.

public interface IProductService 
{
   void DisplayProductDetails();
}

Implement IProductService in our product types.

public class Cloth : IProductService
{
    private string _type { get; set; }

    private int? _price { get; set; }

    public Cloth(string type, int price)
    {
       _type = type;
       _price = price;
    }

    public void DisplayProductDetails()
    {
        Console.WriteLine($"A cloth product of type:{_type}");
    }
}

public class Shoe : IProductService
{
    private string _brand { get; set; }
    private int _size { get; set; }
    private int ? _price { get; set; }

    public Shoe(string brand, int size, int? price)
    {
        _brand = brand;
        _size = size;
        _price = price;
    }

    public void DisplayProductDetails()
    {
        Console.WriteLine($"A shoe product of brand: {_brand} and of size: {_size}");
    }
}

Now, to make Warehouse to store only "Product" type, we will introduce a type constraint in our Warehouse<T> generic class.

public class Warehouse<T> where T : IProductService
{
    private List<T> _items;

    public Warehouse()
    {
        _items = new List<T>();
    }

    public void AddItem(T item)
    {
        _items.Add(item);
        Console.WriteLine("Item added successfully.");
    }

    public void RemoveItem(T item)
    {
        _items.Remove(item);
        Console.WriteLine("Item removed.");
    }

    public void TotalItems()
    {
        int itemsCount = _items.Count;
        Console.WriteLine($"Total items in the ware house: {itemsCount}");
    }

    public void DisplayAllProductDetails()
    {
        foreach(T item in _items)
        {
            item.DisplayProductDetails();
        }
    }
}

In Warehouse class with a type parameter T, we added DisplayAllProductsDetails() method to print product detail. The where T : IProductService constraint specifies that the generic type T must implement the IProductService interface. Warehouse cannot be accept type beside IProductService.

Below, since the Cloth type implements IProductService, Warehouse can store them.

// Create some clothes
Cloth shirt = new("denim jeans", 600);
Cloth jacket = new("leather", 1200);
Cloth sweater = new("woolen", 1800);

// create warehouse to store clothes
Warehouse<Cloth> clothesWarehouse = new();
clothesWarehouse.AddItem(shirt);
clothesWarehouse.AddItem(jacket);
clothesWarehouse.AddItem(sweater);
clothesWarehouse.TotalItems();
Console.WriteLine("-- Remove an item from the warehouse. --");
clothesWarehouse.RemoveItem(sweater);
clothesWarehouse.TotalItems();
clothesWarehouse.DisplayAllProductDetails();

Output:

If we try to store other than IProductService type in our generic Warehouse, we will get compile-time error. Let's look at this by creating a type of Person.

public class People
{
    private string Name { get; set; }
    private int Age { get; set; }
    public People(string name, int age)
    {
        Name = name;

        Age = age;
    }
}

If I try to create Warehouse to store People :

Type constraints ensure the allowance of only certain types, enhancing code safety by preventing inappropriate types from being used and catching potential errors at compile-time rather than runtime. In our case, the IProductService constraint in Warehouse<T> ensures that only IProductService types can be stored in our warehouse.

Generic methods

In addition to generic classes, C# design teams also introduced generic methods. Generic methods offers a powerful way to write functions that work with a variety of data types without sacrificing type safety. They are commonly used in algorithms and collection manipulation functions, where the type of data being processed can vary. Let's dive into an example. We will create a class that will provide us basic services of accounting. We will add a specific method to check if the product is expensive or not.

public class AccountingService
{
    public AccountingService() { }

    /// <summary>
    /// Method to return consumption percentage.
    /// </summary>
    /// <returns></returns>
    public int GetConsumptionPercentage(int totalItem, int currentItem)
    {
        return totalItem/currentItem * 100;
    }

    /// <summary>
    /// Is expensive if more than 1000.
    /// </summary>
    /// <returns></returns>
    public bool IsProductExpensive<T>(T product) where T : IProductService
    {
        return product.GetProductPrice() > 1000;
    }
}

Here, IsProductExpensive<T>(T product) is an generic method that will return the product expensiveness. For sake of type safety, only IProductService type can be checked. I added GetProductPrice() in IProductService. So, each type that implements IProductService will have GetProductPrice(). Cloth and Shoe class will have a method, something like this:

public int GetProductPrice()
{
    return _price ?? 0;
}

Our generic method returns bool. Let's print if a particular cloth is expensive.

Cloth sweater = new("woolen", 1800);
// check if sweater is expensive. Our logic insists > 1000 to be expensive.
AccountingService accountingService = new AccountingService();
var isExpensive = accountingService.IsProductExpensive(sweater);
Console.WriteLine($"Is product expensive:{isExpensive}");

Output:

More generic constraints

C# supports many generic constraints. Following are some of the type constraint:

• new() : The new() constraint in the Warehouse<T> class specifies that the generic type parameter T must have a public parameterless constructor. This means that you can create an instance of T without providing any constructor arguments. However, it doesn't restrict the use of types that have other constructors; it just ensures the existence of a parameterless constructor.

  Syntax to use new() constraint:

    public class Warehouse<T> where T : new()
    {
       // methods
    }
  

Warehouse now can be used by only type that have parameterless constructor. If we recall our Cloth.cs, Shoe.cs, and People.cs, and want to initialize them, they all have public parameterized constructor. Hence, Warehouse<T> where T : new() cannot be used, we will get an compile-time error. They must be have public parameterless parameter.

• Interface <interface name> : Interface constraint ensures generic type parameter to be implemented by the provided interface. Our original Warehouse uses interface type constraint.

  Syntax to use interface type constraint:

    public class Warehouse<T> where T : IProductService
    {
       // methods
    }
  

• Class name (<base class>) : The class name constraint, also known as the base class constraint, ensures that a generic type parameter must be derived from or be the provided class. Our original Warehouse uses interface type constraint.

  Syntax to use base class type constraint:

    public class Warehouse<T> where T : SomeBaseClass
    {
       // methods
    }
  

• notnull : notnull constraint ensures the generic type parameter to be a non-nullable reference or value type.

  Syntax to use notnull type constraint:

    public class Warehouse<T> where T : notnull
    {
       // methods
    }
  

Generic Variance

Generic variance refers to the ability to use a more specific type than originally specified (covariance) or a less specific type than originally specified (contravariance).

Covariance

Covariance allows you to use a more derived type than originally specified. For example, we can create a general Warehouse to store our product (both shoe and cloth). We can achieve this by initializing a Warehouse<IProductService>.

// create shoe
Shoe ggSneaker = new("gucci", 42, 4400);
Shoe airMax = new("air max 90", 42, 4500);
// create cloth
Cloth winterCloth = new("hoodie", 5500);
Cloth winterCloth1 = new("sweater", 1400);
/* covariance example */
Warehouse<IProductService> covarianceWarehouse = new();
// add cloth type 
covarianceWarehouse.AddItem(winterCloth);
covarianceWarehouse.AddItem(winterCloth1);
// add shoe type
covarianceWarehouse.AddItem(ggSneaker);
covarianceWarehouse.AddItem(airMax);
// print total items of covariance ware house
covarianceWarehouse.DisplayAllProductDetails();

Output:

Covariance allows you to treat a Warehouse<Derived> as a Warehouse<Base>, where Derived is a type derived from Base. Here, IProductService serves as the common base interface for both Cloth and Shoe, allowing to use covariance. The out keyword is used to explicitly declare covariance when dealing with generic types in interfaces and delegates. It provides a way to communicate the intent of the generic parameter usage. We can modify our IProductService like so:

public interface IProductService<out T> 
{
    void DisplayProductDetails();
    int GetProductPrice();
}

The out keyword indicates covariance for the generic type parameter T in interfaces or delegates. It allows the type parameter to be used as a more derived (specific) type than originally specified, providing flexibility in scenarios where a more specific type is expected. Now, we need to modify all our IProductService derived classes like so:

Shoe.cs

public class Shoe : IProductService<object>
{
   // methods and properties
}

Cloth.cs

public class Cloth : IProductService<object>
{
   // methods and properties
}

Warehouse.cs

public class Warehouse<T> where T : IProductService<object>
{
   // methods and properties
}

Main method

// create shoe
Shoe ggSneaker = new("gucci", 42, 4400);
Shoe airMax = new("air max 90", 42, 4500);
// create cloth
Cloth winterCloth = new("hoodie", 5500);
Cloth winterCloth1 = new("sweater", 1400);
/* covariance example */
Warehouse<IProductService<object>> covarianceWarehouse = new(); // using object as type argument
// add cloth type 
covarianceWarehouse.AddItem(winterCloth);
covarianceWarehouse.AddItem(winterCloth1);
// add shoe type
covarianceWarehouse.AddItem(ggSneaker);
covarianceWarehouse.AddItem(airMax);
// print total items of covariance ware house
covarianceWarehouse.DisplayAllProductDetails();

Output:

Using object as a type parameter can be a solution in some cases, but it comes with a trade-off. When you use object as the generic type parameter, you sacrifice type safety and lose the benefits of working with specific types. Using "object" doesn't check type, so errors are only discovered at runtime which is a headache when debugging. In our scenario, we want to create a Warehouse that can store various types of products (clothes and shoes), it might be more beneficial to create a non-generic interface. Hence, it is not compulsory to use out keyword to achieve covariance.

Contravariance

Contravariance allows you to use a more base type than originally specified. Contravariance is exactly on the flip side of covariance. For example, with contravariance, you can treat a Warehouse<Product> as a Warehouse<Shoe>. However, you can only add shoes to it, not clothes.

// Contravariant use of Warehouse
Warehouse<Shoe> contravariantWarehouse = new();
contravariantWarehouse.AddItem(partyShoe);
contravariantWarehouse.AddItem(runningShoe);
//contravariantWarehouse.AddItem(nightWear); // Compile time error
contravariantWarehouse.DisplayAllProductDetails();

Here, we created an warehouse of type shoe. We cannot add other than shoe in contravariantWarehouse.

Output:

In this continuation, I've added the creation of a Shoe instance for partyShoe, runningShoe, and a Cloth instance for nightWear. As mentioned, attempting to add nightWear to contravariantWarehouse will result in a compile-time error since contravariantWarehouse is specifically typed to only accept Shoe instances.

Contravariance provides flexibility when dealing with generic types, allowing you to treat a more specific type as a more general type. This can be useful in scenarios where you want to handle a broader range of types while maintaining type safety.

Summary

1. Generics in C#:

   • Introduced in C# version 2.0 for handling types at compile time.

   • Used for creating general-purpose code that works with different types.

   • Commonly used in collections like List<T>, providing type safety and code reusability.

2. Generic Classes:

   • Example of a generic class: Warehouse<T> capable of storing various types of products.

   • Achieves code reusability, flexibility, reduced redundancy, and maintainability.

   • Compiler determines the type of the generic parameter during compilation.

3. Achievements of using generics:

   • Code reusability: Warehouse adapts to various item types.

   • Flexibility: Warehouse stores different types without code modification.

   • Reduced redundancy: Generics eliminate the need for duplicate code.

   • Maintainability: Centralized logic improves code manageability.

4. Type Constraints:

   • Introduced to enforce specific conditions on generic types.

   • Example: Using an interface constraint (IProductService) to ensure only specific types can be stored in the warehouse.

5. Generic Methods:

   • Allows writing functions that work with a variety of data types without sacrificing type safety.

   • Example: AccountingService with a generic method IsProductExpensive<T> using type constraints.

6. More Generic Constraints:

   • new() constraint ensures a public parameterless constructor.

   • Interface and base class constraints restrict types to those implementing a specific interface or derived from a base class.

   • notnull constraint ensures a non-nullable reference or value type.

7. Generic Variance:

   • Covariance allows using a more derived type than originally specified.

   • Contravariance allows using a more base type than originally specified.

   • Example: Using covariance in a Warehouse<IProductService> to store both shoes and clothes.

8. Conclusion:

   • Generics in C# provide a powerful mechanism for creating flexible and reusable code.

   • Type constraints enhance type safety and prevent inappropriate types at compile-time.

   • Generic variance (covariance and contravariance) allows more flexibility in handling types.

Sourcecode.