Socket
Socket is an endpoint which is combination of IP address and port number.
From client <-> server, each end of connection has a socket.
Server usually has a port that’s opened for making connection (e.g. port 80). Then, when client tries to connect with the server, client assigns a random(OS assigns it) port number to make a connection with the server. This port is called ‘Ephemeral Port (Short-lived)’. This ephemeral port allows the same client to make connection with the server with multiple port numbers.
Blocking vs Non-Blocking I/O
There are two types of approaches for input/output processing.
Blocking and Non-Blocking I/O applies to how we read data from the socket and how we write response data to the socket.
#1. Blocking I/O
For server to handle multiple connections concurrently, the server needs to create a thread per connection. When one thread is blocked for waiting data to arrive, server makes progress by switching between threads and reading data from other sockets.
Blocking I/O waits for the I/O operation to complete
#2. Non-Blocking I/O
Uses single thread to handle multiple connections. Typically, one thread per CPU core. Each thread handles multiple connections.
Each request is made of multiple packets. Instead of waiting until the next packet arrives, the I/O thread will process with other request’s packet. This is possible due to OS, which raises an event when a socket is ready for I/O. When the thread gets ready for I/O, it’ll get assigned with other request that can be processed.
Non-blocking I/O does not wait for I/O operation to complete.
Benefit of Non-Blocking I/O: Makes the connection really cheap. The connection is cheap because itself is just a file descriptor and listener.
This allows the server to handle more connections than Blocking I/O. While thread consumes significantly more resources.
Models of how requests are processed
#1. Thread per connection
There are dedicated threads per every connection. The business logic of application is executed inside the worker thread. Each worker thread is basically blockable.
#2. Thread per request with non-blocking I/O
Set of worker threads do not map one-to-one with connections.
A single I/O thread per CPU core can read bytes from several different sockets. Then it writes data to dedicated buffers. When enough bytes are read from a particular socket, meaning that there are enough bytes read to construct an HTTP request, one of the worker threads from the pool starts to process.
- Good for CPU-bound workloads
- Pros and Cons
- Pros: Easier to implement, test and debug applications
- Cons: Each thread consumes resources (memory, CPU). We have to limit the number of concurrent requests(Load shedding and rate limiting)
- ex) Jetty, Nginx, Tomcat
#3. Event Loop
There is a non-blocking I/O thread which is called an event loop. Not only acts as a I/O thread, this thread also acts as a worker thread. Which means this thread executes the application-specific code for request processing for all requests.
This can be thought as a task queue.
After the event loop thread reads the request bytes from the socket, it creates an event or task for processing the request. This can be one or several tasks for a single request. Tasks are submitted to the queue for asynchronous processing. The event queue then takes the next task from the queue and works on it.
- Good for I/O-bound workloads (e.g. large files)
- Pros and Cons
- Pros: Massive amount of active connections. More resilient to sudden traffic spikes (Because no need to make more threads)
- Cons: Increased development and operational complexity.
- ex) Netty, Node.js, Zuul
Notes
- 1 CPU core can run 1 thread at a time
Will the Guru 
Leave a comment