Will the Guru

Thread per Request

• The more threads, the higher throughput
  • However, if the thread count is beyond some point, creating the threads only slow down the application because threads consume CPU/Memory as itself

---

Thread Pool

• Sizing the thread pool is difficult
  • If the thread pool size is too small, throughput suffers.
  • If the thread pool size is too big, resource shortage can occur.

How to size a thread pool?

#1. CPU-bound task

• Definition
  • Tasks that utilizes processor a lot during the execution.
• Optimal size of thread pool
  • No much advantages than having more threads than CPU cores.
  • In theory: The number of CPU cores + 1
  • In practice: Run a load test and observe the level of CPU utilization

#2. I/O-bound task

• Definition
  • Large number of input/output operations, such as reading data from disk or network.
• Optimal size of thread pool
  • Thread pool size can be quite big, as many threads will be waiting for I/O operations to complete, making CPU less busy and allowing it to handle more threads concurrently.
  • In theory: The number of CPU cores * (1 + wait time / service time)
    • wait time: time spent waiting for IO operations to complete (CPU idle time)
    • service time: CPU busy time
  • In practice: Run a load test and observe the level of CPU utilization

Graceful Shutdown

Graceful Shutdown Sequence

1. Thread pool stops accepting any new tasks
2. Thread pool waits for the previously submitted tasks to execute (e.g. several seconds)
3. Remaining tasks are cancelled (be careful here as we need to make sure no tasks are lost)
4. Thread pool terminates.

---

Reference

• https://leetcode.com/explore/interview/card/system-design-for-interviews-and-beyond/726/the-importance-of-queues-in-distributed-systems/4765/