Thousands of Threads and Blocking I/O

• For an NIO based server, the server notifies when some I/O event is ready to be processed, this is then processed; since all I/O is effectively multiplexed, it requires the server to keep track of where each client is within its i/o transaction, i.e. state must be maintained for all clients (unless a stateless protocol is used, e.g. all state is part of the request).
• NIO is not faster than IO, but it can be more scalable, though the scalability is an issue of how efficient the OS is at handling many threads.
• NIO transfers rate can be only 75% of a plain IO connection (several benchmark studies show this sort of comparative maximum rate).
• A multithreaded IO server tends to automatically takes advantage of multiple cores, where an NIO server may explcitly need to hand processing off to a pool of worker threads (though that is the common design)
• On modern OSs, idle threads have not much cost, context switching is fairly efficient, uncontended synchronization is cheap.

• Nonblocking datastructures scale well - ConcurrentLinkedQueue, ConcurrentHashMap, NonBlockingHashMap (&NonBlockingLongHashMap)
• A good architecture throttles incoming requests to the maximum rate the server can handle optimally, otherwise if the server gets overloaded overall request rates as well as individual request service times drop to unnacceptable levels.
• Avoid Executors.newCachedThreadPool as an unbounded number of threads tends to be bad for applications (e.g. more threads get created just when you are already maxxed on CPU).
• If you do mutliple sends per request, use a buffered stream. If one send per request, don't buffer (as you effectively already have).
• Try to keep everything in byte arrays if possible, rather than converting back and forth between bytes and strings.
• In a thread-per-request model, watch for socket timeouts.
• Multithreaded server coding is more intuitive than an event based server.

details - http://www.mailinator.com/tymaPaulMultithreaded.pdf

Fast and Safe concurrency - Actor framework for java
How to get C like performance in java
High Performance Serialization
How much time out if your day does ibm waste