Let’s start right at the beginning. Bear with me, it might get long…
The following snippet of Java code could be used to create a server which receives TCP/IP requests:
class Server implements Runnable {
public void run() {
try {
ServerSocket ss = new ServerSocket(PORT);
while (!Thread.interrupted())
Socket s = ss.accept();
s.getInputStream(); //read from this
s.getOutputStream(); //write to this
} catch (IOException ex) { /* ... */ }
}
}
This code runs as far as the line with ss.accept(), which blocks until an incoming request is received. The accept method then returns and you have access to the input and output streams in order to communicate with the client.
There is one issue with this code. Think about multiple requests coming in at the same time. You are dedicated to completing the first request before making the next call to the accept method. Why? Because the accept method blocks. If you decided you would read a chunk off the input stream of the first connection, and then be kind to the next connection and accept it and handle its first chunk before continuing with the original (first) connection, you would have a problem, because the accept method blocks. If there were no second request, you wouldn’t be able to finish off the first request, because the JVM blocks on that accept method. So, you must handle an incoming request in its entirety, before accepting a second incoming request.
This isn’t so bad, because you can create the ServerSocket with an additional parameter, called the backlog, which tells it how many requests to queue up before refusing further connections. While you are busy handling the first request, subsequent requests are simply queued up.
This strategy would work, although it’s not really efficient. If you have a multicore CPU, you will only be doing work on one core. It would be better to have more threads, so that the load can be balanced across the cores (watch out, this is JVM and OS dependent!).
A more typical multi-threaded server gets built like this:
class Server implements Runnable {
public void run() {
try {
ServerSocket ss = new ServerSocket(PORT);
while (!Thread.interrupted())
new Thread(new Handler(ss.accept())).start();
// one thread per socket connection every thread
// created this way will essentially block for I/O
} catch (IOException ex) { /* ... */ }
}
}
The above code hands off each incoming request to a new thread, allowing the main thread to handle new incoming requests, while spawned threads handle individual requests. This code also balances the load across CPU cores, where the JVM and OS allow it. Ideally, we probably wouldn’t create a thread per new request, but rather hand off the request to a thread pool executor (see the java.util.concurrent package). On the other hand, there are times when a thread per request is required. If the conversation between server and client is longer lasting (rather than a simple HTTP request that is typically serviced in anything from milliseconds to seconds), then the socket can stay open. An example of when this is required are things like chat servers, or VOIP, or anything else where a continual conversation is required. But in such situations, the above code, even though it is multi-threaded, has it’s limits. Those limits are actually because of the threads! Consider the following code:
public class MaxThreadTest {
static int numLive = 0;
public static void main(String[] args) {
while(true){
new Thread(new Runnable(){
public void run() {
numLive++;
System.out.println("running " + Thread.currentThread().getName() + " " + numLive);
try {
Thread.sleep(10000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
numLive--;
}
}).start();
}
}
}
This code creates a bunch of threads, until the process crashes. With 64 MB heap size, it crashed (out of memory) after around 4000 threads, while testing on my Windows XP Thinkpad laptop. I upped the heap size to 256 MB and Eclipse crashed while in debug mode… I started the process from the command line and managed to open 5092 threads, but it was unstable and unresponsive. Interestingly, I upped the heap size to 1 GB, and then I could only open 2658 threads… This shows, I don’t really understand the OS or JVM at this level! Anyway, if we were writing a system to handle a million simultaneous conversations, we would probably need over two hundred servers. But theoretically, we could reduce our costs to less than 10% of that, because we are allowed to open just over 65,000 threads per server (well, say 63,000 by the time we account for all the ports used by the OS and other processes). We could theoretically get away with just having 16 servers per million simultaneous connections.
The way to do this is, is to use non-blocking I/O. Since Java 1.4 (around 2002?), the java.nio package has been around to help us. With it, you can create a system which handles many simultaneous incoming requests using just one thread. The way it works is roughly by registering with the OS to get events when something happens, for example when a new request is accepted, or when one of the clients sends data over the wire.
With this API, we can create a server, which is, sadly, a little more complicated than those above, but which handles lots and lots of sockets all from one thread:
public class NonBlockingServer2 {
public static void main(String[] args) throws IOException {
System.out.println("Starting NIO server...");
Charset charset = Charset.forName("UTF-8");
CharsetDecoder decoder = charset.newDecoder();
CharsetEncoder encoder = charset.newEncoder();
ByteBuffer buffer = ByteBuffer.allocate(512);
Selector selector = Selector.open();
ServerSocketChannel server = ServerSocketChannel.open();
server.socket().bind(new InetSocketAddress(30032));
server.configureBlocking(false);
SelectionKey serverkey = server.register(selector, SelectionKey.OP_ACCEPT);
boolean quit = false;
while(!quit) {
selector.select(); //blocks until something arrives, of type OP_ACCEPT
Set keys = selector.selectedKeys();
for (SelectionKey key : keys) {
if (key == serverkey) {
if (key.isAcceptable()) {
SocketChannel client = server.accept();
if(client != null){ //can be null if theres no pending connection
client.configureBlocking(false);
SelectionKey clientkey = client.register(selector,
SelectionKey.OP_READ); //register for the read event
numConns++;
}
}
} else {
SocketChannel client = (SocketChannel) key.channel();
if (!key.isReadable()){
continue;
}
int bytesread = client.read(buffer);
if (bytesread == -1) {
//whens this happen?
key.cancel();
client.close();
continue;
}
buffer.flip();
String request = decoder.decode(buffer).toString();
buffer.clear();
if (request.trim().equals("quit")) {
client.write(encoder.encode(CharBuffer.wrap("Bye.")));
key.cancel();
client.close();
}else if (request.trim().equals("hello")) {
String id = UUID.randomUUID().toString();
key.attach(id);
String response = id + "\r";
client.write(encoder.encode(CharBuffer.wrap(response)));
}else if (request.trim().equals("time")) {
numTimeRequests++;
String response = "hi " + key.attachment() + " the time here is " + new Date() + "\r";
client.write(encoder.encode(CharBuffer.wrap(response)));
}
}
}
}
System.out.println("done");
}
}
The above code is based on that found here. By reducing the number of threads being used, and not blocking, but rather relying on the OS to tell us when something is up, we can handle many more requests. I tested some code very similar to this to see how many connections I could handle. Windows XP proved its high reliability, when reproducibly and consistently, more than 12,000 connections lead to blue screens of death! Time to move to Linux (Fedora Core). I had no problems creating 64,000 clients all simultaneously connected to my server. Let me re-prase… I didn’t have problems having the clients simply connect and keep the connection open, but getting the server to also handle just 100 requests a second caused problems. Now 100 requests a second on a web server, on hardware which was a 5 year old cheap Dell laptop, sounds quite impressive to me. But on a server with 64,000 concurrent connections, that means each client making a request every ten minutes! Not very good for a VOIP application… The connection speeds also slowed down from around 3 milliseconds with 500 concurrent connections, down to 100 milliseconds with 60,000 concurrent connections.
So, perhaps I better get to the point of this posting? A few days ago, I read about Voxer, and Node.js on The Register. I had difficulty with this article. Why would anyone want to build a framework for Javascript on the server? I have developed plenty of rich clients, and have the experience to understand how to do rich client development. I have also developed plenty of rich internet apps (RIA), which use Javascript, and I can only say, it’s not the best. I’m not some script kiddie or script hacker who doesn’t know how to design Javascript code, and I understand the problems of Javascript development well. And I have developed lots and lots of server side code, mostly in Java and appreciate where Java out punches Javascript.
It seems to me, that the developers of Node.js, and those following it and using it, don’t understand server development. While writing in Javascript might initially be quicker, the lack of tools and libraries in comparison to Java make it a non-competition in my opinion.
If I were a venture capitalist, and knew my money was being spent on application development based on newly developed frameworks, instead of extremely mature technologies, when the mature technologies suffice (as shown with the non-blocking server code above), I would flip out and can the project.
Maybe though, this is why I have never worked at a start up!
To wrap up, let’s consider a few other points. Before anyone says that the performance of my example server was poor because it’s just Java which is slow, let me comment. First of all, Java will always be faster than Javascript. Secondly, using top to monitor the server, I noticed that 50% of the CPU time was spent by the OS working out what events to throw, rather than Java handling those requests.
In the above server, everything runs on one thread. To improve performance, once a request comes in, it could be handed off to a thread pool to respond. This would help balance load across multiple cores, which is definitely required to make the server production ready.
While I’m at it, here is a quote from Node JS’s home page:
"But what about multiple-processor concurrency? Aren’t threads necessary to scale programs to multi-core computers? Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ) which fits well into the current design."
Actually, I’m not so sure… Java on Linux can spread threads across cores, so individual processes are not actually required. And the above statement just proves that Node JS is not mature for building really professional systems – I mean come on, no threading support?!
So, in the interests of completion, here is the client app I used to connect to the server:
public class Client {
private static final int NUM_CLIENTS = 3000;
static Timer serverCallingTimer = new Timer("servercaller", false);
static Random random = new Random();
/**
* this client is asynchronous, because it does not wait for a full response before
* opening the next socket.
*/
public static void main(String[] args) throws UnknownHostException, IOException, InterruptedException {
final InetSocketAddress endpoint = new InetSocketAddress("192.168.1.103", 30032);
System.out.println(new SimpleDateFormat("HH:mm:ss.SSS").format(new Date()) + " Starting async client");
long start = System.nanoTime();
for(int i = 0; i < NUM_CLIENTS; i++){
startConversation(endpoint);
}
System.out.println(new SimpleDateFormat("HH:mm:ss.SSS")
.format(new Date())
+ "Done, averaging "
+ ((System.nanoTime() - start) / 1000000.0 / NUM_CLIENTS)
+ "ms per call");
}
protected static void startConversation(InetSocketAddress endpoint) throws IOException {
final Socket s = new Socket();
s.connect(endpoint, 0/*no timeout*/);
s.getOutputStream().write(("hello\r").getBytes("UTF-8")); //protocol dictates \r is end of command
s.getOutputStream().flush();
//read response
String str = readResponse(s);
System.out.println("New Client: Session ID " + str);
//send a request at regular intervals, keeping the same socket! eg VOIP
//we cannot use this thread, its the main one which created the socket
//simply create another task to be carried out by the scheduler at a later time
//the interval below is 4 minutes, otherwise the server gets REALLY slow handling
//so many requests. This is equivalent to ~260 reqs/sec
serverCallingTimer.scheduleAtFixedRate(
new ConversationContainer(s, str),
random.nextInt(240000/*in the next 4 mins*/),
240000L/*every 4 mins*/);
}
private static String readResponse(Socket s) throws IOException {
InputStream is = s.getInputStream();
int curr = -1;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while((curr = is.read()) != -1){
if(curr == 13) break; //protocol dictates a new line is the end of a response
baos.write(curr);
}
return baos.toString("UTF-8");
}
private static class ConversationContainer extends TimerTask {
Socket s;
String id;
public ConversationContainer(Socket s, String id){
this.s = s;
this.id = id;
}
@Override
public void run() {
try {
s.getOutputStream().write("time\r".getBytes("UTF-8")); //protocol dictates \r is end of command
s.getOutputStream().flush();
String response = readResponse(s);
if(random.nextInt(1000) % 1000 == 0){
//we dont want to log everything, because it will kill our server!
System.out.println(id + " - server time is '" + response + "'");
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Copyright © 2010 Ant Kutschera
