After upgrading to Mountain Lion and thus the latest Vine Server I too was experiencing frequent “Illegal server message type 131” problems that could only be solved by completely killing the server.
Today I was finally annoyed enough by it that I installed XCode and started investigating myself. It’s actually pretty simple, the richClipboardSupport member of the rfbClientRec is uninitialized. When the server process is fresh it defaults to zero, which is fine. The longer the server runs, the higher the probability becomes that it won’t be zero anymore for the next connection and thus triggers the problem.
In the end this little change in rfbserver.c is all that’s needed:
cl = (rfbClientPtr)xalloc(sizeof(rfbClientRec));
- memset(cl, 0, sizeof(rfbClientRec));
This being an uninitialized memory bug it might also cure other problems, who knows.
I’m not in the business of distributing compiled packages, I’m not even a Mac developer, but I hope this fix finds its way into the release version soon for everybody who also suffers from this problem.