Hello all.
This is a very long post, but necessarily so, in order to describe all the very peculiar detail and circumstance! Please please stick with it…
OSXvnc is great, and I use it very successfully both plain (home LAN) and via SSH (everywhere else!), but I occasionally stumble across a problem connecting client to server, especially if the server has been up for a while. Once this occurs, the problem is (fairly) persistent.
My setup is, (most usually but exclusively) CotVNC and OSXvnc. Hardware/software are all various, including PM, PB, Mini, iMac (all PPC) with OS ranging from 10.3.9 - 10.4.8 but I also keep VNC Dimension and VNC Thing to hand, from the time that I was assessing them all. This becomes relavent, as described below. I also use a VNC password.
On the occassions that this error occurs, I go from a completely working scenario, to a completely failing scenario. Upon attempting connection, I get half a connection (remote screen flashes up) but CotVNC imediately reports that the connection was terminated. I have checked and checked all the details and everything is correct. Fast forward a bit (over all the detail below) and I believe that, basically, this involves incorrect characters being transferred during the connect/authenticate process. Details below describe in more detail how I have arrived at this conclusion but of course, I welcome everyone’s input in case I am missing or misunderstanding something!
When I first encountered the problem, my usual ‘solution’ was to restart everything, as I was new to VNC, SSH, port forwarding etc., and figured I had messed something up!
Eventually, I stumbled across another ‘solution’, which was to press the ‘fn’ key a couple of times prior to initiating the connection, and this seemed to work, for the most part.
I also discovered, again by chance, that attempting to connect to OSXvnc using VNC Dimension would be successful (when CotVNC wasn’t) but thereafter, CotVNC would connect fine, as if VNC Dimension had ‘cleared’ something.
One of the above was my usual solution… until this evening when, for the first time (that I recall) neither ‘cleared’ the problem. About to reboot my client machine, I decided to first check the other clients again, and I noticed that…
VNC Thing failed immediately, like CofVNC, but VNC Dimension made a successful connection to the machine. Interestingly, it differs from the other two by asking for the VNC password in a second stage. However, once connected, I was still unable to get past the user login screen on the remote machine, which was reporting that the password was incorrect, which it wasn’t.
I then decided to clear the (pre-entered) usename in the user login window, and start the data entry from scratch. As I entered my username, what followed was a curious collection of strange and bizarre characters. Clearly, this is why the password previously entered (and viewed only as bullets) was being rejected. Ok progress…
So assuming this was happening (to different degrees) across three VNC clients, I initially figured that my client machine (maybe the OS) was the culprit, sending out dodgy characters. This begged one more check. Given exactly the same circumstances (nothing rebooted) what would happen with a freshly booted second client machine trying to access the server…?
Answer… exactly the same! So, I figured it must surely be OSXvnc ‘interpreting’ the characters incorrectly, rather than them being ‘sent’ incorrectly.
One further point of interest, the numerals from the password always brought up the ‘alt’ key equivalent. So, on a UK keyboard entering ‘6’ would yield ‘?’ etc. To make things more bizarre, entering the numbers with the ‘ctrl’ keys pressed did actually yield the correct number. I couldn’t find a way of getting the alpha characters correct 
So, a quick reboot of the OSXvnc server (it’s headless, so SSH in and then sudo shutdown -r now) did the trick and the previously failing clients all worked perfectly! 
I’d be very happy if it was possible to investigate this further. Has anyone else encountered this?!
If you’ve stuck with this post as far as this… many thanks for your patience
Best - John