Ok we have finally identified the source of this bug. This is not a Vine Viewer bug, it is definitely a bug in UltraVNC-SC. Specifically when doing authentication for an RFB 3.8 viewer UltraVNC-SC passes 0 for the number of security types. From the RFB Specification:
If number-of-security-types is zero, then for some reason the connection failed (e.g. the server cannot support the desired protocol version).
UltraVNC-SC is using an unregistered protocol number(which is their prerogative) but since it’s using a version beyond 003.008 it needs to be able to handle the prior documented protocols.
That all being said, we have altered Vine Viewer to attempt to detect the buggy SC connections and responds with an older protocol which allows the connection to complete.
This is available in Vine3.1-Beta3 available here:
http://www.testplant.com/downloads/Vine3.1Beta3.dmg