Sounds like there is a key mismatch somewhere, can you double check that your core has the right “.der” file copy of your server’s key, and your server has the key for your core? Can you post your full terminal command history for loading / setting keys?
I encountered exactly same issue, and I felt the exception is caused by the unencoded data passed into read_coreid function when handling incoming request, which cause the problem of
Error: error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len
For my case: the handshake can be done successfully when core tries to connect to local cloud. However the handshake issues will occur when subsequent request is coming.