Jump to content
  • Advertisement
Sign in to follow this  

RSA Public Key problem c++ client to java server

This topic is 4550 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I have a c++ client that generates a keypair and then sends the public key to the server(java). Server receives key and converts it to "java format" (reverses endian, etc), all this works fine. Then server uses its converted public key to encrypt a message that it sends back to the client. Problem is that when the client decrypts the message, it gets a NTE_BAD_DATA (0x80090005) error and the decryption fails. Encryption/Decryption works fine locally on the client. My guess is that this has something to do with padding/algorithm/ecb/cipher/CRYPT_OAEP... I been working on this for a couple of days now so please, any guru in heaven who can help with this? /********** C++ SIDE ************/ /* Initiate Provider and Key */ #define ENCRYPT_ALGORITHM // The high order WORD 0x0200 (decimal 512) determines the key length in bits. #define KEYLENGTH 0x02000000 // Attempt to acquire a context and a key container. // The context will use Microsoft Enhanced Cryptographic // Provider for the RSA_FULL provider type. CryptAcquireContext(&m_hCryptProv, wszContainerName, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET ); // Call the CryptGenKey method to get a handle to a new exportable key-pair. CryptGenKey(m_hCryptProv, CALG_RSA_KEYX , KEYLENGTH | CRYPT_EXPORTABLE, &m_hCryptKey); /* C++ side configuration now looks like this when prompted with CryptGetKeyParam(...) function */ //ALGORITHM: CALG_RSA_KEYX //BLOCKSIZE: 512 //KEYLEN: 512 //PADDING: PKCS5_PADDING //CIPHERMODE: none /* To export the public key as MSPublicKeyBlob, following is done: */ // This call determines the length of the key blob. CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, NULL, &dwBlobLen); // Do the actual exporting into the key BLOB. CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, pbKeyBlob, &dwBlobLen); /* To test encrypt locally, the function is called as follows */ CryptDecrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen); /* To decrypt with local or remote data, the Encrypt function is called as follows */ CryptEncrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen, dwBufLen); /********** JAVA SIDE ************/ /* Convert MSPublicKeyBlob to Java PublicKey */ private static final byte PUBLICKEYBLOB = 0x06; private static final byte CUR_BLOB_VERSION = 0x02; private static final short RESERVED = 0x0000; private static final int CALG_RSA_KEYX = 0x0000a400; private static final int CALG_RSA_SIGN = 0x00002400; private static final String MAGIC = "RSA1" ; // 0x31415352 public static PublicKey msPkeyToPkey(byte[] mspublickeyblob) { DataInputStream dis = null; int jint = 0 ; // int to build Java int from little-endian ordered byte data int bitlen = 0; int pubexp = 0; //------ Read the "BLOBHEADER" fields ------------- ByteArrayInputStream bis = new ByteArrayInputStream(mspublickeyblob); dis = new DataInputStream(bis); if( dis.readByte() != PUBLICKEYBLOB || dis.readByte() != CUR_BLOB_VERSION || dis.readShort() != RESERVED) { System.out.println("Fail # 1"); return null; } jint = 0; for (int i=0; i<4; i++) jint += dis.readUnsignedByte() *(int)Math.pow(256,i); if(jint != CALG_RSA_KEYX && jint != CALG_RSA_SIGN) { System.out.println("Fail # 2"); return null; } //------ Read the RSAPUBKEY struct members --------- StringBuffer magic = new StringBuffer(4); for (int i=1; i<=4; i++) magic.append((char)dis.readByte()); if(!magic.toString().equals(MAGIC)) { System.out.println("Fail # 3"); return null; } for (int i=0; i<4; i++) bitlen += dis.readUnsignedByte() *(int)Math.pow(256,i); for (int i=0; i<4; i++) pubexp += dis.readUnsignedByte() *(int)Math.pow(256,i); int keysize = bitlen; //----- Finally, get the modulus data, and reverse bytes to get big-endian value -------- byte[] modulus = new byte[bitlen/8] ; //should be this many bytes left int modbytes = dis.read(modulus); if(modbytes != (bitlen/8)) { System.out.println("Fail # 4"); return null; } Utils.reverseEndian(modulus) ; //reverse bytes to put in big-endian order //----- Create the PublicKey from modulus and public exponent -------- RSAPublicKeySpec pubKeySpec = new RSAPublicKeySpec(new BigInteger(1, modulus), BigInteger.valueOf(pubexp)); KeyFactory keyFactory = KeyFactory.getInstance("RSA"); PublicKey pubKey = keyFactory.generatePublic(pubKeySpec); return pubKey; } /* Instantiate a Cipher used for encryption */ cipher = Cipher.getInstance("RSA"); // or possibly "RSA/ECB/PKCS1Padding"? cipher.init(Cipher.ENCRYPT_MODE, key); /* Encrypt data */ cipher.doFinal(bytedata, 0, bytedata.len); The problem is probably to correctly set MODE and PADDING on both sides but it seems everybody has different opinions to which configuration that actually works, and how it can be set up. Is it even possible using the CryptoAPI together with JCE or do I need BouncyCastle, Crypto++ etc.. to find an equal setting in both languages. Any suggestion using some other assymetic encryption system would do just fine if some two-side example code is accessible.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!