I have a c++ client that generates a keypair and then sends the public key to the server(java). Server receives key and converts it to "java format" (reverses endian, etc), all this works fine. Then server uses its converted public key to encrypt a message that it sends back to the client. Problem is that when the client decrypts the message, it gets a NTE_BAD_DATA (0x80090005) error and the decryption fails. Encryption/Decryption works fine locally on the client. My guess is that this has something to do with padding/algorithm/ecb/cipher/CRYPT_OAEP... I been working on this for a couple of days now so please, any guru in heaven who can help with this? /********** C++ SIDE ************/ /* Initiate Provider and Key */ #define ENCRYPT_ALGORITHM // The high order WORD 0x0200 (decimal 512) determines the key length in bits. #define KEYLENGTH 0x02000000 // Attempt to acquire a context and a key container. // The context will use Microsoft Enhanced Cryptographic // Provider for the RSA_FULL provider type. CryptAcquireContext(&m_hCryptProv, wszContainerName, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET ); // Call the CryptGenKey method to get a handle to a new exportable key-pair. CryptGenKey(m_hCryptProv, CALG_RSA_KEYX , KEYLENGTH | CRYPT_EXPORTABLE, &m_hCryptKey); /* C++ side configuration now looks like this when prompted with CryptGetKeyParam(...) function */ //ALGORITHM: CALG_RSA_KEYX //BLOCKSIZE: 512 //KEYLEN: 512 //PADDING: PKCS5_PADDING //CIPHERMODE: none /* To export the public key as MSPublicKeyBlob, following is done: */ // This call determines the length of the key blob. CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, NULL, &dwBlobLen); // Do the actual exporting into the key BLOB. CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, pbKeyBlob, &dwBlobLen); /* To test encrypt locally, the function is called as follows */ CryptDecrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen); /* To decrypt with local or remote data, the Encrypt function is called as follows */ CryptEncrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen, dwBufLen); /********** JAVA SIDE ************/ /* Convert MSPublicKeyBlob to Java PublicKey */ private static final byte PUBLICKEYBLOB = 0x06; private static final byte CUR_BLOB_VERSION = 0x02; private static final short RESERVED = 0x0000; private static final int CALG_RSA_KEYX = 0x0000a400; private static final int CALG_RSA_SIGN = 0x00002400; private static final String MAGIC = "RSA1" ; // 0x31415352 public static PublicKey msPkeyToPkey(byte[] mspublickeyblob) { DataInputStream dis = null; int jint = 0 ; // int to build Java int from little-endian ordered byte data int bitlen = 0; int pubexp = 0; //------ Read the "BLOBHEADER" fields ------------- ByteArrayInputStream bis = new ByteArrayInputStream(mspublickeyblob); dis = new DataInputStream(bis); if( dis.readByte() != PUBLICKEYBLOB || dis.readByte() != CUR_BLOB_VERSION || dis.readShort() != RESERVED) { System.out.println("Fail # 1"); return null; } jint = 0; for (int i=0; i<4; i++) jint += dis.readUnsignedByte() *(int)Math.pow(256,i); if(jint != CALG_RSA_KEYX && jint != CALG_RSA_SIGN) { System.out.println("Fail # 2"); return null; } //------ Read the RSAPUBKEY struct members --------- StringBuffer magic = new StringBuffer(4); for (int i=1; i<=4; i++) magic.append((char)dis.readByte()); if(!magic.toString().equals(MAGIC)) { System.out.println("Fail # 3"); return null; } for (int i=0; i<4; i++) bitlen += dis.readUnsignedByte() *(int)Math.pow(256,i); for (int i=0; i<4; i++) pubexp += dis.readUnsignedByte() *(int)Math.pow(256,i); int keysize = bitlen; //----- Finally, get the modulus data, and reverse bytes to get big-endian value -------- byte[] modulus = new byte[bitlen/8] ; //should be this many bytes left int modbytes = dis.read(modulus); if(modbytes != (bitlen/8)) { System.out.println("Fail # 4"); return null; } Utils.reverseEndian(modulus) ; //reverse bytes to put in big-endian order //----- Create the PublicKey from modulus and public exponent -------- RSAPublicKeySpec pubKeySpec = new RSAPublicKeySpec(new BigInteger(1, modulus), BigInteger.valueOf(pubexp)); KeyFactory keyFactory = KeyFactory.getInstance("RSA"); PublicKey pubKey = keyFactory.generatePublic(pubKeySpec); return pubKey; } /* Instantiate a Cipher used for encryption */ cipher = Cipher.getInstance("RSA"); // or possibly "RSA/ECB/PKCS1Padding"? cipher.init(Cipher.ENCRYPT_MODE, key); /* Encrypt data */ cipher.doFinal(bytedata, 0, bytedata.len); The problem is probably to correctly set MODE and PADDING on both sides but it seems everybody has different opinions to which configuration that actually works, and how it can be set up. Is it even possible using the CryptoAPI together with JCE or do I need BouncyCastle, Crypto++ etc.. to find an equal setting in both languages. Any suggestion using some other assymetic encryption system would do just fine if some two-side example code is accessible.