On ARM processor the alignment of data is important. It can read/write much faster at aligned positions. So 2-byte long value should be on an even address, and a 4-byte long value should be on an address dividable by 4. There are codes for unaligned read/write but they are different, longer and slower. The compiler must find out which code has to be used, and generally it does a very good job finding it out, but a few times fails.
These failed cases always contain a reinterpret_cast. In your case this is the C-style (int*) casting. That kind of casting is typically used in loading/saving data, or receiving/sending data through network. In other places it is typically just bad design, and should be avoided.
So I recommend you to concentrate all your code that uses reinterpret_cast into one or few classes, and handle the problem there.
And you can create a function for this kind of reading/writing similar to your doLittleBigEndianConversion. Like this:
float value = doLittleBigEndianConversion(doUnalignedReading(reinterpret_cast<int*>(ucharPtr + offset));
In that function just use the memcpy trick or read the data byte by byte and assemble it with |.