Thank for your replies

I find it highly unlikely that Terraria encrypts its files. According to 30 seconds of Googling, it just appears to be a custom format, most likely to make loading and streaming easier. That's not encryption and any user with a bit of technical background can get access to the data even if the specification is not public for easier modding.

If you want to prevent casual opening/modifications of your resources give them a non-standard extension (like .myimage instead of .png). That should prevent opening them by simple double-click in Windows. Of course even a user with a minimum of technical understanding can still just drag the file into Gimp.
Alternatively just store multiple files inside some archive format. I would advise against encryption. It will most likely cause you more problems getting it right than a skilled user to grab the decryption keys from your own executable.

I had also the same idea about the custom file extension but that is not enough

Encryption will also slow down your game loader - not much of an issue in a small project, but for a large game it's definitely noticeable. Released games are more concerned about loading something fast then encrypting it - it just so happens that the kinds of things you do to make something load fast, like archiving them, using a custom format, or preprocessing also make them difficult to read by standard programs.

Ok so i give up the encryption method

XNBs are simply files that are generated by the XNA Content Pipeline. They are not encrypted in any way, you can write yourself a viewer without much trouble using XNA or Monogame.

The easiest way that I can think of to "protect" your assets would be to store them in one or multiple archive (7z, zip, ...) and use a custom extension. Then, you change the the archive header so that the files cannot be opened without knowing how to reconstruct the original header.

Somehow I wonder what kind of assets you have that no one must be able to see them.

But zipping file will not make them slower to read ?

But zipping file will not make them slower to read ?

I had also the same idea about the custom file extension but that is not enough

Why not? What are you trying to prevent?

The reality check is that no matter how much effort you put into it, people will be able to undo whatever it is you come up with, so you might as well not waste much of your time with it. I'm sure it could be better spent improving other, more important aspects of your game.

But zipping file will not make them slower to read ?