I have created a custom file format for my engine which uses both standard text and binary data. Eventually I will implement creation of these files in a level or world editor, but for the time being I am editing them by hand. My question is how do I conveniently enter delimiting characters into the file, i.e. null and /n, etc. I have been using a hex editor, but it is not convenient for entering non-0 delimiters, and can introduce bugs if data is input at the wrong location (because I cannot intuitively tell where the delimiter should go, I have to count and use the side-bar text representation of the hex). Is there a better way which would allow me to type in '0' or something so that a normal text editor would know that this is a null and NOT a 0 char?
That sounds painful. Any actual reason you can't keep your data in a more manageable form like all text, or in multiple files so that every file is either binary or text? Trying to squash everything together before you are even using an editor seems premature. It's easy to compress your data together later.
Don't edit binary files by hand, seriously, that's just asking for a whole load of trouble.
If you're going to be using custom binary files a tool for creating and editing those files should be a priority, even if it's only temporary.
When I need to store data in a binary format I first of all work out a design for the format itself, then I write the required tools based on that design, and it's not until that's finished I start generating and working with the files themselves.
This way of working saves you from a whole lot of painful situations afterwards.
At very least I would make a tool that convert formatted text files into the binary format, at least until you can write a proper editor. I don't know the specifics of your format but it shouldn't be too hard to write a tool that will take a say XML or JSON text file and do the conversion. There are parsers available like irrXML