Divide the data however it makes sense for your game.
Maybe having one big file is good for your design. Maybe having lots of little files is good for your designs. Your design, your engine, your decisions.
My preference is to have a single file for each data table while you are developing it. You can have tools that bundle the files up into smaller pieces for the final distribution.
The data we typically just keep in Excel spreadsheets. When the designers change the data they use a macro to extract the data and generate an XML document. Some of the tables were four or five columns wide with only one or two rows initially with sample data, eventually growing 20 or 30 rows. Some of them were several hundred columns wide and many thousand rows. Both the xls and xml files live in version control. Some fancy parts of the game engine (kudos to whoever writes the file monitoring in all the engines I've worked on) monitors the files for changes. When the change is detected, it broadcasts an event to whoever registered the watch on the file; the system that registered it automatically reloads the tables when a change is detected, then sends out an event to the system that the data has been updated. Upon getting the notification, any interested listeners can refresh themselves, so that means the GUI can refresh, the battle rules system can refresh, and anything else that cares can refresh. When the game is packaged for distribution files get moved around and pulled into a compressed file system, and the xml spreadsheets are cleaned up into a more simple binary representation.
Sorry for the distraction on hot reloads. They are important because they can save hundreds of hours and lots of frustration at the end of development. A 10-second refresh is so much nicer than a 5-minute restart.
So ultimately I recommend one end of the pipe has data tables in xls. On the other end of the pipe you have data tables as arrays in code. Magic happens in the middle. Keep each table specific to the task at hand, whatever that task happens to be. The magic of the toolchain in between is entirely up to you.
Edited by frob, 15 March 2014 - 01:24 AM.