Members - Reputation: 242
Posted 14 August 2012 - 02:35 PM
Once again i am struggling with a question i hope to find some answers for.
In my project, i work with a lot of classes, mainly containing other classes, which contain mostly strings and integers. The size of my data is approximately as follows:
in each event, approximately 8 markets
in each market, approximately 100 measures
in each measure, approximately 20 variables (integers and strings).
Since both events and markets also contain a few other variables, the approximate number of variables to store for each class is about 20.000. I intend to store up to a maximum of 1.000 events, therefore storing at maximum about 20 million values.
I wonder what would be the most efficient way to be able to store this data in one session and then load them back to use them in another session. The method i thought of myself was to create a XML file for each event and store the data in that file. Unfortunately i dont have any experience with data storing or using XML whatsoever, so i just wondered if this would be a reasonably effective way to store data. Any insights in this problem would really help me.
Since i am storing the data locally and mainly for myself, data access or protection is absolutely not an issue. Storing and reading speed is also not a really large issue, as long as it takes under a minute. The most important thing to consider is that i am a really unexperienced coder who is looking for a relatively simple solution. Also it should be a flexible method which allows me to add (for example) some measures to a market or remove some variables to a measure.
Thank you in advance for your ideas and opinions.
By the way, i work in C# using Visual Studio.
Members - Reputation: 1104
Posted 14 August 2012 - 06:31 PM
Without more context, it's hard to give good advice, but generally: If you have a lot of data to store, and you want to store it "efficiently", you probably want to use a properly structured binary format, rather than a "readable" format like XML, which requires more memory to store, and more processing power to parse.
Members - Reputation: 242
Posted 15 August 2012 - 11:51 AM
I am not familiar with 'a structured binary format' but i will definitely google into that. If anybody has any other ideas or suggestions, i am happy to hear them.
Members - Reputation: 628
Posted 15 August 2012 - 12:50 PM
Since integers are 4 bytes long, then your file will also be 4 bytes long regardless of how big the integer is.
Now if i stored it as text the number 4,000,000 will take up more space then the number 1.
Just google BinaryWriter there should be some good examples to get you going.