Sign in to follow this  
necreia

Dynamic data loading in C++

Recommended Posts

Hello, this is a hard question to word correctly. I've been working with C++ for many years, and have always found a way 'around' this problem. I'm working on a custom model structure that supports animation through bones. However the problem I have is, loading / saving and working with a dynamic file size. In the past when I wanted to make something, I created a structure that was large enough to hold as big as it will ever be (like a map that could have up to 256x256 tiles on it, I would just save 256x256 and have the unused ones empty). I've always relied on fstream and loaded the data into a prebuilt structure. The question is, if I load this file into memory (for example, into a BYTE* pObject), how do I say, copy the first 64 bytes into 1 variable, then the next 32 bytes into a second variable. I know how to use the copymemory function, but is there a way to set the starting byte offset? Also, for this type of thing, is fstream the way to go? Thanks in advanced!

Share this post


Link to post
Share on other sites
Quote:
Original post by necreia
In the past when I wanted to make something, I created a structure that was large enough to hold as big as it will ever be.


You can load data into a std::vector or std::deque that will grown as the data comes in. No more worrying about data sizes. It is particularly simple if you use a stream iterator (like std::istreambuf_iterator - suitable for loading raw bytes) since you can just pass them to the container constructor (or assign member function).


#include <fstream>
#include <iterator>
#include <deque>

std::ifstream ifs("map.bin",std::ios::binary);
std::istreambuf_iterator<char> begin(ifs);
std::istreambuf_iterator<char> end;

std::deque<char> map_data(begin, end);



The data is then all in memory and you can do whatever you want with it.

Note that a vector holds memory in a contiguous block (i.e. if you do &vec[0], the result is a pointer you can use to access the data), while a deque does not. It is up to you to use one or the other depending on your needs. For very large data files, a deque is preferable. For data that will need to be passed to C APIs, a vector is preferable.

Quote:
The question is, if I load this file into memory (for example, into a BYTE* pObject), how do I say, copy the first 64 bytes into 1 variable, then the next 32 bytes into a second variable.


By hand.

Quote:
I know how to use the copymemory function, but is there a way to set the starting byte offset?


You can do pointer arithmetic. If you pass ptr+32 to the function, then it'll start 32 array elements (e.g. 32 bytes if you have a char*) from the start.

Quote:
Also, for this type of thing, is fstream the way to go?


Definitely. See my code snippet.

Share this post


Link to post
Share on other sites
Quote:
So you've been using c++ for years and you've never used new/delete? Anyways a great point of reference for anything c++ seems to be the c++ FAQ lite. have a look at this


I've used new and delete, but mostly to create new objects at an open nulled pointer...

I've always just used it like this:
struct SOMETHING
{
int SomethingElse;
}

SOMETHING *pToSomething;
pToSomething = new SOMETHING;


----

I had no idea that it could do anything more advanced than that, like loading to and from a storage medium (looking that up now via your link, thanks!).

Fruny: That's exactly what I need, thanks a ton!

Share this post


Link to post
Share on other sites
necreia,

I'm more familiar with how to do this in C than C++, so I'll describe the C method.

First, you need to open a file for reading data. In C, this is done with the followng line:

FILE *fopen( const char *filename, const char *mode );

Once you open the file with fopen you will be given a FILE pointer, you can then use this file pointer to get back the data you need from the file. To get a specific amount of data use the following line:

size_t fread( void *buffer, size_t size, size_t count, FILE *stream );

The first parameter is a buffer to store the data, the second parameter is the size of an item you'd like to read, and the third element is the number of items to read. So for example, if you want to read in 100x 1-byte characters, you might do this:

// read in 100, 1 byte 'blocks'
fread( pBuffer, 1, 100, pFile );

In reality, the size and count variables are often used interchangeably depending on what information you have. That is, some people multiply the size by the count and just pass in the total value for size, passing in 1 for count, etc...

So now that you know HOW to load in an arbitrary amount of data, how can this be used. Here's the classic example of a "gradebook" file. It stores simply a header identifying the number of students and then reads in the grades.

Format:
1 byte // Stores the number of grades in the range of 0-255
X bytes // 1 byte for each student's grade


FILE *pFile = fopen( "GradeBook.txt", "r" );
if( pFile )
{
// Since we know we always store the first byte as the number of
// grades, just read in the single byte
BYTE numGrades = 0;
fread( &numGrades, 1, 1, pFile );

// Now dynamically allocate memory to store the grades
BYTE* pGrades = new BYTE[ numGrades ];

// Now read in the grades
fread( pGrades, 1, numGrades, pFile );
}




The above system can be used for reading in an arbitrary number of elements of any data type. Rather than reading in numGrades bytes I could just as easy read in NumVerts vertices where vertices are an arbitrary sized structure. The only issue with reading in structs is occasional padding problems, but in general should work just fine.

So in Summary, define a "header" at the top of your file which is ALWAYS the same size. Or at least, provide a byte at the beginning for versioning so you know how big the header is. Then use the data loaded from the header to dynamically allocate memory to store the data that you need.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this