Sign in to follow this  
deadlydog

Read entire file in vs read as you go

Recommended Posts

Hey, just a quick question. Right now I am making a 2D starfox type of game where enemies come from the top of the screen down, and I am using a txt file to hold the level information, which is basically just a text file with a list of enemies, what time they should appear at, and where they should be positioned on the screen. Now, as I am currently doing it, I am reading the entire file before the level starts and storing the information in an enemy queue, sorted by the time they should appear at, then each frame I just check if it's time to add an enemy or not. I was thinking that it would be more beneficial to just read the script file as I went, and just not using an enemy queue. This way I could store extra commands in the script file, and just use a switch statement to perform the appropriate actions based on what command was given. This way I could do other things, such as storing when items should appear, when something in the level should change, or whatever. Now, I'm just wondering if this would cause any problems or if there are any downsides to this method over my original, since the txt file would either need to be kept open for the entire duration of the level (it's read-only, but I'm not sure if keeping it open would cause problems or not), so that I wouldn't lose my position within the file, or I would have to save my position within the file and open and close the txt file every frame (which could be very costly). Which method would be best to use? I just want to make sure there's not some big "no no" that I would be doing. Thanks. Dan

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Id probably load the whole file one time, interpret the level/script, and save the interpreted commands into a big array or some other internal level structure in the program's stack. Not keep the file open for the duration; that sounds... problematic, suppose you get a read error, how do you handle it without interrupting the game?.

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
Id probably load the whole file one time, interpret the level/script, and save the interpreted commands into a big array or some other internal level structure in the program's stack. Not keep the file open for the duration; that sounds... problematic, suppose you get a read error, how do you handle it without interrupting the game?.


Yeah, this is pretty much what I'm doing now. Keeping the file open for the duration would definitely be the easiest to do though. Good point with the read error, but right now I do not have any read error handling anyways so whether the error occurred before the level started or during the level wouldn't make much of a difference. Either way I would likely just display the error in a msg window and exit the game. Anyone else have any comments or see any other problems with this method? Thanks.

Share this post


Link to post
Share on other sites
I would put the whole file in a string array and read from there. That doesnt cost too much memory, does it?

Share this post


Link to post
Share on other sites
Quote:
Original post by Gagyi
I would put the whole file in a string array and read from there. That doesnt cost too much memory, does it?


That would depend entirely on the size of the file.

Share this post


Link to post
Share on other sites
File IO is usually not very fast, it could slow your game down if reading as the game is running. However, you could make a stream of some sort so the read position is ahead of the game, that way the game is not waiting on the file io to read what's coming up.

Share this post


Link to post
Share on other sites
Quote:
Original post by tstrimp
Quote:
Original post by Gagyi
I would put the whole file in a string array and read from there. That doesnt cost too much memory, does it?


That would depend entirely on the size of the file.


Hmmm, this idea is interesting. And while it would take a little bit more memory, I can't see it taking too much more than I'm using right now, since currently I am storing all of the information in a big list of structures. Does anybody see any drawbacks to this method? or can you think of any other methods that might be better? Oh, and right now my level files are around 500 - 750 words long, but that could increase once I'm able to store other things in the file besides just which enemies should appear when, but it should stay under 1000 words (by words I mostly mean 1-6 digit numbers).

I assumed that this would be a fairly common problem, as lots of games use scripts to read in game info, so I thought there would be a common, well known solution/method to this problem. Thanks for the replies so far everyone!

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by deadlydog
Hmmm, this idea is interesting. And while it would take a little bit more memory, I can't see it taking too much more than I'm using right now, since currently I am storing all of the information in a big list of structures. Does anybody see any drawbacks to this method?


Your current approach of storing the script information in special structures is better than storying the literal file strings in a giant array.

For one, as you noticed, its less memory. For another, the literal strings need to be interpreted so they can be used in the game. While your structures method (hopefully) is designed to pre-process all that work and instead is adapted for direct use in gameplay (ie, the structs contain numeric values, function pters, etc, stuff that will directly be accessed in your map).


It sounds like what you were already doing is correct...

Share this post


Link to post
Share on other sites
First, abstract the problem away. A Raw_Level is a stream of lines. A Processed_Level is a stream of processed level lines.

Then you can implement a simple and easy Raw_Level and Processed_Level implementation -- whichever is easiest and still correct -- and worry about what kind of buffering you should do later.

Share this post


Link to post
Share on other sites
Might there be a consideration that you have to pre-read the info to load assets like images and sounds (which you would want to preload for the level)??

Storing the data in memory cant be too costly (large) in any case. If most of it is numbers it will usually be alot smaller than the file -- and even a meg of memmory isnt much these days (unless you are on a lowend/old palmtop).

Prereading to verify the level file is intact/valid is also a good idea.

Doing all that you might as well read in into memory at the start.

Share this post


Link to post
Share on other sites
If the file is large, you can load specific sections of the file
based on the current state of the program.

This way, you only load the sections that are needed, without using
to much memory.

The best thing about this method, is that you can reuse the same memory
block, whenever you need to load a new section from the file.

Of course, you will need to add the sections to the file itself.
(Mabey a string token?, for example)

Share this post


Link to post
Share on other sites
Quote:
Original post by deadlydog
Now, as I am currently doing it, I am reading the entire file before the level starts and storing the information in an enemy queue, sorted by the time they should appear at, then each frame I just check if it's time to add an enemy or not.

I was thinking that it would be more beneficial to just read the script file as I went, and just not using an enemy queue. This way I could store extra commands in the script file, and just use a switch statement to perform the appropriate actions based on what command was given. This way I could do other things, such as storing when items should appear, when something in the level should change, or whatever.

I agree with others that you should stick with an internal data structure (which is most likely created by parsing a file at level-load time), rather than parsing a file (or a large string) while running the game. However, if I may take a guess, it seems you might be stuck on how to do this while still maintaining the option to add items, game events, and such, and not just get stuck with enemies.

Here is a suggestion to remedy this problem, using OOP. This is C++, but if you're using something else, hopefully it won't be too hard to convert.

The basic idea I'm thinking of is to have a base class GameStateModifier. This class's primary purpose is to have a virtual function that modifies the game state. The function should accept some representation of the game state that it can modify, and should return a bool representing if it was successfully applied. It should also contain a comparison function to enable sorting by time (which I'm choosing to hide from most of the game). (Note that I'm making all these functions inline, to save space. You can split them into headers and source files if you wish.)
class GameStateModifier
{
public:
GameStateModifier(time_type time) : mTime(time) {}
virtual ~GameStateModifier() {}

virtual bool Apply(GameState& gameState) { return state.Time() >= mTime; }
friend bool operator < (const GameStateModifier& left, const GameStateModifier& right) { return left.mTime < right.mTime; }

protected:
time_type mTime;
};







Now when you load your level data, you'll parse through the data, creating derived classes based on the data and pushing these instances into a sorted container (which should automatically use the operator < () function to do the sorting). Suppose a derived class looks like this:
class EnemySpawner : public GameStateModifier
{
public:
EnemySpawner(time_type time, enemy_type_enum enemyType, vector2 position) :
GameStateModifier(time),
mEnemyType(enemyType),
mPosition(position)
{
}

virtual bool Apply(GameState& gameState)
{
if (GameStateModifier::Apply(gameState))
{
gameState.SpawnEnemy(mEnemyType, mPosition);
return true;
}
else
{
return false;
}
}

protected:
enemy_type_enum mEnemyType;
vector2 mPosition;
};







When parsing, if you found a section of data that represented an enemy spawn, you'd add it to the container of events like this:
//We have to use pointers to get the polymorphic behavior,
//and I'd rather avoid having to manage the memory, so I'm just using smart
//pointers instead.
typedef boost::shared_ptr<GameStateModifier> ModifierHandle;

//Also, we need a comparison functor to enable comparison on pointers, so
//that we can use the algorithm std::upper_bound() later on.
struct ModifierHandleComparer : std::binary_function(ModifierHandle, ModifierHandle, bool)
{
bool operator ()(const ModifierHandle& first, const ModifierHandle& second)
{
return *first < *second;
}
};

//This is our actual container. I chose std::deque because modifying both
//front and back is efficient, and we'll be doing a lot of both, but very
//little modifying of the middle.
//You could also for example use std::set, and specify the ModifierHandleComparer
//as the sort predicate, and avoid having to call std::upper_bound(), since
//the set will do the sorting for you.
std::deque<ModifierHandle> gameModifiers;

//Later on while parsing...
//We've parsed data and know that it is
// an enemy spawn event. The data is
// stored in these variables:
time_type modifierTime;
enemy_type_enum enemyType;
vector2 enemyPosition;

//Now we create the modifier and add it to the container:
ModifierHandle modifier(new EnemySpawner(modifierTime, enemyType, enemyPosition));
gameModifiers.insert(std::upper_bound(modifier), modifier)
//Note that if your file is ordered by time, then the insert will always
//occur at the back of the container (which is efficient). But if anything
//is out of order, the insert will still work, it might just be a little
//slow. I doubt you'd notice a performance problem from a few out of
//place items, though.

//We continue doing this for all the level data, potentially creating all sorts of different GameStateModifier instances.







Finally, while you're running through the game, you can handle these events like this:
//Game loop top
{
//Do other game loop stuff

//We're guaranteed that the modifiers are sorted, with soonest ones in
//front. Just apply the front one and pop it off, until we encounter
//one that doesn't want to be applied yet (since it check the time and
//and saw that it wasn't ready yet).
while (!gameModifiers.empty() && gameModifiers.front()->Apply(gameState))
{
gameModifiers.pop_front();
}

//Do other game loop stuff
} //End of Game Loop







So the game loop part of code doesn't care about what type of modifiers exist in the container. It just tells them to do their stuff, and they do it. The only part of code that needs to worry about the different types is the level data parser, which has to worry about that regardless of how you set this system up.

Oh, and as a disclaimer [smile], don't hate me if some of this stuff doesn't compile. I wrote it on the spot. I did try to take care to look up things though to make sure it is correct.

Share this post


Link to post
Share on other sites
Quote:
Original post by Agony
I agree with others that you should stick with an internal data structure (which is most likely created by parsing a file at level-load time), rather than parsing a file (or a large string) while running the game. However, if I may take a guess, it seems you might be stuck on how to do this while still maintaining the option to add items, game events, and such, and not just get stuck with enemies.

Yeah, you were right that I was trying to decide how best to have my script handle all types of game events, not just enemy spawning. I like your idea of having a Game Modifier class to handle the list of all game events, and then just having all of my actual classes (enemy class, item class, etc) simply inherit it. This way I have a simple container, and I can just do a switch when removing items from the container to see what type they are and take appropriate action.

I really appreciate everyones input on my problem. Thank you to everyone who replied!!

Share this post


Link to post
Share on other sites
Quote:
Original post by deadlydog
This way I have a simple container, and I can just do a switch when removing items from the container to see what type they are and take appropriate action.

Note that I used a virtual function instead of a switch to handle this functionality. It should be faster, as well as conform more fully with the OOP paradigm. And it [opinion]just looks more elegant[/opinion]. But the switch will work, and that is the primary goal, so good luck with whatever you choose.

Share this post


Link to post
Share on other sites
Quote:
Original post by GamerYZ
File IO is usually not very fast, it could slow your game down if reading as the game is running.



Bologna.


Streaming game data live from disk is actually very common in games - even streaming from "slow" disk sources like CD/DVD. Hard drive IO is not that big of a deal when you're talking about such tiny bits of data as deadlydog's level information.

Here's an experiment you should try sometime: write a small program that outputs the numbers 1 to 2,000,000 into a file, one number per line. Time how long it takes to run. Then write a second program that loads each line and sums up the numbers. Time that one.

You may be quite surprised.

Share this post


Link to post
Share on other sites
Quote:
Original post by ApochPiQ
Quote:
Original post by GamerYZ
File IO is usually not very fast, it could slow your game down if reading as the game is running.



Bologna.


Streaming game data live from disk is actually very common in games - even streaming from "slow" disk sources like CD/DVD. Hard drive IO is not that big of a deal when you're talking about such tiny bits of data as deadlydog's level information.

Here's an experiment you should try sometime: write a small program that outputs the numbers 1 to 2,000,000 into a file, one number per line. Time how long it takes to run. Then write a second program that loads each line and sums up the numbers. Time that one.

You may be quite surprised.

I liked your idea to do a practical test and wrote a quick program. On my PC, writing 1000 integers to a file takes 8 milliseconds, and reading in a thousand integers takes 5 milliseconds. If I open the file before writing/reading each integer, and close it after writing/reading each integer, it takes 183 milliseconds to write 1000 integers, and 70 milliseconds to read them. So if I just open the file and leave it open for the duration of the level I'm not really going to see any performance hit, since I'm usually reading in at most 20 integers at any one given time. Also, since the events are ordered by time, I don't have to do a file read every frame, just whenever the "next event start time" has passed. So really this method would cost maybe an extra millisecond every 5 - 30 seconds (however long between events), which seems acceptable to me. And it also allows me to take the quick and easy way out by not having to build a container or list for everything. I can just do a switch on the events as they are read in and take appropriate action.

I could see this approach maybe not being suitable for todays latests and greatest games which have huge level files and tons of events going on all of the time which would need to be loaded, but I think for my simple little game the simplicity is worth the tiny performance hit that shouldn't even be noticable. Comments?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this