Jump to content
  • Advertisement
Sign in to follow this  
  • entries
    25
  • comments
    26
  • views
    26191

Tackling Gastropoda's memory usage - Match One

Sign in to follow this  
Liza Shulyayeva

1006 views

[font=arial]


I've been having to power cycle my Gastropoda Digital Ocean droplet every day lately because something was hogging up all the memory. It was a little annoying to diagnose because I know the problem had to do with the recurring events that are run using a cron job, of which there are quite a few. I know ideally PHP isn't something you would use for a long running server process, but these jobs weren't long running - they were just frequent.

[/font]
[font=arial]


The problem ended up being something very obvious. Every minute I run an event called CheckIdleSnailActions. It loops through all living snails in the world and sees what they're doing. Unfortunately at some point that event began to take more than a minute to run. So when I ran ps aux --sort -rss to sort all running processes by memory usage I saw two instances of CheckIdleSnailActions running at the same time, one taking something like 60% memory and the other ~30%.

[/font]
[font=arial]


I always knew running these checks every minute would see me encounter issues like these and know I need to find a more efficient way of doing this. In the meantime, the first step was to avoid running these events in parallel.

[/font]
[font=arial]


To do this I ended up using lock files. The cron job runs a Dispatcher command called RunRecurringEvents. This command grabs all recurring events that are scheduled to run at this time from the database and runs them. These events include things like:

[/font]

  • CheckIdleSnailActions
  • TickOrientationToNeutral
  • TickStressToNeutral
  • DepleteEnergyByTime
  • ...etc
    [font=arial]


    I created a new directory in app/storage called lockFiles.

    [/font]
    [font=arial]


    I didn't want to stop all of these things from running in parallel for now. The heaviest event by far is CheckIdleSnailActions. This is where we generate the brain, where the neurons do their work for each snail, where we comb through each snail's memories for recognition of an object, retrieve and record mood impact, etc.

    [/font]
    [font=arial]


    When each recurring event is triggered it runs only if there is no lock file matching the event's name in app/storage/lockFiles. So if there is a file called app/storage/lockFiles/CheckIdleSnailActions.txt, the event will not run. If there is no lock file by this name it creates one and starts running, and when it finishes running it just deletes the lock file and allows the next instance of the event to run when it is triggered.

    [/font]
    [font=arial]


    So far I haven't had to power cycle the droplet since this went in. It's not a full solution (I can't even imagine how long it might take for one CheckIdleSnailActions event to run if the world ever has thousands of living snails in it), so this is a work in progress.

    [/font]
Sign in to follow this  


2 Comments


Recommended Comments

This project makes me happy.

 

Thanks! Now that things are hopefully settling down with the cat I am going to get back to it!

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!