Note: This post may make more sense if you also check out the one I posted before Christmas on my blog (which I unfortunately forgot to also post here at the time: http://liza.io/the-brain-scope-is-growing-brainhub/
I've started this post maybe twenty times now, since before Christmas, and each time I keep putting it off, thinking "I'll just blog when I have this next bit done." But each "next bit" is followed by something else, and then something else, into infinity.
So I'll just write an update.
Since before the holidays I've been working on BrainHub and BrainRunner, which I've already written about. Basically - checking each brain every minute as part of the main SnailLife app was becoming unmanageable. All background processes will be moved to complementary apps away from the user-facing SnailLife application. The idea is to have a BrainHub controlling tasks sent to individual BrainRunners (which are hosted on other DigitalOcean or EC2 droplets).
So, here is the first iteration of the BrainHub admin site. The BrainHub has its own database of queued tasks and BrainRunners but also connects to the main SnailLife database and imports a package called SnailLifeCore to be able to get information about brains due for a check and allow admin SnailLife users to log in/control the hub.
It is sort of functional in that BrainHub runs 2 scheduled tasks:
- QueueIdleBrainChecks
- AssignBrainsToBrainRunners
Both of these run every minute. The first gets a list of brains that need to be processed and puts them into the queued_brains table in the brainhub db.
The second, AssignBrainsTobrainRunners, looks for any idle brain runners (brain runners with status code 0) and assigns the next brain in the queue to them. Then, the brain runner checks the brain and reports back to BrainHub with the result, which releases the runner to process the next brain in the queue.
Right now there are some issues - runners don't get consistently released, for example. Should be easy enough to fix, but for now I've added an emergency release button to the admin site (you can see it above).
But right right now I am working on logging. The brain runner creates a new log for each brain task it runs. These then need to be backed up to AWS S3 (as opposed to being stored on the Droplet itself), and then the admin site will display the logs by-task for each runner.
There is a mountain of work to do on this but it feels like I'm sort of over the main hump of setting up the core package and the BrainHub and BrainRunner apps to sit alongside the main SnailLife app.
