Graphics without drivers?
Run your program through a DOS emulator, then you can use all those old-school techniques
Or even just write a bootloader and a basic C kernel and boot it in a VM, that works too. There's even code online that implements a simple VGA driver for an example (1-file) "operating system". I don't think you can directly access the VGA framebuffer under modern operating systems anyway, at least not easily. I would think even enabling the required VGA mode requires your process to be in kernel land to begin with, especially since your address space would be virtualized anyway in userland so your 0xB8000 address wouldn't be of any use.
Well, on Linux you could always try mapping the graphics card's memory into virtual memory through /proc/, but that will probably not produce the results you want 
On a modern PC when your graphics card is in a modern high res full colour mode, the frame buffer might or probably is not at 0Xb8000 in the first place.
The frame buffer doesn't even exist as such a resource any more. They're not directly accessible, they're no longer stored as linear memory, and all of the hardware resources are locked down so that regular apps can't access them without using a driver (for stability and security reasons).
Short of running in an emulator or working on your own driver (which are extremely), there is no way to access the raw hardware.
There are projects like Linux's framebuffer drivers that can give you an experience similar to DOS, but they're still going through drivers.
If you want to do old-school drawing, just draw into a local memory buffer and then blit that to the screen using OpenGL or DirectX. Plenty of folks in the demo scene and hobby game developers do just that.
Trying to answer the OP question directly.
So say we wanted to, from pure code, write text of just colour in certain pixels, how would we do this without going through the drivers?
You would either need to use an operating system that grants direct hardware access, or take control of direct hardware access by writing your own system.
Most modern operating systems restrict access to the physical hardware as described above. Without that kind of abstraction programs could not share resources, nor could the system recover cleanly after a crash or other misbehavior. That is a good thing. Back in the bad old days, and still today on smaller hardware and embedded systems, if a program or component misbehaves often the only good solution was to reboot. With the OS virtualizing access to the hardware a crashed program means everything else can continue, and on the other end of virtualization, a crashed video driver means the screen goes black for a moment then recovers. Any misbehavior is recoverable.
Many embedded systems still have direct hardware access. Old operating systems like DOS allow direct hardware access. You just won't be getting raw hardware access under Windows, Linux, MacOS, or similar modern systems (unless you are writing your own device drivers, then you can get limited access).
I know it isn't accessible from regular user space apps but I assume something like it is mapped in kernel space since he mentioned how he is interested in how an OS has access to it.
Again, there's nothing to map. That's not how modern hardware works. Modern hardware works via command queues and buffer requests, not framebuffer access. Even integrated or "low end" hardware works that way now; the framebuffer may be accessible in memory somewhere but it's often no longer a linear array of pixels like the old days.
Hardware will often implement VGA mode for things like early boot-up and diagnostic, but you often won't get access to many of the "basic" features like multiple monitors, higher color modes, higher resolutions and refresh rates, etc. So you could use the old VGA interface (with an appropriate OS) if you really wanted to, but that's not how the modern OS does things in any way shape or form.
Forget the whole discussion about whether one is in user space or kernel space, as far as I know the entirely of the VGA support is completely gone in the newest hardware because UEFI can cope with the GPU framebuffers directly (and if you ever try to render video through UEFI, you'll only get a framebuffer, text mode doesn't exist anymore).
Forget the whole discussion about whether one is in user space or kernel space, as far as I know the entirely of the VGA support is completely gone in the newest hardware because UEFI can cope with the GPU framebuffers directly (and if you ever try to render video through UEFI, you'll only get a framebuffer, text mode doesn't exist anymore).
Citation? I can't find anything on that, and it would certainly be newsworthy. In Linux land at least I'm sure there would be a lot of discussion if text mode ceased existing.
While UEFI might happen to limit functionality, I have not read or heard anything about nvidia or ati/amd cards dropping support for old modes.
A few years back, 2009 or so, I was curious about that when fixing up a machine. A trip to MSDN's download section got me old dos, and finding an old floppy disk got me some games. Some broke, but I was able to get several EGA-mode games running. One was a tanks game. While things like moving the turrets and adjusting power levels was basically impossibly fast, the EGA graphics worked just fine.
I also know the old text modes continue to work just fine on modern cards. They're not going to do away with the classic 80x25, 80x43, 80x50, and similar text modes.
On a handy linux box with a modern box xrandr says it can use CGI resolutions, those come back to 1981. A bit of Google searching shows the old CGA, EGA, and VGA memory ranges are still reserved for graphics.
I would be surprised if they were removed. My guess is that you could install a new GTX 980 and still use the old bios or vesa commands to reach at least a 640x480x8-bit graphics display and probably older, or if you couldn't change into a text mode and write to the 0xB8000 for your old-school text buffer also introduced back with the CGA over thirty years ago. I would also be surprised if the POST screens were doing something other than the classic CGA text modes for those initial text screens.
