Sign in to follow this  
@

Unity is std::ostream thread-safe?

Recommended Posts

there was a discussion here: http://www.gamedev.net/community/forums/topic.asp?topic_id=275240 but I'd like to elaborate this topic. My logger uses TLS buffer for log-message formatting and std::ofstream for output. Buffering is disabled by std::ofstream::rdbuf()->pubsetbuf(0, 0). The question is: can I use std::ofstream::write() w/o locking? Locking is undesirable because logging is pretty intensive and this can reduce performance.

Share this post


Link to post
Share on other sites
I'm pretty sure that nothing in the standard C++ library is thread-safe by design. C++ as such has no notion of threads, hence the standard can't make any guarantees regarding thread-safety. I'm afraid you'll have to lock if you want to share an ostream object between threads.

Share this post


Link to post
Share on other sites
Quote:

The question is: can I use std::ofstream::write() w/o locking?

Maybe. The results may vary if you need to write more than one atomic piece of data, or if underlying implementation cannot perform fully atomic write.
Quote:
Locking is undesirable because logging is pretty intensive and this can reduce performance.

The file write will lock on its own. File is a sequential device and cannot be accessed concurrently, unless implemented via multiplexing using one of async APIs.


You're solving the right problem but in the wrong way, sort of premature optimization.

1) Do you need each log to be reliably written. In that case you need to use fsync() or similar to cause OS to flush pending writes before proceeding, or fail. In this case, there is no way around blocking. Application cannot proceed if log write fails, which means concurrent logs will need to wait for each other, or each log to their own per-thread file.
2) Are you sure file writes are the problem? Do you generate more than 10MB per second? Can you log to a SSD? Can you log over gigabit network with hardware accelerate card? If not, then it is likely that logging will not be a bottleneck.

To avoid stalls or prevent increased jitter when logging, the preferred way would be to use either async file access (cannot be done via fstream, depends on OS), or an in-memory database running in a separate process.

A user-mode, cross-platform logging can be implemented like this (See services, logging). This uses serial out-of-thread file access. If application crashes, logs will be lost. There is no way around this without blocking.

For bulk logging (10MB+ per second), scatter/gather style logging is possible. Each thread has its own file, each log is timestamped. They can write without impeding each other (if each commit is atomic and completes), but logs must be merged before analyzing. SSD, RAID or network might be preferable for this type anyway.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
The file write will lock on its own. File is a sequential device and cannot be accessed concurrently [...]


Maybe the actual, physical file will but what about the ostream-object representing the file? What if internal state is modified by writing to a file via an ostream object?

Share this post


Link to post
Share on other sites
Quote:
Original post by cache_hit
fyi, Boost.Log is pending review and I think should be accepted pretty soon.


Boost log was pending review some 3 years ago when I first saw it. It has been in development since before then.

I no longer care about stuff coming "real soon". Give me something that has been in use for a year.

Seriously, logging is not some newfangled functionality, but the very core of every application for past 40 years.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
A user-mode, cross-platform logging can be implemented like this (See services, logging). This uses serial out-of-thread file access. If application crashes, logs will be lost. There is no way around this without blocking.
It is possibel to implement a logging server in a separate process, and have that server buffer the logs as necessary. Communicate via local IPC or sockets.

This approach should give one all the advantages of out-of-thread logging, plus logs survive program crashes, at the cost of some implementation complexity.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by cache_hit
fyi, Boost.Log is pending review and I think should be accepted pretty soon.


Boost log was pending review some 3 years ago when I first saw it. It has been in development since before then.

I no longer care about stuff coming "real soon". Give me something that has been in use for a year.

Seriously, logging is not some newfangled functionality, but the very core of every application for past 40 years.


This is a completely new logging library. The last one was abandoned by the author, this one is by a new author and his own implementation, written from the ground up, and the author is more committed to supporting it than the other author.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
It is possibel to implement a logging server in a separate process, and have that server buffer the logs as necessary. Communicate via local IPC or sockets.

This approach should give one all the advantages of out-of-thread logging, plus logs survive program crashes, at the cost of some implementation complexity.


If you do not need a guarantee that each log is written, then there is no problem.

Consider a radiation therapy device which needs to be audited that not only is each log written, but also replicated and stored in a way that cannot be tampered with, plus each log must be kept for a minimum of 10 years in a physically separate location with paper trail, while there might be dozens of logs generated each second in a real-time system.

In more common case however, the following two scenarios are important:
- How bad is it if random swaths of logs get lost or malformed?
- How bad is it if last n logs are not written

Second option is probably much worse, since it will make diagnosing crashes impossible - the only time you really need each log and as many as possible is in the last 5 functions calls that led to crash.

Sockets, out-of-thread writes and even async writes all lead to second case. Even transactional file systems can be a problem, if process fails in the middle of transaction which might be writing a considerable chunk of data.

Simply put, for reliable logging, flush each log to disk after each write, or use a database that implements this. Locking is highly problematic here, since the important logs might be pending write blocked by a lock, while an INFO is being written.


Ugh... performance is the very last issue one needs to deal with... Just keep one log file per thread, and merge them externally. Very few systems actually put any effort in logging, they do well enough just with fwrite(). Performance these days is usually solve by adding more cheap machines.

Share this post


Link to post
Share on other sites
but I'd like to elaborate this topic. My logger uses TLS buffer for log-message formatting and std::ofstream for output. Buffering is disabled by std::ofstream::rdbuf()->pubsetbuf(0, 0). The question is: can I use std::ofstream::write() w/o locking? Locking is undesirable because logging is pretty intensive and this can reduce performance.

Apparently Microsoft's implementation of iostream is thread safe so you can write to it from two threads at once if you want. The other STL containers are okay for simultaneous reads but not writes. You'd probably want some sort of safeguard if you were going to rely on that behavior. Maybe assert(IS_VISUAL_STUDIO) or something.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Forum Statistics

    • Total Topics
      627762
    • Total Posts
      2978972
  • Similar Content

    • By KARTHI
      Free software
       
      1. Lumberyard (Game engine) open-source and free tool
      Amazon Lumberyard is a free cross-platform triple-A game engine developed by Amazon and based on the architecture of Cry Engine, which was licensed from Crytek in 2015.
       
      2. Sculptris (sculpture tool) open-source and free tool
      Sculptris is a virtual sculpting software program, with a primary focus on the concept of modeling clay. It entered active development in early December 2009, and the most recent release was in 2011.
       
      3. Make human (game model creator) open-source and free tool
      Make human is an open-source 3D computer graphics software middleware designed for the prototyping of photorealistic humanoids. It is developed by a community of programmers, artists, and academics interested in 3D modeling of characters.
       
      4. Ipi soft (motion capture software) not free tool
      iPi Motion Capture is a scalable markerless motion capture software tool that supports 1 or 2 Kinect cameras or 3 to 6 Sony PlayStation Eye cameras to track a human action and convert it into a motion capture file
       
       5. Blender (Complete tool) for modeling, texturing and so on (open-source and free tool)
       Blender is a professional, free and open-source 3D computer graphics software toolset used for creating animated films, visual effects, art, 3D printed models, interactive 3D applications and video games.
                 
      6. Audacity (music editor) open-source and free tool
      Audacity is a free open source digital audio editor and recording computer software application, available for Windows, OS X, Linux and other operating systems.
                 
      7. Awesome bump (bump map editor) open-source and free tool (optional)
      Awesome Bump is a free program written using Qt library designed to generate normal, height, specular or ambient occlusion textures from a single image.
       
      8. Faceware (facial animation) not free tool
      Faceware Technologies is an American company that designs facial animation and motion capture technology. The company was established under Image Metrics and became its own company at the beginning of 2012.
       
      9. GIMP (image editing) open-source and free tool
      GIMP is a free and open-source raster graphics editor used for image retouching and editing, free-form drawing, converting between different image formats, and more specialized tasks. Through this you can also create bump maps
       
      10. Meshlab (mesh repair) open-source and free tool (Optional)
      MeshLab is an advanced 3D mesh processing software system that is oriented to the management and processing of unstructured large meshes and provides a set of tools for editing, cleaning, healing.
       
      11. LibreOffice (create documents) open-source and free tool
      LibreOffice is a free and open source office suite, a project of The Document Foundation. It was forked from OpenOffice.org in 2010, which was an open-sourced version of the earlier StarOffice.
                 
      12. Atom (coding software) open-source and free tool
      Atom is a free and open-source text and source code editor for macOS, Linux, and Microsoft Windows with support for plug-ins written in Node.js, and embedded Git Control, developed by GitHub.
                 
       
      Useful websites
      free image
                 
      Reference images will be found on Pinterest
       
      Free Sounds
       
      1.      Freesound.org
      2.      99Sounds.org
      3.      NoiseForFun.com
      4.      Incompetech.com
      5.      OpenGameArt.org
      6.      RaisedBeaches.com
      7.      Musopen.org
      8.      PlayonLoop.com
      9.      Bensound.com
      10.   SoundJay.com
      11.   Dig.ccmixter.org
      12.   Soundgator.com
      13.   Pacdv.com
      14.   Freesfx.co.uk
      15.   Soundtrack.imphenzia.com
      16.   Bxfr.net
       
      Download the free music tracks from these websites
       
      1. http://incompetech.com/music/
      2. http://dig.ccmixter.org/
      3. http://www.joshwoodward.com
      4. http://www.youtube.com/audiolibrary
       
      I hope this information will help full to you. I am got so stress to collect this data so don't waste it 
      🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤗🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔
       
       
    • By rXpSwiss
      Hello,
      I am looking for a team to stop developing project alone and start doing bigger things.  I would like a team that is organized and serious even of it is just a hobby. I don't want a project to be dropped every week and the team to disband after 2 months. Even though it is a hobby the quality and complexity of the game should be on par with a game that would be released on a major platform.

      I have experience with UE4, Unity and making my own engine with OpenGL. I prefer UE4 for 3D and Unity for 2D but I am willing to work with both.
      Every one of the games I worked on (personal project) involve multiplayer and netcode, I used what UE4 provided but for unity I used Forge Remastered which gave me a bigger control on how to optimize everything.
      I play several instruments, know about game design and am able to do small modeling task but I am before everything a programmer.
      I play mostly competitive fighting games these days (SFV, MvCi, GG, T7) but before I started getting serious into these types of games it was mostly FPS (CS1.6, UT2k4, Quake 3) and Dota-like. As you can see I love competitive multiplayer games but I play solo games too. I have a preference for darker theme games(F.E.A.R, HL, Shin Megami Tensei/Persona, Last of Us, SH2, RE7) even though I play everything offering me a challenge or some peace (Don't Starve, Stardew Valley, Darkest Dungeon, Zelda etc).
      I do have several ideas for games with a very clear direction gameplay, marketing and art but I am not looking for a team to work on my ideas, I just have them if needed.
      Feel free to ask questions here or PM me if you prefer.
      Best regards,
      rXp
    • By KARTHI
      Please reply me about this
    • By KARTHI
      Currently I using makehuman to make a 3d models but I am not satisfied with that model. If any use this please reply about the quality and details of models. 
      🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔
    • By KARTHI
      I planned to make a space game so I need a knowledge about 3d planet making so anyone help me to make it possible
      🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔🤔
  • Popular Now