• Advertisement
Sign in to follow this  

Unity is std::ostream thread-safe?

This topic is 2894 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

there was a discussion here: http://www.gamedev.net/community/forums/topic.asp?topic_id=275240 but I'd like to elaborate this topic. My logger uses TLS buffer for log-message formatting and std::ofstream for output. Buffering is disabled by std::ofstream::rdbuf()->pubsetbuf(0, 0). The question is: can I use std::ofstream::write() w/o locking? Locking is undesirable because logging is pretty intensive and this can reduce performance.

Share this post


Link to post
Share on other sites
Advertisement
I'm pretty sure that nothing in the standard C++ library is thread-safe by design. C++ as such has no notion of threads, hence the standard can't make any guarantees regarding thread-safety. I'm afraid you'll have to lock if you want to share an ostream object between threads.

Share this post


Link to post
Share on other sites
Quote:

The question is: can I use std::ofstream::write() w/o locking?

Maybe. The results may vary if you need to write more than one atomic piece of data, or if underlying implementation cannot perform fully atomic write.
Quote:
Locking is undesirable because logging is pretty intensive and this can reduce performance.

The file write will lock on its own. File is a sequential device and cannot be accessed concurrently, unless implemented via multiplexing using one of async APIs.


You're solving the right problem but in the wrong way, sort of premature optimization.

1) Do you need each log to be reliably written. In that case you need to use fsync() or similar to cause OS to flush pending writes before proceeding, or fail. In this case, there is no way around blocking. Application cannot proceed if log write fails, which means concurrent logs will need to wait for each other, or each log to their own per-thread file.
2) Are you sure file writes are the problem? Do you generate more than 10MB per second? Can you log to a SSD? Can you log over gigabit network with hardware accelerate card? If not, then it is likely that logging will not be a bottleneck.

To avoid stalls or prevent increased jitter when logging, the preferred way would be to use either async file access (cannot be done via fstream, depends on OS), or an in-memory database running in a separate process.

A user-mode, cross-platform logging can be implemented like this (See services, logging). This uses serial out-of-thread file access. If application crashes, logs will be lost. There is no way around this without blocking.

For bulk logging (10MB+ per second), scatter/gather style logging is possible. Each thread has its own file, each log is timestamped. They can write without impeding each other (if each commit is atomic and completes), but logs must be merged before analyzing. SSD, RAID or network might be preferable for this type anyway.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
The file write will lock on its own. File is a sequential device and cannot be accessed concurrently [...]


Maybe the actual, physical file will but what about the ostream-object representing the file? What if internal state is modified by writing to a file via an ostream object?

Share this post


Link to post
Share on other sites
Quote:
Original post by cache_hit
fyi, Boost.Log is pending review and I think should be accepted pretty soon.


Boost log was pending review some 3 years ago when I first saw it. It has been in development since before then.

I no longer care about stuff coming "real soon". Give me something that has been in use for a year.

Seriously, logging is not some newfangled functionality, but the very core of every application for past 40 years.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
A user-mode, cross-platform logging can be implemented like this (See services, logging). This uses serial out-of-thread file access. If application crashes, logs will be lost. There is no way around this without blocking.
It is possibel to implement a logging server in a separate process, and have that server buffer the logs as necessary. Communicate via local IPC or sockets.

This approach should give one all the advantages of out-of-thread logging, plus logs survive program crashes, at the cost of some implementation complexity.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by cache_hit
fyi, Boost.Log is pending review and I think should be accepted pretty soon.


Boost log was pending review some 3 years ago when I first saw it. It has been in development since before then.

I no longer care about stuff coming "real soon". Give me something that has been in use for a year.

Seriously, logging is not some newfangled functionality, but the very core of every application for past 40 years.


This is a completely new logging library. The last one was abandoned by the author, this one is by a new author and his own implementation, written from the ground up, and the author is more committed to supporting it than the other author.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
It is possibel to implement a logging server in a separate process, and have that server buffer the logs as necessary. Communicate via local IPC or sockets.

This approach should give one all the advantages of out-of-thread logging, plus logs survive program crashes, at the cost of some implementation complexity.


If you do not need a guarantee that each log is written, then there is no problem.

Consider a radiation therapy device which needs to be audited that not only is each log written, but also replicated and stored in a way that cannot be tampered with, plus each log must be kept for a minimum of 10 years in a physically separate location with paper trail, while there might be dozens of logs generated each second in a real-time system.

In more common case however, the following two scenarios are important:
- How bad is it if random swaths of logs get lost or malformed?
- How bad is it if last n logs are not written

Second option is probably much worse, since it will make diagnosing crashes impossible - the only time you really need each log and as many as possible is in the last 5 functions calls that led to crash.

Sockets, out-of-thread writes and even async writes all lead to second case. Even transactional file systems can be a problem, if process fails in the middle of transaction which might be writing a considerable chunk of data.

Simply put, for reliable logging, flush each log to disk after each write, or use a database that implements this. Locking is highly problematic here, since the important logs might be pending write blocked by a lock, while an INFO is being written.


Ugh... performance is the very last issue one needs to deal with... Just keep one log file per thread, and merge them externally. Very few systems actually put any effort in logging, they do well enough just with fwrite(). Performance these days is usually solve by adding more cheap machines.

Share this post


Link to post
Share on other sites
but I'd like to elaborate this topic. My logger uses TLS buffer for log-message formatting and std::ofstream for output. Buffering is disabled by std::ofstream::rdbuf()->pubsetbuf(0, 0). The question is: can I use std::ofstream::write() w/o locking? Locking is undesirable because logging is pretty intensive and this can reduce performance.

Apparently Microsoft's implementation of iostream is thread safe so you can write to it from two threads at once if you want. The other STL containers are okay for simultaneous reads but not writes. You'd probably want some sort of safeguard if you were going to rely on that behavior. Maybe assert(IS_VISUAL_STUDIO) or something.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
  • Advertisement
  • Popular Tags

  • Advertisement
  • Popular Now

  • Similar Content

    • By 3dmodelerguy
      So I am building a turn based rogue-like (think CDDA). The game is going to have a very large map (up to 1000's x 1000's) however to alleviate most of that I obviously can't render everything so there will just be render a certain radius around the player and just load in and out data as the player moves.
      The next major system I am prototyping is making interactive tiles destructible and pretty much everything will be destructible besides basic landscape (cars, doors, windows, structures, etc. will be destructible)
      While I am only rendering a certain amount of tiles around the player, I want to keep the amount of colliders active at one time to be as small as possible for performance and currently the tilemap tool I use automatically merges colliders together.
      So instead of creating a separate colliders for each of these tiles and having the destructible behavior tied to that object (which my tilemap tool would allow me to do) I was thinking that I would store an array of all the X and Y locations for the interactive tilemap layer and let the tilemap manage the colliders. 
      Then when I hit a collider on the interactive tilemap layer, instead of of getting the behavior for how to deal with the destruction for that tile from that game object, I would pull it from the array I mentioned earlier based on the tile I attempt to interact with which I already have.
      Does this sound like a good approach? Any other recommendations would be welcomed.
    • By NDraskovic
      Hey guys,
      I have a really weird problem. I'm trying to get some data from a REST service. I'm using the following code:
       
      private void GetTheScores() { UnityWebRequest GetCommand = UnityWebRequest.Get(url); UnityWebRequestAsyncOperation operation = GetCommand.SendWebRequest(); if (!operation.webRequest.isNetworkError) { ResultsContainer rez = JsonUtility.FromJson<ResultsContainer>(operation.webRequest.downloadHandler.text); Debug.Log("Text: " + operation.webRequest.downloadHandler.text); } } The problem is that when I'm in Unity's editor, the request doesn't return anything (operation.webRequest.downloadHandler.text is empty, the Debug.Log command just prints "Text: "), but when I enter the debug mode and insert a breakpoint on that line, then it returns the text properly. Does anyone have an idea why is this happening?
      The real problem I'm trying to solve is that when I receive the text, I can't get the data from the JSON. The markup is really simple:
      [{"id":1,"name":"Player1"},{"id":2,"name":"Player2"}] and I have an object that should accept that data:
      [System.Serializable] public class ResultScript { public int id; public string name; } There is also a class that should accept the array of these objects (which the JSON is returning):
      [System.Serializable] public class ResultsContainer { public ResultScript[] results; } But when I run the code (in the debug mode, to get any result) I get an error: ArgumentException: JSON must represent an object type. I've googled it but none of the proposed solutions work for me.
      Also (regardless if I'm in the debug mode or not) when I try to do some string operations like removing or adding characters to the GET result, the functions return an empty string as a result
      Can you help me with any of these problems?
      Thank you
    • By nihitori
      The Emotional Music Vol. I pack focuses on beautiful and esoteric orchestral music, capable of creating truly emotive and intimate moods. It features detailed chamber strings, cello and piano as the main instruments, resulting in a subtle and elegant sound never before heard in video game royalty-free music assets.

      The pack includes 5 original tracks, as well as a total of 47 loops based on these tracks (long loops for simple use and short loops for custom / complex music layering).

      Unity Asset Store link: https://www.assetstore.unity3d.com/en/#!/content/107032
      Unreal Engine Marketplace link: https://www.unrealengine.com/marketplace/emotional-music-vol-i

      A 15 seconds preview of each main track is available on Soundcloud:
       
    • By RoKabium Games
      Another one of our new UI for #screenshotsaturday. This is the inventory screen for showing what animal fossils you have collected so far. #gamedev #indiedev #sama
    • By eldwin11929
      We're looking for programmers for our project.
      Our project is being made in Unity
      Requirements:
      -Skills in Unity
      -C#
      -Javascript
      -Node.js
      We're looking for programmers who can perform a variety of functions on our project.
      Project is a top-down hack-and-slash pvp dungeon-crawler like game. Game is entirely multiplayer based, using randomized dungeons, and a unique combat system with emphasis on gameplay.
      We have a GDD to work off of, and a Lead Programmer you would work under.
      Assignments may include:
      -Creating new scripts of varying degrees specific to the project (mostly server-side, but sometimes client-side)
      -Assembling already created monsters/characters with existing or non-existing code.
      -Creating VFX
      -Assembling already created environment models
      If interested, please contact: eldwin11929@yahoo.com
      This project is unpaid, but with royalties.
       
      ---
      Additional Project Info:
      Summary:
      Bassetune Reapers is a Player-verus-Player, competitive dungeon crawler. This basically takes on aspects of dungeon crawling, but with a more aggressive setting. Players will have the option to play as the "dungeon-crawlers" (called the 'Knights', or "Knight Class", in-game) or as the "dungeon" itself (literally called the 'Bosses', or "Boss Class", in-game). What this means is that players can choose to play as the people invading the dungeon, or as the dungeon-holders themselves.
      Key Features:
      -Intense, fast-paced combat
      -Multiple skills, weapons, and ways to play the game
      -Tons of different Bosses, Minibosses, creatures and traps to utilize throughout the dungeon
      -Multiple unique environments
      -Interesting, detailed lore behind both the game and world
      -Intricate RPG system
      -Ladder and ranking system
      -Lots of customization for both classes s of customization for both classes
  • Advertisement