Jump to content
  • Advertisement
  • Session A Tale of Three Data Schemas


    A Tale of Three Data Schemas
    Ludovic Chabant

    What's a Data Schema?

    Definition: formal structures of the data that your system is working with

    Examples include:

    • Unity script
    • Unreal UClass
    • Blind data in Maya
    • table columns in a rdbms
    • data definition format (DDF) in Frostbite
      • define data separately from logic

    With schemas we define:

    Type "SkinnedMeshAsset"
    Property "float lodScale"

    And the data shows up in the tools. The data can be stored (Frostbite stores in Perforce) and access it at runtime in code.

    Designing data schemas requires a lot of consideration:

    • If you bias too much to runtime you might end up with flattened or nicely packed but content creators can't use it.
    • If you make it too geared toward users, it might be too OOO, take up too much mmeory, not be cache friendly, etc

    But you don't always have to use a single data schema.

    The Three Data Schemas

    1. Runtime data schema

    Game loads it in memory, programmers use it in code.

    2. Storage data schema

    Used to keep things around somewhere. Not to be confused with data format - schema is about what you save. Format is how.

    3. Tools schema

    Normally same as storage schema with possibly a few small differences.

    The data is going to exist in the content creator's mind. How you visualize the schema will determine how the user views it.
    Node graph, property grid, etc. Schema design is closely related to UX design.

    Each data schema has a specific purpose. Runtime - reading, Storage - writing, Tools - user

    Four tales from Frostbite games

    1. Euler is your friend. Cotes is not.

    In the UI - property grid - translation, euler angles, scale. In code it's a 4x4 matrix. On disk, it's a 4x3 matrix.

    Property grid is more user friendly. On the fly conversion not always enough. In animation editor, angular values in degrees, stored and used in radians.

    Sometimes conversions and precision errors can be annoying. User enters 45 degrees, converts to radians internally, UI shows 44.99999.
    Solution: store in degrees, store radians in runtime, display in degrees.

    Expose user friendly data to content creators.

    2. The best of both worlds

    Cinematic tools have a lot of animation curves. Curve schema to content creator is a list of keyframes. Storage schema an array of keyframes, each item a struct with floats time, value, tangents, etc.

    This is efficient for the tool side. OO way to represent curves. Optimized for editing curves. Runtime uses array of structs. More efficient for traversals, memory, etc.

    Example: asset pipeline

    • storage and tools data schema is straightforward relational asset tree
    • runtime is a mix of flattened structures and string pools
    • use different schemas to solve different problems and optimize for use cases

    3. It's All in Your Mind

    Gameplay cameras are a camera state machine and camera modes

    First UI iteration - camera directors and modes open in different document tabs
    Problem: too many tabs open

    Second UI iteration: open camera modes inside same document tab as camera director
    Problem: contnet creators now think camera mode is "owned" by camera director
    Solution: change storage schema to enforce ownership

    Might think this was a UX problem but point of tale is that if you find a UX problem it might actually be a problem in the schema.

    4. Data Finds a Way

    • opening a document - storage to tools schema
    • cooking - tools schema to storage
    • live editing - tools to runtime

    live editing often implemented as

    set <objectid>.<property> = <value>

    This is wrong - assumes tools and storage schema same as runtime schemaStorage:

    class Widget 
      string frobNAme; 


    uint frobName;

    Combine the two:

    meta string frobName;   // meta expands to conditional compilation (#if !defined FINALORCONSOLE)
    uint frobName;

    meta fields used by good amount of pipeline code. Abstract the pipeline code to do it at runtime

    Use conditional compilation to include tools/storage schemas into runtime schemas.

    Conditionally include appropriate pipeline in runtime to transform data on the fly during live edit

    What to do With All of This

    At start of design, usually start with storage shema - only one that is persistent

    • also most expensive to change
    • change runtime? just update pipeline code
    • change storage? lots of existing data to change. have to upgrade data

    1. start with storage schema. looks at ddf changes during a code review
    2. build your tools around it
    3. write your runtime code around it
        - if data is not optimal
        1. adjust runtimes cehma and code as needed
        2. write popeline code to xform data
        3. jump here directly if storage scheme as obviously non-rutime friendly

    1. use conditional compilation to mix schemas
    2. don't forget live editing cases


    User Feedback

    Recommended Comments

    There are no comments to display.

    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!