Jump to content
  • Advertisement

Search the Community

Showing results for tags 'Education'.

The search index is currently processing. Current results may not be complete.


More search options

  • Search By Tags

    Type tags separated by commas.
  • Search By Author

Content Type


Categories

  • Audio
    • Music and Sound FX
  • Business
    • Business and Law
    • Career Development
    • Production and Management
  • Game Design
    • Game Design and Theory
    • Writing for Games
    • UX for Games
  • Industry
    • Interviews
    • Event Coverage
  • Programming
    • Artificial Intelligence
    • General and Gameplay Programming
    • Graphics and GPU Programming
    • Engines and Middleware
    • Math and Physics
    • Networking and Multiplayer
  • Visual Arts
  • Archive

Categories

  • Audio
  • Visual Arts
  • Programming
  • Writing

Categories

  • Game Dev Loadout
  • Game Dev Unchained

Categories

  • Game Developers Conference
    • GDC 2017
    • GDC 2018
  • Power-Up Digital Games Conference
    • PDGC I: Words of Wisdom
    • PDGC II: The Devs Strike Back
    • PDGC III: Syntax Error

Forums

  • Audio
    • Music and Sound FX
  • Business
    • Games Career Development
    • Production and Management
    • Games Business and Law
  • Game Design
    • Game Design and Theory
    • Writing for Games
  • Programming
    • Artificial Intelligence
    • Engines and Middleware
    • General and Gameplay Programming
    • Graphics and GPU Programming
    • Math and Physics
    • Networking and Multiplayer
  • Visual Arts
    • 2D and 3D Art
    • Art Critique and Feedback
  • Community
    • GameDev Challenges
    • GDNet+ Member Forum
    • GDNet Lounge
    • GDNet Comments, Suggestions, and Ideas
    • Coding Horrors
    • Your Announcements
    • Hobby Project Classifieds
    • Indie Showcase
    • Article Writing
  • Affiliates
    • NeHe Productions
    • AngelCode
  • Topical
    • Virtual and Augmented Reality
    • News
  • Workshops
    • C# Workshop
    • CPP Workshop
    • Freehand Drawing Workshop
    • Hands-On Interactive Game Development
    • SICP Workshop
    • XNA 4.0 Workshop
  • Archive
    • Topical
    • Affiliates
    • Contests
    • Technical
  • GameDev Challenges's Topics
  • For Beginners's Forum
  • Unreal Engine Users's Unreal Engine Group Forum
  • Unity Developers's Forum
  • Unity Developers's Asset Share

Calendars

  • Community Calendar
  • Games Industry Events
  • Game Jams
  • GameDev Challenges's Schedule

Blogs

There are no results to display.

There are no results to display.

Product Groups

  • Advertisements
  • GameDev Gear

Find results in...

Find results that contain...


Date Created

  • Start

    End


Last Updated

  • Start

    End


Filter by number of...

Joined

  • Start

    End


Group


About Me


Website


Role


Twitter


Github


Twitch


Steam

Found 16 results

  1. Hello and welcome to this weeks dev diary! Today I'll show a couple of ways to slice up your images into smaller portions, one that doesn't require any plugins and one that need you to download a script. The following tutorial uses the default settings for GIMP, just like a fresh installation with no previous installations that might have retained some of your old settings. The GIMP Version is 2.10.12 The Manual Way Slicing the image manually can be really work intensive, depending how many parts you want to slice your image to and is by no means an efficient way to do it, but will suffice for anyone who wishes to divide their image to 4 x 4 tiles or less. 1. When you have loaded up the image you want to slice into smaller pieces, go to image > Guides > New Guide... 2.a When you click on the "New Guide..." the following box appears. 2.b Now you have to do some math and divide your image into as many parts as you want. You have to do this both horizontally and vertically as many times as you need guides. For this tutorial I'll make the image into 4 x 4 tiles, meaning 3000x3000 pixels in size, as per 12000 / 4 = 3000 - this means that I need to space all the guides 3000 pixels apart from each other, adding 3000 to each position. i.e. the first guide to 3000, the second guide to 6000, and the third guide to 9000. 3. After you have added your guides, your image should look like this; 4. Go to Image > Slice Using Guides - Click it and GIMP will divide the image for you into separate images and switch your view to the last piece. The original image will still be intact. GIMP will use the same filename that your original file has, but add to the row and column number to the end of the name, like seen in the above image filename, on the top right corner. Original filename in this example is; M5213A-texture-12000x12000.png The last piece has an addition at the end of it; M5213A-texture-12000x12000-3-3.png The below image showcases how GIMP organizes the tiles. First number represents the Column and the second number represents the Row; 5. Save each image separately, by going to File > Export As... (or using shortcut of Shift + CTRL + E). The file extension is the same as the original file's, so no need to change any setting when saving the files. The default export path is the one that the original image resides in, so you may want to create a new folder for the image slices. There is also several plugins that helps with saving multiple tabs in GIMP. The Much More Automated Method 1. First, download these plug-ins; ofn-layer-tiles Export Layer 2. Install both plugins into C:\Users\"Your-Username"\AppData\Roaming\GIMP\2.10\plug-ins Note that the "2.10" folder is the version of the GIMP you have installed and will change after each update that changes the first or second version number. Now your plug-ins folder should have these files. 3. After opening GIMP and your image file, remove the ".png" from your layer name. This will ensure that the "Export Layers" plugin names the layers correctly. 4. Go to Layers > Tiles > Split tiles (by rows and columns)... 5. Choose how many rows and columns you want, no need to change other settings, unless you want to specify different naming schemes. Read the plugin instructions for more information on that. For the purpose of this tutorial, I will be splitting the image into 4 x 4 tiles, just like before. By default, the slicing process named the files with the same logic as GIMP, the first number represent the Column and the second one is the ROW. The only difference is, that the plugin starts the numbering from 01 instead of 00, that GIMP starts from. This can be changed in the settings how ever, if this is important to you. To make the numbering start from 0, change "{column1:02d}-{row1:02d}" to "{column0:02d}-{row0:02d}". 6. Press the "OK" button and the plugin will change the image into separate layers. 7. Go to File > Export Layers... Choose the folder you would like to save the images to and your preferred file format. After you are done, click "Export". The Plugin will save each layer as a file and depending on the size of your original file, this could take some time. Conclusion For my needs, these plug-ins are gems. I'm just surprised how come GIMP does not include these kind of functionalities out of the box. But then again, open source programs tend to rely more on community to add more functionality to the core program in the form of plug-ins and scripts. One thing that should be a out-of-the-boc functionality, is the "Save All" option for saving all open documents easily and fast. Hope this was of use to you and I'll see you on the next one! You can check out every possible mid week announcements about the project on these official channels; • YouTube • Facebook • Twitter • Discord • Reddit • Pinterest • SoundCloud • LinkedIn •
  2. Hello and welcome to this weeks Dev Diary! Today, as promised last week, I'll be explaining how Astuvansalmi relates to the last weeks title; " Amūńe, The Old One ". Facts and Speculation Known facts are rare, as the Finnish language, or any language on the Finnic lands, have never had written language until fairly recently. Spoken stories were the only known way that people taught their customs and culture to each other - the way of the land, one could call it, due to it's organic nature. This is also why the names of the gods have changed and split into multitudes over the millennia, hence we can only ques what was the original names of the ancient gods, that were recently called by their modern names. Facts at the Astuvansalmi are as follows; There resides one of the oldest and largest rock painting collection in the northern Europe. It has seemingly had religious significance for many millennia, as there has been found several items from different times. Most significant ones are amber statues of a man, a woman, a boy and a bears head, which points to the sites religious meaning. The human figurines were named as following; Statue of the man is called "Astuvan Ukko", the "Ukko of Astuva". Ukko can mean several things, "Old Man", "Grandfather" or when talking about a god, "The Old One". Statue of the woman is called "Astuvan Akka", the "Akka of Astuva". Akka means "Old Woman", "Grandmother" or a feminine "Old One", when used from a goddess. Also the largest one found. Statue of the boy is called "Astuvan Poika", the "Poika of Astuva". Poika is a straight translation for "Boy", so there's no mystery there. The head of the bear is a reference to the forefathers, as bears were widely considered to be the ancestors of the locals, which has also been called "the Bear Clan". All of the figurines had a hole through them, which indicates that the statues were actually a necklace or a pendant at one time, considered to bring good luck to it's wearer. In what aspects of life the luck was supposed to show, can only be speculated, but since there is an indication of two greater spirits or gods and their son, with a bears head, that could very well mean, that the purpose of the necklace was to bring good fortune in life in general - Abundance of sustenance, fertility and strength from the forefathers - praying for the continuity of the family or clan. Picture of "Astuvan Ukko", photographed by Ritva Bäckman , 1990. One of the pictures on the rock depicts a woman with a bow, who was named "Astuvan Tellervo", Tellervo being the daughter of the god of the forest Tapio. Because it is thought that women were not allowed to hunt, the depiction is considered to be divine in nature. It is also possible that the depicted woman had been considered to be the mythological and/or deified foremother of the clan. The pictographs also depicted many pictures of moose facing west, but one was standing looking towards the east. East was considered to be the living place for the good gods, as the Sun rises from the east. Moose was one of the most sacred animals, as the Sun was considered to being a Golden Moose riding across the sky by many northern cultures at the time, some of the native peoples telling the same story even to this day. The rock also had several paw imprints, that points to the sacred nature of the site. Some of the paw imprints were made with bear paws, which could also mean that the "Karhulaiset", or the Bear Clan had claimed the site for their own, as the bear clan had seemingly a habit to mark their territory with marking the walls with bear paws (According to the Finnish Wikipedia site). It could also be, that the Astuvansalmi was one of the holiest place in the Southern Finnish territory back in the 3000 - 1200 BCE time-period and was regularly fought over (my own speculation). Astuvan Tellervo, by Ohto kokko. a Son of the Gods? Mikael Agricola, the "father of literary Finnish", who created the first written form of Finnish, wrote that according to Tavastians, Väinämöinen is said to have been the god of chants, songs and poetry. In Kalevala he is seen as an ancient hero, born from his mother, Ilmatar and fathered by water. "Väinä" literally means "stream pool", which is also very depicting of being a god of chants, songs and poetry - "the words were just flowing off of my tongue", as one of the sayings go. Also a tidbit that not many might know; Väinämöinen was one of the inspirations for Gandalf, Tom Bombadil and Ents in general, in the Lord of The Rings. It could also be, that the necklace depicted a son of the gods, which there is no certain information about. But if it did depict the son of the gods in the necklace, it could be the earliest sign of Väinämöinen or some similar deity was infact being worshiped on certain situations. This is pure speculation though, but might hold true, if more finding pointing toward this possibility are uncovered. Sadly, as the Finnish soil is rather acidic, not much survives for long periods of time, when it comes to most of the decomposing materials. The oldest fishing net found in the Finnish-Karelian region is the Antrea Net, which dates back to 8540 BCE, making it one of the oldest fishing nets in the world ever found, so there is hope yet. The Connotations of Amūńe Amūńe is Veps for "Ammoinen", "Muinainen", meaning "Ancient", referring in this context to the ancient gods of the Ancient Finns. Veps on the other hand is considered to be the Sanskrit of modern Finnish language, or in other words, a great grandfather of the modern Finnish language, that is considered to have the most similarities with the Late Proto-Finnic language. Thus a fitting common name to depict the old gods (the Old Ones), the oldest speculated culture and/or ancestors of the people of this larger northern Finnic region in general. At first, I was referring to Ukko with the Amūńe, but since I made some further research last week, I learned quite much about how many different age layers there are with the Finnish gods, or any ancient mythology for that matter, as all of the ancient religions bind around the natural order of things, as they work as a way to rationalize and/or depict in metaphors how the world works or what has happened in the past - what occurrences has led the world to this moment or situation in general. The first speculated animal representing Ukko could have well been the White-Tailed Eagle, according to Unto Salo, a 78 years old archeologist and emeritus Professor at the University of Turku. According to him, the roots of the Uralic mythological "Ukkoskotka", or "Thunder Eagle" can be traced remarkably far, or at least to 5000 BC. But it begs a question, as the Native American's has legends of the "Thunderbird", which is also depicted as an eagle. This could indicate that the "Thunder Eagle" may have been a thing even before the first Native American's crossed over to the American continent 30 000 years ago, or sometime between 30 000 and 10 000 years ago, that is estimated to have been the period when multiple waves of Native American's arrived to the American Continent. This is only my own speculation currently, as it might also just be a cultural coincidence. After being known as the "Thunder Eagle" and before the similarities with the Greek Zeus and Roman Jupiter became apparent, Ukko was known as a God of the Hunt and Fertility - of which the fertility aspect can be seen even on the later iterations, as Ukko is later known as the God of Thunder and Rain, rain being one of the necessities for crops to grow. Akka therefore represents most likely the ground in which the crops grow, the mother nature, or equivalent to Gaia, before the more well known pantheon of gods overthrew the old gods, known as Titans in the Greek mythology. Conclusion This research is related to the backgrounds of the upcoming RTS title and the larger mythos behind my other planned works. This is only the tip of the iceberg thus far, as there is still much to uncover from the depths of the past. I still need to gather more material from archeological and language study findings about Finnic pre-history to have a more complete and enticing setting for the game, so stay tuned. Thank you for tuning in, and I'll see you on the next one! You can check out every possible mid week announcements about the project on these official channels; • YouTube • Facebook • Twitter • Discord • Reddit • Pinterest • SoundCloud • LinkedIn • Feature Photo; Painting by Robert Ekman in 1867 called Lemminkäinen tulisella järvellä where Lemminkäinen asks help from Ukko ylijumala with crossing the lake in fire on his route to Pohjolan häät.
  3. So you're a filmmaker entering post-production on your film, a game developer working on music and audio triggers or perhaps a YouTuber wanting to enhance your videos. You need SFX and a music soundtrack but you might not have the budget to hire a professional composer or audio engineer. The question is: Can you still have a great commercial soundtrack and great SFX for your project utilizing only free resources? Definitely. Quick story about why I am writing this article: My career is in electronic music production, but I also love game design and have always wanted to create my own video game, so 2 years ago I decided to produce my first interactive horror visual novel. I was designing everything in-house: music, SFX, graphics, writing, everything. Many of you reading this are probably quite similar to me and are doing most of the work for your project yourself. I could handle the music, the writing, and the code, but there was no avoiding the obvious fact that I was terrible at drawing and I did not have the field-recording equipment necessary to record my own sound effects. I also did not have the budget to hire an audio engineer or an artist to provide me with SFX and the artwork needed for the game, so I had a dilemma. How do I immerse the player in my story without quality sound effects, artwork, and no budget to pay for them? Well, I did what I always do when I have a problem that seems impossible to overcome: I asked the internet. I spent days researching online and long story short I found dozens of websites providing free commercially-usable resources for my project. And imagine my surprise when a few of these free resources...didn't suck! Not only did I end up finishing my game without spending a penny, but I could still sell it! So I got to thinking, what about filmmakers, game developers, and YouTubers who already have great video and artwork but need great music and SFX? Are there equally-powerful free music libraries out there to search and download from? I did my research and once again found some incredible resources that I am going to share with you now! And not just that, but I'm going to help you incorporate and edit these free resources so they don't sound like you just downloaded them off random websites online. Here's what we are going to cover: Where to obtain free, commercially usable music & SFX for your project without sacrificing the quality of your end-product How to edit music and SFX you downloaded from widely different sources to create a unified soundtrack that works with your project How to create loops, fade-ins, fade-outs and layer audio to immerse the audience Licensing, what it means and what restrictions you have when using these resources (not many, I promise) How to change the format of your audio to work with your software Where to find volunteers or paid professionals if you can't find what you need So without boring you to death, let's get started with the best 9 websites to download free, commercially-usable music & SFX! TOP 5 FREE COMMERCIALLY USABLE MUSIC LIBRARIES: (For SFX, Keep Scrolling) JordanWinslow.me - My Personal Library of Hundreds of Electronic & Orchestral Soundtracks Arranged by Category for Free Download - License: Free Commercial Use With Attribution (See Terms of Service on Website) As I said before, I have been producing electronic music for over 14 years now, 5 years of which it has been my primary source of income. So I wanted to put together my own free resource for others to benefit from! These are some of my absolute best soundtracks, many of which are loopable. And I spent many days organizing them by category and mood to make it easier for you to find what you're looking for! And the best thing is, all of the tracks can be listened to without even leaving the page and can be easily downloaded in 1 click! Icons8 - Incredibly Well-Sorted, Professional Library of Hundreds of Songs from Various Artists - License: “Free for a Link” (See Website) Don’t be fooled by this company’s origin: They started off as graphic designers who made icons, thus Icons8. But they have evolved and got their hands on a rather large music library of various artists who have been curated by their team. Naturally, when a library is curated, it is subjective and therefore might not be to your tastes if you disagree with how they select their tracks for inclusion on their website, but it can’t hurt to take a look at their gallery since it is so incredibly well-sorted! Incompetech - A Classic Library of a Few Hundred Songs Ranging from Classic Rock to Jazz - License: Primarily Creative Commons: By Attribution 3.0 Incompetech has been around for quite some time and is recommended by many other bloggers because it provides a convenient category system. Unfortunately, the tracks are not organized by mood or tonal characteristics other than genre, so you will find tracks with the instruments you are looking for, but it may take some digging to find the appropriate mood you are looking for. dig cc mixter - A Massive Library of Non-Categorized Music Submitted by Various Artists - License: Primarily Creative Commons: By Attribution 3.0 Unfortunately dig cc mixter, though a MASSIVE resource with tons of great tracks, is not a great place to find music in the specific genre you are looking for as there are no categories and no search functions to speak of. If you are willing to spend the time looking through thousands of tracks though, you will find a few hidden gems on here that will fit perfectly in your project! Josh Woodward - 200+ Primarily Acoustic & Electric Guitar Driven Songs - License: Commercial Use With Attribution (See Website for Terms) The best part about Josh Woodward’s free commercially usable library is that he has tagged each and every song with different moods, themes, and styles to make it very easy for Filmmakers, Game Developers & Artists to search through tracks to find songs in the mood they are looking for! Honorable Mention: Partners in Rhyme - A Little Over 100 Free Unsorted Music Loops - License: Free for a “Thanks” or With Attribution (Vague, See Terms on Website) This library is more of a last resort as the tracks are unsorted and not as high quality as others on this list, however free is free and these tracks would be suitable for app developers or creators who are looking for this type of sound. TOP 3 FREE COMMERCIALLY USABLE SFX LIBRARIES: ZapSplat - 27,000+ Searchable Sound Effects Recorded by Professionals - License: Free Commercial Use With Attribution ZapSplat is by far my favorite free SFX resource. When I first discovered their website it had far fewer audio files and a much less appealing logo design. It appears they are dedicated to growth because they have completely redesigned their branding and added thousands of audio clips to their website! I personally used this resource in the development of my Horror Visual Novel titled “The Watchers.” Soundeffects+ - Over 5000 Free Sound Effects Sorted by 16 Categories - License: Free Commercial Use With Attribution (See Website for More Details) Soundeffects+ offers a large library sorted by the categories visible in the screenshot above. See something you’re looking for? Click the link and go check it out. Otherwise, keep scrolling! Videvo - 400 Free Sound Effects Sorted by Over 20 Categories - License: Complicated, each sound effect has it’s own license and it varies. Check each sound page for the license. Videvo is primarily a stock video provider with many free video clips, but they are also breaking into sound as well and have an expanding library of 440 clips which isn’t much but they are very well sorted so it should be easy to find something unique for your project. Make sure you check their licensing page as their licensing is quite complex compared to other websites. In the above 9 websites, you should be able to obtain all of the sound effects and music for your project as long as you are willing to put in the time filtering and searching through these libraries to find what you are looking for. It may seem like a daunting task, but I have done it personally myself in my game development and have had great success! My suggestion to you is download anything that sounds interesting to you at the time, even if you are unsure if you can use it in your project or not, and copy-paste the license information into a .txt file so you don’t forget to give proper attribution. Part 2: How to Choose the Right Music & SFX for Each Scene, and How to Edit Audio to Achieve Your Goals Once you have a selection of songs or SFX for your project it's time to edit. Since most of you will be using many different types of software I am only going to cover how to edit music in 3rd party FREE software, namely Audacity. Don't knock it, Audacity is very powerful software and unless you're considering a career in audio engineering, music production, sound design or mixing, this is probably the only tool you'll ever need. If you want more professional audio editing tools I highly suggest iZotope's RX6 software as it allows you to do incredible things such as take backgrounds out of one scene and move them into another, repair poor recordings and dubbing, and more. How to loop music that wasn't originally recorded as a loop: To achieve this the easiest method is to create a soft fade-in and fade-out on the track. You can experiment with different values but 1-2 seconds on each end usually suffices unless the music is louder or more complex, then you can try up to 4 seconds on each end or even more for atmospheric loops. Here is an easy to follow video tutorial on fade-ins, fade-outs and looping audio: https://youtu.be/ryLpfVecUDs How to make everything sound cohesive, as if everything was designed specifically for your project: Keep in mind, layering audio is an incredibly easy, yet very powerful tool at your disposal. You can loop one audio track while another one continues to play underneath it to keep the player from noticing the loop. You can even create elaborate scenes with chattering people, blowing wind, ambient tones, and musical accompaniment. All of these types of atmospheres can be downloaded at the above free resources! It is important to consider the stylistic and tonal changes of the music you downloaded when switching from one song to another. Don't just go from a percussive action track straight into a somber atmospheric melody. Transitioning is key: utilize fade-ins and fade-outs during most, if not all of your audio changes so the experience draws the audience further into your story rather than taking their focus off the screen and into the audio. Oh yeah, and... Epic Music Does Not Make a Boring Scene More Epic! I think there is a huge problem in the video game industry specifically (filmmakers don't scoff, it's a problem in your industry too, but perhaps less pronounced) where game developers think if they make the music louder and louder and more and more epic it will somehow make the game more fun or the experience more immersive. Well, it doesn't. Many times have I been playing through a game or watching a film where the audio is 10X more dramatic than what is happening on screen and it makes me just want to mute it or turn it down. This is not the experience you want to give your audience, trust me. Consider the emotion of every scene before you place any music and ensure that listening to the music by itself gives you the feeling you want the player to have, but don't expect the music and sound effects to do the work for you on making the scene enjoyable and immersive! Once you have a rough draft of your soundtrack & SFX library, go back to Step 1 and make SURE you didn’t miss any audio that may be in other categories you didn’t listen to that might fit the scenes you’re working on. Part 3: Obtain the Appropriate License to Use the Music & SFX and Ensure You Have Given Proper Attribution Whenever you’re working with royalty free music & SFX you always have to keep in mind that just because the music is free doesn’t mean you don’t have to cite the author. For example, if you’re using my personal Royalty Free Music Catalog I linked to earlier then this part is very simple: If you will not make money from your project directly or indirectly (this includes advertisements and YouTube monetization) then all you have to do is put “Music Downloaded From https://JordanWinslow.Me/RoyaltyFreeMusic” in your credits, description or somewhere easily visible in your project. If you will make money from your project directly or indirectly, simply fill out the Commercial License Request Form found on the website and enter in the title of your project and your project’s information for EACH project you will require music for. All of the sites I linked above have very similar licensing agreements, so just read up on the individual website before you download, and ensure you create a .txt document with all the links you need so you don’t forget! The best part about all of the above libraries is that almost every song and SFX clip you download can be legally edited, looped, layered, remixed and changed any way you see fit! The only restriction is you cannot sell or distribute your edited or remixed audio clips as standalone clips if they were your own because technically the author still retains copyright ownership over the files. But that does not mean you can't sell your film or video game with the edited audio! If you are confused, double-check the licensing page on each website to be sure. And that’s how you spend time instead of money to create a custom soundtrack for your film, video game or YouTube video! What to Do if you Still Haven’t Found What You’re Looking For, or the Audio You Downloaded isn’t the Correct Format If the audio you downloaded isn’t in the correct format for your software, you can use the free open source tool Audacity to convert it by using the "Export" menu to change the format of your audio or use this free online audio converter. Keep in mind that certain audio formats like .mp3 have restrictions on where they can be used. I recommend .ogg since it is an open source audio format with great quality and compression. Now if for some reason you don’t find the music or SFX you need in those libraries of thousands of songs and SFX, it’s probably time to consider looking for a volunteer composer or simply hiring a professional. You can find low-cost audio engineers and composers on websites like Fiverr and Upwork, but keep in mind that quality products do not often come with low price tags, be wary of anything that seems "too good to be true" because it probably is. And be sure to listen to their portfolio thoroughly before making a decision! I know it’s not easy to make a career out of your passions when you’re on a limited budget, believe me, just read My Story if you want to know how I spent 6 years in poverty before becoming a successful electronic music producer & composer. But I guarantee if you put the time into finding music and SFX in the above libraries, or looking for a great volunteer, you can get your project done at no cost other than the hardware and software you purchased! If you have any questions, feel free to reach out to me any time! See you later, creators! Electronic Music Producer, Composer & Audio Engineer https://JordanWinslow.Me
  4. It is good for programmers to understand what goes on inside a processor. The CPU is at the heart of our career. What goes on inside the CPU? How long does it take for one instruction to run? What does it mean when a new CPU has a 12-stage pipeline, or 18-stage pipeline, or even a "deep" 31-stage pipeline? Programs generally treat the CPU as a black box. Instructions go into the box in order, instructions come out of the box in order, and some processing magic happens inside. As a programmer, it is useful to learn what happens inside the box. This is especially true if you will be working on tasks like program optimization. If you don't know what is going on inside the CPU, how can you optimize for it? This article is about what goes on inside the x86 processor's deep pipeline. Stuff You Should Already Know First, this article assumes you know a bit about programming, and maybe even a little assembly language. If you don't know what I mean when I mention an instruction pointer, this article probably isn't for you. When I talk about registers, instructions, and caches, I assume you already know what they mean, can figure it out, or will look it up. Second, this article is a simplification of a complex topic. If you feel I have left out important details, please add them to the comments at the end. Third, I am focusing on Intel processors and the x86 family. I know there are many different processor families out there other than x86. I know that AMD introduced many useful features into the x86 family and Intel incorporated them. It is Intel's architecture and Intel's instruction set, and Intel introduced the most major feature being covered, so for simplicity and consistency I'm just going to stick with their processors. Fourth, this article is already out of date. Newer processors are in the works and some are due out in a few months. I am very happy that technology is advancing at a rapid pace. I hope that someday all of these steps are completely outdated, replaced with even more amazing advances in computing power. The Pipeline Basics From an extremely broad perspective the x86 processor family has not changed very much over its 35 year history. There have been many additions but the original design (and nearly all of the original instruction set) is basically intact and visible in the modern processor. The original 8086 processor has 14 CPU registers which are still in use today. Four are general purpose registers -- AX, BX, CX, and DX. Four are segment registers that are used to help with pointers -- Code Segment (CS), Data Segment (DS), Extra Segment (ES), and Stack Segment (SS). Four are index registers that point to various memory locations -- Source Index (SI), Destination Index (DI), Base Pointer (BP), and Stack Pointer (SP). One register contains bit flags. And finally, there is the most important register for this article: The Instruction Pointer (IP). The instruction pointer register is a pointer with a special job. The instruction pointer's job is to point to the next instruction to be run. All processors in the x86 family follow the same pattern. First, they follow the instruction pointer and decode the next CPU instruction at that location. After decoding, there is an execute stage where the instruction is run. Some instructions read from memory or write to it, others perform calculations or comparisons or do other work. When the work is done, the instruction goes through a retire stage and the instruction pointer is modified to point to the next instruction. This decode, execute, and retire pipeline pattern applies to the original 8086 processor as much as it applies to the latest Core i7 processor. Additional pipeline stages have been added over the years, but the pattern remains. What Has Changed Over 35 Years The original processor was simple by today's standard. The original 8086 processor began by evaluating the instruction at the current instruction pointer, decoded it, executed it, retired it, and moved on to the next instruction that the instruction pointer pointed to. Each new chip in the family added new functionality. Most chips added new instructions. Some chips added new registers. For the purposes of this article I am focusing on the changes that affect the main flow of instructions through the CPU. Other changes like adding virtual memory or parallel processing are certainly interesting and useful, but not applicable to this article. In 1982 an instruction cache was added to the processor. Instead of jumping out to memory at every instruction, the CPU would read several bytes beyond the current instruction pointer. The instruction cache was only a few bytes in size, just large enough to fetch a few instructions, but it dramatically improved performance by removing round trips to memory every few cycles. In 1985, the 386 added cache memory for data as well as expanding the instruction cache. This gave performance improvements by reading several bytes beyond a data request. By this point both the instruction cache and data cache were measured in kilobytes rather than bytes. In 1989, the i486 moved to a five-stage pipeline. Instead of having a single instruction inside the CPU, each stage of the pipeline could have an instruction in it. This addition more than doubled the performance of a 386 processor of the same clock rate. The fetch stage extracted an instruction from the cache. (The instruction cache at this time was generally 8 kilobytes.) The second stage would decode the instruction. The third stage would translate memory addresses and displacements needed for the instruction. The fourth stage would execute the instruction. The fifth stage would retire the instruction, writing the results back to registers and memory as needed. By allowing multiple instructions in the processor at once, programs could run much faster. 1993 saw the introduction of the Pentium processor. The processor family changed from numbers to names as a result of a lawsuit--that's why it is Pentium instead of the 586. The Pentium chip changed the pipeline even more than the i486. The Pentium architecture added a second separate superscalar pipeline. The main pipeline worked like the i486 pipeline, but the second pipeline ran some simpler instructions, such as direct integer math, in parallel and much faster. In 1995, Intel released the Pentium Pro processor. This was a radically different processor design. This chip had several features including out-of-order execution processing core (OOO core) and speculative execution. The pipeline was expanded to 12 stages, and it included something termed a 'superpipeline' where many instructions could be processed simultaneously. This OOO core will be covered in depth later in the article. There were many major changes between 1995 when the OOO core was introduced and 2002 when our next date appears. Additional registers were added. Instructions that processed multiple values at once (Single Instruction Multiple Data, or SIMD) were introduced. Caches were introduced and existing caches enlarged. Pipeline stages were sometimes split and sometimes consolidated to allow better use in real-world situations. These and other changes were important for overall performance, but they don't really matter very much when it comes to the flow of data through the chip. In 2002, the Pentium 4 processor introduced a technology called Hyper-Threading. The OOO core was so successful at improving processing flow that it was able to process instructions faster than they could be sent to the core. For most users the CPU's OOO core was effectively idle much of the time, even under load. To help give a steady flow of instructions to the OOO core they attached a second front-end. The operating system would see two processors rather than one. There were two sets of x86 registers. There were two instruction decoders that looked at two sets of instruction pointers and processed both sets of results. The results were processed by a single, shared OOO core but this was invisible to the programs. Then the results were retired just like before, and the instructions were sent back to the two virtual processors they came from. In 2006, Intel released the "Core" microarchitecture. For branding purposes, it was called "Core 2" (because everyone knows two is better than one). In a somewhat surprising move, CPU clock rates were reduced and Hyper-Threading was removed. By slowing down the clock they could expand all the pipeline stages. The OOO core was expanded. Caches and buffers were enlarged. Processors were re-designed focusing on dual-core and quad-core chips with shared caches. In 2008, Intel went with a naming scheme of Core i3, Core i5, and Core i7. These processors re-introduced Hyper-Threading with a shared OOO core. The three different processors differed mainly by the size of the internal caches. Future Processors: The next microarchitecture update is currently named Haswell and speculation says it will be released late in 2013. So far the published docs suggest it is a 14-stage OOO core pipeline, so it is likely the data flow will still follow the basic design of the Pentium Pro. So what is all this pipeline stuff, what is the OOO core, and how does it help processing speed? CPU Instruction Pipelines In the most basic form described above, a single instruction goes in, gets processed, and comes out the other side. That is fairly intuitive for most programmers. The i486 has a 5-stage pipeline. The stages are - Fetch, D1 (main decode), D2 (secondary decode, also called translate), EX (execute), WB (write back to registers and memory). One instruction can be in each stage of the pipeline. There is a major drawback to a CPU pipeline like this. Imagine the code below. Back before CPU pipelines the following three lines were a common way to swap two variables in place. XOR a, b XOR b, a XOR a, b The chips starting with the 8086 up through the 386 did not have an internal pipeline. They processed only a single instruction at a time, independently and completely. Three consecutive XOR instructions is not a problem in this architecture. We'll consider what happens in the i486 since it was the first x86 chip with an internal pipeline. It can be a little confusing to watch many things in motion at once, so you may want to refer back to the diagram above. The first instruction enters the Fetch stage and we are done with that step. On the next step the first instruction moves to D1 stage (main decode) and the second instruction is brought into fetch stage. On the third step the first instruction moves to D2 and the second instruction gets moved to D1 and another is fetched. On the next stage something goes wrong. The first instruction moves to EX ... but other instructions do not advance. The decoder stops because the second XOR instruction requires the results of the first instruction. The variable (a) is supposed to be used by the second instruction, but it won't be written to until the first instruction is done. So the instructions in the pipeline wait until the first instruction works its way through the EX and WB stages. Only after the first instruction is complete can the second instruction make its way through the pipeline. The third instruction will similarly get stuck, waiting for the second instruction to complete. This is called a pipeline stall or a pipeline bubble. Another issue with pipelines is some instructions could execute very quickly and other instructions would execute very slowly. This was made more visible with the Pentium's dual-pipeline system. The Pentium Pro introduced a 12-stage pipeline. When that number was first announced there was a collective gasp from programmers who understood how the superscalar pipeline worked. If Intel followed the same design with a 12-stage superscalar pipeline then a pipeline stall or slow instruction would seriously harm execution speed. At the same time they announced a radically different internal pipeline, calling it the Out Of Order (OOO) core. It was difficult to understand from the documentation, but Intel assured developers that they would be thrilled with the results. Let's have a look at this OOO core pipeline in more depth. The Out Of Order Core Pipeline The OOO Core pipeline is a case where a picture is worth a thousand words. So let's get some pictures. Diagrams of CPU Pipelines The i486 had a 5-stage pipeline that worked well. The idea was very common in other processor families and works well in the real world. The Pentium pipeline was even better than the i486. It had two instruction pipelines that could run in parallel, and each pipeline could have multiple instructions in different stages. You could have nearly twice as many instructions being processed at the same time. Having fast instructions waiting for slow instructions was still a problem with parallel pipelines. Having sequential instruction order was another issue thanks to stalls. The pipelines are still linear and can face a performance barrier that cannot be breached. The OOO core is a huge departure from the previous chip designs with their linear paths. It added some complexity and introduced nonlinear paths: The first thing that happens is that instructions are fetched from memory into the processor's instruction cache. The decoder on the modern processors can detect when a large branch is about to happen (such as a function call) and can begin loading the instructions before they are needed. The decoding stage was modified slightly from earlier chips. Instead of just processing a single instruction at the instruction pointer, the Pentium Pro processor could decode up to three instructions per cycle. Today's (circa 2008-2013) processors can decode up to four instructions at once. Decoding produces small fragments of operations called micro-ops or u-ops. Next is a stage (or set of stages) called micro-op translation, followed by register aliasing. Many operations are going on at once and we will potentially be doing work out of order, so an instruction could read to a register at the same time another instruction is writing to it. Writing to a register could potentially stomp on a value that another instruction needs. Inside the processor the original registers (such as AX, BX, CX, DX, and so on) are translated (or aliased) into internal registers that are hidden from the programmer. The registers and memory addresses need to have their values mapped to a temporary value for processing. Currently 4 micro-ops can go through translation every cycle. After micro-op translation is complete, all of the instruction's micro-ops enter a reorder buffer, or ROB. The ROB currently holds up to 128 micro-ops. On a processor with Hyper-Threading the ROB can also coordinate entries from multiple virtual processors. Both virtual processors come together into a single OOO core at the ROB stage. These micro-ops are now ready for processing. They are placed in the Reservation Station (RS). The RS currently can hold 36 micro-ops at any one time. Now the magic of the OOO core happens. The micro-ops are processed simultaneously on multiple execution units, and each execution unit runs as fast as it can. Micro-ops can be processed out of order as long as their data is ready, sometimes skipping over unready micro-ops for a long time while working on other micro-ops that are ready. This way a long operation does not block quick operations and the cost of pipeline stalls is greatly reduced. The original Pentium Pro OOO core had six execution units: two integer processors, one floating-point processor, a load unit, a store address unit, and a store data unit. The two integer processors were specialized; one could handle the complex integer operations, the other could solve up to two simple operations at once. In an ideal situation the Pentium Pro OOO Core execution units could process seven micro-ops in a single clock cycle. Today's OOO core still has six execution units. It still has the load address, store address, and store data execution units, the other three have changed somewhat. Each of the three execution units perform basic math operations, or instead they perform a more complex micro-op. Each of the three execution units are specialized to different micro-ops allowing them to complete the work faster than if they were general purpose. In an ideal situation today's OOO core can process 11 micro-ops in a single cycle. Eventually the micro-op is run. It goes through a few more small stages (which vary from processor to processor) and eventually gets retired. At this point it is returned back to the outside world and the instruction pointer is advanced. From the program's point of view the instruction has simply entered the CPU and exited the other side in exactly the same way it did back on the old 8086. If you were following carefully you may have noticed one very important issue in the way it was just described. What happens if there is a change in execution location? For example, what happens when the code hits an 'if' statement or a 'switch" statement? On the older processors this meant discarding the work in the superscalar pipeline and waiting for the new branch to begin processing. A pipeline stall when the CPU holds one hundred instructions or more is an extreme performance penalty. Every instruction needs to wait while the instructions at the new location are loaded and the pipeline restarted. In this situation the OOO core needs to cancel work in progress, roll back to the earlier state, wait until all the micro-ops are retired, discard them and their results, and then continue at the new location. This was a very difficult problem and happened frequently in the design. The performance of this situation was unacceptable to the engineers. This is where the other major feature of the OOO core comes in. Speculative execution was their answer. Speculative execution means that when a conditional statement (such as an 'if' block) is encountered the OOO core will simply decode and run all the branches of the code. As soon as the core figures out which branch was the correct one, the results from the unused branches would be discarded. This prevents the stall at the small cost of running the code inside the wrong branch. The CPU designers also included a branch prediction cache which further improved the results when it was forced to guess at multiple branch locations. We still have CPU stalls from this problem, but the solutions in place have reduced it to the point where it is a rare exception rather than a usual condition. Finally, CPUs with Hyper-Threading enabled will expose two virtual processors for a single shared OOO core. They share a Reorder Buffer and OOO core, appearing as two separate processors to the operating system. That looks like this: A processor with Hyper-Threading gives two virtual processors which in turn gives more data to the OOO core. This gives a performance increase during general workloads. A few compute-intensive workflows that are written to take advantage of every processor can saturate the OOO core. During those situations Hyper-Threading can slightly decrease overall performance. Those workflows are relatively rare; Hyper-Threading usually provides consumers with approximately double the speed they would see for their everyday computer tasks. An Example All of this may seem a little confusing. Hopefully an example will clear everything up. From the application's perspective, we are still running on the same instruction pipeline as the old 8086. There is a black box. The instruction pointed to by the instruction pointer is processed by the black box, and when it comes out the results are reflected in memory. From the instruction's point of view, however, that black box is quite a ride. Here is today's (circa 2008-2013) CPU ride, as seen by an instruction: First, you are a program instruction. Your program is being run. You are waiting patiently for the instruction pointer to point to you so you can be processed. When the instruction pointer gets about 4 kilobytes away from you -- about 1500 instructions away -- you get collected into the instruction cache. Loading into the cache takes some time, but you are far away from being run. This prefetch is part of the first pipeline stage. The instruction pointer gets closer and closer. When the instruction pointer gets about 24 instructions away, you and five neighbors get pulled into the instruction queue. This processor has four decoders. It has room for one complex instruction and up to three simple instructions. You happen to be a complex instruction and are decoded into four micro-ops. Decoding is a multi-step process. Part of the decode process involved a scan to see what data you need and if you are likely to cause a jump to somewhere new. The decoder detected a need for some additional data. Unknown to you, somewhere on the far side of the computer, your data starts getting loaded into the data cache. Your four micro-ops step up to the register alias table. You announce which memory address you read from (it happens to be fs:[eax+18h]) and the chip translates that into temporary addresses for your micro-ops. Your micro-ops enter the reorder buffer, or ROB. At the first available opportunity they move to the Reservation Station. The Reservation Station holds instructions that are ready to be run. Your third micro-op is immediately picked up by Execution Port 5. You don't know why it was selected first, but it is gone. A few cycles later your first micro-op rushes to Port 2, the Load Address execution unit. The remaining micro-ops wait as various ports collect other micro-ops. They wait as Port 2 loads data from the memory cache and puts it in temporary memory slots. They wait a long time... A very long time... Other instructions come and go while they wait for their micro-op friend to load the right data. Good thing this processor knows how to handle things out of order. Suddenly both of the remaining micro-ops are picked up by Execution Ports 0 and 1. The data load must be complete. The micro-ops are all run, and eventually the four micro-ops meet back in the Reservation Station. As they travel back through the gate the micro-ops hand in their tickets listing their temporary addresses. The micro-ops are collected and joined, and you, as an instruction, feel whole again. The CPU hands you your result, and gently directs you to the exit. There is a short line through a door marked "Retirement". You get in line, and discover you are standing next to the same instructions you came in with. You are even standing in the same order. It turns out this out-of-order core really knows its business. Each instruction then goes out of the CPU, seeming to exit one at a time, in the same order they were pointed to by the instruction pointer. Conclusion This little lecture has hopefully shed some light on what happens inside a CPU. It isn't all magic, smoke, and mirrors. Getting back to the original questions, we now have some good answers. What goes on inside the CPU? There is a complex world where instructions are broken down into micro-operations, processed as soon as possible in any order, then put back together in order and in place. To an outsider it looks like they are being processed sequentially and independently. But now we know that on the inside they are handled out of order, sometimes even running braches of code based on a prediction that they will be useful. How long does it take for one instruction to run? While there was a good answer to this in the non-pipelined world, in today's processors the time it takes is based on what instructions are nearby, and the size and contents of the neighboring caches. There is a minimum amount of time it takes to go through the processor, but that is roughly constant. A good programmer and optimizing compiler can make many instructions run in around amortized zero time. With an amortized zero time it is not the cost of the instruction that is slow; instead it means it takes the time to work through the OOO core and the time to wait for cache memory to load and unload. What does it mean when a new CPU has a 12-stage pipeline, or 18-stage, or even a "deep" 31-stage pipeline? It means more instructions are invited to the party at once. A very deep pipeline can mean that several hundred instructions can be marked as 'in progress' at once. When everything is going well the OOO core is kept very busy and the processor gains an impressive throughput of instructions. Unfortunately, this also means that a pipeline stall moves from being a mild annoyance like it was in the early days, to becoming a performance nightmare as hundreds of instructions need to wait around for the pipeline to clear out. How can I apply this to my programs? The good news is that CPUs can anticipate most common patterns, and compilers have been optimizing for OOO core for nearly two decades. The CPU runs best when instructions and data are all in order. Always keep your code simple. Simple and straightforward code will help the compiler's optimizer identify and speed up the results. Don't jump around if you can help it, and if you need to jump, try to jump around exactly the same way every time. Complex designs like dynamic jump tables are fancy and can do a lot, but neither the compiler or CPU will predict what will be coming up, so complex code is very likely to result in stalls and mis-predictions. On the other side, keep your data simple. Keep your data in order, adjacent, and consecutive to prevent data stalls. Choosing the right data structures and data layouts can do much to encourage good performance. As long as you keep your code and data simple you can generally rely on your compiler's optimizer to do the rest. Thanks for joining me on the ride. updates 2013-05-17 Removed a word that someone was offended by