Code organization without the limitations of files

1,875

September 11, 2014 12:40 PM

Firsty, I really don't agree with you about it being a problem. It's normal to have to alter multiple files for a single change, and I'm quite happy with that.

If you are really worried about code bloat and repetition on code blocks, restructure your classes in a nice OO way. Then turn on link time optomisation.

Ok that out of the way...

If you want to work in a different way, you have to write some code that sits in front of your compiler. This is not a new principle, most game engines use an editor instead of just a mass of source code.

You can make your "editor" for lack of a better word work in any way you want, then when you hit the magic build button, it spits out files that can be compiled by a traditional compiler.

This , again, is not a new idea.

Your chance to come up with something that is new, is what happens when you press the magic button.

What would you like? If you can dream it, someone can code it.

Fiddler

859

September 11, 2014 12:51 PM

I remember someone bringing up an idea about storing code in some sort of loosely-defined database, rather than plain text files. This was in GD maybe a decade ago so good luck finding a reference to the old thread, but the idea does have some merit.

Source control could operate on individual items (add class foo, rename function bar, move that code from x to z) instead of plain non-semantic text. More complicated to implement, certainly, but it would be really interesting to what new programming capabilities could arise then.

The real challenge is that this would require very tight integration between the programming language, the IDE and the source control system. It would not work with C++, but it might just be feasible with something like Go or C# (Roslyn!)

[OpenTK: C# OpenGL 4.4, OpenGL ES 3.0 and OpenAL 1.1. Now with Linux/KMS support!]

Ryan_001

3,477

September 11, 2014 01:25 PM

I remember someone bringing up an idea about storing code in some sort of loosely-defined database, rather than plain text files. This was in GD maybe a decade ago so good luck finding a reference to the old thread, but the idea does have some merit.

Source control could operate on individual items (add class foo, rename function bar, move that code from x to z) instead of plain non-semantic text. More complicated to implement, certainly, but it would be really interesting to what new programming capabilities could arise then.

The real challenge is that this would require very tight integration between the programming language, the IDE and the source control system. It would not work with C++, but it might just be feasible with something like Go or C# (Roslyn!)

That's the first thing I thought of as well.

Another thought, since we're throwing things out there, was to perhaps store it in some sort of AST format. In that format search for related code would be relatively trivial I imagine.

SmkViper

5,396

September 11, 2014 02:43 PM

Maybe I'm just getting tied up in the example provided by the OP with fireballs and such (which I agree with Sean, is not the correct thing to do in code, that's a data issue) but it seems to me that code is a collection of algorithms. You group together related ones (same class, same "system", whatever) close to eachother in some manner (maybe same file, maybe multiple files across a directory) to more easily see them side-by-side.

However if you're asking "I want to see all the code that executes when I cast a fireball" then... well... that's what a debugger is for, surely? You're asking a run-time question.

On the flipside, I think the IDE is the correct way to go about this and I could see someone making a tool where you could somehow input some parameters into a function, and then have it "run" the code, collecting all code hit into a giant pseudo-function that you could look at all at once. Heck, IDE and profiling tools already exist that can give you whole call trees which then let you jump to each function in the code.

Course, I have no idea how you'd even begin to edit the function from that standpoint...

In the end, code organization is something you have to decide for yourself. And whatever you pick will be ideal for task X, but will suck for task Y. You simply cannot organize code in such a manner that all possible queries on said codebase are represented. Even storing functions individually won't work because functions may have conditions which change what code inside them executes based on parameters - which means pulling the function in pulls in code you're not interested in because of pre-conditions.

Get a good IDE and organize your code for the most common use case. In my case, that is to group systems into folders and classes into single pairs of h/cpp files. Then let the IDE and other tools parse the code and provide you the data you want for less-common tasks like "what calls me?" or "who do I call?" or "If Bob is blue, then how does metal smell?"

_the_phantom_

11,263

September 11, 2014 03:17 PM

However if you're asking "I want to see all the code that executes when I cast a fireball" then... well... that's what a debugger is for, surely? You're asking a run-time question.

Not really; you come to a new system, you are trying to add 'something' to it but in order to do so you need to follow the flow of the code and this can get complicated FAST when moving across multiple dependent and independent files.

I've had to deal with this a lot recently and my current method of dealing with it is to use the 'pin tab' functionality of VS to track files I've been in (and if jumping among those files even that isn't great) as well as noting down the flow on paper... all of which is a bit of a faff.

Even a small example, adding ADPCM support for Android in UE4, resulted in my having four or five files pin'd while trying to track all the logic going on.

What would have made this easier would have been anything which could have integrated the code fragments into a single view (or a sub-set of them into a couple of views if that made sense), bonus points for being able to display call linkage either in the same view or in a 'map' view so I can jump around and see where things go to/come from.

So, in the sound case, I wanted to see how 'createbuffer' was called; my current solution was to use VAX to 'find references' and open the correct file from there. What would have been 'nicer' would be same initial flow ('find references') but instead of files the output would be a list of tagged versions so that I could say 'ahah! AndroidAudioDevice!', double click on it and get that function inserted below the current code (which I could then optionally drag/drop above so that the flow made more sense) so now I can see call site and destination in the same view.

By the end of this session I might have 7 or 8 functions in the chain on screen which would have meant having 4 or 5 files open but instead all viewable on screen in a single session.

----

This does feel like something which needs to be also tackled at the language level however; many existing languages rely on the concept of a file and file system structure to control compile order or dependency information which could cause problems and, ultimately, not make it useful for things like C or C++.

(Although, for languages like that you could maintain the existing file setup and have the IDE work magic to give you the new views and handle changing the original data when you update the functions).

Version control would either have to be based on 'fragments' too or you'd have to provide a compatibility layer so that while the fragment 'void foo()' doesn't sit in a file normally the comp layer shoves it into one for version tracking - however being able to track on the fragment level would probably be better, bonus points for allowing a checkin to reference multiple fragments; "changed Audio to support ADPCM" for example would reference the 7 or 8 fragments changed to support it which makes it clear what has changed in that check in.

I think the idea has merits, but you will need to build it in from Day 1 and it would require retooling for a few things as well as a fundamental shift in how people think about code and structure.

I could see it making life easier however when you are tasked with doing Things in a large code base.

ferrous

6,164

September 11, 2014 06:43 PM

There was a video floating about of a former Visual Studio developer that was writing an IDE that was similar to what was described. Instead of showing files, it would show functions, so as you followed a bit of code, like, say you were debugging, instead of showing you the file, it would just show you the function you're currently in, above it would be the function it was called from and all its code, etc. Basically, a very verbose callstack.

SmkViper

5,396

September 11, 2014 08:10 PM

However if you're asking "I want to see all the code that executes when I cast a fireball" then... well... that's what a debugger is for, surely? You're asking a run-time question.

Not really; you come to a new system, you are trying to add 'something' to it but in order to do so you need to follow the flow of the code and this can get complicated FAST when moving across multiple dependent and independent files.

True - but again, I think this is still more of a "let's write a tool that will do that" thing then a "how do I organize a file system which shouldn't have anything to do with my program but does because that's how the C compilation model works".

And again, whatever you do to organize will work well for a set of questions you want to ask of your code base, but not others, so you're going to at least want multiple views on the data depending on the work you're doing.

Washu

7,836

September 13, 2014 09:32 PM

Not really; you come to a new system, you are trying to add 'something' to it but in order to do so you need to follow the flow of the code and this can get complicated FAST when moving across multiple dependent and independent files.

I've had to deal with this a lot recently and my current method of dealing with it is to use the 'pin tab' functionality of VS to track files I've been in (and if jumping among those files even that isn't great) as well as noting down the flow on paper... all of which is a bit of a faff.

This is certainly an area where I can see something like tags being helpful. The ability to immediately query for something (assuming you know the tag), have a browser of tags, be able to search on tags, etc. could speed up the finding of these functional bits immensely faster. At the moment you're kind of stuck reading through a function and then "Go To Definition" on various calls or looking at the parent caller to find out where things are going and how the logic is flowing in order to make a change to a single piece of functionality.

Even a small example, adding ADPCM support for Android in UE4, resulted in my having four or five files pin'd while trying to track all the logic going on.

What would have made this easier would have been anything which could have integrated the code fragments into a single view (or a sub-set of them into a couple of views if that made sense), bonus points for being able to display call linkage either in the same view or in a 'map' view so I can jump around and see where things go to/come from.

So, in the sound case, I wanted to see how 'createbuffer' was called; my current solution was to use VAX to 'find references' and open the correct file from there. What would have been 'nicer' would be same initial flow ('find references') but instead of files the output would be a list of tagged versions so that I could say 'ahah! AndroidAudioDevice!', double click on it and get that function inserted below the current code (which I could then optionally drag/drop above so that the flow made more sense) so now I can see call site and destination in the same view.

By the end of this session I might have 7 or 8 functions in the chain on screen which would have meant having 4 or 5 files open but instead all viewable on screen in a single session.

Yes, I can see how this might work... use edit controls in a parent view. Have the edit controls be inline and expandable so that they're fairly transparent (makes the text area look like a single file) but with a header notation on each function/method/class/etc. that indicated its physical location. i.e. something like <tt>@/src/spells/elites/fierygreatsword/FieryRush.cs#433</tt>.

This does feel like something which needs to be also tackled at the language level however; many existing languages rely on the concept of a file and file system structure to control compile order or dependency information which could cause problems and, ultimately, not make it useful for things like C or C++.

(Although, for languages like that you could maintain the existing file setup and have the IDE work magic to give you the new views and handle changing the original data when you update the functions).

C# and Java already have the ability to annotate code with attributes, so you could simply implement a Tag attribute in C# (for example), and you've got a quarter of the problem solved. The other quarter is building the UI pieces into visual studio.

Version control would either have to be based on 'fragments' too or you'd have to provide a compatibility layer so that while the fragment 'void foo()' doesn't sit in a file normally the comp layer shoves it into one for version tracking - however being able to track on the fragment level would probably be better, bonus points for allowing a checkin to reference multiple fragments; "changed Audio to support ADPCM" for example would reference the 7 or 8 fragments changed to support it which makes it clear what has changed in that check in.

I think the idea has merits, but you will need to build it in from Day 1 and it would require retooling for a few things as well as a fundamental shift in how people think about code and structure.

I could see it making life easier however when you are tasked with doing Things in a large code base.

Probably a compatibility layer at first, so that you could have dual workflows. The tag based one as it was implemented and grew into maturity, and the default workflow people are currently used to.

I am surprised to see so little discussion on this, I think the idea is certainly worth exploring with current languages and we certainly have the capability to implement it now at least on some platforms...

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.

ApochPiQ

23,138

Author

September 13, 2014 11:15 PM

It's a very human trait to endure ridiculous amounts of inconvenience when that inconvenience is familiar and comfortable. Changing to a better - but different - workflow is often too scary to "justify" leaving behind the immense pain that must be endured in the known, "safe" workflow.

Frankly, though, I'm tired of the way code organization and exploration is forced to happen in contemporary languages. I'm going to solve this problem even if the majority of programmers are content to metaphorically punch themselves in the gonads all day. If grep and Find All References are good enough for someone, then so be it. They aren't good enough for me.

I think there are fundamentally several intertwined issues going on here that are worth considering:

Finding a home for newly written code
Viewing call graphs: how does code flow into and out of a given function?
Viewing related code, where "related" is an arbitrary concept/axis
Viewing structurally proximal code, i.e. stuff in a single file/module/unit of some kind
Treating related code as if it were proximal

It's starting to sound like the right direction is to leave the general file-based organizational hierarchy intact, but layer some kind of viewing filters on top of that. It would take serious IDE/compiler integration but I think it's totally doable especially in the context of a novel language.

I have a half-formed mental image of a "Build Code View" dialog that allows cherry-picking code chunks from a variety of sources:

A tree-view of directories, files, and code units within each file
A list of known tags and/or a tree-view of tag "regions" or "spaces"
Call graphs centered on arbitrary code units in the program

So I open up this thing, click on whatever chunks of code make sense, and I get a new Project tree entry called "Some View Foo" which contains exactly the code I selected. I'm not entirely sure what happens if I write new code into this view, but that seems like a minor question overall.

Then I can share my views with anyone else on the team, for example, and they can pop open a pre-selected perspective of the project that completely details, say, all the network messages transmitted by the system, or all the custom UI element implementations, or whatever.

I kind of like the idea of making views orthogonal to the file structure. It makes cooperating with version control a lot easier, although it doesn't get the cool benefits like function-level revision history. Maybe it's worth digging deeper to come up with a fundamental way to solve both issues, or maybe I should be happy to solve one thing at a time...

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

Washu

7,836

September 13, 2014 11:59 PM

It's a very human trait to endure ridiculous amounts of inconvenience when that inconvenience is familiar and comfortable. Changing to a better - but different - workflow is often too scary to "justify" leaving behind the immense pain that must be endured in the known, "safe" workflow.

Frankly, though, I'm tired of the way code organization and exploration is forced to happen in contemporary languages. I'm going to solve this problem even if the majority of programmers are content to metaphorically punch themselves in the gonads all day. If grep and Find All References are good enough for someone, then so be it. They aren't good enough for me.

Especially since Find All References doesn't work 100% even in managed language code bases.

I think there are fundamentally several intertwined issues going on here that are worth considering:
Finding a home for newly written code
Viewing call graphs: how does code flow into and out of a given function?
Viewing related code, where "related" is an arbitrary concept/axis
Viewing structurally proximal code, i.e. stuff in a single file/module/unit of some kind
Treating related code as if it were proximal
It's starting to sound like the right direction is to leave the general file-based organizational hierarchy intact, but layer some kind of viewing filters on top of that. It would take serious IDE/compiler integration but I think it's totally doable especially in the context of a novel language.

Well, the Visual Studio SDK is quite powerful, you could probably relatively quickly build a tag based plugin (assuming that's the route one went) for any of the managed languages probably with only a few hours of investment. But for a language like C++... I honestly don't see it happening.

I have a half-formed mental image of a "Build Code View" dialog that allows cherry-picking code chunks from a variety of sources:
A tree-view of directories, files, and code units within each file
A list of known tags and/or a tree-view of tag "regions" or "spaces"
Call graphs centered on arbitrary code units in the program
So I open up this thing, click on whatever chunks of code make sense, and I get a new Project tree entry called "Some View Foo" which contains exactly the code I selected. I'm not entirely sure what happens if I write new code into this view, but that seems like a minor question overall.

Then I can share my views with anyone else on the team, for example, and they can pop open a pre-selected perspective of the project that completely details, say, all the network messages transmitted by the system, or all the custom UI element implementations, or whatever.

I kind of like the idea of making views orthogonal to the file structure. It makes cooperating with version control a lot easier, although it doesn't get the cool benefits like function-level revision history. Maybe it's worth digging deeper to come up with a fundamental way to solve both issues, or maybe I should be happy to solve one thing at a time...

Code Views, like database views.

That would be much more doable in non-managed languages. As for the source control integration and version history: Why not? You can run blame on a single line of code, so no reason you couldn't run that blame against the file for the set of lines that makes up the function and simply cut out all lines that are not that function.

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.

Code organization without the limitations of files

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Code organization without the limitations of files

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines