Sign in to follow this  
shurcool

On Source Code, Design and Implementation

Recommended Posts

Today, the source code of a software project encodes *both* the design and the implementation of the app.

The design describes [b]how the app looks, behaves and what it does (in theory).[/b]

The implementation is nothing more than a way of [b]making the design a reality[/b] on a given platform. It uses known tools (CPU instructions arranged in a certain order, software libraries, programming languages, programming tools, etc.) to make the app real and working, rather than a design that is working only in someone's imagination.

Do you guys think it's a problem that source code encompasses both things? That they are inseparable, intertwined components?

I think so. Ideally, IMO, these two highly distinct concepts should be precisely that, distinct.

If I have a certain app that does something and looks a certain way, why should it matter what its implementation is? Why should it matter if I programmer A or programmer B wrote the implementation? Hell, the implementation might even be created by some sophisticated computer program.

The problem holding us back is... it's not easy. How do we represent design? Source code is a pretty good representation of implementation. But how do you represent design? It can reside in a designer's head, but that's not good enough. It can be scribbled on a piece of paper, or described in a lengthy design doc. But that's not good enough either, IMO. It has to be more dynamic, so that it can be linked up with a matching implementation and run on a computer/mobile device near you.

These are the right questions that we should be asking and trying to answer... IMO. Edited by shurcool

Share this post


Link to post
Share on other sites
Well I was going to "Report" your thread, so it can be moved to the General Programming forum. However, it seems that the "Report" function is for thread that are truly negative. With that said, this is not a Lounge question, but a General Programming forum question. And should be moved to that forum, where it would be better served.

Share this post


Link to post
Share on other sites
One could argue that the UI is the design and the code-behind (web or desktop) is the implementation of the design. You have software such as Windows Workflow Foundation that that tries to meld the design with the implementation. I'm sure by now that there is software that ties UML designs to the code itself. Basically presents a graphical tree/chart of your code and its flow. But it sounds like what you are describing is something similar to MVC (or similar paradigms).

Share this post


Link to post
Share on other sites
Fair enough. I didn't post it there because I thought it was too abstract for a forum like General Programming where real-world problems are being discussed, but I do want a programming-friendly audience to see it (hence GameDev.net).

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343008198' post='4962106']
Fair enough. I didn't post it there because I thought it was too abstract for a forum like General Programming where real-world problems are being discussed, but I do want a programming-friendly audience to see it (hence GameDev.net).
[/quote]
No, this is definitely General Programming forum material. That forum now handles Software Engineering and Practical Programming issues.

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343002840' post='4962087']
Today, the source code of a software project encodes *both* the design and the implementation of the app.
[/quote]
First, let's clarify exactly what you mean be "design," as that term can apply to a lot of things. Is the the UI design? Is it the workflow design? Is it the architectural design of the code? etc.

I'm going to assume you are aiming more at the UI design. I'd go with what Alpha_ProgDes said and say that what you want is probably something like a [url="http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller"]model-view-controller[/url] (MVC) design. It's entirely possible (and common) to design the UI (the view) using WYSIWYG or markup tools (i.e. without programming anything), implement a functional GUI-less back end that actually does all the work (the model) in some programming language, and glue the view and model together with a bit of code (the controller).

Share this post


Link to post
Share on other sites
There have been several attempts to address this, with varying degrees of success. Most notable is obviously the web model.

And other technologies also follow the markup model, such as WPF/silverlight, XUL and QML.

Share this post


Link to post
Share on other sites
[quote]If I have a certain app that does something and looks a certain way, why should it matter what its implementation is? Why should it matter if I programmer A or programmer B wrote the implementation? [b]Hell, the implementation might even be created by some sophisticated computer program.[/b][/quote]

You mean like how compilers generates machine code from human-readable source code? ;) In all seriousness, that's exactly what the whole concept of the separation between source code and machine code is for - just at a lower level than you're talking about.

So, what you're essentially proposing is some way to take the "design" of a program and produce a program from that, without worrying about what the implementation actually is, correct? Let's pretend that you've created some sophisticated design tool that lets designers specify a "design" and lets the computer figure out the "implementation". Since the "design" is now a direct specification of the program's look and behaviour, your "design" has now become the "implementation." So we're back at the beginning again.

I would say that if you're creating some sort of mechanism by which the design of a program is codified (which in my view is all that separates "design" from "ideas"), and the level of codification is sufficient that the machine can generate an implementation of the design for you, then all you're doing is creating a very high level programming language. In fact, I would say that a sufficiently formalized language intended for the design and specification of behaviour is the very [i]definition[/i] of a programming language, no matter what medium it uses (textual in the case of traditional programming languages) and no matter what the compiled result is. Consequently, any design codified in that language is by definition "source code." Edited by Oberon_Command

Share this post


Link to post
Share on other sites
[quote name='Cornstalks' timestamp='1343009420' post='4962111']
[quote name='shurcool' timestamp='1343002840' post='4962087']
Today, the source code of a software project encodes *both* the design and the implementation of the app.
[/quote]
First, let's clarify exactly what you mean be "design," as that term can apply to a lot of things. Is the the UI design? Is it the workflow design? Is it the architectural design of the code? etc.[/quote]
By design I really mean everything about how the app looks, feels, and behaves. This includes the UI, of course, but also the behaviour. When you click on a button, or swipe something, the displayed pixels change in some fashion. This is all design in this context, basically everything that's "the front of the scene."

The implementation is everything that's behind the scenes, like the language that was used, the widget toolkit, the rendering API, the choice of sorting algorithm, etc.

What if we found some way to cleanly separate these two things, such that the implementation could be interchanged while keeping the design untouched. So a user wouldn't even notice any change from their perspective (aside from potentially increased performance, etc.).

One potential advantage of this is that the implementation stuff could be improved separately from the design, and reused across multiple projects. Kinda like libraries, I suppose.

Oberon_Command made some very good points. I have to keep thinking about this. [img]http://public.gamedev.net//public/style_emoticons/default/smile.png[/img]

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343016419' post='4962128']
What if we found some way to cleanly separate these two things, such that the implementation could be interchanged while keeping the design untouched. So a user wouldn't even notice any change from their perspective (aside from potentially increased performance, etc.).
[/quote]
That's what the MVC architecture is all about. I don't know if I can go into much detail or not, so I won't go into too much, but take the company I work for, Sorenson Media, and our flagship product Squeeze, a video transcoding program. It's no secret we use Qt, but the key is that we are able to separate the GUI (everything you see and work with) from the transcoding engine (everything you don't see that actually transcodes the video). We can design the GUI, workflow, etc. in Qt designer. We can develop a separate GUI-less transcoding engine. Then it's just a matter of writing a little bit of glue code to connect the GUI to the transcoding engine, and walla, you have Squeeze Desktop. We just released version 8.5 recently, and while the GUI didn't change much, there were some significant backend changes made in the compression engine that allowed it to run up to 3x faster with some things. A user wouldn't realize the significant changes, because what they see and interact with is mostly the same. Alternatively, there were a few GUI and workflow changes that didn't require us to rewrite the engine, as they're separated.

Another benefit is that we can use the compression engine in other products of ours, like our cloud server, without booting up a GUI version of Squeeze Desktop.

Separating the front-end (what the user sees and interacts with) and the back-end (what actually does the work) is generally good practice, and there are several patterns and design architectures created specifically with that goal in mind. ChaosEngine noted some of the technologies that strive to make this easy.

I don't know how familiar you are with the MVC software design, but if you're not too familiar with it I suggest trying to get familiar with it by creating a program with C# + WPF or C++ + Qt (or whatever combo you may choose).

I think the really interesting question is what alternatives to the MVC software design exist, and what are good practices for separating the front-end from the back-end.

I suppose how you represent and design the front-end is a good enough question though. My answer would be: we can represent design with flowcharts documenting workflows, WYSIWYG editors for actual design and creating the GUI, markup languages for defining layout, etc. Then it's just a matter of gluing this to the code (in Qt, this is done with signals and slots), more or less.

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343016419' post='4962128']
By design I really mean everything about how the app looks, feels, and behaves. This includes the UI, of course, but also the behaviour. When you click on a button, or swipe something, the displayed pixels change in some fashion. This is all design in this context, basically everything that's "the front of the scene."
[/quote]
Well you are delving into the realm of MVVM which handles things in alot of ways similar to what you are describing. Sounds like you want a Designer tool program (but with far more capability than what we have today). You should take a look at Microsoft's Expression Blend (for use with Silverlight and WPF both MVVM platforms). It does some pretty cool things. Edited by Alpha_ProgDes

Share this post


Link to post
Share on other sites
Thanks for the insightful replies guys, I've got some thinking to do about this. Need to make my ideas more concrete.

Share this post


Link to post
Share on other sites
On my [url="https://plus.google.com/106017796906508928693/posts/MYjHKThichK"]Google+ post[/url], someone asked me to describe a sample scenario of how this approach might be used. Here's what I wrote:

[quote name='shurcool']This is somewhat off the top of my head, but here's a hypothetical sample scenario of how this approach might be used to create a new app by an app designer.

I think the best metaphor to describe the process would be similar to a sculptor sculpting a shape out of some material. He makes changes to it until it looks like his target object.

You start by creating a new app template project with a window. The app is running so you can see how it looks like and make changes to it. You decide the default background color is no good and change it to be green (via some interface, suppose a voice recognition engine). Then you add 2 buttons and describe what clicking on each one does.

Assuming the implementation for all the features you request is already present and is found, you don't have to write any code. If not, you do.

The difference between this approach and what you can already create using Visual Studio or Xcode is that for a designer to create the above program it would take 3-5 minutes and it doesn't require any coding from his part. The coding gets outsourced to coders who write it from scratch or find existing work in a library. The designer doesn't have to know or go through the API to make changes to his app, he just describes them in natural language (or via drawings, or some other means).

I'm really making this up as I go, so it is what it is. [img]http://public.gamedev.net//public/style_emoticons/default/biggrin.png[/img] Perhaps the vision described here is too hard to create today, but I care more about whether the vision is good and as long as it can be created eventually... In other words, I'm working on the design first (before implementation). ;)

Edit: I realize the above scenario is an oversimplification... It describes a very static app. What makes most apps hard to code is their highly dynamic behaviour. What happens when a button is pressed might vary wildly depending on a complex state. Perhaps these difficulties can be overcome by having the system interpolate/fill in the blanks/guess the missing behaviour (and the app designer can correct it if it's wrong).[/quote]

Share this post


Link to post
Share on other sites
Well, there is the whole field of Software Architecture in academia that tries to deal with this [img]http://public.gamedev.net//public/style_emoticons/default/tongue.png[/img]

Let me give you a few papers you might be interested in. First, the discipline was founded by Perry & Wolf in [url="http://dl.acm.org/citation.cfm?doid=141874.141884"]Foundations for the study of software architecture[/url]. There are some interesting concepts, like views ([url="http://dl.acm.org/citation.cfm?doid=216591.216611"]4+1[/url] seems to be the most popular), styles (and older [url="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=566427"]classificaiton[/url]), and ADLs (yet [url="http://dl.acm.org/citation.cfm?id=331527"]another popular classification[/url]). You briefly touch on [url="http://www.omg.org/cgi-bin/doc?omg/03-06-01.pdf"]what[/url] [url="http://www.omg.org/mda/"]model-driven[/url]development is - which is implemented by [url="http://www.eclipse.org/modeling/emf/"]EMF[/url]and [url="http://wiki.eclipse.org/Eclipse4/RCP/Architectural_Overview"]used in eclipse 4[/url].

Now, most of these papers are paid. If you are in university, chances are you can get them for free via institutional login. Now let me expand a little bit on this [img]http://public.gamedev.net//public/style_emoticons/default/smile.png[/img] Most of the links will be to the free Wikipedia. The [url="http://en.wikipedia.org/wiki/Software_architecture"]introductory article on software architecture[/url] seems to give a good overview on the topic, with links to free content.

If you study SA or Software Engineering (SE), chances are, you will be amazed how much different academic and industrial approaches are. I sometimes feel that scientists are like children playing in the sand (assuming a perfect world), but that is OK I guess.

Let me start first with what you call separation of "design" and "implementation". There has been a lot of work on that. One of the accepted ideas is that you can attack the problem by explicitly describing your architecture. You do that in an [url="http://en.wikipedia.org/wiki/Architecture_description_language"]Architecture Description Language[/url]. They differ in shape and form, but most commonly there is a high-level programming language, which can describe the basic architectural items: components and connectors. Ideally, they would also support describing varying styles and/or "dynamic" vs "static" architectures. Then you somehow invoke a compiler that verifies the syntax, and runs some validation checks to confirm that there are no logical problems with your architecture. Since ADLs are formal languages, formal analysis can be performed on the model, thus possibly discovering race conditions or other conflicts.

Now, there are different opinions as to what follows ADLs, or how they should be implemented.

One view is that you should couple architecture and implementation. You design the architecture, and then write the implementation. You can have run-time and compile-time checks for architectural conformance. This makes certain that the two are kept in synch, as they reside in the same codebase, thus share compiler and runtime environment. The most notable example in academia for this is called ArchJava ([url="http://www.cs.cmu.edu/~garlan/17811/Readings/aldrich-icse02.pdf"]paper[/url] (free), [url="http://archjava.fluid.cs.cmu.edu/"]web site[/url]), which is specific to to Java, but there are attempts to port it for other languages.

Another view is that you should keep them separate. You then generate your code (or at least a skeleton) from the architectural model. An expansion of that view is Object Management Group (OMG)'s Model Driven Architecture (MDA). These are the same guys that did UML and CORBA. You design your application, and then a tool, through a series of transformations and using predefined templates, generates the source code for you. Very notable example here is EMF. It seems that IBM favors this approach, as it is used in eclipse - a real-world piece of software - that many people use on a daily basis. I never thought anybody will go that route, but it seems that if there is will, there is a way. I guess this comes somewhat close to what the thread opener had in mind.

Yet another view on ADLs is that you should keep them separate, and limit their use to just checking the saneness of your model, making sure you don't make trivial mistakes. The idea is to take advantage of more accessible form of formal methods. The new wave of ADLs, most notably byADL ([url="http://dl.acm.org/citation.cfm?doid=1806799.1806816"]paper[/url] - paid, [url="http://byadl.di.univaq.it/"]site[/url]), focus on you creating DSL specific to your project or area (e.g. web programming). The advantage is that it won't force you to describe your architecture in terms that don't make sense, and it will cover your specific needs.

What many people seem to be getting at is called [url="http://en.wikipedia.org/wiki/Software_architecture#Examples_of_architectural_styles_and_patterns"]architectural styles and patterns[/url] in academia. The waters are murky as to what is the difference between styles and patterns, as some authors say they are mostly interchangeable, while others point out that patters are bigger in terms of scale and abstraction. Anyhow, MVC is sometimes classified as a style, and as is [i]probably [/i]the most well-known and widely used style in programming. Some people mentioned MVVC, which is most notably featured in WPF/Silverlight.

Now, as to my own opinion on the subject... I just don't like the idea of automatically generated code. I don't think it is reasonable to specify the architecture up-front (or to expect it to be correct). It usually evolves, and keeping some description up to date is IMO waste of time and effort better spent on another iteration of that evolution. There are many tools that automatically extract (UML) class diagrams from your code, so I don't see the benefit of yet another description. The main problem I see with the academic approach is that they mix high-level and low-level concepts. A lot. They work on tiny projects (tested on 20-30k LOC - that is less that what a cart + check out workflow is); work is done in unrealisic ways (some papers change the freaking code of the Java standard library - hello distribution nightmare), and plain childish attempts are made (look up AliasJava for an example of what I mean - unleash the complexity of C++ onto Java).

From my point of view, you can, and should, briefly document your architecture and doc-comment the code. In your nightly builds, build the automated documentation (javadoc for Java/sandcastle or ndoc for .NET) and generate a few automated class diagrams. That, plus a modern IDE, would be enough for a competent developer to cope with even quite big code bases.

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343016419' post='4962128']
By design I really mean everything about how the app looks, feels, and behaves. This includes the UI, of course, but also the behaviour. When you click on a button, or swipe something, the displayed pixels change in some fashion. This is all design in this context, basically everything that's "the front of the scene."
[/quote]
It looks to me like we are fighting a bit of a wordgame. Design and Implementation are both very broad "words", the distinction between design and implementation, in the way you seem to use it, is more subjective.


[quote name='shurcool' timestamp='1343016419' post='4962128']
The implementation is everything that's behind the scenes, like the language that was used, the widget toolkit, the rendering API, the choice of sorting algorithm, etc[/quote].
What algorithm you should use definitly belongs to the design.

Let's make an example: I need a program to manage roughly 1000 documents.
Now you have your visionary modeling language to design a program that manages an arbitrary amount of documents. And now an interpreter should "implement" your design to manage 1000 documents.
My friend needs a similar program to manage about 5 million documents, you would rerun your interpreter and he would have his program.

This cannot go well, you would use a different datastructure to hold the different amount documents, not even thinking about algorithms. You have two completely different problems to solve. In other words you need a different design if you want to make both people happy.


Design and implementation are two different things, but designing something without having an implementation in mind is not very useful, at least from my experience. I once had someone design a beautiful GUI Prototype in axure, this person had no idea about the implementation and it ended up being impossible to use.
I say your design will probably differ for your target implementation. Your program architecture should be different for using C or C++.

[quote name='shurcool' timestamp='1343016419' post='4962128']
What if we found some way to cleanly separate these two things, such that the implementation could be interchanged while keeping the design untouched. So a user wouldn't even notice any change from their perspective (aside from potentially increased performance, etc.).
[/quote]
Depending on your perspective, even the behaviour might be an "implementation". Sometimes all you care about is what goes in and what comes out, everything else is unimportant.

All benefits you described so far are achieved by good program architecture. MVC was mentioned and it is the most prominent, but there are more.


E: embarassing typeos removed Edited by Bluefirehawk

Share this post


Link to post
Share on other sites
How is this discussion?

That separating presentation from functionality is good has been a known thing in the business world for what? 30 years?

The closest thing you're likely looking for is modern CSS, and webdev in general. Depending on what platform I'm using, I get a different ESPN.com. Depending on what site I go to, I get google maps presented with different information. Many, many different designs provide me with information from the facebook API.

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343016419' post='4962128']
What if we found some way to cleanly separate these two things, such that the implementation could be interchanged while keeping the design untouched. So a user wouldn't even notice any change from their perspective (aside from potentially increased performance, etc.).
[/quote]

As others have mentioned, what you're describing more or less is the MVC pattern. However, I think most definitions of MVC are either flawed or incomplete. Conceptually, a View layer is purely for presenting and user interaction, but too often 'back-end' details end up bleeding into the UI (in my experience, anyway). Also, I think most people get too hung up on abstract conflicts such as: Does this functionality belong in the model or the controller? Should I have a skinny model or a fat model? Etc.

Like I said, it seems incomplete. In order to make the idea of MVC useful, in my opinion, the UI should not be treated as not just a separate 'component' of the application, but rather a separate application unto itself which communicates with a back-end application via a clearly defined 'front-end' API. Conceptually, this results in leaner applications which are easier to reason about.

Share this post


Link to post
Share on other sites
[quote name='Alpha_ProgDes' timestamp='1343555473' post='4964227']
@shurcool: Did we answer your question? Or help you arrive at an answer? Just wondering.
[/quote]
Alpha_ProgDes, this thread has been extremely insightful and helpful to me. I wouldn't say it has "answered" the question, simply because the question is too broad to be completely answered.

That said, I wanted to add something.

Some people have suggested that basically what I'm describing is a "high-level language". Yes and no. I think the key difference between my pitched idea and a typical high-level language is that in a HL language, you don't really have control over how something happens on the low-level. You just say, "get this done" and hope the language figures it out in a way that's appropriate.

Sometimes this works, if the compiler/language is smart, and if you don't really care how something gets done.

However, with my approach, you can explicitly specify the low-level implementation for any high-level construct, if you wish (sometimes, you have no choice but to do it). "Get this high-level thing done, and do it this way."

Does that offer an advantage over the standard HL languages where you have no such control?

Share this post


Link to post
Share on other sites
[quote name='shurcool' timestamp='1343755898' post='4964927']
Some people have suggested that basically what I'm describing is a "high-level language". Yes and no. I think the key difference between my pitched idea and a typical high-level language is that in a HL language, you don't really have control over how something happens on the low-level. You just say, "get this done" and hope the language figures it out in a way that's appropriate.

Sometimes this works, if the compiler/language is smart, and if you don't really care how something gets done.

However, with my approach, you can explicitly specify the low-level implementation for any high-level construct, if you wish (sometimes, you have no choice but to do it). "Get this high-level thing done, and do it this way."[/quote]

What you're describing is broadly similar to the idea of using inline assembly in C or C++, which is something games in particular used to do a lot back in the days of 16-bit DOS. As in that scenario, I would argue that the high-level and low-level aspects of the language are actually using two separate languages with different idioms and ways of approaching the problem.

[quote]Does that offer an advantage over the standard HL languages where you have no such control?[/quote]

For what you're trying to do, not really. Consider that the objection in the OP to the idea of source code was that design and implementation should be distinct. If you're "falling back to the lower levels" to implement a particular feature, then you're blurring the line between the high- and low-levels - between the [i]design[/i] and the [i]implementation[/i], which is exactly the opposite of what you're trying to enforce. If it becomes idiomatic to specify the low-level implementation for a high-level construct in any given program (and for performance-critical programs like games, it will), then idiomatically the "design" and "implementation" will not be distinct the way you want, which means that practically speaking your language will offer no advantages over the existing solutions you object to in the OP.

Share this post


Link to post
Share on other sites
[quote name='Oberon_Command' timestamp='1343757184' post='4964930']
What you're describing is broadly similar to the idea of using inline assembly in C or C++, which is something games in particular used to do a lot back in the days of 16-bit DOS. As in that scenario, I would argue that the high-level and low-level aspects of the language are actually using two separate languages with different idioms and ways of approaching the problem.[/quote]

Functionally, it would be quite similar, yes. But in terms of usability (for the programmer), I don't think there's much similarity... Sorry, it's hard for me to elaborate right now.

[quote name='Oberon_Command']
For what you're trying to do, not really. Consider that the objection in the OP to the idea of source code was that design and implementation should be distinct. If you're "falling back to the lower levels" to implement a particular feature, then you're blurring the line between the high- and low-levels - between the [i]design[/i] and the [i]implementation[/i], which is exactly the opposite of what you're trying to enforce. If it becomes idiomatic to specify the low-level implementation for a high-level construct in any given program (and for performance-critical programs like games, it will), then idiomatically the "design" and "implementation" will not be distinct the way you want, which means that practically speaking your language will offer no advantages over the existing solutions you object to in the OP.
[/quote]

When I talk about high-level and low-level constructs, I am referring to the implementation aspect only. A high-level line of code might be "sort these numbers" while a low-level one might be the C++ code to do bubble sort.

I think it's best to compare the high-level functions to an API of a standard library, while the low-level stuff would be the implementation of said functions.

Another way to look at this might be to create an interface that allows to easily move things into a library (which is easily shared across projects/developers). Right now, creating a library involves a lot of overhead work, making it easier to simply leave any potentially-library-level code as baked into your current project, rather than extracting it for better reuse.

On another hand, I'm starting to realize that if you're really creating a general purpose program, it might be hard to separate the design from the implementation, and source code might be a good way to represent design after all. It's not the same if you're creating a real-time simulation (a game) vs. a GUI app.

P.S. Please excuse me if I'm starting to say really stupid things, as this whole thing is becoming a mess in my head and I'm working on figuring things out. Part of the problem is I had to take a break from this for a while, and by now I've forgotten the exact context of this thread (plus it probably changed direction a bit by now)... Edited by shurcool

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this