Introduction This topic was started as a way to discuss a practical, real language that will aid in solving real programming challenges. Some of these challenges are already old and well-understood in the field. Others, like multiprocessing, are partly understood, but will become increasingly more important in the near future. This discussion arose from the observation that currently popular languages (an oft-mentioned culprit being C++) do not solve these needs well. Other languages may address these needs in very elegant and effective ways, and yet these languages have failed to achieve mainstream usage in most problem domains. Our goal is to critically analyze the problems that we face, the solutions to these problems, and practical ways to integrate those solutions into a new language. Our goal is not to develop the "perfect" language, or to build anyone's "dream language." Rather, we accept the reality that any language will have to have some ugly, rough edges. At some point, a language must disappoint someone. We are not here to make the ultimate language for everything. However, we do believe that a language can be developed which is both a major improvement on existing tools, and is also suitable for a very wide variety of problem domains. Whereas many highly elegant languages are, realistically speaking, only used in narrow problem spaces, we intend to devise an elegant, rich, and powerful language that is suitable for many problem domains, and is appealing to programmers working in those domains. (Note: this post has been modified to clarify and better express our plans. It has changed with the course of discussion to reflect the changing vision of the language and its goals. I fully expect it to continue to change more over time, as described below in the Philosophical Goals.) Philosophical Goals These are the cornerstones of our effort. These ideals guide our planning and discussion. They are intended to provide a sort of shibboleth by which we decide which things are good for our language, and which are not. If a decision is to be made regarding the language, it should be made in light of these principles.

• Seek a careful balance between pragmatism and idealism. By nature, our investigation is seeking to apply the best knowledge of programming theory to the domain of real-world problems. We are interested above all in a language that effectively aids programmers in accomplishing their work, be it academic research, embedded applications, business logic, or games. While pragmatic concerns are intensely important to our efforts, it is also vital to remain grounded in solid theory. We seek not just an elegant and self-consistent design, but also a design that gives us powerful tools to accomplish the creation of software.
• Produce a real language for real work. At all times we should be mindful of the fact that we are not engaging merely in a mental exercise. We do not seek to produce an idealistic language that exists only in specification papers and committee meetings. Instead, we aim to grow a language organically from the experience, insight, and desires of real programmers who work on real code. This language is being designed with the very real goal of implementing it and using it for doing work. Without this emphasis, our endeavor may be mentally stimulating, but will ultimately not be useful. Driving for an actual realization of our ideals is vital.
• Use the language early, use the language often. As noted, one of our goals is to produce a real language that is viable for real work. Therefore, it is important that as often as possible we test our designs by doing real work in the language. During the early stages we can expect the language to change rapidly; however, even once this pace slows later in the language's life, it is vitally important to make sure that the design remains consistent with real usage. It is in our best interests to seek to implement the language as quickly as possible, and use the implementation heavily. By using our own creation, we ensure that our designs are truly good. Moreover, we offer ourselves the opportunity to refine our designs based on real feedback from real use of the language. We must truly eat our own dogfood, and we must be prepared to adjust the formula to taste.
• Provide the tools to develop abstraction and express it clearly. One of the fundamental aims of this project is to assist programmers in clearly communicating their intentions to the machine. It is unfortunately common with popular language tools today for a programmer to "know" much more about the semantics and expectations of a system than he is able to tell the compiler. We aim to reduce this gap in knowledge, and give the programmer tools and a vocabulary by which to express abstract notions precisely and thoroughly.
• Flexibility is paramount: assume the programmer knows his needs well. It is important that we retain a degree of flexibility. Despite its flaws, C++ has remained a powerful force in the programming world because it permits the programmer a very high degree of control. We must be willing to trust programmers with this control, because in many problem domains, such control is important. The language should exist to support the programmer's efforts, not constrain or limit them.
• Reliability is paramount: the programmer is fallible. This is a careful balance with the previous goal. While flexibility is important, safety is also equally important. Productivity comes by removing menial concerns. Abstraction and encapsulation aid productivity by removing the concern of duplicating technical and implementation details. Garbage collection aids productivity by removing the concern of memory management. Wherever possible, we should seek to increase productivity by removing the mundane technical concerns of computing, and allow the programmer to focus on solving the problems of his domain. However, it is important to permit the programmer to be concerned with these things, should the need arise.
• Build what is needed. Build tools to develop the rest. Feature creep and bloat are very real dangers. We must avoid the temptation to add every possible feature under the sun. Rather, we should seek to provide a powerful toolkit by which programmers can assemble a library of useful features themselves. Moreover, where it is practical to do so, we should permit programmers to bend and modify these features to suit their individual needs. Users of the language should be encouraged to develop reusable abstractions and share them with the community as a whole. Some fundamental features should be codified as "standard" elements of either the language or the core library of features that ships with a compliant implementation of the language.
• Favor programmer productivity and efficiency, even if it makes implementation of language features difficult. Language features only need to be implemented once. After that, they may be used freely. We should strive to implement useful, powerful, and elegant features, even if they are hard or expensive. Our investment of time and effort in developing the language's features will pay itself back many times over in saved time and effort by the users of the language - which, hopefully, should someday include us as well.

Practical Challenges Since we are aiming to develop a real language for use in real work by real people, we must remain mindful of the real issues that we face. There are challenges all along the way that must be addressed and considered if we are to succeed.

• Target Audience We cannot be all languages to all programmers. Such is not practically - if even theoretically - possible. However, we seek to provide sufficient flexibility that the resulting language is useful for a wide variety of people. We aim to allow programmers who are experienced with traditional imperative languages to transition easily and comfortably into this language. We aim to permit functional programmers to use their preferred style in order to gain the advantages offered by the functional approach. We aim to permit a wide range of development paradigms and philosophies while encouraging and facilitiating safe, reliable, and productive practices. Ideally, we would like to provide the tools for programmers to work close to hardware, in domains where this is important (embedded systems, realtime systems such as 3D games, etc.) while also offering a rich library of abstractions, so that the language is useful in higher level logic domains (business logic, web logic, simulation logic, etc.). Where existing languages succeed in crossing many paradigms and methodologies, we seek to cross many levels of abstraction; it is our belief that by offering a unified toolkit for developing on many such levels, each level may benefit by being coherent and consistently expressed with the other levels.
• Platform Support For better or worse, a programming language will not survive as a viable tool without interoperability with the Windows platform. The ability to interoperate with the family of Unixes is also important, although to a slightly lesser degree. A central goal should be interop with the Win32 API. Ideally, a layer for interacting with the .Net platform should also be developed as quickly as is practical. The language must facilitate the development of interop layers with the Unixes as well.
• Execution Model Agnosticism The language should be permitted to execute under many implementations: interpreters, compiled bytecode, compiled machine code, et. al. This allows us to use and develop the language early on (since interpreters and bytecode systems are fairly easy to build) while allowing us to eventually target hardware directly (compiling to machine code). This ability is important, because assuming the existence of a VM layer or interpreter violates the usefulness of programming close to hardware. Additionally, ability to compile to machine code (or possibly widely adopted VM formats such as MSIL) is critical for performance, since it is doubtful that we will be able to build a VM system that can rival commercial products.
• Early and Widespread Buy-in Just as we must test and use the language while it is being developed, we must also provide incentives for others to do so as well. Enlarging the pool of users and spreading the word is critical. Without widespread support and interest, the language will not be able to compete with established technologies. Achieving this buy-in requires that we be able to offer people the ability to use the language early on. It is vital that, as soon as possible, programmers are able to start writing real, useful code in the language, even if it is just interpreted or bytecode compiled.
• Growth by Supplementing, Not Replacing, Existing Tools Realistically, the "big language" to compete with in terms of buy-in is C++. Other languages are also important in various domains, but for Windows applications and games in particular, C++ continues to be the dominant player. This is primarily due to inertia: huge amounts of time and money are invested in existing C++ systems, making it impractical to replace them with systems in a new language. We cannot hope to gain adoption by demanding that people discard C++ and use this language instead. Rather, we must gain adoption by supplementing C++. Interoperability with C-style APIs is mandatory. Providing the tools to build binding layers with C++ systems (similar to embedded-language projects like boost::python) is essential. Programmers must be able to take this new language and add it to their toolset slowly, at a pace which is practical and comfortable for them. We cannot replace the existing Big Players overnight, because the inertia of existing code is too large. However, with careful planning, we can make it more cost-effective to use this new language for new code, rather than writing new code in old languages. Over time this will amount to a gradual shift in language emphasis. During this transition, though, it is important to emphasize compatibility - this language must be usable in concert with existing tools like C++. Demanding a full replacement of existing code will not work.

Language Elements and Features This is an overview of the features and capabilities that we have chosen to build into the language.

• Favor high degrees of abstraction and transparency
• Permit low edgrees of abstraction and allow technical concreteness (programming close to the hardware)
• Maximize the ability to express behavioral semantics of a system, as well as the data in that system
• Provide rich introspection and metaprogramming capabilities
• Encourage the use of nested abstraction layers to build domain-specific "dialects" on top the base language, via metaprogramming

Type System This section deals with the type system of the language: its nature, features, and limitations.

• Employ static typing and strong static-time checking of types.
• Encourage the definition of abstract datatype semantics, and methods for converting between semantic types.
• Provide rich features for dimensional analysis.
• Semantic datatypes are defined by the set of values they may store. A variable of a given semantic type may not be assigned a value which is not in its set. Doing so should cause a static, compile-time error (if possible), a warning if the behavior is not guaranteed to be reliable based on the defined semantics of the type, and a run-time error should an invalid assignment be attempted.
• Semantic type validation is performed by a boolean validation routine that returns either "true" for valid data or "false" for invalid data. This is done instead of explicitly defining semantics in the language syntax to allow for maximum flexibility in the way validation is performed. Validation routines are encouraged to be kept simple so that they can be checked at compile time. To help enforce this, validation routines are not permitted to interact with data outside the value being validated. This ensures that validation can be checked statically.
• Support for generic values is accomplished via a subset hierarchy. At the root of this hierarchy is the notion of Anything. All language concepts are a type of Anything (this is done to allow for higher-order functions, introspection, and other metaprogramming capabilities). Various data concepts express restrictions over the broad Anything. Such restrictions include numeric types, container types, and (eventually) objects. In addition, individual restrictions may be further subdivided by semantic types. For instance, numeric types can be split into integers, reals, complex numbers, and so on. A type may implicitly take the place of any superset of its own semantic set; an Integer is always substitutable for a Number, and a Number is always substitutable for Anything. Types may not be substituted across the hierarchy to sibling types, or down the hierarchy to child types, unless a specific semantic conversion function has been defined for the particular conversion being attempted.
• Support type inference as much as possible. Whenever a type can be statically inferred, this should be done automatically by the language. The simple rule of thumb for the compiler is "infer what I can; complain about what I cannot." If a type cannot be inferred, it must be explicitly specified by the programmer.

Additional points will be added as final conclusions are reached in the discussion. The Current State of Affairs I have personally taken on the responsibility of bootstrapping the language and delivering an implementation as quickly as my limited spare time allows. A running commentary of my progress is available via my journal, which can be reached by clicking the Journal link below this post. When major events in Epoch's development arise, I will drop a post in here. For the minor bits, keep an eye on my journal. In the meantime, everyone please feel free to continue discussion and making suggestions here. Optional, But Encouraged Reading The best starting place is to read this thread itself, as it contains the vast bulk of the history of the discussion, and all of the important conclusions and issues are at least mentioned here. This rest of this is just some background material to help illuminate the train of thought that started this project. With the exception of Tim's presentation, don't take any of this as the word of an expert. Heck, don't even take it as the word of anyone who has the first clue what he's babbling on about [wink] The Pragmatist's Programming Language Wishlist Additional explanations and thoughts Tim Sweeney's presentation on languages (PDF) An early concept for abstraction layering Some early thoughts of mine on dialectic language families Some early thoughts on integrating expressivity of abstraction and automated testing My thoughts on domain-specific languages and the need for expressivity [Edited by - ApochPiQ on March 10, 2008 5:38:52 PM]

Hmm, not really what I was driving for. What I'd like to see is a sort of "default mode" where garbage collection, abstract numeric types, and other such features are normal. In other words, it should take no effort for me to write a program that is garbage collected, uses transparent type promotion to avoid overflow, checks array boundaries automatically, etc. etc. Those things are extremely important to having good abstraction capabilities in a language, and should not be neglected in any way.

My contention, though, is that these features need to be permitted to co-exist with the capability of working with a high degree of concreteness. It should take extra effort to specify technical limitations like 32 bits of storage for a numeric type - but not too much extra. It is simply important, in my mind, that those capabilities exist.

A language is only as portable as its most expressive level of concreteness. PHP has no technical semantics, and as such is dependent on its execution engine to handle those semantics. Therefore, PHP will never be used for embedded logic development, or doing low-level video hardware interactions. My emphasis on the concreteness aspect is primarily borne from the pragmatic realm of 3D game engines; writing a 3D engine effectively, especially a cutting-edge one, requires concreteness. That's why I think C++ continues to be so popular in game development - it offers such concreteness.

Direct hardware interaction is important also because it allows for portability, as outlined in my second journal entry. Think of an OS built in Foo; the lowest-level, hardware-interaction elements can be done nicely in Foo, without sacrificing any of the expressivity and contractual safety nets, while upper layers can be built entirely in abstract onion-shell combinations of functionality. IMHO the only real hope for native OS "awareness" of concepts like JIT execution models and global garbage collection is to have such interoperability between layers of abstraction, within a single language family.



I definitely agree on metaprogramming. I've been pondering the concept of building "objects" not based on static, code-time classes, but with a more dynamic and fluid structure. This has been inspired largely by evolutional's experiments in tag-oriented data organization; the main challenge I see at this point is figuring out the logical extension of that sort of paradigm to logic organization as well. But in general, yes, introspection and metaprogramming are extremely important. Foo should let me build entire syntactic subdialects for domain-specific purposes.

We don't need a silver-bullet paradigm or a silver-bullet "reinvention" of OO; I don't think such a thing exists. We need a way to express concepts like OO at a fundamental level, dynamically, as you've described.

Expressing range and validation constraints on data types is very important, I think.

I'd take it a step further, though. First of all, I think data type restrictions should be explicitly stated as formal declarations. Having the restrictions inline in every single variable declaration is going to get messy, and is counterproductive if you ever need to tweak the restrictions. Secondly, I think there should be an implicit concept of "validation" in each data type:

DataType oddnumber Basis:Integers Validation(x):{ x % 2 == 1 };

Of course if complex validation is not needed it can be omitted. Such validation should be done as much as possible at compile-time (looking for potential conflicts, etc.) and have a well-defined model for validation at run-time. Off the top of my head, the easiest way to do this is to have validation occur as a pre-copy step when evaluating an assignment. For instance:

// Given some function Foo()
oddnumber x = Foo();

Foo() is first invoked and its return value obtained. Then, the return is passed to the validator of oddnumber, which is a magical boolean function or something. The validator detects that Foo() has returned an even number, and a run-time error is raised to signal the offense.

At compile-time, if Foo() is simply flagged as returning "Integer" rather than an explicit oddnumber, the compiler should raise a warning that Foo() cannot be guaranteed to return compatible data. Conversions between user-defined data semantic types should basically follow traditional conversion-warning rules from languages like C++ (where, for instance, assigning an int into a bool makes compilers grumpy).



Now of course there is the concept of introspection and metaprogramming, where it becomes important to be able to query a user-defined semantic type about its nature. For instance, I should be able to write code like:

if(2 IsValid oddnumber)
message("barf!");

This has two main benefits. First, and most obviously, it allows for easy testing of the semantics and validation of types. Secondly, it allows the program logic itself to adapt at run-time to the capabilities of data types. Since all programs in the FooLanguage express semantics in the same way, it should then be possible to transparently marshal data between programs without needing explicit converter code for each combination of two interacting programs. There's also some cool metaprogramming stuff that can be done with types introspection; I'll have to poke my brain a bit and see if I can come up with a good example.


Side node on syntax
I don't really like the C/C++ syntax format. In fact, I don't like the syntax philosophy of most languages. The philosophy tends to seem like "memorize this set of magical symbols and words, and then figure out how to make them interact." Inconsistent semantics and quirks make it hard to memorize what certain magic characters are for. (Exhibit A: the eternally annoying angle-bracket semantics in C++)

Personally, if I'm going to go to all the effort to construct a language from scratch, I'm going to make an attempt at a consistent, logical, and readable syntax. A beginner coming into the language should be able to figure out the bulk of the syntax elements just from their names. We've been pushing the merits of self-descriptive identifiers in code for years... now let's take it the next logical step and start looking at self-descriptive syntax, too.

I can't speak for anyone else, but I'd much rather exchange some verbosity and a little extra typing for extremely clear code.


To pick on your examples, I don't agree with the use of brackets to specify type range. Yes, this is a well-known convention in mathematics, but we're not in mathematics here. In practicality, if we lean towards a C-style syntax, brackets are now given extra responsibility, since they work as the array-indexing operator, and are sort of the de-facto standard for expressing the generic indexing concept in C++. The bottom line is that I now have to know multiple possible roles of the [] symbols.

That doesn't seem like such a huge deal to anyone who has been doing real-world programming for any length of time, so I understand that this looks like a really anal bit of nitpickery to many. In fact I had a similar discussion earlier today about a related issue, and was alone in a group of 5 coders in advocating verbosity, simplicity, and consistency over obtuse abbreviations and symbol-overloading. My gut feeling is... yeah, it's a tiny thing to have to memorize, but why should I have to memorize it at all? Tiny things add up; when a language is full of such tiny, "insignificant" things, we get C++, which is a damned confusing language (syntactically) on the best of days.

The main reasons I hear for using tight, compact symbol-hackery instead of clear natural-language-style syntax are:
- English-like syntax takes too much typing
- Verbose syntax takes up a lot of screen real-estate and can inhibit readability

The first objection is frankly laziness, as far as I'm concerned. Typing is such a trivial subset of the work of programming that I honestly don't care about it. My feeling is that unless the amount of typing work increases at least fivefold, typing work is not even a consideration. So what if I have to press six keys instead of two?

The second objection has merit, and as a matter of fact it's something I think is important when approaching the syntax design of a language. At-a-glance perception of code is very important. Highly experienced programmers are sometimes said to be able to "see clean code" just by looking at the formatting and general pattern of certain syntax elements. Screen real-estate I don't think is really that big of a deal, especially since the eye is remarkably good at filtering out irrelevant syntactic fluff to get to the "good stuff" once one gets used to a language.

Overall readability, though, is certainly an issue. My personal view (again, can't speak for anyone else, and I know many would disagree) is that a natural-language syntax is more pleasant to look at in general than a highly symbolic one. I know of few beginners who would say that Lisp is more readable or obvious than BASIC. Yeah, again, those distinctions largely go away once one invests a lot of time and effort learning to "read" a language. All I'm saying is... why should we even have to? I'm lazy. I don't want to learn to recognize Lisp syntax. I already did all the work of learning to read English, so why can't I take advantage of that existing ability to read (parts of) my code?


Anyways... I need to cut this short (as if it were short to begin with...) or I'll babble all night [grin]

OK, one last thought, and then I'm done for a while, I promise [smile]


Another area of datatype semantics that could be extremely useful is manually specifying interactions between datatypes. In the Tim Sweeney thread I talked about Conversion functions that can be used to intelligently munge between data types without risk of losing valuable information or accuracy.

However, a challenge comes up in cases like this:

DataType oddnumber Basis:Integers Validation(x):{ x % 2 == 1 };
DataType evennumber Basis:Integers Validation(x):{ x % 2 == 0 };

oddnumber x = 3;
evennumber y = 2;
y = x;

What can we detect and whine about at compile time?

• We can compile the validation functions for each data type


• Validation functions can be required to be effects-free by the language


• Since validation has no side-effects, we can validate data at compile time


• We can validate that 3 and 2 are acceptable for oddnumber and evennumber, repsectively

Therefore, we can validate at compile time, before the program even starts running, that the first four lines of this code are never going to cause problems or violate their own semantics.

Unfortunately, the fifth line is a problem. We cannot, at compile time, detect that these two data types are mutually exclusive. (Typical reduction to the Halting Problem, blah blah blah, etc.) The best we can do is note that both oddnumber and evennumber have explicit validators, and therefore issue a warning during compilation:

Warning W47948: You might be doing something really retarded on line 5.

However, any human reviewer can see immediately that any assignment between evennumber and oddnumber is going to barf. The two are mutually exclusive.


There are two ways to handle this in the language. The first is to be pessimistic and assume that conversion between semantic types never makes sense unless there is an explicit conversion function, or a chain of conversion functions, by which one semantic type can be transformed to another. The second is to warn about any such conversions, but allow the conversions to be flagged specifically as errors by means of an additional declaration:

MutuallyExclusiveDataSemantics oddnumber,evennumber;

Now the compiler knows that if it ever sees an oddnumber getting assigned to an evennumber, it can vomit outright and demand that the programmer fix his mistake.

I'm not sure off the top of my head, but at the moment I favor the second model. Yeah, it requires more typing again, but it has a very important bonus: it clearly defines the semantics to human readers. Sure, the compiler can figure out that oddnumber=evennumber is a dumb idea in either approach. However, by specifically expressing that these two types are mutually exclusive, we now codify more information that the human programmer can use.

When considering a system, there are two sets of knowledge that are important: the set of what the compiler/language knows about the system, and the set of what the programmer knows about the system. The idea here is to minimize the difference between these two data sets, by giving linguistic constructs that allow explicit expression of as much of that knowledge as possible. This is really the same principle behind requiring DBC concepts in the language itself.

Quote:Original post by Falling Sky
[snip thoughts on brevity of accessors, mutators, initialization]

Sounds like your friend needs C# [smile]

In general, I agree - these types of things are preferable. However, I think that may be a bit far down the road at the moment, especially considering that we don't even really have a plan for how OO concepts are implemented in the first place. Given the leanings of Sneftel's suggestion earlier (and my heartfelt agreement with it), I think it's safe to say that the OO philosophy will be quite different from C++, which might make some of these subtleties moot.