Back to General and Gameplay Programming

A pragmatic language for the future

Mike Lewis · 2008-03-10T17:43:52

Introduction This topic was started as a way to discuss a practical, real language that will aid in solving real programming challenges. Some of these challenges are already old and well-understood in the field. Others, like multiprocessing, are partly understood, but will become increasingly more important in the near future. This discussion arose from the observation that currently popular languages (an oft-mentioned culprit being C++) do not solve these needs well. Other languages may address these needs in very elegant and effective ways, and yet these languages have failed to achieve mainstream usage in most problem domains. Our goal is to critically analyze the problems that we face, the solutions to these problems, and practical ways to integrate those solutions into a new language. Our goal is not to develop the "perfect" language, or to build anyone's "dream language." Rather, we accept the reality that any language will have to have some ugly, rough edges. At some point, a language must disappoint someone. We are not here to make the ultimate language for everything. However, we do believe that a language can be developed which is both a major improvement on existing tools, and is also suitable for a very wide variety of problem domains. Whereas many highly elegant languages are, realistically speaking, only used in narrow problem spaces, we intend to devise an elegant, rich, and powerful language that is suitable for many problem domains, and is appealing to programmers working in those domains. (Note: this post has been modified to clarify and better express our plans. It has changed with the course of discussion to reflect the changing vision of the language and its goals. I fully expect it to continue to change more over time, as described below in the Philosophical Goals.) Philosophical Goals These are the cornerstones of our effort. These ideals guide our planning and discussion. They are intended to provide a sort of shibboleth by which we decide which things are good for our language, and which are not. If a decision is to be made regarding the language, it should be made in light of these principles. Seek a careful balance between pragmatism and idealism. By nature, our investigation is seeking to apply the best knowledge of programming theory to the domain of real-world problems. We are interested above all in a language that effectively aids programmers in accomplishing their work, be it academic research, embedded applications, business logic, or games. While pragmatic concerns are intensely important to our efforts, it is also vital to remain grounded in solid theory. We seek not just an elegant and self-consistent design, but also a design that gives us powerful tools to accomplish the creation of software. Produce a real language for real work. At all times we should be mindful of the fact that we are not engaging merely in a mental exercise. We do not seek to produce an idealistic language that exists only in specification papers and committee meetings. Instead, we aim to grow a language organically from the experience, insight, and desires of real programmers who work on real code. This language is being designed with the very real goal of implementing it and using it for doing work. Without this emphasis, our endeavor may be mentally stimulating, but will ultimately not be useful. Driving for an actual realization of our ideals is vital. Use the language early, use the language often. As noted, one of our goals is to produce a real language that is viable for real work. Therefore, it is important that as often as possible we test our designs by doing real work in the language. During the early stages we can expect the language to change rapidly; however, even once this pace slows later in the language's life, it is vitally important to make sure that the design remains consistent with real usage. It is in our best interests to seek to implement the language as quickly as possible, and use the implementation heavily. By using our own creation, we ensure that our designs are truly good. Moreover, we offer ourselves the opportunity to refine our designs based on real feedback from real use of the language. We must truly eat our own dogfood, and we must be prepared to adjust the formula to taste. Provide the tools to develop abstraction and express it clearly. One of the fundamental aims of this project is to assist programmers in clearly communicating their intentions to the machine. It is unfortunately common with popular language tools today for a programmer to "know" much more about the semantics and expectations of a system than he is able to tell the compiler. We aim to reduce this gap in knowledge, and give the programmer tools and a vocabulary by which to express abstract notions precisely and thoroughly. Flexibility is paramount: assume the programmer knows his needs well. It is important that we retain a degree of flexibility. Despite its flaws, C++ has remained a powerful force in the programming world because it permits the programmer a very high degree of control. We must be willing to trust programmers with this control, because in many problem domains, such control is important. The language should exist to support the programmer's efforts, not constrain or limit them. Reliability is paramount: the programmer is fallible. This is a careful balance with the previous goal. While flexibility is important, safety is also equally important. Productivity comes by removing menial concerns. Abstraction and encapsulation aid productivity by removing the concern of duplicating technical and implementation details. Garbage collection aids productivity by removing the concern of memory management. Wherever possible, we should seek to increase productivity by removing the mundane technical concerns of computing, and allow the programmer to focus on solving the problems of his domain. However, it is important to permit the programmer to be concerned with these things, should the need arise. Build what is needed. Build tools to develop the rest. Feature creep and bloat are very real dangers. We must avoid the temptation to add every possible feature under the sun. Rather, we should seek to provide a powerful toolkit by which programmers can assemble a library of useful features themselves. Moreover, where it is practical to do so, we should permit programmers to bend and modify these features to suit their individual needs. Users of the language should be encouraged to develop reusable abstractions and share them with the community as a whole. Some fundamental features should be codified as "standard" elements of either the language or the core library of features that ships with a compliant implementation of the language. Favor programmer productivity and efficiency, even if it makes implementation of language features difficult. Language features only need to be implemented once. After that, they may be used freely. We should strive to implement useful, powerful, and elegant features, even if they are hard or expensive. Our investment of time and effort in developing the language's features will pay itself back many times over in saved time and effort by the users of the language - which, hopefully, should someday include us as well. Practical Challenges Since we are aiming to develop a real language for use in real work by real people, we must remain mindful of the real issues that we face. There are challenges all along the way that must be addressed and considered if we are to succeed. Target Audience We cannot be all languages to all programmers. Such is not practically - if even theoretically - possible. However, we seek to provide sufficient flexibility that the resulting language is useful for a wide variety of people. We aim to allow programmers who are experienced with traditional imperative languages to transition easily and comfortably into this language. We aim to permit functional programmers to use their preferred style in order to gain the advantages offered by the functional approach. We aim to permit a wide range of development paradigms and philosophies while encouraging and facilitiating safe, reliable, and productive practices. Ideally, we would like to provide the tools for programmers to work close to hardware, in domains where this is important (embedded systems, realtime systems such as 3D games, etc.) while also offering a rich library of abstractions, so that the language is useful in higher level logic domains (business logic, web logic, simulation logic, etc.). Where existing languages succeed in crossing many paradigms and methodologies, we seek to cross many levels of abstraction; it is our belief that by offering a unified toolkit for developing on many such levels, each level may benefit by being coherent and consistently expressed with the other levels. Platform Support For better or worse, a programming language will not survive as a viable tool without interoperability with the Windows platform. The ability to interoperate with the family of Unixes is also important, although to a slightly lesser degree. A central goal should be interop with the Win32 API. Ideally, a layer for interacting with the .Net platform should also be developed as quickly as is practical. The language must facilitate the development of interop layers with the Unixes as well. Execution Model Agnosticism The language should be permitted to execute under many implementations: interpreters, compiled bytecode, compiled machine code, et. al. This allows us to use and develop the language early on (since interpreters and bytecode systems are fairly easy to build) while allowing us to eventually target hardware directly (compiling to machine code). This ability is important, because assuming the existence of a VM layer or interpreter violates the usefulness of programming close to hardware. Additionally, ability to compile to machine code (or possibly widely adopted VM formats such as MSIL) is critical for performance, since it is doubtful that we will be able to build a VM system that can rival commercial products. Early and Widespread Buy-in Just as we must test and use the language while it is being developed, we must also provide incentives for others to do so as well. Enlarging the pool of users and spreading the word is critical. Without widespread support and interest, the language will not be able to compete with established technologies. Achieving this buy-in requires that we be able to offer people the ability to use the language early on. It is vital that, as soon as possible, programmers are able to start writing real, useful code in the language, even if it is just interpreted or bytecode compiled. Growth by Supplementing, Not Replacing, Existing Tools Realistically, the "big language" to compete with in terms of buy-in is C++. Other languages are also important in various domains, but for Windows applications and games in particular, C++ continues to be the dominant player. This is primarily due to inertia: huge amounts of time and money are invested in existing C++ systems, making it impractical to replace them with systems in a new language. We cannot hope to gain adoption by demanding that people discard C++ and use this language instead. Rather, we must gain adoption by supplementing C++. Interoperability with C-style APIs is mandatory. Providing the tools to build binding layers with C++ systems (similar to embedded-language projects like boost::python) is essential. Programmers must be able to take this new language and add it to their toolset slowly, at a pace which is practical and comfortable for them. We cannot replace the existing Big Players overnight, because the inertia of existing code is too large. However, with careful planning, we can make it more cost-effective to use this new language for new code, rather than writing new code in old languages. Over time this will amount to a gradual shift in language emphasis. During this transition, though, it is important to emphasize compatibility - this language must be usable in concert with existing tools like C++. Demanding a full replacement of existing code will not work. Language Elements and Features This is an overview of the features and capabilities that we have chosen to build into the language. Favor high degrees of abstraction and transparency Permit low edgrees of abstraction and allow technical concreteness (programming close to the hardware) Maximize the ability to express behavioral semantics of a system, as well as the data in that system Provide rich introspection and metaprogramming capabilities Encourage the use of nested abstraction layers to build domain-specific "dialects" on top the base language, via metaprogramming Type System This section deals with the type system of the language: its nature, features, and limitations. Employ static typing and strong static-time checking of types. Encourage the definition of abstract datatype semantics, and methods for converting between semantic types. Provide rich features for dimensional analysis. Semantic datatypes are defined by the set of values they may store. A variable of a given semantic type may not be assigned a value which is not in its set. Doing so should cause a static, compile-time error (if possible), a warning if the behavior is not guaranteed to be reliable based on the defined semantics of the type, and a run-time error should an invalid assignment be attempted. Semantic type validation is performed by a boolean validation routine that returns either "true" for valid data or "false" for invalid data. This is done instead of explicitly defining semantics in the language syntax to allow for maximum flexibility in the way validation is performed. Validation routines are encouraged to be kept simple so that they can be checked at compile time. To help enforce this, validation routines are not permitted to interact with data outside the value being validated. This ensures that validation can be checked statically. Support for generic values is accomplished via a subset hierarchy. At the root of this hierarchy is the notion of Anything. All language concepts are a type of Anything (this is done to allow for higher-order functions, introspection, and other metaprogramming capabilities). Various data concepts express restrictions over the broad Anything. Such restrictions include numeric types, container types, and (eventually) objects. In addition, individual restrictions may be further subdivided by semantic types. For instance, numeric types can be split into integers, reals, complex numbers, and so on. A type may implicitly take the place of any superset of its own semantic set; an Integer is always substitutable for a Number, and a Number is always substitutable for Anything. Types may not be substituted across the hierarchy to sibling types, or down the hierarchy to child types, unless a specific semantic conversion function has been defined for the particular conversion being attempted. Support type inference as much as possible. Whenever a type can be statically inferred, this should be done automatically by the language. The simple rule of thumb for the compiler is "infer what I can; complain about what I cannot." If a type cannot be inferred, it must be explicitly specified by the programmer. Additional points will be added as final conclusions are reached in the discussion. The Current State of Affairs I have personally taken on the responsibility of bootstrapping the language and delivering an implementation as quickly as my limited spare time allows. A running commentary of my progress is available via my journal, which can be reached by clicking the Journal link below this post. When major events in Epoch's development arise, I will drop a post in here. For the minor bits, keep an eye on my journal. In the meantime, everyone please feel free to continue discussion and making suggestions here. Optional, But Encouraged Reading The best starting place is to read this thread itself, as it contains the vast bulk of the history of the discussion, and all of the important conclusions and issues are at least mentioned here. This rest of this is just some background material to help illuminate the train of thought that started this project. With the exception of Tim's presentation, don't take any of this as the word of an expert. Heck, don't even take it as the word of anyone who has the first clue what he's babbling on about [wink] The Pragmatist's Programming Language Wishlist Additional explanations and thoughts Tim Sweeney's presentation on languages (PDF) An early concept for abstraction layering Some early thoughts of mine on dialectic language families Some early thoughts on integrating expressivity of abstraction and automated testing My thoughts on domain-specific languages and the need for expressivity [Edited by - ApochPiQ on March 10, 2008 5:38:52 PM]

General and Gameplay Programming Programming Unity

Started by ApochPiQ March 27, 2006 07:18 PM

609 comments, last by ApochPiQ 16 years, 1 month ago

ApochPiQ

23,138

Author

March 10, 2008 05:43 PM

It's baaaaack! I'm doing some work on a bootstrapping interpreter which I've named Fugue. Watch the development progress (such as it is) in my journal.

In the meantime, everyone please feel free to continue with discussion here. I can't promise to keep up with everything 100%, but I'll do the best I can. The input of the community in shaping this language has been very valuable to me from the very beginning - especially since there's a lot of people out there who know far more about programming languages than I do - so go ahead and speak your minds.

Oh, if anyone is interested in getting builds/code for Fugue, let me know and I'll start uploading them periodically to the SourceForge site (which currently is down, but I'll get it running again).

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

A pragmatic language for the future

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

A pragmatic language for the future

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines