19 replies to this topic

[Signature]

Hi all!

As a programmer I am really frustrated with the state of the art.
Interfaces are often set in stone. Hardcoded software is very hard to refactor. Libraries were developed with a different focus than what you actually need etc.
If you are not off the charts smart starting and finishing a complex project is extremely hard.

After looking into CASE I started wondering if things need to be that difficult or if it might be the tools that make things harder.

What do you think about the following thoughts ... why isn't IT moving in the following direction:

UML is pretty mature and NoSQL is on the rise ... there could be a UML management service that
- makes the definition of models (UMLDL) and instances (UMLML) that go with the models possible
- allows developers and users to browse the models (UMLQL) if they have the rights to make the query (UMLCL)
- allows syndication of defined models
- is multidimensional (layers that can be parametrized)
- allows streaming (given changing parameters)
- comes with concurrent version support (maybe as a dimension)
- would be a low level infrastructure for a new generation of tools

Most developers would not have to worry about UML ... because there would be new authoring tools that can be customized and expanded easily ... getting the meta data as a model and JSON objects that they understand because they have the meta data.

Guess the google video about "Apps without borders" really got me thinking.

I don't see any show-stoppers ... what do you guys think? And does anybody know of an effort based on similar thoughts?

I started working on a Java implementation but I guess the concepts (like a RFC) should really be done by smarter people.

This should give you a basic idea of how I want to access data:

// UMLQL

fetch dimensions
tagged as tags
for query

fetch queries
tagged as tags

fetch model
select types tagged as tags
from queries
add dimension using parameters

fetch instances
for model
ordered by values
limit to number

stream query
synchronizing parameters for dimension

// UMLML

insert for model
set values 
connect instance as relation
in dimension with parameters

update instances
set values if (not) in instances 
connect instances as relation if (not) in instances

delete instances

// UMLDL

create dimension

create subject as type
relate subjects as relation

alter subject
relate subjects as relation

drop subjects

store select as query

Rock on everybody!
AndyT

Edited by Signature, 28 May 2012 - 06:07 AM.

[Antheus]

UML is pretty mature

UML has several problems:
- impedance mismatch (it works the way managers work, not how software is naturally written)
- developed, authored and owned by private for-profit committee - they make money off training and product sales, not improving productivity
- overhead added by UML can only be covered by large companies, think 500+ development teams
- it is used for risk aversion and establishing paper trail as well as process certification, it completely fails if software usability is at premium
- CASE are used for business intelligence and business process management, and even there the generic package isn't widely used, there are better toolings provided by other vendors. They are not a good fit for actual software (code-related) development. They work best when managers define various processes and workflows, off which managers under them base their work. If you look at eclipse forums, you'll notice there are more posts in workflow and BI forum than all other topics combined. CASE is not for developers, it's for managers who don't do technical work.

NoSQL is on the rise

No... Not really. Maybe for 2 person web shops.

There is no database in enterprise that cannot be handled by SQL (DB2, Oracle or MSSQL - others do not exist), there is today sufficient hardware available to scale these vertically for any real world data set.

ETL and ad-hoc analysis works on different data sets, but again, for enterprise usage problems lie elsewhere and they are currently catered at both high-end (again IBM and Oracle) as well as low-end (S3-based solutions).

SAP is also in there somewhere. Surprisingly, it tends to work quite well, so it gets little press because of lack of snafus.

I started working on a Java implementation

Does it work with every existing technology and standard? Out-of-box? If no, then it's of no use. It's the first thing you will get asked.

UML is both a mess and "success", because Rational and IBM made sure it works with every dead and living standard and technology stack. The result is multi-gigabyte monstrocity built on top of Eclipse (and previously Visual Age).

- makes the definition of models (UMLDL) and instances (UMLML) that go with the models possible
- allows developers and users to browse the models (UMLQL) if they have the rights to make the query (UMLCL)

I remember working on something like that.
- there was abstract model
- after it got complex, these models got auto generated, so meta language was developed for describing the models
- this meta language got too complex, so meta-meta language was create (no joke)

Meanwhile, actual users and developers gave up on it and used 5 line text files instead 500,000 LOC meta-meta framework. Application as it was meant to be written was 500+ MB of source + 500MB of generated code at the time I left.

BTW: the thing won an award from one of these CASE groups.


But the most important thing to understand here is that problems inherent here are not technical. It's complex because of the way enterprise world works.


For a modern take on enterprise software, I'd look at salesforce. They are considerably smaller in scope than players above, but they are based from ground up on web-centric approach and are investing a lot into more dynamic/live approach.

Edited by Antheus, 28 May 2012 - 06:39 AM.

[Tribad]

I am doing most work with UML.
But I will not use IBM Rhapsody, former Rational, because this tool lacks a lot of features I need.
Whenever it comes to questions why UML is not used in software development I am getting the same answers.

These are the highlights:

- we are developing in C (not object oriented)
- code generation does not what I want.
- I can use any graphics tool for documentation
- refactoring can be done in eclipse easily aswell.

I am doing development in C and C++ as well and using UML all the time. Code generation works for me perfectly and not only creates the code but also the build files.
Shure. Any graphic is usable as long as everyone understand what you are thinking. Using a standard way of descibing something as pictures makes software more understandable.
Refactoring with an UML-Tool even means restructuring. Moving classes around, re-connecting dependencies and all such things. This is hard to realize with a eclipse only environment.

I am a fan of UML. But there are only a few tools that hit my feature list. I am fast even with small systems. And always have the possibility to create documents from the tools.

[Signature]

impedance mismatch (it works the way managers work, not how software is naturally written)

Well, even as a programmer ... I wish software would follow the model of whatever it is the software is meant for.

No... Not really. Maybe for 2 person web shops.

There is no database in enterprise that cannot be handled by SQL (DB2, Oracle or MSSQL - others do not exist), there is today sufficient hardware available to scale these vertically for any real world data set.
ETL and ad-hoc analysis works on different data sets, but again, for enterprise usage problems lie elsewhere and they are currently catered at both high-end (again IBM and Oracle) as well as low-end (S3-based solutions).

Well I was referring to what google does with their page rank (Bigtable?) and I saw NoSQL in the min. qualifications of quite a few open posts. But I think soon most projects and organisations will struggle with performance and complexity.
The company I work at struggles with updating data after changes to the meta data (application business model) and with interfaces to other systems all the time.

And I don't see why stuff is hardcoded at all these days.
If that was not the case things would just fall into place and syndication would become possible.

The syndication possibilities are what I miss in SQL systems. A UML management system could probably be built on top of a SQL system.
I want IT to get to a point where I drag and drop a human, a piano and a camera from Wikipedia into Blender and possibly the whole concept of multibody dynamics.
UML models in XML representation should make that possible ... if the applications that are connected that way work with the same model or define a mapping strategy for the models.

Does it work with every existing technology and standard? Out-of-box? If no, then it's of no use. It's the first thing you will get asked.

But the most important thing to understand here is that problems inherent here are not technical. It's complex because of the way enterprise world works.

For a modern take on enterprise software, I'd look at salesforce. They are considerably smaller in scope than players above, but they are based from ground up on web-centric approach and are investing a lot into more dynamic/live approach.

Well I don't want to sell to companies. I want a movement by innovators for innovators ... that the business world will have to follow.
I don't see why such a service could not be used in any field (the RFC could allow different kinds of connectors, even combinations of TCP and UDP for data that has to be streamed ... maybe even servlets could provide the service).

Any projects that connects to a database (networking capability) and can parse xml should be able to use the system.

Edited by Signature, 28 May 2012 - 01:49 PM.

[LorenzoGatti]

And I don't see why stuff is hardcoded at all these days.

Words like hardcoded are a strong signal of MDA-flavoured magical thinking.
In the real world, if software does something useful and meaningful, it's because someone took the trouble to write it, and if software does something "automatically", something that is not "hardcoded", it's because what has been programmed is sufficiently abstract and general and well designed, which implies even more programming effort and/or skill.

[Signature]

Words like hardcoded are a strong signal of MDA-flavoured magical thinking.

Not sure I understand what you are saying. The importand part is a proper design ... I agree.
Why shouldn't data model and application logic be designed in UML and the actual programs rendered and deployed by CASE services on the fly?

Why does a programmer have to write the code? Why are there binary distributions of isolated programs all over the place when most programs could run (and share) scripts these days?

[LorenzoGatti]

Why shouldn't data model and application logic be designed in UML and the actual programs rendered and deployed by CASE services on the fly?

Why does a programmer have to write the code? Why are there binary distributions of isolated programs all over the place when most programs could run (and share) scripts these days?

Other disturbing watchwords: "deployed on the fly", as if tool automation didn't require close supervision and reduced the need for testing; "have to write the code", as if one could get away with writing less than is needed and letting a magic wand CASE tool take care of the details.

While using appropriate programming languages can make writing the same program less expensive, or even qualitatively shift programming tasks from impractical to easy, scripting is no free lunch. The price of smaller and simpler scripts is an increase of complexity of the scriptable system and an increase of its implicit information and inflexible assumptions.

I don't want to insist on why the right place of UML is on whiteboards, as a communication tool for people, and why it is terrible as a DSL; even with the favourable assumptions of writing your scripts in a good programming language, the kind of infrastructure you propose is doomed because, both technologically and economically, it only makes sense where there are many applications that are both very complex and very similar, which is an extremely small and possibly empty niche.

• If the complex and similar applications are few, straightforward modular design allows one to implement variations cleanly, without the need for general scripting systems or complex code generation.
• If many complex applications are moderately similar, they are usually made independently, by sharing as many tools and common pieces as possible.
  SAP installations, consisting of generic modules that do things almost every customer needs and lots of code for highly valuable customization that the whole system has been designed to allow and manage, are a good example of this category.
• If many similar applications aren't very complex, or at least their differences aren't, tools and languages can be simple and specialized.
  Game mods and custom levels are a good example; how would you involve UML in the creation of images, 3D models, map layouts, and so on?

[Signature]

What facts prove that automation requires close supervision?

Assume we had one model describing the whole world through syndication, where is the showstopper that prevents applications to use that middleware to share data and create an "Applications Without Borders" user experience?

Yes, it only makes sense where there are many applications that are both very complex and very similar ... but that is my vision for the whole internet. We should not write stuff that comes to mind but EXACTLY what is needed to reach our goals.
I don't see any substance in your arguments, yet.

Btw. I updated the project with a first take on possible Milestones for the best guess implementation of a UML management system.

Edited by Signature, 30 May 2012 - 03:12 AM.

[frob]

It is worth reiterating that these currently are communications tools.

There have been hundreds of similar tools over the decades. It is possible to auto-generate stubs and pieces of software, but ultimately they have failed at becoming a final implementation. CASE tools peaked in the 90s, and nearly 20 years of maturity later they are still relegated to a method of communication rather than a method of final implementation.



Perhaps you do have the 'killer app' idea. People come up with those occasionally.

The systems you are talking about are stable and mature and have been developed for decades by countless teams of experts and geniuses. I don't know if your app idea will transform the worlds of software development, software communication, CASE tools, or even become the new 'vision for the whole internet'. It is certainly possible. Somebody has to have the transformative idea, and perhaps that person is you.

Just keep in mind that many people and many businesses have walked that path before, some investing and earning billions of dollars along the way, but none have discovered what you describe.

So nothing PROVES it isn't possible. There is just a lot of evidence that it isn't feasible.

Perhaps you have discovered a way around barriers that Rational, IBM, Oracle, HP, and a bunch of other companies and research groups could not. But please understand the reasons why so many are skeptical.

[Signature]

What I have not seen yet is an attempt at bringing the data storage and meta modeling together as a service for higher level applications.
I realize that ones guts have to tell us that there is something wrong with the idea. Maybe there is a fault in the concept. I was mostly just screening if anybody here can spot one.

I do understand why people are skeptical, I am just a little disappointed that nobody wants to just play the thought ...
I am not complaining ... I am trying to challenge the opposing forces to play the thought and find a substantial flaw :-)

Hope I did not offend anyone. I will keep working on the proof of concept but I could really use any kind of input and don't want to waste time and energy if this is not worth pursuing.

[Tribad]

I think I missed something.
What do you want to make differently than UML does?

In your very first post you used a language again. If you want a more powerfull language than Java or C++ or something similar create one.

[Signature]

Yeah I did not do a good job of describing what I have in mind.

The idea is defining a system that combines syndication of UML models and storing data that is in compliance with the model.
Both the model and the data can be accessed and modified by clients. The system would be a middleware that wraps the actual databases and removes the business logic from most programs. And the UMLM systems would need to be able to connect to others for syndication.

Most programs would not need to know the data stucture any more. They would ask that service what model they should work with (through UMLQL) and then collect data (also through UMLQL). Model and data can be defined and altered (with UMLML for instances / data, UMLDL for the model). UMLCL would add a possibility for permission management.

I imagine the specification for UMLMS could be general like that and implemented in pretty much any language and support pretty much any kind of database.

The Java implementation I mentioned was the project I have started to test the waters and to move towards a proof or disproof of concept.
And I should not have mentioned scripting without defining what context I think this would play a role in.
The system would be a low level and tools built on top of that middleware would need much less hardcoded business logic.

I also noticed that I broke one of my rules ... I have not quite tried to understand first (before seeking to be understood). SAP was mentioned twice. I just noticed I don't quite know how well that can be used to define business models that go beyond ... well ... business. Guess I will do some more research based on the input I already got.

Edited by Signature, 30 May 2012 - 07:50 AM.

[Tribad]

Maybe you should have a look at OpenAmeos UML-Tool.
The Tool stores the meta model in a database and the tool itself is completly independent from something like the UML-Meta-Model.
AFAIK it is based on Software-Through-Pictures from Aonix.
If you download and install the UML-Tool you get a lot of documentation about how it works and you get all scripts that make up the UML-Tool with it.
The UML-Tool has been created with a lot of scripts that define how the data should be handled.

Have a look at it. Its a funny flexible system

[swiftcoder]

The idea is defining a system that combines syndication of UML models and storing data that is in compliance with the model.

XML schemas, SQL tables with constraints - they are both examples of carrying structure along with the data, in a parseable form. What does moving this metadata to UML actually gain us? (keeping in mind that UML is designed for human readability, not machine comprehension)

Both the model and the data can be accessed and modified by clients.

Can you give me a real-world example where client modification of the model is beneficial? (and by beneficial, I mean worth the risk that changes will invalidate all other client software)

The system would be a middleware that wraps the actual databases and removes the business logic from most programs.

How exactly does the existence of a published model remove the need for business logic? I may be missing something essential, but if my program relies on the username, I have the alternative of using data['username'] vs data[model['username']]. The latter allows me to more easily adapt equivalent data from a different source - but that could equally well be accomplished by a stream filter/whatever, and it in no way impacts my actual logic.

[Signature]

Maybe it is important to understand where I am coming from.

1.) As a reader of Abundance and fan of XPrize.org and Singularity University I believe in DIY innovation, barrier free environments and collaboration.
What bothers me is that we never think about how to pool our ideas and data. All efforts are isolated - to some degree they are brought together in universities but that is not good enough. And we don't remove borders by applying the DEAL (definition, elimination, automation and liberation) principles. It is extremely hard to contribute to big projects. There should be better authoring tools and those tools should play well together.
Eventually computers will need to "understand" the world ... think "Turing" test. The big challenges will require syndication and exchange of data (power grid, sensors - http://www.nokiasensingxchallenge.org etc.).

2.) As a programmer I hate my job right now. Too many clients for the size of the company, too many products and too many different versions of our software and products that are hard to refactor and don't play well together.
The one good thing we have is a code generator that creates stubs for forms from xml descriptions.

So I went from wondering: "why can't we also create DAOs from meta descriptions?"
to "why can't we define the whole software in an abstract way and generate most of the source code with the help of builder services?".

There were many times where we hated that we were stuck with our software architecture.
We would have loved to sell all our products in one suite edition, sell a "Single session, multiple users" standalone version of a product or plugin or even sell one feature of one of our products as a cheaper product.
So I wanted a CASE tool that knows about tiers, roles, use cases, layers, classes, interfaces, functions and structograms ... and that allows users to define (capturing as data, not hardcoded) what to do with any business model in order to realize a certain kind of build.

Then I stated to wonder "why can't we use the same business model that our clients use?" ... for example share our model with the help of a model service.

UML is more than just class diagrams ... you can define what a service is supposed to look like etc.



And combining both trains of thoughts I ended up thinking that a syndication of UML management systems is the way to go ... imagine most current LAMP and WAMP server added that middleware and all different kinds of authoring tool projects started using it.
How different are our applications really? How many use the concept of scene graphs? Would a service that knows physics and updates a generic scene graph streamed from and then to a client make sense?

Go from there ... and I think the questions that arise are just small details:
* render software as binary builds?
* turn OS into a browser that can deploy any application in any way with generated scripts?
* ...

And we would not even have to share all data if that is what you are afraid of now.
A new program could embed a UML management system and establish a P2P connection to the same program running on a friends computer.
They could exchange instances for a model they fetch from the cloud without sharing those instances with anybody else.
Companies could still develop proprietary models by working with their intranet cloud and not sharing their additions to the public model.

[LorenzoGatti]

The driving force behind this combination of naivety and basic MDA/SOA delusion is the OP's bad experience with a badly designed software product line, where copy&paste reigns supreme and tools like a generator of "stubs" encourage massive duplication of write-only and very boring code.

Improving this situation is a laudable intent, but if the current mess is considered "hard to refactor" it means that it's already exceeding the technical ability of the whole company: hoping to make a sophisticated CASE tool that understands and automates the whole product line design is obviously more difficult than a more gradual and simple redesign effort, and therefore it cannot be considered anything but wishful thinking.

Serious software engineering would consider the difficulty of writing such a CASE tool (high, and higher than alternatives), its cost (obviously in the bet-the-company range), its risk of failure (horrible, as it is only useful if complete and flawless), its provided value (limited and uncertain) and figure out something more humble and practical, like a component-based plan that starts by imposing some order on existing code and replaces modules with better designed and more configurable ones over time.

But naive software engineering is able to neglect practical constraints (like time, money and skill), turn a blind eye to difficulties (for example, assuming that UML is useful) and dream of far less realistic systems: instead of a tool for the specific sort of products the OP's company is making, which could contain a lot of valuable complex design and genuine knowledge of "business models", a silver bullet tool for everything, able to model and understand generic processes: it would be so generic that it would go full circle to a combination of programming language and runtime platform, like e.g. Java, and it would be no better than the currently vague language design and libraries it would consist of.

The final layer of delusion is a compound free lunch expectation: the AI will put together working software that satisfies requirements without supervision; from "business models" which will be simple to write; sparing programming effort by reusing other people's "syndicated" models and data whenever possible. It goes without saying that all three points range from completely undemonstrated to fantastic.

Edited by LorenzoGatti, 01 June 2012 - 06:28 AM.

[Signature]

It would not be one silver bullet tool. UML model syndication would be one addition to our existing infrastructure. Like I said, any language could use that service it it can use a database system.

It would be fully scalabe ... every person and enterprise could decide how to use it - in other words we could actually decide to go for a humble approach. There days we need to employ specialists and come up with a model that has been defined millions of times already by other groups and individuals ... I don't see what's humble about that.

Through devide and conquer things would be simpler not more complex or complicated.

[ChristerSwahn]

It would be fully scalabe ... every person and enterprise could decide how to use it - in other words we could actually decide to go for a humble approach. There days we need to employ specialists and come up with a model that has been defined millions of times already by other groups and individuals ... I don't see what's humble about that.

I've got to ask, have you actually developed software AND the related processes for a big organization, and seen it through to production/roll-out?

Many similar subprocesses have been defined formally or informally by many organizations yes. But the devil is in the details, and process details depend on their context. Working with processes you have to make sure they integrate at all levels of abstraction which means you can't design/define them in isolation. Even a single human intelligence isn't enough. For instance I've seen banks overhauling processes and they have large teams devoted to the "paper designs" for years, and they're in constant debate with the rest of the organization how everybody and every system are to work together.

Through devide and conquer things would be simpler not more complex or complicated.

Which means: Don't try to design a single system/process/framework to solve everything! ;-)

[Signature]

Designing a single software is not the plan. The plan is offering a shared infrastructure that offers the best way to bring carefully designed software components together - as a "seamless software user experience".
Those software components would have to be designed in an abstract form in order to allow different kinds of deployments - and I see a lot of room for automation and elimination that would automatically be exposed with that kind of infrastructure.

Yeah I have worked in a big company where I saw what different teams (support, development and operating departments) struggle with. And I currently work in a small company as a programmer where I have witnessed and contributed to many projects and development cycles.

And I have come to the conclusion that teams and individuals struggle with problems and have to answer questions that should not have come up in the first place, all the time.

Edited by Signature, 01 June 2012 - 08:28 AM.

[Tribad]

What you are trying to do is creating a language to define abstract components and give them some parameter to fit into a problem.

That is what any good software engineer does.

If I want to count the number of children crossing the street, I can use the abstract code fragment that increments a variable whenever a child crosses the street.
But to make that possible you need someone, the software engineer, who is able to do the abstraction. The software engineer identifies the requirements for the software and take the code fragment for such a counting loop out of his database named brain and use it.
So what you are trying here to do is to teach a piece of software the interpretation/abstraction of any requirement that may come to mind and let the software decide how to solve it.

That is A.I at its best.

How often software engineers struggle depends on there experience they have.

My opinion.