The State of MDE Adoption
But, as we look to the horizon of a decade hence we see no silver bullet. There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity. — Brooks (p. 181) (Brooks 1974)
You cannot reduce the complexity of your problem by increasing the complexity of your language. — Wirth (attributed)
One of the most critical aspects of any new technological approach is the evaluation of its performance on the field — that is, an evidence-based analysis of the impact of its application under real-world conditions. It is particularly important to those embarking on the design of a new software development methodology, itself based on model-driven principles, to understand MDE's benefits and drawbacks from a practical standpoint. This chapter combines a review of MDE adoption literature with a limited amount of new research, highlighting lessons learned and barriers to entry, with the objective of clarifying MDE's current state of practice. As with theory (cf. State of the Art in Code Generation), these findings will be used as input to MASD's requirements gathering process (cf. FIXME Chapter).1
The document is organised as follows. Section Limitations sets out the reach of the analysis to be presented. The analysis proper is composed of two sections: Section How Widely Adopted is MDE? characterises the popularity of MDE within the software industry while Section Empirical Analysis of Adoption Literature teases out the broad themes that emerge from its adoption. Lastly, the chapter concludes with a brief discussion that summarises our findings (Section Discussion). Let us begin by laying out the scope and limits of the effort.
Limitations
Our analysis is constrained by two types of impediments: intrinsic limitations of the model-driven approach, and shortcomings related to the way adoption research has been conducted. The next two sections delve into each of these categories respectively.
Intrinsic Limitations to MDE
As recently as 2011, Hutchinson et al. (Hutchinson et al. 2011) warned that MDE had historically lacked evidence to back many of its claims (Hutchinson et al. 2011).2 These difficulties led Whittle et al. to admit that "[…] there remains a lack of clarity on whether or not model-driven engineering (MDE) is a good way to develop software." Their wariness is not entirely unjustified. In the previously mentioned paper, Hutchinson et al. identified three key reasons for why, in practice, MDE may have a detrimental effect in the development of software systems:
- Higher abstraction levels may not lead to better software. Citing experiments in psychology and psychology of programming, they point out that individuals find thinking in abstract terms hard and, in general, there is a tendency to prefer exemplars over abstract conceptualisations.
- MDE activities may have both positive and negative effects. Code generation is offered as an example, as it may have a positive effect on productivity but also negative consequences — including the time required to develop the models for code generation as well as the possible need to integrate manual code with generated code.3 Nor is code generation the only MDE activity with this conflicting property, making it a very thorny issue: "How the balance between these two effects is related to context, and what might lead to one outweighing the other, is simply not known." (Hutchinson, Rouncefield, and Whittle 2011)
- Determining the right approach is hard. Given the proliferation of MDE variants, tools and frameworks4, it is very hard to evaluate benefits and drawbacks in a rigorous way. Choices are highly dependent on context and thus difficult to compare in a fair manner. This is a theme we will return to in the next section.
Torchiano et al. (Torchiano et al. 2012) are even more critical of the status quo, managing to combine the challenges of practice with those of theory into one clear and incisive diagnosis (emphasis ours):
[MD*] is considered to be still evolving and not yet completely mature. The first success stories were heard a long time ago but the knowledge to make those successes consistently repeatable is still missing. Being the discipline not yet fully understood, and the underlying knowledge not yet codified, expertise is the only resource we can rely on when a MD* solution is designed.
Their criticism evokes the idea of a practitioner mastering a body of knowledge, as we have defended (Craveiro 2021b) (Section 2.2) — a scenario that is inherently unsuitable for repeatable and comparative analysis as it is highly dependent on context.5 When taken together, all of these open questions raise even further the significance of adoption research. Unfortunately, it too faces several challenges.
Adoption Literature Limitations
The literature reveals a litany of experience reports and empirical studies on MDE application (Andolfato et al. 2014) (Shirtz, Kazakov, and Shaham-Gafni 2007) (Paige and Varró 2012) (Clark and Muller 2012), and to these we have added our own (Craveiro 2021a). Taken together, they form a useful but somewhat fractious breeding ground from which to extract universal answers as to the suitability of MDE. Hutchinson et al. had already raised red flags when performing a similar exercise (emphasis ours):
As many of our respondents admit, quantification of the benefits and failures of MDE is complex and difficult. […] [The] evidence for our understanding of MDE at this stage derives from the quality and perceptiveness of our descriptions and our analysis rather than any simple mathematization. (Hutchinson, Rouncefield, and Whittle 2011)
Much the same said Mohagheghi and Dehlen (Mohagheghi and Dehlen 2008), who, whilst identifying threats to the validity of their own work, inadvertently enumerated the challenges faced by anyone risking a similar undertaking. These are as follows:
- Small sample size. Though not lacking on variety, the number of reports found in the literature is insufficient to reach a "generalization to a population or theory." (Mohagheghi and Dehlen 2008)
- Survivorship bias. In their opinion — which is also ours — successes are more likely to be reported than failures; therefore, the literature may portrait an inaccurate picture of the most likely outcome of MDE projects.
- Incentives for biased reporting. Research projects with external financing may report biased results, downplaying negative outcomes. Companies may also behave in a similar fashion to avoid negative publicity. Incentives amplify the survivorship bias.
- Interference from competitive advantages. Conversely, companies may avoid publishing results in order to keep a competitive advantage against their peers.
- Large-scale projects are under-reported. The success of MDE on small-scale projects such as academic research may not be indicative of its success on large industrial projects. Results for large-scale projects are not as frequent as those of small-scale projects.
- Lack of baseline data. Most companies do not report baseline data, which makes evaluations subjective. In addition, it is very difficult to obtain relevant and unbiased baseline data.
- Lack of quantitative data. Most studies focus on the qualitative aspects of the experience and neglect a quantitative analysis. More generally, there is a lack of well-defined dimensions for the gathering of quantitative data, for reasons such as those outlined by Hutchinson et al. above.
From what has been stated thus far, it seems clear that it is very difficult to use the literature to make universal statements about MDE adoption with any degree of confidence.6 Indeed, we argue that most universal statements about MDE are devoid of meaning from a scientific perspective because they must be localised to the specifics of a context and can only be extrapolated to other contexts with a great deal of care; even then, they would still be riddled with reservations. In other words, it is extremely difficult to compare MDE projects because they are highly sensitive to local conditions — conditions which are not readily replicable — much less lend themselves to easy aggregation at an industry-wide level. The essence of the problem was captured by Rumpe earlier, who had stated:
The pressing problems that we tackle in the software and system modeling research domain can be classified as "wicked problems": we learn more about the nature of the problems we tackle through experimentation with proposed solutions. Rigorous evaluation of these solutions invariably entails costly and lengthy experimentation in industrial contexts. Experiments that seek to evaluate solutions based on novel or radically different ideas are particularly difficult to sell to potential industrial partners because the risks are not well-understood by all involved. Even with committed industrial partners, the wide variations in industrial development environments makes it difficult (if not foolhardy) to extrapolate the results beyond the specific industries. Despite the difficulties, there is no getting away from the reality that evaluation is key to developing progressively better solutions to wicked problems. (France 2008)
In this light, even though empirical studies are not particularly useful in proving or disproving universal claims, they are still extremely valuable because they capture the themes emerging from within application. From these we can build conceptual tooling to augment MDE's body of knowledge — best practices, patterns, guidelines and the like7 — and, for the purposes of MASD, these observations can guide the requirements gathering process (cf. FIXME Chapter). On the main, it is in this spirit that the MDE adoption literature is to be understood within this dissertation. All of that said, we shall start by attempting to tackle one universal question — limitations described here notwithstanding. We will do so for two reasons. Firstly, because we believe it offers far-reaching insights into the application of modeling in general and therefore to MASD. Secondly, because it can be answered — even if only broadly. So it is to that pivotal question we turn to next.
How Widely Adopted is MDE?
High expectations were set out in a seminal presentation by Bézivin (Bézivin 2003), where he ambitiously declared model engineering to be the future of object technology, and outlined a twenty year horizon for its maturing. As we sail at speed towards Bézivin's evaluation date, it is increasingly clear that the adoption curves of object technology and model technology are distinct: the former became the mainstay of software engineering within less than twenty years of its inception, whereas the latter is yet to achieve similar levels of exposure. These thoughts are echoed by Mussbacher et al., who lamented that (emphasis ours):
[…] MDE is arguably still a niche technology. It has not been adopted as widely as popular programming languages such as Java and C#, and, whilst some modeling languages like the UML have become widespread, they are often not used to their full potential and the use of models to automatically generate systems is still relatively rare. (Mussbacher et al. 2014)
In sharp contrast with the hopeful tone of the past8, the literature now has a downcast mood9, perhaps reflecting the realisation that MDE "[…] is currently not as widespread in industry as the modeling community hoped for." (Mussbacher et al. 2014) This state of affairs is all the more surprising when one considers that the claims associated with MDE are often decisive factors in an industrial setting.10 There is therefore a clear disconnect between promises and adoption, perchance not unrelated to the difficulties in evidencing grandiose universal assertions (cf. Section Limitations).
Our objective for this section is twofold. First, we want to substantiate or disprove Mussbacher et al.'s claims by representing, however broadly, MDE's state of practice, because we believe it offers considerable insights as to the applicability of its vision to the reality of industrial software engineering. Second, we want to perform Bézivin's evaluation by characterising the direction of travel of MDE adoption — again, in broad strokes11,12 — because it gauges industry's reaction to it. The next two sections analyse evidence from multiple sources to position MDE within this frame. Evidence was collected at two different scales: the macro-scale — that is, across the software engineering profession — and the micro-scale — that is, small samples and empirical studies. With this in mind, let us start our analysis from a high vantage point.
Analysis of Evidence at a Macro-Scale
Whilst the literature does provide detailed quantitative data for MDE adoption at small sample sizes (cf. Section Analysis of Evidence at a Micro-Scale), measurements that place the approach in an industry-wide context are harder to come by. In the above cited paper, Mussbacher et al. used data from search engine queries to demonstrate, somewhat amusingly, that "MDE is simply not considered cool". Unfortunately, their dataset was insufficient for our purposes; however, their methodology was promising, so we extended it to other large data sources freely available on the Internet.
Two such sources were used for our analysis. The first, Google Trends13, allows measuring interest over time from a search engine perspective.14 We started by gathering evidence to support Mussbacher et al.'s claim of MDE being a niche technology by comparing interest in MDE and UML to interest in the suggested programming languages — Java and C#.15 Figure 1 does imply that, compared to Java and C#, both UML and MDE are fairly niche from the perspective of search engine querying.
Figure 1: Google searches for Java, C#, UML and MDE. Source: Author's plot using Google Trends data (January 2004 to August 2018)
Next, we analysed how queries related to MDE have evolved in the time dimension. For this we plotted interest over time for six different search terms related to MDE, with the results captured by Figure 2. The data allows us to make some general observations, as the graph clearly shows a marked decline from a peak around 2004, and a sharp descent soon thereafter; furthermore, it lacks any obvious upticks during the latter years, which would indicate some form of revival as the industry gets to grips with the approach.
It is important to understand that there are numerous limitations with our analysis, such as the clipped time horizon (available data excludes the period from 2000 to 2004), the reliance on Google as the sole data source (searches may have been performed on other search engines, though Google's dominant market position mitigates this risk), the exclusion of queries using acronyms (adding MDE, MDA, etc. to the report caused false positives), and so forth. Nonetheless, in the absence of better data, it serves as a coarse approximation.
Figure 2: Google searches related to MDE. Source: Author's plot using Google Trends data (January 2004 to August 2018).
Our second data source was Stack Overflow16, a question and answer website popular with software engineers17, which provides access to statistical data via its Insights query interface.18 Questions in Stack Overflow are tagged by its users to facilitate searching and aggregation19, and the distribution of tags can be analysed via Insights. The absence of evidence was informative in itself, as no tags could be located with regards to "model-driven", "MDE", "MDD", "MDSE" or any other MDE variant described in this dissertation. UML was the only available tag related to modeling, and the resulting data is plotted in Figure 3. The number of questions for UML as a percentage of total questions reached a maximum of 0.12% and has declined to around 0.02%.
Figure 3: Questions tagged with UML on Stack Overflow. Source: Stack Overflow Insights from 2009 to July 2018.
We then tried to establish UML's position when compared to Java and C#. The results, plotted in Figure 4, are in line with the findings from Google Trends: UML is quite niche when compared to popular programming languages.
As with Google Trends, it is important to highlight the limitations of Stack Overflow as a data source, since it excludes software engineers who need not ask questions about MDE (e.g. experienced practitioners) as well as those who use other sources of information (commercial product support, other web forums), questions may be incorrectly classified, questions may be tagged against specific tools rather than modeling terms, the data range is narrow (only covers the period from 2009 to 2018) and so forth. Similarly to Google Trends, these limitations were deemed acceptable for our purposes.
Figure 4: Questions tagged with Java, C# or UML on Stack Overflow. Source: Stack Overflow Insights from 2009 to July 2018.
In summary, the picture emerging from the analysed data is in overall agreement with those who view MDE as a niche technology, facing a trend of decreasing interest in industry. We shall now contrast these findings with evidence at the opposite end of the scale.
Analysis of Evidence at a Micro-Scale
A change of perspective is often revealing when addressing wicked problems (cf. Section Limitations). On what the authors claimed was the first wide-range industry study of its kind, Whittle et al. (Whittle, Hutchinson, and Rouncefield 2014) surveyed 450 MDE practitioners with the objective of characterising MDE practice, and their findings are in stark contrast to those described thus far. The authors were already aware of this discrepancy, stating (emphasis ours):
Some claim that the application of MDE to software engineering is minimal. MDE, they argue, is only used by specialists in niche markets. Our data refutes such claims, however. We have found that some form of MDE is practised widely, across a diverse range of industries (including automotive, banking, printing, web applications etc.
The unmistakable conclusion of Whittle et al.'s study is that MDE is widely used in industry, but in ways that are extremely difficult to quantify in the aggregate. For example, their analysis points out that practitioners prefer creating DSL over using general purpose modeling languages such as UML, and implement these DSL using a dizzying array of tools, techniques and frameworks (emphasis ours): "We found no consensus on which modeling languages or tools developers use — they cited over 40 modeling languages and over 100 tools as 'regularly used' in our survey." Whittle et al.'s analysis builds upon earlier work from Petre (Petre 2013), who showed that, in a sample of 50 software designers, very few used UML, and those that did — a total of 11 — used it only for "selective" purposes.20 Petre's findings are in agreement with the evidence from our macro-analysis. The picture that emerges from both of these studies is one of great diversity and fragmentation, which further reinforces our own views of MDE as a diffused body of knowledge (Craveiro 2021b) (Chapter 2). Furthermore, these results are not isolated; a related study by Hutchinson et al., with 250 respondents (Hutchinson et al. 2011), found similar heterogeneous patterns.
One can, of course, question the exact meaning of the expression "practised widely", given the small sample size and the fact that Whittle et al. surveyed only MDE practitioners — all of which makes it rather difficult to place their work in the context of the wider industry. In addition, as previously demonstrated, the boundaries of MDE are rather porous (Craveiro 2021b) (Chapter 2) so the criteria for classifying any given project as an "MDE project" — in their words, "some form of MDE" — is not entirely free of ambiguity; in the limit, any project using code generation or a DSL could be construed as a "MDE project", though that, perhaps, may not be in the spirit of the endeavour.21 Nonetheless, even when taking these validity threats into consideration, it is clear that their work provides undeniable evidence of MDE adoption across a variety of scenarios; more so than its niche status would imply.
Clark and Muller (Clark and Muller 2012) uncover a second reason that may help explain why it is difficult to find evidence of MDE use in the large, when they conclude that "the spirit of model-driven technology is very alive, although absorbed by mainstream programming environments […]." In their view, there has been a steady flow of ideas and concepts from MDE's body of knowledge to traditional programming environments, and this, they suggest, is a trend that is set to continue or even accelerate due to commercial demands (emphasis ours): "The next generation of companies making use of model-driven technologies might be more successful if they manage to hide model-driven technology, embedding it as a competitive advantage."22
Taking all of these views into account, the evidence at the micro-scale is consistent with the idea of MDE's body of knowledge being digested and repurposed into a series of technological changes that are fit for specific purposes. And, now that both macro and micro cases have been presented with contradictory results, we must attempt to reconcile these viewpoints.
Discussion
The evidence at a macro-level reinforces Mussbacher et al.'s intuition of MDE's niche status, whilst the evidence at the micro-level provides support for the idea that MDE is in widespread use but, crucially, not in accordance to its original vision.23 Both statements are not incompatible. Our opinion is that this vision is yet to come to pass, and its now unlikely to do so within the Bézivin horizon. Further: the limited available data seems to point out that the industry is moving away from that direction altogether, and is instead choosing the path of dispersing the MDE body of knowledge into small and localised solutions, and this may ultimately prove to be the enlightenment that has been long sought. If so, Cook's words now sound eerily prophetic (Cook 2006) (emphasis ours):
The notion that all of software development will somehow be replaced by modelling is at least as mistaken as "objects are just there for the picking". […] Specific kinds of models are useful in specific tasks; the modelling language used for a specific task must be designed to be fit for that task. Today’s increasing interest in Domain Specific Languages, rather than general-purpose modelling languages, clearly recognizes this.
With these wise words, to which we subscribe fully, we conclude our outline of the industry-wide adoption picture. It is now time to turn our attention to the analysis of themes emerging from MDE's application, in order to gain a better understanding of the specific challenges faced by those using the approach.
Empirical Analysis of Adoption Literature
A great deal of insightful information can be extracted from the MDE adoption literature, much of which is relevant to our work. The information is, however, in a form that is troublesome to analyse due to its qualitative nature. In order to address this problem, we decided to deploy the classification system put forward by Whittle et al. in (Whittle et al. 2013), and subsequently improved upon (Whittle et al. 2017). There, they describe it as "a loose taxonomy of tool-related considerations, based on empirical industry data, which can be used to reflect on the tooling landscape as well as inform future research on MDE tools."
Our previous brushes with the taxonomy revealed it to be malleable, amenable not only for the analysis of MDE tooling but also of MDE adoption issues in general (Craveiro 2021a) (Craveiro 2021c). This is likely a byproduct of the close relationship between MDE application and its tooling, for, as Mohagheghi and Dehlen perceptively noted, "[s]upporting MDE with a comprehensive tool environment is crucial, as many of the techniques promoted as necessary in MDE strongly depend on proper tool support." (Mohagheghi and Dehlen 2008)
Each of the sections below tackle one of the four top-level categories in the taxonomy. They start with a brief description of the category, in order to contextualise those unfamiliar, and proceed to analyse relevant issues gleaned from the adoption literature. As with Section How Widely Adopted is MDE?, the objective is not to perform an exhaustive review but instead to capture overarching themes that are pertinent to the present dissertation. In addition, due to its qualitative nature, we have chosen to rely on the same approach as did Whittle et al. — namely, the use of extensive quoting from original sources in order to register more faithfully the underlying themes.
Technical Factors
Technical issues that affect the adoption of MDE, such as the features available in tooling and their integration with development environments, were prominent in the literature. In particular, there seems to be a noticeable mismatch between developer expectations and functionality available in tools, as Clark and Muller explain (emphasis ours):
Industry would like to use MDD as a shrink-wrapped black-box process. Current technologies expose a great deal of the inner workings of PIM, PSM and transformation design. Developers feel that they need to have a detailed knowledge of all aspects of the technology which undermines its commercial value compared to the use of more trusted mature technologies such as compilers. (Clark and Muller 2012)
The compiler metaphor is particularly apt, since developers are used to code generators that behave in a fashion similar to compilers (Craveiro 2021c). Whittle et al.'s findings echoed similar thoughts, revealing that the available functionality often exceeds the typical needs of software engineers:
Our interviewees emphasized tool immaturity, complexity and lack of usability as major barriers. Usability issues can be blamed, at least in part, on an over-emphasis on graphical interfaces: "… I did an analysis of one of the IBM tools and I counted 250 menu items." More generally, tools are often very powerful, but it is too difficult for users to access that power; or, in some cases, they do not really need that power and require something much simpler: "I was really impressed with the power of it and on the other hand I saw windows popping up everywhere… at the end I thought I still really have no idea how to use this tool and I have only seen a glimpse of the power that it has." (Whittle et al. 2017)
The reliance on graphical interfaces appears to be a common complaint from developers: "There is a large amount of evidence that software engineers prefer textual representations for system artifacts rather than diagrams." (Clark and Muller 2012) The underlying thread that unifies all these observations is an impedance mismatch between how developers would like tools to behave versus how tool developers view the role of those tools. Whittle et al. express a concern for the egregious "[…] lack of consideration for how people work and think: 'basically it’s still the mindset that the human adapts to the computer, not vice-versa.'" (Whittle et al. 2017)
They go on to explain that many practitioners addressed this mismatch by creating their own special purpose tooling, with reportedly better results (emphasis ours):
The majority of our interviewees were very successful with MDE but all of them either built their own modeling tools, made heavy adaptations of off-the-shelf tools, or spent a lot of time finding ways to work around tools. The only accounts of easy-to-use, intuitive tools came from those who had developed tools themselves for bespoke purposes. Indeed, this suggests that current tools are a barrier to success rather than an enabler and "the fact that people are struggling with the tools… and succeed nonetheless requires a certain level of enthusiasm and competence." (Whittle et al. 2017)
Nevertheless, it is important not to underestimate the immense effort required to create industrial-grade MDE tooling, even when experienced practitioners are involved, as Paige and Varró's work amply demonstrates (Paige and Varró 2012), and so does Andolfato et al.'s (Andolfato et al. 2014). Speaking in the context of tool vendors, Clark and Muller put the matter in more forceful terms (emphasis theirs): "Tool development is expensive. However much you think it will cost in terms of time and effort to develop a business based on a modelling tool, multiply by 10. Be prepared to be patient and support development through other activities." (Clark and Muller 2012) Internal tool development does benefit from a narrower focus, of course, but it is no less difficult; the cost and the associated risk in developing new tools for a complex problem domain such as modeling — in most cases, a proposition that is entirely unrelated to the main business activity and the current engineering skill-set — is an obvious barrier to MDE adoption. Our own work sheds light on some of the challenges developers faced:
Interviewees developed an appreciation for the difficulty of creating a general purpose code generator: "[…] It was a massively ambitious project, right? [To] [b]uild a general purpose code generator, is a very, very difficult thing."
Once the magnitude of the task was understood, a natural process of de-scoping started to take place: "But when you say, 'I want to write a code generator that is going to work for everything', well then now you need to define what everything is. And how do you define everything? […] You can't, so you say 'right I'm guessing I'm going to need lists, but I'm guessing I won't really need dictionaries, I'm guessing it will [be] good enough just to have public setters but let's not worry about public/private, everything will be public and that's […] something reasonable that I can write a code generator for, within a year and a half and […] that'll have to do.' […]" (Craveiro 2021a) (Section 6.1)
Thus, the trade-offs between off-the-shelf tooling and internal tool development are very complex, and highly dependent on situational context. Whittle et al.'s words summarise the issues with technical factors rather aptly (emphasis ours):
It is ironic that MDE was introduced to help deal with the essential complexity of systems, "but in many cases, adds accidental complexity". Although this should not be surprising […], it is interesting to describe this phenomenon in the context of MDE. For the technical categories, in almost every case, interviewees gave examples where the category helped to tackle essential complexity, but also other examples where the category led to the introduction of accidental complexity. So, interviewees talked about the benefits of code generation, but, at the same time, lamented the fact that "we have some problems with the complexity of the code generated […] we are permanently optimizing this tool." (Whittle et al. 2017)
Internal Organisational Factors
The adoption of MDE takes place in the context of an organisation with its own distinctive structure, processes and procedures, as well as a unique culture. The literature clearly highlights both the importance and the difficulty in integrating MDE related infrastructure with what precedes it:
One interviewee described how the company’s processes had to be significantly changed to allow them to use the tool: a lack of control over the code generation templates led to the need to modify the generated code directly, which in turn led to a process to control these manual edits. Complexity also arises when fitting an MDE tool into an existing tool chain: "And the integration with all of the other products that you have in your environment…" Despite significant investment in providing suites of tools that can work together, this is clearly an area where it is easy to introduce accidental complexity. (Whittle et al. 2017)
This stumbling block was also observed by Mohagheghi and Dehlen: "Integrating a tool suite that satisfies these requirements into a coherent environment is evidently a challenge. In the MODELWARE project, a wide range of tools were used, but all partners experienced problems with instability of the tools and their integration." (Mohagheghi and Dehlen 2008)
Interestingly, whilst bespoke development facilitates tooling integration as the requirements are very specific to an organisation, it often has a clear downside with regards to the sustainability of tooling engineering because it lacks an alignment with core business activities. In general, developers have very little appetite for extraneous activities on an already congested development schedule, as we witnessed first-hand:
A key point was the difficulty in justifying continued investment on a bespoke code generator from a business perspective: "Its quite product-y? So it almost feels like, you know, its something which [we] should be buying in or open source […]. Not what you want to be focusing your interest on, if you can […] avoid it." (Craveiro 2021a) (Section 6.5)
Finally, there are also challenges due to the immaturity of the formal processes associated with MDE, as Mohagheghi and Dehlen note:
The importance of utilizing a defined process in software engineering has been known for several years. However, most "tried and tested" processes are not tailored for MDE, which does not make any assumptions on the software development process or the design methodology. Baker et al. report that many teams in Motorola encountered major obstacles in adopting MDE due to the lack of a well-defined process, lack of necessary skills and inflexibility in changing the existing culture […]. (Mohagheghi and Dehlen 2008)
External Organisational Factors
MDE adoption is also impaired by factors that are external to the organisation applying it. As already highlighted (cf. Section Analysis of Evidence at a Micro-Scale), the MDE tool offering is very large and diverse, bringing with it its own problems such as a difficulty in deciding on the appropriate tooling for a given project — much as did CASE before it (cf. Computer Aided Software Engineering (CASE)). In addition, once tools are identified, there are always fears of vendor lock-in, as Mohagheghi and Dehlen report: "[t]he vendor lock-in problem persuades some users to use open source tools such as the Eclipse framework. Others combine third-party products with self-developed tools […], or develop their own tools […]." (Mohagheghi and Dehlen 2008)
Relying on a vendor may also be problematic if the vendor is small, as Paige and Varró demonstrate by vividly narrating the many lessons they've learned whilst creating and shutting down two MDE tool vending start-ups (Paige and Varró 2012). Our key take-away from their work is that the software tooling market is generally very competitive and MDE tooling is no different, so, as part of any feasibility analysis, it is very important to take into consideration the sustainability of vendors far into the future. This problem is exacerbated because interoperability between tools of different vendors is not yet a fully resolved issue, even in the presence of mature industry standards such XML Metadata Interchange (XMI). In (Lundell et al. 2006), Lundell et al. analysed the impact of XMI on heterogeneous tooling environments and, whilst being generally positive about the standard, they also helped explain why interoperability remains such a thorny issue (emphasis ours):
In considering the results of the tests it should be noted that anything short of complete success is of limited value in practice. The work involved in repairing significant semantic loss in an interchanged model is often considered infeasible for industrial strength models. With this in mind, from the perspective of legacy systems and tool lock-in, the new generation of modelling tools has not generally improved prospects for importing existing models exported from earlier tools. (Lundell et al. 2006)
As a result, practitioners are often wary of finding themselves involved with what France and Rumpe called the "the DSL-Babel challenge"; that is, the fear that the "[…] use of many DSLs can lead to significant interoperability, language-version and language-migration problems." (France and Rumpe 2007)
Social Factors
An influential aspect of MDE adoption is concerned with issues of control and trust, particularly with regards to the tools of third-party vendors or of those produced by large internal tooling teams with a degree of independence from their end-users. Whilst vendors have historically tried to promote holistic top-down solutions — very much in line with the MDE vision (cf. Section How Widely Adopted is MDE?) — the adoption literature shows that small-scale, bottom-up approaches tends to yield better results in practice, as Whittle et al. explain (emphasis ours):
Our findings also lead us to believe that most successful MDE practice is driven from the ground up. MDE efforts that are imposed by high-level management typically struggle; interviewees claimed that top-down management mandates fail if they do not have the buy-in of developers first. As a result, there are fewer examples of the use of MDE to generate whole systems. Rather than following heavyweight top-down methodologies, successful MDE practitioners use MDE as and when it is appropriate and combine it with other methods in a very flexible way. (Whittle, Hutchinson, and Rouncefield 2014)
Regardless of whether tools are bespoke or sourced from external vendors, a common approach taken by developers to handle tooling inadequacies is to subvert the tools in order to achieve some desired behaviour:
These and other speculative features [immutability, factory methods] were not optional, perhaps in order to restrict variability, so as a consequence software engineers started to make use of them best they could: "I think its a good point, you just learn to live with what you got, right? […] These are the […] constraints that we have, so we're going to have to […] live with those constraints. […] And you find a way, right?" (Craveiro 2021a) (Section 6.3)
Whittle et al. recorded eerily similar experiences: "A second example is a company that mandated the use of a commercial MDE tool. However, the developers could not get the tool to fit their processes, and, under pressure to 'make things work’, they hacked it, messed with the generated code, and circumvented it when they had to." (Whittle, Hutchinson, and Rouncefield 2014)
And it is with those telling words — with pragmatism overriding theory to get things done — that we must end our swift excursion through Whittle et al.'s taxonomy, as well as through the larger terrain of MDE adoption literature. Let's us now gather a set of lessons learned from the analysis.
Discussion
The previous sections presented themes unveiled by empirical analysis in the MDE adoption literature. Of these, we would like to highlight those that are of key importance to MASD:
- Usability is often a concern with MDE tooling. There is often a mismatch between what developers expect of a tool and what vendors view as the role of the tool. Empirical evidence has shown this is a barrier for adoption.
- MDE tools are expensive to develop and maintain. Whilst bespoke tooling has a better fit, it is very difficult to develop due to the need for specialist skills and the costs involved.
- Developers prefer bottom-up and incremental approaches. Management and tool vendors seem to prefer top-down approaches, but software engineers prefer to integrate new tools and approaches incrementally, experimenting with them over time and increasing their usage as their mastery of the tool improves.
- Integration with existing tooling is a challenge. Software engineers typically use a variety of tools to achieve their goals, and continually pick up new tools over their career. In general, tools that integrate with existing toolsets are favoured over tools that force wholesale changes to development workflows.
Our conclusion from the analysis of MDE adoption is that there is a gap in the literature for methodology and tooling that are better aligned with how practitioners actually use MDE rather than researcher and tool developer's expectations. Now that the gap has been established, it is the role of the next chapter to distil these and other findings into a set of requirements designed to address this impedance mismatch.
Bibliography
Footnotes:
It may be argued that our deep interrogation of MDE is excessive. However, it is our firm opinion that its application — and the definition of methodologies based on it — can only be done successfully once its key strengths and weaknesses are well understood, and this requires looking at both theory and practice. To our knowledge, this is an undertaking thus far absent from the literature.
The original wording by Hutchinson et al. is rather more intriguing (emphasis ours): "Although MDE claims many potential benefits in terms of gains in productivity, portability, maintainability and interoperability, it has been developed largely without empirical support for these claims."
Our personal experience mirrors this effect quite closely (Craveiro 2021a) (Section 6 and 7).
As detailed in (Craveiro 2021b), Chapter 2 — and Section 2.4 in particular.
Indeed, in this sense we are closer to McBreen's Software Craftsmanship (McBreen 2002) rather than to Software Engineering. There is much to be said for McBreen's idea that, on the main, the creation of high-quality software may not be amenable to repeatable engineering processes, but sadly such discussion falls outside the remit of this dissertation.
Manuscripts such as Völter's "MD* Best Practices" (Völter 2009) capture well the idea.
For a partial timeline of events related to MDE, see Clark and Muller (Clark and Muller 2012), Section 2, "The MDD Landscape".
Articles such as Bell's "Death by UML Fever" (Bell 2004), Thomas' "MDA: Revenge of the Modelers or UML Utopia?" (Thomas 2004) and France et al.'s "Model-driven development using UML 2.0: promises and pitfalls" (France et al. 2006) accurately depict the zeitgeist.
The MDA Guide is well aware of this, stating (emphasis ours): "Automation reduces the time and cost of realizing a design, reduces the time and cost for changes and maintenance and produces results that ensure consistency across all of the derived artefacts." (Group 2014)
The Hype Cycle Model (Linden and Fenn 2003) is often deployed in this context — e.g. (Brambilla, Cabot, and Wimmer 2012) (p. 22), Torchiano et al. (Torchiano et al. 2012) — and understandably so, for, in the words of Torchiano et al., "[h]ype is frequently associated to software development processes/techniques until (sic.) they are not yet mainstream and fully understood; we think it is also the case for modeling and MD*." We opted for a dissenting view nonetheless, siding instead with those with concerns about the model such as Dedehayir and Steinert (Dedehayir and Steinert 2016). Our main qualm is the difficulty in determining whether we are in the model's final moment (i.e. the so-called "plateau of productivity") or if we have entered a perpetual descent in the "through of disillusionment". Without adequate quantitative datasets, statements on this regard are problematic to substantiate and therefore we do not believe the model brings any additional clarity towards the evolution of MDE adoption.
Note that due to the limitations already described (cf. Section Section Limitations), the purpose of this analysis is not a rigorous determination, but merely the use of multiple sources to get a sense of where the theory has led us thus far.
Google Trends defines interest over time as follows: "Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means that there was not enough data for this term."
These languages rank 2nd and 5th respectively in the TIOBE Index, at the time of writing (TIOBE 2021). The TIOBE Index offers a measure of popularity for programming languages.
According to internal data, Stack Overflow has a total of 16 million questions, 25 million answers, 9.2 million active users and over 9.8 million visits per day (Exchange 2018).
Stack Overflow defines tags as follows: "A tag is a word or phrase that describes the topic of the question. Tags are a means of connecting experts with questions they will be able to answer by sorting questions into specific, well-defined categories." (Overflow 2018)
Petre defines selective in the following, somewhat recursive, manner: "UML is used in design in a personal, selective, and informal way, for as long as it is considered useful, after which it is discarded." (Petre 2013)
A small sample of the inclusion criteria in the reviewed material should suffice to give a flavour of this dilemma. Hutchinson et al. stated (Hutchinson et al. 2011): "The study takes a deliberately broad interpretation of MDE, as it is intended to be exploratory. Therefore, all variants of MDE are covered, including both domain-specific modeling languages (DSMLs) and UML-based methods. […] The only hard criterion for excluding/including data was that the company must have been using models as a primary development artifact (sic)." For their part, Mohagheghi and Dehlen (Mohagheghi and Dehlen 2008) limit their study to approaches that generate "models, code and other artifacts from models".
These very words were inspirational to the MASD approach.
This vision is articulated clearly by France and Rumpe:
In the MDE vision, domain architects will be able to produce domain specific application development environments (DSAEs) using what we will refer to as MDE technology frameworks. Software developers will use DSAEs to produce and evolve members of an application family. A DSAE consists of tools to create, evolve, analyze, and transform models to forms from which implementation, deployment and runtime artifacts can be generated. Models are stored in a repository that tracks relationships across modeled concepts and maintains metadata on the manipulations that are performed on models. (France and Rumpe 2007)