In my experience, the two things that to the largest degree has improved software development productivity the last 20 years is revision control and continuous integration. Back in the nineties, the many did not use version control at all, or used cvs or commercial alternatives that all made it a pain to merge the changes from different developers. Later svn came along, and then finally git and other distrubuted version control systems came along and made it very much easier for a large number of programmers to work on one codebase.
Continuous integration, and in the tools for assisting doing that has made it much easier to make sure you able to build the software you are developing at any time. At less organized places it could be weeks or even months where there did not exist a workable version of the software, and integration was a pain. Sure this still happens some places, but now it is totally avoidable.
Unfortunately, almost none of the academic studies seems to take the tools seriously unless you are talking about compilers. I think that tools still is the area where most gains in software development productivity can be found.
I agree. I think the next steps towards improved software is TDD and functional programming. Possibly along with higher levels of test automation as well.
When a system can be automatically proven to meet the requirements a business has of it, then much of the need for "reconciling perspectives" is removed. The perspectives that matter is the business perspective of what they need, and the technical perspective of how to build it. Automated tests in between those two will help solidify the matching of those perspectives.
Although I haven't been in the industry for what others may call a long time, I feel that after working at a few large enterprises, this is something that upper management should read and understand in order to better relate to those actually performing the development of projects.
Agile is the newest and most widespread buzzword in my current role and its practices are being shoved down every team's throat. Those who adopt look like better performing teams to management and become the favorite and most flaunted teams so much to be idolized by others.
Meanwhile some of the smartest and proven teams are clearly producing immense value, and ignore a lot of new methodology mumbo jumbo to continue operating the way they have been for years.
It's an interesting dynamic, and I believe this paper captured a lot of what I've been experiencing, but was unable to convey into words.
Are you saying that the high-performing teams are continuing to work as they were, and the other, presumably lower-performing, teams being made to act differently? That doesn't sound too bad, at least on its face.
working on stories instead of layers you push out finished ux and features early on, and that's what both manager loves but ultimately kills the speed after six month to a year, because those need to be maintained forever but it's very rare to actually squeeze teme to refactor and even if you do, speed slows down as you refactor
The team spending less on the ui and more on making sure they get the datamodel right and the abstraction thight can quickly built an ui over the stable platform and overall will have better speed past a year/year and an half depending on the project
Agile is not about improving performance - its about removing the burden of efficiency proof from management. Pre-agile every manager had to be constantly on the prowl to get "numbers" and dates from teams, to gain a metric, to show progress, to demonstrate them actually having a place and job in the whole cooperate process.
Now they can point at realized features and make a funny progress bar from ice-box to Tested.
That this progress bar might hide tech-debt, that some realized modules might have to be silently rewritten to accommodate follow up stories- and thus that one last feature might be as expensive as all that have gone before.
Well that is treacherous developers, slacking off.
It also trades strong global deadlines of the "your project must be done by X" type for weak local deadlines of the "this feature should be done by X" type. That is custom-made for projects that face scope creep.
But, yes, it's impossible for developers to create an overall plan, or for the developers to decide to pay back technical debit. But can still be done by the client, but those normally don't.
These are mostly solved problems (minus omniscience or scale) with just handfuls of books that represent most parts of the solution.
Workaday environments suffer impediments to SE application not solvable by tooling. Many large organizations run de jure or de facto Need-to-Know (NtK) departments. I have witnessed Functional or Class Decomposition and Module Hiding repurposed for NtK atomized development teams not knowing shared OUTCOME semantics. Organizations and people turn SE abstraction boundaries into NtK semantic ASYMMETRIES, i.e., SECRECIES.
That does not work well, efficiently, or predictably. Anyone can rightfully detect motivations for both FSF/OSS and Skunkworks structured "startup" teams where all wood is behind the same arrow.
Other problems are lack of SE training. There are big tugs in finite training days between reading/writing skills, problem domain maths, tool churn, machine models, scheduling techniques and so on. Many HN Skunkworks styles teams are in VERY good shape compared to 20 years ago or contemporary organizations trying to turn SE into contract management.
I believe the title is misleading out of context, I think it is meant to lack conclusions or clear statements and simply outline some areas for discussion ... a proposition paper?
I'm not familiar with what that really means, just jumping to an assumption about the intent of the paper that I think was not made clear in the title as is.
In that case I won't waste my time reading this, but submit my outline of a Useful Theory of Software Engineering:
It comes down to making a good choice in each of the following areas. (My choice is in parens, along with my comments.)
Cloud Platform
(For historical reasons I am on Azure. I have some experience with Google. It does not hurt of course to work in several, but the cognitive overhead of gaining a real in depth devops knowledge on very many platforms will be overwhelming for many people, especially as the vendors invent stuff on the fly.)
Relational Database Management System
(Strive to learn SQL and 1 or 2 platforms, mine is SQL Server, as best as possible. There is only one useful data model, and it is the relational model. You will store data with other means over your career, but learning them is mostly trivial, and if your are intentionally using a different data model than relational, you should have a good reason. Also learn to data model using natural keys. This is a huge topic unto itself. The main justification is it will simplify the queries you end up using. A friend of mine is working on a book on nat keys. There is a real shortage of good literature on this topic.)
An application programming language
(Mine is F#. Learn one or 2 languages well, chosen because the compiler and canonical workflows do so much to keep you out of trouble. You will have to know 1 or 2 scripting languages our of necessity. Given the other areas you need real expertise in, I am not a fan of spending a lot of time with the language of the month club. Become an expert in the language(s) you use. It's a career-long pursuit.)
Project Management
(Recently saw a good overview video of the "no estimates" approach. I was sceptical before I saw it. Not really impressed by Agile, but I don't want to wade into that swamp.)
Specification Language
(A new area for me. I am interested in TLA+, but I'm still learning it. The idea of a specification language that delivers an unambiguous spec at an arbitrary level of granularity that you can manipulate and refactor is very appealing to my sense of propriety. We spend too much time going straight to prototype in our favorite language, refactoring it, etc. TLA+ forces you to focus on actual spec and not implementation details. The prototype approach leads us all to confuse implementation and spec.)
I think the call was for
something more than 'throw this shit togheter hope it stick'
And, to be perfectly honest, the technological stack is maybe the minor part of engineering a software system, both because compositions at different scales are already well defined and because the actual complexities are outside software selection and well into the user-software-developer interactions and specifically in non functional requirementd.
This isn't really a theory but a set of tools that help you do your job. A theory would gives a means to make predictions about software engineering projects. It would naturally give us a way of working such that projects don't fail.
But I don't disagree with you. It's important to learn those things.
The systematic study of methods for producing high quality software. Your comment is equivalent to calling a flexural test machine 'materials engineering'.
Granted. I'll throw out the cloud platform, but keep the others.
1) There is no substitute for the relational model. Clearly it is a paradigm for building mathematical objects.
2) I didn't strongly advocate my choice of F#. Strongly-typed, functional first, ML syntax language is the sweet spot today. In theory this predicts faster development and fewer bugs. All software assemblies are mathematical objects, those built with strongly-typed, functional languages tend to be more humanly comprehendible.
3) No estimates allows practitioners to make useful projections of the course of a project. (Of the 4, this one has the weakest claim to belong to theory, yet I will argue useful projections are a kind of prediction.)
4) TLA+, or something like it, separates spec from implementation detail and makes "predictions" (at arbitrary level of granularity) of what a completed software component will do.
It's not just you. I found it very abstract and... theoretical, even compared to other theoretical CS papers. The wording is vague and verbose in some parts, and definitely has a feel of being padded and "business-speak"-ish.
Even the body...as I was reading it, when I kept seeing new acronyms defined and used it sounded almost comedic. It reminded me of some talk or something I've seen somewhere where the made up acronyms end up being really funny description words like ("and this we call a PAININTHEASS process"). Can't remember what that was though.
In my experience, the two things that to the largest degree has improved software development productivity the last 20 years is revision control and continuous integration. Back in the nineties, the many did not use version control at all, or used cvs or commercial alternatives that all made it a pain to merge the changes from different developers. Later svn came along, and then finally git and other distrubuted version control systems came along and made it very much easier for a large number of programmers to work on one codebase.
Continuous integration, and in the tools for assisting doing that has made it much easier to make sure you able to build the software you are developing at any time. At less organized places it could be weeks or even months where there did not exist a workable version of the software, and integration was a pain. Sure this still happens some places, but now it is totally avoidable.
Unfortunately, almost none of the academic studies seems to take the tools seriously unless you are talking about compilers. I think that tools still is the area where most gains in software development productivity can be found.
I agree. I think the next steps towards improved software is TDD and functional programming. Possibly along with higher levels of test automation as well.
When a system can be automatically proven to meet the requirements a business has of it, then much of the need for "reconciling perspectives" is removed. The perspectives that matter is the business perspective of what they need, and the technical perspective of how to build it. Automated tests in between those two will help solidify the matching of those perspectives.
Although I haven't been in the industry for what others may call a long time, I feel that after working at a few large enterprises, this is something that upper management should read and understand in order to better relate to those actually performing the development of projects.
Agile is the newest and most widespread buzzword in my current role and its practices are being shoved down every team's throat. Those who adopt look like better performing teams to management and become the favorite and most flaunted teams so much to be idolized by others.
Meanwhile some of the smartest and proven teams are clearly producing immense value, and ignore a lot of new methodology mumbo jumbo to continue operating the way they have been for years.
It's an interesting dynamic, and I believe this paper captured a lot of what I've been experiencing, but was unable to convey into words.
Are you saying that the high-performing teams are continuing to work as they were, and the other, presumably lower-performing, teams being made to act differently? That doesn't sound too bad, at least on its face.
Flashy != Productive
working on stories instead of layers you push out finished ux and features early on, and that's what both manager loves but ultimately kills the speed after six month to a year, because those need to be maintained forever but it's very rare to actually squeeze teme to refactor and even if you do, speed slows down as you refactor
The team spending less on the ui and more on making sure they get the datamodel right and the abstraction thight can quickly built an ui over the stable platform and overall will have better speed past a year/year and an half depending on the project
Agile is not about improving performance - its about removing the burden of efficiency proof from management. Pre-agile every manager had to be constantly on the prowl to get "numbers" and dates from teams, to gain a metric, to show progress, to demonstrate them actually having a place and job in the whole cooperate process.
Now they can point at realized features and make a funny progress bar from ice-box to Tested. That this progress bar might hide tech-debt, that some realized modules might have to be silently rewritten to accommodate follow up stories- and thus that one last feature might be as expensive as all that have gone before. Well that is treacherous developers, slacking off.
It also trades strong global deadlines of the "your project must be done by X" type for weak local deadlines of the "this feature should be done by X" type. That is custom-made for projects that face scope creep.
But, yes, it's impossible for developers to create an overall plan, or for the developers to decide to pay back technical debit. But can still be done by the client, but those normally don't.
A similar paper: http://alistair.cockburn.us/Characterizing+people+as+non-lin...
The book "Peopleware" is also worth reading.
Unfortunately it seems that nobody has done any serious anthropology on software production with a view to improving it.
Is there more anywhere?
I think the ideas are right, but after telling me so, the paper enda without a theory.
I was able to find an open version of the paper mentioned from ResearchGate: https://www.researchgate.net/profile/Steve_Adolph/publicatio...
Here's you two on a theory of programming:
http://www.cs.toronto.edu/~hehner/aPToP/
https://bertrandmeyer.com/2015/07/06/new-paper-theory-of-pro...
Ooh a free book. Thank you, this looks promising :) have you read it?
Theory of Programs was a nice link. Other one was above my head.
I agree. Given the author, it makes more sense with some sense of DbC but it's a slog.
There's quite a few papers on Reconciling Perspectives but behind Elsevier paywalls.
Reference [21] is written by the same author and is a much more detailed thesis about the theory of "Reconciling Perspectives".
I haven't read all of it, but it's available on the University of British Columbia website:
https://open.library.ubc.ca/cIRcle/collections/ubctheses/24/...
These are mostly solved problems (minus omniscience or scale) with just handfuls of books that represent most parts of the solution.
Workaday environments suffer impediments to SE application not solvable by tooling. Many large organizations run de jure or de facto Need-to-Know (NtK) departments. I have witnessed Functional or Class Decomposition and Module Hiding repurposed for NtK atomized development teams not knowing shared OUTCOME semantics. Organizations and people turn SE abstraction boundaries into NtK semantic ASYMMETRIES, i.e., SECRECIES.
That does not work well, efficiently, or predictably. Anyone can rightfully detect motivations for both FSF/OSS and Skunkworks structured "startup" teams where all wood is behind the same arrow.
Other problems are lack of SE training. There are big tugs in finite training days between reading/writing skills, problem domain maths, tool churn, machine models, scheduling techniques and so on. Many HN Skunkworks styles teams are in VERY good shape compared to 20 years ago or contemporary organizations trying to turn SE into contract management.
> These are mostly solved problems (minus omniscience or scale) with just handfuls of books that represent most parts of the solution.
Could you list some of those books? Would love to add them to my reading queue.
Is it just me, or does this paper lack any real "meat"?!
I believe the title is misleading out of context, I think it is meant to lack conclusions or clear statements and simply outline some areas for discussion ... a proposition paper?
I'm not familiar with what that really means, just jumping to an assumption about the intent of the paper that I think was not made clear in the title as is.
In that case I won't waste my time reading this, but submit my outline of a Useful Theory of Software Engineering:
It comes down to making a good choice in each of the following areas. (My choice is in parens, along with my comments.)
Cloud Platform (For historical reasons I am on Azure. I have some experience with Google. It does not hurt of course to work in several, but the cognitive overhead of gaining a real in depth devops knowledge on very many platforms will be overwhelming for many people, especially as the vendors invent stuff on the fly.)
Relational Database Management System (Strive to learn SQL and 1 or 2 platforms, mine is SQL Server, as best as possible. There is only one useful data model, and it is the relational model. You will store data with other means over your career, but learning them is mostly trivial, and if your are intentionally using a different data model than relational, you should have a good reason. Also learn to data model using natural keys. This is a huge topic unto itself. The main justification is it will simplify the queries you end up using. A friend of mine is working on a book on nat keys. There is a real shortage of good literature on this topic.)
An application programming language (Mine is F#. Learn one or 2 languages well, chosen because the compiler and canonical workflows do so much to keep you out of trouble. You will have to know 1 or 2 scripting languages our of necessity. Given the other areas you need real expertise in, I am not a fan of spending a lot of time with the language of the month club. Become an expert in the language(s) you use. It's a career-long pursuit.)
Project Management (Recently saw a good overview video of the "no estimates" approach. I was sceptical before I saw it. Not really impressed by Agile, but I don't want to wade into that swamp.)
Specification Language (A new area for me. I am interested in TLA+, but I'm still learning it. The idea of a specification language that delivers an unambiguous spec at an arbitrary level of granularity that you can manipulate and refactor is very appealing to my sense of propriety. We spend too much time going straight to prototype in our favorite language, refactoring it, etc. TLA+ forces you to focus on actual spec and not implementation details. The prototype approach leads us all to confuse implementation and spec.)
I think the call was for something more than 'throw this shit togheter hope it stick'
And, to be perfectly honest, the technological stack is maybe the minor part of engineering a software system, both because compositions at different scales are already well defined and because the actual complexities are outside software selection and well into the user-software-developer interactions and specifically in non functional requirementd.
This isn't really a theory but a set of tools that help you do your job. A theory would gives a means to make predictions about software engineering projects. It would naturally give us a way of working such that projects don't fail.
But I don't disagree with you. It's important to learn those things.
This does not describe software engineering, as that term is academically defined.
and that is?
The systematic study of methods for producing high quality software. Your comment is equivalent to calling a flexural test machine 'materials engineering'.
Granted. I'll throw out the cloud platform, but keep the others.
1) There is no substitute for the relational model. Clearly it is a paradigm for building mathematical objects.
2) I didn't strongly advocate my choice of F#. Strongly-typed, functional first, ML syntax language is the sweet spot today. In theory this predicts faster development and fewer bugs. All software assemblies are mathematical objects, those built with strongly-typed, functional languages tend to be more humanly comprehendible.
3) No estimates allows practitioners to make useful projections of the course of a project. (Of the 4, this one has the weakest claim to belong to theory, yet I will argue useful projections are a kind of prediction.)
4) TLA+, or something like it, separates spec from implementation detail and makes "predictions" (at arbitrary level of granularity) of what a completed software component will do.
Lots of systems software has no place for any kind of RDBMS, I'm not really sure I understand what you mean by "no substitute".
Yep. Didn't think through writing that, did I?
It's not just you. I found it very abstract and... theoretical, even compared to other theoretical CS papers. The wording is vague and verbose in some parts, and definitely has a feel of being padded and "business-speak"-ish.
The abstract reads like something that could be generated by an automated paper generator.
Even the body...as I was reading it, when I kept seeing new acronyms defined and used it sounded almost comedic. It reminded me of some talk or something I've seen somewhere where the made up acronyms end up being really funny description words like ("and this we call a PAININTHEASS process"). Can't remember what that was though.
(2013)