jwilk 7 years ago

> This repo illustrates a way to manage a project using just plain Bourne sh, available on all Unix systems since 1977.

No, this code uses functions, which didn't exist in the original Bourne shell. They were introduced only in 1984.

Source: https://www.in-ulm.de/~mascheck/bourne/#variants

  • jperkin 7 years ago

    It also uses the "local" keyword, which is non-portable.

    People like to bash the autotools, but they exist for exactly this reason - they already spent decades working through all these gritty details to ensure the scripts generated are portable. To paraphrase Spencer "Those who do not understand autotools are doomed to re-invent them, poorly."

    • rossy 7 years ago

      That's the argument for autotools, but I wholeheartedly disagree. Untested code can be assumed not to work, so if a build script doesn't work on a certain platform, what's the likelihood that the C program it's building will work on that platform? When you see a poorly-written autoconf script checking for basic things like size_t, you have to wonder if the program it's building really supports pre-ANSI C. This wastes real time on not-so-uncommon platforms like Cygwin and MSYS2, where spawning subshells is slow and running ./configure can take minutes.

      So if you've never tested your program on a platform where /bin/sh doesn't understand local, feel free to use local in the build script. Bashisms like local are safer anyway. Variable scoping is good. (Though I guess the author's claim of it working in a 70s era Bourne shell is still wrong.)

      • jancsika 7 years ago

        > This wastes real time on not-so-uncommon platforms like Cygwin and MSYS2, where spawning subshells is slow and running ./configure can take minutes.

        That's true, but in my case it turned out to be insignificant. While changing all the autoconf junk in > 100 plugins could move the total build time from 50 minutes down to 40, I'd need an order of magnitude decrease in build time to warrant the work. I say order of magnitude because its only at a 5 minute build time that the entire dev process would see a paradigm shift-- at that point I could, with a straight face, stop shipping Linux binaries and just suggest that users compile the software themselves. (Of course that's not great usability for a whole class of Ubuntu users, but at build time == 5 minutes some other dev could easily come along and package things up with ease.)

        Also, the extra wasted time under Windows isn't the fault of autoconf. Try doing `git any_subcommand` in msys2 and you'll get latency so high you'd think it were a practical joke.

        Edit: added parenthetical

      • convolvatron 7 years ago

        you're missing another huge problem with autoconf and friends.

        if it doesn't just magically work, you're largely sol.

        sometimes you can put in a basic build in its place, or find some environment variables to fiddle with. but usually..

    • akkartik 7 years ago

      Uh-uh. Pray tell me an actual extant platform without `local`.

      Debian dash deems 3 POSIX extensions basic enough to be in /bin/sh -- including `local`. https://www.debian.org/doc/debian-policy/ch-files.html#s-scr...

      I don't care about some spherical-cow ideal of portability. It has its own costs -- like autotools. This is why OP makes no mention of POSIX or portability.

      (This exchange regurgitates https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake..., albeit with more mutual sneering.)

      • jperkin 7 years ago

        Any OS which use the Korn shell as /bin/sh, for example Solaris 10, 11, and illumos:

          $ /bin/sh -c "fail() { local f; }; fail"
          /bin/sh[1]: local: not found [No such file or directory]
          $ /bin/sh --version
            version         sh (AT&T Research) 93t+ 2010-03-05
          $ uname -rsv
          SunOS 5.11 joyent_20160721T174127Z
        
        It wouldn't work on Solaris 9 and older either as the original /bin/sh also doesn't support "local", but as those releases are EOL it's fine to discount them.

        Whether you regard these platforms as extant depends on your point of view. Most people wouldn't, and that's fine. We'd naturally disagree ;)

        • akkartik 7 years ago

          That's good to know, actually. Thanks! Sorry about my tone before. Are there any other platforms that actually benefit from autotools, that you're aware of?

        • JdeBP 7 years ago

          OpenBSD uses the Korn shell for /bin/sh and this is not true there.

              $/bin/sh -c "fail() { local f; }; fail"
              $
          
          OpenBSD's /bin/sh defines several aliases as standard.

              $alias local
              local=typeset
              $
          
          M. Agaram has stated elsewhere that xe was using OpenBSD sh as a poor man's POSIX conformance test.
          • jwilk 7 years ago

            > M. Agaram

            Who's that?

            • akkartik 7 years ago

              That's my last name. Perhaps the M stands for "Monsieur" :)

  • feelin_googley 7 years ago

    I just rewrote it without functions in about 5 min. It probably takes less than a minute for a faster typist.

    He may have used functions to keep this to a single file and I doubt his 1977 comment (now changed to "30 years") was to be taken literally.

    Anyone aware of the overzealous use of more complex build systems in situations where they are not necessary would understand his point. It is possible to build portable software without autoconf and makefiles.

    Example: http://nacl.cr.yp.to

  • akkartik 7 years ago

    Thanks for the correction!

edejong 7 years ago

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA).

No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG, you're constantly traversing from one side to the other, making the build 'work'.

  • brudgers 7 years ago

    There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository.

    Alternatives are not replacements.

    • edejong 7 years ago

      Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones.

      A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, CMake, Maven, SBT and many others.

      • moosingin3space 7 years ago

        To add to this, Google spent a lot of time and effort on building a custom build system (Bazel) to make this sort of dependency tracking in a mono-repo easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell scripts any day, but the OP appears to have been focusing on extremely simple projects, where shell scripts are okay.

        • brudgers 7 years ago

          Right, "works for me" doesn't mean "you must do it.".

      • brudgers 7 years ago

        My apologies for not expressing myself as clearly as I would like.

        Some systems don't need to scale. Make and SBT etc. have an impedance mismatch with some projects due to their learning curve and the current sophistication of the development team.

        The most important feature of any development tool is not whether it is best practice, it is whether it works. For example, Tarn Adams doesn't use version control for Dwarf Fortress. [1] That doesn't mean Microsoft should have switched the Windows codebase to zip files instead of Git any more than a single developer should start boiling the small lake of SBT just because it is better.

        Sometimes, it is easier and faster and better to just build the tool that is needed. And of course sometimes it isn't.

        [1]: https://www.reddit.com/r/IAmA/comments/1avszc/im_tarn_adams_...

  • antibjarne 7 years ago

    Maybe they're not using a language with insanely bloated compile times?

    • edejong 7 years ago

      It's not just about compile time. Couple of examples:

      - Parallel downloading / uploading of artifacts

      - Multi-step transformation (old-style: lexer to C to Object file to linker to deployment). You can turn this around, but generally you want each output of your lexer to be input to the C compiler and so on.

      - Testing (especially brute-force testing might take time)

      And in modern situations you have the same, with Typescript => Javascript => Minimisation => Combination => Deployment (roughly).

      It really helps to think of this as a dependency chain and use a partial-order planner to find a possible linearization and execute.

vog 7 years ago

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems?

To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.)

Python is a much cleaner language than any shell language, especially with regard to error handling. (yes, I'm aware of "set -eu") Also, if you occationally need some more elaborated data structures or algorithms, you can express these in a straight forward way, rather than a series of complicated (and usually very fragile) text processing steps.

  • majewsky 7 years ago

    I see you've not yet encountered Alpine-based Docker containers.

      $ docker run --rm -ti alpine sh             
      / # bash
      sh: bash: not found
      / # python
      sh: python: not found
      / # perl
      sh: perl: not found
      / # ruby
      sh: ruby: not found
    
    Sure, you can install all of these with a single `apk add`, but it makes your container much fatter, i.e. every CI job that has to pull the image will be taking a longer time. Same for when you push the image into a far-away datacenter.
    • wrs 7 years ago

      In a base Alpine container, it would be awfully hard to build anything interesting. You are probably going to need to install some build tools first anyway. (Thus the two-stage container pattern.)

    • user5994461 7 years ago

      It's one of the reason that alpine should not be used outside of tests and toys projects. It cannot be substituted to an usual distribution.

  • swiley 7 years ago

    Python never works right for me, having to deal with yet another dependency just to use the build system is obnoxious.

    Just use make. It's not hard and it works even for fairly large projects.

  • edwinyzh 7 years ago

    Speaking of using Python or Node.js as a language for a build process, there were so many times it failed when I trying to run the build script included in the github open source project I found interesting and downloaded to my local file system, and usually it's a project not written in my familiar language, and since I'm NOT familiar with Python nor node.js, so when the build script emit something like a dependency error, after several tries without success, I usually just give up.

  • akkartik 7 years ago

    Totally agreed; I'd use a different language if I needed even an array.

  • nailer 7 years ago

    Also most current Unixes have at least Bash. Bourne shell is surprisingly rare outside BSD.

    • kingmanaz 7 years ago

      Or Korn, which is better than Bash, IMHO.

  • adtac 7 years ago

    I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example?

    Array is the only one that I have found lacking, in my experience.

    • dom0 7 years ago

      > I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example?

      Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).

    • vog 7 years ago

      If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph).

      You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/

      And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.

      • svckr 7 years ago

        Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :)

            $ cat > f
            a c
            a d
            b c
            ^D
            $ tsort f
            a
            b
            d
            c
        • vog 7 years ago

          The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)

      • chenglou 7 years ago

        Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with.

        Is there any similar build system? I know of https://github.com/droundy/fac, but it's not cross-platform

        • dom0 7 years ago

          tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is

          1) reasonably easy to use 2) very fast 3) very correct

          Most build systems fail 3), usually 2) and often also 1).

          • CJefferson 7 years ago

            What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.

      • akkartik 7 years ago

        Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.

    • chubot 7 years ago

      Hash tables are useful for expressing build variants -- some examples at the end of this comment:

      https://news.ycombinator.com/item?id=15043633

      debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.

    • johnt15 7 years ago

      Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.

    • cbcoutinho 7 years ago

      I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.

boomlinde 7 years ago

This leaves the topological sorting of the tasks to the programmer? This is neat, but that's one very useful feature of make that's missing.

  • akkartik 7 years ago

    Yes, that's true. I'm still noodling on that, primarily to enable parallel builds.

    Outside of parallel builds, however, I actually prefer an explicit sequence of steps to so-called "declarative syntax". The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway. And it's better for the reader as well to see the commands laid out in order, when they need to debug breakage on their setup. People should stop pretending to be Vulcans and just write a script for building their project, and then sprinkle in `older_than` directives as an optional optimization. Building from scratch should be the case to optimize for readability, rather than some exceptional situation.

    [1] https://github.com/zetavm/zetavm/commit/4e5e2e9d8b

    [2] https://github.com/tekknolagi/stackx/commit/5999202423

    [3] Fun fact: when you try to build OpenBSD's userland from source, the top-level Makefile always unconditionally calls `make clean`. I've seen this pattern a few times: large projects find themselves up shit creek in the build system, and it's difficult enough to test and debug the dependency graph that they give up all performance and retreat to just recompiling everything, just to be safe.

    • vog 7 years ago

      > The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway

      This is why the more elaborate build systems recognize dependencies automatically. For Makefiles, usually some sub-Makefile is generated automatically and then included. For language-specific build systems like OPAM (OCaml) or Cargo (Rust), this is one of their core features: As a programmer, don't care about maintaining dependency definitions! Whenever a source file is changed, the build system recalculates the relevant part of the dependency graph as well.

zimpenfish 7 years ago

How does this compare to e.g. @apenwarr's minimal `do`?

pskocik 7 years ago

I'm also using (for now) something based on these time checks (also pure posix + local, but local is extremely portable (tests well on 7 shells), though mine's a little more involved (automatic build job timing, controlled concurrency, robust failure/cancellation with autodeletion, fancy logging, filterable implicit dependencies, dependency + counting). It scales well to about 1000 source files. Then the lag starts to feel noticable.

reacweb 7 years ago

One of the benefits I see for this approach is the possibility to handle spaces in filenames (AFAIK very difficult with gnumake). Sadly the provided function does not accept spaces in filenames.

sgt 7 years ago

Somehow I find it slightly amusing that I see ./a.out in 2017.

lauretas 7 years ago

"notabug.org"... interesting.

Anyway, how does this compare, features-wise, with GNU Make? For example with make you have `./configure` files, do I also have them here?

  • majewsky 7 years ago

    > For example with make you have `./configure` files, do I also have them here?

    The concept of `./configure` is 100% orthogonal to Makefiles. You can have Makefiles without `./configure`, and you can have `./configure` that outputs something other than a Makefile. For example, I could easily imagine a `./configure` that prints a shell script like the one in the submission.

  • gardnr 7 years ago

    Looking as the sh code, it appears that it will work without tab characters. https://stackoverflow.com/a/2131227/1144060

    So there's that.

    • akkartik 7 years ago

      As it happens, I was moved to create this standalone repo showing off my current build setup after writing this comment: https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me...

      So yes, eliminating tabs (or rather the execrable requirement of using tabs) was absolutely a prime goal.

      • Symbiote 7 years ago

        Put this in ~/.editorconfig

            [Makefile]
            indent_style = tab
        
        And install the plugin, which is available for most editors and IDEs.

        http://editorconfig.org/

      • falcolas 7 years ago

        It's not a requirement - I give you: .RECIPEPREFIX

        > If you prefer to prefix your recipes with a character other than tab, you can set the .RECIPEPREFIX variable to an alternate character

        That said, how hard is it really to disable tab expansion in your editor? I have a block of configuration dedicated to the different indentation requirements of different programming and configuration languages. In more modern editors, I don't even have to do anything at all to edit Makefiles with tabs - they come pre-configured to do the right thing.

  • beojan 7 years ago

    Configure scripts are part of autotools, not make.

  • akkartik 7 years ago

    The whole goal of this approach is to get out of the vicious cycle of build files generating other build files, so no `configure`.

    `configure` is not in itself a feature. What purpose does it fulfill for you? Then we can discuss how we may serve that purpose in some more lightweight manner.

    • blux 7 years ago

      I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'. This works in some cases, but certainly not all of them. configure is one of them.

      Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that.

      • akkartik 7 years ago

        "I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'."

        You're certainly right about that:

        http://akkartik.name/about

        http://akkartik.name/post/libraries2

        https://lobste.rs/s/to8wpr/configuration_files_are_canary_wa...

        :)

        "This works in some cases, but certainly not all of them."

        This too I'm totally willing to accept. I believe most abstractions are prematurely frozen and so poorly designed. But this makes me liable to err too far in the other direction.

        "configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that."

        Here we part ways. I think `configure` is one of the easiest cases for me to be sure of, and it's because I don't care about "all previously supported platforms". Many if not all of them are obsolete. Also, as rossy pointed out elsewhere in this thread (https://news.ycombinator.com/item?id=15045076), it's impossible to know today if there's a bug in autotools for some rare platform.

        We live in a world very different from the one autotools were built for, and it's almost a monoculture at this point. All that crap is utterly unnecessary today. I don't care about building software on all possible platforms, all I care about is building it on this one platform in front of me right now.

        The most frustrating thing about autotools is that it's unclear which platform each individual check is concerned with. I have a whole mess of rules from 20 years ago -- and zero rationale for any of the rules. That makes even the few rules that are worth having a net liability because I can't separate them from all the useless ones.

TheDong 7 years ago

> The build script is easy to repurpose to your needs

No, it's not. This isn't under a free or open source license. The lack of license means that only an idiot would use it for anything.