I'm one of the developers of esy, would be happy to answer any questions about it.
I have background in JS ecosystem and always liked how npm/yarn allows you to have a set of dependencies installed per project which do not conflict with any other global installations. Then this goes even beyond runtime dependencies and you have tools like eslint/flow and others installed per project in node_modules.
Now enter native development and you won't find the same workflow there. This is what esy tries to solve. It focuses now on Reason programming language but conceptually it works with other platforms/tools too (for example some of our dependencies are C/C++ code and it works nicely with esy too).
Also I want to highlight the following points:
- All dependencies (and devDependencies) are local to a project (like with npm/yarn)
- The cache for built packages is shared between different projects - if you built OCaml 4.06.0 compiler once then the other project will just reuse it (this works similar to Nix package manager)
- To execute some tool within the isolated project environment you prefix it with esy command: "esy ocaml", "esy vim", "esy code"
- esy can install packages from npm, opam (package registry for OCaml ecosystem), github
- esy allows you to develop multiple packages at once (see linking workflow in docs).
- finally... we support Windows (besides Linux and macOS)!
Hi Andrey, can you tell me if this understanding is correct - ? I can use Esy to write native OCaml/Reason, but has no use in a compile-to-js environment - there it is npm/yarn. With Esy I will be using Dune as my build tool, but instead of the opam binary I can use Esy itself which can pull packages from the opam repository. It takes care of both packages and compiler environment just like opam.
The `link` directive is much more ergonomic than `npm link`. Nice.
Is there anything in Esy that would make porting native OCaml packages (like say https://github.com/c-cube/qcheck) to Bucklescript?
It would also be wonderful to see it in action through a short video tutorial.
> I can use Esy to write native OCaml/Reason, but has no use in a compile-to-js environment - there it is npm/yarn.
If you use js_of_ocaml (a compiler from OCaml/Reason to JS) then esy works with it very well (with the help from dune). With Bucklescript (another OCaml/Reason to JS compiler) it's more complicated (see below).
> With Esy I will be using Dune as my build tool, but instead of the opam binary I can use Esy itself which can pull packages from the opam repository. It takes care of both packages and compiler environment just like opam.
This is correct!
But not necessarily dune (though this is a very good choice), you can use almost any build system you want.
> Is there anything in Esy that would make porting native OCaml packages (like say https://github.com/c-cube/qcheck) to Bucklescript?
Unfortunately not, esy works on level higher than build systems, it doesn't built code itself but rather calls an actual build system to do the workd (dune, make, cmake, ...). Bucklescript is unique as it tries to own the whole stack and thus you have to make code compatible with its build system (bsb) manually.
One thing which esy could help though right now with Bucklescript is providing native tools like ppx'es (preprocessors for Reason and OCaml code) or tools for editors like Merlin (autocompletion and etc.).
> It would also be wonderful to see it in action through a short video tutorial.
Awesome work! I've not had the greatest experience with Opam in the past for the reasons you mention - after using Cargo for a long time, it was a bit of a shock! This could do wonders to improve things!
Some questions:
- Is there any issues with sharing the same config file with another package manager? Ie. `package.json`?
- On a similar vein, why JSON, and not a config format that supports comments, like TOML?
- Will it be able to do Coq projects as well? I've not had a great experience trying to use Opam with it, and had to resort to using submodules. D:
> - On a similar vein, why JSON, and not a config format that supports comments, like TOML?
Yeah, we thought of that but decided to use JSON for now as it's easy to publish
esy projects to npm - you already have package.json ready for publish! We might
add support for comments to package.json soon though.
> - Will it be able to do Coq projects as well? I've not had a great experience trying to use Opam with it, and had to resort to using submodules. D:
I think it's possible now (I only use it install Coq itself to work through Software
Foundations) but I'm not sure about the details yet.
But considering that Coq ecosystem is moving to dune (the build system of choice
in OCaml/Reason native ecosystem) and esy supports dune very well I think Coq is
going to have a first class treatment soon in esy!
EDIT: re JS compat: you can esy.json instead of package.json and then esy will use it instead. That way you can use package.json with yarn/npm.
thanks for yor work on esy, everytime it comes up I have a look at the documentation and the latest developments. It looks like a nice project, but I'm still a little confused about its purpose. You mention your JS background, so is it all about familiarity to npm? Or are there other things you gain when you run esy on top of dune?
And my other question? It doesn't run with bucklescript - right? Are there specific reasons?
> It looks like a nice project, but I'm still a little confused about its purpose. You mention your JS background, so is it all about familiarity to npm?
I think familiarity with npm is important here!
I mentioned that npm allows you to have all dependencies of the project local.
Now if we want the same experience for a native project it gets more complex: we can't just install OCaml/Reason/C/C++ code into node_modules, it's not going to work.
So what esy does is it constructs for each package a "perfectly" isolated build environment based on dependencies declared in package.json, which is then used to produce build output (executable or library). Such environment contains things like `$PATH`, `$OCAMLPATH`, `$PKG_CONFIG_PATH` and etc, so these are all environment variable which C/OCaml/Reason/... software can understand.
Now because such isolated build environment depends only on the package.json (I simplify a bit here... later on that) and the build output depends only on its build environment we can easily compute a cache key for each package build output just by looking at package.json of the package.
So esy has smart cache for builds which makes each package build work like this:
let KEY = hash(package.json) # this is very cheap
if ~/.esy/store/KEY
done
else
build(package.json, ~/.esy/store/KEY) # this is not cheap!
It means in practice that if you have completely separate esy projects which
depend on the same version of `ocaml` (or `gcc` or any other package) then esy
will build that version of `ocaml` just once. So those "every local"
dependencies are super cheap once you have esy cache warm.
(the key isn't really hash(package.json) but more like hash(package.json +
esy.lock) as we need to resolve dependency constraints to concrete versions
first using "esy install" command)
> Or are there other things you gain when you run esy on top of dune?
The correct comparison is to dune on top of opam, I think (opam, the command line program, not opam, the registry as esy supports opam registry too!):
- with opam you manage switches manually (switches are like installed package sets), if you need a switch per project - you create a local switch per project
- with opam every time you create switch you need to compile everything from scratch, with esy we try to reuse as much as possible from the build cache
- you use dune with esy the same way you use dune with opam but you need to prefix dune invocations with esy: "esy dune ...", so that way dune sees esy build environment
- with esy you can pull packages from npm/opam/github/tarballs/...
- there's mechanism called `link:`[2] which allows to develop multiple packages as parts of the same project (so called monorepo workflow).
- ...
Some of these things are possible with opam too (not sure about smart cache though and I think it's pretty important) but with esy we've tried to provide the same experience as when you develop with npm/yarn (of course we wanted to solve gigantic node_modules issue and thus we have pnp, and flaky dependency constraints and thus we have a proper SAT solver, the same used by opam). So you might consider this a different UX to develop native code which is focused on Reason/OCaml.
> And my other question? It doesn't run with bucklescript - right? Are there specific reasons?
It doesn't integrate with bucklescript very well now but you still can use esy with bucklescript projects so you can install Merlin (IDE assistant) or other tools from Reason / OCaml ecosystem.
The way how bucklescript works right now isn't compatible with esy - for example we use pnp[1] installation mechanism (same as yarn) so we don't produce node_modules and thus we are fast and package sources are cached globally - for now bucklescript doesn't understand how resolve package sources from pnp.js runtime and thus it can't work with esy directly.
If bucklescript implements support with pnp then it will be perfectly usable with esy but for now you can use package.json for npm/yarn for bucklescript dependencies and esy.json (esy looks for esy.json before package.json) for
native dependencies.
> - with opam every time you create switch you need to compile everything from scratch, with esy we try to reuse as much as possible from the build cache
Yes, that's definitely useful.
> [...] for now you can use package.json for npm/yarn for bucklescript dependencies and esy.json (esy looks for esy.json before package.json) for native dependencies.
So by acting like we use esy on Bucklescript, when we actually use npm, we get a very similar result?
> So by acting like we use esy on Bucklescript, when we actually use npm, we get a very similar result?
Yeah, we use npm in that case just to fetch deps and layout them in node_modules which bucklescript understands. Otherwise we use esy for native built tools (ocamlmerlin, preprocessors and etc).
Just wanted to say thank you for all your work on this, Andrey!
I've been using for developing native projects with ReasonML / OCaml (on Windows!) - a framework called Revery [1] (a fast, native alternative to Electron) and a re-write for our Oni [2] text editor using this platform. It really wouldn't have been possible to pursue these paths without esy.
We're also starting to see other cross-platform Reason-native projects pop-up that I'm very excited about, like Brisk (native OSX UI w/ Reason/OCaml) [3] and Fastpack (native JS bundler - a faster webpack) [4].
Some of my favorite features are the following:
- Esy gives the convenience of NPM / Yarn, but for native development. Importantly, it works great cross-platform - OSX/Linux/Windows. IMO The ease of cross-platform development is one of the JS+Node+NPM workflows greatest strengths, and esy provides this for native development.
- Esy gets faster as you use it. It has smart caching for compiled artifacts - taking into account the full set of transitive dependencies.
- The linking / resolution workflow is incredibly useful for native development. For Revery, we'd often need to try fixes across our GLFW/Fontkit stack, and the link feature made this simple.
I'm biased as I've contributed a little bit to the esy, but I've found the approach to development really interesting architecturally - esy is like a 'React' for dependency management - treating the dependency tree as a "pure function" of (dependency tree -> lock -> build environment) with "memoization" to re-use compiled artifacts.
These are tough problems to solve in native... and I'm really impressed with your work Andrey! Thank you!
This looks very nice! I wonder if it might be easy(ish) to wrap it as a plug-in for asdf? There's already an opam/ocaml plug-in - but from comments here it sounds like it might be possible to write a wrapper similar to the ruby plug-in that re-uses a lot of the tooling for rbenv/build. (as does the rust plug-in, and others)
- esy builds are happening in an isolated environment with only specified dependencies available
- esy executes build processes under sandboxing (so builds can't write into arbitrary locations, we have some work to do here though)
- esy has feature which allows to rewrite binaries to remove prefix paths from them.
As for package signing+verification (assuming you are talking about source code) we do only integrity checks now for packages downloaded from npm and via opam.
Hi,
I'm one of the developers of esy, would be happy to answer any questions about it.
I have background in JS ecosystem and always liked how npm/yarn allows you to have a set of dependencies installed per project which do not conflict with any other global installations. Then this goes even beyond runtime dependencies and you have tools like eslint/flow and others installed per project in node_modules.
Now enter native development and you won't find the same workflow there. This is what esy tries to solve. It focuses now on Reason programming language but conceptually it works with other platforms/tools too (for example some of our dependencies are C/C++ code and it works nicely with esy too).
Also I want to highlight the following points:
- All dependencies (and devDependencies) are local to a project (like with npm/yarn)
- The cache for built packages is shared between different projects - if you built OCaml 4.06.0 compiler once then the other project will just reuse it (this works similar to Nix package manager)
- To execute some tool within the isolated project environment you prefix it with esy command: "esy ocaml", "esy vim", "esy code"
- esy can install packages from npm, opam (package registry for OCaml ecosystem), github
- esy allows you to develop multiple packages at once (see linking workflow in docs).
- finally... we support Windows (besides Linux and macOS)!
Hi Andrey, can you tell me if this understanding is correct - ? I can use Esy to write native OCaml/Reason, but has no use in a compile-to-js environment - there it is npm/yarn. With Esy I will be using Dune as my build tool, but instead of the opam binary I can use Esy itself which can pull packages from the opam repository. It takes care of both packages and compiler environment just like opam.
The `link` directive is much more ergonomic than `npm link`. Nice.
Is there anything in Esy that would make porting native OCaml packages (like say https://github.com/c-cube/qcheck) to Bucklescript?
It would also be wonderful to see it in action through a short video tutorial.
> I can use Esy to write native OCaml/Reason, but has no use in a compile-to-js environment - there it is npm/yarn.
If you use js_of_ocaml (a compiler from OCaml/Reason to JS) then esy works with it very well (with the help from dune). With Bucklescript (another OCaml/Reason to JS compiler) it's more complicated (see below).
> With Esy I will be using Dune as my build tool, but instead of the opam binary I can use Esy itself which can pull packages from the opam repository. It takes care of both packages and compiler environment just like opam.
This is correct!
But not necessarily dune (though this is a very good choice), you can use almost any build system you want.
> Is there anything in Esy that would make porting native OCaml packages (like say https://github.com/c-cube/qcheck) to Bucklescript?
Unfortunately not, esy works on level higher than build systems, it doesn't built code itself but rather calls an actual build system to do the workd (dune, make, cmake, ...). Bucklescript is unique as it tries to own the whole stack and thus you have to make code compatible with its build system (bsb) manually.
One thing which esy could help though right now with Bucklescript is providing native tools like ppx'es (preprocessors for Reason and OCaml code) or tools for editors like Merlin (autocompletion and etc.).
> It would also be wonderful to see it in action through a short video tutorial.
Agreed. Hope to have something soon.
Awesome work! I've not had the greatest experience with Opam in the past for the reasons you mention - after using Cargo for a long time, it was a bit of a shock! This could do wonders to improve things!
Some questions:
- Is there any issues with sharing the same config file with another package manager? Ie. `package.json`?
- On a similar vein, why JSON, and not a config format that supports comments, like TOML?
- Will it be able to do Coq projects as well? I've not had a great experience trying to use Opam with it, and had to resort to using submodules. D:
> - Is there any issues with sharing the same config file with another package manager? Ie. `package.json`?
esy can install JS dependencies just fine but we are supporting plug'n'play installations only (see docs https://esy.sh/docs/en/node-compatibility.html for more info on that).
> - On a similar vein, why JSON, and not a config format that supports comments, like TOML?
Yeah, we thought of that but decided to use JSON for now as it's easy to publish esy projects to npm - you already have package.json ready for publish! We might add support for comments to package.json soon though.
> - Will it be able to do Coq projects as well? I've not had a great experience trying to use Opam with it, and had to resort to using submodules. D:
I think it's possible now (I only use it install Coq itself to work through Software Foundations) but I'm not sure about the details yet.
But considering that Coq ecosystem is moving to dune (the build system of choice in OCaml/Reason native ecosystem) and esy supports dune very well I think Coq is going to have a first class treatment soon in esy!
EDIT: re JS compat: you can esy.json instead of package.json and then esy will use it instead. That way you can use package.json with yarn/npm.
hi Andrey,
thanks for yor work on esy, everytime it comes up I have a look at the documentation and the latest developments. It looks like a nice project, but I'm still a little confused about its purpose. You mention your JS background, so is it all about familiarity to npm? Or are there other things you gain when you run esy on top of dune?
And my other question? It doesn't run with bucklescript - right? Are there specific reasons?
Ben
Hi Ben,
> It looks like a nice project, but I'm still a little confused about its purpose. You mention your JS background, so is it all about familiarity to npm?
I think familiarity with npm is important here!
I mentioned that npm allows you to have all dependencies of the project local.
Now if we want the same experience for a native project it gets more complex: we can't just install OCaml/Reason/C/C++ code into node_modules, it's not going to work.
So what esy does is it constructs for each package a "perfectly" isolated build environment based on dependencies declared in package.json, which is then used to produce build output (executable or library). Such environment contains things like `$PATH`, `$OCAMLPATH`, `$PKG_CONFIG_PATH` and etc, so these are all environment variable which C/OCaml/Reason/... software can understand.
Now it looks like this (super simplified!):
Now because such isolated build environment depends only on the package.json (I simplify a bit here... later on that) and the build output depends only on its build environment we can easily compute a cache key for each package build output just by looking at package.json of the package.So esy has smart cache for builds which makes each package build work like this:
It means in practice that if you have completely separate esy projects which depend on the same version of `ocaml` (or `gcc` or any other package) then esy will build that version of `ocaml` just once. So those "every local" dependencies are super cheap once you have esy cache warm.(the key isn't really hash(package.json) but more like hash(package.json + esy.lock) as we need to resolve dependency constraints to concrete versions first using "esy install" command)
> Or are there other things you gain when you run esy on top of dune?
The correct comparison is to dune on top of opam, I think (opam, the command line program, not opam, the registry as esy supports opam registry too!):
- with opam you manage switches manually (switches are like installed package sets), if you need a switch per project - you create a local switch per project
- with opam every time you create switch you need to compile everything from scratch, with esy we try to reuse as much as possible from the build cache
- you use dune with esy the same way you use dune with opam but you need to prefix dune invocations with esy: "esy dune ...", so that way dune sees esy build environment
- with esy you can pull packages from npm/opam/github/tarballs/...
- there's mechanism called `link:`[2] which allows to develop multiple packages as parts of the same project (so called monorepo workflow).
- ...
Some of these things are possible with opam too (not sure about smart cache though and I think it's pretty important) but with esy we've tried to provide the same experience as when you develop with npm/yarn (of course we wanted to solve gigantic node_modules issue and thus we have pnp, and flaky dependency constraints and thus we have a proper SAT solver, the same used by opam). So you might consider this a different UX to develop native code which is focused on Reason/OCaml.
> And my other question? It doesn't run with bucklescript - right? Are there specific reasons?
It doesn't integrate with bucklescript very well now but you still can use esy with bucklescript projects so you can install Merlin (IDE assistant) or other tools from Reason / OCaml ecosystem.
The way how bucklescript works right now isn't compatible with esy - for example we use pnp[1] installation mechanism (same as yarn) so we don't produce node_modules and thus we are fast and package sources are cached globally - for now bucklescript doesn't understand how resolve package sources from pnp.js runtime and thus it can't work with esy directly.
If bucklescript implements support with pnp then it will be perfectly usable with esy but for now you can use package.json for npm/yarn for bucklescript dependencies and esy.json (esy looks for esy.json before package.json) for native dependencies.
[1]: https://github.com/yarnpkg/rfcs/pull/101 [2]: https://esy.sh/docs/en/linking-workflow.html
thanks for your explanations.
> - with opam every time you create switch you need to compile everything from scratch, with esy we try to reuse as much as possible from the build cache
Yes, that's definitely useful.
> [...] for now you can use package.json for npm/yarn for bucklescript dependencies and esy.json (esy looks for esy.json before package.json) for native dependencies.
So by acting like we use esy on Bucklescript, when we actually use npm, we get a very similar result?
> So by acting like we use esy on Bucklescript, when we actually use npm, we get a very similar result?
Yeah, we use npm in that case just to fetch deps and layout them in node_modules which bucklescript understands. Otherwise we use esy for native built tools (ocamlmerlin, preprocessors and etc).
Just wanted to say thank you for all your work on this, Andrey!
I've been using for developing native projects with ReasonML / OCaml (on Windows!) - a framework called Revery [1] (a fast, native alternative to Electron) and a re-write for our Oni [2] text editor using this platform. It really wouldn't have been possible to pursue these paths without esy.
We're also starting to see other cross-platform Reason-native projects pop-up that I'm very excited about, like Brisk (native OSX UI w/ Reason/OCaml) [3] and Fastpack (native JS bundler - a faster webpack) [4].
Some of my favorite features are the following:
- Esy gives the convenience of NPM / Yarn, but for native development. Importantly, it works great cross-platform - OSX/Linux/Windows. IMO The ease of cross-platform development is one of the JS+Node+NPM workflows greatest strengths, and esy provides this for native development.
- Esy gets faster as you use it. It has smart caching for compiled artifacts - taking into account the full set of transitive dependencies.
- The linking / resolution workflow is incredibly useful for native development. For Revery, we'd often need to try fixes across our GLFW/Fontkit stack, and the link feature made this simple.
I'm biased as I've contributed a little bit to the esy, but I've found the approach to development really interesting architecturally - esy is like a 'React' for dependency management - treating the dependency tree as a "pure function" of (dependency tree -> lock -> build environment) with "memoization" to re-use compiled artifacts.
These are tough problems to solve in native... and I'm really impressed with your work Andrey! Thank you!
- [1] https://github.com/revery-ui/revery
- [2] https://github.com/onivim/oni
- [3] https://github.com/briskml/brisk
- [4] https://github.com/fastpack/fastpack
> I'm biased as I've contributed a little bit to the esy,
Well, Bryan is responsible for the Windows support in esy, not a bit!
This looks very nice! I wonder if it might be easy(ish) to wrap it as a plug-in for asdf? There's already an opam/ocaml plug-in - but from comments here it sounds like it might be possible to write a wrapper similar to the ruby plug-in that re-uses a lot of the tooling for rbenv/build. (as does the rust plug-in, and others)
https://github.com/asdf-vm/asdf
I'm having a hard time figuring out if esy supports reproducable builds, and package signing+verfication.
And if reproducable builds are supported, what format is used to store the build information so downloaded packages can be reproduced.
esy tries hard to enable reproducible builds:
- esy builds are happening in an isolated environment with only specified dependencies available
- esy executes build processes under sandboxing (so builds can't write into arbitrary locations, we have some work to do here though)
- esy has feature which allows to rewrite binaries to remove prefix paths from them.
As for package signing+verification (assuming you are talking about source code) we do only integrity checks now for packages downloaded from npm and via opam.
EDIT: wording
And how is the build env recorded for timestamps and env variables? Is all of this standardized?
>we do only integrity checks now for packages downloaded from npm and via opam
Which is fine, but esy should be capable of accepting signatures and checking them against a given key.
> And how is the build env recorded for timestamps and env variables? Is all of this standardized?
We do record the build env now in esy's own format, you run:
to see the build plan (a-la Nix derivation) which is a JSON with build commands and environment.As regarding timestamps we don't set SOURCE_DATE_EPOCH yet but we definitely plan to address that.