I think ucontext is an excellent starting point for a general implementation. You just abstract it with a thin veneer and adopt faster implementations as needed where applicable.
It's slower than Boost Context for sure, but still around 10x faster than (ab)using Pthreads on Linux. Until someone releases a faster standalone C library it's still the fastest portable solution for projects like Cixl that can't afford dragging C++ around.
There's already many fast, portable, standalone C libraries for this. For instance, see table 1 of https://www.gnu.org/software/pth/rse-pmt.ps. The assembly code for a context switch is pretty minimal (on most platforms, it's just setjmp and longjmp) if you don't try manage the signal state. I would be surprised if there was significant variance in performance (other than whether it chooses to switch the signal mask). Additionally, many language runtimes directly support it without any extra effort on your part. So if you choose to instead use one of those language, you get a fast portable solution without needing to do any extra work to pick a support library (for example, D-Lang LDC, PyPy, Go, Julia).
I implemented coroutines for C with assembly [1] (x86 32 and 64 bit). I took advantage of the calling convention to cut down on the amount of state to save (4 registers for x86 32b and six for x86 64b). Mixing this with signals is probably unwise [2]. So far I've tested the code on Linux and Mac OS-X and it works (although I might not use it for C++ either).
The shortest contest switch sequence I could come up on x86-64 is three instructions:
xchg %rsp, %rdx
leaq 1f(%rip), %rax
jmp *%rsi
1:
It it expect the target stack ptr/ip pair to be in rdx/rsi and saves the current stack ptr and ip in rdx/rax. It does not save any register and uses gcc asm clobbers to instruct the compiler to save any other register.
Code at [1]. The comments about hacks and ub is because I'm trying to transparently propagate exceptions across coroutines, otherwise the stack switching us fairly robust (although GCC specific).
Not faster than dealing directly with ucontext from what I've seen; many wrap it directly and the rest tend to emulate using signals, setjmp and prayers.
ucontext modifies the signal mask, requiring a syscall. That’s very expensive (as shown by the boost benchmark above), and usually unnecessary. “emulate” is a rather negative sounding way to describe running effective the same code as ucontext does (which also happens to typically be the same as sigsetjmp) - it’s not like ucontext has some privledged permissions. It’s “just” a context switch.
I was under the impression that Boost Context is more than that; at least that's what the amount of assembler code tells me; but I'll be the first to admit I don't have much patience for deciphering modern C++. I get that it's possible to invoke the same functionality as ucontext minus signal masks manually, but I'm not convinced it would save enough cycles to pay for the added complexity.
Slightly OT, but do people know of any good small language which compiles to C? I'm not looking for big languages like C++, Nim, or Rust, but something which is essentially just C, but with some minimal modernization, such as perhaps being able to import modules, and doing away with header files.
I realize the Forth & Lisp heritage might be a bit hard on the brain for those who haven't been exposed before, but what you're asking for is otherwise more or less what Cixl is aiming for.
Wow, cixl is a great thing! I love the Dijkstra quote against complexity that serves as inspiration for the language. However, you find this :
> To build Cixl yourself, you'll need a reasonably modern GCC and CMake installed.
I was very taken aback by this. Really? Why do you really need to depend on the cmake monstrosity? It's just a few lines of C without dependencies that they are distributing!
If there is something about cixl that will scare people is the ridiculous dependence on cmake, not its clean syntax.
I hear you; I'm not overly excited with CMake either, or the C build system story in general.
I spent several years trying to bend regular make into something I was happy with; then I spent several years on top of that using Rake, since at least it allowed me to say what I mean in a sane language. Compared to Rake, CMake is at least semi-standard, provides some kind of macro for most things I want to do, and mostly stays out of my way.
But you definitely have a point when it comes to simplicity and project fit. If someone would be willing to step up and help translate the makefile into something that doesn't look horrible, I'd be more than happy to let it go. Otherwise we'll have to wait until I get enough round tuits, which could take a while given how much remains to be done in Cixl.
Often it is disheartening to bend make to your bidding. But cixl has a neatly arranged tree and the makefile to build it is trivial. The following lines suffice (and by running 'make test -j' you compile everything in parallel and run all the tests) :
CFLAGS = -Isrc -O2
LDLIBS = -ldl -lm
SRC = $(shell ls src/cixl/*.c src/cixl/lib/*.c)
OBJ = $(SRC:%.c=%.o)
src/main : src/main.o $(OBJ)
clean : ; $(RM) $(OBJ) src/main
test: src/main ; for i in tests/*; do ./src/main<$$i; done
I just do not understand its point. Cmakefiles are much, much uglier than makefiles. And it takes several steps to compile a program, instead of a single one. And it seems larger than the operating system itself.
I do not know what is "the percentage system", but I doubt it can be so wonderful to make me forget about all the other unnecessary problems.
It may be justified to use cmake when it is really needed. But the cixl interpreter can be built by a five-line makefile.
Ah, I see, a bit more tentativity has been filed away for when I finally check it out.
I was regarding how it outputs progress indication, like
[ 16%] Building C object ...
[ 91%] Linking C executable ...
etc (many lines elided). Back on the slow machine I was using when I was very new to Linux, progress indication ("how soon can I stop freaking out about this maybe not working") was very nice to have :D
But (10 years on) I can understand it being icky to use. I guess, ugly as it is, it's a little more structured than automake is, and maybe that's its redeeming quality.
Maybe build systems are the dead-centre of a "sour spot" (opposite of sweet spot) in computer science, with no truly nice solutions out there? (Genuinely curious. The exponential profusion of JS build systems suggests it's rather hard to solve.)
The only explanation I have is that
- Maybe the author is used to cmake
- Maybe this project has Big Goals And Ventures™ syndrome - and, hence, all the enterprise build systems. (Hopefully not, it looks kinda nice)
> Maybe build systems are the dead-centre of a "sour spot" (opposite of sweet spot) in computer science, with no truly nice solutions out there?
I think it's the opposite. It is a very, very sweet spot for which many essentially perfect solutions exist. Then, it attracts all kinds of extreme bikeshedding and produces the monsters we see here and there.
If you're serious about writing code; I strongly recommend learning some C, Forth & Common Lisp to get an idea of what's possible. Once you have those under your belt, appearances matter less and Cixl will look less crazy banana.
Hmm. How would you do away with header files in a simple way if you wanted something similarly-scoped to C? Have the compiler vacuum in every source file at once to pick up all references, then discard via dead code elimination? That'd kill incremental compilation :/
(I'm genuinely curious about the answer to this question - I'm also looking for something similar to C)
Incremental compilation creates .o files for every compiled .c file, and so it could also create a .h file at the same time (perhaps fused in the .o file).
I haven't looked at Nim in detail, but since Nim does garbage collection, it didn't read to me like a thin layer over C. For instance, is it possible to instantiate a struct on the stack? I'd also have liked the compiled C code to be readable, which if I'm not mistaken, Nim doesn't do.
> For instance, is it possible to instantiate a struct on the stack?
Yeah, it is. You can easily avoid the GC as long as you don't mind managing memory yourself (unfortunately this means that you cannot use the stdlib either, or at least parts of it, if you disable the GC with the `--gc:none` flag you'll get warning messages from the compiler about procs that need the GC which is nice).
> I'd also have liked the compiled C code to be readable, which if I'm not mistaken, Nim doesn't do.
Yeah, that's one thing that Nim definitely doesn't do. I don't think there are many languages that do. It would be an interesting experiment to see how pretty you could make the generated C by either fixing things in the compiler or just running a C formatter on it.
> unfortunately this means that you cannot use the stdlib either
This is the kind of friction I expect from a language which uses GC, for my usecase. I think it might make sense for Nim to support a GC-less stdlib for people who do not want GC in their code.
Worth noting that ucontext is quite slow (at least on linux): https://www.boost.org/doc/libs/1_67_0/libs/context/doc/html/...
It's fast enough for many applications.
I think ucontext is an excellent starting point for a general implementation. You just abstract it with a thin veneer and adopt faster implementations as needed where applicable.
It's slower than Boost Context for sure, but still around 10x faster than (ab)using Pthreads on Linux. Until someone releases a faster standalone C library it's still the fastest portable solution for projects like Cixl that can't afford dragging C++ around.
There's already many fast, portable, standalone C libraries for this. For instance, see table 1 of https://www.gnu.org/software/pth/rse-pmt.ps. The assembly code for a context switch is pretty minimal (on most platforms, it's just setjmp and longjmp) if you don't try manage the signal state. I would be surprised if there was significant variance in performance (other than whether it chooses to switch the signal mask). Additionally, many language runtimes directly support it without any extra effort on your part. So if you choose to instead use one of those language, you get a fast portable solution without needing to do any extra work to pick a support library (for example, D-Lang LDC, PyPy, Go, Julia).
I implemented coroutines for C with assembly [1] (x86 32 and 64 bit). I took advantage of the calling convention to cut down on the amount of state to save (4 registers for x86 32b and six for x86 64b). Mixing this with signals is probably unwise [2]. So far I've tested the code on Linux and Mac OS-X and it works (although I might not use it for C++ either).
[1] https://github.com/spc476/C-Coroutines
[2] In my not-so humble opinion, using signals at all is not wise.
The shortest contest switch sequence I could come up on x86-64 is three instructions:
1:It it expect the target stack ptr/ip pair to be in rdx/rsi and saves the current stack ptr and ip in rdx/rax. It does not save any register and uses gcc asm clobbers to instruct the compiler to save any other register.
Code at [1]. The comments about hacks and ub is because I'm trying to transparently propagate exceptions across coroutines, otherwise the stack switching us fairly robust (although GCC specific).
[1] https://github.com/gpderetta/delimited/blob/master/delimited...
Signals, done correctly, are hard. I agree with your NSHO for the most part. Neat code!
Not faster than dealing directly with ucontext from what I've seen; many wrap it directly and the rest tend to emulate using signals, setjmp and prayers.
I would love to be wrong though...
ucontext modifies the signal mask, requiring a syscall. That’s very expensive (as shown by the boost benchmark above), and usually unnecessary. “emulate” is a rather negative sounding way to describe running effective the same code as ucontext does (which also happens to typically be the same as sigsetjmp) - it’s not like ucontext has some privledged permissions. It’s “just” a context switch.
I was under the impression that Boost Context is more than that; at least that's what the amount of assembler code tells me; but I'll be the first to admit I don't have much patience for deciphering modern C++. I get that it's possible to invoke the same functionality as ucontext minus signal masks manually, but I'm not convinced it would save enough cycles to pay for the added complexity.
Saving the signal mask easily cost hundreds to thousand cycles. One or two order of magnitude more than the rest of contex switching.
For the record: I just benchmarked GnuPth with NULL sigmask against ucontext for the example in the post and it's slightly (3.4/3.2s) slower.
Slightly OT, but do people know of any good small language which compiles to C? I'm not looking for big languages like C++, Nim, or Rust, but something which is essentially just C, but with some minimal modernization, such as perhaps being able to import modules, and doing away with header files.
Cixl? :)
I realize the Forth & Lisp heritage might be a bit hard on the brain for those who haven't been exposed before, but what you're asking for is otherwise more or less what Cixl is aiming for.
https://github.com/basic-gongfu/cixl#compiling
Wow, cixl is a great thing! I love the Dijkstra quote against complexity that serves as inspiration for the language. However, you find this :
> To build Cixl yourself, you'll need a reasonably modern GCC and CMake installed.
I was very taken aback by this. Really? Why do you really need to depend on the cmake monstrosity? It's just a few lines of C without dependencies that they are distributing!
If there is something about cixl that will scare people is the ridiculous dependence on cmake, not its clean syntax.
I hear you; I'm not overly excited with CMake either, or the C build system story in general.
I spent several years trying to bend regular make into something I was happy with; then I spent several years on top of that using Rake, since at least it allowed me to say what I mean in a sane language. Compared to Rake, CMake is at least semi-standard, provides some kind of macro for most things I want to do, and mostly stays out of my way.
But you definitely have a point when it comes to simplicity and project fit. If someone would be willing to step up and help translate the makefile into something that doesn't look horrible, I'd be more than happy to let it go. Otherwise we'll have to wait until I get enough round tuits, which could take a while given how much remains to be done in Cixl.
Often it is disheartening to bend make to your bidding. But cixl has a neatly arranged tree and the makefile to build it is trivial. The following lines suffice (and by running 'make test -j' you compile everything in parallel and run all the tests) :
would you accept such a patch in your project?Not bad; my make-fu was never that spectacular to begin with and it hasn't been aging well, this helps me a lot.
Sure thing, if you feel like giving the entire makefile the same treatment I'd be delighted to accept it.
Thanks for taking the time!
What's so bad about cmake? The first time I ever used it, I was very pleasantly impressed by the percentage system.
I take it that cmake tried to tackle autotools, partially succeeded, and partly succumbed to autotools' all-devouring complexity? :)
I just do not understand its point. Cmakefiles are much, much uglier than makefiles. And it takes several steps to compile a program, instead of a single one. And it seems larger than the operating system itself.
I do not know what is "the percentage system", but I doubt it can be so wonderful to make me forget about all the other unnecessary problems.
It may be justified to use cmake when it is really needed. But the cixl interpreter can be built by a five-line makefile.
Ah, I see, a bit more tentativity has been filed away for when I finally check it out.
I was regarding how it outputs progress indication, like
etc (many lines elided). Back on the slow machine I was using when I was very new to Linux, progress indication ("how soon can I stop freaking out about this maybe not working") was very nice to have :DBut (10 years on) I can understand it being icky to use. I guess, ugly as it is, it's a little more structured than automake is, and maybe that's its redeeming quality.
Maybe build systems are the dead-centre of a "sour spot" (opposite of sweet spot) in computer science, with no truly nice solutions out there? (Genuinely curious. The exponential profusion of JS build systems suggests it's rather hard to solve.)
The only explanation I have is that
- Maybe the author is used to cmake
- Maybe this project has Big Goals And Ventures™ syndrome - and, hence, all the enterprise build systems. (Hopefully not, it looks kinda nice)
> Maybe build systems are the dead-centre of a "sour spot" (opposite of sweet spot) in computer science, with no truly nice solutions out there?
I think it's the opposite. It is a very, very sweet spot for which many essentially perfect solutions exist. Then, it attracts all kinds of extreme bikeshedding and produces the monsters we see here and there.
Ooh, okay.
What do you think are the best/"essentially perfect" solution(s)?
(Said as someone who just figured out his first Makefile a couple days ago :P)
make is essentially perfect for compiling small and moderately sized projects
Yeah, the syntax is pretty out-of-the-world for me, but I'll give it a go. Thanks!
If you're serious about writing code; I strongly recommend learning some C, Forth & Common Lisp to get an idea of what's possible. Once you have those under your belt, appearances matter less and Cixl will look less crazy banana.
Ion (https://github.com/pervognsen/bitwise/tree/master/ion) is a relatively new language with round-trippable conversion to C as a design goal.
Ooooh, bitwise looks really interesting...
Hmm. How would you do away with header files in a simple way if you wanted something similarly-scoped to C? Have the compiler vacuum in every source file at once to pick up all references, then discard via dead code elimination? That'd kill incremental compilation :/
(I'm genuinely curious about the answer to this question - I'm also looking for something similar to C)
Incremental compilation creates .o files for every compiled .c file, and so it could also create a .h file at the same time (perhaps fused in the .o file).
Ooooh, duh. Takes notes
Nim is small enough to be used for this purpose. You can easily use it as a nice layer on top of C.
Why do you consider it too big?
I haven't looked at Nim in detail, but since Nim does garbage collection, it didn't read to me like a thin layer over C. For instance, is it possible to instantiate a struct on the stack? I'd also have liked the compiled C code to be readable, which if I'm not mistaken, Nim doesn't do.
> For instance, is it possible to instantiate a struct on the stack?
Yeah, it is. You can easily avoid the GC as long as you don't mind managing memory yourself (unfortunately this means that you cannot use the stdlib either, or at least parts of it, if you disable the GC with the `--gc:none` flag you'll get warning messages from the compiler about procs that need the GC which is nice).
> I'd also have liked the compiled C code to be readable, which if I'm not mistaken, Nim doesn't do.
Yeah, that's one thing that Nim definitely doesn't do. I don't think there are many languages that do. It would be an interesting experiment to see how pretty you could make the generated C by either fixing things in the compiler or just running a C formatter on it.
> unfortunately this means that you cannot use the stdlib either
This is the kind of friction I expect from a language which uses GC, for my usecase. I think it might make sense for Nim to support a GC-less stdlib for people who do not want GC in their code.
This is giving me D flashbacks.
I think you want Zig!
Zig looks amazing! Thank you! While it doesn't compile to C, you read my mind that I actually wanted a C replacement rather than compile to C.
Great! Yeah, that's what I was guessing, glad that I guessed correctly.
Isn't Zig based on LLVM rather than compiled to C?
Yes, I didn’t assume that was the highest order bit here.
Zig explicitly does not depend on the C stdlib
Yeah, this is true, I was responding mostly to the semantics aspect rather than that part.