Isn't Python 3 exactly a way to unify the async approaches? (not saying a good/bad way, just a way) We've had a few projects trying to do async in many ways over the years: stackless, gevent, twisted, asyncio, curio, ...
async in Python 3 goes from "there's lots of solutions, which should I choose?" to "there's an obvious solution in standard library (unless you really want something else for a very specific reason)"
The great thing and the awful thing about open source is that anyone can change what it means to be the "Zen of Python."
> There should be one—and preferably only one—obvious way to do it
I feel this can be interpreted many ways. Sure, async means there is more than one way you can do requests to get web data now, and more than one way you should structure your app - but not every app needs to do API calls async, and sometimes the complexity is not worth it.
Other times, yes, of course you need to do requests async. They're taking three minutes each and you have to do twenty of them.
I've been writing web apps using Tornado since 2013 and that has been an excellent experience.
But I'm not sure if I have the same opinion of asyncio. To be honest, I don't even know if I understand it completely. There are so many concepts and IMHO they're not really intuitive, so it's difficult to put pieces together and find out how those individual concepts relate to each other or work with each other.
On the surface it looks simple, but it's really like going down a rabbit hole. The more I try to understand it, the more I agree with Armin Ronacher's article on asyncio [1].
There was a post on HN that was asking about real-world examples of asyncio code that wasn't `asncio.sleep()`. I think my project is pretty good example for that - and a strange example too because it involved a lot of hardware interfaces.
I've been meaning to open-source the project ever since we axed it. I was the only real programmer on the job, so I'm itching for feedback on it. So if anyone's interested I'll post it here.
I had actually written it 3 times over. First as raw socket server, then in twisted, then on top of asyncio.
So yes, asyncio is a bit of a mess. It's documentation is a bit odd and I have several gripes. What helped we was realising that there was an 'async way' and an 'callback way' to everything in asyncio. So not mixing the two together and ignoring the callback parts really _really_ helped.
I don't know if this is true, but I recall reading somewhere that asyncio was not designed with coroutines initially and this explains the dual-personality of the library. I can't find any evidence that asyncio existed before 3.3, so I can't back that up.
I would be interested in seeing and reviewing your asyncio work gwillz! I have a small (750 sloc) network service implemented with asyncio and ran into some design problems around modularly handling healthchecking, exception handling/monitoring, and auto recovery/restart. I was unable to find much in the way of large software written with asyncio to learn better patterns from, so ended up with what, to me, feel like mediocre solutions. My email is in my profile.
Yes, and it is...not great. It's generally fine as far as the language goes (considering it's a retrofit), but the builtin asyncio library is not the way to go in my opinion. trio is far better for just aping a lot of the Erlang way of things.
But even then it's hard to debug, or work out ideas in the REPL, testing is a bit more complex than it should have to be, etc. There are probably better languages to reach for (I'd love to see rust firm up a one true way of async).
FWIW, I was at a talk by an asyncio core developer at EuroPython and he said they are looking very closely at how to improve the asyncio API in upcoming 3.8/3.9 releases. In particular, looking at good ideas they can pull in from libraries such as trio.
Yes, and it has been a good experience so far. Mostly we've been doing applications involving several RabbitMQ producers/consumers, a HTTP server and database access here and there. It relies on the fact that there already are asyncio-enabled libraries for the particular technologies we routinely use.
Note that our use case might be different to many in this thread - in contrast to HTTP, we write mainly event-driven message-oriented applications. It is often practical to have e.g. multiple different consumers within one process - without async, we'd need to resort to threads which brings a plethora of new problems. Async Python does offer a good solution for this.
That said, there are a lot of places where async could be improved.
Language-wise it's pretty okay, but the asyncio library contains too many concepts (some perhaps introduced historically and now deprecated) and the docs aren't doing very good job explaining those. IO loop management could be simplified and some of the related functions should have different, better names. Error messages and stack traces could also be better when async is involved. As far as testing goes, pytest-asyncio could also enjoy some love.
But overall, no major complaints. Few catches aside, it works pretty much the same as in C# or JavaScript syntax-wise, which is what you would expect from this approach to asynchronous programming.
You might want to ask this question over at https://github.com/home-assistant/home-assistant. They have been porting most of their codebase from threads and such to Python3 async over the past months/years. I'm sure the've hit almost all corners of async doing that.
I did some small contributions for HA components and didn't have to get deep into async just yet, but my general feeling is that it is less 'natural' that async programming in a language like Go. But Go has the bigger advantage is was born with async in mind, so it doesn't lag behind that much with 3rd party libraries support like Python does.
Honestly, no.
Despite all the supposed advantages of async, I haven't found a use for it.
When it comes to web apps a nginx + uwsgi + a sync framework is an unbeatable combination.
The only times I have used async were when I made website scrapers and telegram bots for And if I was to scrape websites now, I would just use scrapy. My favorite telegram bot framework is synchronous too.
There are some use cases in web programming. For example, if a request handler (view, in Django's terminology) is in turn calling out some external endpoints, you don't want those external requests to block other requests to your API. For things like those, async is actually quite nice.
It really depends on how responsive your endpoints are. If they're really slow (or slowly streaming data), you can probably do hundreds/thousands via thread pools. (depends how you do the dispatch / locking) You can go orders of magnitude higher with async. (as in epoll - specific async frameworks may have their own bottlenecks)
Yes, in https://apility.io all the backend services, API endpoints at the edge and parsing services take 'advantage' of async in Python >3.
My opinion is -like somebody said- it breaks the Zen of Python. When I started using it I had doubts about moving to a more async 'friendly' language. But I decided to stick with my old friend Python... even when I don't recognize him very much after some changes after version 3.
Yeah I'm using it plenty for web services where I've got a concurrent app but most data processing is done upstream.
It's pretty easy to get started with and with the new keywords simple enough to teach. The testing story is a bit horrible due to how coroutines work though, having to create asynchronous True and False when patching returns for example is a pain.
Using it for what you might describe as API proxies (take requests, fetch a bunch of other APIs, combine/transform their results to a different format, deliver to client). Works for that case well, but it's admittedly basically the perfect application for it.
Can not really tell what I am working on, but so far I am more or less happy with it.
It is not as simple as advertised though ("just use async/await"). In order to write stable, performant applications you have to understand more or less the underlying mechanisms of asyncio (what are tasks/futures, how does their lifecycle work). Then there are quite a few things which work not as expected and you have to understand them as well (e.g. exception handling). If you get into the nitty-gritty details you will have a good experience with asyncio. If you don't you better stay away from it or you will have a hard time.
Performance wise it seems pretty good so far btw.. Did not really hit its limit yet though.
I used it to write an arbitrage trading bot. I learned python and asyncio just for this, and it is working as intended.
Asyncio with aiohttp is used to capture price data from 4 websockets and handle requests to exchanges via their APIs.
Python was easy to learn with “learning python the hard way”, asyncio was also easy to learn thanks to lots of available material online, and it looked less confusing than threading.
Again, speaking as a beginner, I quite like asyncio because to me it looks just like synchronous code. To me it’s easier to think about and keep track of the control flow and shared data than with threading.
Asyncio looked like the easiest and fastest way to accomplish my goal, and it worked.
We had a couple services in Twisted. Aptly named. It was a big reason I've enjoyed Go so much.
One was an in-house DB abstraction/API that handled multiple DB hosts, read/write query routing, connection pooling, query queueing, metrics, and more. The other was a link forwarding service that took encoded links, decoded and tracked them and their metadata, and forwarded the user.
Yes, we are using it to build a set of system configuration and monitoring daemons, controlled through D-Bus.
It's okay, it can be a a little confusing at first but worth the time investment required to understand it. The API has been improved with all of the recent 3.x dot releases. The new run command in 3.7 should make it a lot easier for folks to get started.
I have implemented the Raft algorithm with asyncio. Although this was an academic exercise and I would choose something else for real world usage, I was pleasantly surprised with the performance: https://github.com/simonacca/zatt
A lot of machine learning datasets only include metadata with a link to the original source (e.g., a full resolution image which might be too wasteful to include with the dataset).
So the first step is to write a crawler and async is a really fast way to do that because everything is network IO bounded.
I love python but if I wanted a fast async app, I would prefer to write it in go with goroutines. I don't like the async programming style where every function involved in the logic chain has to be async. Goroutines with channels are just nicer.
Python is awesome for a lot of things but go really shines for async style programs.
Bit of an unconstructice rant, but this always bothered me: Async is the nail in the coffin of The Zen of Python.[1]
Open a python REPL, type:
…There should be one—and preferably only one—obvious way to do it
…
It has not been true for years. Python 3 especially, took a sledgehammer to that ole poem.
See http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... which is about this effect. I am reminded of that blog post every single time I work with async in Python :/
[1] https://www.python.org/dev/peps/pep-0020/
Isn't Python 3 exactly a way to unify the async approaches? (not saying a good/bad way, just a way) We've had a few projects trying to do async in many ways over the years: stackless, gevent, twisted, asyncio, curio, ...
async in Python 3 goes from "there's lots of solutions, which should I choose?" to "there's an obvious solution in standard library (unless you really want something else for a very specific reason)"
The great thing and the awful thing about open source is that anyone can change what it means to be the "Zen of Python."
> There should be one—and preferably only one—obvious way to do it
I feel this can be interpreted many ways. Sure, async means there is more than one way you can do requests to get web data now, and more than one way you should structure your app - but not every app needs to do API calls async, and sometimes the complexity is not worth it.
Other times, yes, of course you need to do requests async. They're taking three minutes each and you have to do twenty of them.
The story of "import this": https://www.wefearchange.org/2010/06/import-this-and-zen-of-...
There's also:
from __future__ import braces
and
import antigravity
I've been writing web apps using Tornado since 2013 and that has been an excellent experience.
But I'm not sure if I have the same opinion of asyncio. To be honest, I don't even know if I understand it completely. There are so many concepts and IMHO they're not really intuitive, so it's difficult to put pieces together and find out how those individual concepts relate to each other or work with each other.
On the surface it looks simple, but it's really like going down a rabbit hole. The more I try to understand it, the more I agree with Armin Ronacher's article on asyncio [1].
[1]: http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio...
Yes - and it was great fun!
There was a post on HN that was asking about real-world examples of asyncio code that wasn't `asncio.sleep()`. I think my project is pretty good example for that - and a strange example too because it involved a lot of hardware interfaces.
I've been meaning to open-source the project ever since we axed it. I was the only real programmer on the job, so I'm itching for feedback on it. So if anyone's interested I'll post it here.
I had actually written it 3 times over. First as raw socket server, then in twisted, then on top of asyncio.
So yes, asyncio is a bit of a mess. It's documentation is a bit odd and I have several gripes. What helped we was realising that there was an 'async way' and an 'callback way' to everything in asyncio. So not mixing the two together and ignoring the callback parts really _really_ helped.
I don't know if this is true, but I recall reading somewhere that asyncio was not designed with coroutines initially and this explains the dual-personality of the library. I can't find any evidence that asyncio existed before 3.3, so I can't back that up.
I would be interested in seeing and reviewing your asyncio work gwillz! I have a small (750 sloc) network service implemented with asyncio and ran into some design problems around modularly handling healthchecking, exception handling/monitoring, and auto recovery/restart. I was unable to find much in the way of large software written with asyncio to learn better patterns from, so ended up with what, to me, feel like mediocre solutions. My email is in my profile.
Yes, and it is...not great. It's generally fine as far as the language goes (considering it's a retrofit), but the builtin asyncio library is not the way to go in my opinion. trio is far better for just aping a lot of the Erlang way of things.
But even then it's hard to debug, or work out ideas in the REPL, testing is a bit more complex than it should have to be, etc. There are probably better languages to reach for (I'd love to see rust firm up a one true way of async).
FWIW, I was at a talk by an asyncio core developer at EuroPython and he said they are looking very closely at how to improve the asyncio API in upcoming 3.8/3.9 releases. In particular, looking at good ideas they can pull in from libraries such as trio.
Maybe Python 4 will mark a maturing of asyncio?
Yes, and it has been a good experience so far. Mostly we've been doing applications involving several RabbitMQ producers/consumers, a HTTP server and database access here and there. It relies on the fact that there already are asyncio-enabled libraries for the particular technologies we routinely use.
Note that our use case might be different to many in this thread - in contrast to HTTP, we write mainly event-driven message-oriented applications. It is often practical to have e.g. multiple different consumers within one process - without async, we'd need to resort to threads which brings a plethora of new problems. Async Python does offer a good solution for this.
That said, there are a lot of places where async could be improved.
Language-wise it's pretty okay, but the asyncio library contains too many concepts (some perhaps introduced historically and now deprecated) and the docs aren't doing very good job explaining those. IO loop management could be simplified and some of the related functions should have different, better names. Error messages and stack traces could also be better when async is involved. As far as testing goes, pytest-asyncio could also enjoy some love.
But overall, no major complaints. Few catches aside, it works pretty much the same as in C# or JavaScript syntax-wise, which is what you would expect from this approach to asynchronous programming.
You might want to ask this question over at https://github.com/home-assistant/home-assistant. They have been porting most of their codebase from threads and such to Python3 async over the past months/years. I'm sure the've hit almost all corners of async doing that.
I did some small contributions for HA components and didn't have to get deep into async just yet, but my general feeling is that it is less 'natural' that async programming in a language like Go. But Go has the bigger advantage is was born with async in mind, so it doesn't lag behind that much with 3rd party libraries support like Python does.
Honestly, no. Despite all the supposed advantages of async, I haven't found a use for it. When it comes to web apps a nginx + uwsgi + a sync framework is an unbeatable combination. The only times I have used async were when I made website scrapers and telegram bots for And if I was to scrape websites now, I would just use scrapy. My favorite telegram bot framework is synchronous too.
There are some use cases in web programming. For example, if a request handler (view, in Django's terminology) is in turn calling out some external endpoints, you don't want those external requests to block other requests to your API. For things like those, async is actually quite nice.
Are you talking about blocking other requests to the request handler or to external endpoints?
Because both can be solved in a sync server via thread pools.
It really depends on how responsive your endpoints are. If they're really slow (or slowly streaming data), you can probably do hundreds/thousands via thread pools. (depends how you do the dispatch / locking) You can go orders of magnitude higher with async. (as in epoll - specific async frameworks may have their own bottlenecks)
You can mix asynchronous code with Django?
You kick off an async task in celery or use some other way to keep track of it. Then return a key, and have your front end poll for status
There are things you can do to achieve async in Django. But I only used Django terminology because Django is more popular than other frameworks.
Yes, in https://apility.io all the backend services, API endpoints at the edge and parsing services take 'advantage' of async in Python >3.
My opinion is -like somebody said- it breaks the Zen of Python. When I started using it I had doubts about moving to a more async 'friendly' language. But I decided to stick with my old friend Python... even when I don't recognize him very much after some changes after version 3.
Yeah I'm using it plenty for web services where I've got a concurrent app but most data processing is done upstream.
It's pretty easy to get started with and with the new keywords simple enough to teach. The testing story is a bit horrible due to how coroutines work though, having to create asynchronous True and False when patching returns for example is a pain.
Using it for what you might describe as API proxies (take requests, fetch a bunch of other APIs, combine/transform their results to a different format, deliver to client). Works for that case well, but it's admittedly basically the perfect application for it.
Yes.
Can not really tell what I am working on, but so far I am more or less happy with it.
It is not as simple as advertised though ("just use async/await"). In order to write stable, performant applications you have to understand more or less the underlying mechanisms of asyncio (what are tasks/futures, how does their lifecycle work). Then there are quite a few things which work not as expected and you have to understand them as well (e.g. exception handling). If you get into the nitty-gritty details you will have a good experience with asyncio. If you don't you better stay away from it or you will have a hard time.
Performance wise it seems pretty good so far btw.. Did not really hit its limit yet though.
I’m a beginner with python as my first language.
I used it to write an arbitrage trading bot. I learned python and asyncio just for this, and it is working as intended.
Asyncio with aiohttp is used to capture price data from 4 websockets and handle requests to exchanges via their APIs.
Python was easy to learn with “learning python the hard way”, asyncio was also easy to learn thanks to lots of available material online, and it looked less confusing than threading.
Again, speaking as a beginner, I quite like asyncio because to me it looks just like synchronous code. To me it’s easier to think about and keep track of the control flow and shared data than with threading.
Asyncio looked like the easiest and fastest way to accomplish my goal, and it worked.
We had a couple services in Twisted. Aptly named. It was a big reason I've enjoyed Go so much.
One was an in-house DB abstraction/API that handled multiple DB hosts, read/write query routing, connection pooling, query queueing, metrics, and more. The other was a link forwarding service that took encoded links, decoded and tracked them and their metadata, and forwarded the user.
Yes, we are using it to build a set of system configuration and monitoring daemons, controlled through D-Bus.
It's okay, it can be a a little confusing at first but worth the time investment required to understand it. The API has been improved with all of the recent 3.x dot releases. The new run command in 3.7 should make it a lot easier for folks to get started.
I have implemented the Raft algorithm with asyncio. Although this was an academic exercise and I would choose something else for real world usage, I was pleasantly surprised with the performance: https://github.com/simonacca/zatt
A lot of machine learning datasets only include metadata with a link to the original source (e.g., a full resolution image which might be too wasteful to include with the dataset).
So the first step is to write a crawler and async is a really fast way to do that because everything is network IO bounded.
Async programming in python? yes, as part of distributing parameter finding for genetic algorithms and backtesting.
Does it use 'asyncio'? Nope.
I would if there was an async sql server driver... Thread pools don't cut it.
I don't have my hopes high for this one though.
Check out the asphalt project if you'd like to give a native asyncio framework a try.
I love python but if I wanted a fast async app, I would prefer to write it in go with goroutines. I don't like the async programming style where every function involved in the logic chain has to be async. Goroutines with channels are just nicer.
Python is awesome for a lot of things but go really shines for async style programs.