chrisweekly 6 years ago

My favorite tool in this space is [lnav](https://lnav.org) which has an embedded sqlite engine and works with all kinds of structured data. It might obviate the need for datasette, or maybe complement it in a scripted workflow....

donut 6 years ago

Nice. Looking forward to trying this out.

Reminds me of http://openrefine.org/.

  • aorth 6 years ago

    Indeed! I was just thinking that this could be a substitute for viewing where I don't want to have to fire up OpenRefine, but for editing, GREL, reconciliation, etc OpenRefine is still king! I've just started playing with reconciliation in OpenRefine via conciliator + Solr. Pretty cool.

    https://github.com/codeforkjeff/conciliator

obelix_ 6 years ago

I see something about uploading the CSV file. Not clear where the sqlite dB is being created. Is it local? Does the browser create it? Ideally I would like to run this locally when I am playing with data.

What's been the largest file tested? Basically what's the max rows/cols it can handle fast?

  • simonw 6 years ago

    You can run it locally - that's the default way to use it. "pip3 install datasette", create your SQLite database file and run "datasette mydb.db" to start exploring.

    I've run it successfully on SQLite files up to a GB in size and theoretically it should work with much bigger files than that.

    The https://publish.datasettes.com/ tool works by taking your uploaded CSVs and running my "csvs-to-sqlite" script on them, then deploying the datasette application alongside that new SQLite database file.

    • obelix_ 6 years ago

      Thanks! Looks pretty useful. Will be checking it out soon.

hprotagonist 6 years ago

i’ve been keeping tabs (ha!) on this for some time.

keep up the good work! it is incredibly useful for me to be able to rapidly poke around a new data set and get a lay of the land.

Perhaps equally importantly, being able to do this in a meeting with the semi-higher ups, live, makes you look fiendishly smart.

gigatexal 6 years ago

This is awesome. Is there something like this for Postgres?

chrisweekly 6 years ago

Nice publish workflow, leveraging Zeit `now`.