GNU recutils: Plain text database

On the topic of plain text databases, I have been playing around with wordnet(basically the backbone data structure of a thesaurus ) and while there are a lot of perfectly good libraries to handle the data I was having fun building my own. One interesting thing is that the original data format has the byte offset of every record and link baked into the data, this makes it trivial to avoid having to load the whole thing into memory and you can directly seek to the record in question thus making it a plain text database. Admittedly one only good for reading as writes would have to rebuild all the indexes.

https://wordnet.princeton.edu/documentation/wndb5wn

Honestly now days the whole thing can be trivially loaded into memory but back when the project was started this was much more of a concern, I do know that once I figured this out I started re writing my program to see how little memory I could use, It was a lot of fun to use access patterns other than "load the whole thing into memory"

https://news.ycombinator.com/item?id=22153665 505 points, 143 comments, 6 years ago

https://news.ycombinator.com/item?id=31832564 155 points, 52 comments, 3 years ago

https://news.ycombinator.com/item?id=15302035 105 points, 46 comments, 8 years ago

   w3m `apt-cache show recutils | awk '/Homepage/ { print $2 }'`

I use the .rec format whenever I want a database maintained in git/github. The format is ideal if you want reasonable data diffs.

Amazingly rugged design that somehow places man and machine on equal footing by throwing us back to the time of casette futurism.

Amazingly poor choice of logo.

I love recutils. The database format is simple enough, it has a bunch of options for constraints, and it has Bash integration and a great Emacs mode to search, edit and verify the integrity of the database.

Sure, it's not as fast as SQLite or bigger systems, but often it's enough for smaller projects.

Tortoise sex is a bold choice for a logo, but certainly memorable.

I like plaintext database files, and this looks way more readable than csv. But what I'd find even more interesting than improved readability would be features like atomicity. There are many tools for joins and other basic operations on tables stored in various text formats, but the real gap to 'proper' databases are ACID properties.

With recutils csv2rec I have my invoices list.

With recutils recsel | recfmt -f template.fodt I have my invoices.

soffice and curl to generate .pdf s and email them off.

With recutils recset I have my invoices status updated. Done.

For those who get blocked by gnu.org with a 403 (older Firefox) or an even sillier "Too Many Requests" error (older Safari) need to override their user agents strings to "curl" to make the site load again.

In 2010 I remember people being very proficient with this at Amazon.

I really enjoying the toolset to query logs etc...

Good memories.

Reminds me of https://news.ycombinator.com/item?id=45458455

In the AI era, the rec file seems to be a great choice for formatting text that will be feed into LLMs. Imagine converting an HTML table into a RAG file, the context will be much clearer.

Thanks for the submission. I had no idea that existed, but I am definitely going to use this now.

As a similar, yet more powerful data format I started using Nickel (https://nickel-lang.org). It has very sophisticated typing and transformation features. Highly recommended!

2 turtles fucking is certainly a logo choice...

[flagged]

It's a bummer that the library and utilities are GPLv3 - really limits adoption, because it means that product developers can't build it into the kinds of small embedded Linux systems where it would really shine.