Searching And Saving With Plumber
TL;DR, NOW SHOW ME THE DAMN CODE
Standing On The Shoulders Of !/bin/sh
Several years ago, after playing with dmenu a lot, dsearch was born. It was designed to behave as an omni-search bar, staying out of the way and leaving the browser out of most of the searches I did on a computer.
It grew, it grew and grew and grew - incorporating more and more bangs (a la DuckDuckGo)
!yt|!pl
- searching Youtube for videos or playlists!code
- search local code bases!w
- search Wikimedia sites!lyrics
- search for the lyrics of a currently playing song or to match a query!song
- search my music library!g
- search Google
I think you get the point. Anything which could be queried for went through dsearch. It popped up, and got me to my answers, largely without ever needing a browser open for the task. This allowed me to focus on a task, without a large context switch.
Saving Private Files
dsearch became my omni-everything, eventually. There was !rec
to start a recording, !snap
to take a screeshot, !next|!play|!pause
, and it was out of hand. But there was also !save
.
!save
was interesting. I saved things through dmenu! Appended music/videos to playlists, added RSS feeds to my RSS feed aggregator, and stored PDF files to a specific directory with launcher entries, to name some examples. It was powerful, trustworthy, and just worked.
But Plan9 Is Really Cool
I was using Plan9 a lot, and dearly missed dsearch while I was there. I started working on replacement binaries1 to give me at least some of the features back, mostly in Go:
Eventually, after making a few toy file servers I realised I could expose most of my search endpoints as files. searchfs was started, and cat /n/search/youtube/myquery
would list the results of the query.
Under the hood, searchfs would shell out to ytcli, wkcli, gcli; it was fairly circuitous. Invoking the binaries themselves was generally faster, though the eventual goal was to have it run on a very fast desktop computer, shared out on the network to make use of very low latency of a wired connection, and caching results so subsequent identical queries could be made.
It Always Felt Hacky.
I never actually used this, other than in testing. It just wasn’t beneficial to me. Two issues were, there was no method of interface discovery. This was solved, on paper at least: ls /n/search
would show a list of all endpoints, and ls /n/search/someendpoint
would show all the various filters for a search.
There was no dmenu, no easy way to call this that would invoke the search and get the heck out of my way when it was done - it wasn’t close to dsearch. Cool idea though!
Plumbing
I won’t do any justice trying to explain the plumber, and plenty of others have before.
Storage
Plumb allows arbitrary targets to be set, and Plumber allows arbitrary listeners. Using this, I wrote store which is a glorified plumb -d store
that fetches the remote mimetype, and storage. This let me do a few awesome things. Downloading a gist, given a web link is simple.
# These are the plumbing rules used to match this
type is text/plain
dst is store
data matches '$protocol/$urlchars'
data matches 'https://gist.github.com/($urlchars)'
data set 'https//gist.githubusercontent.com/$1/raw'
attr add 'filename=/usr/halfwit/notes/gist/$1'
plumb to storage
plumb client storage
$ store https://gist.github.com/halfwit/somehash
2
Et viola! My computer has a file named /usr/halfwit/notes/gist/halfwit/somehash
! Similarly, rules can be written for any sort of thing I wanted to store. Classy, simple, and the only thing left is to add the store
command to Rio (WIP).
Searches
A small alias (or script) can be used to wrap plumb
, such as alias search="plumb -t query"
in your shell profile. After that, it’s just rules. Preference is king here, and I lean back to my dsearch tokens
query='[a-zA-Z0-9_\-./:,;@ ]+'
# Any type of query you wish to handle would require an entry
type is query
data matches '!yt ($query)'
data set $1
plumb start ytcli search $1
type is query
data matches '!g ($query)'
data set $1
plumb start gcli search $1
# [...]
This allows many things:
- Rewriting rules
search '!yt mything' | dmenu -l 10 | plumb
- Serve up from a faster computer, to slower ones
Future
As this gets more use, I’m certain I’ll end up writing many listeners for the plumber. storage
does the right thing for an arbitrary file, video
comes to mind to send a video to your preferred (networked?) player. This is, in my opinion much cleaner than this, this, or adding features to this.
1 These mostly require API keys, and it’s definitely a compromise that I’ve made. Things exist like Youtube-DL and Googler to be able to search these services without needing keys
2 This requires use of setting the destination port explicitly, so it doesn’t collide with normal plumbs