Discussion topic 1: vim or emacs for personal wiki, etc?

Instead of my more typical how-to, this posting will aim to solicit input from readers. I realize I may be inviting some sort of flame war, but rest assured that my intentions are sincere: I really am largely ignorant of the respective virtues and and flaws of the two utilities on which I want to solicit input, having barely dabbled in either. My hope is that I might get input here on which will be the more worthwhile one to put further effort into learning.

First, a bit about my aims. I set up for myself some time ago a personal wiki--a vehicle for keeping track of inspiring ideas, tasks on which I am now working, or will need at some point in the future, to work, and a receptacle for various tech tips I have employed and which I may need again in the future to use, but which I have difficulty remembering. I wanted the wiki to be accessible to me, not just at home, but from the internet as well. Much as I wanted to keep the wiki's set-up and maintenance simple, at the time I deemed that deploying it under a web-serving scenario would be required. To that end, I implemented the MoinMoin wiki on a machine I administered.

That scenario has worked out acceptably well over the last few years. But it is now time to take that machine out of service. So I will be needing to reconstitute my wiki and so am revisiting the matter of how I will set up and administer it.

Having a preference for simple, resource-frugal utilities, I am hoping I might migrate my wiki to some command-line interface. The overhead and complexity of the web server most wikis involve is not really justified for my use case: in fact, I might be engaging in a bit of hyperbole in claiming that I use what I have as a real wiki--it's used more like just an organizer.

Under my best-case envisioned scenario, I could either ssh into my machine to consult and/or modify my wiki, or perhaps even host it at a shell account to which I have access. It's an appealing thought and one I hope I will soon be able to implement.

So far as I can tell, the two candidate command-line tools I might use for this are vimwiki and emacs in org-mode. And I must admit that my experience with both has been very slight. In fact, I've tried to avoid using either vim or emacs, typically gravitating to nano for the sorts of needs either of those utilities might otherwise fulfill. Perhaps emacs will be slightly more preferable since development on the vimwikiplugin seems to have ceased a little over 4 years ago, while emacs org-mode seems to have a quite active and extensive user and development base.

Both utilities, with their arcane interfaces and keystroke options have left me baffled and even trapped on more than one occasion. Having a few years of command-line interaction under my belt, I did recently manage a bit of experimentation with emacs org-mode--at least enough to convince me that it could be a suitable new vehicle for my wiki.

I had pretty much written off vim as a possible vehicle since, in past attempts to utilize it, I have found it even more obtuse and intractable than emacs. But that situation recently changed somewhat when I realized that one of the best tools for doing some routine maintenance on one of my Arch systems employs vimdiff. Having used that a few times, I can now say that I've recently managed, under the guise of vimdiff, to use vim successfully for some system maintenance tasks.

And just today I learn that emacs has its own diff implementation--ediff--as well. So emacs might also be serviceable in the system-maintenance capacity, should I decide that it will be more worthwhile to try and better learn emacs org-mode.

Bottom line here is that it looks as though I am going to be using one or other of these utilities routinely, so it is time I started learning it better. And I can, at the same time, use whichever I will be learning better, as the new vehicle for my wiki.

So I am looking for guidance and recommendations on which is likely better to suit my needs and disposition--or whether I might even have overlooked some other command-line utility for creating an maintaining a personal wiki. I should state that I am unlikely ever to do any sort of programming, so whatever may be the relative advantages of either with respect to coding, will be largely irrelevant for me. Rather, I would be using them for perhaps some simple editing functions, and mostly for some routine maintenance tasks (comparing updated config files with files already on my system) and for managing my wiki.

Let the discussion begin.

Afterthought: perhaps even creating a markdown file containing my wiki's material, then converting that to html for viewing with elinks/lynx could even work? In other words, a sort of homebrew solution?

11th installment: lynx; your own personal google scraper

Ok, I'll admit it: there's certainly hyperbole in this entry's title. What I'm doing with the text-mode browser lynx isn't really scraping--it's just something that bears some conceptual (in my view) similarities. It might appear similar because what I've done is to come up with a way of invoking lynx (or any other text-mode browser for that matter), with search terms already entered, from the command line. The end product is just the text results google finds relative to your query--sans all the bells and whistles google's search portal has been foisting on us in recent years. Why is this a significant accomplishment? Well, consider the following.

Background

Have you found google's search portal to be increasingly cluttered and bothersome? I certainly have. Things like pop-out previews do nothing for me but create distraction, and auto-completion is far more often an irritation to me than a help: as a liberal estimate, perhaps 25% of my searches have benefited from the auto-completion feature. For what it's worth, if google wished to provide better service to users like me, they would create two separate search portals: one would be a fuzzy-feely search portal for those who might be uncertain as to what they're seeking and who could benefit from auto-completion and/or pop-out previews; the other would be google's old, streamlined search page and would involve little more than short text summaries and relevant links.

Once upon a time there was a google scraper site at www.scroogle.org--billing itself more as a search anonymizer than as an interface unclutterer--that provided a results page pretty much like the old google one. I used to use scroogle in the days before google introduced some of the more irritating "enhancements" that now plague their site, and came to appreciate above all its spartan appearance. But, alas, scroogle closed its doors in mid-2012 and so is no longer an option. I've been stuck since, resentfully, using google.

In a recent fit of frustration, I decided to see whether there might be any other such scrapers around. As I searched, I wondered as well whether one might not be able to set up their own, personal scraper, on their own personal computer: I had certainly heard and read about the possibilities for conducting web searches from the command line, and this seemed a promising avenue for my query. I ended up finding some results that, while providing but a primitive approximation, look like they may nonetheless have given me a workable way to do the sort of pseudo-scraping I need. Thus, the following entry.

More about the task

Conducting web searches from the command line is another way of describing the task I aimed to accomplish. Granted, doing this sort of thing is nothing especially new. surfraw, for example, created by the infamous Julian Assange, has been around for a number of years and more properly fits into the category of web-search-from-the-command-line utilities than does the solution I propose--which just invokes a text-mode browser. There are actually several means of doing something that could be classified as "searching the web from the command line" (google that and you'll see), including the interesting "google shell" project, called "goosh."

Still, the solution I've cobbled together using bits found in web searches, and which involves a bash function that calls the text-mode browser lynx, seemed on-target enough and something worth writing an entry about. Details below.

The meat of the matter: bash function

To begin with, some credits. The template I cannibalized for my solution is found here: I only did some minor modifications to that code so that it would work more to my liking. There's another interesting proposition in that same thread, by the way, that uses lynx--though it pipes output through less. I tried that one and it got me thinking in the direction of using lynx for this. But I liked the way the output looked in lynx much more than when piped through less, so I decided to try further adapting the bash function for my uses and came up with the following.

The bash function outlined at that site actually uses google search and calls a graphical browser to display the output. The graphical browser part was the one I was trying to obviate so that would be the first change to make. I mostly use elinks these days for text-mode browsing, but having revisited lynx while experimenting with the other solution posed there, I decided I would try it out. And I must say that it does have an advantage over elinks in that URL's can be more easily copied from within lynx (no need to hold down the shift key).

I could not get the google URL given in that example to work in my initial trials, however. This is likely owing to changes google has made to its addressing scheme in the intervening interval since that post was made. So I first used a different URL from the search engine startpage.

After some additional web searching and tweaking, I was finally able to find the correct URL to return google search results. Though that URL is likely to change in the future, I include it in the example below.

What I have working on this system results from the code below, which I have entered into my .bashrc file:

search() { search="" echo "$1" for term in $*; do search="$search%20$term" done lynx "http://google.com/search?q=$search" #for startpage results try "https://startpage.com/do/search/?q=$search" }

Once that has been entered, simply issue . .bashrc so that your system will re-source your .bashrc file, and you're ready for command-line web searching/pseudo-scraping. To begin searching, simply enter the new terminal command you just created, search, followed by the word or phrase you wish to search for on google: search word, search my word, search "my own word", search my+very+own+word, or seemingly just about any other search term or phrase you might otherwise enter into google's graphical search portal seem to work fine.

lynx will then open in the current terminal to the google search results page for your query. You can have a quick read of summaries or follow results links. Should any of the entries merit graphical inspection, you can copy and paste the URL into your graphical browser of choice.

You'll probably want to tell lynx (by modifying the relevant option in lynx.cfg) either to accept or reject all cookies so as to save yourself some keystrokes. If you do not do so, it will, on receiving a cookie, await your input prior to displaying results. Of course you could use any other text-mode browser--such as w3m, the old links or xlinks, retawq, netrik, or any other text-mode-browser candidates as well.

Suggestions for improvements to my solution or offerings of alternative approaches will be appreciated. Happy pseudo-scraping/command-line searching!

AFTERTHOUGHT: I happened upon some other interesting-looking bash functions at another site that are supposed to allow other types of operations from the command line; e.g., defining words, checking weather, translating words. These are rather dated, though (2007), and I couldn't get them to work. Interpreting their workings and determing where the problem(s) lie is a bit above my pay grade: anyone have ideas for making any of these functions once again operable?

Third installment: In praise of newsbeuter

As I've mentioned previously, I'm a T.E.T. (trailing edge technology) type of guy. Rather than chasing after the latest, greatest, electronics must-haves, I'm content to sit on the sidelines watching developments, trying to scheme up ways of getting dated tech to approximate what the new gadgets can do, or sometimes even pouncing on something a bit dated when prices to drop by 50% or more once the novelty's worn off.

That sort of curmudgeonly attitude, augmented by being something of a technological late-bloomer (never touched a computer prior to age 30), it may come as scant surprise that, until fairly recently, I'd done little to nothing involving feed reading. I do recall, when first hearing about RSS, that it seemed like something well-suited to my needs and disposition. But initial attempts at utilizing it did not get very far.

Feed reading seemed attractive to me initially because, given my penchant for T.E.T. and all things low-resource, my preference for things like news reading, which I do a fair amount of, has been, for some years now, largely to use non-graphical applications. What I mean by that is that, when I want to know, via the internet, what's happening in the world at large, I want mostly words, not pictures. And most especially what I don't want when I'm looking for news is advertisements.

Well, advert aversity and my preponderating graphical ambivalence happen to go together quite well with low-resource computing--whether as cause or symptom I don't know. These proclivities led me, for several years, to do most of my news reading under text-mode browsers--primarily links/elinks and friends. I was quite pleased with that arrangement since it obviated a good deal of the bothersome advertising and allowed me to pick and choose which of the more newsworthy graphics I wished to view.

I could, for example, set up links/elinks to use a low-resource program like feh to view the graphics of my choice. That, of course, was in the days when in-line images were the standard for web pages, rather than more modern irritations like java scripts or the even more egregious flash. Should, in those bygone days, any news story hold sufficient graphical interest, I could always copy the URL into a graphical browser and have a gander at the site in its full monty manifestation.

As the web has continued its inexorable march toward chimerical perfection, text-mode browsing--as alluded to above--has become less and less practicable. For some time, one of my primary sources for news was the collection of headlines at the yahoo main page--which used to display quite acceptably in elinks. But yahoo finally sold what remained of its soul to the devil of progress and vaulted itself yet further into the 21st century, updating its main page and thereby causing it to render quite terribly under elinks.

I was adrift for a time news-wise in the wake of the demise of the old yahoo main page. I eventually settled on google news as my news page--once I discovered the proper user-agent incantation; one that would give me the mobile version of the page, containing just a clickable headline along with a line or two from the story. The full version of the page, containing whole opening-paragraphs from the various news stories of the day, was way more news than I was looking for.

But I do like to consult more than one news source each day: for example, I typically look at some Linux news site (Lxer is my preference these days), some boxing headlines, and some MMA news as well. I was ending up with 3 or 4 tabs open in elinks, refreshing each site as I wanted to check for new stories. Which didn't work too badly, by the way.

Recently, though, I decided to have another go at learning about and seeing whether RSS might suit better my news-reading needs. What I ended up discovering was an RSS feed reader that would turn out to scratch, in the most satisfactory way probably humanly possible, the wildest insatiable itch I barely even knew I had (if that statement makes no sense, read on: it might later--though I'm making no guarantees :)). That RSS application was newsbeuter.

Will it come as any surprise if I intimate that newsbeuter is a console program? Probably not. Not only can this application be used from a pseudo-terminal, it can even be used straight from the console! Now that's right up my T.E.T. alley.

"Ok, so what is it?" you ask. Well, it's a terminal application for reading RSS feeds, an application that was developed by an Austrian (as in, fellow countryman of our own American Terminator/Governator) fellow, Andreas Krennmair. The interface, such as it is, quite reminiscent of vi/vim. As he tells us, the name

"Newsbeuter" is a pun on the German word "Wildbeuter", which means "hunter-gatherer". During the stone age, people hunted and gathered their food, and these days, they hunt and gather news and information. Credits for this idea goes to Clifford Wolf, who submitted it to a little competiton that was started when I got aware that the original name would violate French and European registered trademarks.

The screenshot below offers a glimpse at what newsbeuter looks like once it's running:

Each of the entries you see in the screenshot represents an RSS feed. You use the up and down arrows to navigate them, and press the enter key to open the feed, which shows you a list of items or headlines for that feed. Note also the numeral pairs in parentheses, which shows how many items are within that group and how many of those have already been read.

Once you've highlighted one of the items using the up/down arrows and pressed the enter key, you'll be presented by something like what's seen in the screenshot below:

Note the blue bar across the bottom of the window that gives a list of keys that can be pressed and offers a description of what each does.

Again, the up/down arrow keys are used to navigate the headlines, and pressing the enter key on the highlighted headline opens a view that shows the first few lines of the story and that looks as follows (and yes, it's running in a screen session):

Here's where the real magic comes in for us command-line commandos. The application is configured to open the system's default browser for viewing the full story which, on most systems, will be Firefox or some other graphical monstrosity. But newsbeuter offers configuration options that will cause a different, more sane browser to open when the "o" (open) key is pressed, allowing you to view the story in a text-mode browser such as my preferred news reading application, elinks.

To configure newsbeuter to use a more sane text-mode browser, just open (or create, if it's not already on your system) ~/.newsbeuter/config and add the entry html-renderer elinks, and you're all set to read your news stories using elinks. Of course lots of other options can be set within this configuration file--which you can read about at the application's documentation page at http://newsbeuter.org/doc/newsbeuter.html.

Actually, a step that needs to be taken before even the tried-and-true elinks will be of any use to you in reading the news stories linked to, is to configure your feeds. To do that, you open the file ~/.newsbeuter/urls and enter in the URL's of the feeds you wish to monitor. Once that's done, you have a minimal configuration for reading news from RSS feeds with the big boys.

I must confess to being as happy as a clam with this new news-reading scheme--actually, happier. No clam grovelling in the cold mud in the murky depths could ever experience the warmth and satisfaction I've gotten from finding this RSS feed manager, which offers a way to link it up with my favored text-mode browser. So I'm actually happier than a clam--way happier (take that, you smug little bivalve molluscs).

I wish I would have discovered this solution long ago; but part of the reason I didn't was because, simply stated, I'm still very much in learning mode with GNU/Linux system set-up and administration. Had I discovered newsbeuter, say 3 or 4 years ago, I may very well have been unable to puzzle out how to configure and use it.

There are, of course, other command-line RSS feed readers. Perhaps some even work about as well as newsbeuter. I simply haven't looked further yet since what I have looks like it answers to all my needs.

One alternate project that looks interesting is rss2email. I may later have a look at it, but at the moment it's hard for me to imagine what advantages it would have for my purposes over newsbeuter. This entry, from the now--sadly--defunct blog "Motho ke motho ka botho," reviews a few others.

Thus concludes my paean to newsbeuter, the mutt of RSS readers: actually, I used mutt for about a year and gave up on it, going back to Alpine--so I'm more inclined to call it the Alpine of RSS readers. But, whatever.

All hail Andreas Krennmair, command-line RSS manager programmer extraodinaire! He deserves a Nobel prize for command-line RSS feed reader programming! Buy the man a beer in lieu of the Nobel prize he's been stinted if ever you meet him!