Eighth Installment: compress and encrypt/decrypt a directory

I recently visited a relative who is studying in the natural sciences and who, surprisingly, is even less capable in certain technical aspects of computing than I am. He was trying to create, on his Mac, a script that would run as a cron job, and asked me for some pointers. Though I know the basics about cron and was willing to pitch in, I wasn't so sure about the script: you see, calling my bash skills rudimentary would be high praise. Nonetheless I decided that, with some web searching, I might be able to assist with that, too. Sure enough, I was able to find just the sort of information that would help us create a script that would tar and compress, then encrypt, a target directory. Details--shamelessly lifted from various locales on the web--are included below.

Over the years that I've been using Linux I have, of course, read more than a few articles that describe methods of encrypting files or partitions. Most recently, for example, there appeared on Lxer an article that described a clever way of encrypting a local directory that then gets backed up to some cloud storage service like dropbox. I've bumped up against the issue of encryption when doing fresh installations as well, as it has been the case for some time now that an option is given on installation for many Linux distros of encrypting, for example, the /home directory.

Despite reading at least some of those articles with interest, I did not feel the need to implement such encryption on my own systems. So it was not until someone else asked my assistance in doing something like this that I actually tried it myself. As you will see, it was actually fairly simple to implement. But first, a few caveats.

I'll skip any details in the following description regarding the cron aspect of this project--not that I could provide a whole lot of enlightenment anyway--other than to say that it's a handy way to make programs or processes run on a set schedule on computers than run *nix. One way I've used it is to cause a weather map, which I've set up as the desktop background on one of my computers, to update every 10 minutes--look for a future entry in this blog on how I managed that.

I'll also not speak in any depth about another ingredient in this recipe--tar--other than to say that it is an abbreviation for for "tape archive." I myself do not understand its workings terribly well, though I've used it on several occasions. I will mention on a related note, however, that, in my research, I ran across articles that used another, similar, utility--dd (an abbreviation for "disk dump")--to create compressed and encrypted archives. But I did not follow up on the dd option and so cannot post any further information about how that was done.

Finally, I can't speak in any depth about the program I used for doing the encryption--openssl--or about another program with which I experimented and which also does encryption--gpg. But I promise, despite those rather glaring deficits, that I will describe something I managed to accomplish and which you, too, should be able to accomplish by following the steps outlined.

Perhaps in some future entry for this blog I'll be able to further explore tar, dd, and/or cron. But for now I'm going to focus my attention mainly on the option we ended up using, which involved mainly tar and openssl.

The relative in question, as I mentioned, works in the natural sciences. He has a directory of his ongoing work that he wants to back up regularly, but to which he does not want anyone else to have access. His choice for backing up beyond his own PC, is to use dropbox. So the task was, as mentioned, to compress and encrypt the target directory: moving it to the location on the local machine where dropbox would find it so as to back it up will also not be covered in this write-up, though that step did end up being part of his final resolution.

So, what's left? It was quite easy to find directions of the web for doing all this. I pretty much went with the first workable solution I found, which came from the linuxquestions forum (the relevant thread can be found here).

The incantation we used was as follows:

tar -cj target-dir | openssl enc -aes128 -salt -out target-dir.tar.bz2.enc -e -a -k password

What that line does may be evident to most, but I will offer a bit of review nonetheless. The target directory is first tar'red and compressed with bzip, (the c option stands for "create" and the j option specifies that the created file should be compressed with bzip2) then it is piped to openssl for encryption. The word "password" is, obviously, to be replaced by whatever password the user chooses.

One possible drawback to this method, as pointed out in the thread from which I lifted it, is that the encryption password gets entered, in plain text, right on the command line (which is slightly less of an issue with a cron script such as we were creating). Thus, anyone who can gain access to the machine can, by using the command line history, see what the encryption password was. Since someone gaining access to his computer and viewing the command line history was not a concern for the fellow I was helping, this is the solution we implemented. But that potential concern can be easily remedied by simply leaving off the -k password switch at the end, which has the effect of prompting the user for a password, which does not get echoed to the command line.

To decrypt the file, the following command--which prompts for the password--is used:

openssl enc -aes128 -in target-dir.tar.bz2.enc -out target-dir.tar.bz2 -d -a

The file can then be uncompressed and untar'red. This part of the process could likely be reduced from two steps (decryption, then uncompression/untar'ing) to one by using a pipe, but since it was presumed, for purposes of this project, that the file would act simply as insurance against loss--the need for ever actually recovering the content being very unlikely--I did not pursue streamlining that aspect.

I did manage to find and test a couple of other variants which I will offer here as well. The second variant was found here, and follows:

tar -cj target-dir | openssl enc -e -a -salt -bf -out target-dir.blowfish

It is much the same as the first variant, though it uses a different encryption method called blowfish. I am uncertain which of these two encryption schemes is considered better. To decrypt the compressed directory, the following command is used:

openssl enc -d -a -bf -in target-dir.blowfish -out target-dir-decrypt.tar.bz2

Finally, I discovered yet another variant, details about which can be found here. A sample of how to use this one is as follows:

tar -cjf target-dir.tar.bz2 target-dir/ | gpg -r user -e target-dir.tar.bz2

As will be noted, this variant uses gpg to encrypt the directory. Of course user must be replaced by the name of someone who has a valid gpg key on the system, usually the primary user of said machine or account.

An interesting feature I discovered about this method is that a time-sensitive gpg key can be created, i.e., one that expires after a certain interval. If I understand correctly how this works, once the key expires, the directory can no longer be decrypted.* This feature should, obviously, be used with care.

Decrypting the directory can be done in the following way:

gpg --output target-dir.tar.bz2 --decrypt target-dir.tar.bz2.gpg

The same two-step process of decrypting, then untar'ing/uncompressing, applies to these two methods as well.

This sums up what I have to offer in this entry. Now that winter is upon us northern-hemispherers, I may be able to post more frequent entries. There are a few things I've been wanting to document for some time now.

* Correction: an anonymous commenter writes of my claim that the key expiration makes decrypting the file no longer possible that "Unfortunately not. The expired key is no longer trusted but is still functional."

Seventh installment: imagemagick trick 1 of (?)

During a recent vacation, I needed to convert some digital images taken by a relative's camera from a raw format into something that would display better on my laptop. As usual, imagemagick came to the rescue. Details are included below.

imagemagick has been called something like "the swiss army knife of image manipulation utilities." I've utilized this application on many occasions over my years as a GNU/Linux user and been amazed to have as yet found nothing that it cannot do. Moreover, it's a CLI-friendly tool, which makes it doubly appealing for someone like me who favors using the least resource-intensive application for any given job.

I should mention that, not only was I converting these images to a different format with which my laptop could better deal, desirable, but resizing them was needed as well. You see, each photo taken by this fairly high-end camera came in at about 25 megabytes, and the dimensions of each image were probably in thousands of pixels. So I wanted to both resize and convert them; but, based on past experience with imagemagick, I was sure it would be up to the job.

I was surprised, after some initial searching on how to accomplish this task, at how simple this task would turn out to be. After mounting the Secure Digital card on my laptop, I copied over all the images--which had the .CR2 extension, by the way--to a temporary directory on my laptop.

The incantation I then used, after cd'ing to the temporary directory where the images resided, was mogrify -format jpg -geometry 1024x768 *.CR2. It should be fairly obvious what this does, but just in case it's not, let me explain. The mogrify command is part of imagemagick's image manipulating routines and can be invoked to do a number of things with an image, but in this case the -format switch, followed by the jpg extension, tells it to convert from one format to another.

Though most readers of this blog are likely to know this already, it will be worth mentioning that the asterisk preceding the file extension is the wildcard symbol. Thus, all files in that directory with the designated extension get converted. Incidentally, they are given the same prefix as the raw .CR2 files, but after conversion have the .jpg extension instead of the .CR2 extension.

I should mention here that, although raw formats are not especially new, I did run across some indications while researching how to do this, that older versions of imagemagick might not be able to operate on such formats. But I believe any version that's come out in the last 2 or 3 years should understand and be able to manipulate these raw formats.

The -geometry switch tells imagemagick to resize the image from something in the thousands of pixels down to a size that displays better, and takes up a whole lot less disk space, on this laptop. The switch does also have a somewhat undesirable effect on photos taken in vertical rather than horizontal orientation, since, as written to disk, these are taller than they are wide (whereas photos taken in horizontal orientation are wider than they are tall): the program thus makes the height of the photo 768 pixels, while the width, being kept proportional, turns out to be quite a bit less than that. But for my purposes--which was simply being able to view quickly these fresh photos on my laptop--these somewhat under-sized photos did not pose any particular problem.

The process of converting many photos this way can be lengthy, but, as you have seen, it is quite simple. I would simply start the process, then either do some other task in another desktop while awaiting completion, or just set the laptop aside while it was running the process. After about a half hour or so, I was ready to view photos.

Above is a sample of one of the converted images. Oh, and by the way, the "(?)" in the title of this entry is meant to indicate the multitude of other imagemagick tricks, as yet unknown to a relatively casual user such as myself, there are likely to be.

Addendum to the first installment: yet more on screencasting with ffmpeg

With a recent upgrade (apt-get dist-upgrade to 12.04, custom-built Ubuntu) to my office machine, I started having serious audio/video sync issues when producing screencasts using recordmydesktop. As you may (or may not) recall, I record lectures on my computer and use, as a sort of visual aid, an on-screen whiteboard, where I type key words or phrases about which I'm speaking: that's what I capture from my screen when I'm recording these screencasts. Well, with the updated recordmydesktop, the text on my on-screen whiteboard would begin to appear several seconds before the audio about the word or phrase was indicating its appearance.

Oddly, the opposite was happening with the updated ffmpeg when running the screencasting incantation with which I'd earlier experimented. The on-screen text I was typing into the whiteboard was lagging a bit behind the audio.

I tried introducing a number of alternative switches into the commands I was issuing when using both recordmydesktop and ffmpeg. But to no avail: I couldn't get rid of the sync problems with either.

During the course of my searches aimed at resolving these issues, I ran across a crude script on the Ubuntu forums that someone had cobbled together, a script which uses ffmpeg, but which separately records video and audio, joining the two streams together, as a final step, into a final output file. I think this joining of an audio and video file are called, in electronic multimedia circles, "muxing," by the way. I decided the script was worth a try.

And, what do you know, after figuring out how to use the script, my tests indicated that it caused audio and video to be in nearly perfect sync. Thus, the answer to my newly-appeared screencasting issues was resolved.

I hoped to solicit improvements to the script but have so far not managed to find much help. Probably the weirdest thing about this script, which likely demonstrates the inexperience of the script's creator, is the fact that, once ffmpeg is invoked and the recording begins, you're supposed to enter, into the same terminal where ffmpeg is running, a file name, then hit the "enter" key as the singal to stop the recording and begin the joining of the audio and video files.

All this while you're seeing in the terminal the standard ffmpeg prompt that tells you to hit control-c to stop the recording. Confusing, to say the least--and made even more confounding by the fact that you can't actually see the text you're entering when you go to type the file name.

Despite those shortcomings, since the results produced by this script exceed anything else I've been able to accomplish, I think I'm going to stick with the script for now. I have made a couple of tweaks, mainly so as to make it record what are called "lossless" files--files that are produced with minimal processing (for example no compression), and which are therefore quite large. I have to re-encode my output files before uploading them in any case, so it's best to start with better-quality files.

Without further ado, then, I present the tweaked version of the script I've found:
#!/bin/bash #vzybilly #these are temp files aud="aud.flac" vid="vid.mkv" #grab audio & pid ffmpeg -f alsa -ac 2 -i plughw:1,0 -acodec flac $aud & audPID=$! #grab screen & pid ffmpeg -f x11grab -s "830x660" -r "24" -i :0.0+227,130+nomouse -threads 0 -vcodec libx264 -preset ultrafast -crf 0 $vid & vidPID=$! #wait, till name given (that means stop) read -p "Stop by giving an Output video name?" out #alternate, more coherent way of output file naming--requires zenity #out="`zenity --entry --title="Video muxing script" --text="Please type a file name (sans extension) into the blank and click 'Ok' to stop the recording"`.mkv" #stop audio and video with pids kill -n 2 $audPID kill -n 2 $vidPID echo "Saving to $out" #combine to the target output file ffmpeg -i $aud -i $vid -acodec copy -vcodec copy "$out" #purge the temp files rm $aud rm $vid
I should mention that I discovered a slightly less incoherent way of soliciting file-name input. Though it is commented out in the version of the script you see above, I intend to use this until such time as the script can be further improved ( to use it you uncomment the line that begins out= and comment out the line that reads read -p: note that you must have zenity installed for this modification to work).

In a related item, as I've mentioned earlier, the disadvantage to using ffmpeg for screencasting is that there is no built-in provision for pausing. Well, apparently someone has proposed a kind of workaround for that--see this thread for further details. I've not tried that method and don't really understand how it works, so I cannot attest to its efficacy.

What I have tried is simply stopping, then restarting a new file when a pause is necessary. That's definitely more cumbersome than pausing, and, furthermore, it requires the additional step of somehow joining what could be thought of a separate "vignettes" into a single "episode."

The good news I can report on that front is that I've found another script that was created precisely to join such separate files. I've tried some tests with it and it has worked for me quite well. It's called mmcat and it can be found here.

I'd like to post more about the plughw switch seen in the above script and which I needed to introduce in order to record through a new USB sound device I've added to my computer. But I don't really understand well what differentiates it from the more standard hw switch. So I won't speak to that matter further in this entry. :)

That about sums things up so far as recent screencasting developments on my front is concerned. Do you have any suggestions for improving the screencasting script I found? If so, please pipe in. Any other suggestions for pausing ffmpeg screencast recording? Please let me/us know.

Addendum to the first installment: more on screencasting with ffmpeg

Long time no blog. I got busy with a lot of other things, not least of which was various bicycling trips and associated maintenance projects. Amazing how much time and energy those things can take.

With the academic year approaching, coming up with a resolution to my screencast file-size problems has taken on renewed urgency. To reiterate, a recent major hardware upgrade on my computer led to a twofold increase in the size of screencast video files I produce on this machine--which in turn led to upload problems at my course web site (file size limits). It looks like the resolution may well be provided by ffmpeg.

So I began once again grappling in earnest with the increased file-size issue. I've actually even found a resolution, should I decide to switch over from the more capable recordmydesktop to the nice but somewhat feature-lacking ffmpeg.

Before launching into a description of that resolution, however, I need to make quick note of a new entry into the field of GNU/Linux screencasting applications. The new kid on the block is called Kazam and it looks promising. I've not yet been able to get it running--and I'm not in a big rush to do so, since I favor command-line tools. But it's received some praises and looks like a promising application. That noted, I now return to a description of the resolution I've discovered for the issue of screencast file sizes made using ffmpeg's x11grab.

First and foremost, I discovered a fairly lengthy thread on the Ubuntu forums that deals with screencasting usingffmpeg. It's chock full of all kinds of interesting tips for screencasting with ffmpeg. For example, a switch is documented there that allows you to exclude the mouse cursor from your screencast: you simply add :0.0+nomouse to the command you use to start your screencast. I'd definitely need that in order to use ffmpeg as the utility to record my lectures.

Also described there is a way of doing what might be called "pseudo pausing," which really means you just stop your recording at the point where you need to pause, then start up again with a new recording. You then need to concatenate the files--not anywhere near as convenient as recordmydesktop's pausing capability, but something that'll do in a pinch.

It was from that thread that I derived what seems like a pretty good resolution to my file size issues. What I've discovered is that I can record my files in a lossless format (mkv is recommended in that thread)--which results in pretty gigantic file sizes (ca. 13 megabytes per minute on this system)--then transcode them into flash format while dramatically decreasing the file size: a one-minute test file I made actually came in at about 1.2 megabytes after re-encoding using this method. Since what I need is a file that comes in at about 2 megabytes per minute, this could work well for my circumstances.

The key command that's allowed me to reduce dramatically screencast file sizes to an acceptable level while at the same time converting to a format that works well at my hosting site uses . . . you guessed it, ffmpeg. Here it is:  

ffmpeg -i output.mkv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flv

I do still hope I can find a way of reliably shrinking the files recordmydesktop outputs, but my research on that has, to date, not been fruitful. There's a utility called oggResize that's supposed to allow you to easily resize .ogv files, but it puts the audio out of sync with the video. And it appears that tool is not under active development. It's likely resizing of .ogv files can also be done with ffmpeg, but I've not yet managed to arrive at the appropriate incantation for doing that. I'll keep digging though, and if I come up with a resolution, I'll post it in this blog.

As a final note on things I've learned in this round of screencast research, I should also mention a command-line tool called ffcast someone has developed. This looks like a bash script that first invokes a tool for selecting an area of the screen, then calls ffmpeg to record the screencast. I've also not tried that one but it's on my list to look into in the future.

POSTSCRIPT: I decided to run on an .ogv file created by recordmydesktop the same command I'd used to shrink the lossless .mkv file to see what the results would be. I'm pleased to report that the results are positive: the .ogv file, when re-encoded as an .flv using that method, shrunk to about one third the size (the 1 minute .ogv test file was 4.2 megabytes and it came in as a 1.2 megabyte .flv file after re-encoding). So I can continue using recordmydesktop to record my lecture screencasts--albeit with the penalty of having to do an additional round of encoding/re-encoding (recordmydesktop already takes quite some time to encode a screencast into .ogv format unless you use the --on-the-fly-encoding switch, which I'm now likely to start using). So, the command I used to re-encode the .ogv file to .flv is:

ffmpeg -i output.ogv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flv

MUCH LATER POSTSCRIPT: I've discovered that ogv files work much better at my hosting site than do flv files, so I began looking into ways of transcoding/shrinking to that format. The incantation I discovered, through trial and error, and which reduces screencast file sizes to about 1.5 megabytes per minute (roughly one seventh of the original, lossless, size), is as follows:

ffmpeg -i infile.mkv -r 13 -acodec libvorbis -ab 48k -ac 2 -vcodec libtheora -preset slow -wpredp 0 -crf 22 -threads 0 outfile.ogv

There are two crucial bits here. The -ab 48k option is one of them: that reduces the audio bit rate, which brings the file size down by quite a lot. The other is the -r 13 option, which reduces the frame rate to 13 frames per second and thus shrinks the file size yet further. A 48 minute long lossless screencast I created, for example, was reduced to a 73 megabyte ogv file using this incantation.

Sixth installment: establishing wifi connections from the command line

Reader's Digest condensed version

Want to get on a wifi network fairly quickly and painlessly from the command line? Try network-manager's command-line cousin, cnetworkmanager. A command like sudo cnetworkmanager -C essid-of-network --unprotected (for unprotected networks) will do it. For further details on how I discovered this and for further remarks on cnetworkmanager's capabilities, read below.

Unabridged version

Trailing edge technologist that I am, I'm also a fan of minimalist desktops. I've used mainly ion2/3, dwm, and evilwm for day-to-day computing--though I've tried out many others like JWM, icewm, fluxbox, blackbox, twm, openbox, etc.

After my netbook's keyboard went on the fritz subsequent to having water squirted on it by a loved one (I can say she's a loved one in hindsight), I decided to dig out an old laptop and press it back into service. It is, of course, a single-core machine with a Celeron 2.4 Ghz CPU and only 512 MB RAM. So, another ideal candidate for a minimalist desktop.

I wanted to get the machine up in a hurry so I chose to install a lighter-weight *buntu on it--namely, Lubuntu. That variant uses the lightweight LXDE desktop.

After I'd gotten it set up and used it for awhile, I decided to try paring back the system resource usage yet further. The main means toward that end was to install the evilwm window manager which, though I understand it is not quite as lightweight as dwm, nevertheless looked like a good choice for this machine.

It took some trickery to be able to load into the evilwm desktop once the machine was booted. Lubuntu, you see, uses the LXDM login manager, and I was unable to create a valid menu entry for an evilwm session. So I used the quick and dirty method: cobble together an .xinitrc for evilwm, kill LXDM, then issue startx from the command line.

I'm in the initial stages of possibly replacing LXDE with evilwm on this laptop, but I've gotten at least partway there. I want to recount in this entry how I managed to get wifi connections working under evilwm. But first, a relevant digression.

I used to use command-line routines on another old laptop some years ago when connecting to wifi networks. That involved using commands such as iwlist and iwconfig, then issuing a dhcp request on the wifi NIC. At a certain point, however, my trusted command line routines stopped working for reasons I could not understand.

Searching for alternatives, I discovered wicd. I actually doubted it would work since, as I understood it, wicd was simply a series of python scripts that invoked the same command-line binaries I'd been unsuccessfully attempting to use.

But wicd did work, and to my delight, I even discovered that it had a curses, terminal interface: I thus needed not be bothered with using a pesky gui utility to employ wicd. I subsequently came to rely on wicd for establishing and managing wifi connections.

Getting back to evilwm; those who are familiar with that window manager will be aware how minimalistic it is: there are no taskbars, no system trays, no pop-up menus. No, all evilwm does is display the desktop background you specify and allow you to open graphical applications (invoked from a terminal or by using something like xbindkeys and gmrun).

That sparseness presented an issue with wifi connections once I started switching from LXDE to evilwm: you see, Lubuntu has set up this system such that it uses network-manager for managing wifi connections, and network-manager, in turn, expects there to be a system tray in which it can reside. Given that architecture, I could see no way in which I could use network-manager for my wifi connections while running evilwm.

My first assumption was that my old standby, wicd, would allow me to manage wifi connections under evilwm. So I installed it. But it would not, no way, no how, allow me reliably to connect to wifi networks.

I am uncertain why this problem exists. I read that it was related to some conflict with network-manager, and that the solution was to uninstall network-manager. But doing that did not resolve my problems: wicd would, for the most part, give me a "bad password" error and would not connect to the network: perhaps once in every thirty tries I could get on-line, but the other twenty-nine times, I would get the bad password error.

Short story long, I had to abandon wicd and look into other ways of connecting my laptop to wifi networks using command line utilities. My attempts at finding an alternative to wicd finally resulted in the discovery that there is actually a command line version of network-manager called cnetworkmanager.

The happy ending to this story is that cnetworkmanager's man page is fairly simple and easy to comprehend for the non-specialist. I was, in very short order, able to bring up my wifi network connection using cnetworkmanager. Below are salient details.

In order to find the name of your wifi network interface and, at the same time, scan for essid's of detected networks, the command cnetworkmanager -a -d run in a terminal does the trick (-a is the option that lists available networks, -d is the switch that tells you the names of wifi interfaces on the machine being used). With that information it is fairly simple--though super-user privileges are required for this--to a network as follows.

For WPA-encrypted networks, the command sudo cnetworkmanager -C essid-name --wpa-pass=password will do the trick. For unencrypted networks, sudo cnetworkmanager -C essid --unprotected does the job.

It's a bit clunky, but provides a workable solution that gets me by for now.

An explanation of html tags used at blogspot

This is just a sort of filler entry, for my own reference. It will contain an explanation of html tags blogspot posts use, and is intended to help me utilize sane editors for creating my blog entries rather than the ungainly blogger web interface. See below.

Here's a first paragraph. It consists in two sentences and, after the pariod, an extra carriage return so as to space the paragraph apart from the following paragraph.

Here's a second paragraph. It, too, consists in two sentences followed by an extra carriage return--as do all subsequent paragraphs in this test.

Here's a block quote. As all paragraphs should, it consists in at least two sentences.

Now, here's a paragraph after the block quote. Once again, we see two sentences.

Now, for a fourth paragraph that contains a link. The link is here.

Moving on to the fifth paragraph, we'll enter some bold, the italic, text. Here's the bold. Here's the italic. And here's some underlined text.

For the sixth paragraph, we'll change the font size. Let's try what they call a "large" font. Now, back to "normal" font.

In the seventh paragraph, we'll change the font face. There's not much to chose from, so here's arial. Now, back to the "default" font.

Now, we'll try inserting a numbered list, then a bulleted list. Here's the numbered list, in a new paragraph (apparently, the extra carriage return is not needed at the beginning or end of lists):

1. item 1
2. item 2
3. item 3

Now, the bulleted list, in a new paragraph:

• item 1
• item 2
• item 3

I'd like to insert a table, but I don't see any way to do that. So, let's try some strikethrough. Very impressive.

And, after this eleventh paragraph, a pseudo-list of a few lines of text, each on a separate line:

one
two
three

Finally, just for kicks, let's insert in this twelfth paragraph what they call a "jump break" (not sure what that is, but . . .) 

Addendum to the first installment: alternate screencast applications

As you may recall, in the inaugural entry for this blog I described a crude mock-up I'd cobbled together, using Linux utilities, for producing screencast video lectures along the lines of those found on the khanacademy web site. I recently had occasion to explore some alternate screencast applications, since ogv files ceased playing back sanely on one of my systems. That issue was ultimately addressed by "upgrading" the video card in my aging Pentium 4 computer (with an old 128 MB nvidia card I got for $15). But not before I investigated some alternatives to recordmydesktop.

Of course there are various options for screencasting in Linux: I've read about Istanbul, Byzanz, vnc2flv, and some others, for example. I've so far gravitated toward recordmydesktop because it "just worked" and because it can be used from the command-line. But because of the video card issue mentioned above, I did recently decide to explore a couple of other alternatives which I'll write about in this entry, namely xvidcap and ffmpeg.

I should mention at the outset that, as those who read the first installment of this blog will be aware, I need something that not only records a video of a part of the desktop (actually, a particular application window in my case), but that allows for sound recording as well: after all, these are lecture videos I'm producing. So any screencast application that does not have built-in sound recording isn't going to work for my purposes.

Screencasting with ffmpeg

Most readers of this blog will have heard of the indispensable video-manipulating program ffmpeg. It's possible to do a truly amazing amount of manipulation and creation of videos using that program. And, as it turns out, it's even possible to do screencasting with it (in fact, ffmpeg may actually be used "behind the scenes" by recordmydesktop--though I'm not fully certain about that).

Some recent research on the web resulted for me in some successful experiments in screencasting with ffmpeg. The main ingredient in this screencasting capability is a switch called x11grab. Run from the command line, an ffmpeg screen/sound capture session would be invoked something like this:

ffmpeg -f alsa -ac 2 -ab 48k -i hw:0,0 -f x11grab -r 20 -s 800x600 -i :0.0+227,130 -acodec libmp3lame -vcodec libx264 -vpre lossless_ultrafast -threads 0 output.avi

That command will output an avi file called, appropriately enough, output.avi that is 800x600 resolution and that grabs a portion of the screen that is 227 pixels from the left edge and 130 pixels from the top edge (about where I open the application I wish to record on this machine). The sound portion is transcoded into mp3 by libmp3lame--something I discovered reduces the final size of the file considerably.

I believe ffmpeg can output just about any video format, including ogg theora, as recordmydesktop does. In order for that to be implemented, the -vcodec argument needs to be modified to, in the case of an ogg theora file, to -vcodec libtheora, I believe, while the extension of the output file's name would need, obviously, to be changed to .ogv.

As those who are familiar with ffmpeg will know, I'm barely scratching the surface of the tip of the iceberg here as far as its capabilities go. In fact, ffmpeg's man page has got to be one of the most voluminous and daunting of them, and it's only within the last couple of years that I've begun to be able to make any sense of it. But don't presume that I mined the information above mostly from the man page: rather, I found a working sample command on the web, then went back to the man page to try and better understand how it works and how I might tweak it for my purposes. I'm really still on a very rudimentary level when it comes to understanding and using ffmpeg.

Now, though this ffmpeg solution works quite well, it's turned out to not really be usable for me. This is because, as great as it is, the ffmpeg process cannot be paused while it's running: you either have to stop it and resume anew if you have some reason to pause, or else you'll need to edit the resulting video and cut out  the extraneous portions. In case it's not apparent, it's really useful during a lecture to be able to pause and resume.

So, as well as it worked in my experiments, I decided that, until I can come up with some way to pause the process, then resume, ffmpeg is not going to work for me as well as recordmydesktop does.

Screencasting with xvidcap

Which bring my to the next screencasting application, xvidcap. There's a lot to like about xvidcap. It's a graphical application, but with a very minimal interface--just the type of gui application that appeals to me. It draws a nice, visible perimeter around the area you're recording, and the perimeter can be easily manipulated with the mouse to enclose whatever quadrangular area you'd like. And it does allow for pausing and resuming of the screencast.

A screenshot of xvidcap (running on someone else's machine)

Furthermore, it's able to output a few video formats, among which are mpeg and flv. It looks like a great application and I really wish I could use it. But there's a major problem with sound: you see, xvidcap was developed during an era when oss (Open Sound System) was the standard Linux sound server, but oss is now deprecated in favor of alsa (Advanced Linux Sound Architecture).

In theory, it's possible to emulate oss with alsa. That's what you'll read on the internet, anyway. But the fact of the matter is that, as Linux moves further and further away from the old oss architecture, emulation of it has become harder and harder. I spent quite a few hours trying to implement it on this system, so happy was I with what I'd seen of the video-capturing end of xvidcap. I tried a number of purported solutions I found in my web searches. But none of them enabled me to record sound alongside the video I was capturing with xvidcap. So, I've reluctantly given up on it for now.

Those are the two alternate screencasting programs I've toyed with lately, and with mixed results. Both seem to have their advantages as compared to recordmydesktop, but in the end, it looks as though neither is going to be able to displace it. If you have any sort of pointers for addressing either of the issues I experienced in my attempts to use those applications, I'd be most delighted to hear about them. So please do offer your input.

Segmenting video files

To wrap up this posting, I need first of all to confide that I finally did "upgrade" my old work PC. I replaced the aging Pentium 4 with a not-quite-so elderly dual core machine. So video recording, and especially playback, goes quite a bit more smoothly on this "new" machine.

One odd thing has cropped up regarding video, though. The problem is that, despite the fact that I use the same exact recordmydesktop command to record my screencasts, the resulting files are now almost twice as large as they had been on the old machine: instead of being on the order to 2 megabytes per minute, they're now closer to 4 megabytes per minute.

My suspicion is that the video hardware could be to blame, since I had a pretty old 32 megabyte ATI (PCI) video card in the old machine, while the newer machine has an nVidia 128 megabyte card (PCI-express). I've tried switching video card modules from nv to vesa, but that doesn't seem to affect the resulting video size.

This becomes a problem because I have a file size limit on the site where I must upload my lectures, and they're now routinely going to exceed the size limit. So, until I can figure out how to reduce the video file sizes back closer to what they were, I've had to come up with a work-around--namely to split the video files into parts that, individually, do not exceed the size limit. And for this, ffmpeg once again comes to the rescue.

In order to split my lecture files, I'm using the following command:

 ffmpeg -ss 00:00:00 -i in.ogv -t 00:30:00 -vcodec copy -acodec copy out.ogv

What that does is to start at the beginning of the file (the -ss 00:00:00 part), go 30 minutes into the file, then copy that section to a new file called out.ogv. So, for an hour-long lecture, after having done that to split off the first 30 minutes into a new file, the same command would be run again, except that-ss 00:00:00 would be replaced by-ss 00:30:00. Then, the two parts could be appropriately named (with part1 or part2 in the name, as appropriate) and uploaded.

That's all for this entry. If anyone has tips or recommendations about what's been discussed above or about anything else related to screencasting on Linux, please pipe in.

Afterthought: here's a link that offers some technical details on the sound hardware in my new computer: http://people.atrpms.net/~pcavalcanti/alsa-1.0.15rc2_snd-hda-intel.html#final . I was looking at that as I was trying to work out how I might get xvidcap functioning on this system.

Addendum to fifth installment: prescience?; more on xmodmap

Prescience?

Ok, grappling with this xmodmap issue has caused me to revisit the issue of inputting non-latin characters under Linux, something I dealt with a few years ago. Finally, with helpful hints gotten in on-line forums and reading various README files, I had some success. I observe now that I displayed a bit of prescience back then as well.

For posterity's sake, I'm pasting here the resolution I came up with--the prescient bit (written almost 8 years ago) highlighted in bold:

. . . I'm taking the other approach I mentioned earlier (editing XF86Config-4 to allow for keyboard map/font switching for X apps).  He's also helped me out with this. . . . there is an additional line possible in the keyboard stanza of XF86Config-4 (this refers to 4.3.0something and may not apply to earlier releases): the XkbVariant" line.  In order to get the phonetic Russian and polytonic layouts (or at least a semblance of polytonic) I want, I've added the "XkbVariant" line with ",phonetic,polytonic" following.  That has gotten me most of the way to where I need to be in entering unicode input into Xapps.  It also works in a console if you have the right fonts installed.  As an additional detail, I should mention that I think you must have a unicode locale specified in your system for this to work.  In my case, I've set locale to en_US.UTF-8 (how to set that varies by distribution: dpkg-reconfigure locales is how Debian does it).  I'll provide the relevant section of my XF86Config-4 for those interested:

Section "InputDevice"
Identifier      "Generic Keyboard"
Driver          "keyboard"
Option          "CoreKeyboard"
Option          "XkbRules"      "xfree86"
Option          "XkbModel"      "pc104"
Option          "XkbLayout" "us,ru,el"
Option          "XkbVariant" ",phonetic,polytonic"
Option          "XKbOptions" "grp:alt_shift_toggle,grp_led:scroll"
EndSection

Note the "grp_led:scroll" segment too, which causes the scroll lock LED on the keyboard to light up when I'm in one of the alternate keyboard/font mappings.  In summary, I've written this in the hope it may help someone else wanting to do this under Linux. . . . my little research/how to project here could likely be itself obsoleted pretty shortly.

Now, as those of you who've been working with GNU/Linux this long will know, we no longer edit GUI configuration files by hand: these are to be auto-generated by the system. But sometimes, for whatever crazy reason, you'll need to do some tricky thing like I was doing then so that you can switch keyboard layouts on the fly.

In current circumstances, for example, my work computer uses a very minimalist desktop that includes no keyboard-layout switching utility/applet--something Gnome, KDE, XFCE4, and the like, do include. So, if I want to implement keyboard-layout switching there, I'm going to need a solution like the one I implemented several years ago.

I suppose it's still possible to hand-edit the GUI configuration file--now called xorg.conf, as we may all know--but that, I believe, is not the canonical method. Instead, what's now expected is that modifications such as the one above be entered into a different configuration file, from whence they get loaded into the xorg.conf file the system automatically generates.

That file is located (on Debian-based systems) in /etc/default and is called, appropriately enough, "keyboard." The content needed to implement the modifications listed above--along with further context from that file--will look as follows:

# If you change any of the following variables and X is configured to
# use this file, then the changes will become visible to X only if udev
# is restarted. You may need to reboot the system.

# The following variables describe your keyboard and can have the same
# values as the XkbModel, XkbLayout, XkbVariant and XkbOptions options
# in /etc/X11/xorg.conf.

XKBMODEL="pc105"
XKBLAYOUT="us,ru,el"
XKBVARIANT=",phonetic,polytonic"
XKBOPTIONS=""ctrl:nocaps,grp:alt_shift_toggle,grp_led:scroll,terminate:ctrl_alt_bksp""

# If you don't want to use the XKB layout on the console, you can
# specify an alternative keymap. Make sure it will be accessible
# before /usr is mounted.
# KMAP=/etc/console-setup/defkeymap.kmap.gz

There's my demonstration of prescience for the year (I hope that doesn't change again within the year). Now, on to a revisitation of the xmodmap issue.

More on xmodmap
Turns out the xmodmap issue is in no way unique to XFCE4: rather, it seems to affect many, and perhaps all, graphical environments. As a moderator from the Arch Linux forums put it almost exactly two years ago, "This is an unsolved problem for quite some time" (see this thread for a bit of further detail).

The symptom on the machine I'm trying this with is that it xmodmap will not initiate when the GUI comes up, despite my attempts at placing commands intended to invoke it in relevant start-up files: xmodmap will only take effect when run manually after the graphical environment is running. Even then, it causes 100% CPU usage for several seconds while it is initiating (this is a pretty low-resource system, at only 800 Mhz, which partly explains the delay).

What is the nature of this problem? Why can't this re-mapping or re-binding be done in connection with the user's log-in and graphical environment's initiation? Does it have something to do with evdev which, as I understand it, is what implements the system's auto-configuration of peripherals like keyboard and mouse? I simply do not understand, unfortunately.

But this matter seems to have foiled greater minds than my own. As a case in point, while trying to resolve this and similar problems, I bumped into the following example in the Xsession man page--which even seemed calculated to address specifically the problem I was trying to resolve:

Here is an example of how one might write a script, named 40custom_load-xmodmap, to invoke xmodmap(1):

SYSMODMAP="/etc/X11/Xmodmap"
USRMODMAP="$HOME/.Xmodmap"

   if [ -x /usr/bin/X11/xmodmap ]; then
      if [ -f "$SYSMODMAP" ]; then
           xmodmap "$SYSMODMAP"
      fi
     fi

    if [ -x /usr/bin/X11/xmodmap ]; then
        if [ -f "$USRMODMAP" ]; then
            xmodmap "$USRMODMAP"
        fi
     fi

Well, I tried that. I created a file with that content, put it in the recommended directory (/etc/X11/Xsession.d) and gave it the same permissions as the other files in that directory. But the xmodmap modifications I'd stipulated were not in effect once the graphical environment came up: I still had to run xmodmap manually.

The only thing approaching a resolution I've found that's calimed to work, is a really ugly looking hack I read about on the Archbang forums (you can read about it here). I haven't tried it yet and so cannot confirm that it works: and I hope to avoid trying it. I really hope there's something a bit less kludgy that'll resolve the issue.

As for things I might attempt, I've thought of entering each of my eight modifications on separate lines in the .xsession or .xinitrc file to see whether that'll cause the modifications to take effect when the user logs in--though that solution isn't much prettier than the one outlined on the Archbang forum. If I did do that, I assume the lines would take the form xmodmap -e “keycode 12 = 3 numbersign 3 numerosign” &. If I try that and it's any more successful, I'll report back about it.

Meantime, I'm still looking for more informed helpers with this issue. Got any insights?

Fifth installment: "fun" with xmodmap

In case it's not evident I'll point out at the outset that this entry is facetiously entitled (as the scare quotes might indicate). What I did with xmodmap wasn't exactly fun. Maybe mildly rewarding, finally. But also a bit perturbing.

Why perturbing? Because, after finally discovering a means of accomplishing the goal at which I was aiming, I bumped into what appears to be a long-standing bug that thwarts the resolution about half of the time. I'm still looking for a better resolution but, at least for the time being, I have a mostly workable solution.

Anyway, back to describing the task at hand, which I record here in case it may benefit anyone else and so that the lesson might stick with me longer.

So, I needed to help someone get all needed keys working on a keyboard that would, for the most part, be used for creating non-latin content. For what it's worth, the user will be switching back and forth between English and the non-latin language occasionally.

The target machine has XFCE4 installed as graphical environment. And it happens to be a "new-world Mac" that runs Linux--Debian, to be precise--a factoid which may prove fodder for a future entry. And yes, it's the powerpc architecture, and yes, it has a Mac keyboard (but not a Mac mouse).

XFCE4 has facilities for keyboard switching, which I hoped would make my job a little easier. Its graphical utilities did allow for selection of the needed encoding and layout, so I got a good part of the way using that. The user brought along stickers for the keyboard's keys which could be placed on the appropriate latin-lettered keys so that, being a non-touch-typist, it would be clear which letters should be typed in.

Almost all the letter keys all worked fine under the initial configuration. The problem was that the numerical keys across the top of the keyboard, when shifted, were supposed to be printing certain signs, and the stickers and the actual signs were not matching up. Something like 8 keys were affected. That needed to be fixed.

On looking into what could be done about this state of affairs, xmodmap began to look like it could provide an appropriate solution. So, I dove in. I did get it to work for the most part, though it seems XFCE4 cannot play nicely with xmodmap, so I had to introduce a kludge to be used on certain occasions (see here and here for some dated descriptions of the issues).

It was a bit mind-numbing trying to figure out exactly how to deploy xmodmap, given that I'd never used it before and that I was trying to make it handle two different encodings. The command xmodmap -pke >.Xmodmap-pke proved very helpful, since it created a text file of the current keyboard mapping.

Looking over that output I was able to identify the target keys. Some further reading and experimentation enabled me to find the right syntax to be used for modifying the current mapping.

It seems the canonical way of causing xmodmap to modify the current keyboard mapping is to create a file named .Xmodmap in the user's home directory. The window manager or graphical environment is supposed to read that file on starting and modify the mapping that will be used for those keys within the graphical environment.

For information on how the entries in that file should look, you should, of course, consult the man page--which, in this case, actually contains examples (many man pages, for reasons I as non-programmer can neither fathom nor appreciate, do not).

I did, of course, find helpful information in my web searches as well. The initial red herrings that needed to be sifted out were related to the fact that virtually all pages I found dealing with the matter of re-mapping macintosh keyboards were focused on re-mapping system keys like control, alt, or the "Apple key." I did not find any information dealing with the specific issue I was researching, namely, on re-mapping number keys.

But, as I observed, using xmodmap -pke gave me the data that, once I'd figured out the relevant portions, enabled me to do the required mapping. I needed, for example, the 3 key, when shifted in the non-latin mapping, to produce not the hash or number sign (i.e., #), but something that looks like the following: №. That, as I discovered, is called the numerosign.

Likewise the 7 key, when shifted in the non-latin mapping, needed to produce the question mark. The question-mark/slash key, on the other hand, needed to produce a period and a comma in the non-latin mapping, the latter when shifted.

The output of xmodmap -pke indicated that the 3 key corresponded to what to xmodmap is keycode 12 (xmodmap uses the terms keycodes and keysims to designate the various keys found on the keyboard but I do not claim to understand what is the technical difference between those terms/concepts). The Xmodmap-pke file I had created showed the following line for keycode 12:

12 0x0033 (3) 0x0023 (numbersign) 0x0033 (3) 0x0023 (numbersign)

From looking at other entries in Xmodmap-pke I was able to discern that the second half of the various lines which make up that list referred to the non-latin mappings for this keyboard, while the first half referred to the default, latin, mapping. So 0x0033 (3) 0x0023 (numbersign) is the output the key generates when the latin mapping is in effect, 3 being output when the keyboard is unshifted, and the hash sign being output when it's shifted.

The second half of the line is where my problem lay, then. In its present configuration, the keyboard was mapped to output the same 3 and hash character in the non-latin mapping as it was configured to output in the latin one. So I set about changing that.

I found some helpful samples for testing this at http://mac.linux.be/content/tuning-ubuntu-macintosh-key-mapping-and-3-mouse-button-emulation-0 . Specifically, I discovered that keyboard mapping can be changed sort of on-the-fly from the command line by issuing a command like xmodmap -e “keycode 12 = 3 numbersign 3 numerosign”. Once I did that, the 3 key, when shifted under the non-latin mapping, did, in fact, cause the desired № to appear.

That provides a good way to test the re-mapping to make sure it's working as desired. But if you want the change to be permanent, as I did, you have to construct an .Xmodmap file in your home directory that has all the relevant re-mapping lines in it and which should be read and processed each time the graphical environment starts. Entries in the .Xmodmap,file, by the way, look as follows: keycode 12 = 3 numbersign 3 numerosign. My .Xmodmap file consists in 8 such lines.

Once I had all relevant entries in that file I expected that the mapping modifications would be set and working once the graphical environment had fully loaded. In practice, however, I found that only some of the time was that the case: in many instances I had to open a terminal and type xmodmap .Xmodmap for the re-mapping to go into effect. Further internet searching revealed to me that there is some sort of incompatibility between xmodmap and XFCE4 that causes them to not work together very well. So I created a graphical shortcut that would run xmodmap .Xmodmap when the user clicked on it so as to get the desired mapping to take effect. Not a very pretty solution, but one that works for the time being.

There may be other solutions to this re-mapping issue. The xmodmap solution is one of the first I came across and pursued, so I can't speak to any of the other possible solutions. If you know of some other, possibly more effective one, I'd appreciate hearing about it.

NOTES TO SELF: setxkbmap provides some interesting key re-mapping options as well, a list of which can be found in /usr/share/X11/xkb/rules/base.lst. Another possibility I ran across suggests editing the relevant mapping file in /usr/share/X11/xkb/symbols/, but those files make little sense to me and I'm hesitant to go fiddling with them.

Fourth installment: How to write a business letter using TeX/LaTeX

I continue my experiments with TeX/LaTeX. There's just something I like about working with mark-up. I suppose that for most people, the mark-up would hopelessly complicate things. But I find the opposite to be the case for me.

That dates back probably at least as far as the first word processor I used with regularity--WordPerfect (WP hereafter) 4.0. It was also a console application, one that I managed to learn my way around fairly easily. It ran on a 286 with a whopping 20 megabytes of hard drive space and probably not more than 2 megabytes of RAM.

One of the great things about that version of WP was a feature it had called "reveal codes." If some aspect of the document you were working on looked weird and you couldn't correct by selecting various canned options, you could use the "reveal codes" feature to examine the situation behind the scenes. Despite the fact that my computer knowledge was extremely paltry back then, I don't think I ever confronted a situation where, by revealing those codes, I wasn't able to straighten things out.

Skip forward about a year to when I had finally gotten rid of the old computer and was using a "modern" word processor--Word 6.0. Imagine my surprise and dismay when, having issues with documents I was creating with that version of Word, I discovered that the program had no "reveal codes" feature. I was stuck with whatever Word wanted to give me, and that was that (a more technical explanation of the matters at issue can be read here).

So, my interest in TeX/LaTeX perhaps traces, ultimately, to the frustrations I encountered long ago with using more "modern" word processing applications. Whatever the motivations, I continue to experiment--and am now even having modest successes--with TeX/LaTeX.

Recently, I had another opportunity to continue my experimentation with  TeX/LaTeX. I needed to write an official, business-type letter and decided to try writing it with TeX/LaTeX.

I must admit there were also other motivating factors--not least of which was that, since I write so few of these types of letters, I can never remember the proper format and need, each time I write one, to do some web research aimed at refreshing my memory about that prior to composing the letter. TeX/LaTeX could be of assistance here since, I assumed, the letter document class should already be set to output the proper format--which did, indeed, turn out to be the case.

The down side was that it took me an inordinately long time to compose this fairly short letter: but that was, in part, because I wasted a lot of time in a largely fruitless attempt to utilize what was supposed to be a new-and-improved letter class, something called scrlttr2, a component of what is called the "KOMA bundle." Had I just used the old letter document class and not spent so much time trying to get scrlttr2 to do what I wanted, it probably would have taken me about half as long. A long time to write just one letter but, given that it was my first try using this application and that I had some spare time to spend on it, not too bad.

I was successful in the end. Below I'll paste a template for the letter I wrote, one that contains a nice tweak that causes a scanned signature to be printed.

Without further ado, here's the template I used (some explanation below):

\documentclass[12pt]{letter}
\usepackage{fullpage}
\usepackage[english]{babel}
\usepackage{graphicx} %so that signature graphic can be included
%\signature{Name Surname}
\address{My Street \\ My City zip \\ Phone: 555-555-5555 \\ e-mail: me@mydomain.whrvr}
\begin{document}
\begin{letter}{Recipient \\ Recipient's street \\ Recipient's city zip\\ country}
\opening{To whom it may concern:}
This letter is written for the purpose of \ldots

If you have any questions, or would like to see some additional documentation, please do not hesitate to contact me.
\closing{Sincerely,\\
\fromsig{\includegraphics[width=55mm]{Mysig.jpeg}}\\
\fromname{My Name}
}
% \ps{P.S. Here goes your ps.}
% \encl{Enclosures.}
\end{letter}
\end{document}

The first thing I'd like to explain is the use of the graphicx package, which is what allows for the graphic of the signature file to be included. Simply scan your signature and save it as a jpg file (other image formats may work, though jpg is the only one I tested). It will need to be saved within the directory from which you're compiling your letter. Stipulate the size you want for the signature, and pdflatex will include it once you compile.

The other fields in the template should be self-explanatory. I've commented out some of them (the percentage sign is the comment character in TeX/LaTeX), either because they did not apply to the letter I was writing--for example, the PS and enclosure options at the end--or perhaps because it was something that I tried but that didn't have the desired result (the "signature" option in the preamble).

The template should work as is. Just save it as a .tex file, enter your values in place of the generic information, scan your signature and save it to the same directory, then run pdflatex on the .tex file. You should end up with a nicely-formatted business letter.

EDIT: Thanks to those commenters who pointed out the the missing \end{document} tag at the letter's end--not sure how I left that out but it's now been added.

Third installment: In praise of newsbeuter

As I've mentioned previously, I'm a T.E.T. (trailing edge technology) type of guy. Rather than chasing after the latest, greatest, electronics must-haves, I'm content to sit on the sidelines watching developments, trying to scheme up ways of getting dated tech to approximate what the new gadgets can do, or sometimes even pouncing on something a bit dated when prices to drop by 50% or more once the novelty's worn off.

That sort of curmudgeonly attitude, augmented by being something of a technological late-bloomer (never touched a computer prior to age 30), it may come as scant surprise that, until fairly recently, I'd done little to nothing involving feed reading. I do recall, when first hearing about RSS, that it seemed like something well-suited to my needs and disposition. But initial attempts at utilizing it did not get very far.

Feed reading seemed attractive to me initially because, given my penchant for T.E.T. and all things low-resource, my preference for things like news reading, which I do a fair amount of, has been, for some years now, largely to use non-graphical applications. What I mean by that is that, when I want to know, via the internet, what's happening in the world at large, I want mostly words, not pictures. And most especially what I don't want when I'm looking for news is advertisements.

Well, advert aversity and my preponderating graphical ambivalence happen to go together quite well with low-resource computing--whether as cause or symptom I don't know. These proclivities led me, for several years, to do most of my news reading under text-mode browsers--primarily links/elinks and friends. I was quite pleased with that arrangement since it obviated a good deal of the bothersome advertising and allowed me to pick and choose which of the more newsworthy graphics I wished to view.

I could, for example, set up links/elinks to use a low-resource program like feh to view the graphics of my choice. That, of course, was in the days when in-line images were the standard for web pages, rather than more modern irritations like java scripts or the even more egregious flash. Should, in those bygone days, any news story hold sufficient graphical interest, I could always copy the URL into a graphical browser and have a gander at the site in its full monty manifestation.

As the web has continued its inexorable march toward chimerical perfection, text-mode browsing--as alluded to above--has become less and less practicable. For some time, one of my primary sources for news was the collection of headlines at the yahoo main page--which used to display quite acceptably in elinks. But yahoo finally sold what remained of its soul to the devil of progress and vaulted itself yet further into the 21st century, updating its main page and thereby causing it to render quite terribly under elinks.

I was adrift for a time news-wise in the wake of the demise of the old yahoo main page. I eventually settled on google news as my news page--once I discovered the proper user-agent incantation; one that would give me the mobile version of the page, containing just a clickable headline along with a line or two from the story. The full version of the page, containing whole opening-paragraphs from the various news stories of the day, was way more news than I was looking for.

But I do like to consult more than one news source each day: for example, I typically look at some Linux news site (Lxer is my preference these days), some boxing headlines, and some MMA news as well. I was ending up with 3 or 4 tabs open in elinks, refreshing each site as I wanted to check for new stories. Which didn't work too badly, by the way.

Recently, though, I decided to have another go at learning about and seeing whether RSS might suit better my news-reading needs. What I ended up discovering was an RSS feed reader that would turn out to scratch, in the most satisfactory way probably humanly possible, the wildest insatiable itch I barely even knew I had (if that statement makes no sense, read on: it might later--though I'm making no guarantees :)). That RSS application was newsbeuter.

Will it come as any surprise if I intimate that newsbeuter is a console program? Probably not. Not only can this application be used from a pseudo-terminal, it can even be used straight from the console! Now that's right up my T.E.T. alley.

"Ok, so what is it?" you ask. Well, it's a terminal application for reading RSS feeds, an application that was developed by an Austrian (as in, fellow countryman of our own American Terminator/Governator) fellow, Andreas Krennmair. The interface, such as it is, quite reminiscent of vi/vim. As he tells us, the name

"Newsbeuter" is a pun on the German word "Wildbeuter", which means "hunter-gatherer". During the stone age, people hunted and gathered their food, and these days, they hunt and gather news and information. Credits for this idea goes to Clifford Wolf, who submitted it to a little competiton that was started when I got aware that the original name would violate French and European registered trademarks.

The screenshot below offers a glimpse at what newsbeuter looks like once it's running:

Each of the entries you see in the screenshot represents an RSS feed. You use the up and down arrows to navigate them, and press the enter key to open the feed, which shows you a list of items or headlines for that feed. Note also the numeral pairs in parentheses, which shows how many items are within that group and how many of those have already been read.

Once you've highlighted one of the items using the up/down arrows and pressed the enter key, you'll be presented by something like what's seen in the screenshot below:

Note the blue bar across the bottom of the window that gives a list of keys that can be pressed and offers a description of what each does.

Again, the up/down arrow keys are used to navigate the headlines, and pressing the enter key on the highlighted headline opens a view that shows the first few lines of the story and that looks as follows (and yes, it's running in a screen session):

Here's where the real magic comes in for us command-line commandos. The application is configured to open the system's default browser for viewing the full story which, on most systems, will be Firefox or some other graphical monstrosity. But newsbeuter offers configuration options that will cause a different, more sane browser to open when the "o" (open) key is pressed, allowing you to view the story in a text-mode browser such as my preferred news reading application, elinks.

To configure newsbeuter to use a more sane text-mode browser, just open (or create, if it's not already on your system) ~/.newsbeuter/config and add the entry html-renderer elinks, and you're all set to read your news stories using elinks. Of course lots of other options can be set within this configuration file--which you can read about at the application's documentation page at http://newsbeuter.org/doc/newsbeuter.html.

Actually, a step that needs to be taken before even the tried-and-true elinks will be of any use to you in reading the news stories linked to, is to configure your feeds. To do that, you open the file ~/.newsbeuter/urls and enter in the URL's of the feeds you wish to monitor. Once that's done, you have a minimal configuration for reading news from RSS feeds with the big boys.

I must confess to being as happy as a clam with this new news-reading scheme--actually, happier. No clam grovelling in the cold mud in the murky depths could ever experience the warmth and satisfaction I've gotten from finding this RSS feed manager, which offers a way to link it up with my favored text-mode browser. So I'm actually happier than a clam--way happier (take that, you smug little bivalve molluscs).

I wish I would have discovered this solution long ago; but part of the reason I didn't was because, simply stated, I'm still very much in learning mode with GNU/Linux system set-up and administration. Had I discovered newsbeuter, say 3 or 4 years ago, I may very well have been unable to puzzle out how to configure and use it.

There are, of course, other command-line RSS feed readers. Perhaps some even work about as well as newsbeuter. I simply haven't looked further yet since what I have looks like it answers to all my needs.

One alternate project that looks interesting is rss2email. I may later have a look at it, but at the moment it's hard for me to imagine what advantages it would have for my purposes over newsbeuter. This entry, from the now--sadly--defunct blog "Motho ke motho ka botho," reviews a few others.

Thus concludes my paean to newsbeuter, the mutt of RSS readers: actually, I used mutt for about a year and gave up on it, going back to Alpine--so I'm more inclined to call it the Alpine of RSS readers. But, whatever.

All hail Andreas Krennmair, command-line RSS manager programmer extraodinaire! He deserves a Nobel prize for command-line RSS feed reader programming! Buy the man a beer in lieu of the Nobel prize he's been stinted if ever you meet him!

A second addendum to the second installment

I really do have things other than outlining with nano to write about. Really I do. For example, there's the project of installing the Tinycore distribution on some older machines in our computer lab to write about--something I did about a month ago and about which I've already started an article; there's an article about the newsbeuter rss client; one about how to set a weather map as the desktop background; and so forth.  But I've gone on kind of a jag with this nano project lately, and it's complex and foreign enough to me that if I don't record it now, I'm liable to forget important details. So, you're forced to endure another installment on it. :)

What follows may be the last entry on this topic for awhile--we'll see whether any enhancements will be forthcoming soon. If not, I'll probably move on to some of those other important writing projects after this entry. Anyway, on to the topic at hand.

With help from some generous folks over at the linux questions forum, I now have a 3 scripts to share. All the scripts--one written in perl, one in python, and one in awk--add mark-up to the nano outlines I create, mark-up that causes them, once pdflatex has been run on them, to transform into documents that print nicely on paper. I'll paste below the code for each of the three scripts so that, not only will I be less likely to forget how all this works, but also for the possible benefit of others who want to do something similar to what I'm doing. I'll start with the perl script.

The following perl script will add TeX/LaTeX mark-up to the outline files I create with nano. The code can be seen in the following graphic (thanks, blogspot, for making it such a PITA to post code snippets here that I have upload graphic files in order to show them, and sincere thanks to formatmysourcecode for providing the real and effective solution evident below):

#!/usr/bin/perl
# run outl2tex.pl as follows: outl2tex.pl file.outl > file.tex

print <<END
\\documentclass[14pt]{extarticle}
\\usepackage{cjwoutl}
\\usepackage[top=1in,bottom=1in,left=1in,right=1in]{geometry}
\\usepackage{underlin}
\\setlength{\\headsep}{12pt}
\\pagestyle{myheadings}
\\markright{\\today{\\hfill ***Header*title*here***\\hfill}}
\\linespread{1.3} % gives 1.5 line spacing
\\pagestyle{underline}
\\begin{document}
\\begin{outline}[new] 
END
;

while (<>) {
  s/^(\t*)=(.*)/"$1\\outl{".((length $1) + 1)."}$2"/e;
  print;
}

print <<END
\\end{outline}
\\end{document} 
END
;

(the code pasting above was made easy and possible by the source code formatter available at http://formatmysourcecode.blogspot.com/)

As will be obvious, I've named the script outl2tex.pl. As should also be clear, the script is to be run by issuing the command outl2tex.pl name_of_outl_file.outl > name_of_tex_file.tex

The next script--actually scripts--I'll show are the python ones. Why two? Well, it seems python is a rapidly-developing scripting language--something I found out when trying to run the first script that was created for me: it returned an error on my machine. At the same time, it ran fine on the machine of the fellow who created it.

Well, as it turns out, I have version 3.2.2 installed on my machine, while he developed the script on a machine that had version 2.6 on it. So, the following script is the one he originally wrote and that ran on his computer, and one which should work for python version 2.x (he wrote it for 2.6, I ran it successfully with version 2.7, but whether it works for all 2.x versions I cannot say for certain).

#!/usr/bin/python
# for use with versions 2.x of python
# run outl2tex.py as follows: outl2tex.py file.outl > file.tex

import sys
import re

if( len( sys.argv ) != 2 ):
    print >> sys.stderr, "{0} requires one filename to process.".format( sys.argv[0].split('/')[-1] )
    sys.exit( 1)

try:
    rawOutline = open( sys.argv[1], 'r' )
except:
    print >> sys.stderr, "Unable to open {0} for reading".format( sys.argv[1] )
    sys.exit( 2 )

print ( '\\documentclass[14pt]{extarticle}\n'
        '\\usepackage{cjwoutl}\n'
        '\\usepackage[top=1in,bottom=1in,left=1in,right=1in]{geometry}\n'
        '\\usepackage{underlin}
        '\\setlength{\\headsep}{12pt}
        '\\pagestyle{myheadings}\n'
        '\\markright{\\today{\\hfill ***Header*title*here***\\hfill}}\n'
        '\\linespread{1.3} % gives 1.5 line spacing\n'
        '\\pagestyle{underline}
        '\\begin{document}\n'
        '\\begin{outline}[new]\n' )

for inputLine in rawOutline:
    reMatches = re.match( r"(\t*)=(.*)", inputLine )
    if( reMatches == None ):
        print inputLine.rstrip()
    else:
        tabCount = len( reMatches.group(1).split('\t') )
        print "{0}\\outl{{{1:d}}}{2}".format( reMatches.group(1), tabCount, reMatches.group(2) )

print ( '\\end{outline}\n'
        '\\end{document}\n' )

This script is run in essentially the same way as the perl one.

Next, the python 3.x script. As with the 2.x version listed above, I've confirmed that this one runs using version 3.2.2, though I am uncertain whether it can be expected to run under all 3.x versions of python.

#!/usr/bin/python
# for use with versions 3.x of python
# run outl2tex.py as follows: outl2tex.py file.outl > file.tex

import sys
import re

if( len( sys.argv ) != 2 ):
    print ( "{0} requires a filename to process.".format( sys.argv[0].split('/')[-1] ), file=sys.stderr )
    sys.exit( 1 )

try:
    rawOutline = open( sys.argv[1], 'r' )
except:
    print ( "Unable to open {0} for reading".format( sys.argv[1] ), file=sys.stderr )
    sys.exit( 2 )

print ( '\\documentclass[14pt]{extarticle}\n'
        '\\usepackage{cjwoutl}\n'
        '\\usepackage[top=1in,bottom=1in,left=1in,right=1in]{geometry}\n'
        '\\usepackage{underlin}
        '\\setlength{\\headsep}{12pt}
        '\\pagestyle{myheadings}\n'
        '\\markright{\\today{\\hfill ***Header*title*here***\\hfill}}\n'
        '\\linespread{1.3} % gives 1.5 line spacing\n'
        '\\pagestyle{underline}
        '\\begin{document}\n'
        '\\begin{outline}[new]\n' )

for inputLine in rawOutline:
    reMatches = re.match( r"(\t*)=(.*)", inputLine )
    if( reMatches == None ):
        print ( inputLine.rstrip() )
    else:
        tabCount = len( reMatches.group(1).split('\t') )
        print ( "{0}\\outl{{{1:d}}}{2}".format( reMatches.group(1), tabCount, reMatches.group(2) ) )

print ( '\\end{outline}\n'
        '\\end{document}\n' )

Finally, here's the awk script, and it's a long 'un (unfortunately this script is now broken owing to some updates I made to the template file):

#!/usr/bin/awk -f
#
# -v tab=8
#       set tab stops at every eight columns (the default).
#
# -v template=template.tex
#       set the path to the LaTeX template file.
#
# run outl2tex.awk as follows: outl2tex.awk file.outl > file.tex

# Convert tabs to spaces.
function detab(detab_line) {

    if (length(tabsp) != tab) {
    }

    while ((detab_pos = index(detab_line, "\t")) > 0)
        detab_line = substr(detab_line, 1, detab_pos - 1) substr(tabsp, detab_pos % tab) substr(detab_line, detab_pos + 1)

    return detab_line
}

BEGIN {
    # Set tab width to default, unless set on the command line.
    if (tab < 1)
        tab = 8

    # Set template name to default, unless set on the command line.
    if (length(template) < 1)
        template = "template.tex"

    # Record separator is a newline, including trailing whitespace.
    RS = "[\t\v\f ]*(\r\n|\n\r|\r|\n)"

    # Field separator is consecutive whitespace.
    FS = "[\t\v\f ]+"

    # Configuration -- parsed from magic comments.
    split("", config)
    config["tab"] = tab
    config["template"] = template

    # We are not working on anything yet.
    template = ""
    header = ""
    footer = ""
    split("", outline)
    outline[0] = 1
    maxspaces  = 0
    CURR = ""
}

CURR != FILENAME {

    # Empty line?
    if ($0 ~ /^[\t ]*$/)
        next        

    # Configuration comment?
    if ($0 ~ /^%[\t ]*[A-Za-z][0-9A-Za-z]*[\t ]*:/) {
        name = $0
        sub(/^%[\t ]*/, "", name)
        sub(/[\t ]*:.*$/, "", name)
        value = $0
        sub(/^[^:]*:[\t ]*/, "", value)

        # Make the name case-insensitive.
        temp = name
        name = ""
        for (i = 1; i <= length(temp); i++) {
            c = substr(temp, i, 1)
            uc = toupper(c)
            lc = tolower(c)
            if (uc != lc)
                name = name "[" uc lc "]"
            else
                name = name c
        }

        config[name] = value
        next
    }

    # Comment line (skipped)?
    if ($0 ~ /^[\t ]*%/)
        next

    # This is the first line of actual content.
    CURR = FILENAME

    # Set up tabs as currectly specified.
    tab = int(config["tab"])
    tabsp = "                "
    while (length(tabsp) < tab)
        tabsp = tabsp tabsp
    tabsp = substr(tabsp, 1, tab)

    # Have we used a template yet?
    if (length(template) < 1) {
        # No, read it.
        template = config["template"]
        if (length(template) < 1) template = "-"
        OLDRS = RS
        RS = "(\r\n|\n\r|\r|\n)"

        while ((getline line < template) > 0) {
            # Content marker line?
            if (line ~ /^[\t\v\f ]*[Cc][Oo][Nn][Tt][Ee][Nn][Tt][\t\v\f ]*$/)
                break

            # Outline level definition?
            if (line ~ /^%[\t ]*\\outl{/) {
                level = line
                sub(/^[^{]*{/, "", level)
                sub(/}.*$/, "", level)
                level = int(level)

                line = detab(line)
                sub(/\\.*$/, "", line)
                sub(/%/, "", line)
                spaces = length(line)
                outline[spaces] = level
                if (spaces > maxspaces)
                    maxspaces = spaces
                continue
            }

            # Default value definition?
            if (line ~ /^%[\t ]*[A-Z][0-9A-Za-z]*:/) {
                name = line
                sub(/^%[\t ]*/, "", name)
                sub(/[\t ]*:.*$/, "", name)
                value = line
                sub(/^[^:]*:[\t ]*/, "", value)

                # Make the name case-insensitive.
                temp = name
                name = ""
                for (i = 1; i <= length(temp); i++) {
                    c = substr(temp, i, 1)
                    uc = toupper(c)
                    lc = tolower(c)
                    if (uc != lc)
                        name = name "[" uc lc "]"
                    else
                        name = name c
                }

                # If not in config already, set.
                if (!(name in config))
                    config[name] = value
                continue
            }

            # Comment line?
            if (line ~ /^[\t ]*%/)
                continue

            # Ordinary header line. Remove comment.
            sub(/[\t ]%.*$/, "", line)
            header = header line "\n"
        }

        # The rest belongs to footer.
        while ((getline line < template) > 0)
            footer = footer line "\n"

        close(template)
        RS = OLDRS

        # Fill in the outline levels.
        level = outline[0]
        for (spaces = 1; spaces < maxspaces; spaces++)
            if (spaces in outline)
                level = outline[spaces]
            else
                outline[spaces] = level

        # Replace all known ~Name~ in the template.
        for (name in config) {
            gsub("~" name "~", config[name], header)
            gsub("~" name "~", config[name], footer)
        }

        # Replace all other ~Name~ entries in the template with empty strings.
        gsub(/~[A-Z][0-9A-Za-z]*~/, "", header)
        gsub(/~[A-Z][0-9A-Za-z]*~/, "", footer)

        # Emit the template.
        printf("%s", header)
    }
}

/^[\t ]*=/ {
    line = $0
    prefix = index(line, "=") - 1

    # Indentation size in spaces.
    spaces = length(detab(substr(line, 1, prefix)))

    # Find out the outline level for this indentation.
    if (spaces > maxspaces)
        level = outline[maxspaces]
    else
        level = outline[spaces]

    # Add outline level definition.
    line = substr(line, 1, prefix) "\\outl{" level "}" substr(line, prefix + 2)

    printf("%s\n", line)
    next
}

{   printf("%s\n", $0)
}

END {
    printf("%s", footer)
}

As you'll note, this script is quite a bit more complex than the others. It expects, for example, that the TeX template is located in the directory in which the script is run, since it reads values from there (neither the perl nor the python scripts expect such a template file). If it doesn't find the template, it won't process the .outl file.

Furthermore, certain variables can be entered on the command line with this script--unlike the others: the number of spaces that make up a tab space can be fed to it, or an alternate name for the template file--even the title can be set as a command line option using -v title="Title". The title can likewise be entered as a commented line at the top of the outline file after the fashion % Title: A Nice Outline (the percentage sign is how commented lines are to be formed using TeX/LaTeX mark-up).

That about wraps up this second addendum to outlining with nano. The only other addition I can make for the time being is to point out that I've slightly modified the TeX/LaTeX preamble for my outlines, adding in the header, instead of the author's name, mark-up that will print there today's date (\today)--information more apropos for my purposes. See below the slightly modified template:

\documentclass[14pt]{extarticle}
\usepackage{cjwoutl}
\usepackage[top=1in,bottom=1in,left=1in,right=1in]{geometry}
\usepackage{underlin}
\setlength{\headsep}{12pt}
\pagestyle{myheadings}
\markright{\today{\hfill ***Header*title*here***\hfill}}
\linespread{1.3} % gives 1.5 line spacing
\pagestyle{underline} %this along with underlin package give hrule in header
\begin{document}
\begin{outline}[new]
%\outl{1}
%       \outl{2}
%               \outl{3}
%                       \outl{4}
%                               \outl{5}
%                                       \outl{6}
%                                               \outl{7}
%                                                       \outl{8}
%                                                               \outl{9}
%                                                                       \outl{10}
\end{outline}
\end{document} 

I've also altered slightly my .nanorc file so that it includes my preferred "pseudo-bullet"--the equals sign. What that means is that any line that does not begin with the equals sign--regardless of whether it's indented or not--does not get color highlighting applied. Lines that do begin with the equals sign--indented or not--do get color highlighting applied. The relevant section of the updated .nanorc is:

syntax "outl" "\.outl$"
color brightwhite "(^)=.*$"
color brightred "(^[[:blank:]])=.*$"
color brightgreen "(^[[:blank:]]{2})=.*$"
color magenta "(^[[:blank:]]{3})=.*$"
color brightblue "(^[[:blank:]]{4})=.*$"
color brightyellow "(^[[:blank:]]{5})=.*$"
color cyan "(^[[:blank:]]{6})=.*$"
color brightred "(^[[:blank:]]{7})=.*$"
color brightgreen "(^[[:blank:]]{8})=.*$"
color magenta "(^[[:blank:]]{9})=.*$"
color brightblue "(^[[:blank:]]{10})=.*$"

That's pretty much it. The only other modification that might be done involves quotation marks: since TeX/LaTeX expects, not " but `` (double back-ticks) as the opening quotation marks, my files produce the wrong sort of open quotation mark (back-facing instead of front-facing). If one of the above scripts could be modified to detect the opening quotation marks and replace them with the needed double back-ticks, it would make processing of the outline files more complete (ugh, I've just realized I have a similar problem with single quotes or what are sometimes called apostrophes). Otherwise, a search-and-replace will be in order prior to running pdflatex on the converted outlines.

Ya know, come to think of it, working with TeX/LaTeX is a real PITA. The final output is stunning, and no word processor I've ever dealt with comes anywhere close to matching it aesthetically. But it's still a PITA. And that's gonna to have to do it for now.