Eighth Installment: compress and encrypt/decrypt a directory

I recently visited a relative who is studying in the natural sciences and who, surprisingly, is even less capable in certain technical aspects of computing than I am. He was trying to create, on his Mac, a script that would run as a cron job, and asked me for some pointers. Though I know the basics about cron and was willing to pitch in, I wasn't so sure about the script: you see, calling my bash skills rudimentary would be high praise. Nonetheless I decided that, with some web searching, I might be able to assist with that, too. Sure enough, I was able to find just the sort of information that would help us create a script that would tar and compress, then encrypt, a target directory. Details--shamelessly lifted from various locales on the web--are included below.

Over the years that I've been using Linux I have, of course, read more than a few articles that describe methods of encrypting files or partitions. Most recently, for example, there appeared on Lxer an article that described a clever way of encrypting a local directory that then gets backed up to some cloud storage service like dropbox. I've bumped up against the issue of encryption when doing fresh installations as well, as it has been the case for some time now that an option is given on installation for many Linux distros of encrypting, for example, the /home directory.

Despite reading at least some of those articles with interest, I did not feel the need to implement such encryption on my own systems. So it was not until someone else asked my assistance in doing something like this that I actually tried it myself. As you will see, it was actually fairly simple to implement. But first, a few caveats.

I'll skip any details in the following description regarding the cron aspect of this project--not that I could provide a whole lot of enlightenment anyway--other than to say that it's a handy way to make programs or processes run on a set schedule on computers than run *nix. One way I've used it is to cause a weather map, which I've set up as the desktop background on one of my computers, to update every 10 minutes--look for a future entry in this blog on how I managed that.

I'll also not speak in any depth about another ingredient in this recipe--tar--other than to say that it is an abbreviation for for "tape archive." I myself do not understand its workings terribly well, though I've used it on several occasions. I will mention on a related note, however, that, in my research, I ran across articles that used another, similar, utility--dd (an abbreviation for "disk dump")--to create compressed and encrypted archives. But I did not follow up on the dd option and so cannot post any further information about how that was done.

Finally, I can't speak in any depth about the program I used for doing the encryption--openssl--or about another program with which I experimented and which also does encryption--gpg. But I promise, despite those rather glaring deficits, that I will describe something I managed to accomplish and which you, too, should be able to accomplish by following the steps outlined.

Perhaps in some future entry for this blog I'll be able to further explore tar, dd, and/or cron. But for now I'm going to focus my attention mainly on the option we ended up using, which involved mainly tar and openssl.

The relative in question, as I mentioned, works in the natural sciences. He has a directory of his ongoing work that he wants to back up regularly, but to which he does not want anyone else to have access. His choice for backing up beyond his own PC, is to use dropbox. So the task was, as mentioned, to compress and encrypt the target directory: moving it to the location on the local machine where dropbox would find it so as to back it up will also not be covered in this write-up, though that step did end up being part of his final resolution.

So, what's left? It was quite easy to find directions of the web for doing all this. I pretty much went with the first workable solution I found, which came from the linuxquestions forum (the relevant thread can be found here).

The incantation we used was as follows:

tar -cj target-dir | openssl enc -aes128 -salt -out target-dir.tar.bz2.enc -e -a -k password

What that line does may be evident to most, but I will offer a bit of review nonetheless. The target directory is first tar'red and compressed with bzip, (the c option stands for "create" and the j option specifies that the created file should be compressed with bzip2) then it is piped to openssl for encryption. The word "password" is, obviously, to be replaced by whatever password the user chooses.

One possible drawback to this method, as pointed out in the thread from which I lifted it, is that the encryption password gets entered, in plain text, right on the command line (which is slightly less of an issue with a cron script such as we were creating). Thus, anyone who can gain access to the machine can, by using the command line history, see what the encryption password was. Since someone gaining access to his computer and viewing the command line history was not a concern for the fellow I was helping, this is the solution we implemented. But that potential concern can be easily remedied by simply leaving off the -k password switch at the end, which has the effect of prompting the user for a password, which does not get echoed to the command line.

To decrypt the file, the following command--which prompts for the password--is used:

openssl enc -aes128 -in target-dir.tar.bz2.enc -out target-dir.tar.bz2 -d -a

The file can then be uncompressed and untar'red. This part of the process could likely be reduced from two steps (decryption, then uncompression/untar'ing) to one by using a pipe, but since it was presumed, for purposes of this project, that the file would act simply as insurance against loss--the need for ever actually recovering the content being very unlikely--I did not pursue streamlining that aspect.

I did manage to find and test a couple of other variants which I will offer here as well. The second variant was found here, and follows:

tar -cj target-dir | openssl enc -e -a -salt -bf -out target-dir.blowfish

It is much the same as the first variant, though it uses a different encryption method called blowfish. I am uncertain which of these two encryption schemes is considered better. To decrypt the compressed directory, the following command is used:

openssl enc -d -a -bf -in target-dir.blowfish -out target-dir-decrypt.tar.bz2

Finally, I discovered yet another variant, details about which can be found here. A sample of how to use this one is as follows:

tar -cjf target-dir.tar.bz2 target-dir/ | gpg -r user -e target-dir.tar.bz2

As will be noted, this variant uses gpg to encrypt the directory. Of course user must be replaced by the name of someone who has a valid gpg key on the system, usually the primary user of said machine or account.

An interesting feature I discovered about this method is that a time-sensitive gpg key can be created, i.e., one that expires after a certain interval. If I understand correctly how this works, once the key expires, the directory can no longer be decrypted.* This feature should, obviously, be used with care.

Decrypting the directory can be done in the following way:

gpg --output target-dir.tar.bz2 --decrypt target-dir.tar.bz2.gpg

The same two-step process of decrypting, then untar'ing/uncompressing, applies to these two methods as well.

This sums up what I have to offer in this entry. Now that winter is upon us northern-hemispherers, I may be able to post more frequent entries. There are a few things I've been wanting to document for some time now.

* Correction: an anonymous commenter writes of my claim that the key expiration makes decrypting the file no longer possible that "Unfortunately not. The expired key is no longer trusted but is still functional."

Seventh installment: imagemagick trick 1 of (?)

During a recent vacation, I needed to convert some digital images taken by a relative's camera from a raw format into something that would display better on my laptop. As usual, imagemagick came to the rescue. Details are included below.

imagemagick has been called something like "the swiss army knife of image manipulation utilities." I've utilized this application on many occasions over my years as a GNU/Linux user and been amazed to have as yet found nothing that it cannot do. Moreover, it's a CLI-friendly tool, which makes it doubly appealing for someone like me who favors using the least resource-intensive application for any given job.

I should mention that, not only was I converting these images to a different format with which my laptop could better deal, desirable, but resizing them was needed as well. You see, each photo taken by this fairly high-end camera came in at about 25 megabytes, and the dimensions of each image were probably in thousands of pixels. So I wanted to both resize and convert them; but, based on past experience with imagemagick, I was sure it would be up to the job.

I was surprised, after some initial searching on how to accomplish this task, at how simple this task would turn out to be. After mounting the Secure Digital card on my laptop, I copied over all the images--which had the .CR2 extension, by the way--to a temporary directory on my laptop.

The incantation I then used, after cd'ing to the temporary directory where the images resided, was mogrify -format jpg -geometry 1024x768 *.CR2. It should be fairly obvious what this does, but just in case it's not, let me explain. The mogrify command is part of imagemagick's image manipulating routines and can be invoked to do a number of things with an image, but in this case the -format switch, followed by the jpg extension, tells it to convert from one format to another.

Though most readers of this blog are likely to know this already, it will be worth mentioning that the asterisk preceding the file extension is the wildcard symbol. Thus, all files in that directory with the designated extension get converted. Incidentally, they are given the same prefix as the raw .CR2 files, but after conversion have the .jpg extension instead of the .CR2 extension.

I should mention here that, although raw formats are not especially new, I did run across some indications while researching how to do this, that older versions of imagemagick might not be able to operate on such formats. But I believe any version that's come out in the last 2 or 3 years should understand and be able to manipulate these raw formats.

The -geometry switch tells imagemagick to resize the image from something in the thousands of pixels down to a size that displays better, and takes up a whole lot less disk space, on this laptop. The switch does also have a somewhat undesirable effect on photos taken in vertical rather than horizontal orientation, since, as written to disk, these are taller than they are wide (whereas photos taken in horizontal orientation are wider than they are tall): the program thus makes the height of the photo 768 pixels, while the width, being kept proportional, turns out to be quite a bit less than that. But for my purposes--which was simply being able to view quickly these fresh photos on my laptop--these somewhat under-sized photos did not pose any particular problem.

The process of converting many photos this way can be lengthy, but, as you have seen, it is quite simple. I would simply start the process, then either do some other task in another desktop while awaiting completion, or just set the laptop aside while it was running the process. After about a half hour or so, I was ready to view photos.

Above is a sample of one of the converted images. Oh, and by the way, the "(?)" in the title of this entry is meant to indicate the multitude of other imagemagick tricks, as yet unknown to a relatively casual user such as myself, there are likely to be.

Addendum to the first installment: yet more on screencasting with ffmpeg

With a recent upgrade (apt-get dist-upgrade to 12.04, custom-built Ubuntu) to my office machine, I started having serious audio/video sync issues when producing screencasts using recordmydesktop. As you may (or may not) recall, I record lectures on my computer and use, as a sort of visual aid, an on-screen whiteboard, where I type key words or phrases about which I'm speaking: that's what I capture from my screen when I'm recording these screencasts. Well, with the updated recordmydesktop, the text on my on-screen whiteboard would begin to appear several seconds before the audio about the word or phrase was indicating its appearance.

Oddly, the opposite was happening with the updated ffmpeg when running the screencasting incantation with which I'd earlier experimented. The on-screen text I was typing into the whiteboard was lagging a bit behind the audio.

I tried introducing a number of alternative switches into the commands I was issuing when using both recordmydesktop and ffmpeg. But to no avail: I couldn't get rid of the sync problems with either.

During the course of my searches aimed at resolving these issues, I ran across a crude script on the Ubuntu forums that someone had cobbled together, a script which uses ffmpeg, but which separately records video and audio, joining the two streams together, as a final step, into a final output file. I think this joining of an audio and video file are called, in electronic multimedia circles, "muxing," by the way. I decided the script was worth a try.

And, what do you know, after figuring out how to use the script, my tests indicated that it caused audio and video to be in nearly perfect sync. Thus, the answer to my newly-appeared screencasting issues was resolved.

I hoped to solicit improvements to the script but have so far not managed to find much help. Probably the weirdest thing about this script, which likely demonstrates the inexperience of the script's creator, is the fact that, once ffmpeg is invoked and the recording begins, you're supposed to enter, into the same terminal where ffmpeg is running, a file name, then hit the "enter" key as the singal to stop the recording and begin the joining of the audio and video files.

All this while you're seeing in the terminal the standard ffmpeg prompt that tells you to hit control-c to stop the recording. Confusing, to say the least--and made even more confounding by the fact that you can't actually see the text you're entering when you go to type the file name.

Despite those shortcomings, since the results produced by this script exceed anything else I've been able to accomplish, I think I'm going to stick with the script for now. I have made a couple of tweaks, mainly so as to make it record what are called "lossless" files--files that are produced with minimal processing (for example no compression), and which are therefore quite large. I have to re-encode my output files before uploading them in any case, so it's best to start with better-quality files.

Without further ado, then, I present the tweaked version of the script I've found:
#!/bin/bash #vzybilly #these are temp files aud="aud.flac" vid="vid.mkv" #grab audio & pid ffmpeg -f alsa -ac 2 -i plughw:1,0 -acodec flac $aud & audPID=$! #grab screen & pid ffmpeg -f x11grab -s "830x660" -r "24" -i :0.0+227,130+nomouse -threads 0 -vcodec libx264 -preset ultrafast -crf 0 $vid & vidPID=$! #wait, till name given (that means stop) read -p "Stop by giving an Output video name?" out #alternate, more coherent way of output file naming--requires zenity #out="`zenity --entry --title="Video muxing script" --text="Please type a file name (sans extension) into the blank and click 'Ok' to stop the recording"`.mkv" #stop audio and video with pids kill -n 2 $audPID kill -n 2 $vidPID echo "Saving to $out" #combine to the target output file ffmpeg -i $aud -i $vid -acodec copy -vcodec copy "$out" #purge the temp files rm $aud rm $vid
I should mention that I discovered a slightly less incoherent way of soliciting file-name input. Though it is commented out in the version of the script you see above, I intend to use this until such time as the script can be further improved ( to use it you uncomment the line that begins out= and comment out the line that reads read -p: note that you must have zenity installed for this modification to work).

In a related item, as I've mentioned earlier, the disadvantage to using ffmpeg for screencasting is that there is no built-in provision for pausing. Well, apparently someone has proposed a kind of workaround for that--see this thread for further details. I've not tried that method and don't really understand how it works, so I cannot attest to its efficacy.

What I have tried is simply stopping, then restarting a new file when a pause is necessary. That's definitely more cumbersome than pausing, and, furthermore, it requires the additional step of somehow joining what could be thought of a separate "vignettes" into a single "episode."

The good news I can report on that front is that I've found another script that was created precisely to join such separate files. I've tried some tests with it and it has worked for me quite well. It's called mmcat and it can be found here.

I'd like to post more about the plughw switch seen in the above script and which I needed to introduce in order to record through a new USB sound device I've added to my computer. But I don't really understand well what differentiates it from the more standard hw switch. So I won't speak to that matter further in this entry. :)

That about sums things up so far as recent screencasting developments on my front is concerned. Do you have any suggestions for improving the screencasting script I found? If so, please pipe in. Any other suggestions for pausing ffmpeg screencast recording? Please let me/us know.

Addendum to the first installment: more on screencasting with ffmpeg

Long time no blog. I got busy with a lot of other things, not least of which was various bicycling trips and associated maintenance projects. Amazing how much time and energy those things can take.

With the academic year approaching, coming up with a resolution to my screencast file-size problems has taken on renewed urgency. To reiterate, a recent major hardware upgrade on my computer led to a twofold increase in the size of screencast video files I produce on this machine--which in turn led to upload problems at my course web site (file size limits). It looks like the resolution may well be provided by ffmpeg.

So I began once again grappling in earnest with the increased file-size issue. I've actually even found a resolution, should I decide to switch over from the more capable recordmydesktop to the nice but somewhat feature-lacking ffmpeg.

Before launching into a description of that resolution, however, I need to make quick note of a new entry into the field of GNU/Linux screencasting applications. The new kid on the block is called Kazam and it looks promising. I've not yet been able to get it running--and I'm not in a big rush to do so, since I favor command-line tools. But it's received some praises and looks like a promising application. That noted, I now return to a description of the resolution I've discovered for the issue of screencast file sizes made using ffmpeg's x11grab.

First and foremost, I discovered a fairly lengthy thread on the Ubuntu forums that deals with screencasting usingffmpeg. It's chock full of all kinds of interesting tips for screencasting with ffmpeg. For example, a switch is documented there that allows you to exclude the mouse cursor from your screencast: you simply add :0.0+nomouse to the command you use to start your screencast. I'd definitely need that in order to use ffmpeg as the utility to record my lectures.

Also described there is a way of doing what might be called "pseudo pausing," which really means you just stop your recording at the point where you need to pause, then start up again with a new recording. You then need to concatenate the files--not anywhere near as convenient as recordmydesktop's pausing capability, but something that'll do in a pinch.

It was from that thread that I derived what seems like a pretty good resolution to my file size issues. What I've discovered is that I can record my files in a lossless format (mkv is recommended in that thread)--which results in pretty gigantic file sizes (ca. 13 megabytes per minute on this system)--then transcode them into flash format while dramatically decreasing the file size: a one-minute test file I made actually came in at about 1.2 megabytes after re-encoding using this method. Since what I need is a file that comes in at about 2 megabytes per minute, this could work well for my circumstances.

The key command that's allowed me to reduce dramatically screencast file sizes to an acceptable level while at the same time converting to a format that works well at my hosting site uses . . . you guessed it, ffmpeg. Here it is:  

ffmpeg -i output.mkv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flv

I do still hope I can find a way of reliably shrinking the files recordmydesktop outputs, but my research on that has, to date, not been fruitful. There's a utility called oggResize that's supposed to allow you to easily resize .ogv files, but it puts the audio out of sync with the video. And it appears that tool is not under active development. It's likely resizing of .ogv files can also be done with ffmpeg, but I've not yet managed to arrive at the appropriate incantation for doing that. I'll keep digging though, and if I come up with a resolution, I'll post it in this blog.

As a final note on things I've learned in this round of screencast research, I should also mention a command-line tool called ffcast someone has developed. This looks like a bash script that first invokes a tool for selecting an area of the screen, then calls ffmpeg to record the screencast. I've also not tried that one but it's on my list to look into in the future.

POSTSCRIPT: I decided to run on an .ogv file created by recordmydesktop the same command I'd used to shrink the lossless .mkv file to see what the results would be. I'm pleased to report that the results are positive: the .ogv file, when re-encoded as an .flv using that method, shrunk to about one third the size (the 1 minute .ogv test file was 4.2 megabytes and it came in as a 1.2 megabyte .flv file after re-encoding). So I can continue using recordmydesktop to record my lecture screencasts--albeit with the penalty of having to do an additional round of encoding/re-encoding (recordmydesktop already takes quite some time to encode a screencast into .ogv format unless you use the --on-the-fly-encoding switch, which I'm now likely to start using). So, the command I used to re-encode the .ogv file to .flv is:

ffmpeg -i output.ogv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flv

MUCH LATER POSTSCRIPT: I've discovered that ogv files work much better at my hosting site than do flv files, so I began looking into ways of transcoding/shrinking to that format. The incantation I discovered, through trial and error, and which reduces screencast file sizes to about 1.5 megabytes per minute (roughly one seventh of the original, lossless, size), is as follows:

ffmpeg -i infile.mkv -r 13 -acodec libvorbis -ab 48k -ac 2 -vcodec libtheora -preset slow -wpredp 0 -crf 22 -threads 0 outfile.ogv

There are two crucial bits here. The -ab 48k option is one of them: that reduces the audio bit rate, which brings the file size down by quite a lot. The other is the -r 13 option, which reduces the frame rate to 13 frames per second and thus shrinks the file size yet further. A 48 minute long lossless screencast I created, for example, was reduced to a 73 megabyte ogv file using this incantation.

Sixth installment: establishing wifi connections from the command line

Reader's Digest condensed version

Want to get on a wifi network fairly quickly and painlessly from the command line? Try network-manager's command-line cousin, cnetworkmanager. A command like sudo cnetworkmanager -C essid-of-network --unprotected (for unprotected networks) will do it. For further details on how I discovered this and for further remarks on cnetworkmanager's capabilities, read below.

Unabridged version

Trailing edge technologist that I am, I'm also a fan of minimalist desktops. I've used mainly ion2/3, dwm, and evilwm for day-to-day computing--though I've tried out many others like JWM, icewm, fluxbox, blackbox, twm, openbox, etc.

After my netbook's keyboard went on the fritz subsequent to having water squirted on it by a loved one (I can say she's a loved one in hindsight), I decided to dig out an old laptop and press it back into service. It is, of course, a single-core machine with a Celeron 2.4 Ghz CPU and only 512 MB RAM. So, another ideal candidate for a minimalist desktop.

I wanted to get the machine up in a hurry so I chose to install a lighter-weight *buntu on it--namely, Lubuntu. That variant uses the lightweight LXDE desktop.

After I'd gotten it set up and used it for awhile, I decided to try paring back the system resource usage yet further. The main means toward that end was to install the evilwm window manager which, though I understand it is not quite as lightweight as dwm, nevertheless looked like a good choice for this machine.

It took some trickery to be able to load into the evilwm desktop once the machine was booted. Lubuntu, you see, uses the LXDM login manager, and I was unable to create a valid menu entry for an evilwm session. So I used the quick and dirty method: cobble together an .xinitrc for evilwm, kill LXDM, then issue startx from the command line.

I'm in the initial stages of possibly replacing LXDE with evilwm on this laptop, but I've gotten at least partway there. I want to recount in this entry how I managed to get wifi connections working under evilwm. But first, a relevant digression.

I used to use command-line routines on another old laptop some years ago when connecting to wifi networks. That involved using commands such as iwlist and iwconfig, then issuing a dhcp request on the wifi NIC. At a certain point, however, my trusted command line routines stopped working for reasons I could not understand.

Searching for alternatives, I discovered wicd. I actually doubted it would work since, as I understood it, wicd was simply a series of python scripts that invoked the same command-line binaries I'd been unsuccessfully attempting to use.

But wicd did work, and to my delight, I even discovered that it had a curses, terminal interface: I thus needed not be bothered with using a pesky gui utility to employ wicd. I subsequently came to rely on wicd for establishing and managing wifi connections.

Getting back to evilwm; those who are familiar with that window manager will be aware how minimalistic it is: there are no taskbars, no system trays, no pop-up menus. No, all evilwm does is display the desktop background you specify and allow you to open graphical applications (invoked from a terminal or by using something like xbindkeys and gmrun).

That sparseness presented an issue with wifi connections once I started switching from LXDE to evilwm: you see, Lubuntu has set up this system such that it uses network-manager for managing wifi connections, and network-manager, in turn, expects there to be a system tray in which it can reside. Given that architecture, I could see no way in which I could use network-manager for my wifi connections while running evilwm.

My first assumption was that my old standby, wicd, would allow me to manage wifi connections under evilwm. So I installed it. But it would not, no way, no how, allow me reliably to connect to wifi networks.

I am uncertain why this problem exists. I read that it was related to some conflict with network-manager, and that the solution was to uninstall network-manager. But doing that did not resolve my problems: wicd would, for the most part, give me a "bad password" error and would not connect to the network: perhaps once in every thirty tries I could get on-line, but the other twenty-nine times, I would get the bad password error.

Short story long, I had to abandon wicd and look into other ways of connecting my laptop to wifi networks using command line utilities. My attempts at finding an alternative to wicd finally resulted in the discovery that there is actually a command line version of network-manager called cnetworkmanager.

The happy ending to this story is that cnetworkmanager's man page is fairly simple and easy to comprehend for the non-specialist. I was, in very short order, able to bring up my wifi network connection using cnetworkmanager. Below are salient details.

In order to find the name of your wifi network interface and, at the same time, scan for essid's of detected networks, the command cnetworkmanager -a -d run in a terminal does the trick (-a is the option that lists available networks, -d is the switch that tells you the names of wifi interfaces on the machine being used). With that information it is fairly simple--though super-user privileges are required for this--to a network as follows.

For WPA-encrypted networks, the command sudo cnetworkmanager -C essid-name --wpa-pass=password will do the trick. For unencrypted networks, sudo cnetworkmanager -C essid --unprotected does the job.

It's a bit clunky, but provides a workable solution that gets me by for now.

An explanation of html tags used at blogspot

This is just a sort of filler entry, for my own reference. It will contain an explanation of html tags blogspot posts use, and is intended to help me utilize sane editors for creating my blog entries rather than the ungainly blogger web interface. See below.

Here's a first paragraph. It consists in two sentences and, after the pariod, an extra carriage return so as to space the paragraph apart from the following paragraph.

Here's a second paragraph. It, too, consists in two sentences followed by an extra carriage return--as do all subsequent paragraphs in this test.

Here's a block quote. As all paragraphs should, it consists in at least two sentences.

Now, here's a paragraph after the block quote. Once again, we see two sentences.

Now, for a fourth paragraph that contains a link. The link is here.

Moving on to the fifth paragraph, we'll enter some bold, the italic, text. Here's the bold. Here's the italic. And here's some underlined text.

For the sixth paragraph, we'll change the font size. Let's try what they call a "large" font. Now, back to "normal" font.

In the seventh paragraph, we'll change the font face. There's not much to chose from, so here's arial. Now, back to the "default" font.

Now, we'll try inserting a numbered list, then a bulleted list. Here's the numbered list, in a new paragraph (apparently, the extra carriage return is not needed at the beginning or end of lists):

1. item 1
2. item 2
3. item 3

Now, the bulleted list, in a new paragraph:

• item 1
• item 2
• item 3

I'd like to insert a table, but I don't see any way to do that. So, let's try some strikethrough. Very impressive.

And, after this eleventh paragraph, a pseudo-list of a few lines of text, each on a separate line:

one
two
three

Finally, just for kicks, let's insert in this twelfth paragraph what they call a "jump break" (not sure what that is, but . . .) 

Addendum to the first installment: alternate screencast applications

As you may recall, in the inaugural entry for this blog I described a crude mock-up I'd cobbled together, using Linux utilities, for producing screencast video lectures along the lines of those found on the khanacademy web site. I recently had occasion to explore some alternate screencast applications, since ogv files ceased playing back sanely on one of my systems. That issue was ultimately addressed by "upgrading" the video card in my aging Pentium 4 computer (with an old 128 MB nvidia card I got for $15). But not before I investigated some alternatives to recordmydesktop.

Of course there are various options for screencasting in Linux: I've read about Istanbul, Byzanz, vnc2flv, and some others, for example. I've so far gravitated toward recordmydesktop because it "just worked" and because it can be used from the command-line. But because of the video card issue mentioned above, I did recently decide to explore a couple of other alternatives which I'll write about in this entry, namely xvidcap and ffmpeg.

I should mention at the outset that, as those who read the first installment of this blog will be aware, I need something that not only records a video of a part of the desktop (actually, a particular application window in my case), but that allows for sound recording as well: after all, these are lecture videos I'm producing. So any screencast application that does not have built-in sound recording isn't going to work for my purposes.

Screencasting with ffmpeg

Most readers of this blog will have heard of the indispensable video-manipulating program ffmpeg. It's possible to do a truly amazing amount of manipulation and creation of videos using that program. And, as it turns out, it's even possible to do screencasting with it (in fact, ffmpeg may actually be used "behind the scenes" by recordmydesktop--though I'm not fully certain about that).

Some recent research on the web resulted for me in some successful experiments in screencasting with ffmpeg. The main ingredient in this screencasting capability is a switch called x11grab. Run from the command line, an ffmpeg screen/sound capture session would be invoked something like this:

ffmpeg -f alsa -ac 2 -ab 48k -i hw:0,0 -f x11grab -r 20 -s 800x600 -i :0.0+227,130 -acodec libmp3lame -vcodec libx264 -vpre lossless_ultrafast -threads 0 output.avi

That command will output an avi file called, appropriately enough, output.avi that is 800x600 resolution and that grabs a portion of the screen that is 227 pixels from the left edge and 130 pixels from the top edge (about where I open the application I wish to record on this machine). The sound portion is transcoded into mp3 by libmp3lame--something I discovered reduces the final size of the file considerably.

I believe ffmpeg can output just about any video format, including ogg theora, as recordmydesktop does. In order for that to be implemented, the -vcodec argument needs to be modified to, in the case of an ogg theora file, to -vcodec libtheora, I believe, while the extension of the output file's name would need, obviously, to be changed to .ogv.

As those who are familiar with ffmpeg will know, I'm barely scratching the surface of the tip of the iceberg here as far as its capabilities go. In fact, ffmpeg's man page has got to be one of the most voluminous and daunting of them, and it's only within the last couple of years that I've begun to be able to make any sense of it. But don't presume that I mined the information above mostly from the man page: rather, I found a working sample command on the web, then went back to the man page to try and better understand how it works and how I might tweak it for my purposes. I'm really still on a very rudimentary level when it comes to understanding and using ffmpeg.

Now, though this ffmpeg solution works quite well, it's turned out to not really be usable for me. This is because, as great as it is, the ffmpeg process cannot be paused while it's running: you either have to stop it and resume anew if you have some reason to pause, or else you'll need to edit the resulting video and cut out  the extraneous portions. In case it's not apparent, it's really useful during a lecture to be able to pause and resume.

So, as well as it worked in my experiments, I decided that, until I can come up with some way to pause the process, then resume, ffmpeg is not going to work for me as well as recordmydesktop does.

Screencasting with xvidcap

Which bring my to the next screencasting application, xvidcap. There's a lot to like about xvidcap. It's a graphical application, but with a very minimal interface--just the type of gui application that appeals to me. It draws a nice, visible perimeter around the area you're recording, and the perimeter can be easily manipulated with the mouse to enclose whatever quadrangular area you'd like. And it does allow for pausing and resuming of the screencast.

A screenshot of xvidcap (running on someone else's machine)

Furthermore, it's able to output a few video formats, among which are mpeg and flv. It looks like a great application and I really wish I could use it. But there's a major problem with sound: you see, xvidcap was developed during an era when oss (Open Sound System) was the standard Linux sound server, but oss is now deprecated in favor of alsa (Advanced Linux Sound Architecture).

In theory, it's possible to emulate oss with alsa. That's what you'll read on the internet, anyway. But the fact of the matter is that, as Linux moves further and further away from the old oss architecture, emulation of it has become harder and harder. I spent quite a few hours trying to implement it on this system, so happy was I with what I'd seen of the video-capturing end of xvidcap. I tried a number of purported solutions I found in my web searches. But none of them enabled me to record sound alongside the video I was capturing with xvidcap. So, I've reluctantly given up on it for now.

Those are the two alternate screencasting programs I've toyed with lately, and with mixed results. Both seem to have their advantages as compared to recordmydesktop, but in the end, it looks as though neither is going to be able to displace it. If you have any sort of pointers for addressing either of the issues I experienced in my attempts to use those applications, I'd be most delighted to hear about them. So please do offer your input.

Segmenting video files

To wrap up this posting, I need first of all to confide that I finally did "upgrade" my old work PC. I replaced the aging Pentium 4 with a not-quite-so elderly dual core machine. So video recording, and especially playback, goes quite a bit more smoothly on this "new" machine.

One odd thing has cropped up regarding video, though. The problem is that, despite the fact that I use the same exact recordmydesktop command to record my screencasts, the resulting files are now almost twice as large as they had been on the old machine: instead of being on the order to 2 megabytes per minute, they're now closer to 4 megabytes per minute.

My suspicion is that the video hardware could be to blame, since I had a pretty old 32 megabyte ATI (PCI) video card in the old machine, while the newer machine has an nVidia 128 megabyte card (PCI-express). I've tried switching video card modules from nv to vesa, but that doesn't seem to affect the resulting video size.

This becomes a problem because I have a file size limit on the site where I must upload my lectures, and they're now routinely going to exceed the size limit. So, until I can figure out how to reduce the video file sizes back closer to what they were, I've had to come up with a work-around--namely to split the video files into parts that, individually, do not exceed the size limit. And for this, ffmpeg once again comes to the rescue.

In order to split my lecture files, I'm using the following command:

 ffmpeg -ss 00:00:00 -i in.ogv -t 00:30:00 -vcodec copy -acodec copy out.ogv

What that does is to start at the beginning of the file (the -ss 00:00:00 part), go 30 minutes into the file, then copy that section to a new file called out.ogv. So, for an hour-long lecture, after having done that to split off the first 30 minutes into a new file, the same command would be run again, except that-ss 00:00:00 would be replaced by-ss 00:30:00. Then, the two parts could be appropriately named (with part1 or part2 in the name, as appropriate) and uploaded.

That's all for this entry. If anyone has tips or recommendations about what's been discussed above or about anything else related to screencasting on Linux, please pipe in.

Afterthought: here's a link that offers some technical details on the sound hardware in my new computer: http://people.atrpms.net/~pcavalcanti/alsa-1.0.15rc2_snd-hda-intel.html#final . I was looking at that as I was trying to work out how I might get xvidcap functioning on this system.