Category: Hacks

THIS. CHANGES. EVERYTHING. – Useful Bash/*nix Tricks I Never Stumbled Across in the Last 15 Years

A thread at /r/linux sought out to reveal all the magic ways of increasing productivity under Linux (or other *nix based OS-es), and as most people I thought that there wouldn’t be much news here.

But I was wrong. So very, very wrong.

  1. disown – a way to disown a process, making it continue running in the background if you have to log out or close a long running session over ssh because you’re going somewhere, but want to keep the currently running process still running. If you’ve ever thought “why the fsck didn’t I run this under screen?”, then this trick is for you. This is a new future, and I’m proud to be a part of it.
  2. CTRL+r in bash – allows you to search your bash history buffer. I’ve known about this, I’ve just never picked up the habit of actually using it. Will do that now.
  3. ssh-copy-id – Appends your public key to the authorized_keys file at the destination computer.
  4. man ascii – the manual page entry for ascii contains an ascii table, right there in your terminal.
  5. xargs ‐‐max-procs and parallel – allows you to duplicate the functionality of xargs, but in parallel. Starts up all the processes at the same time, instead of starting them one by one.

Head over to the thread for other goodies such as a sudo alias for writing files when you’ve opened them without the correct permissions directly in vim.

Missing Statistics in OpenX Again – This Time in 2.8.7

After upgrading to OpenX 2.8.7 from 2.4.1 our statistics suddenly seemed to have vanished. Debugging an issue like this isn’t just straight forward, but after digging through google searches, wiki pages at OpenX and, well, reading the source (brrrrrrrrr), I think I’ve nailed it.

After upgrading to 2.8.7 the DeliveryLog plugin didn’t get installed – which meant that no delivery / clicks / impressions were logged. After discovering that this had been moved to a plugin I tried simply unzippping the plugin and copying the files to the plugins/-directory. This seemed to make OpenX recognize the plugin if I went to “groups” in the plugin menu, but not under the “plugin” menu. Another problem was the fact that it didn’t actually log anything, which could be considered a problem.

All the Google searches had shown that OpenX had changed the logging format to a new table structure (named buckets), but they don’t provide of restoring / creating the bucket tables if they don’t exist, and they don’t give any error about the bucket tables missing if the plugin doesn’t load. I couldn’t find anything at all about how the tables should look and which tables should be installed, but I finally tried to simply install the plugin through the web interface (Log in as Administrator -> Select Plugins in the top menu) by uploading the zip file directly, and then FINALLY the post install script ran. That created the tables (I’ll dump the definitions later if someone needs them), and after reloading the ads the bucket tables started getting values.

Now we’ll just have to hope that they actually gets aggregated into something useful as well..

PS: I’m less than impressed by the OpenX upgrade procedure, it always seem to fsck up some detail that leaves your installation in limbo, without being able to detect that something has gone wrong and provide a way to resolve the issue. I understand that they need to – and want to – focus on their pay product, so well, I’ll keep having to fix things manually for a while, but Google’s Doubleclick for Small Businesses may see a new customer soon.

Evolution & Exchange: Unable to retrieve message

Some time after upgrading to Ubuntu 11.10 I ended up with the dreaded “Unable to retrieve message” in Evolution (which I use for Exchange connectivity). This has usually corrected itself by simply restarting Evolution, but this time nothing would help. I stumbled across a thread that provided a few ways to possibly solve the issue, but the .evolution directory didn’t contain any live installation in Ubuntu.

Turns out the directory is:

.local/share/evolution

As both my mailstore and address book lives on the Exchange server, I decided to just move the evolution directory to a new name and recreate the evolution directory from scratch. This takes a bit of time while Evolution indexes everything, but after a while everything were back to normal.

E:Error, pkgProblemResolver::Resolve generated breaks

While attempting to upgrade to Ubuntu 11.10 (Oneiric) from 11.04, do-release-upgrade refused to do anything useful. The only message it felt like delivering was “E:Error, pkgProblemResolver::Resolve generated breaks”. Googling didn’t turn up much, but a forum thread (which I seem to have lost now) suggested (among other attempts) to remove any references to external (3rd party) APT repositories. I thought do-release-upgrade did this by itself, but apparently not …

Commenting out the external repositories in /etc/apt/sources.list and in /etc/apt/sources.list.d/* solved the problem (I had spotify, dropbox and Google Chrome there), allowing do-release-upgrade to do its thing.

Getting Mac OS X-fonts to Work on Windows (Through Ubuntu)

This will be a fairly technical post, so if you’re not in the mood or don’t know what a bash script or Ubuntu is, this might not be the post for you.

Useful Ubuntu packages: fontforge (install with apt-get install fontforge)
Useful PERL Libraries: Mac::AppleSingleDouble (install with cpan install Mac::AppleSingleDouble)

First; Mac OS X stores the actual font data as a secondary data stream / file that’s attached to the actual font that the user can see. This requires the user sending the font to zip the contents before sending, so that both files can be included. The interesting files will be located in the __MACOSX folder and not in the actual folder containing the font (this folder will only contain the files with a 0 byte size). These files in __MACOSX will be the ones we’re going to town with.

Doing “file” on any file from this directory will probably yield “AppleDouble encoded Macintosh file”.

You can extract the actual contents of these files with the following awesome PERL snippet from a post at superuser:

perl -MMac::AppleSingleDouble -e 'for(@ARGV) {
    $a = new Mac::AppleSingleDouble($_);
    if(open $f, ">", $_.".rsrc") {
        binmode $f;
        print $f $a->get_entry(2);
        close $f;
    }
}' FILENAME_HERE

This will create a FILENAME_HERE.rsrc file in the current directory. This is the actual file that were stored as side channel data, without the other metadata associated with it.

In my case the resulting file was indicated to be a “Mac OSX datafork font, PostScript” by “file”, and after this I tried two things – one worked, one failed.

The first that failed:
In the archive there was also two other small datafiles. I tried extracting these and renaming the files to .pfb and .pfm (to use them as PostScript Type 1 fonts on Windows). This did not work. I’m not really sure where this failed at the moment, as I didn’t really feel like digging myself further down into the Windows Font subsystem. I also tried running the files through fondu, but didn’t get anywhere (it seems I really should run this under Mac OS X to be able to generate the proper .pfb and .afm files).

The second that worked (sort of):
fontforge is able to import a PostScript/dfont file without the associated metadata file. This might break kerning and a few other issues (fontforge will guess, so it won’t be half-bad), but will give you a resulting TTF file that actually works and can be used for most of the things that you’d use the font for. It was good enough for our use. Simply start fontforge, open the file and select file -> Generate Font afterwards.

Hopefully I won’t have to do this any time soon again. Fonts. Prrffftttt.

rsyslogd stuck at eating 100% (or more) CPU after upgrading to Ubuntu Natty Narwhal

This might also happen after upgrading to maverick, so don’t ignore the explanation even if you’re a version or two behind (.. or reading this at a much later time and we’ve all switched to implants).

Apparently the reason for rsyslogd getting stuck is a mismatch between how the kernel provides access to rsyslogd and what rsyslogd expects. If rsyslogd fails to get access to elements in the proc file system (/proc/kmsg was suggested in a bug thread), it locks up and spews out error messages at a great rate.

From /var/log/syslog

Apr 29 08:04:08 ubuntu kernel: Cannot read proc file system: 1 - Operation not permitted.
Apr 29 08:05:08 ubuntu kernel: last message repeated 13208405 times
Apr 29 08:06:08 ubuntu kernel: last message repeated 13297682 times
Apr 29 08:07:08 ubuntu kernel: last message repeated 14241325 times
Apr 29 08:08:09 ubuntu kernel: last message repeated 14397034 times
Apr 29 08:08:43 ubuntu kernel: last message repeated 7302035 times

Yes, that’s about 62 million error messages in less than 5 minutes. This demands quite a bit of CPU.

The reason for this is that the kernel API changed somewhere between the current Ubuntu version (2.6.38 in Natty) (and possibly the one in Maverick) and the one I was running (2.6.31). When rsyslogd runs under the latter, everything goes haywire. The solution is to make sure your kernel is upgrade to the most recent version – and that you’re actually running it.

First, stop rsyslogd to make your system a bit more responsive again:

sudo service rsyslogd stop

Updating Ubuntu should already have installed the newest kernel versions, but you might have told Ubuntu to use the existing configuration file instead of overwriting it when you updated (I almost do that automagically, which left me a couple of kernel versions behind). You can re-run this process and get grub to use an updated kernel version:

sudo update-grub

This might ask you again about whether you want to overwrite the current configuration file, and will also allow you to inspect the differences between the currently installed file and the one that update-grub wants to install. See if there are any significant changes (pay attention to information such as which partitions to use for booting), and if looks OK – allow the file to be replaced.

update-grub will then update your boot sequence with the new configuration file, and after rebooting (press ESC if you need to see the grub menu to make any changes), your new kernel should be running smoothly and rsyslogd should hopefully behave properly again.

Ubuntu Natty and Native Linux Spotify – “There is a problem with your sound card”

After updating to the most recent version of Ubuntu, The Natty Narwhal, Spotify decided it didn’t want to play music any longer. The only message in the UI itself was the usual “There is a problem with your sound card”, which isn’t very helpful if you want to actually try to find a solution.

Starting Spotify from the command line gave a few new messages, ending with:

E [snd:298] playbackError(12)

According to a a thread on getsatisfaction, this seems to be caused by a mismatch in different versions of the offline file cache (My naive guess is that Spotify uses the kernel identificator or some other settings about the machine in deciding the key to use for encrypting the offline storage – when this suddenly doesn’t match any longer, it refuses to play any music).

Closing Spotify, deleting the ~/.cache/spotify/offline.bnk file (~ expands to your home directory), solves the issue and allows Spotify to fill your ears yet again.

Update: This also seem to fix any issue where Spotify fails to get the external IP through upnp (at least that’s one of the error messages): 

E [upnp:521] ip: error getting external ip 0
I [http:840] Result 404 Not Found

Building Varnish on RedHat Enterprise Linux 4.0 (RHEL4)

We’re switching to Varnish as our reverse proxy service (and leaving mod_proxy under Apache), and as our web frontends still run RHEL4, there was a few small things that crept up while attempting to build Varnish from source (the supplied RPM packages are only for RHEL5).

You’ll need to have the pcre packages installed:

pcre pcre-devel

When running configure you might still get a notice about pcre not being available:

checking for PCRE... no
configure: error: Package requirements (libpcre) were not met:

Package libpcre was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpcre.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libpcre' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables PCRE_CFLAGS
and PCRE_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

You can solve this by setting the PCRE_CFLAGS and PCRE_LIBS environment variables:

export PCRE_LIBS="-L/usr/lib -lpcre"
export PCRE_CFLAGS=-I/usr/include/pcre

Re-run configure and run make, and things should hopefully build properly.

Parse a DSN string in Python

A simple hack to get the different parts of a DSN string (which are used in PDO in PHP):

def parse_dsn(dsn):
    m = re.search("([a-zA-Z0-9]+):(.*)", dsn)
    values = {}
    
    if (m and m.group(1) and m.group(2)):
        values['driver'] = m.group(1)
        m_options = re.findall("([a-zA-Z0-9]+)=([a-zA-Z0-9]+)", m.group(2))
        
        for pair in m_options:
            values[pair[0]] = pair[1]

    return values

The returned dictionary contains one entry for each of the entries in the DSN.

Update: helge also submitted a simplified version of the above:

driver, rest = dsn.split(':', 1)
values = dict(re.findall('(\w+)=(\w+)', rest), driver=driver)

Introducing Mismi – Amazon Price Comparison for Norwegian Customers

My main project in December was Mismi – a service that compares the total price of items from Amazon.com and from Amazon.co.uk for Norwegians. The solution is built on top of the Zend_Service_Amazon class (with a few extensions of my own).

The reasoning behind making the service is that there are several factors that are in play when deciding whether to order a product from the US or from the UK: the exchange rate for GBP and USD, the shipping cost, the delivery situation for the item and whether the item is sold in the store at all.

The user enters a list of the URLs to the products they’re considering purchasing from an Amazon-store, press submit and get a list back of which items are in stock, where the item is the cheapest and what the total sum of an order placed at the store would be. In addition I added a alpha stage feature just before Christmas which will also tell you the “optimum” set of items for the orders – “order item 1,4,7,9 from .com, item 2,3,5,6,8 from .co.uk”. This took quite a bit of hacking – you also have to consider the initial price of shipping, shipping for each item and other fun things.

Feel free to play with it over at mismi.e-mats.org. It’s in Norwegian, but it should be easy to understand anyhow with the description above.