rsyslogd stuck at eating 100% (or more) CPU after upgrading to Ubuntu Natty Narwhal

This might also happen after upgrading to maverick, so don’t ignore the explanation even if you’re a version or two behind (.. or reading this at a much later time and we’ve all switched to implants).

Apparently the reason for rsyslogd getting stuck is a mismatch between how the kernel provides access to rsyslogd and what rsyslogd expects. If rsyslogd fails to get access to elements in the proc file system (/proc/kmsg was suggested in a bug thread), it locks up and spews out error messages at a great rate.

From /var/log/syslog

Apr 29 08:04:08 ubuntu kernel: Cannot read proc file system: 1 - Operation not permitted.
Apr 29 08:05:08 ubuntu kernel: last message repeated 13208405 times
Apr 29 08:06:08 ubuntu kernel: last message repeated 13297682 times
Apr 29 08:07:08 ubuntu kernel: last message repeated 14241325 times
Apr 29 08:08:09 ubuntu kernel: last message repeated 14397034 times
Apr 29 08:08:43 ubuntu kernel: last message repeated 7302035 times

Yes, that’s about 62 million error messages in less than 5 minutes. This demands quite a bit of CPU.

The reason for this is that the kernel API changed somewhere between the current Ubuntu version (2.6.38 in Natty) (and possibly the one in Maverick) and the one I was running (2.6.31). When rsyslogd runs under the latter, everything goes haywire. The solution is to make sure your kernel is upgrade to the most recent version – and that you’re actually running it.

First, stop rsyslogd to make your system a bit more responsive again:

sudo service rsyslogd stop

Updating Ubuntu should already have installed the newest kernel versions, but you might have told Ubuntu to use the existing configuration file instead of overwriting it when you updated (I almost do that automagically, which left me a couple of kernel versions behind). You can re-run this process and get grub to use an updated kernel version:

sudo update-grub

This might ask you again about whether you want to overwrite the current configuration file, and will also allow you to inspect the differences between the currently installed file and the one that update-grub wants to install. See if there are any significant changes (pay attention to information such as which partitions to use for booting), and if looks OK – allow the file to be replaced.

update-grub will then update your boot sequence with the new configuration file, and after rebooting (press ESC if you need to see the grub menu to make any changes), your new kernel should be running smoothly and rsyslogd should hopefully behave properly again.

Ubuntu Natty and Native Linux Spotify – “There is a problem with your sound card”

After updating to the most recent version of Ubuntu, The Natty Narwhal, Spotify decided it didn’t want to play music any longer. The only message in the UI itself was the usual “There is a problem with your sound card”, which isn’t very helpful if you want to actually try to find a solution.

Starting Spotify from the command line gave a few new messages, ending with:

E [snd:298] playbackError(12)

According to a a thread on getsatisfaction, this seems to be caused by a mismatch in different versions of the offline file cache (My naive guess is that Spotify uses the kernel identificator or some other settings about the machine in deciding the key to use for encrypting the offline storage – when this suddenly doesn’t match any longer, it refuses to play any music).

Closing Spotify, deleting the ~/.cache/spotify/offline.bnk file (~ expands to your home directory), solves the issue and allows Spotify to fill your ears yet again.

Update: This also seem to fix any issue where Spotify fails to get the external IP through upnp (at least that’s one of the error messages):

E [upnp:521] ip: error getting external ip 0
I [http:840] Result 404 Not Found