Escaping Characters in a Solr Query / Solr URL

We’re using our own Solr library at Derdubor at the moment, but we’ve only been using it for indexing content. The query part was never standardized in our common library as we usually used an alternative output format, but during the last days that has changed. We now have a parser for the default XML outputter and we’re also supporting facets and field queries (or constraints as they’re abstracted as in our library).

This means that we’re feeding content into the query that may contain foreign characters, in particular those who have special meaning in a Solr query. You can find the complete list of characters that need to be escaped in a SOLR or Lucene query in the Lucene manual.

To escape the characters we use this very simple and stupid PHP method:

    static public function escapeSolrValue($string)
    {
        $match = array('\\', '+', '-', '&', '|', '!', '(', ')', '{', '}', '[', ']', '^', '~', '*', '?', ':', '"', ';', ' ');
        $replace = array('\\\\', '\\+', '\\-', '\\&', '\\|', '\\!', '\\(', '\\)', '\\{', '\\}', '\\[', '\\]', '\\^', '\\~', '\\*', '\\?', '\\:', '\\"', '\\;', '\\ ');
        $string = str_replace($match, $replace, $string);

        return $string;
    }

We used a regular expression first, but the sheer amount of backslashes made it a regular .. hell … to read. So to make it easier for the persons maintaining this in the future, we went the easy to read / easy to maintain road for this one.

Table 100% Width and Margins

While hacking aboooooot today I found the need for making a table behave like a regular block element. I have a section floated to the right of a table, and the table should occupy the rest of the available spot. Making the table’s width 100% would make it adjust it size according to its parent instead of the available place (.. with margin-right set to the width of the other element + whitespace).

The best solution I’ve found so far is to create a wrapper div around the table, and then setting the table width to 100%. This makes the table adjust its size according to the parent – which now is a block element.




...

PHP: Fatal error: Can’t use method return value in write context

Just a quick post to help anyone struggling with this error message, as this issue gets raised from time to time on support forums.

The reason for the error is usually that you’re attempting to use empty or isset on a function instead of a variable. While it may be obvious that this doesn’t make sense for isset(), the same cannot be said for empty(). You simply meant to check if the value returned from the function was an empty value; why shouldn’t you be able to do just that?

The reason is that empty($foo) is more or less syntactic sugar for isset($foo) && $foo. When written this way you can see that the isset() part of the statement doesn’t make sense for functions. This leaves us with simply the $foo part. The solution is to actually just drop the empty() part:

Instead of:

if (empty($obj->method()))
{
}

Simply drop the empty construct:

if ($obj->method())
{
}

Fixing Stuttering Audio With ffmpeg and Quicktime

We’re using ffmpeg to encode videos for flash on several of the sites I’m involved with, and this usually works flawless. From time to time there’s however certain video files that makes something go wrong, usually small issues like stuttering at random places. Today I decided to go bug hunting and try to find out exactly what triggered this behaviour in one of our recent videos.

After quite a bit of debugging and re-encoding the offending video segment (which limits the rate of debug attempts, as you have to wait a couple of minutes ++ for each encode), I decided to try to simply use -acodec copy for the audio codec. The Quicktime file already used AAC as its codec, and the file we were encoding also used AAC. The stuttering disappeared! This indicates that the sound encoding of the process were to blame for the stuttering, so if you’re having a sound related problem, try to copy the input codec to the output source.

As libfaac and libfaad were the two only involved libraries of the encoding and decoding process, the first thing to try were to check if there were any new versions of these libraries available. And lo’ and behold, both libfaac and libfaad had been updated since the versions included in our ubuntu version (no real shocker there, as things in the audio and video codec world moves with a rather high velocity).

I removed the packages (sudo apt-get remove libfaac-dev libfaad-dev) from Ubuntu, downloaded the new libfaac and libfaad versions, compiled (./configure && make) and installed them (sudo make install), and then recompiled ffmpeg with the new libraries in place. ffmpeg then complained about libfaac.so.2 missing, but a quick run of ldconfig (sudo ldconfig) fixed that issue.

Re-encoding the file yet again – and wooho! The offending file now works as it should. This probably also solve the issue we’ve been seeing for several other files too. The new versions of libfaac and libfaad solved the issues we were having.

BTW: There’s a small fix needed to make libfaac compile with gcc4.

Making Spotify, Wine and PulseAudio Play Together

In what has now become a series on this marvellous blog, we’re diving into how to make PulseAudio play nice with Spotify running under Wine. After yesterday’s article about getting any sound through PulseAudio at all, the next issue that surfaced was that the output from Spotify running under Wine were just garbled noise. This worked with the previous ALSA or OSS setup, but with PulseAudio everything went haywire!

Luckily the massive wisdom of teh intarwebs came to the rescue again, this time through an almost three year old post by Paul Betts: “Make Wine and PulseAudio Get Along“.

The solution is to first use padsp winecfg to configure padsp for a specific process, then use padsp to redirect any access to /dev/dsp while the provided application runs.

First run:

padsp winecfg

Then add padsp to your spotify startup script (or if you do this manually, just .. remember to type it. OK?):

padsp wine "C:\program files\spotify\spotify.exe"

Restart Spotify and enjoy the massive collection of great music!

Missing Devices or Sound Card in Pulse Audio

After getting the missing sound control panel in Ubuntu back yesterday, the next problem that turned up was that no actual applications would play any sound – other than the annoying beep! beep! beep!

The reason is that after the pulseaudio configuration has been installed, pulseaudio it self may be missing all it device drivers and other packages.

The solution for this issue can be found at the page for PulseAudio at the Ubuntu Wiki.

Install the missing PulseAudio packages through APT:

sudo apt-get install libasound2-plugins "pulseaudio-*" paman padevchooser paprefs pavucontrol pavumeter

Restart the application that needs sound support and everything should work. If you’re still not getting any sound, check the PulseAudio device configuration (by default installed under Applications, Sound and Video, PulseAudio Device Chooser) and that your device shows up there (mine didn’t before installing the above packages).

When the Ubuntu Sound Control Panel Doesn’t Show Up

After upgrading from Ubuntu 9.0 to 9.10 the other day, I suddenly got a very annoying beep every time I saved a file in NetBeans. Horrible stuff! Disabling it in NetBeans seems impossible (and Firefox makes the same sound when it encounters an error), so removing it in the Ubuntu Sound Configuration Panel sounded (!) like the best option.

One problem: The Sound Control Panel (when selecting System -> Preferences -> Sound) didn’t open. It attempts to load, but then disappears without a trace. I couldn’t find a notice in any of the logfiles either, so my only hope of a remedy was the global debugging power of teh intarwebs.

This was actually harder than I imagined, as I had to wade through large amounts of documentation of how to open the control panel at all. I had found the menu item, but it didn’t work.

Luckily someone had documented several issues at a page named “Sound Solutions for Ubuntu 9.04“, and while I didn’t have the exact problems listed there, they mentioned the required packages for pulseaudio and the GUIs.

Installing the padevchooser package should pull in all the dependencies:

root@ubuntu:~# apt-get install padevchooser

(Use sudo apt-get install padevchooser if you’re not root)

At the same split second the installer finished, the sound configuration dialog opened and I were able to switch to the NO SOUND scheme in Ubuntu. No beeps!

dpkg –configure Error for mysql-server-5.1

After starting the year by upgrading my Ubuntu desktop at work to 5.1 (and spending a couple of hours after borking my new kernel), my mysql-server had become lost in the transition.

Reinstalling the mysql-server package gave a cryptic error message:

Setting up mysql-server-5.1 (5.1.37-1ubuntu5) ...
Installing new version of config file /etc/init.d/mysql ...
 * Stopping MySQL database server mysqld                                 [ OK ] 
Warning: found usr.sbin.mysqld in /etc/apparmor.d/force-complain, forcing complain mode
 * Starting MySQL database server mysqld                                 [fail] 
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing mysql-server-5.1 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.1; however:
  Package mysql-server-5.1 is not configured yet.
dpkg: error processing mysql-server (--configure):
 dependency problems - leaving unconfigured
Processing triggers for menu ...
No apport report written because the error message indicates its a followup error from a previous failure.
                          Errors were encountered while processing:
 mysql-server-5.1
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

Quite weird, yes. Trying to find anything in the mysql-logs didn’t yield anything either (both were empty), but luckily the syslog provided an answer this time. Among loads of other messages regarding mysql the answer hid from general view:

Jan  7 14:30:48 ubuntu mysqld_safe[20249]: 100107 14:30:48 [ERROR] /usr/sbin/mysqld: unknown option '--skip-bdb'
Jan  7 14:30:48 ubuntu mysqld_safe[20249]: 100107 14:30:48 [ERROR] Aborting

Taking a look at the mysql config file in /etc/mysql/my.cnf confirmed this:

#
# * BerkeleyDB
#
# Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
skip-bdb

Apparently not just the support ceased, but also the skip-bdb parameter. That seems a bit odd – I’d suggest at least keeping the parameter around for another release cycle, but anyhow:

#
# * BerkeleyDB
#
# Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
# skip-bdb

And suddenly everything went in the right general direction!

Missed Schedule for Posts in WordPress

As I started queuing the posts for the previous run of “Ready for 2010“-articles, I came across a problem with my WordPress installation. The scheduled articles didn’t show up when they were scheduled, and the only thing shown in the WordPress administration interface were a message about “Missed Schedule”. No shit, sherlock.

The reason behind the message is that the wp-cron.php file didn’t run as it should. WordPress usually tries to run this every now and then by inserting a reference to the file through the web site. Apparently this behavior was borked on my blog. I have a perfectly working cron implementation on my server, so instead of relying on WordPress to do some kind of magic to insert a reference to the file and kick off the processing with a web request, I added a reference to wp-cron.php in my usual crontab.

I have no idea how often wp-cron really should be run, but decided that a five minute resolution was enough for my use. The crontab entry is included here:

*/5 * * * * cd <directory of blog> && php wp-cron.php

This runs the cron script from the proper directory, and seems to work fine.

One Possible Reason for Missing Munin Graphs

We’re currently expanding our munin reporting cluster at Derdubor, but after installing munin-node on one of our servers we never got any graphs. The only section available on the munin server was “Other”, and that didn’t contain any information at all (which indicates that you’re not getting any response from the server).

The first step I make when trying to debug a munin connection is to telnet into the munin port, as this confirms that the two servers are able to talk to each other and that the munin daemon listens to the correct interface and port.

# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
#

The connection was established, but then munin closed the connection as soon as it was created. This usually means one thing: the host you’re connecting from isn’t added to the cidr_allow list or the allow list, or in the denied hosts list. This time it meant neither, the host was added and we didn’t have any denied hosts list.

The next step was to take a look at the munin-node.log in /var/log/munin (at least under under debian).

The last message was:


User "ejabberd" in configuration file "/etc/munin/plugin-conf.d/munin-node" nonexistant. Skipping plugin. at /usr/sbin/munin-node line 615, line 83.
Something wicked happened while reading "/etc/munin/plugins/munin-node". Check the previous log lines for spesifics. at /usr/sbin/munin-node line 261, line 83.

We don’t have ejabberd installed, but the ejabberd config reference was apparently added to the configuration file in /etc/munin/plugin-conf.d/munin-node. This made our version of munin-node barf, as the user it reference wasn’t available.

Next step was to remove the section from the file and restarting munin-node:

/etc/init.d/munin-node restart

After restarting munin, I did the telnet check again:

# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at example.com
.
fetch load
load.value 0.02
.
quit
Connection closed by foreign host.
#

Wait 10 – 15 minutes and you should start seeing graphs again – if this actually were your problem. Probably not (and then you should probably read Debuggning Munin Plugins and other documentation on the Wiki). But if it were, you’ll be happy happy joy joy now.