Tag: Hacks

Console / shell script runs twice when using pyramid.paster and bootstrapping

You’ve just created your new, exceptional shell script to maintain your pyramid application from the console, when you discover that everything runs twice. Usually not a very good idea, but .. why?

The issue is probably that you’re running pyramid with the scan option, which requires all modules in the path of your pyramid application to be imported. This will also import your console script, and if you haven’t placed everything into a function and added a check to see if the script has been invoked directly, you’re fscked!

The easy way out is to put your code into a function:

def main():
    from pyramid.paster import bootstrap
    env = bootstrap('../../development.ini')

[snip]

if __name__ == '__main__':  
    main()

The last if-test check if this is the main file that has been invoked, and if true, calls main and launches your script. This should hopefully solve the issue!

THIS. CHANGES. EVERYTHING. – Useful Bash/*nix Tricks I Never Stumbled Across in the Last 15 Years

A thread at /r/linux sought out to reveal all the magic ways of increasing productivity under Linux (or other *nix based OS-es), and as most people I thought that there wouldn’t be much news here.

But I was wrong. So very, very wrong.

  1. disown – a way to disown a process, making it continue running in the background if you have to log out or close a long running session over ssh because you’re going somewhere, but want to keep the currently running process still running. If you’ve ever thought “why the fsck didn’t I run this under screen?”, then this trick is for you. This is a new future, and I’m proud to be a part of it.
  2. CTRL+r in bash – allows you to search your bash history buffer. I’ve known about this, I’ve just never picked up the habit of actually using it. Will do that now.
  3. ssh-copy-id – Appends your public key to the authorized_keys file at the destination computer.
  4. man ascii – the manual page entry for ascii contains an ascii table, right there in your terminal.
  5. xargs ‐‐max-procs and parallel – allows you to duplicate the functionality of xargs, but in parallel. Starts up all the processes at the same time, instead of starting them one by one.

Head over to the thread for other goodies such as a sudo alias for writing files when you’ve opened them without the correct permissions directly in vim.

Missing Statistics in OpenX Again – This Time in 2.8.7

After upgrading to OpenX 2.8.7 from 2.4.1 our statistics suddenly seemed to have vanished. Debugging an issue like this isn’t just straight forward, but after digging through google searches, wiki pages at OpenX and, well, reading the source (brrrrrrrrr), I think I’ve nailed it.

After upgrading to 2.8.7 the DeliveryLog plugin didn’t get installed – which meant that no delivery / clicks / impressions were logged. After discovering that this had been moved to a plugin I tried simply unzippping the plugin and copying the files to the plugins/-directory. This seemed to make OpenX recognize the plugin if I went to “groups” in the plugin menu, but not under the “plugin” menu. Another problem was the fact that it didn’t actually log anything, which could be considered a problem.

All the Google searches had shown that OpenX had changed the logging format to a new table structure (named buckets), but they don’t provide of restoring / creating the bucket tables if they don’t exist, and they don’t give any error about the bucket tables missing if the plugin doesn’t load. I couldn’t find anything at all about how the tables should look and which tables should be installed, but I finally tried to simply install the plugin through the web interface (Log in as Administrator -> Select Plugins in the top menu) by uploading the zip file directly, and then FINALLY the post install script ran. That created the tables (I’ll dump the definitions later if someone needs them), and after reloading the ads the bucket tables started getting values.

Now we’ll just have to hope that they actually gets aggregated into something useful as well..

PS: I’m less than impressed by the OpenX upgrade procedure, it always seem to fsck up some detail that leaves your installation in limbo, without being able to detect that something has gone wrong and provide a way to resolve the issue. I understand that they need to – and want to – focus on their pay product, so well, I’ll keep having to fix things manually for a while, but Google’s Doubleclick for Small Businesses may see a new customer soon.

Parse a DSN string in Python

A simple hack to get the different parts of a DSN string (which are used in PDO in PHP):

def parse_dsn(dsn):
    m = re.search("([a-zA-Z0-9]+):(.*)", dsn)
    values = {}
    
    if (m and m.group(1) and m.group(2)):
        values['driver'] = m.group(1)
        m_options = re.findall("([a-zA-Z0-9]+)=([a-zA-Z0-9]+)", m.group(2))
        
        for pair in m_options:
            values[pair[0]] = pair[1]

    return values

The returned dictionary contains one entry for each of the entries in the DSN.

Update: helge also submitted a simplified version of the above:

driver, rest = dsn.split(':', 1)
values = dict(re.findall('(\w+)=(\w+)', rest), driver=driver)

Space, Page Up, Page Down, Home, End Broken for Navigation in Firefox

While I initially thought I had gone crazy or were still dreaming, my fear was true – all my keyboard page navigation had stopped working in Firefox (3.6.3). I tried to move onto to the next by pressing space or page down, my hands trembled as I discovered that neither arrow keys or home / end keys worked. The friendly internet suggested that I should turn Caret Browsing (F7) off (.. so try that if you’re in this situation), but that just told me that I were turning Caret Browsing on (with a proper warning). No luck.

After a bit of tinkering I disabled all my add-ons (previously known as extensions), and voilá, everything worked as it were supposed to! Further problem hunting revealed the culprit: The Del.icio.us Bookmarks Add-on. Curse you, del.icio.us! Disabling the Add-on and re-enabling all the other addons solved the issue, and I’m now happily browsing the internet one page size at the time again.

svn: Can’t convert string from native encoding to ‘UTF-8’

The error “svn: Can’t convert string from native encoding to ‘UTF-8′” suddenly made it impossible to update one of the projects on our staging servers. The project contains loads of file under SVN control, and several data directories which up to this time wasn’t svn:ignore’d. One of the files in one of these directories had norwegian letters in ISO-8859-1 in its filename (which didn’t work in the project anyhow, as it was something left around from earlier).

This single file borked svn from actually being able to update or do anything useful with the actual files under SVN control. When Subversion analyzed the directory structure to check which files it should attempt to update, it would just barf before seeing any files with the error message about the file name not being in UTF-8. You’d think it would be better to ignore errors for filenames that aren’t a part of svn and that you’re not trying to add, but there’s probably a good reason for this behaviour.

Anyways: The solution: delete the file. We didn’t use it anyway. There’s also a good chapter in the SVN Book about localization issues which contain information about how you can solve the issue by changing your active character set.

Fixing Issue With PHPs SoapClient Overwriting Duplicate Attribute and Tag Names

The setting:

An SOAP request contains an Id attribute – and an element with the exact name in the response (directly beneath the element containing the attribute – an immediate child):


  foobar

The problem is that the generated result object from the SoapClient (at least of PHP 5.2.12) contains the attribute value, and not the element value. In our case we could ignore the z:Id attribute, as it was simply an Id to identify the element in the response (this might be something that ASP.NET or some other .NET component does).

Our solution is to subclass the internal SoapClient and handle the __doRequest method, stripping out the part of the request that gives the wrong value for the Id field:

class Provider_SoapClient extends SoapClient
{
    public function __doRequest($request, $location, $action, $version)
    {
        $result = parent::__doRequest($request, $location, $action, $version);
        $result = preg_replace('/ z:Id="i[0-9]+"/', '', $result);
        return $result;
    }
}

This removes the attribute from all the values (there is no danger that the string will be present in any other of the elements. If there is – be sure to adjust the regular expression). And voilá, it works!

A Simple Smarty Modifier to Generate a Chart Through Google Chart API

After the longest title of my blog so far follows one of the shortest posts.

The function has two required parameters – the first one is provided automagically for you by smarty (it’s the value of the variable you’re applying the modifier to). This should be an array of objects containing the value you want to graph. The only required argument you have to provide to the modifier is the method to use for fetching the values for graphing.

Usage:
{$objects|googlechart:”getValue”}

This will dynamically load your plugin from the file modifier.googlechart.php in your Smarty plugins directory, or you can register the plugin manually by calling register_modifier on the template object after you’ve created it.

function smarty_modifier_googlechart($points, $method, $size = "600x200", $low = 0, $high = 0)
{
    $pointStr = '';
    $maxValue = 0;
    $minValue = INT_MAX;
    
    foreach($points as $point)
    {
        if ($point->$method() > $maxValue)
        {
            $maxValue = $point->$method();
        }

        if ($point->$method() < $minValue)
        {
            $minValue = $point->$method();
        }
    }

    if (!empty($high))
    {
        $maxValue = $high;
    }

    $scale = 100 / $maxValue;

    foreach($points as $point)
    {
        $pointStr .= (int) ($point->$method() * $scale) . ',';
    }

    $pointStr = substr($pointStr, 0, -1);

    // labels (5)
    $labels = array();

    $steps = 4;
    $interval = $maxValue / $steps;

    for($i = 0; $i < $steps; $i++)
    {
        $labels[] = (int) ($i * $interval);
    }

    $labels[] = (int) $maxValue;

    return 'http://chart.apis.google.com/chart?cht=lc&chd=t:' . $pointStr . '&chs=' . $size . '&chxt=y&chxl=0:|' . join('|', $labels);
}

The function does not support the short version of the Google Chart API Just Yet (tm) as it is an simple proof of concept hack made a few months ago.

How To Dismantle An Atomic HTTP Query .. String.

Following up on yesterday’s gripe about PHPs (old and now useless) automagic translation of dots in GET and POST parameters to underscores, today’s edition manipulates the query string in place instead of returning it as an array.

This is useful if you have a query string you want to pass on to another service, and for some reason the default behaviour in PHP will barf barf and barf. That might happen because of the dot translation issue or that some services (such as Solr) rely on a parameter name being repeatable (in PHP the second parameter value will overwrite the first).

function http_dismantle_query($queryString, $remove)
{
    $removeKeys = array();

    if (is_array($remove))
    {
        foreach($remove as $removeKey)
        {
            $removeKeys[$removeKey] = true;
        }
    }
    else
    {
        $removeKeys[$remove] = true;
    }

    $resultEntries = array();
    $segments = explode("&", $queryString);

    foreach($segments as $segment)
    {
        $parts = explode('=', $segment);

        $key = urldecode(array_shift($parts));

        if (!isset($removeKeys[$key]))
        {
            $resultEntries[] = $segment;
        }
    }

    return join('&', $resultEntries);
}

I’m not really sure what I’ll call the next function in this series, but there sure are loads of candidates out there.

Getting Dots to Work in PHP and GET / POST / COOKIE Variable Names

One of the oldest and ugliest relics of the register_globals era of PHP are the fact that all dots in request variable names gets replaced with “_”. If your variable was named “foo.bar”, PHP will serve it to you as “foo_bar”. You cannot turn this off, you cannot use extract() or parse_str() to avoid it and you’re mostly left out in the dark. Luckily the QUERY_STRING enviornment (in _SERVER if you’re running mod_php, etc) contains the raw string, and this string contains the dots.

The following “”parser”” is a work in progress and does currently not support the array syntax for keys that PHP allow, but it solves the issue for regular vars. I will try to extend this later on to do actually replicate the functionality of the regular parser.

Here’s the code. No warranties. Ugly hack. You’re warned. Leave a comment if you have any good suggestions regarding this (.. or know of an existing library doing the same..).

function http_demolish_query($queryString)
{
    $result = array();
    $segments = explode("&", $queryString);

    foreach($segments as $segment)
    {
        $parts = explode('=', $segment);

        $key = urldecode(array_shift($parts));
        $value = null;

        if ($parts)
        {
            $value = urldecode(join('=', $parts));
        }

        $result[$key] = $value;
    }

    return $result;
}

(OK, that’s not the real function name, but it’s aptly named to be the nemesis of http_build_query)