Mats Lindh » PHP

Solr Response Empty from PHP, but Works in Browser or CURL?

Mats — Wed, 19 Oct 2011 11:20:59 +0000

Weird issue that I think I’ve stumbled upon earlier, but yet again reared it’s head yesterday. Certain application containers (possibly Jetty in this case) will for some reason not produce any output from Solr (or other applications I’d guess) if the request is made with HTTP/1.0 as the version identifier (“GET /…/ HTTP/1.0″ as the first line of the request). The native HTTP support in PHP identifies itself as HTTP/1.0 as it doesn’t support request chunking, which then turns into a magical problem with requests that used to work, but doesn’t work any longer (the response is just zero bytes in size – all other headers are identical) – but still works as expected if you open them in your browser.

The solution is to either gamble on the server not sending any chunked responses and then setting protocol_version in the stream context that you pass to the file retrieving function (the list of HTTP wrapper settings (.. I don’t think it’s a good idea to define protocol_version as float, but .. well.)), or use cURL instead. The Solr pecl extension uses cURL internally, so it’s not affected by this issue.

Parse a DSN string in Python

Mats — Mon, 31 Jan 2011 11:46:12 +0000

A simple hack to get the different parts of a DSN string (which are used in PDO in PHP):

def parse_dsn(dsn):
m = re.search("([a-zA-Z0-9]+):(.*)", dsn)
values = {}
if (m and m.group(1) and m.group(2)):
values['driver'] = m.group(1)
m_options = re.findall("([a-zA-Z0-9]+)=([a-zA-Z0-9]+)", m.group(2))
for pair in m_options:
values[pair[0]] = pair[1]
return values

The returned dictionary contains one entry for each of the entries in the DSN.

Update: helge also submitted a simplified version of the above:

driver, rest = dsn.split(':', 1)
values = dict(re.findall('(\w+)=(\w+)', rest), driver=driver)

Solr, Tomcat and HTTP/1.1 505 HTTP Version Not Supported

Mats — Wed, 19 May 2010 10:48:50 +0000

During today’s hacking aboot I came across the above error from our Solr query library. The error indicates that some part of Tomcat was unable to parse the “GET / HTTP/1.1″ string – where it is unable to determine the “HTTP/1.1″ part. A problem like this could be introduced by having a space in the query string (and it not being escaped properly), so that the request would have been for “GET /?a=b c HTTP/1.1″. After running through both the working and non-working query through ngrep and wireshark, this did however not seem to be the problem. My spaces were properly escaped using plus signs (GET /?a=b+c HTTP/1.1).

There does however seem to be a problem (at least with our version of Tomcat – 6.0.20) which results in the +-s being resolved before the request is handed off to the code that attempts to parse the header, so even though it is properly escaped using “+”, it still barfs.

The solution:

Use %20 to escape spaces instead of + signs; simply adding str_replace(” “, “%20″, ..); in our query layer solved the problem.

Simple PHP Hack to Print an Integer in Binary Form

Mats — Mon, 05 Apr 2010 21:29:56 +0000

Today: Using my own blog as a simple pastebin to remember a helper function I wrote while debugging a good selection of binary operations on integers in PHP. This will simply output the provided value as bit values (0 / 1). Wrap it in

-s if you’re running it in a web context.
$len allows you do change how many bits it will print (starting from lsb – least significant bit), $blockSize adjust the amount of bits before throwing in a space.




function printBinary($val, $len = 32, $blockSize = 8)


{


    print("\n");


 


    for($i = $len - 1; $i >= 0; $i--)


    {


        print(($val & 0x1<<$i) ? 1 : 0);


 


        if ($i%$blockSize == 0)


        {


            print (" ");


        }


    }


 


    print ("\n");


}

Fixing Issue With PHPs SoapClient Overwriting Duplicate Attribute and Tag Names

Mats — Fri, 05 Feb 2010 13:48:59 +0000

The setting:

An SOAP request contains an Id attribute – and an element with the exact name in the response (directly beneath the element containing the attribute – an immediate child):

z:Id="i123">
>foobar>
>

The problem is that the generated result object from the SoapClient (at least of PHP 5.2.12) contains the attribute value, and not the element value. In our case we could ignore the z:Id attribute, as it was simply an Id to identify the element in the response (this might be something that ASP.NET or some other .NET component does).

Our solution is to subclass the internal SoapClient and handle the __doRequest method, stripping out the part of the request that gives the wrong value for the Id field:

class Provider_SoapClient extends SoapClient
{
public function __doRequest($request, $location, $action, $version)
{
$result = parent::__doRequest($request, $location, $action, $version);
$result = preg_replace('/ z:Id="i[0-9]+"/', '', $result);
return $result;
}
}

This removes the attribute from all the values (there is no danger that the string will be present in any other of the elements. If there is – be sure to adjust the regular expression). And voilá, it works!

Avoid Escaping Spaces in the Query String in a Solr Query

Mats — Sat, 30 Jan 2010 22:24:14 +0000

Following up on the previous post about escaping values in a Solr query string, it’s important to note that you should not escape spaces in the query itself. The reason for this is that if you escape spaces in the query “foo bar”, the search will be performed on the term “foo bar” itself, and not with “foo” as one term and “bar” as the other. This will only return documents that has the string “foo bar” in sequence.

The solution is to either remove the space from the escape list in the previous function – and use another function for escaping values where you actually should escape the spaces – or break up the string into “escapable” parts.

The code included beneath performs the last task; it splits the string into different parts delimited by space and then escapes each part of the query by itself.

$queryParts = explode(' ', $this->getQuery());
$queryEscaped = array();
foreach($queryParts as $queryPart)
{
$queryEscaped[] = self::escapeSolrValue($queryPart);
}
$queryEscaped = join(' ', $queryEscaped);

A Simple Smarty Modifier to Generate a Chart Through Google Chart API

Mats — Fri, 29 Jan 2010 22:48:55 +0000

After the longest title of my blog so far follows one of the shortest posts.

The function has two required parameters – the first one is provided automagically for you by smarty (it’s the value of the variable you’re applying the modifier to). This should be an array of objects containing the value you want to graph. The only required argument you have to provide to the modifier is the method to use for fetching the values for graphing.

Usage:
{$objects|googlechart:”getValue”}

This will dynamically load your plugin from the file modifier.googlechart.php in your Smarty plugins directory, or you can register the plugin manually by calling register_modifier on the template object after you’ve created it.

function smarty_modifier_googlechart($points, $method, $size = "600×200", $low = 0, $high = 0)
{
$pointStr = '';
$maxValue = 0;
$minValue = INT_MAX;
foreach($points as $point)
{
if ($point->$method() > $maxValue)
{
$maxValue = $point->$method();
}
if ($point->$method() < $minValue)
{
$minValue = $point->$method();
}
}
if (!empty($high))
{
$maxValue = $high;
}
$scale = 100 / $maxValue;
foreach($points as $point)
{
$pointStr .= (int) ($point->$method() * $scale) . ',';
}
$pointStr = substr($pointStr, 0, -1);
// labels (5)
$labels = array();
$steps = 4;
$interval = $maxValue / $steps;
for($i = 0; $i < $steps; $i++)
{
$labels[] = (int) ($i * $interval);
}
$labels[] = (int) $maxValue;
return 'http://chart.apis.google.com/chart?cht=lc&chd=t:' . $pointStr . '&chs=' . $size . '&chxt=y&chxl=0:|' . join('|', $labels);
}

The function does not support the short version of the Google Chart API Just Yet ™ as it is an simple proof of concept hack made a few months ago.

How To Dismantle An Atomic HTTP Query .. String.

Mats — Thu, 28 Jan 2010 20:48:40 +0000

Following up on yesterday’s gripe about PHPs (old and now useless) automagic translation of dots in GET and POST parameters to underscores, today’s edition manipulates the query string in place instead of returning it as an array.

This is useful if you have a query string you want to pass on to another service, and for some reason the default behaviour in PHP will barf barf and barf. That might happen because of the dot translation issue or that some services (such as Solr) rely on a parameter name being repeatable (in PHP the second parameter value will overwrite the first).

function http_dismantle_query($queryString, $remove)
{
$removeKeys = array();
if (is_array($remove))
{
foreach($remove as $removeKey)
{
$removeKeys[$removeKey] = true;
}
}
else
{
$removeKeys[$remove] = true;
}
$resultEntries = array();
$segments = explode("&", $queryString);
foreach($segments as $segment)
{
$parts = explode('=', $segment);
$key = urldecode(array_shift($parts));
if (!isset($removeKeys[$key]))
{
$resultEntries[] = $segment;
}
}
return join('&', $resultEntries);
}

I’m not really sure what I’ll call the next function in this series, but there sure are loads of candidates out there.

Getting Dots to Work in PHP and GET / POST / COOKIE Variable Names

Mats — Wed, 27 Jan 2010 22:22:21 +0000

One of the oldest and ugliest relics of the register_globals era of PHP are the fact that all dots in request variable names gets replaced with “_”. If your variable was named “foo.bar”, PHP will serve it to you as “foo_bar”. You cannot turn this off, you cannot use extract() or parse_str() to avoid it and you’re mostly left out in the dark. Luckily the QUERY_STRING enviornment (in _SERVER if you’re running mod_php, etc) contains the raw string, and this string contains the dots.

The following “”parser”" is a work in progress and does currently not support the array syntax for keys that PHP allow, but it solves the issue for regular vars. I will try to extend this later on to do actually replicate the functionality of the regular parser.

Here’s the code. No warranties. Ugly hack. You’re warned. Leave a comment if you have any good suggestions regarding this (.. or know of an existing library doing the same..).

function http_demolish_query($queryString)
{
$result = array();
$segments = explode("&", $queryString);
foreach($segments as $segment)
{
$parts = explode('=', $segment);
$key = urldecode(array_shift($parts));
$value = null;
if ($parts)
{
$value = urldecode(join('=', $parts));
}
$result[$key] = $value;
}
return $result;
}

(OK, that’s not the real function name, but it’s aptly named to be the nemesis of http_build_query)

Retrieving URLs in Parallel With CURL and PHP

Mats — Sun, 24 Jan 2010 10:00:50 +0000

As we’ve recently added support for querying Solr servers in parallel, one of the things we added was a simple class to allow us to query several servers at the same time. The CURL library (which has a PHP extension) even provides an abstraction layer for doing the nitty gritty work for you, as long as you keep track of the resources. The code beneath is based on examples in the documentation and a few tweaks of my own.

The code beneath is licensed under a MIT license. You can also download the file (gzipped).

class Footo_Content_Retrieve_HTTP_CURLParallel
{
/**
* Fetch a collection of URLs in parallell using cURL. The results are
* returned as an associative array, with the URLs as the key and the
* content of the URLs as the value.
*
* @param array $addresses An array of URLs to fetch.
* @return array The content of each URL that we've been asked to fetch.
**/
public function retrieve($addresses)
{
$multiHandle = curl_multi_init();
$handles = array();
$results = array();
foreach($addresses as $url)
{
$handle = curl_init($url);
$handles[$url] = $handle;
curl_setopt_array($handle, array(
CURLOPT_HEADER => false,
CURLOPT_RETURNTRANSFER => true,
));
curl_multi_add_handle($multiHandle, $handle);
}
//execute the handles
$result = CURLM_CALL_MULTI_PERFORM;
$running = false;
// set up and make any requests..
while ($result == CURLM_CALL_MULTI_PERFORM)
{
$result = curl_multi_exec($multiHandle, $running);
}
// wait until data arrives on all sockets
while($running && ($result == CURLM_OK))
{
if (curl_multi_select($multiHandle) > -1)
{
$result = CURLM_CALL_MULTI_PERFORM;
// while we need to process sockets
while ($result == CURLM_CALL_MULTI_PERFORM)
{
$result = curl_multi_exec($multiHandle, $running);
}
}
}
// clean up
foreach($handles as $url => $handle)
{
$results[$url] = curl_multi_getcontent($handle);
curl_multi_remove_handle($multiHandle, $handle);
curl_close($handle);
}
curl_multi_close($multiHandle);
return $results;
}
}

Download the file.