Mats Lindh » Programming http://e-mats.org Where desperate is just another word for a regular day. Wed, 01 Feb 2012 14:42:48 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 A Gentle Introduction to Gearman and its Concepts http://e-mats.org/2011/08/a-gentle-introduction-to-gearman-and-its-concepts/ http://e-mats.org/2011/08/a-gentle-introduction-to-gearman-and-its-concepts/#comments Mon, 01 Aug 2011 21:21:54 +0000 Mats http://e-mats.org/?p=935 Gearman (an anagram for “Manager”) is a system for farming out work units to several different servers (or several processes on one server), allowing the calling code to do something completely different while the task is performed. Gearman is not intended for inter-process communication, but is a way to tell other processes that there are work available, and letting these processes (called workers) grab a piece of work for themselves.

One of the common themes that show up at the gearman IRC channel on freenode is an attempt to understand what gearman is and how everything fits together. I’ll try to explain the different concepts and what the different responsibilities of a working gearman infrastructure are. There’s also a “Getting Started” guide on the Gearman web site with a bit of example code and installation instructions, so you might want to keep that open in another tab. So here we go: a simple gearman tutorial explaining the concepts and not just throwing example code your way.

There are three core components of a gearman installation. These are a client (someone requesting a task to be performed), a worker (someone performing a task) and the server (which coordinates tasks between clients and workers). All these three components need to be running for you to be able to something useful with gearman. It’s worth noting that I’ll use name “task” for a single item to be performed, you’ll also see this named ‘function’ (which is the name of the actual function the task asks to be performed – a server offers several “functions” that a client can call). Some APIs might also refer to a “task” as a collection of functions to be called. I’ll use the first definition; a task is a call to a function on the server, together with the data for the task and a task identifier. Several subsequent tasks will call the same function.

I’ll go a bit more in detail about each of these components, but it’s important that you understand how everything is interconnected first. An exchange of messages between the different parts can be illustrated as follows:

client -> server: ask server to perform a task
server acknowledges request and assigns an identificator to the request
server -> all workers: tell workers registered for the task that there is work to be performed
worker -> server: I'll perform the task you just told us about
server -> worker: ok, go ahead, here's the information about the task.
worker -> server: here's the result of the task performed
server -> client: here's the result of the task you asked me to get someone to do for you

The idea behind the server telling all the workers that there are work available is to let the worker that responds fastest to actually get the task, as it’s assumed that this is the worker with the least load on the server it’s running on (as it responds quickly, the server is not busy doing other things). As I wrote above, the worker is the piece of code actually doing the work – the worker performs the task that a client has submitted to the gearman server.

You’ll find that most of Gearman is designed according to the same principle – keep stuff simple. The server only needs to keep track of which workers perform which functions, and then let the workers grab a task when it becomes available.

The Gearman Client

In Gearman the client is the piece of code that connects to the server and asks for a task to be performed. This can be a dynamic web page (running in python, ruby, PHP, perl or another language with a suitable Gearman library), a completely application that connects to Gearman, a worker (to submit a new task or to divide the current task into several smaller tasks to be performed by other workers) or a combination of the above. The important part is that this is simply a client – it has a task that needs to be handled, and it’ll ask the Gearman server to find someone who can perform the task.

The client can be run in synchronous (blocking) or asynchronous (non-blocking) mode. The first will make the client wait until the task has been performed by a worker (and if no worker is available, it’ll wait indefinitely or until reaching a timeout in the client), while the latter will simply fire-and-forget the task to the Gearman server (the server will confirm that the task has been received) and then go on its merry way afterwards. The Gearman server will provide a task identification value which the asynchronous client can use to query the current state of the task it asked to be performed (as long as the actual worker provide such updates).

A small example of how a client might work (using PHP):

  1. <?php
  2. $client = new GearmanClient();
  3. $client->addServer('localhost', 4730);
  4.  
  5. $arguments = array(
  6.     'url' => 'http://www.example.com/',
  7. );
  8.  
  9. $client->addTaskBackground('fetchURL', json_encode($arguments));
  10.  
  11. $client->runTasks();

This will submit a request to a Gearman server running on the same machine as the script, asking for the function “fetchURL” to be run, and including an array of arguments to the function (you could simply include just the URL, but I find that this way is easier to extend in the future – and using JSON for data exchange makes the worker code more programming language independent). This code uses addTaskBackground to submit the task to be performed in an asynchronous manner. We’re not interested in the result of this task in this particular piece of code – the worker will either provide the result through other means (storing it in a database, in memcache, call an API function telling us that it’s finished) or perhaps we’re not interested in the result at all, just that we’ve attempted to perform the task. If you’re using the synchronous interface, the data returned from the worker will be returned to your code as the return value from the client.

As you can see, the client code is very, very simple. There is no actual work being performed here, we’re just telling the server that we’d like some work to be performed for us.

The Gearman Worker

The Gearman worker is where all the actual work (.. who’d guess) is performed. This is the application that receives a notice that it has to wake up and do a bit of hard work, and which actually goes out and does just that. What kind of work it does depends on what you’re using Gearman for, but a couple of use cases could be to resize an image into smaller sizes (such as thumbnails), to convert an uploaded video into another format for a specific device, sending notification emails, updating an internal search engine such a Solr and quite a few other tasks. As long as the task is not important for the application to continue running (no need for waiting for an E-mail to be delivered if you’re going to show a “Your information has been saved” message), then Gearman (and other alternative message queues) is a valid solution.

You’ll run each worker as its own process. A worker can perform several different functions (although you should (usually) stay away from multi-threading to perform them at the same time). This means starting several copies of the same worker if you want to allow for more than one worker performing a task at the same time (i.e., if you want to send 30 e-mails in parallel), you’ll start each worker as separate processes (30 workers in that case). There are several daemons and frameworks that can help you manage the number of processes available depending on server and task load, such as supervisord and GearmanManager (a PHP daemon). Another possible solution is to use screen to start several workers, which also will allow you to attach to the output of any worker at any time.

How the worker performs its work is up to the worker itself. In most cases you’ll have to write a bit of code to expose your code as a Gearman function (so that clients can submit tasks to perform that function), but this code will usually just instantiate the worker framework from the Gearman library you’re using, letting you register what functions you’ll be able to perform and attaching callbacks telling the library what part of your own code should be called when a request to perform a task arrives.

A simple example modified from the Gearman Getting Started guide:

  1. <?php
  2. $worker = new GearmanWorker();
  3. $worker->addServer("localhost", 4730);
  4. $worker->addFunction("fetchURL", "fetch_url");
  5.  
  6. while ($worker->work());
  7.  
  8. function fetch_url($job)
  9. {
  10.     $arguments = json_decode($job->workload());
  11.  
  12.     if (!empty($arguments['url']))
  13.     {
  14.         print("Fetching " . $arguments['url'] . "\n");
  15.         return file_get_contents($arguments['url']);
  16.     }
  17. }

The $worker->work() method call will wait until a work arrives, then execute the callback as defined in the addFunction call. addFunction instructs the worker to tell the gearman server that this worker is able to perform any tasks calling the “fetchURL” function. The callback provided to the library (“call this PHP function (‘fetch_url’) when tasks want to call ‘fetchURL’”) will then receive the job object containing information about the job (task) to be performed. The workload() method returns the workload – the information we included in addition to which function to call in the client example. The server receives the workload from the client and then sends it to the worker together with the task information.

Since our client calls the server using the asynchronous interface it’ll not wait for the worker to return the web page contents, but by using ->do() or one of the other foreground methods in the PHP Gearman library.

The Gearman Server

The Gearman Server used is usually the C version of the server. There’s also a PERL version, but these days the C server is the one being actively developed. There’s not much to say about the server, you usually just start it and let it run by itself, doing what it was supposed to do all along.

I’ve got one simple suggestion if you’re just playing around with Gearman for the first time: start the server with the -vvv option. This will make gearmand a lot noisier, and will allow you to see clients registering themselves with the server, pinging the server and getting a bit more information about what’s happening inside the server process.

You’ll also want to provide an IP address that the gearman server should bind to – by default it binds to all interfaces, and since gearmand does not have any authentication built in by default, you don’t want to expose your server to the whole world.

Here’s an example of how we start gearmand at one of our servers:

  1. screen -d -m -S gearmand /usr/local/sbin/gearmand -L 127.0.0.1 -p 4730 -vvv

You can drop the part related to screen if you just want to play with gearmand:

  1. /usr/local/sbin/gearmand -L 127.0.0.1 -p 4730 -vvv

If you have gearmand in your path and not in the same location as us, drop /usr/local/sbin :-) This will bind gearmand to your localhost and use the default port (earlier the default port was something other than 4730, so we provide it just in case).

Making it all come together

The easiest way to play around with gearman is to simply open three terminal windows: one for gearmand with logging turned on, one for your worker and its output and the last window for a client sending a task request to gearmand (you can use the ‘gearman’ binary for this, just be sure to include any data in an appropriate format). As you submit a task for a function that the worker has registered, you should see it pick it up and then start processing the task as soon as possible. After a while (depending on how you’ve implemented your worker and what function it performs) the result should appear in your client.

Our production setups usually use a web application (PHP or python/django) as the client in the above scenario. The functions are usually long running tasks, such as analysing GPS paths, encoding videos and downloading files or internal web site analytics (where we just want to get things logged and not wait for the actual logging to complete). The web application submits a request to gearmand as soon as a file has been received, with a payload of the path to the file to be processed. The workers perform their function and then store the information back into the database or to disk, then usually call a web service to tell the web application that the work has been performed and any internal state can be updated to include (and show) the result of the task.

Message queues (such as Gearman) has become one of the core technologies behind many modern web applications (and non-web applications for that matter), so there’s really no reason to avoid at least playing around a bit with it and adding another possible tool to your future options.

]]> http://e-mats.org/2011/08/a-gentle-introduction-to-gearman-and-its-concepts/feed/ 2 Creating / Generating a URL Friendly Snippet in Django http://e-mats.org/2011/03/creating-generating-a-url-friendly-snippet-in-django/ http://e-mats.org/2011/03/creating-generating-a-url-friendly-snippet-in-django/#comments Sat, 19 Mar 2011 12:15:11 +0000 Mats http://e-mats.org/?p=866 When generating URLs that say something about the content it contains, you usually have the need to create url-friendly strings (such as “creating-generating-a-url-friendly-snippet-in-django” for this post). Earlier today I had the need build something like that in Django, but came up empty handed for a while – simply because I didn’t realize that they’re called slugs in the Django documentation (which in turn inherited it from the print paper world). That helped quite a bit, and here’s the things you need to know:

If you want your model to perform validation and automagically add a database index for the slug, put in a SlugField in your model. You could use a regular CharField, but this will not provide any validation of the field in the admin interface (and you’d have to add db_index yourself). This will however not populate the field with a value, but this stack overflow question has a few pointers on how you could implement that. I’ll cite the interesting parts here:

  1. from django.template.defaultfilters import slugify
  2.  
  3. class test(models.Model):
  4.     q = models.CharField(max_length=30)
  5.     s = models.SlugField()
  6.  
  7.     def save(self, *args, **kwargs):
  8.         if not self.id:
  9.             self.s = slugify(self.q)
  10.  
  11.         super(test, self).save(*args, **kwargs)

This will populate the slug field the first time the entry is saved, and will not update the slug value if the object itself is updated.

You can however also handle this yourself in the get_absolute_url of your object, if you don’t need to perform any lookups on the value (although you could still save it to avoid regenerating it later). This will also give you the current slug for the entry (again, if you only use it as a part of the URL and do not validate it in any way):

  1. from django.template.defaultfilters import slugify
  2.  
  3. def test(models.Model):
  4.     q = models.CharField(max_length=30)
  5.  
  6.     @models.permalink
  7.     def get_absolute_url(self):
  8.         return ('reviewerer.views.sites_detail', [str(self.id), slugify(self.q)])

Hopefully that will help you solve your issue .. or at least search for the correct term next time :-)

]]> http://e-mats.org/2011/03/creating-generating-a-url-friendly-snippet-in-django/feed/ 0 Parse a DSN string in Python http://e-mats.org/2011/01/parse-a-dsn-string-in-python/ http://e-mats.org/2011/01/parse-a-dsn-string-in-python/#comments Mon, 31 Jan 2011 11:46:12 +0000 Mats http://e-mats.org/?p=852 A simple hack to get the different parts of a DSN string (which are used in PDO in PHP):

  1. def parse_dsn(dsn):
  2.     m = re.search("([a-zA-Z0-9]+):(.*)", dsn)
  3.     values = {}
  4.    
  5.     if (m and m.group(1) and m.group(2)):
  6.         values['driver'] = m.group(1)
  7.         m_options = re.findall("([a-zA-Z0-9]+)=([a-zA-Z0-9]+)", m.group(2))
  8.        
  9.         for pair in m_options:
  10.             values[pair[0]] = pair[1]
  11.  
  12.     return values

The returned dictionary contains one entry for each of the entries in the DSN.

Update: helge also submitted a simplified version of the above:

  1. driver, rest = dsn.split(':', 1)
  2. values = dict(re.findall('(\w+)=(\w+)', rest), driver=driver)
]]> http://e-mats.org/2011/01/parse-a-dsn-string-in-python/feed/ 1 Introducing Mismi – Amazon Price Comparison for Norwegian Customers http://e-mats.org/2011/01/introducing-mismi-amazon-price-comparison-for-norwegian-customers/ http://e-mats.org/2011/01/introducing-mismi-amazon-price-comparison-for-norwegian-customers/#comments Mon, 10 Jan 2011 21:40:19 +0000 Mats http://e-mats.org/?p=831 My main project in December was Mismi – a service that compares the total price of items from Amazon.com and from Amazon.co.uk for Norwegians. The solution is built on top of the Zend_Service_Amazon class (with a few extensions of my own).

The reasoning behind making the service is that there are several factors that are in play when deciding whether to order a product from the US or from the UK: the exchange rate for GBP and USD, the shipping cost, the delivery situation for the item and whether the item is sold in the store at all.

The user enters a list of the URLs to the products they’re considering purchasing from an Amazon-store, press submit and get a list back of which items are in stock, where the item is the cheapest and what the total sum of an order placed at the store would be. In addition I added a alpha stage feature just before Christmas which will also tell you the “optimum” set of items for the orders – “order item 1,4,7,9 from .com, item 2,3,5,6,8 from .co.uk”. This took quite a bit of hacking – you also have to consider the initial price of shipping, shipping for each item and other fun things.

Feel free to play with it over at mismi.e-mats.org. It’s in Norwegian, but it should be easy to understand anyhow with the description above.

]]> http://e-mats.org/2011/01/introducing-mismi-amazon-price-comparison-for-norwegian-customers/feed/ 0 mod_jk and Internal Server Error (HTTP 500) http://e-mats.org/2011/01/mod_jk-and-internal-server-error-http-500/ http://e-mats.org/2011/01/mod_jk-and-internal-server-error-http-500/#comments Tue, 04 Jan 2011 12:26:11 +0000 Mats http://e-mats.org/?p=823 We’ve extended our previously single Solr-node to a few nodes in a cluster. This allows us to run queries against one node while updating or configuring another, distributing the load across several servers (although we’re not there yet load wise) and being able to handle any out of memory or other critical errors.

While Solr supports querying several cores or distributing the queries internally, we decided to move the load balancing and handling of failed nodes higher up in the hierarchy. We’re now doing simple load balancing and handling of failed nodes by using mod_jk in our existing Apache-based environment. mod_jk also handles failed servers without any administrator interaction. We were already using mod_jk for our main web frontend, and since we use Tomcat as our application container for Solr, things should be a breeze!

Well, no. After copying our existing mod_jk setup, configuring our new workers and restarting Apache, all I got was the well known 500 INTERNAL SERVER ERROR. Here’s the worker configuration file:

worker.list=loadbalancer,status

worker.solr1.port=8009
worker.solr1.host=10.0.0.4
worker.solr1.type=ajp13
worker.solr1.lbfactor=1
worker.solr1.cachesize=10

worker.solr2.port=8009
worker.solr2.host=10.0.0.5
worker.solr2.type=ajp13
worker.solr2.lbfactor=4
worker.solr2.cachesize=10

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=solr1,solr2
worker.loadbalancer.sticky_session=0

worker.status.type=status

This provides us with two solr servers and one status worker (the status worker is responsible for providing a simple web interface for enabling/disabling/seeing the status of the other workers), configured with a 1:4 load balancing (the second server has quite a bit more memory available for Solr).

I provided the configuration of the workers through the JkWorkersFile configuration setting (in a VirtualHost block… don’t do that):

JkWorkersFile conf/workers.properties

I’d also enable debug logging to attempt to find the problem (still in a VirtualHost block):

JkLogFile logs/mod_jk.log
JkLogLevel debug
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

Other mod_jk settings (in the VirtualHost block) were:

JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"
JkShmFile logs/jk.shm
JkMount /* loadbalancer

<Location /jkstatus>
	JkMount status
	Order deny,allow
        Deny from all
        Allow from 127.0.0.1
</Location>

Still no solution. Peeking at the log files mod_jk provided, I were able to deduce the following:

[debug] map_uri_to_worker::jk_uri_worker_map.c (525): Attempting to map context URI '/jkstatus'
[debug] map_uri_to_worker::jk_uri_worker_map.c (550): Found an exact match status -> /jkstatus
[debug] jk_handler::mod_jk.c (1920): Into handler jakarta-servlet worker=status r->proxyreq=0
[debug] wc_get_worker_for_name::jk_worker.c (111): did not find a worker status
[info]  jk_handler::mod_jk.c (2071): Could not find a worker for worker name=status

This indicates that mod_jk was unable to find a worker matching the name I provided in the JkMount statement above; status. Weird. I added some garbage characters to the “JkWorkersFile” setting, and Apache complained that it were unable to find the workers file. Changed it back, reloaded, and still nothing. It was apparently unable to find the worker. The map did however work, as it tried to launch a worker.

Looking back at the start up sequence of mod_jk, the following were found in the log:

[debug] build_worker_map::jk_worker.c (236): creating worker ajp13
[debug] wc_create_worker::jk_worker.c (141): about to create instance ajp13 of ajp13
[debug] wc_create_worker::jk_worker.c (154): about to validate and init ajp13
[debug] ajp_validate::jk_ajp_common.c (1922): worker ajp13 contact is 'localhost:8009'
[debug] ajp_init::jk_ajp_common.c (2047): setting endpoint options:
[debug] ajp_init::jk_ajp_common.c (2050): keepalive:        0
[debug] ajp_init::jk_ajp_common.c (2054): timeout:          -1
[debug] ajp_init::jk_ajp_common.c (2058): buffer size:      0
ajp_init::jk_ajp_common.c (2062): pool timeout:     0
[debug] ajp_init::jk_ajp_common.c (2066): connect timeout:  0
[debug] ajp_init::jk_ajp_common.c (2070): reply timeout:    0
[debug] ajp_init::jk_ajp_common.c (2074): prepost timeout:  0
[debug] ajp_init::jk_ajp_common.c (2078): recovery options: 0
[debug] ajp_init::jk_ajp_common.c (2082): retries:          2
[debug] ajp_init::jk_ajp_common.c (2086): max packet size:  8192
[debug] ajp_create_endpoint_cache::jk_ajp_common.c (1959): setting connection pool size to 1 with min 0

It took a bit of time, but I realized that this tells me that mod_jk created _a default_ worker named ajp13. Apparently it was not reading my worker file at all, but it still complained if I changed the file name. You’d think that the setting which loads the configuration file would work when it complains when it doesn’t. But .. well. After an hour of attempting to find out why the workers didn’t load, revising the workers file to a minimal example, trying with just a single status worker, I concluded that my workers file was correct, and obviously mod_jk found it when it attempted to load it.

Then I suddenly discovered the small notice in the mod_jk configuration manual:

JkWorkersFile: This directive is only allowed once. It must be put into the global part of the configuration.

JkWorkersFile can not be defined in a <VirtualHost> section. It will NOT complain if you do it, it’ll just never define any workers. It will complain if the file doesn’t exist, even if it never tries to actually load it.

Confusing.

Moving the JkWorkersFile statement out from the <VirtualHost> block and to the LoadModule statement instead solved the issue. This is also the case for JkWorkerProperty.

]]> http://e-mats.org/2011/01/mod_jk-and-internal-server-error-http-500/feed/ 1 Patch for Max Iterations for a foreach Block in Smarty http://e-mats.org/2010/08/patch-for-max-iterations-for-a-foreach-block-in-smarty/ http://e-mats.org/2010/08/patch-for-max-iterations-for-a-foreach-block-in-smarty/#comments Mon, 09 Aug 2010 17:35:27 +0000 Mats http://e-mats.org/?p=807 After running into a need for a max iteration count on a foreach block tonight and seeing that several others have had the need during the years, I’ve created a simple patch to add max= as an attribute to the foreach block. I tried to search the archives for a reason why this hadn’t already been included, so feel free to ignore this patch if there are proper reasons why this isn’t available as an argument. There are cases where a simple break in the loop is more efficient than making a copy with array_slice if you need the same data several places but in different slice sizes.

The patch also contains three tests to test the max attribute.

The patch is available here: smarty.foreach.max.patch. The patch is against the current SVN trunk of 2010-08-08.

Example:

{foreach item=x from=[0,1,2,3,4,5,6,7,8,9] max=5}{$x}{/foreach}

Output:

01234

]]> http://e-mats.org/2010/08/patch-for-max-iterations-for-a-foreach-block-in-smarty/feed/ 0 Unbreak My Hea.. Firefox Ctrl Click Please! http://e-mats.org/2010/03/unbreak-my-hea-firefox-ctrl-click-please/ http://e-mats.org/2010/03/unbreak-my-hea-firefox-ctrl-click-please/#comments Thu, 11 Mar 2010 21:57:34 +0000 Mats http://e-mats.org/?p=765 When we launched Gamer.no over a year ago, we had to come up with a wallpaper advertising solution in a rush (everything were a rush back then as we built and launched a site from scratch (after disagreements between the previous owner and Gamer) in just under four days (or 96 hours)). While this solution has worked .. good enough .. it has always had a few irky bugs that I’ve never really had the right inspiration to uncover the cause of. Usually I’ve spent an hour and decided that the time wasn’t worth it at the moment and then moved onto something else, but today! Today is a glorious day!

The bug has been fixed!

The wallpaper element is placed around the main content div, which sadly also makes the wallpaper element receive any click elements that the main content div receives. This leads to the wallpaper getting clicked and the wallpaper ad window opening regardless of where people click – which will get very, very annoying very quick. So to battle this issue the original solution was to call .stopPropagation() on the evt object in a click handler for the main content div. This solved the issue and everyone rejoiced! However, all was not perfect in paradise.

Some time later we discovered that the .stopPropagation() fix borked ctrl-click a link in Firefox. Other browsers handled it just fine, but Firefox were obviously not happy. Not happy at all. Mad and going on a killing spree it shot down the proposed fixes from both myself and other people who had a brief look at the code. It wasn’t a big issue as we only run the wallpaper code for small intervals of time and people didn’t complain (maybe we were some of the few who had the issue).

Today I decided to have a look at the issue again, and finally I realized that we had been way to focused on our call to .stopPropagation(). Everyone had been planning how we could get .stopPropagation to do what we wanted it to do – after all – the issue was that stopPropagation didn’t behave when we ctrl-clicked in Firefox. But wait.

If you instead think of the original problem; the window.open gets triggered when people click the inner element instead of the outer, there may be alternative solutions to using stopPropagation. And yes, THAT was quite a simple fix. Instead of trying to stop the event from bubling up through the cloud.. let’s just set a status variable that tells the code handling the wallpaper click that THIS CLICK IS NOT FOR YOU BAD HANDLER GO AWAY LET OTHER GROWNUPS HANDLE THIS. So that I did.

  1. $(document).ready(function () {
  2.     innerClick = false;
  3.     $('#wallpaper').click(function() {
  4.         if (innerClick)
  5.         {
  6.             innerClick = false;
  7.             return true;
  8.         }
  9.        
  10.         window.open("..");
  11.     });
  12.     $('#content').click(function(evt) {
  13.         innerClick = true;
  14.     });
  15. });

As soon as I actually spent some time on what we were trying to solve instead of what seemed like the cause of the issue .. everything went better than expected.

]]> http://e-mats.org/2010/03/unbreak-my-hea-firefox-ctrl-click-please/feed/ 2 Fixing Issue With PHPs SoapClient Overwriting Duplicate Attribute and Tag Names http://e-mats.org/2010/02/fixing-issue-with-phps-soapclient-overwriting-duplicate-attribute-and-tag-names/ http://e-mats.org/2010/02/fixing-issue-with-phps-soapclient-overwriting-duplicate-attribute-and-tag-names/#comments Fri, 05 Feb 2010 13:48:59 +0000 Mats http://e-mats.org/?p=750 The setting:

An SOAP request contains an Id attribute – and an element with the exact name in the response (directly beneath the element containing the attribute – an immediate child):

  1. <res z:Id="i123">
  2.   <Id>foobar</Id>
  3. </res>

The problem is that the generated result object from the SoapClient (at least of PHP 5.2.12) contains the attribute value, and not the element value. In our case we could ignore the z:Id attribute, as it was simply an Id to identify the element in the response (this might be something that ASP.NET or some other .NET component does).

Our solution is to subclass the internal SoapClient and handle the __doRequest method, stripping out the part of the request that gives the wrong value for the Id field:

  1. class Provider_SoapClient extends SoapClient
  2. {
  3.     public function __doRequest($request, $location, $action, $version)
  4.     {
  5.         $result = parent::__doRequest($request, $location, $action, $version);
  6.         $result = preg_replace('/ z:Id="i[0-9]+"/', '', $result);
  7.         return $result;
  8.     }
  9. }

This removes the attribute from all the values (there is no danger that the string will be present in any other of the elements. If there is – be sure to adjust the regular expression). And voilá, it works!

]]> http://e-mats.org/2010/02/fixing-issue-with-phps-soapclient-overwriting-duplicate-attribute-and-tag-names/feed/ 0 Avoid Escaping Spaces in the Query String in a Solr Query http://e-mats.org/2010/01/avoid-escaping-spaces-in-the-query-string-in-a-solr-query/ http://e-mats.org/2010/01/avoid-escaping-spaces-in-the-query-string-in-a-solr-query/#comments Sat, 30 Jan 2010 22:24:14 +0000 Mats http://e-mats.org/?p=744 Following up on the previous post about escaping values in a Solr query string, it’s important to note that you should not escape spaces in the query itself. The reason for this is that if you escape spaces in the query “foo bar”, the search will be performed on the term “foo bar” itself, and not with “foo” as one term and “bar” as the other. This will only return documents that has the string “foo bar” in sequence.

The solution is to either remove the space from the escape list in the previous function – and use another function for escaping values where you actually should escape the spaces – or break up the string into “escapable” parts.

The code included beneath performs the last task; it splits the string into different parts delimited by space and then escapes each part of the query by itself.

  1. $queryParts = explode(' ', $this->getQuery());
  2. $queryEscaped = array();
  3.  
  4. foreach($queryParts as $queryPart)
  5. {
  6.     $queryEscaped[] = self::escapeSolrValue($queryPart);
  7. }
  8.  
  9. $queryEscaped = join(' ', $queryEscaped);
]]> http://e-mats.org/2010/01/avoid-escaping-spaces-in-the-query-string-in-a-solr-query/feed/ 1 A Simple Smarty Modifier to Generate a Chart Through Google Chart API http://e-mats.org/2010/01/a-simple-smarty-modifier-to-generate-a-chart-through-google-chart-api/ http://e-mats.org/2010/01/a-simple-smarty-modifier-to-generate-a-chart-through-google-chart-api/#comments Fri, 29 Jan 2010 22:48:55 +0000 Mats http://e-mats.org/?p=739 After the longest title of my blog so far follows one of the shortest posts.

The function has two required parameters – the first one is provided automagically for you by smarty (it’s the value of the variable you’re applying the modifier to). This should be an array of objects containing the value you want to graph. The only required argument you have to provide to the modifier is the method to use for fetching the values for graphing.

Usage:
{$objects|googlechart:”getValue”}

This will dynamically load your plugin from the file modifier.googlechart.php in your Smarty plugins directory, or you can register the plugin manually by calling register_modifier on the template object after you’ve created it.

  1. function smarty_modifier_googlechart($points, $method, $size = "600×200", $low = 0, $high = 0)
  2. {
  3.     $pointStr = '';
  4.     $maxValue = 0;
  5.     $minValue = INT_MAX;
  6.    
  7.     foreach($points as $point)
  8.     {
  9.         if ($point->$method() > $maxValue)
  10.         {
  11.             $maxValue = $point->$method();
  12.         }
  13.  
  14.         if ($point->$method() < $minValue)
  15.         {
  16.             $minValue = $point->$method();
  17.         }
  18.     }
  19.  
  20.     if (!empty($high))
  21.     {
  22.         $maxValue = $high;
  23.     }
  24.  
  25.     $scale = 100 / $maxValue;
  26.  
  27.     foreach($points as $point)
  28.     {
  29.         $pointStr .= (int) ($point->$method() * $scale) . ',';
  30.     }
  31.  
  32.     $pointStr = substr($pointStr, 0, -1);
  33.  
  34.     // labels (5)
  35.     $labels = array();
  36.  
  37.     $steps = 4;
  38.     $interval = $maxValue / $steps;
  39.  
  40.     for($i = 0; $i < $steps; $i++)
  41.     {
  42.         $labels[] = (int) ($i * $interval);
  43.     }
  44.  
  45.     $labels[] = (int) $maxValue;
  46.  
  47.     return 'http://chart.apis.google.com/chart?cht=lc&amp;chd=t:' . $pointStr . '&amp;chs=' . $size . '&amp;chxt=y&amp;chxl=0:|' . join('|', $labels);
  48. }

The function does not support the short version of the Google Chart API Just Yet ™ as it is an simple proof of concept hack made a few months ago.

]]> http://e-mats.org/2010/01/a-simple-smarty-modifier-to-generate-a-chart-through-google-chart-api/feed/ 0