«
»

Escaping Characters in a Solr Query / Solr URL

We’re using our own Solr library at Derdubor at the moment, but we’ve only been using it for indexing content. The query part was never standardized in our common library as we usually used an alternative output format, but during the last days that has changed. We now have a parser for the default XML outputter and we’re also supporting facets and field queries (or constraints as they’re abstracted as in our library).

This means that we’re feeding content into the query that may contain foreign characters, in particular those who have special meaning in a Solr query. You can find the complete list of characters that need to be escaped in a SOLR or Lucene query in the Lucene manual.

To escape the characters we use this very simple and stupid PHP method:

  1.     static public function escapeSolrValue($string)
  2.     {
  3.         $match = array('\\', '+', '-', '&', '|', '!', '(', ')', '{', '}', '[', ']', '^', '~', '*', '?', ':', '"', ';', ' ');
  4.         $replace = array('\\\\', '\\+', '\\-', '\\&', '\\|', '\\!', '\\(', '\\)', '\\{', '\\}', '\\[', '\\]', '\\^', '\\~', '\\*', '\\?', '\\:', '\\"', '\\;', '\\ ');
  5.         $string = str_replace($match, $replace, $string);
  6.  
  7.         return $string;
  8.     }

We used a regular expression first, but the sheer amount of backslashes made it a regular .. hell … to read. So to make it easier for the persons maintaining this in the future, we went the easy to read / easy to maintain road for this one.

Tags: , , , ,

This entry was posted on Wednesday, January 20th, 2010 at 11:00 and is filed under PHP, Solr. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

6 Responses to “Escaping Characters in a Solr Query / Solr URL”

  1. Shalin Shekhar Mangar Says:
    January 22nd, 2010 at 22:04

    You can verify this function against Solr’s Java client ClientUtils.escapeQueryChars. See http://svn.apache.org/repos/asf/lucene/solr/trunk/src/solrj/org/apache/solr/client/solrj/util/ClientUtils.java

  2. Mats Says:
    January 23rd, 2010 at 23:44

    Good point. I’ve used SolrJ quite a bit before, but I never thought about validating it against the same behaviour. SolrJ also escapes ” and ; which were missing from my list. I’ve added them now.

    Thanks for the update!

  3. Avoid Escaping Spaces in the Query String in a Solr Query | Mats Lindh Says:
    January 30th, 2010 at 23:24

    [...] up on the previous post about escaping values in a Solr query string, it’s important to note that you should not escape spaces in the query itself. The reason for [...]

  4. Satish Says:
    November 5th, 2010 at 18:19

    ClientUtils class is escaping the space also. For example:

    Input: hello there
    Expected: hello there
    Actual: hello\ there

    This is giving problem as the final string will become as hello\+there when sent over HTTP.

    Regards,
    Satish.

  5. binit Says:
    August 18th, 2011 at 08:10

    Very helpful post.
    @Shalin Shekhar Mangar Clientutils was good food too ;)

  6. Alex Says:
    December 7th, 2011 at 14:41

    This also gives bad errors for date facets

    For example ..

    Your method turns the query into this…
    http://localhost:8080/test/select?q=fqdn\:b\*&facet=on&facet.date.start=NOW&facet.date.end=2012\-02\-05T13\:37\:29\+00\:00Z&facet.date=ending&facet.date.gap=\+7DAY&rows=25&wt=json

    Making SOLR do this in the error log!

    INFO: [] webapp=/test path=/select params={facet.date.start=NOW&facet=on&q=fqdn\:b\*&facet.date=ending&facet.date.gap=\+7DAY&wt=json&facet.date.end=2012\-02\-05T13\:36\:21\+00\:00Z&rows=25} hits=0 status=400 QTime=1
    07-Dec-2011 13:37:29 org.apache.solr.common.SolrException log
    SEVERE: org.apache.solr.common.SolrException: date facet ‘end’ is not a valid Date string: 2012\-02\-05T13\:37\:29\ 00\:00Z

Leave a Reply