Followup on The Missing Statistics in OpenX

After my previous post about the missing OpenX statistics because of crashed MySQL-tables, I got a very nice and helpful comment from one of the OpenX developers. To put it one single word: awesome. If you’re ever going to run a company and have to look after your customers (even if you release your project as open source), simply do that. People will feel that someone are looking out for them.

Anyways, as promised, this were supposed to be a follow up. We didn’t manage to get the impressions statistics back, but the missing clicks returned after repairing the tables. The tip from Arlen didn’t help either, but I have a few suggestions for how to make the script easier to use.

I were kind of perplexed about how I could give the dates for the time interval it was going to rebuild the statistics. The trick was to change two define()-s in the top of the code. Not very user friendly, so I made a small change to use $argc and $argv instead. That way I could do:

  1.     php regenerateAdServerStatistics.php "2008-06-01 10:00:00" "2008-06-01 10:59:59"

instead of having to edit the file and changing the defines every time. After doing this simple change, I could also write a small helper script that ran the regenerateAdServerStatistics.php file for all the operation intervals within the larger interval (an operation interval is an hour, while my interval were several days).

So, here it is, regenerateForPeriod.php:

  1. <?php
  2.     if (empty($argc) || ($argc < 3))
  3.     {
  4.         exit("Usage: regenerateForPeriod.php <startdate> <enddate>");
  5.     }
  6.  
  7.     $start = $argv[1];
  8.     $end = $argv[2];
  9.  
  10.     $start_ts = strtotime($start);
  11.     $end_ts = strtotime($end);
  12.  
  13.     if (!$start_ts || !$end_ts || ($start_ts >= $end_ts))
  14.     {
  15.         exit("Invalid dates.");
  16.     }
  17.  
  18.     $current_ts = mktime(date('H', $start_ts), 0, 0, date('m', $start_ts), date('d', $start_ts), date('y', $start_ts));
  19.  
  20.     while($current_ts < $end_ts)
  21.     {
  22.         system('php regenerateAdServerStatistics.php "' . date('Y-m-d H', $current_ts) . ':00:00" "' . date('Y-m-d H', $current_ts) . ':59:59"');
  23.         $current_ts += 3600;
  24.     }
  25. ?>

This runs the renegerateAdServerStatistics.php script for each operation interval. If your ad server uses a larger interval than 3600 seconds, change the setting to a more appropriate value. Before doing this, you’ll want to remove the sleep(10) and the warning in regenerateAdServerStatistics.php, so that you don’t have to wait 10 seconds for each invocation of the script. I removed the warning and sleep altogheter, but hopefully someone will commit a command line parameter to regenerateAdServerStatistics.php that removes the delay. I didn’t have time to clean up the code and submit an official patch today, but if there is interest, leave a comment and I’ll consider it.

Tags: , ,

11 Responses to “Followup on The Missing Statistics in OpenX”

  1. Cedders Says:

    Thanks for this. Probable small error in the code:

    date(‘h’) should presumably be date(‘h’, $start_ts)

  2. Mats Says:

    Good catch. It’s a while since I wrote this, but I think you’re right. I’ve updated the code above to reflect this. Thanks!

  3. kolio Says:

    Hi, I don’t understand where to create this file, you have to insert some code?

  4. Mats Says:

    The file was created in the same directory as regenerateAdServerStatistics.php which is part of the standard OpenX distribution.

    If you want to keep the file in another location you have to change the path in the system() call.

  5. Mattias Says:

    date(‘h’, $start_ts) should be date(‘H’, $start_ts) ?

  6. Mats Says:

    Yes, thanks for spotting that. I’ve updated the code.

  7. Mattias Says:

    Running the script for 95 000 000 rows in phpads_data_raw_ad_impression takes a while… =)

  8. Kris Noble Says:

    Neat solution – you should definitely submit to the OpenX team for consideration if it’s not already been done, the frequency of table crashes with OpenX/MySQL makes it worthwhile having this tool to hand so that recovery is as quick as possible..

  9. Mats Says:

    Haven’t submitted any patches to OpenX, but if I get some spare time I’ll try to get one together later. Feel free to submit the change and claim the credit, tho. :-)

  10. POed Says:

    It is a piece of crap. Has been, is, and will be. These guys operate in some kind of geek fog with no perception of the end user. They don’t even realize that Openx is pure feces. Time is of no value to them, because it is our time. Better scripts come from high school kids.

  11. Oliver Says:

    Saved my Weekend!

Leave a Reply