The Countable Interface

The Countable interface at it’s simplest level allows you to pass objects that implement it to PHP’s native count function. The interface is, on the surface, extremely easy to implement. You only need to add the Countable::count() method to implement the interface in your class and this method only needs to return an integer. If you pass an object that implements Countable to the PHP count function the PHP interpreter will automatically call the count method in the object, returning the value from it to you to use.

Why is Countable so Useful?

Like all of the predefined interfaces and SPL functionality the real power of countable comes from the ability to effectively override and re-implement core PHP functionality with your own logic. I’ll give three brief examples here of ways that I’ve personally used Countable recently.

Accessing the State of a Private or Protected Property

One of the simplest uses of countable is to return the count of an array that’s held as a protected or private property of an object. Take the following example:

class SomeClass implements Countable {

 protected $_data = array();

 public function doSomething()
 {
//Code to change $_data here
 }

 public function count()
 {
return count($this->_data);
 }
}

In this example some data is stored in the object as an array and the Countable:Count method simply returns the count of that array. While this is obviously a very simple example it can be very powerful. I’ve been doing a lot of work with a web service in the last few weeks and have used code very much like this. After a web service call the resulting XML is processed using XPath and the resulting array is stored in a protected property. The count method would then return the number of results of the web services call. Using count in this way obviously makes a lot of sense when coupled with implementing Iterator or ArrayAccess to allow access to the data held in the protected property.

Using Countable to Signal the State of an Object

Countable can also be used to signal the state of an object. Imagine the above code, amended slightly to this:

class SomeOtherClass implements Countable {

protected $_data;

public function doSomething()
{
//Code to change $_data here
}

public function count()
{
return is_null($this->_data) ? 0 : 1;
}
}

In this case count will only ever return 0 or 1, depending on whether the protected property $_data has a value set in it or not. Using this code could look something like this:

$obj = new SomeOtherClass();

$obj->doSomething();

if (count($obj)) {

//$_data is not null, proceed accordingly.

}

I’ve also used this recently with a web service call. In this case the web service call was expected to return a single value, which is stored in $_data. Implementing countable like this simply signifies to the calling code if a value has been successfully retrieved or not.

Using Countable with a Database Table

Another interesting use of Countable is with a class that maps to a database table in some way. Imagine a case where you have a database table where values are only considered ‘live’ if some condition is met. Implementing countable on an object that encapsulates the db table can then be used to flag if the table contains any ‘live’ values. The following example uses Zend_Db_Table:

class AnotherClass extends Zend_Db_Table implements Countable {

 public function count()
 {
$select = $this->select(Zend_Db_Table::SELECT_WITH_FROM_PART)
->from('some_table', array('Count' => 'COUNT(*)'))
->where('SomeCondition = SomeValue');
$row = $this->fetchRow($select);
return (int) $row->Count;
 }
}

In your calling code you would then simply do this:

$obj = new AnotherClass();

if (count($obj)) {

//Table has 'live' values. Do something here

}

This can be a very useful way of working with a database table.

A Word of Caution…

The only limit to the way Countable can be used is the imagination of the developer, and this is also a danger. All developers are familiar with using the count() function to get hold of the number of values in an array. When implementing Countable the further you move the count() method away from this functionality the less intuitive the code becomes and the greater the chance that the developer using it might make a mistake. This is true of all of the predefined interfaces however. While they provide great power the further away from core PHP functionality you get while implementing them in your own code the less intuitive they become. However, don’t let this stop you! The next time you go to write a method called something like ‘getItemCount()’ in a class don’t do it! Implement the Countable interface instead and start hooking into the power of the core PHP engine.

API Format

The SalesNet API is SOAP based and consists of 8 separate endpoints, each of which allows you to operate on different parts of a companies data such as deals, accounts, contacts, etc. Before you can use any of these services it’s necessary to authenticate against a separate security endpoint. This returns a token which is valid for between 1-12 hours (depending on the account setup in SalesNet). This token must then be set in a SOAP header for use in requests to any of the main SalesNet endpoints. Each of the API methods accepts many arguments to control precisely the data that is returned and the only thing I can suggest to anyone looking to use the API is to refer to the documentation. One of the quirks of the API is that data is returned as XML, embedded in a stdClass object. For example, a call to the GetDeals method of the deals endpoint returns a stdClass object (say $obj), and to access the response data the following would be needed:

$response = $obj->GetDealsResult->any;

In the above example $response would now include the XML data ‘payload’, which can then be parsed. It’s often not that easy to parse the XML returned as it seems to be an XML representation of a Microsoft ADODB recordset, which in some cases is not even properly formed XML. I’ve had luck using SimpleXML and Xpath to extract the nodes I’m after however.

The Code

To work with the various SalesNet API’s I created two classes, one of which handles authentication and the other of which acts as a proxy to catch and pass SOAP calls through to the API. Both of the classes make use of ‘lazy loading’ and will only connect to SalesNet as and when a method is called that requires a SOAP call.

The Authentication Class

My goal in the authentication class was to simplify authenticating against the SalesNet security endpoint. The object accepts a company name, username and password and will retrieve a security token from SalesNet. This is then set in a PHP SoapHeader object and is returned for use whenever the doLogin() method is called. Internally the class will keep track of when the token expires and will automatically fetch a new token from SalesNet as and when required. My thought is that an object of this class can be serialized and cached to be stored and shared between many requests as a security token can be valid for up to 12 hours.

The SoapProxy Class

The SoapProxy class provides a proxy to trap calls to the many SalesNet API methods. Since there are so many methods cross so many endpoints it would be impossible to map these out to individual methods so I chose to use __call() instead to pass calls through to a PHP SoapClient object. The class stores the URL endpoints of the various services as constants and an API endpoint can be set when an object is instantiated or before any method is called. The object also accepts an instance of the Authentication class and uses this internally to set the security SoapHeader for API calls.

A Brief Example

So, enough talk! As I said earlier full code can be found on GitHub but here’s a brief example of how to use the classes to call the GetDeals method of the Deals endpoint.

<?php
use WebServices\Soap\SalesNet\Authentication, WebServices\Soap\SalesNet\SoapProxy;

try {
    $auth = new Authentication();
    $auth->setCompanyLogin('COMPANY LOGIN HERE')
         ->setUserName('USERNAME HERE')
         ->setPassword('PASSWORD HERE');

    $soap = new SoapProxy(SoapProxy::DEALS, $auth);
    $args = array(
        //Set up method arguments here
    );
    $response = $soap->GetDeals($args);
} catch (BadMethodCallException $e) {
    /**
     * Errors will be caught here.
     * The BadMethodCallException can have a previous exception of a SoapFault if there is one.
     */
}

That’s all there is to it. To use other endpoints simply change the constant passed to the SoapProxy constructor and call another Soap method, with the correct arguments of course. Let me know of any comments on this code or improvements people can see. Enhancements and fixes on GitHub are welcome.

Google Site Search

Google Site Search can be used to create a customised search engine for your website. The costs start at $100 per year (at the time of writing) for a website of up to 1,000 pages and this allows 250,000 searches per year. The search engine can be queried in a number of ways, one of which is via a REST web service. The service seems to be ideal for small to medium websites. Larger websites would probably benefit from using a service such as Solr or Lucene to create a search index on the sites’ web server, which may well provide a more customisable solution. For sites where it’s not possible to create such an index, or the owner doesn’t want to go to the effort, Google Site Search seems to be ideal.

The Google Custom Search Class

The class I’ve written can be downloaded from PHPClasses along with a file giving some examples of it’s use. The class can use cURL or the PHP HTTP stream wrapper to perform the web service request, with the constructor deciding on which method to use based on what the server has installed. The response from Google, consisting of search results and optional spelling suggestions, is processed using SimpleXML into arrays which are set as properties of the object. The search results XML contains the results as HTML and the class allows a user to specify how this should be handled. The user can ask for the results to be made available as plain text (no HTML and no entities encoded), plain text with all HTML special characters entity encoded, or with the HTML that Google returns left in. The user can specify which character encoding, defaulting to ISO-8859-1, a search query uses and results are returned in the same encoding. The Google API also allows many parameters to be set in the request to the service (see http://www.google.com/cse/docs/resultsxml.html#wsRequestParameters for full documentation). The class allows an array of key => value pairs to be passed which are then set as query string parameters, allowing the search to be further customised. Finally, any errors generated in the process cause a RuntimeException to be thrown which has to be caught outside of the object. Below are some examples of using the class.

try {
		//A simple example, default parameters in constructor and no extra parameters set.
		$search = new GoogleCustomSearch('CSE Key Here');
		$search->query('some search text');
		//The results property contains the search results, spellingSuggestions contains any spelling suggestions from Google.
		var_dump($search->results);
		var_dump($search->spellingSuggestions);

		//Example with results returned with HTML from Google left in and character encoding set to UTF-16. Result text will also be encoded in UTF-16
		//NOTE: When using a custom character encoding the search query must also be encoded using this character set.
		//The object defaults to character encoding in iso-8859-1 and assumes that queries passed to it are also encoded in this character set.
		$search = new GoogleCustomSearch('CSE Key Here', 'utf-16', GoogleCustomSearch::HTML);
		$search->query('some search text');
		var_dump($search->results);
		var_dump($search->spellingSuggestions);

		//Use the same object to perform an extra search, but with some different parameters
		$search->charEncoding = 'iso-8859-1';
		$search->processText = GoogleCustomSearch::ENTITIES_ENCODED; //Strips HTML but leaves HTML special characters entity encoded.
		$search->query('another query');
		var_dump($search->results);
		var_dump($search->spellingSuggestions);

		//Example using an array to set extra, custom options in the query string used to send a request to Google.
		//See http://www.google.com/cse/docs/resultsxml.html#WebSearch_Query_Parameter_Definitions for a full list of query parameter options that may be passed in the request.
		$opts = array(
			'lr' => 'lang_fr', //Request search results only for French language pages.
			'cr' => 'countryCA', //Request search results only for a particular country, in this case Canada
			'num' => 7 //Limit the search to a maximum of 7 results
		);
		//Create an object using default search options
		$search = new GoogleCustomSearch('CSE Key Here');
		//Set the custom options for the search.
		//These can also be set by passing the $opts array as the final argument in the object constructor.
		$search->opts = $opts;
		$search->query('yet another query');
		var_dump($search->results);
		var_dump($search->spellingSuggestions);
	}
	catch (RuntimeException $e) {
		//Handle any exceptions raised by the object here.
		echo $e->getMessage();
	}

I hope somebody finds the class to be useful. It’s still a version 1 release and I can see two major ways that it could be improved. Firstly, the handling of error response from the API is a little basic. I need to do some more research into error response codes that the API generates and use that to make the error messages set in any RuntimeException a little more specific. Secondly the response XML contains much more information than I am currently processing. I could make much more of it available as properties of the object and/or make the XML returned from the service available as a property of the object to allow a user to process it in any way they wish. If anyone has any problems, comments or suggestions post them below and I’ll get back to you.

What is LiveDocx?

The Live Docx service allows a programmer to take a document template and through a SOAP based web service populate it with content. The resulting document can then be retrieved to the web server in one of many common formats including PDF, MS Word .doc and .docx and RTF. Best of all the service is completely free at the lowest level, but depending on the requirements of your application it may be more appropriate to use one of the paid for levels of service. The templates can be created in .doc, .docx, .rtf and .txd formats and use mail merge to populate the content. The great advantage is that you can use all of the features of a word processing program to design your document, ending up with a fully professional quality document. Best of all you don’t even need to write any code to access the web service. The latest release of the Zend Framework (1.10) includes a component, Zend_Service_LiveDocx, to access and work with the service and there’s also a .NET library to download for Microsoft developers.

Creating Templates

As mentioned earlier the LiveDocx service uses mail merge fields to perform its magic. You create a document that contains one or more merge fields and then assign content to these fields programmatically at run time. The LiveDocx servers replace the mail merge fields with the content in the same way as a word processing program would, making the completed document available to you. Templates can either be stored directly on the LiveDocx servers or on your own webserver. The SOAP service allows to you specify the name of the template you want to use and whether it is stored locally or remotely. Which one is  chosen will largely depend on the application but storing templates on your own web server does mean that the template must be sent to LiveDocx along with the content before the completed document can be retrieved. This obviously significantly increases the amount of information that needs to be transmitted over the internet but the demands of your application may dictate that templates be stored locally.

Using Zend_Service_LiveDocx

The Zend Framework LiveDocx component is very easy to use and complete documentation can be found here. The Zend Framework download also includes several examples of using the service. Basically you instantiate a Zend_Service_LiveDocx_MailMerge object, setting your username, password and the template you wish to use, before assigning content to the object in much the same way that you would with Smarty. You then call a createDocument() method to produce the document on the remote server followed by a retrieveDocument() method to get the document produced back as a string. The latter method takes an argument which specifies the type of file that should be created. That’s basically it. The code below is a simple view class that I wrote for my application that uses the Zend_Service_LiveDocx to produce a document.

<?php
 /**
 * View for the download plan functionality. Uses the LiveDocx web service and the ZendFramework LiveDocx SOAP integration to produce a file.
 * @author Jeremy Cook
 * @version 1.0
 * @see http://framework.zend.com/manual/en/zend.service.livedocx.html
 */
 class planView {
 /**
 * Array of content to be used in the plan
 *
 * @var array
 */
 protected $content = array();
 /**
 * The type of file to be requested from LiveDocx
 *
 * @var string
 */
 protected $fileType;
 /**
 * Name of the template on LiveDocx to use
 *
 * @var string
 */
 protected $template;
 /**
 * Username for LiveDocx
 *
 * @var string
 */
 protected $username;
 /**
 * Password for LiveDocx
 *
 * @var string
 */
 protected $password;
 /**
 * Name to give the downloaded file
 *
 * @var string
 */
 protected $filename;
 /**
 * Constructor
 *
 * @param array $content
 * @param string $fileType Type of file to be requested from LiveDocx
 * @param string $template Name of the remote template to use on LiveDocx
 * @param string $filename Name to use for the file produced by LiveDocx
 * @param string $username
 * @param string $password
 * @return planView
 */
 public function __construct (array $content, $fileType, $template, $filename, $username, $password) {
    $this->content = $content;
    $this->fileType = $fileType;
    $this->template = $template;
    $this->filename = $filename;
    $this->username = $username;
    $this->password = $password;
 }
 /**
 * Method to produce the plan document
 *
 * @throws Zend_Service_LiveDocx_Exception
 * @throws RuntimeException
 */
 public function getPlan () {
    require_once 'Zend/Service/LiveDocx/MailMerge.php';
    $doc = new Zend_Service_LiveDocx_MailMerge();
    $doc->setUsername($this->username)->setPassword($this->password);
    $doc->setRemoteTemplate($this->template);
    foreach ($this->content as $key => $value) {
       $doc->assign($key, $value);
    }
    $doc->createDocument();
    $document = $doc->retrieveDocument($this->fileType);
    unset($doc);
    $output = fopen('php://output', 'w');
    if (!$output)
       throw new RuntimeException('Unable to bind PHP output as a file handle');
    switch ($this->fileType) {
       case 'pdf':
          header ('Content-Type: application/pdf');
          break;
       case 'doc':
       case 'docx':
          header ('Content-Type: application/vnd.ms-word');
          break;
    }
    $filename = str_replace(' ', '_', $this->filename) . ".{$this->fileType}";
    header ("Content-Disposition: attachment; filename=\"$filename\"");

    //Make sure the browser doesn't cache the file
    //Set headers to ensure the document is not cached.
    header ("Expires: Mon, 26 Jul 1990 05:00:00 GMT");
    header ("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
    header ("Cache-Control: no-cache, must-revalidate");
    header ("Pragma: no-cache");
    //Write the document to php output
    fwrite($output, $document);
    fclose($output);
 }
 }
?>

Some ‘Gotchas’

The two main problems I had in using the LiveDocx service had nothing to do with LiveDocx or Zend_Service_LiveDocx (which has great documentation). The problems I had were in making MS Word do what I wanted in the template creation. Admittedly this may have more to do with the fact that I’d never created a mail merge template before but I thought I’d mention the two issues here. The issues were in adding merge fields to the document and creating bookmarks.

Adding Merge Fields

Merge fields are denoted by the text {MERGEFIELD fieldName} (where ‘fieldName’ is the unique name of the field). What the documentation doesn’t say however is that the curly braces cannot be simply typed on the keyboard. In Word you need to enter ctrl + F9 to get a special set of merge field curly braces. In Word 2007 you can also go to the insert tab then Quick Parts, [field] and select merge field. Add a name for the merge field and it is inserted into the document where your cursor is placed. This second method makes the field look like «fieldName» but this is completely equivalent to the first method with the curly braces.

Adding Bookmarks

With LiveDocx a bookmarked section allows you to iterate over a section of a template, assigning content from an array to merge fields inside the bookmark on each iteration. This enables you to produce sections of documents that have the same structure but different content. This is obviously very handy as varying amounts of content can be assigned to the section of the template. One of the examples in the Zend Framework utilises this to produce a telephone bill where a varying number of calls can be assigned to a bookmarked section, producing a table of all the calls in the final document. The array has to be multi-dimensional, with the top level array being numeric and each nested array being associative where the key is the name of the merge field and the value is the content to be inserted. What the documentation doesn’t tell you is how to create the bookmarks (there’s supposed to be a screenshot in the documentation which is missing) and I spent a few frustrating hours before I finally found out how to do it. Here are the steps.

1. Go to the insert tab in Word and select bookmark.
2. Create a new bookmark that begins with blockStart_ (ie. blockStart_bookmarkName).
3. The bookmark will be invisible in the document but go ahead and add your merge fields with whatever formatting you like.
4. After the last merge field for the section add another bookmark but start it with blockEnd_ (ie. blockEnd_bookmarkName).
5. When you assign content to the bookmarked section make sure that the array is assigned to the name of the bookmark, like the following:

<?php

//Assume $doc is a Zend_Service_LiveDocx_MailMerge object and $array is the multidimensional array to be assigned to the bookmark.

$doc->assign('bookmarkName', $array);

?>

Conclusion

I hope you find this brief introduction to LiveDocx useful. I’ve found it to be a great service that’s easy to use and I’ll certainly use it again the next time I need to produce professional quality documents from within PHP.

To display the local store information I needed to go through a number of distinct steps in the code.

1. Determine the visitors location from their IP address.
2. If the visitor is in Canada calculate the distance to their closest store.
3. If they’re within 100km of a store display information on that store on the homepage.
4. If the user is not in Canada, is more than 100km away from a store or if there is any kind of error display a generic message about store locations.

Getting the Users Location

Determining a visitors location from their IP address can be done through a number of GeoIp services, some available for free and some paid for. Ideally I wanted to use a PECL PHP extension to do the location lookup but this was not possible as I could not persuade my web host to install this. I fell back to using a free web service provided by ipinfodb for the lookup. I found a class on PHPClasses that used this web service, which I substantially rewrote to serve my purposes. The code that performs the lookup in the class is below:

$strAPIURL = $this->apiUrl . "?ip=" . urlencode ($this->remoteAddress);
 //Make a call to the api and fetch the result.
 $ch        = curl_init ($strAPIURL);
 curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
 $xmlResult = curl_exec ($ch);
 curl_close ($ch);

 if ($xmlResult && strlen ($xmlResult) > 2) {
 //Process the result
 $result = simplexml_load_string ($xmlResult);

 if ((string)$result->Status == 'OK') {
 $this->countryName   = (string)$result->CountryName;
 $this->countryCode   = (string)$result->CountryCode;
 $this->cityName      = (string)$result->City;
 $this->zipPostalCode = (string)$result->ZipPostalCode;
 $this->regionName    = (string)$result->RegionName;
 $this->regionCode    = (string)$result->RegionCode;
 $this->timezone      = (int)$result->Timezone;
 $this->gmtOffset     = (int)$result->Gmtoffset;
 $this->dstOffset     = (int)$result->Dstoffset;
 $this->lat           = (float)$result->Latitude;
 $this->long          = (float)$result->Longitude;
 //Set the ipSniffed flag to true
 $this->ipSniffed     = true;
 }
 }

Once the call to the web service has been completed the information returned is stored in public properties and a flag is set to show that a successful lookup was performed. My class uses filter_var() to make sure a valid IP address that is not in a private range is passed to it, throwing an InvalidArgument Exception if not. Of course, if an exception is thrown I fall back to my default position of displaying a generic message about store locations.

Finding the Closest Store

Once I have the users location from their IP address and I’ve determined that they’re visiting from Canada I proceed to calculate the distance to their closest store. This is done with a database query. Information about all of the stores is stored in a database table along with the latitude and longitude of each store. Using the latitude and longitude returned from the web service I can then calculate the users distance to the nearest store. This is done in the following code:

protected function getClosestStore ($lat, $long) {
 $sql = <<< _SQL_

SELECT StoreId, CONCAT_WS(', ', Address, City, CONCAT(Province, ' ', PostCode)) AS Address, Lat, Lon, ROUND(6371 * 2 * ASIN(SQRT(POWER(SIN((:lat - abs(lat)) * pi()/180 / 2), 2) +
COS(:lat * pi()/180 ) * COS(abs(lat) * pi()/180) *  POWER(SIN((:lon - lon) * pi()/180 / 2), 2) )),2)
AS Distance
FROM Stores
HAVING Distance <= 100
ORDER BY Distance Limit 1
_SQL_;
 try {
 $db = db::getConn();
 $st = $db->prepare($sql);
 $st->execute(array(':lat' => $lat, ':lon' => $long));
 $this->storeInfo = $st->fetch(PDO::FETCH_ASSOC);
 }
 catch (PDOException $e) {
 error_log($e->getMessage());
 }
 }

This PHP in this code simply connects to the database (db::getConn() is a static method that returns a singleton instance of a PDO object), performs the query and stores the result in the storeInfo property. If no result is returned from the database the fetch() method will return false, which then tells me that a user is not within 100 km of a store. The real meat of this code is in the SQL query. It selects some information about the store and then uses some math to calculate the distance to the nearest store. This is done using the Haversine formula, which calculates the distance between two sets of latitude and longitude, taking into account the curvature of the earth. I’m not going to try to explain the math (mostly because I don’t fully understand it myself!) but there is a Wikipedia article on the formula for anyone who is interested. The query then limits the results to stores that are within 100km, orders the results by distance to make sure the closest one is listed first and then returns the first result. If a result is returned I then record the StoreId of the closest store in a cookie, which is then used when the visitor visits the site again. This is to cut down on processing time for subsequent request for the home page and means that I won’t need to hit the ipinfodb web service on every request for the index page of the site.

Limitations of this Solution

There are three major limitations to this solution that I can see: the accuracy of GeoIP services, the database query and saving the local store information in a cookie.

GeoIP services claim about an 80% accuracy when tracking the geographic location of an IP address down to the city level (accuracy is about 95% for finding the country a user is visiting from). For example I am writing this in Guelph, Ontario but the IP address I am connected to the internet with resolves geographically to the nearby city of Kitchener. This is not a major problem for my application as the client only has 35 stores across Canada and I am just trying to find the closest one to a user. Given the small number of stores chances are that I will hit on the closest one, even allowing for only 80% accuracy in GeoIP tracking. For other applications this could be a problem. The W3C has a draft Geolocation API which some browsers (such as Firefox) are starting to implement. Using this (perhaps with some AJAX) could help to get a more accurate fix on some users locations but would open up some privacy concerns. For this application I display links that allow a user to manually select their closest store if the GeoIP tracking is wrong, hopefully alleviating any problems caused by the 80% accuracy.

The SQL query as I’ve written it would have performance issues if a large number of locations were being stored. To calculate the distance the query has to find the distance to every store held in the database, only then narrowing it down to the closest. This is not a problem currently as the client only has 35 locations, but if hundreds or thousands of locations were being stored the query would be extremely innefficient and take a long time to execute. In the course of my research I did read something about limiting the search to a radius (100 km in this case). This would involve several queries and probably a stored procedure to carry them out in. As I was dealing with a small number of locations I decided against this for simplicity. For another solution involving more locations I would probably go with this approach.

Saving the information on the users local store in a cookie makes the application more efficient but it potentially slightly degrades the user experience. In a scenario where a user travels and expects to see information on the closest store to where they currently are this would not work. I made a judgement call here and decided that it was better to go with the more efficient approach for this client, but in other cases this may not be so. Of course, if I could install the PECL GeoIP extension I wouldn’t need to store the cookie. Looking up a users location using this extension would be no more complicated than calling some PHP functions. The GeoIP information would be stored locally as part of the extension. This would be quicker than creating a call to and processing a response from a web service. Unfortunately this was not an option for me on this occasion. This problem can be overcome by the user manually selecting their closest store, overriding the mechanism I programmed.

All in all I am fairly happy with the solution I arrived at for this problem. It enables me to tailor content on the page based on where a user is. My example is a fairly simple one but it would be possible to do far more sophisticated things using GeoIP such as automatically displaying content in different languages depending on which country a visitor is coming from or displaying prices in different currencies. Due to the less than 100% accuracy of GeoIP mechanisms would always need to be provided to allow a user to override the conclusions found by the code though.

Thankfully all of the main social networks provide web service API’s and using these it’s possible to track just about any metric that you would like. As an example I’ve put together a short (about 100 lines including some nice formatting and comments) script that carries out a search on Twitter, using the Twitter search API, and then extracts keywords from each tweet returned, using the Yahoo term extraction API. The results are sent to the user as a CSV file containing columns with the date the tweet was posted, the author, the tweet text and the keywords extracted for that tweet. Popular keywords are then presented at the bottom of the file. This is by no means production ready code but is really a quick and dirty test to show the kind of thing that is possible. Here’s a brief look at how it works.

require 'termExtractor.php';

try {
 //Perform the initial search on Twitter. Also set the number of tweets to return as the maximum allowed per page (100)
 $search = '"Demo Camp Guelph" OR #dcg';
 $query  = http_build_query(array (
 'q'     => $search,
 'lang'  => 'en',
 'rpp'   => 100,
 'since' => date('Y-m-d', strtotime('5 days ago'))
 ));
 $url    = 'http://search.twitter.com/search.json?';
 $ch     = curl_init($url . $query);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 $tweets = json_decode(curl_exec($ch));

The beginning of the script includes a class that I wrote to handle working with the Yahoo term extraction API. It then sets up a few variables and carries out an initial search using the Twitter API. I’ve chosen as my search query text relating to a meeting of developers that took place in Guelph last week. Documentation on the Twitter search API is pretty good and includes information on the format of the search terms, including modifiers that are allowed. My search looks for any tweets posted in the last five days with the text ‘Demo Camp Guelph’ or the hashtag ‘#dcg’. It also sets the number of results to return to 100. This is the maximum number that can be returned from a request but Twitter provides a mechanism whereby searches with more than 100 results can be paged over.

//Open the PHP output as a file handle
 $output = fopen('php://output', 'w');
 if (!$output)
 throw new Exception('Unable to bind PHP output as a file handle');
 //Set headers to tell the browser a csv attachement is coming. Also set cache control to no-cache to ensure browser does not cache the results file.
 header ('Content-Type: application/csv');
 header ('Content-Disposition: attachment; filename="Twitter_Search.csv"');
 header ('Cache-Control: no-cache, must-revalidate');
 //Add headings for the columns
 fputcsv($output, array (
 'Posted On',
 'Author',
 'Tweet',
 'Keywords'
 ));
 if (!isset($tweets->results) || !count($tweets->results)) {
 //No search results. Output an error message and exit
 fputcsv($output, array('Sorry, there are no results that match that search'));
 curl_close ($ch);
 fclose ($output);
 exit;
 }

This code binds the output of the script to a file handle, allowing me to present the output of the script as a file to the browser. HTTP headers are also set to tell the browser to expect a csv file and that it should be treated as an attachment. Some headings for columns are then written and if there are no search results the script outputs a message and exits.

//Set up the termExtractor object, an array to hold the keywords results and a flag to show that we're in the first iteration of the do while loop
 $keys = new termExtractor(YAHOO_API_KEY);
 $keywords_array = array();
 $first = true;
 //Loop over the results
 do {
 if (!$first) {
 //Break the query string for the next page into its' parts. Use substr() to remove the initial ?
 parse_str (substr($tweets->next_page, 1));
 $query = http_build_query(array (
 'q'      => $q,
 'page'   => $page,
 'max_id' => $max_id,
 'lang'   => 'en',
 'rpp'    => 100
 ));
 curl_setopt($ch, CURLOPT_URL, $url . $query);
 $tweets = json_decode(curl_exec($ch));
 }
 foreach ($tweets->results as $result) {
 //Result text from Twitter is utf-8 encoded and htmlentity encoded.
 $result->text = html_entity_decode(utf8_decode($result->text));
 $keywords     = $keys->doTextQuery($result->text);
 //If a single keyword was returned turn it into an array to save repetition of code.
 if (is_string($keywords) && strlen($keywords)) {
 $keywords = array($keywords);
 }
 if (is_array($keywords)) {
 foreach ($keywords as $word) {
 //Check that the keyword does not appear in the initial search term.
 switch (!stristr($search, $word)) {
 case true:
 //If the keyword has already been set in the array increment the count by 1
 if (array_key_exists($word, $keywords_array)) {
 $keywords_array[$word]++;
 } else {
 $keywords_array[$word] = 1;
 }
 break;
 default:
 continue;
 }
 }
 $keywords = implode(', ', $keywords);
 }
 fputcsv($output, array (
 $result->created_at,
 $result->from_user,
 $result->text,
 $keywords
 ));
 }
 $first = false;
 } while (isset($tweets->next_page));

This code sets up the term extraction object and then loops over the results while there is a next_page available. Keywords are output to the CSV file alongside each tweet and also stored in an array for later use. Keywords which appear in the original search query are filtered out of the results array and using a do while loop ensures that the first set of search results are output even if there are no other results after that.

//Destroying the $keys object triggers the destructor which disposes of the cURL resource in the object
$keys = NULL;
//Free up the cURL resource
 curl_close ($ch);
 fputcsv($output, array('Keywords for this search:'));
 fputcsv($output, array (
 'Phrase',
 'Number of Occurences'
 ));
 foreach ($keywords_array as $word => $count) {
 if ($count >= 3) {
 fputcsv($output, array (
 $word,
 $count
 ));
 }
 }
 fclose ($output);
} catch (Exception $e) {
 echo $e->getTraceAsString();
}

The final part of the code processes the stored keywords and outputs any keyword that appears three times or more in the tweets that have been found.

The code here is obviously not optimal but is merely meant as an example of what is possible. It would be very easy to abstract the logic out of this script into a Twitter search class, making the code portable and reusable. The code also takes quite a long time to execute, depending on the search term used, the number of tweets returned and the time frame to return results for. The problem seems to be in the code that calls the Yahoo web service. Every ‘page’ of the Twitter search results returns up to 100 tweets and the text of each of these must then be passed onto Yahoo for analysis. This means that for each call to the Twitter search API there are up to 100 calls to the Yahoo term extraction service. This can easily cause the script to ‘time out’. In its’ present form it would probably be best run once a day to return search results for the last 24 hours, with the results being emailed to someone to analyse. It may be possible to amalgamate the calls to the Yahoo service into one. The keyword extraction itself is somewhat rudimentary and could probably be easily improved, depending on what a client was looking to do.

What is interesting to me is how powerful this could be. It could be possible to build automated scripts that run on a schedule, tracking campaigns on Twitter. If the text analysis could be made more sophisticated it would be possible to alert someone only if certain trends appeared, allowing them to then make appropriate responses. This would then become a very useful Twitter tracking tool. Similar tools could be built for other social networking sites.

As an example of what is possible this script serves its’ purpose. For a production environment there are lots of ways in which it could be improved but the possibilities are there. Source code and an example of the script output can be downloaded here but due to the issues with time outs I can’t make the script itself available to request from a browser. I hope someone will be inspired to come up with their own, better, Twitter analysis tools and will consider sharing it with the rest of us.