Even though there are others idea that have tried accomplishing the same goal as HipHop for PHP, I believe it is quite safe to assume that Facebook has a large enough user-base to produce code that is solid enough to run and can run well.

The announcement has been made on the Facebook blog earlier today, and tonight there is going to be the video tech talk that everybody can watch:

This evening we’re hosting a small group of developers to dive deeper into HipHop for PHP and will be streaming this tech talk live. Check back here around 7:30pm Pacific time if you’d like to watch.

A few questions come to mind even though we haven’t seen the code just yet. My main concern though is the one of buffer overflows and the security implications of turning PHP code into C++. As they say on the blog, it took nearly 18 months before having a relatively stable version and 3 developers. This is a very short lapse of time to develop a solution used by so many.

Another interest of mine related to this release is how does it compete with the likes of phc or roadsend php. If it does at all.

However I have noticed on their blog that Facebook has also developed HPHPi which seems to let you use HipHop but without having to actually compile your code before running it (The concept seems a bit like APC’s stat on and off switch from the few lines of description), which seems like a quite interesting idea for the development stages.

If you have ever used Python and Memcached, you probably have been looking for a library to connect to your Memcached server. And I’m sure you stumbled on the Tummy.com library which has basically no official online documentation.

Code is the best documentation:

That’s right… download the tar.gz file, go to the folder where you downloaded it, extract the content of the archive file, open your favorite editor and read the comments of the memcache.py file. It’s rough, it’s ugly, it takes time and it’s boring, but, it’s a proven way of doing it. Many nerds and geeks will tell you it’s the BEST WAY IN THE WORLD!

Sorry, I strongly disagree with that. When you install the library with the likes of apt-get (apt-get install python-memcache) or any other dependency management tool, you are lost because you have no direct code to look at (Unless you are an uber geek and decide to go look at the location of your installed files, grep for memcache and read the .py from there. Of course this is also very inconvenient and very unproductive).

Don’t use it at all:

Google for anything else? Realize that everything seems to be pointing back to that Tummy link… ohh there are a few other libraries alright, you could use the one from Twisted  or maybe cmemcache which is an interesting one but yet another one that doesn’t seem to have any documentation.

Keep digging and get lucky:

You can keep digging in order to find documentation for that solution you absolutely need and then get lucky and find a link. In the case of Python memcached, if you are lucky enough you find a link on some mysql page… why? Not quite sure but it’s there.

You use it and decide to write an article about it, just like I am doing:

Here we go, Python Memcache basic usage, how to install python-memcache (on Ubuntu), how to connect to your memcache server(s), how to add a new key to the cache, how to retrieve data from memcache and how to delete (lazy delete) from memcache.

1

apt-get install python-memcache

Once everything went ok, you need to test your connection and your memcache server so simply run the python interface:

1
2

>>> import memcache
>>> s = memcache.Client(["127.0.0.1:11211"])

Of course if your memcache server is on another host and another port, you need to adjust the line above.

So now let’s test adding data to the server:

1
2
3

>>> import memcache
>>> s = memcache.Client(["127.0.0.1:11211"])
>>> s.set("name", "david");

Now you have a name set in your memcache server and if you instantiate a new connection, you should be able to retrieve this “name” key.

1
2
3

>>> import memcache
>>> s = memcache.Client(["127.0.0.1:11211"])
>>> name = s.get("name");

You should get “david” back from that. Here’s an example on how you could actually use that in a file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import memcache
"""
Example class
 
This is simply an example class used for the memcache demonstration
 
package: echolibre.examples
"""
class Example():
    """Example of memcache tester"""
    hostname = ""
    server   = ""
    def __init__(self, hostname="127.0.0.1", port="11211"):
        self.hostname = "%s:%s" % (hostname, port)
        self.server   = memcache.Client([self.hostname])
 
    def set(self, key, value, expiry=900):
        """
        This method is used to set a new value
        in the memcache server.
        """
        self.server.set(key, value, expiry)
 
    def get(self, key):
        """
        This method is used to retrieve a value
        from the memcache server
        """
        return self.server.get(key)
 
    def delete(self, key):
        """
        This method is used to delete a value from the
        memcached server. Lazy delete
        """
        self.server.delete(key)
 
if __name__ == '__main__':
    sample = Example("127.0.0.1");
    sample.set("name", "david");
    retrieved = sample.get("name");
    print retrieved

So simply enough you can invoke “python example.py” or do something like

1
2
3
4
5

>>> from example import Example
>>> e = Example("127.0.0.1")
>>> e.set("name", "david")
>>> print e.get("name")
>>> e.delete("name")

You are now ready to use memcache from your python code/classes/scripts so I hope you enjoyed this and that it’ll be useful. I’ll be posting a second article about the multi delete, multi set, and extended methods that you can’t find anywhere. In the meantime, they are mine! Mouhahahaha! (evil)

What has been keeping me busy? I have presented at 3 conferences (PHP Tek, DPC and OSCON), took a week vacation in Iceland with Chris Shiflett and Andrei Z, moved to a new house, work on a very fun but stressful project among other interesting things that have happened.
A rough calculation told me that since mid May, around the time of PHP Tek, I have been away from my home a total of 38 days, almost 50%, not bad at all!

To touch base quickly on each of the conferences I spoke at:

PHP Tek

Located in Chicago, as it has for the previous years, it was a great conference as always. A very good community presence at the conference, which makes it very enjoyable to attend, both to see quality talks given by some of the best PHP speakers in the world and meet old friends and make some new, the best MTA conference so far.

I gave a whole day tutorial on caching and performance with Scott MacVicar and then gave a talk on frontend caching. Both went over fantastically, a lot of user interaction, my talk was full and the tutorial great fun with loads of people in it.

No doubt the best PHP conference in the states at the moment, a must go to for anyone living close (or even afar)

Tutorial slides

1. http://www.slideshare.net/helgith/caching-for-cash-part-1
2. http://www.slideshare.net/smacvicar/caching-for-cash-benchmarking-and-profiling
3. http://www.slideshare.net/smacvicar/caching-for-cash-caching
4. http://www.slideshare.net/helgith/caching-for-cash-part-4

Talk slides:
http://www.slideshare.net/helgith/coping-with-cyber-monday

Dutch PHP Conference (DPC)

As before DPC was held by Ibuildings in the RAI in the beautiful city of Amsterdam.  DPC seems growing every year I see it and was not suffering from any recession problems this year like many events have, if anything they made things even grander than before.

DPC is to Europe what PHP Tek is to the States, a world class conference with high quality speakers and talks, without losing sight of the community. It’s one of those conferences where I expect I will go every year for unforeseen time and try to drag along as many friends as possible, great fun!

I gave my Caching for Cash tutorial with Scott MacVicar, where we sold out and the room was so full that people had to stand around or sit on desks way back in the room, we even brought in couple of extra seats.

The tutorial went over very well, our best performance so far! (pun intended)

The slides can be seen at

1. http://www.slideshare.net/helgith/cashing-for-cash-part-1-dpc-2009
2. http://www.slideshare.net/smacvicar/caching-for-cash-benchmarking-and-profiling
3. http://www.slideshare.net/smacvicar/caching-for-cash-caching
4. http://www.slideshare.net/helgith/caching-for-cash-part-4-dpc-2009

OSCON

This was the first time I go to OSCON so all I have to compare it against are the stories I hear from other people, but there was a problem with basing my expectations on those stories and that was due to the fact OSCON moved from Portland to San Jose, a move a lot of people were not happy with.

I had high hopes but going in I must admit I didn’t really feel the “awesomeness” that a lot of people had been talking about, while a good conference with amazing speakers and good expo, it seemed like a shadow of it’s former self. It was the first year they had a decline in attandence, 500 less or so, no doubt due to location change.

This being one of the biggest tech conferences in the world, there were very good talks in there, ranging from design to programming to hardwares, it is not to be shuned even tho it changed location and has a higher than average entrance fee. You will always get value for you money and meet great people.

As for what I saw at the conference, I poked my head into random talks about various topics but the one that sits best in my mind is the one Jon Tan gave on “Grokkin Design”, great material but due to the fact that OSCON only has 45 minute talks (which I do not understand) he could only touch lightly on the subject.

I gave my “Website releases made easy with the PEAR installer”, it went over nicely but suffered from the whole 45 minutes talk problem, where I originally designed this talk to be at least an hour.
Furhter info on my talk can be found at http://en.oreilly.com/oscon2009/public/schedule/detail/8126 and the slide can be gotten from here http://www.slideshare.net/helgith/website-releases-made-easy-with-the-pear-installer-oscon-2009

I would like to thank Chris, Sean and Jon for allowing me to crash with them and Andrei for being a fantastic host and taking us all over the place for great food and drinks, it really added that extra spark to a good trip

Upcoming:

I will also be presenting soon at ZendCon (http://www.zendcon.com) where I will talk about frontend caching and Epicenter in Dublin (http://www.epicenter.ie) where I will talk about frontend caching  and how to deploy your websites with the PEAR installer.

And hopefully couple of other conferences around the world.

See you there!

a venture-capitalist television programme that originated in Japan where the format is owned by Sony. The format, which now airs internationally, consists of entrepreneurs pitching their ideas in order to secure investment finance from business experts — the “Dragons”.

As some may already know, in 2009 the television show began in Ireland on RTÉ ONE. This post covers the technical considerations encountered when a web site / application appears on national television.

Some background

[Just want the tech tips without the intro?]

One of our clients, Rent Collectors.ie , appeared on the Dragon’s Den (#ddire) on 2 April 2009. This site is hosted on a single virtual private server (VPS) so it has limited memory, CPU and disk space. It runs a version of Ubuntu, PHP 5.2.3, Apache 2.2.4 and a MySQL 5.0.45 (The famous LAMP stack). Those are the stable versions with the operating system version we are running and in order to keep consistency with dependency management, we stuck to it but you should always use the latest stable versions.

RentCollectors is a rather simple service with a good few database interactions and a complex but well designed backend for multiple agents from anywhere in the country to work with. Prior to that day, extreme performances weren’t a concern. Developing with caching in mind, making the site scalable if it has to be scaled, making sure it could hold high loads was more like a theory we applied to the architecture when developing it.

So what happened? Well a few days before the television show, the client warned us that that website would be under high pressure as it was going to be on national television later that week. I decided to make sure that the server, considering it’s hardware capabilities, would hold the load of being publicised on TV.

I started by benchmarking the website with Apache Benchmark to find out how much it could take. Turned out to be handling about 35 pages per second. That made my bat phone ring. 35 requests per second is all but acceptable for a website - that is going to be on national television. So knowing that we hadn’t turned on different levels of caching I realized that in a rather short amount of time - meaning that no rewrite was needed, I managed to bump the server to up to 185 requests per seconds without hogging the server (and still being able to access the website from other computers from other networks). 185 requests per second is not extreme, far from it, but for a single vps server, being to handle the load was the most important things. We could have simply plugged in a few other webservers and scale horizontally/out but I was convinced that we could hold the load with that single server. It was a challenge and a test.

Vertically Scaling a single VPS

So instead of scaling out, we decided to scale vertically/up. That basically involves adding CPU or RAM (memory) to a single system. What we did is we created more daemon processes to Apache and added more reserved(spare) child threads. This gave us the buffer we needed as the load was going up.

Interesting, but then we decided to change the MySQL settings to be able to cache more database results. That way, the MySQL query cache wouldn’t have to do all the work when the exact same queries would be executed. Before you go on your server and increase the query cache on your MySQL servers, please make sure that you know what you are doing. Having the query cache enabled and running is good when the tables do no change too often. This is good for webpages that generate content pages, navigations, footers, etc. If a table gets modified, the cache gets flushed. One way or the other, before you start using the MySQL query cache, I suggest you read the mysql documentation about it and in more general terms you should also read the MySQL Performance Blog, you will most likely learn new things about MySQL and performance.

So after optimizing the MySQL settings we realized that each public page were highly unlikely to be modified during the television show. So we took a different direction and as well as caching at the MySQL level, we thought that if we could rule out the MySQL server completely, it could be even better.

So we setup APC and used our EcholibreApc class. We first used APC to cache the bytecode and then we set the apc.stat setting to “0″. apc.stat is set to “1″ by default and this means that each time a cached script is requested, it verifies if the cache has changed then returns the cached version. By making apc.stat set to “0″ we skip that step meaning that it doesn’t try to see if the cached entry has been modified. Beware! It’s much faster but requires you to restart your webserver if you make a change to a PHP file and you want it to take effect. APC runs by default with the “apc.shm_size” set to “30″. This means that APC will be using 30 megs of your shared memory. Just remember that memory is cheap and it’s easy to add, in our case we made the shared memory size bigger in order to make sure that we could store all what we needed. So after all this, we only had the database working every 30 minutes. Which was reducing the response time and server load/memory usage considerably.

Here’s the very simple EcholibreApc class — APC utility — in case you are interested or want to use it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

/**
 * Echolibre Apc Class
 *
 * A very very simplistic class that acts as a layer to the
 * advanced PHP Cache.
 *
 * One of the reasons of this is that APC doesn't store arrays
 * that are more than "1" level deep. This class wraps around
 * json_encode and json_decode. If you want to know why we use
 * json* instead of serialize well you should run your own micro-
 * benchmarks. You could be surprised ;-)
 *
 * @author  David Coallier
 * @see     http://php.net/apc
 * @version 0.1.2
 * @license New BSD
 */
class EcholibreApc
{
    /**
     * Fetch a variable
     *
     * This static method will fetch a value
     * from the apc cache and either return
     * false if the variable doesn't exist in the
     * cache or a json_decoded array/object.
     *
     * @param  string $name   The name of the variable to fetch from the cache
     * @return mixed  Either bool false or a json_decoded object.
     */
    public static function fetch($name)
    {
        $var = apc_fetch($name);
 
        if ($var == false) {
            return false;
        }
 
        return json_decode($var);
    }
 
    /**
     * Store a variable
     *
     * This static method will store a json_encoded string
     * in the apc cache for the amount of time supplied by
     * the third argument.
     *
     * @param string $name  The name of the variable to store
     * @param mixed  $value The mixed variable to store in the cache
     * @param string $time  The time length to store the variable in cache.
     * @return void
     */
    public static function store($name, $value, $time = 3600)
    {
        apc_store($name, json_encode($value), $time);
    }
 
    /**
     * Delete a variable
     *
     * This static method will delete a variable
     * from the apc cache.
     *
     * @param string $name  The name of the variable to delete from the cache.
     * @return void
     */
    public static function delete($name)
    {
        apc_delete($name);
    }
}

So after doing the Apache tweaks, MySQL tweaks and the APC/PHP tweaks, we simply made sure the sessions were not read and wrote to disk by making sessions saved to /dev/shm (shared memory) and that was it for backend changes.

Final Precautions

We looked at what else could be done in order to improve the user experience. Helgi Þormar Þorbjörnsson and his expertise on frontend caching came in very handy as we decided to cache more data on the user’s browser, GZIP encode the content being sent to the browser, reduce the footprint of the javascripts and CSS stylesheets. So after compacting the multiple javascript files we had and running a few stress tests, we ended up with a website and a server that was in very good conditions and most likely ready to be on television.

The decisive night arrived, the Dragon’s Den was on television, our turn came, we started seeing connections to port 80 (netstat –nat | grep :80 | wc -l) going up, and from a few servers around the world I was still able to access the site as if no-one was online. about 15-20 minutes later, the connection count was gone back to the usual and no sign of downtime whatsoever. We had made it, without a single glitch. Dragon’s Den is on at 22h00 and after this test, it was time to sleep. A very peaceful night, much more peacefully than the number of connections rising on the webserver

I have been studying and working with web standards (XHTML, RDF, ATOM, RSS) for well over 4 years now, something I am glad of,  because recently something struck me. Conventions are arising, for example DOAP, SKOS and others, that are built on top of the Resource Description Framework otherwise known as RDF, if they aren’t, they are usually built on something very similar or related.

In this blog post I am going to be using the FOAF standard as an example and base for my proposal. First of all, the Friend Of A Friend standard (FOAF) is a project aimed at creating machine readable pages that describe people. It covers all basis of human interactions and behaviours. From basic profile information - name, mailbox, title, homepage, img, depiction, surname, given name, family name, firstname - to more detailed information as such as web blog, based near, geekcode, publications, etc.  As you can see on their standard description page, it also covers the following personal aspects: Online Account / IM, Projects / Groups and Documents and Images.

For instance, if a web application was to describe me using FOAF it could look something like this:

1
2
3
4
5

<foaf:Person rdf:about="#davidc" xmlns:foaf="http://xmlns.com/foaf/0.1/">
  <foaf:name>David Coallier</foaf:name>
  <foaf:homepage rdf:resource="http://echolibre.com" />
  <foaf:img rdf:resource="/images/david.jpg" />
</foaf:Person>

The Problem

Whilst such a standard is clearly easy for a computer to read and does seem a logical fit, it’s not the easiest for a developer to read. These days, web users are looking for performance, simplicity, ease of use, and so are the developers creating web applications. Considering that XML is heavy to parse using current technologies (Javascript mostly), it makes very little sense for developers to make web applications that are going to be slower due to parsing complex XML nodes. However, JSON (JavaScript Object Notation) is a lightweight data-interchange format made to carry data over networks with a very small footprint. It has seen massive adoption across the web and is used in widget apps, web apps and various other systems.

As I was talking at OSS Bar Camp, something struck me. Developers need a standard. Fact. We need standards for the same reason the industrial revolution had need of machine part standards: to reduce the amount of different solutions to learn in order to achieve the same goal by having everyone do what they do in a standard way. Some may see standards as a way to prevent innovation, I see them as a way to innovate. See “why are web standards important” by the W3C, a good read.

So, developers need standards. Great, now what currently exists? Development standards (IDE, Documentation, tools, design patterns, unit testing, etc.), Output standards (XHTML, CSS, etc), XML based standards (Namespaces, Schemas, XPath, XQuery, XSLT, DOM, XML Base, RDF(s), etc.), usual web architectural principles, and many more. One thing that is missing though is the standards that allow developers to easily and rapidly work with each other’s web application.

If “Web 2.0″ was characterised by the democratisation of content, we feel strongly that the next stage of web evolution, “Web 3.0″ for want of a better word, will be characterised by the democratisation of data and applications.

The Solution

A standard would make that democratisation a little easier. I’ve decided that I am going to be working on this over the next few months with the help of a few others as such as Ed Finkler from the CERIAS (Also the creator of Spaz and all round awesome guy).

So, today I’m putting forward the need for PJFS - The Practical JSON Format Standard. It will strive to make heavy XML based standards more developer friendly and lighter by creating new, fresh and adapted Practical JSON Formatted Standardized outputs.

For instance, the FOAF example I drew above, is a great example of the need for PJFS. Consider the following from a developers perspective:

1
2
3
4
5
6
7

{
    "Person": {
        "name":"David Coallier",
        "homepage":"http:\/\/echolibre.com",
        "img":"http:\/\/echolibre.com\/images\/david.jpg"
    }
}

It’s easy to read but also easy to parse. It’s fast and reliable. I am well aware of the implications of this post and I do not underestimate the complexity of existing standards. I think each one of them, as complex as they can be, are needed and are something we should all aim to use. However, computers are not completely independent ( just yet!) and the middlemen (developers and users) should not be forgotten.

The implications of JSON formatting standards mean that it’s adoption will make things easier for developers by reducing the amount of work they have to do, and remove the learning curve on new object structures for every web service the want to use.

The first task I will start working on is a PJFS for micro-blogging web services. The likes of identi.ca have already started in the general direction by “copying” the behaviours of the Twitter API, however many other micro-blogging platforms are still very independent and a unified standard could help the tools developers creating more flexible tools that would cover more networks.

So, remember where you heard about PJFS first It’s there to create standard object names, properties, variables, class members names to JSON elements so that developers can expect something identically formed when requesting JSON information from a webservice.

If you’d like to be involved leave a comment or catch me on twitter - @davidcoallier.

This past weekend we went along to Dublin’s OSS BarCamp. There were some really great talks, Stuart Langridge’s Javascript presentation was a highlight. Jaime Hemmett’s talk on using Git for version control was useful and informative. Paul Biggar’s talk on PHC, the Open Source PHP Compiler, was thought provoking and certainly caused a few grumbles from PHP guys like myself in the audience ;). JD and myself managed to catch up with him after and talk through some of his ideas over a drink.

I also got to give a talk on how APIs will be the cornerstone of the next stage in the evolution of the web - web 3.0 if you will.

This was my first time talking about a subject that didn’t directly involve PHP.  I’m planning on doing a similar talk at BarCamp Belfast at the end of the month, so if you were present and have suggestions on how my presentation could be improved feel free to leave a comment below.

Andrea Trasatti (@andreatrasatti) made the valid point that a small company and startup are not the same thing. Also I’ll have to add a list of successful startups towards the end of the presentation to support my central argument that APIs can extend a customer base and service subscriber network.

So, my slides from the presentation are below, please do let me know what you think. As a company, we were delighted to be able to sponsor the event, and it was great to be able to connect to speakers and attendees alike over a few drinks after the event.