shad's blog

Here is my easy way to know who unfollows me on Twitter. Or more precisely, how to monitor the changes on your followers list.

# cd /etc/cron.daily ; ln -s /path/of/your/script twitter_changes

I have just released a new minor version of CirruxCache. This new version includes several bugfixes and third party library upgrades.

An easy way to use Google Appengine as a personal CDN! CirruxCache 0.4.1 is out, so let me show you a set of new cool features!

CirruxCache provides a software solution to dynamically cache HTTP objects on Google Appengine (using the Datastore and the Memcache services).

Since the last release, I received a lot of good comments and smart suggestions. Fortunately, Some of you increased the size of my todo-list and a bunch of nice ideas are still in the box.

Let’s take a look at this release:

• Admin panel now includes a config helper (find here screenshots and documentation).
• Object flushing has been fixed in the admin panel, give it a try!
• The Cache service now includes a “Content Prefetching” feature. When it is enabled, HTML pages are interpreted. All associated content are loaded in the cache before the user request them. So it speeds up the first page loading. It is experimental for the moment, but it works (it is enabled on this blog!).
• A new Image service enables you to delegate image transformation to CirruxCache. For example, you won’t need to upload images previews or thumbnail on your website anymore, let CirruxCache generate them and cache them on the fly! (All images on this page are generated this way).
• Config has been simplified: you don’t need to edit the main “app.py” file. It is located in a file named “config.py”. On top of that, all internal services (admin, cron, store) are now included by default, you can forget them! (Install documentation page has been updated).
• A lot of bugfixes which improved the caching mechanism.

I hope you will enjoy this new release, check it out on the download page.

I would like to share with you the Users page, I try to list some of you using CirruxCache. This pagedoes not have much success, please take a look!

Finally, I warmly encourage you to subscribe to the CirruxCache group, it is really cool to get help and to provide advices to other users.

Spread the word!

 First of all, install the required ubuntu packages:

 $ sudo aptitude install libssl-dev libsqlite3-dev

Then, fetch python2.5 source tarball:

 $ wget http://python.org/ftp/python/2.5/Python-2.5.tar.bz2
 $ tar xjf Python-2.5.tar.bz2

Before compiling, we need to tweak the file Python-2.5/Modules/Setup.dist (line 207) as following:

 _ssl _ssl.c \
 -DUSE_SSL -I/usr/include/openssl \
 -L/usr/lib -lssl -lcrypto

Compile and install:

 $ mkdir /opt/python2.5
 $ ./configure --prefix=/opt/python2.5
 $ make $ sudo make install

Finally, after having deployed the appengine framework package, we need to modify shebangs:

 $ cd /path/to/appengine
 $ find . -name '*.py' -exec sed -i "s/#!\/usr\/bin\/env python/#!\/opt\/python2.5\/python/" {} \;
 $ chmod -R 755 .

I have just written some documentation about the Store WebService of CirruxCache. This little page will help you to upload big files on CirruxCache in the same way than the current admin panel does.

http://code.google.com/p/cirruxcache/wiki/StoreAPI

Just to remember, the Store WebService has been developed to bypass the 1MB limitation of appengine. It is not possible to dynamically cache objects with a size greater than 1MB, in this case, you need to store explicitly such an object using this WebService. It uses the Blobstore service, so file size cannot exceed 2GB.

I am really glad to announce a new major release of CirruxCache.

This new release includes the following changes:

• A storage webservice: store big files (<= 2GB) on the Blobstore in order to deliver them through CirruxCache. This feature is useful to bypass the 1MB limit on appengine.
• An admin panel that enables users to flush objects, manage big files and see some statistics about the resources used.
• Bugfixes

It is really important to note there are few limitations on the panel admin:

• There is no error reporting on the flush panel (it only displays the number of objects trying to be flushed).
• Storage manager displays a "500 Internal Error" when uploading. It only happens when you don't have a billing account (the Blobstore is only available on billing accounts, refer to appengine).

These two limitations will be improved in the next release, and there will be more informations in the statistics panel.

The Storage WebService will be documented really soon, but you can access the admin panel through "http://your.cirruxcache.app/_admin/"

I make the most of this opportunity to announce some changes on the project website:

• Roadmap page
• Users list: this page has just been started, so do not hesitate to notify me to add you.

I hope you will enjoy this new release.

CirruxCache 0.2.2 has just been released. It contains some bugfixes (thanks to Devattas to have reported errors on Datastore latency). Webpy has been updated to the last version.

I have also updated the documentation, especially I brought more details on Point of Presence configuration and usage of cron tasks for garbage collection.

Finally, some of the users reported me that there is a real problem with the cached object size limit (currently 1MB). I am working on the solution, I will take advantage of the new Blobstore service on AppEngine to store objects. Maybe I will keep the Datastore only for meta-data. This solution will raise the cache object limit to 50MB.

Stay tuned :)

I have just released a new version (0.2.1) of CirruxCache. To remember:

CirruxCache provides a software solution to dynamically cache HTTP objects on Google Appengine (using the Datastore and the Memcache services).

This new version includes an interesting set of features:

• allow object flushing from restricted IP
• configure a PoP (Point of Presence) according to a virtual host
• several behaviors (cache, redirect, forward)

In more details, the last feature is the ability to configure a point of presence to differ from a classical caching mechanism. For example, I may want to configure "/admin/*" on my website to be redirected on the origin without caching.

Of course, this release includes several bugfixes, especially a fix on the "Expires" HTTP header which improves the caching performances.

Do not hesitate to test this new version and to comment any bugs or any suggestions.

Webpy is a tiny web framework. I use it a lot for my web-services applications. In general, I let my web server (lighttpd) to handle virtual hosting. But as you may know, I am working on a CDN solution on top of Google App Engine, named CirruxCache. In that case, while I have absolutely no control on the server configuration, I need to handle virtual hosting from the code. Webpy maps urls by iterating through a tuple. So my solution is quite simple: wrapping the tuple to override the __iter__ function according to an environment variable (HTTP_HOST). Let's take this basic webpy example, without vhosting:

import web

urls = ('/(.*)', 'hello')

class hello(object):
def GET(self, name):
if not name:
name = 'World'
return 'Hello, %s' % name

if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
Let's add the VhostMapper class:

import web

urls = {
'default' : ('/(.*)', 'hello'),
'my-vhost.domain.tld' : ('/(.*)', 'helloVhost')
}

class hello(object):
def GET(self, name):
if not name:
name = 'World'
return 'Hello, %s !' % name

class helloVhost(object):
def GET(self, name):
return 'Hello %s' % web.ctx.environ['HTTP_HOST']

class VhostMapper(object):
def __iter__(self):
url = urls['default']
if 'HTTP_HOST' in web.ctx.environ:
vhost = web.ctx.environ['HTTP_HOST']
if vhost in urls:
url = urls[vhost]
return iter(url)

if __name__ == "__main__":
app = web.application(VhostMapper(), globals())
app.run()
Finally, you can use curl or wget to test your vhosts:

$> curl -H "Host: my-vhost.domain.tld" http://localhost:8080/

It is not so early to announce that the next version of CirruxCache will handle virtual hosting :) I am sure this simple hack can be easily reproduced to use virtual hosting in some other Rest frameworks.