The good and bad of Queuing

London Scalability Group • 16th July 2012. #ldnscale

Alister Bulman

self::about()

self::current()

Scaling PHP-based systems, with related technologies. Currently working on a 'secret project' in our secret volcano base, half-way up the Centre Point tower....

(OK, so it's the 13th floor, and we are eye-to-eye with a crane-driver)
I might be able to tell you more very soon

self::prev()

Programming PHP since 1999, PHP v3, (>12 years ago). Previously in everything from a ZX81, BBC/Electron, 6502 to 8086 Assembler, Pascal and C/C++ before PHP

Programming since 198-cough

Classic uses of a queue

image/video resizing
fetching from an external service (eg, RSS feeds, or Twitter)
Twitter uses various queues to move data around internally

various queue systems

Amazon SQS
beanstalkd
LJ's Gearman - and its follow-on superset, TheSwartz
RabbitMQ, written in Erlang (recently bought by Springsource/VMWare) - built in London
Sparrow, Starling, and now Kestrel by Twitter
other heavyweight MQs, Apache's ActiveMQ, Tibco, WebSphere MQ 'message bus'
and many, many others
- including SQL databases
Special mention of ZMQ

What I've done with queues

250,000 emails from a dating website
PEAR's Mail_Queue, 50 at a time
self-limited the sending - LdAvg < 2
Later: why the general case of using a DB isn't the best idea
Why Beanstalkd?
~3.5million requests a day from Twitter - 100 at a time
we topped out at 40req/s with SQS (across the atlantic)
- still did millions req/day with SQS
Beanstalk had priorities - controlled what data was next

Hang on to your hats

Beanstalkd live demo

Putting 1000 items in - and then taking them out

want to see it again?
100,000 JSON-strings items entered in ~70 seconds.
In production.
On Amazon EC2.

Here's the good

asyncronous!

Msmq Many long running, or intensive tasks could be done in the background

- Background or Parallel Processing

processing images or videos
Sending out emails
Fetching information in the background - like RSS feeds
Preparing information for later use

The classic problem is image uploads. Even happy-snap digital cameras can produce an image that is 3-4+ megabytes. You can't just store that and end up resizing it in the browser. You have to resize it, but you also can't do it in the same PHP process that uploaded it - it takes too much time - and memory.

Even the pic from the 13th floor was 780KB on disk, at least 15MB memory - just to view. To resize, could be 30MB or more.

Don't even try to resize 5+ GB of 1080p quality HD video

Use of queues at Flickr
- the most popular use isn't even about resizing images

The Advantages

Parallel and/or asyncronous work
- it's too much to do while we wait
- do it in the background, while we are doing other things
Scales easily - run more workers
- if the work can be done independently of data
- workers can be run almost anywhere within your cluster
- particular jobs can also be directed to specific machines
Call functionality written in other languages
With a good infrastructure, adding in new tasks are trivial

The Bad

Over enthusiasm

too many steps can lead to balancing problems
- more than just two might be too many
- consider merging the steps - don't have so many
Different stages of your system will run at different speeds, and you'll end up with upstream tasks creating ever-growing backlogs when they feed into slower consumers. ... the only real solution is carefully tuning the speed of the original inputs to match the available resources. This is not fun.
-- Pete Warden: June 2011: Queues are the devils own data structures - Pete thinks Hadoop can handle a lot of this.

The Bad, II

If things go wrong

You have to be very careful of failures
hard to see what happens when they fail
worst case: they fail, and go back to the queue -
where they fail over and over again
put all failures into a 'sin bin', and fix ruthlessly

The Bad, III

Setting up and running

some ideas and HowTo's at http://phpscaling.com/tag/queues/ - Doing the work elsewhere - sidebar "running the worker"
Upstart or Supervisord, and shell scripts to start/keep things running.
Be able to shutdown workers at will (shedding load)
I'm still looking for better ways

Queueing via a DB :: Anti-pattern

An obvious early step
Not actually so well suited, especially with many transactions
high transaction costs - locking to ensure only one worker can get a message
PEAR's Mail_Queue can work very well with small number of 'workers'
A lot simpler that throwing large numbers of large messages into a formal queue

Polling
Locking
Data growth

If you are deleting items from the queue as they are being sent, then you know the DB can be good for millions of records

Beanstalkd

Many language bindings are available

Quite easy to write, as it's a simple text-based format, ala Memcached

The daemon is equally dumb as a classic MemcacheD. Talking to multiple servers is done from the client-side.

Blocking
I'll wait here till you have something

BeanstalkD was written in C for a Facebook app called 'Causes' - which was written in RubyOnRails

Not to be confused with the 'Beanstalkapp.com', which is remote SVN & GIT version control

OR WITH Amazon's Beanstalk to start EC2 and other AWS services

What to send through the queue?

'Clean URLs':
- /tasks/image/resize/filename/example.jpg/sizeX/640/sizeY/480
- /tasks/member/logintasks/id/12345
- /tasks/mail/fetchcounts/id/12345
PHP's serialize/deserialize
JSON
- {"module":"task","controller":"image","action":"resize", "params":{"filename":"example1.jpg"}}
plain text: RESTART, STOP, UPDATE-COUNTS

64KB default limit.

In Summary

Job queuing systems can give a lot of flexibility to pass messages into background processing
Background tasks are not subject to the same limits as a webpage would be - time or memory
Limited by your imagination on how to use it
Getting it right is not easy

Thanks

London Scalability Group
Beanstalkd FAQ and github pages at http://kr.github.com/beanstalkd/
Pheanstalk (php library) on Github: http://github.com/pda/pheanstalk
Livejournal's scaling and the technology that got me interested in all of this (oscon presentation, 2005, PDF link)

Questions ?

This will be posted soon at http://abulman.co.uk/tag/presentation/

The Good and Bad of Queuing

London Scalability Group • 16th July 2012