Timothee Groleau @ mig33
2012-07-11
French! Living in Singapore since 1999
Software Engineer at mig33, caring for the web team
Started in web frontends (Flash 5,6,7,8 + ActionScript)
Moved to JS + PHP
Enjoys servers, linux, cli, open source
Shameless plug: http://panocraft.com
Social networking platform targeting emerging markets, primary on low-end mobile devices
Founded 2005, yet still (somewhat) in startup mode
Experimentations on all fronts: Technology, Processes, Development practices. e.g. Practicing with internal rest services before publishing a public rest API.
66m Registered Users
1.4m Daily Actives Users
130k Concurrent Users at peak
Operations:
Data:
Best software is worth nothing if your infra and ops sucks
Jedi-Level ops team in Australia ( Spry: http://www.sprybts.com/ )
mig33's own ops team in KL
DevOps team in Engineering in SG, bridges app and infra knowlege
Puppet for the (almost) entire infra (both SJC + AWS) http://puppetlabs.com/
Nagios for monitoring and alerts http://www.nagios.org/
Cacti for near real time visualization http://www.cacti.net/
mig33 is an open source friendly company! And we're running all these on CentOS boxes
mig33 is an open source friendly company! And we're running all these on CentOS boxes
16+2+2+2+2 web servers
Vanilla Apache + module (e.g. mod_security, mod_rewrite, mod_usertrack)
Vanilla PHP packages + php.ini tweaks + some PHP modules (gettext, xcache, imagick)
Multiple usage:
we server clusters: migcore, login, migbo, wordpress, devcenter
Custom PHP framework served us well
Custom framework has multiple view support built-in (wap/ajax/midlet/touch)
Introducing Code Igniter
Usual practices
shared packages help reduce duplicated code
Reduce Disk IO + latency
2 enemies of performance: io and latency
Several types of cache:
Juggling what we put in what cache
conditions change, what seemed like a good idea might no hold true in the future, be willing to shuffle things around
Usual Cache strategies:
XCache | Memcached | Redis | |
---|---|---|---|
cross-server synchronization | no | yes | yes |
latency | no | yes | yes |
data structure | yes | no | yes |
nested data structure | yes | no | no |
requires serialization | no | yes (mostly) | yes (sometimes) |
supports expiry-based invalidation | yes | yes | yes |
supports content-change invalidation | not easily cross servers | yes | yes |
XCache still has a place as local server cache, best suited for:
Content change invalidation really only works with synchronized caches
Leave as little as possible for the web server to do. Let your app framework work for you.
example: application logic in url rewrite rules
Create a (safe) system to let you manage out-of releases changes (scales your team)
example of out of release changes: marketing messages, landing pages' content, promotions, email templates
Small specific apps/services
Lock down access to internal services to what's needed ONLY.
A fast no-SQL key/value store, with lots of nifty features
Sponsored by vmware
designed to address some shortcomings of existing system (e.g. memcached)
remote data structure + replication are probably the biggest gains
Used by:
Persistence
Remote data structures
example: removing an element from a list or set, in memcached, read, remove write. if another process was adding an element in the meanwhile, the write would overwrite the newly added element. work around in that case: distributed locks.
persistence allows us to store some authoritative data into redis, but we only do that for stuff that do not require ACID and are also not critical to lose or miss for a period of time
Expiry
Atomic operations + Transactions
transactions are performed with an optimistic locking mechanism
Pipelining
Search/List keys (including wildcards)
Dead easy Master/Slave setup
Customizable persistence behavior: AOF / Bg Save rules
Statistics: num ops, memory used, etc.
Single threaded, control over cpu core utilization
Solid! 2 years running, no problem
memory bound, no built-in system for horizontal scaling
responsability of apps to implement consistent hashing, or something else
burden on libs and ops to maximize resource usage
persistence compromise: AOF is slow(er), snapshots risks data loss
No value search
No joins
No read-only slaves (being introduced in 2.6!)
String/Blobs
number/strings
this is where you store your json, sir!
Hashes
key/value value in a key/value store! nesting not supported, but numeric field operations supported as well
Lists
mostly useful for queues
Sets
great for banned lists, user likes, incredible performance with operation like SISMEMBER, funky things like SPOP
supports huge number of entries
remote set operations! union intersection, etc.
Sorted Sets
power of both lists and sets!
go and die "ORDER BY"
expiry is at key level only!
PHP: predis: https://github.com/nrk/predis/
Python: redis-py: https://github.com/andymccurdy/redis-py/
Java: jredis: https://github.com/alphazero/jredis
More clients here: http://redis.io/clients/
/* query string parameters optional */ $instance = new Predis_Client('redis://myhost:12345?connection_timeout=2&read_write_timeout=3"); $instance->set("min_level", 10); $new_count = $instance->incr("landing_page_view"); $key = "visitors"; $value = "gurmit"; $max_age_seconds = 7 * 24 * 60 * 60; $now = time(); /* pipeline support for insert value, trim by age, and mark for expiration */ $pipe = $instance->pipeline() ->zremrangebyscore($key, '-inf', $now - $max_age_seconds) ->zadd($key, $now, $value) ->expire($key, $max_age_seconds) ->execute();
Complicated examples, you don't need to worry about it yet
value could contain more data provided it is deterministic, e.g. '{"username":"bob", "userid":1234, "country":"france"}
values could contain more data provided it is deterministic, e.g. '{"username":"bob", "userid":1234, "country":"france"}, just need to ensure that the data blob is always serialized in a fixed manner.
Quik Robin, to the command line!
there's a complex slow query to get the contestants, we run it once a day and iterate through the resultset to inject to into the redis pool
Redis is memory bound, we need an architecture that scales horizontally, more ram, more servers
Typical approach to clustering:
2 huge servers: 24 cores, 192GB RAM
64 shards + directory instances
Master/Slave across the 2 boxes
IPVS to hide infra complexity:
Self-contained cluster definition, 3 redis keys in the directory is all it takes:
Client lib need only be aware of the directory master and slaves
Shard assignments for an entity in the directory is a blob
Cluster and lib is entity-agnostic
Lib exposes a minimalistic api to interact with the cluster
applications should always have sane default values!
Keys with strict naming convention to allow movement of all keys belonging to an entity
Quick Robin, to the command line!
Sorry PHP, this one's using python :$
Fast
redis-benchmark ftw!
Memory fragmentation is a problem (<2.4)
Live restart of instances in a master/slave setup is hard
Delete on read/write + garbage collection
Multiple instances on the same server: disable all bg save rules – make your own save routine
Key names take memory too, use an efficient naming convention
Don't use auto-serialization in blobs, be explicit
Expiry is at key level, not for hash members
Redis can't give you memory info per key or per key namespace.
Getting rid of memcached -> redis in caching-only mode ftw!
redis is a superset of memcached; app speed from atomic ops on data types for speed and getting speed on set/get
http://antirez.com/post/redis-memcached-benchmark.html/
Services! Multiple versions of the same lib is a bad idea
Migrations of services / features
live migration: migrate data as it's being used
speed up process and handle data migration for inactive users
The future:
Table of Contents | t |
---|---|
Exposé | ESC |
Full screen slides | e |
Presenter View | p |
Source Files | s |
Slide Numbers | n |
Toggle screen blanking | b |
Show/hide slide context | c |
Notes | 2 |
Help | h |