#TALK

Scaling up Twitter systems

Scaling up Twitter systems to support millions of users and millions of lines of heterogeneous code.

Other

Twitter is one of the most heavily trafficked sites on the internet, with over 240 million active users and serving over 500 million Tweets per day. Twitter, as a company and as a service has seen remarkable growth in its eight years of existence. Scaling its services to meet high traffic demands and scaling our development tools to support millions of lines of heterogeneous code have both been challenging and rewarding.

Historically, nearly all of Twitter's traffic had been served by a monolithic application written in Ruby on Rails. Our growing user base, rapid development cycles and changing business needs all contributed to redesigning our monolithic codebase and move to a services oriented architecture.

In this talk, I will attempt to explain how Twitter scaled to support a) millions of users and b) millions of lines of code and how the community can leverage from these experiences and open source tools being developed.

How to scale to support millions of users? For higher performance, scalability and resilience to failure, we worked on the migration of Ruby on Rails based monolithic Twitter services to Services Oriented Architecture (SOA), where the services are JVM processes, mostly written in Scala and communicate with one another asynchronously over a network connection.

How to scale to support millions of lines of heterogeneous code? Pants Build system at Twitter is a state-of-the-art build tool designed to support polyglot services. This tool is open sourced and is being actively worked on and rapidly adopted in the community.