In the field, distributed systems are the new norm. This talk discusses tactics and strategy for productionizing distributed systems with a little bit about what the future will hold.
_-_Version_250c4.jpg?1387210602)
The Netflix OSS Cloud stack is clearly a great set of components for building a cloud infrastructure and platform—if you are Netflix. But how does that architecture work for other businesses? Learn how at Riot we leveraged Netflix OSS Cloud tools and platform components to create a complete infrastructure for hosting our global game platform. Maybe it can work for you too.
This session will describe the libraries, services and tools from the Netflix OSS stack that we used, our adaptations and extensions, and how we put them together with a nifty Java library stack to form a cohesive platform.
On the Java development side, we adopted Dropwizard, added in some of the Netflix OSS libraries plus some of our own magic sauce to create a simple framework for quick development of robust, cloud-ready Java services.
We adopted Aminator’s pre-baked image model, but extended baking to use Chef to leverage our existing cookbook library. And instead of using Chef at launch for bootstrapping and configuration, we use Eureka for service discovery and an built an Archaius backend service for dynamic configuration.
We use Asgard as our general cloud portal, but have extended it with user authorization to fit our engineering environment. And, we use Edda and the Simian Army for security and conformance auditing, as well as routine cleanup tasks.
ZooKeeper is everywhere these days. It’s a core component of the Hadoop ecosystem. Your favorite startup probably uses it internally. But as every good skeptic knows, just because something is popular doesn’t mean you should use it. In this talk I will go over the core uses of ZooKeeper in the wild and why it is suited to these use cases. I will also talk about systems that don’t use ZooKeeper and why that can be the right decision. Finally I will discuss the common challenges of running ZooKeeper as a service and things to look out for when architecting a deployment.
Map Reduce begat Hadoop begat Big Data. NoSQL moved us away from the stricture of monolithic storage architectures to fit-for-purpose designs. But, Houston, we still have a problem – architects are still designing systems like they did in the ‘70s. Yet most systems are still designed for store-then-compute rather than to observe, orient, decide and act on in-flight data. Rather than just taking a spoonful of stream processing from large web companies with well intended open source engines, such as the Lambda Architecture, let’s look at the crazy architectures in high frequency trading such as Complex Event Processing (CEP). They got the techniques right, but the solution wrong – it didn’t scale. This talk isn’t about big, slow data; this talk isn’t about small, fast data. This talk takes a nuanced path through most of the rest which sits somewhere in between, with tradeoffs aplenty.
There has been a lot of talk about Linux containers recently, and how they can help you beat “Dependency Hell”. Docker is one such technology that is using Linux containers to better deploy your applications. In the last year Docker has taken the DevOps world by storm. I’ll tell you what Docker is, how it works, and how to add it to your development toolbox.
Understanding application responsiveness and latency is critical not only for delivering good application behavior but also for maintaining profitability and containing risk. However, good characterization of bad data is useless. If measurements of response time present false or misleading latency information, even the best analysis can lead to wrong operational decisions and poor application experience. In this talk, Gil Tene (CTO, Azul Systems) will demonstrate and discusses some common pitfalls, false assumptions and problematic measurement techniques that lead to dramatically incorrect reporting results, and will explain how these false measurements naturally occur using the most common measurement methods and tools in use today. We will discuss the coordinated data omission problem, and ways to work around it, and will introduce and demonstrate how simple and recently open sourced tools can be used to improve and gain higher confidence in both latency measurement and reporting.
We rely on distributed databases and queues to store and process data reliably, but many fall short of their marketing promises. I’ve spent the last year building tools to stress and analyze popular databases
during network partitions, uncovering everything from undocumented behavior to catastrophic data loss. In this installment of the Jepsen project, we’ll explore what “strongly consistent” really means, and
find out whether various databases live up to their claims.
Boundary was founded on a relatively simple idea: what if we were to use modern distributed data processing techniques to tackle a moribund industry; IT Monitoring? Further, we figured that if we could tackle the piece of the stack that outputs the most data, the network, then we could take on any data that our customers would care to throw at us. Along the way we built and threw away several databases, and changed from a database model to a streaming model, and are building something slightly different than all of that today. I will talk about our motivations along the way, how the shape and the overall volume of the data drove our architecture decisions. And, of course, I’ll talk about how we were able to derive value from a dataset that’s been deemed useless more times than I can count.
Marathon is a framework built on Apache Mesos that provides a fault-tolerant and elastic scale-out architecture for any long-lived application like Ruby on Rails/node.js-based web applications or traditional J2EE servers like Tomcat. Marathon can run applications without modification, and supports running Docker containers as well. It provides a simple REST API for controlling the app lifecycle and allows scripting of custom deploy and release policies. Marathon automatically responds to failures and makes sure your apps keep running forever. It’s the init.d for your data center.
In this talk, we’ll cover how Marathon solves these common devops concerns:
Deploying many apps on large clusters Service discovery Scaling apps with demand Providing developers with a self-serve interface for launching apps Automatically handling software, machine, and rack failures Reducing infrastructure complexity
Cloud computing is not a new paradigm and has been utilized by organization for a long time as private cloud computing. Outsourcing cloud computing has rapidly become more desirable due to higher availability and flexibility of compute resource that is cost effective. Organization are adopting hybrid model for running services, that is, utilizing private and public clouds for their work load. However, hybrid cloud adoption requires to rethink the deployment model; organizations split their data and applications between public and private cloud in unique ways and must solve multiple challenges. How do organization provide security between the realm of greater oversight and control in private cloud and the unknowns in the public cloud? Do the patterns for securing a service in a hybrid cloud deployment look different than used in traditional private cloud? This presentation will cover some of the challenges and patterns to make hybrid cloud secure and redundant.
Groupon recently completed a year-long project to migrate its U.S. web traffic from a monolithic Ruby on Rails application to a new multi-application stack with substantial results.
Groupon’s entire U.S. web frontend had been a single Rails codebase from its inception in 2008. The frontend codebase quickly grew large, which made it difficult to maintain and challenging to ship new
features. As a solution to this gigantic monolith, we decided to re-architect the frontend by splitting it into small, independent and more manageable pieces. At the center of this project, we rebuilt each
major section of the website as an independent application. We also rebuilt the infrastructure to make all the independent apps work together. Interaction Tier (I-Tier) was the result.
Learn about how Groupon achieved this great architecture migration and the business results it is driving.
All distributed systems are constrained by their ability to move data between components. We will talk about the engineering discipline required to achieve low latency and high throughput. Examples will be given from low latency C++ and high throughput python components used in SIG’s trading and data analysis stack. SIG is a proprietary trading firm headquartered in Philadelphia. No prior trading experience is assumed.
We are forced to solve many problems with respect to handling data. Many concepts work in our current models, many do not. Picking the wrong model can lead to massive amounts of accidental complexity.
“Polyglot data” should not be confused with “polyglot persistence.” The former refers to using multiple data representations in a single system (ie OLAP, Graph, Lucene, CEP…). Choosing ElasticSearch or
Command Sourcing with a CQRS implementation is different than choosing Redis for caching or Mongo for unstructured data. This talk will look at how to reach the point where you stop thinking about how to force
your problem into your predefined thinking and how to reach a place where you focus on how to choose the right model for the problem!