StackShare Editors

By early 2017, Airbnb was seeing around 75 million searches each day. In the midst of that growth, the team decided to rearchitect the web experience.

In the prior architecture, each page of the user’s journey, from searching, to viewing listings, to booking, was delivered standalone via Rails. The goal of the redesign was to maximize speed and fluidity.

In the new configuration, Hypernova is used to server-render React, and Redux is used for all API data and “globals,” like authentication state and experiment configurations.

Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

One of the earliest public references to Slack’s stack comes from a Twitter conversation. The Slack account states that “the messaging server is java, the app is php, db is mysql and solr for search,” and that uploaded files are “Stored on S3, but private files require authentication so requests go through the app.”

In 2014, PagerDuty struggled with safely releasing reliable mobile applications to users due to some issues with how the code was being packaged and handled.

PagerDuty’s mobile apps are hybrid and used Cordova to share code between platforms. Coding was straightforward but packaging was not, as a separated Gulp-based build process was also being used. The PagerDuty team took a page from Java and started creating software artifacts.

Rather than checking in transformed code or publishing modules to NPM, the team started creating zipped-up build artifacts, which coincided perfectly with GitHub's Releases feature which arrived in 2013. So despite JavaScript lacking a standard packaged app format like a JAR, PagerDuty was still able to improve the build times and sizes of their mobile apps.

The early data ingestion pipeline at Pinterest used Kafka as the central message transporter, with the app servers writing messages directly to Kafka, which then uploaded log files to S3.

For databases, a custom Hadoop streamer pulled database data and wrote it to S3.

Challenges cited for this infrastructure included high operational overhead, as well as potential data loss occurring when Kafka broker outages led to an overflow of in-memory message buffering.

From a StackShare community member: "We are looking to rewrite our outdated front-end with TypeScript. Right now we have a mix of CoffeeScript and vanilla JavaScript. I have read that adopting TypeScript can help enforce better code quality, and best practices. I also heard good things about Flow (JS). Which one would you recommend and why?"

From a StackShare Community member: “I’m a freelance web developer (I mostly use Node.js) and for future projects I’m debating between npm or Yarn as my default package manager. I’m a minimalist so I hate installing software if I don’t need to- in this case that would be Yarn. For those who made the switch from npm to Yarn, what benefits have you noticed? For those who stuck with npm, are you happy you with it?"

From a StackShare Community member: “We’re about to start a chat group for our open source project (over 5K stars on GitHub) so we can let our community collaborate more closely. The obvious choice would be Slack (k8s and a ton of major projects use it), but we’ve seen Gitter (webpack uses it) for a lot of open source projects, Discord (Vue.js moved to them), and as of late I’m seeing Spectrum more and more often. Does anyone have experience with these or other alternatives? Is it even worth assessing all these options, or should we just go with Slack? Some things that are important to us: free, all the regular integrations (GitHub, Heroku, etc), mobile & desktop apps, and open source is of course a plus."

The Slack desktop app was originally written us the MacGap framework, which used Apple’s WebView to host web content inside of a native app frame. As this approach continued to present product limitations, Slack decided to migrate the desktop app to Electron. Electron is a platform that combines the rendering engine from Chromium and the Node.js runtime and module system. The desktop app is written as a modern ES6 + async/await React application.

For the desktop app, Slack takes a hybrid approach, wherein some of the assets ship as part of the app, but most of their assets and code are loaded remotely.

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.