How Stream Built a Modern RSS Reader With JavaScript

12,311
Stream
Build scalable feeds, activity streams & chat in a few hours instead of months.

By Nick Parsons, Sr. Developer Advocate, Stream


Winds 2.0 by Stream


Winds started out as a simple example app for Stream, but thanks to an outpouring of support from our awesome community we decided to focus more time and energy on the project. The initial response around Winds 2.0 has exceeded all of our expectations. Since launching in mid-May the application ranked #1 on Hacker News for over a day, has 5,200 stars (and counting) on GitHub and became a trending app on Product Hunt.

Going into it, there was no way to anticipate how popular Winds 2.0 would become - would it be an absolute hit or an epic failure and waste of time?? The team enjoys building example apps so I knew it wouldn’t be a total loss, and it’s been rewarding to see this new iteration being used so extensively in the first month after release.

The tech stack for Winds is completely different from that of Stream. You may have seen the blogpost that StackShare wrote about how Stream powers the activity feeds for 300 million users using Go, RocksDB and Raft. Winds, however, is based on Node.js, MongoDB Atlas, Express, PM2, Bull, Babel and React.

To get started with Winds 2.0, you can try the web version or download the application here, or if you feel more adventurous head over to GitHub and spin it up locally. Next up, let’s talk a little bit about the Winds 2.0 stack and why we chose to go with the technologies we did (and why we chose to build Winds in the first place!).


RSS is a Broken Experience 😭

We realize that many RSS power users are developers, designers and journalists. One of our goals with Winds is to answer the questions we had been asking ourselves: What if a community of developers and designers could create an RSS experience that’s simplistic and polished? Could this reverse the downward spiral of less users taking advantage of the technology and more publications dropping support?

The future of RSS is uncertain at best. Our hope with this project is to make a contribution to #ReviveRSS.


Why JavaScript/Node…? 🤔

Another core goal for Winds is to enable a wide range of developers to contribute. We want it to be easy for anyone to be able to notice something they don’t like about their RSS/Podcast experience and easily submit a pull request with a fix or addition.

If you were brave enough to explore the codebase, you probably noticed that we’re using JavaScript for everything – both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this example app. What’s funny about JavaScript is how many people complain about it being an inadequate language. Sure, it has its quirks – single threaded, callback hell, etc.– but we believe that it’s possible to build great software in any language.

For Winds, JavaScript has been a great choice to foster a community around the project. More importantly, JavaScript’s maturity has started to shine with the added support of the Async/Await syntax.

Sure... there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on Hacker News about Node.js.); however, without writing completely in JavaScript, we would not have seen the results we did. Here’s a quick breakdown of some of the reasons why we chose JavaScript:

  • Nearly every developer knows or can, at the very least, read JavaScript
  • With ES6 and Node.js v10.x.x, it’s become a very capable language
  • Async/Await is powerful and easy to use (Async/Await vs Promises)
  • Babel allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet)
  • Yarn allows us to consistently install packages quickly (and is filled with tons of new tricks)


DevOps 🖥️

It’s rare that you hear about how a company deploys and manages code. Being that Winds 2.0 is open-source, we wanted to share a few of the tools we use to get the job done when it comes to getting our code from our machines up to the server.

The web version of Winds is statically hosted on S3 with CloudFront. In all, it costs us a few dollars a month to host. Every desktop version of the application is deployed inside of Electron, allowing us to bridge the gap between web and desktop.

As for the back-end API, that’s another story. We have a rather decent deploy flow going on to ensure stability and maintainability. Here’s the rundown:

  1. All code is stored on GitHub
  2. We manually kick off builds on AWS using a combination of Fabric and Boto
  3. CloudFormation create a fresh Winds environment consisting of EC2 instances, Auto Scaling Groups (ASG), Application Load Balancer (ELB), and a Redis instance
  4. AWS CCM stores and retrieves the various configurations required at boot (e.g. current version, etc.)
  5. Dotenv & Environment variables are stored in Puppet and CCM
  6. Once all EC2 instances are available, a Puppet script runs and applies the configuration on all live instances (in apply mode)
  7. PM2 boots, automatically starting the various Node.js processes we need to keep our application alive (API and Workers)

For logging metrics, we use a combination of StatsD + Graphite + Grafana.


Winds 2.0 by Stream


Understanding Electron ⚡

We wanted to experiment with building an Electron app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong.

Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with electron-builder (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult. Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on GitHub) and we released not only to all of our release targets but to the web, too.


Testing in JavaScript 🗳️

JavaScript is still the wild west to a degree. It’s rather un-opinionated, especially if you’re using Express, so we had to roll our own testing framework to get the job done. Our API, which is built with Express, uses a combination of various Node.js modules. Here’s a list of the tools we use for testing:

  • Mocha as a testing framework
  • Chai as an assertion library
  • Sinon as our mocking library
  • Nock as the HTTP mocking library
  • mock-require as a module mocking library
  • Istanbul as our test coverage tool

Bonus: Here’s an actual example of our test runner.

The combination of test modules we chose our team to move fast with multiple developers working on various feature sets at the same time, without bringing down the API.


Front End 🛠️

React is a phenomenal framework, and in our opinion, has won the battle against other frameworks such as Angular and Ember. Given its updated MIT license, it’s perfect for the Winds 2.0 project.

The main stack that we use for Winds 2.0 is pretty straightforward:

Main Stack

Now let’s look at some of the front-end modules we used to make Winds 2.0 a reality:

Interesting Modules


Back End 🛠️

When you’re building a large application, you generally rely on many libraries and tools to increase the code quality, time to market, etc. With that said, Winds too relies on many libraries and tools. Below are several, but not all, that we use:

FeedParser

FeedParser is a rather complex Node.js module that in our opinion, is the backbone of the project. It handles most of the inconsistencies found in RSS feeds and spits out a “cleansed” version of the feed. Without this module, we would be writing a lot of if/else statements… and that’s no fun.

Franc-Min

Franc-Min is a language detection module that we utilize for determining the language of a feed. This might sound like a small task; however, it’s in fact, a large part of our personalization engine. For example, we only recommend feeds in English to users who speak English. Same with other languages.

Bull

Bull helps keep the Winds 2.0 queue structurally sound with the help of Redis. It comes with a super easy API and supports multiple queues, which is a perfect fit for our use-case. Additionally, there are several open-source monitoring tools on their GitHub page that provide insight into what is happening behind the scenes.

ES6

JavaScript w/ ES6 enables our team to write minimalist code for a wide range of people. All of the code in Winds 2.0 is 100% JavaScript (with the exception of a handful of bash scripts to help with deploy workflows). The team is currently migrating much of the functionality in the codebase to utilize Async/Await to reduce the number of lines of code.

Yarn

Yarn is absolutely amazing. It’s an incredibly fast package manager built specifically for JavaScript. On top of that, it’s 100% open-source and nearly always available, due to it’s caching mechanisms. We’ve used npm in the past, and although it works just fine, the team here at Stream prefers Yarn.

Axios

Axios is a Promise based HTTP client for the browser and Node.js. We actually use it on both the front and back-end for various tasks. For example, all front-end HTTP requests flow through a wrapper to Axios. And for the back-end, we utilize Axios to inspect the size of the file prior to sending them through the parsing process and then off to the database for storage – this ensures large files don’t bring down our worker processes. If you haven’t checked out Axios, you definitely should.

Commander

Commander is another Node.js module, this time providing full support for building command-line interfaces. Yes, that’s right, Winds has a CLI that we use for various tasks such as testing feeds, dropping RSS feeds, and more!

Babel

Babel “allows us to use the next generation of JavaScript, today”. Essentially, if a feature, such as imports isn’t available in a particular JavaScript (front-end and/or back-end), we can still use it by leveraging Babel.

Express

Express is used to power our API. Compared to other frameworks out there, it truly shines when under stress. Honestly speaking, our team has experience with most, if not all, of the JavaScript frameworks and we find Express to be the easiest to work with. It’s regularly maintained, has open-source components, and it’s awesome. 😎

Sentry

Sentry allows for real-time crash reporting for our back- and front-end. What blows us away is how granular you can get with Sentry. Their features help us identify and debug errors and give us insight on when to fix or rollback. When it comes to firefighting, this tool definitely wins the market.

Algolia

Algolia provides lightning-fast (literally) search for our application. In under 2ms, our users can discover RSS feeds and podcasts to read or listen to. They even have components for frameworks such as React and Angular to make the integration better. This is by far one of our favorite; however, we like them all. 😛

Stream

Stream is a key resource to Winds 2.0, as it provides news feeds and activity streams for our users and even machine learning-based personalization. Without Stream, we would not be able to serve up suggested content to our users as we currently do.

MongoDB Atlas

MongoDB Atlas is a phenomenal DBaaS, allowing us to worry about acquiring users, while MongoDB worries about uptime. It’s identical to hosting your own cluster, except MongoDB provides you with a dashboard and a URI to connect to. With MongoDB Atlas, there’s no more worrying about cluster health, monitoring, etc.

Mongoose

Mongoose is a powerful ODM that allows us to define rich models within our MongoDB environment. Generally speaking, NoSQL databases are schemaless (meaning they have or require no form); however, with MongoDB, it’s always a good idea to specify a schema so you can index and organize your data properly. This allows for easy updates and efficient queries.

PM2

PM2 is a Node.js process manager. It allows us to ensure uptime, and scale processes as we need. The CLI for the project is dead simple, which allowed our team to pick it up on the fly.


Final Thoughts 😁

To get started with Winds 2.0, you can try the web version or download the application here. If you’re feeling more adventurous head over to GitHub and spin it up locally.

RSS is in a vicious circle. Winds is a community effort to help turn the tide and #ReviveRSS. Contributions are always much appreciated. To discuss new features check out the official Slack channel for Winds.

If you’re curious to know a little more about Stream and how our API works, we have an easy 5-minute API tour that will walk you through the process of building scalable activity feeds.

Stream
Build scalable feeds, activity streams & chat in a few hours instead of months.
Tools mentioned in article
Open jobs at Stream
Backend Software Developer
Amsterdam;

We are looking for a full time Senior Software Engineer to join our development team. Job duties will include working on Stream's core API technology as well as designing and building high-performance software.

What you will be doing

Most of your day will be dedicated to software design, research, and coding. On typical projects, you will have a lot of freedom and you will be paired with another team member. Our team is made up of very experienced engineers, some with more than 10 years of experience. By working together you will learn from each other along the way. You will have an enormous impact on making our API service faster, more scalable and more flexible.

You will add new features to the service and find ways to make the existing ones perform orders of magnitude faster. Our customers have millions of users; they use Stream for mission critical features such as showing content and exposing core functionality of their application. Building stable and reliable software is not just an option: as a member of the development team, you will design and write state-of-the-art software, follow best practices, measure everything and be responsible for deployment to production. You will also spend part of your time talking to our customers and help them to use Stream in their app.

The challenges

  • Distributed databases: we built our own data store for feeds and for chat
  • Real-time messaging
  • High performance: our API responses are in the 10ms range
  • High scalability: we use sharding, master-master, and master-slave to ensure scalability
  • High availability: our entire infrastructure is designed and operated to survive entire datacenter crashes
  • Multi-region: we deploy our service on 4 different continents

You have

  • 5+ years of backend development experience
  • Experience with at least one (preferably few) of the following languages: Go, Rust, Java, C, C++, Erlang
  • Proficiency in Go language is strongly preferred
  • Experience with high traffic and high performance applications
  • Solid knowledge of relational databases
  • Experience with building HTTP APIs
  • Experience managing your own projects and work in a team

Our tech stack

At Stream we use a wide collection of technologies to offer highly optimized and available features to our customers. Over the years we have experimented with different programming languages, frameworks, databases, and libraries. Here is a short list of the technology that we currently use. Do not worry if you do not master them all or if you do not see your favorite tool or language, you will have the chance to be exposed to most and to convince us to expand the list:

  • Go, gRPC, RocksDB, Python
  • Postgresql, RabbitMQ
  • AWS, Puppet, CloudFormation
  • Grafana, Graphite, ELK, Jaeger
  • Redis, Memcached

What we have to offer you

Stream employees enjoy some of the best benefits in the industry:

  • A team of exceptional engineers
  • The chance to work on OSS projects
  • A competitive salary
  • 28 days paid time off plus paid Dutch holidays 
  • Company equity
  • A pension scheme
  • A generous Learning and Development budget
  • Commute expenses to Amsterdam covered or option to use a company bike within the city
  • Gym membership of choice covered and weekly pilates sessions in the office
  • MacBook Pro or another development setup
  • Healthy team lunches and plenty of snacks
  • A generous relocation package
  • An office in the heart of Amsterdam
  • The opportunity to attend or present to global conferences and meetups
  • The possibility to visit our office in Boulder, CO

Our culture

Stream has a casual social culture, our team is diverse and we all have different backgrounds.

Our talented developers are highly technical and collaborative, which makes Stream a great place to learn and improve your skills. When it comes to software engineering our culture is oriented towards ownership and quality: our goal is to deliver stable software.

If you are interested in becoming a part of what we do, apply now!

No recruiters/agencies please

VP of Engineering
Amsterdam,or

We are looking for a highly skilled VP of Engineering to lead our technical execution and objectives. Between our Feed and Chat products, Stream provides services for more than a billion end users.

What you will be doing

You will work directly with CEO and CTO and be responsible for the execution and delivery of our technical objectives. Our engineering team operates a large infrastructure deployed in 10+ regions, we build SDKs for more than 10 different platforms and the API serves traffic for over a billion end-users.

Why Join Stream as VPE

We want to become the AWS of application components, we currently power Chat and Feeds for over a billion end-users. Our API is used by applications with hundreds of millions of users.

  • Large engineering team ~ 50% of the entire company is part of the engineering department
  • We build and release SDK on all major platforms
  • Modern tech stack: we use Golang, RocksDB, Raft
  • The engineering team is very solid

Responsibilities

  • Own execution and delivery of all technical objectives
  • Hiring and growing the best talent
  • Lead planning, prioritization, and design of new features for API, SDK, and other tech teams
  • Report directly to CTO and work closely with CEO
  • Budget and organize all hiring for the engineering department

About You

You are a VP of Engineering, CTO, or technical Co-Founder with more than 10 years of experience in software development.

As a manager, you maintain a hands-on approach and stay involved with technical challenges, you are passionate about building software and technical teams.

You have

  • Prior experience as VP of engineering, CTO, or as a technical Co-Founder
  • More than 8 years of experience managing software engineering teams
  • Experience managing 30+ developers
  • Hired top talent and grew them under your management
  • CS degree or similar/relevant experience
  • Worked for more than 5 years as a software engineer

Bonus points

  • Startup experience
  • Saas & API products
  • Mobile development background

Our tech stack (for dev positions / engineering dept. roles)

At Stream we use a wide collection of technologies to offer highly optimized and available features to our customers. Here is a shortlist of the technology that we currently use:

  • Go, gRPC, RocksDB, Python
  • Postgresql, RabbitMQ
  • AWS, Puppet, CloudFormation
  • Grafana, Graphite, ELK, Jaeger
  • Redis, Memcached

What we have to offer you

Stream employees enjoy some of the best benefits in the industry:

  • A team of exceptional engineers (developer positions only)
  • The chance to work on OSS projects (developer positions only)
  • A competitive salary
  • 28 days of paid time off plus paid Dutch holidays
  • Company equity
  • A pension scheme
  • A generous Learning and Development budget
  • Commute expenses to Amsterdam covered or the option to use a company bike within the city
  • Gym membership of choice covered and weekly pilates sessions in the office
  • A Macbook Pro and, if necessary, a home office setup package
  • Healthy team lunches and plenty of snacks
  • A generous relocation package
  • An office in the heart of Amsterdam or the possibility to work fully remote
  • The opportunity to attend or present to global conferences and meetups
  • The possibility to visit our office in Boulder, CO

Our culture

Stream has a casual social culture, our team is diverse and we all have different backgrounds. Our talented developers are highly technical and collaborative, which makes Stream a great place to learn and improve your skills. When it comes to software engineering our culture is oriented towards ownership and quality: our goal is to deliver stable software.

If you are interested in becoming a part of what we do, apply now!

No recruiters/agencies please

Technical Director of Engineering
Amsterdam, or

We are looking for a highly skilled technical director of engineering to lead and participate to the execution of our technical roadmap. Between our Feed and Chat products, Stream provides services for more than a billion end users. Internally we have many more technical areas, services and small products.

What you will be doing

You will report to CTO and be responsible for the execution of API products. Inside our engineering, the first two levels of management are still hands-on. You will divide your time between managing projects, and managers as well as working directly on our API product and infrastructure.

Why Join Stream as Technical Director of Engineering

We want to become the AWS of application components, we currently power Chat and Feeds for over a billion end-users. Our API is used by applications with hundreds of millions of users.

  • Large engineering team ~ 50% of the entire company is part of the engineering department
  • Modern tech stack: we use Golang, RocksDB, Raft, Protobuf, Prometheus
  • The engineering team is very solid
  • Stay hands-on

Responsibilities

  • Execution and delivery of API projects
  • Manage a team of developers and team leads
  • Hiring and growing the best talent
  • Lead planning, prioritization, and design of new API features

About You

You worked as a director, CTO, technical Co-Founder, tech lead, or team lead in the past and have several years of experience in software development.

As a manager, you maintain a hands-on approach and stay involved with technical challenges, you are passionate about building software and technical teams.

You have

  • More than 5 years of experience managing software engineering teams
  • Experience managing developers and leading complex technical projects
  • Hired top talent and grew them under your management
  • CS degree or similar/relevant experience
  • Worked for more than 8 years as a software engineer

Bonus points

  • Startup experience
  • Prior experience as CTO or as a technical Co-Founder
  • Saas & API products
  • Mobile development background

Our tech stack

At Stream we use a wide collection of technologies to offer highly optimized and available features to our customers. Here is a shortlist of the technology that we currently use:

  • Go, gRPC, RocksDB, Python
  • Postgresql, RabbitMQ
  • AWS, Puppet, CloudFormation
  • Grafana, Graphite, ELK, Jaeger
  • Redis, Memcached

What we have to offer you

Stream employees enjoy some of the best benefits in the industry:

  • A team of exceptional engineers (developer positions only)
  • The chance to work on OSS projects (developer positions only)
  • A competitive salary
  • 28 days of paid time off plus paid Dutch holidays
  • Company equity
  • A pension scheme
  • A generous Learning and Development budget
  • Commute expenses to Amsterdam covered or the option to use a company bike within the city
  • Gym membership of choice covered and weekly pilates sessions in the office
  • A Macbook Pro and, if necessary, a home office setup package
  • Healthy team lunches and plenty of snacks
  • A generous relocation package
  • An office in the heart of Amsterdam or the possibility to work fully remote
  • The opportunity to attend or present to global conferences and meetups
  • The possibility to visit our office in Boulder, CO

Our culture

Stream has a casual social culture, our team is diverse and we all have different backgrounds. Our talented developers are highly technical and collaborative, which makes Stream a great place to learn and improve your skills. When it comes to software engineering our culture is oriented towards ownership and quality: our goal is to deliver stable software.

If you are interested in becoming a part of what we do, apply now!

Stream provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.

No recruiters/agencies please

Senior iOS SDK Developer
Boulder, CO or

For our Chat SDK team we are seeking a Senior iOS SDK Developer to work on our native iOS SDK libraries. This role is 100% remote, preferably in American time zones (MST, ET or close).

What you will be doing

As an iOS SDK Developer, you strive to simplify how applications are built. You will lead by example, raise the team’s technological proficiency, and influence the rest of the team towards best practices and mature technological choices.

You will write and maintain our open-source SDKs that are used by hundreds of our customers and consumed by hundreds of millions of their users.

Most of your day will be dedicated to software design, research, and coding. You will work with modern iOS technologies, such as Swift, SwiftUI, and Combine, with a heavy focus on code quality, API design, testing, and CI/CD processes.

You have

  • Passion for writing simple and testable code, and for designing APIs meant to be used by other developers
  • Deep knowledge of UIKit, experience with SwiftUI, modular UI elements, and design systems
  • Proficiency in Swift and the iOS development ecosystem
  • An enthusiasm for all kinds of iOS testing: unit, integration, UI, snapshot tests

Bonus points

  • Knowledge of CoreData
  • Experience with developing SDK libraries or maintaining open-source projects
  • Familiarity with reactive patterns, experience with RxSwift or Combine
  • Experience with Fastlane and CI/CD best practices

What we have to offer you

Stream employees enjoy some of the best benefits in the industry:

    • A team of exceptional engineers 
    • The chance to work on OSS projects 
    • A combination of 36 days per year in PTO and public holidays
    • Company equity
    • Remote work flexibility
    • Fitness stipend
    • Lunch budget of €25 per week
    • A Macbook Pro and work-from-home stipend
    • The opportunity to attend or present to global conferences and meetups
    • The possibility to visit our offices in Boulder, CO and Amsterdam, NL
    • Legal employment in your country of residence

Our culture

Stream has a casual social culture, our team is diverse and we all have different backgrounds. Now, Stream is a team of over 140+ peers from over 35 countries across the globe.

We value transparency, aim for excellence, and support each other on our way to new victories.

Our team consists of the strongest talents all over the world, which makes Stream a great place to learn and improve your skills.

When it comes to software engineering our culture is oriented towards ownership and quality: our goal is to deliver stable software.

If you are interested in becoming a part of what we do, apply now!

Salary: Our salary ranges are based on national averages. We have wide ranges so we can be flexible and determine compensation based on a number of factors including the candidate's skills, level of experience, and location (which can be US or a remote location that aligns with MST time zone). For the US based candidates, the salary range for this position is up to $110,000. Compensation at all other locations will be based on the factors as stated above.

Stream provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.

 

Verified by
Building cool things on the internet 🛠️
Software Engineer
Software Engineer, Data Science
Javascript Developer
Software Architect
Account Executive
You may also like