How Stream Built a Modern RSS Reader With JavaScript

13,383
Stream
Build scalable feeds, activity streams & chat in a few hours instead of months.

By Nick Parsons, Sr. Developer Advocate, Stream


Winds 2.0 by Stream


Winds started out as a simple example app for Stream, but thanks to an outpouring of support from our awesome community we decided to focus more time and energy on the project. The initial response around Winds 2.0 has exceeded all of our expectations. Since launching in mid-May the application ranked #1 on Hacker News for over a day, has 5,200 stars (and counting) on GitHub and became a trending app on Product Hunt.

Going into it, there was no way to anticipate how popular Winds 2.0 would become - would it be an absolute hit or an epic failure and waste of time?? The team enjoys building example apps so I knew it wouldn’t be a total loss, and it’s been rewarding to see this new iteration being used so extensively in the first month after release.

The tech stack for Winds is completely different from that of Stream. You may have seen the blogpost that StackShare wrote about how Stream powers the activity feeds for 300 million users using Go, RocksDB and Raft. Winds, however, is based on Node.js, MongoDB Atlas, Express, PM2, Bull, Babel and React.

To get started with Winds 2.0, you can try the web version or download the application here, or if you feel more adventurous head over to GitHub and spin it up locally. Next up, let’s talk a little bit about the Winds 2.0 stack and why we chose to go with the technologies we did (and why we chose to build Winds in the first place!).


RSS is a Broken Experience 😭

We realize that many RSS power users are developers, designers and journalists. One of our goals with Winds is to answer the questions we had been asking ourselves: What if a community of developers and designers could create an RSS experience that’s simplistic and polished? Could this reverse the downward spiral of less users taking advantage of the technology and more publications dropping support?

The future of RSS is uncertain at best. Our hope with this project is to make a contribution to #ReviveRSS.


Why JavaScript/Node…? 🤔

Another core goal for Winds is to enable a wide range of developers to contribute. We want it to be easy for anyone to be able to notice something they don’t like about their RSS/Podcast experience and easily submit a pull request with a fix or addition.

If you were brave enough to explore the codebase, you probably noticed that we’re using JavaScript for everything – both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this example app. What’s funny about JavaScript is how many people complain about it being an inadequate language. Sure, it has its quirks – single threaded, callback hell, etc.– but we believe that it’s possible to build great software in any language.

For Winds, JavaScript has been a great choice to foster a community around the project. More importantly, JavaScript’s maturity has started to shine with the added support of the Async/Await syntax.

Sure... there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on Hacker News about Node.js.); however, without writing completely in JavaScript, we would not have seen the results we did. Here’s a quick breakdown of some of the reasons why we chose JavaScript:

  • Nearly every developer knows or can, at the very least, read JavaScript
  • With ES6 and Node.js v10.x.x, it’s become a very capable language
  • Async/Await is powerful and easy to use (Async/Await vs Promises)
  • Babel allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet)
  • Yarn allows us to consistently install packages quickly (and is filled with tons of new tricks)


DevOps 🖥️

It’s rare that you hear about how a company deploys and manages code. Being that Winds 2.0 is open-source, we wanted to share a few of the tools we use to get the job done when it comes to getting our code from our machines up to the server.

The web version of Winds is statically hosted on S3 with CloudFront. In all, it costs us a few dollars a month to host. Every desktop version of the application is deployed inside of Electron, allowing us to bridge the gap between web and desktop.

As for the back-end API, that’s another story. We have a rather decent deploy flow going on to ensure stability and maintainability. Here’s the rundown:

  1. All code is stored on GitHub
  2. We manually kick off builds on AWS using a combination of Fabric and Boto
  3. CloudFormation create a fresh Winds environment consisting of EC2 instances, Auto Scaling Groups (ASG), Application Load Balancer (ELB), and a Redis instance
  4. AWS CCM stores and retrieves the various configurations required at boot (e.g. current version, etc.)
  5. Dotenv & Environment variables are stored in Puppet and CCM
  6. Once all EC2 instances are available, a Puppet script runs and applies the configuration on all live instances (in apply mode)
  7. PM2 boots, automatically starting the various Node.js processes we need to keep our application alive (API and Workers)

For logging metrics, we use a combination of StatsD + Graphite + Grafana.


Winds 2.0 by Stream


Understanding Electron ⚡

We wanted to experiment with building an Electron app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong.

Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with electron-builder (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult. Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on GitHub) and we released not only to all of our release targets but to the web, too.


Testing in JavaScript 🗳️

JavaScript is still the wild west to a degree. It’s rather un-opinionated, especially if you’re using Express, so we had to roll our own testing framework to get the job done. Our API, which is built with Express, uses a combination of various Node.js modules. Here’s a list of the tools we use for testing:

  • Mocha as a testing framework
  • Chai as an assertion library
  • Sinon as our mocking library
  • Nock as the HTTP mocking library
  • mock-require as a module mocking library
  • Istanbul as our test coverage tool

Bonus: Here’s an actual example of our test runner.

The combination of test modules we chose our team to move fast with multiple developers working on various feature sets at the same time, without bringing down the API.


Front End 🛠️

React is a phenomenal framework, and in our opinion, has won the battle against other frameworks such as Angular and Ember. Given its updated MIT license, it’s perfect for the Winds 2.0 project.

The main stack that we use for Winds 2.0 is pretty straightforward:

Main Stack

Now let’s look at some of the front-end modules we used to make Winds 2.0 a reality:

Interesting Modules


Back End 🛠️

When you’re building a large application, you generally rely on many libraries and tools to increase the code quality, time to market, etc. With that said, Winds too relies on many libraries and tools. Below are several, but not all, that we use:

FeedParser

FeedParser is a rather complex Node.js module that in our opinion, is the backbone of the project. It handles most of the inconsistencies found in RSS feeds and spits out a “cleansed” version of the feed. Without this module, we would be writing a lot of if/else statements… and that’s no fun.

Franc-Min

Franc-Min is a language detection module that we utilize for determining the language of a feed. This might sound like a small task; however, it’s in fact, a large part of our personalization engine. For example, we only recommend feeds in English to users who speak English. Same with other languages.

Bull

Bull helps keep the Winds 2.0 queue structurally sound with the help of Redis. It comes with a super easy API and supports multiple queues, which is a perfect fit for our use-case. Additionally, there are several open-source monitoring tools on their GitHub page that provide insight into what is happening behind the scenes.

ES6

JavaScript w/ ES6 enables our team to write minimalist code for a wide range of people. All of the code in Winds 2.0 is 100% JavaScript (with the exception of a handful of bash scripts to help with deploy workflows). The team is currently migrating much of the functionality in the codebase to utilize Async/Await to reduce the number of lines of code.

Yarn

Yarn is absolutely amazing. It’s an incredibly fast package manager built specifically for JavaScript. On top of that, it’s 100% open-source and nearly always available, due to it’s caching mechanisms. We’ve used npm in the past, and although it works just fine, the team here at Stream prefers Yarn.

Axios

Axios is a Promise based HTTP client for the browser and Node.js. We actually use it on both the front and back-end for various tasks. For example, all front-end HTTP requests flow through a wrapper to Axios. And for the back-end, we utilize Axios to inspect the size of the file prior to sending them through the parsing process and then off to the database for storage – this ensures large files don’t bring down our worker processes. If you haven’t checked out Axios, you definitely should.

Commander

Commander is another Node.js module, this time providing full support for building command-line interfaces. Yes, that’s right, Winds has a CLI that we use for various tasks such as testing feeds, dropping RSS feeds, and more!

Babel

Babel “allows us to use the next generation of JavaScript, today”. Essentially, if a feature, such as imports isn’t available in a particular JavaScript (front-end and/or back-end), we can still use it by leveraging Babel.

Express

Express is used to power our API. Compared to other frameworks out there, it truly shines when under stress. Honestly speaking, our team has experience with most, if not all, of the JavaScript frameworks and we find Express to be the easiest to work with. It’s regularly maintained, has open-source components, and it’s awesome. 😎

Sentry

Sentry allows for real-time crash reporting for our back- and front-end. What blows us away is how granular you can get with Sentry. Their features help us identify and debug errors and give us insight on when to fix or rollback. When it comes to firefighting, this tool definitely wins the market.

Algolia

Algolia provides lightning-fast (literally) search for our application. In under 2ms, our users can discover RSS feeds and podcasts to read or listen to. They even have components for frameworks such as React and Angular to make the integration better. This is by far one of our favorite; however, we like them all. 😛

Stream

Stream is a key resource to Winds 2.0, as it provides news feeds and activity streams for our users and even machine learning-based personalization. Without Stream, we would not be able to serve up suggested content to our users as we currently do.

MongoDB Atlas

MongoDB Atlas is a phenomenal DBaaS, allowing us to worry about acquiring users, while MongoDB worries about uptime. It’s identical to hosting your own cluster, except MongoDB provides you with a dashboard and a URI to connect to. With MongoDB Atlas, there’s no more worrying about cluster health, monitoring, etc.

Mongoose

Mongoose is a powerful ODM that allows us to define rich models within our MongoDB environment. Generally speaking, NoSQL databases are schemaless (meaning they have or require no form); however, with MongoDB, it’s always a good idea to specify a schema so you can index and organize your data properly. This allows for easy updates and efficient queries.

PM2

PM2 is a Node.js process manager. It allows us to ensure uptime, and scale processes as we need. The CLI for the project is dead simple, which allowed our team to pick it up on the fly.


Final Thoughts 😁

To get started with Winds 2.0, you can try the web version or download the application here. If you’re feeling more adventurous head over to GitHub and spin it up locally.

RSS is in a vicious circle. Winds is a community effort to help turn the tide and #ReviveRSS. Contributions are always much appreciated. To discuss new features check out the official Slack channel for Winds.

If you’re curious to know a little more about Stream and how our API works, we have an easy 5-minute API tour that will walk you through the process of building scalable activity feeds.

Stream
Build scalable feeds, activity streams & chat in a few hours instead of months.
Tools mentioned in article
Open jobs at Stream
Staff Backend Engineer (Golang)
Amsterdam or (EU and UK)
<section> <section> <p>We are seeking a skilled Staff Software Developer to join our team. This role is open remotely (if you're EU or UK-based) or in our Amsterdam office (relocation is possible).</p> <h2><strong>What you will be doing</strong>&nbsp;</h2> <p>You'll focus on one of the most used Products: Chat, as well as the brand new Video &amp; Audio API.</p> <p>A big portion of your day will be dedicated to software design, research, and coding.</p> <p>On typical projects, you will have a lot of freedom and you will be paired with another team member.</p> <p>Our team is made up of very experienced engineers, some with more than 10 years of experience.</p> <p>By working together you will learn from each other along the way. Not only that, you will have an enormous impact!</p> <p>Our customers have millions of users; they use Stream for mission-critical features such as showing content and exposing the core functionality of their application.</p> <h2><strong>Responsibilities</strong></h2> <ul> <li>Add new features to the service and find ways to make the existing ones perform orders of magnitude faster</li> <li>Help make our API service faster, more scalable and more flexible.</li> <li>Write clean, efficient, and well-documented code</li> <li>Design and write state-of-the-art software, follow best practices, measure everything and be responsible for deployment to production</li> <li>Engage with customers and help them to use Stream in their app</li> </ul> <p><strong>The challenges:</strong></p> <ul> <li><span style="text-decoration: underline;">Distributed databases:</span> we built our own data store for feeds and for chat</li> <li><span style="text-decoration: underline;">Real-time messaging</span></li> <li><span style="text-decoration: underline;">High performance: </span>our API responses are in the 10ms range</li> <li><span style="text-decoration: underline;">High scalability:</span> we use sharding, master-master, and master-slave to ensure scalability</li> <li><span style="text-decoration: underline;">High availability:</span> our entire infrastructure is designed and operated to survive entire datacenter crashes</li> <li><span style="text-decoration: underline;">Multi-region:</span> we deploy our service on 4 different continents</li> </ul> <h2><strong>About you</strong><span style="font-weight: 400;">&nbsp;</span></h2> <p><strong>You have:</strong></p> <ul> <li data-stringify-indent="0" data-stringify-border="0">7+ years of backend development experience</li> <li data-stringify-indent="0" data-stringify-border="0">Proficiency in Go language is strongly preferred</li> <li data-stringify-indent="0" data-stringify-border="0">Experience with high traffic and high performance applications</li> <li data-stringify-indent="0" data-stringify-border="0">Solid knowledge of relational databases</li> <li data-stringify-indent="0" data-stringify-border="0">Experience with building HTTP APIs</li> <li data-stringify-indent="0" data-stringify-border="0">Experience managing your own projects and work in a team</li> </ul> <p><strong>Bonus points:</strong></p> <ul> <li>Experience with JavaScript and web development frameworks such as React or Angular</li> <li>Experience with one (preferably few) of the following languages: Rust, Java, C, C++, Erlang, Node.js, Python</li> <li>Experience with message queues such as RabbitMQ</li> <li>Experience with automated testing and continuous integration/continuous deployment (CI/CD)</li> <li>Experience with designing and building REST API’s</li> <li>Experience with cloud-based platforms (e.g. AWS, Azure)</li> <li>Bachelor's degree in Computer Science, Engineering or a related field.<br><br></li> </ul> <p><strong>Our tech stack:</strong></p> <p>At Stream we use a wide collection of technologies to offer highly optimised and available features to our customers. Over the years we have experimented with different programming languages, frameworks, databases, and libraries.</p> <p>Here is a short list of the technology that we currently use.</p> <p>Do not worry if you do not master them all or if you do not see your favourite tool or language, you will have the chance to be exposed to most and to convince us to expand the list:</p> <ul> <li>Go, Python, NodeJS&nbsp;</li> <li>Postgresql, CockroachDB&nbsp;</li> <li>AWS, Puppet, CloudFormation&nbsp;</li> <li>Grafana, Graphite, ELK, Jaeger&nbsp;</li> <li>Redis, Memcached</li> </ul> <h2><strong>Why join Stream?</strong></h2> <ul> <li style="font-weight: 400;"><strong>History of success.</strong><span style="font-weight: 400;"> From Amsterdam to Boulder and Techstars in-between, Stream has raised over $58.25M to build the best Chat Messaging &amp; Activity Feed infrastructure available, with best-in-class support.</span></li> <li style="font-weight: 400;"><strong>Freedom and endless growth opportunities.</strong><span style="font-weight: 400;"> As a rapidly growing startup (since 2020 we have gone from 30 to 150 employees), Stream gives you unique personal and professional growth opportunities. The opportunity of true ownership and accountability has a massive impact on your career. These are the things you can rarely experience in huge corporations.</span></li> <li style="font-weight: 400;"><strong>Be on the front line of progress and innovation.</strong><span style="font-weight: 400;"> While working with cutting-edge technology, we are passionate about tackling difficult tech problems at scale and creating reusable components for them, empowering engineering teams to ship apps faster, more securely, and with a better user experience.</span></li> <li style="font-weight: 400;"><strong>They believe in us:</strong><span style="font-weight: 400;"> Stream is backed by leading VC companies (Felicis Ventures, GGV Capital, 01.Advisors, Techstars, Arthur Ventures), including backers like Dick Costolo (01 Advisors, ex-CEO of Twitter), Olivier Pomel (CEO of Datadog), Tom Preston-Werner (Co-Founder of GitHub), Nicolas Dessaigne (Co-Founder of Algolia), Johnny Boufarhat (Founder and CEO of Hopin).</span></li> </ul> <h2><strong>What we have to offer you</strong></h2> <p><span style="font-weight: 400;">Stream employees enjoy some of the best benefits in the industry:</span></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">A team of exceptional engineers&nbsp;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">The chance to work on OSS projects </span><strong><em>&nbsp;</em></strong></li> <li style="font-weight: 400;"><span style="font-weight: 400;">28 days paid time off plus paid Dutch holidays</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Company equity</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">A pension scheme</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Remote work flexibility</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">A Learning and Development budget</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Commute expenses to Amsterdam covered or the option to use a company bike within the city</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Fitness stipend&nbsp;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Monthly in-office chair massages by a professional</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">MacBook Pro&nbsp;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Healthy team lunches and plenty of snacks</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">A generous relocation package</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">An office in the heart of Amsterdam</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">The opportunity to attend or present at global conferences and meetups</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">The possibility to visit our office in Boulder, CO</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Parental leave paid at 100%</span></li> </ul> <p><strong><span class="discussion-id-822efe3d-614e-4adc-9991-ea81cdc89918 notion-enable-hover" data-token-index="0" data-reactroot="">Note: </span><span class="discussion-id-822efe3d-614e-4adc-9991-ea81cdc89918 notion-enable-hover" data-token-index="1" data-reactroot="">this list of benefits applies to Netherlands-based employees and is adjusted per your location of residence</span><span class="notion-enable-hover" data-token-index="2" data-reactroot="">.</span></strong></p> <h2><strong>Our culture</strong></h2> <p><span style="font-weight: 400;">Stream has a casual social culture, our team is diverse and we all have different backgrounds. Now, Stream is a team of over 130+ peers from over 35 countries across the globe.</span></p> <p><span style="font-weight: 400;">We value transparency, aim for excellence, and support each other on our way to new victories.</span></p> <p><span style="font-weight: 400;">Our team consists of the strongest talents worldwide, making Stream a great place to learn and improve your skills.&nbsp;</span></p> <p><span style="font-weight: 400;">When it comes to software engineering, our culture is oriented towards ownership and quality: our goal is to deliver stable software.</span></p> <p><span style="font-weight: 400;">If you are interested in becoming a part of what we do, apply now!</span></p> <p><em><span style="font-weight: 400;">Stream provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.</span></em></p> <p><em><span style="font-weight: 400;">This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.</span></em></p> <p><strong><em>No recruiters/agencies please</em></strong></p> </section> <p>&nbsp;</p> </section>
Verified by
Building cool things on the internet 🛠️
Software Engineer, Data Science
Account Executive
Software Engineer
Software Architect
Javascript Developer
Marketing Manager
You may also like