By Nick Parsons, Sr. Developer Advocate, Stream
Winds started out as a simple example app for Stream, but thanks to an outpouring of support from our awesome community we decided to focus more time and energy on the project. The initial response around Winds 2.0 has exceeded all of our expectations. Since launching in mid-May the application ranked #1 on Hacker News for over a day, has 5,200 stars (and counting) on GitHub and became a trending app on Product Hunt.
Going into it, there was no way to anticipate how popular Winds 2.0 would become - would it be an absolute hit or an epic failure and waste of time?? The team enjoys building example apps so I knew it wouldn’t be a total loss, and it’s been rewarding to see this new iteration being used so extensively in the first month after release.
The tech stack for Winds is completely different from that of Stream. You may have seen the blogpost that StackShare wrote about how Stream powers the activity feeds for 300 million users using Go, RocksDB and Raft. Winds, however, is based on Node.js, MongoDB Atlas, Express, PM2, Bull, Babel and React.
To get started with Winds 2.0, you can try the web version or download the application here, or if you feel more adventurous head over to GitHub and spin it up locally. Next up, let’s talk a little bit about the Winds 2.0 stack and why we chose to go with the technologies we did (and why we chose to build Winds in the first place!).
RSS is a Broken Experience 😭
We realize that many RSS power users are developers, designers and journalists. One of our goals with Winds is to answer the questions we had been asking ourselves: What if a community of developers and designers could create an RSS experience that’s simplistic and polished? Could this reverse the downward spiral of less users taking advantage of the technology and more publications dropping support?
The future of RSS is uncertain at best. Our hope with this project is to make a contribution to #ReviveRSS.
Another core goal for Winds is to enable a wide range of developers to contribute. We want it to be easy for anyone to be able to notice something they don’t like about their RSS/Podcast experience and easily submit a pull request with a fix or addition.
- With ES6 and Node.js v10.x.x, it’s become a very capable language
- Async/Await is powerful and easy to use (Async/Await vs Promises)
- Yarn allows us to consistently install packages quickly (and is filled with tons of new tricks)
It’s rare that you hear about how a company deploys and manages code. Being that Winds 2.0 is open-source, we wanted to share a few of the tools we use to get the job done when it comes to getting our code from our machines up to the server.
The web version of Winds is statically hosted on S3 with CloudFront. In all, it costs us a few dollars a month to host. Every desktop version of the application is deployed inside of Electron, allowing us to bridge the gap between web and desktop.
As for the back-end API, that’s another story. We have a rather decent deploy flow going on to ensure stability and maintainability. Here’s the rundown:
- All code is stored on GitHub
- We manually kick off builds on AWS using a combination of Fabric and Boto
- CloudFormation create a fresh Winds environment consisting of EC2 instances, Auto Scaling Groups (ASG), Application Load Balancer (ELB), and a Redis instance
- AWS CCM stores and retrieves the various configurations required at boot (e.g. current version, etc.)
- Dotenv & Environment variables are stored in Puppet and CCM
- Once all EC2 instances are available, a Puppet script runs and applies the configuration on all live instances (in apply mode)
- PM2 boots, automatically starting the various Node.js processes we need to keep our application alive (API and Workers)
Understanding Electron ⚡
We wanted to experiment with building an Electron app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong.
Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with electron-builder (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult. Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on GitHub) and we released not only to all of our release targets but to the web, too.
- Mocha as a testing framework
- Chai as an assertion library
- Sinon as our mocking library
- Nock as the HTTP mocking library
- mock-require as a module mocking library
- Istanbul as our test coverage tool
Bonus: Here’s an actual example of our test runner.
The combination of test modules we chose our team to move fast with multiple developers working on various feature sets at the same time, without bringing down the API.
Front End 🛠️
React is a phenomenal framework, and in our opinion, has won the battle against other frameworks such as Angular and Ember. Given its updated MIT license, it’s perfect for the Winds 2.0 project.
The main stack that we use for Winds 2.0 is pretty straightforward:
Now let’s look at some of the front-end modules we used to make Winds 2.0 a reality:
- React-audio-player is a nice React interface to the core audio element API
- React-waypoint for scrolling events to handle automatic pagination
- React-dropzone for easy OPML file imports
- React-image for flawless image fallbacks in the event we don’t have an image stored
Back End 🛠️
When you’re building a large application, you generally rely on many libraries and tools to increase the code quality, time to market, etc. With that said, Winds too relies on many libraries and tools. Below are several, but not all, that we use:
FeedParser is a rather complex Node.js module that in our opinion, is the backbone of the project. It handles most of the inconsistencies found in RSS feeds and spits out a “cleansed” version of the feed. Without this module, we would be writing a lot of if/else statements… and that’s no fun.
Franc-Min is a language detection module that we utilize for determining the language of a feed. This might sound like a small task; however, it’s in fact, a large part of our personalization engine. For example, we only recommend feeds in English to users who speak English. Same with other languages.
Bull helps keep the Winds 2.0 queue structurally sound with the help of Redis. It comes with a super easy API and supports multiple queues, which is a perfect fit for our use-case. Additionally, there are several open-source monitoring tools on their GitHub page that provide insight into what is happening behind the scenes.
Axios is a Promise based HTTP client for the browser and Node.js. We actually use it on both the front and back-end for various tasks. For example, all front-end HTTP requests flow through a wrapper to Axios. And for the back-end, we utilize Axios to inspect the size of the file prior to sending them through the parsing process and then off to the database for storage – this ensures large files don’t bring down our worker processes. If you haven’t checked out Axios, you definitely should.
Commander is another Node.js module, this time providing full support for building command-line interfaces. Yes, that’s right, Winds has a CLI that we use for various tasks such as testing feeds, dropping RSS feeds, and more!
Sentry allows for real-time crash reporting for our back- and front-end. What blows us away is how granular you can get with Sentry. Their features help us identify and debug errors and give us insight on when to fix or rollback. When it comes to firefighting, this tool definitely wins the market.
Algolia provides lightning-fast (literally) search for our application. In under 2ms, our users can discover RSS feeds and podcasts to read or listen to. They even have components for frameworks such as React and Angular to make the integration better. This is by far one of our favorite; however, we like them all. 😛
Stream is a key resource to Winds 2.0, as it provides news feeds and activity streams for our users and even machine learning-based personalization. Without Stream, we would not be able to serve up suggested content to our users as we currently do.
MongoDB Atlas is a phenomenal DBaaS, allowing us to worry about acquiring users, while MongoDB worries about uptime. It’s identical to hosting your own cluster, except MongoDB provides you with a dashboard and a URI to connect to. With MongoDB Atlas, there’s no more worrying about cluster health, monitoring, etc.
Mongoose is a powerful ODM that allows us to define rich models within our MongoDB environment. Generally speaking, NoSQL databases are schemaless (meaning they have or require no form); however, with MongoDB, it’s always a good idea to specify a schema so you can index and organize your data properly. This allows for easy updates and efficient queries.
PM2 is a Node.js process manager. It allows us to ensure uptime, and scale processes as we need. The CLI for the project is dead simple, which allowed our team to pick it up on the fly.
Final Thoughts 😁
RSS is in a vicious circle. Winds is a community effort to help turn the tide and #ReviveRSS. Contributions are always much appreciated. To discuss new features check out the official Slack channel for Winds.