How Mixmax Uses Node and Go to Process 250M Events a day

2,776
Mixmax
Bringing email into the 21st century

Background

Mixmax is the product that your team uses to communicate with the outside world. What Slack did for internal communication, we’re doing for email and external communication.

Building a communication platform means processing a TON of data. Our backend, built primarily in Node and Go, processes up to 250M events a day with 200k/minute at peak load. As the glue for an organization’s communication, not only are we processing a huge number of internal events, but we’re also processing data from external sources like CRMs and ATSs totalling 3.2 million events and amounting to a data volume exceeding 14 GB each hour. We've already scaled our platform up 2x in the past 3 months and plan to grow another 10x this year, all while maintaining strict "three 9's" uptime that our customers expect, as they rely on Mixmax all day to get their work done.

I’m the Head of Platform Engineering at Mixmax, which means that I spend most of my time supporting and unblocking our engineering teams. I’ve spent most of my time working in SaaS with a stint in security.

Mixmax Engineering

Our engineering team currently consists of 15 engineers with highly varied backgrounds. Today, everyone on the team is a full-stack engineer, although all of us have our own strengths (i.e. Elasticsearch, MongoDB, devops, security, etc). It makes for an amazing mix with everyone bringing their own superpower to the table. Our team is also highly distributed, with engineers in Australia, Canada, Mexico, and the US.

We’re self-organized into a constantly varying number of teams. We have two evergreen teams, our Core and Support teams, which are responsible for the two pillars of engineering departments - stability and quality. Beyond those two teams, we create teams around our product priorities. This means each product team lives for the duration of the development life-cycle, and no longer. This dynamic nature allows us to more seamlessly share and distribute knowledge across the team so that we’re all constantly learning and growing. Teams are also cross-functional, which helps us have consistent and open feedback between everyone in engineering, product and design. Each team defines what their success and failure metrics are, as well as they will measure their own progress (some teams do two week sprints, some do Kanban, etc). The one constant is that every team agrees and publicizes the metrics that they use to monitor their own success.

Orthogonal to our teams, we have two guilds: our web guild and our platform guild. Our guilds are for helping us improve our and develop our individual strengths, not for siloing knowledge. Our guilds focus on elevating best practices for their areas of ownership, as well as helping to mentor and provide safety nets for members outside their guild. One clear distinction that we draw, is that we explicitly ensure that guild members are not the only ones during work that would fall into their areas. First and foremost, all engineers are expected to focus on their team, helping them to achieve their goals - as an example, this means that “platform work” is not meant to be worked on by only platform guild members.

Initial architecture and application evolution

Mixmax was originally built using Meteor as a single monolithic app. As more users began to onboard, we started noticing scaling issues, and so we broke out our first microservice: our Compose service, for writing emails and Sequences, was born as a Node.js service. Soon after that, we broke out all recipient searching and storage functionality to another Node.js microservice, our Contacts service. This practice of breaking out microservices in order to help our system more appropriately scale, by being more explicit about each microservice’s responsibilities, continued as we broke out numerous more microservices.

This resulted in a system with many Node.js microservices and one still fairly large Meteor service. All of these Node.js services did, and still do, run on Elastic Beanstalk in AWS as we optimized for developer velocity by using a managed deployment platform. The Meteor app ran in Galaxy, which had necessitated that we use a subdomain-based microservice approach for that main Meteor app to talk to the other microservices.

As we began to scale super quickly, with more and more customers joining the platform, we started to see that the Meteor app was still having a lot of trouble scaling due to how it tried to provide its reactivity layer. To be honest, this led to a brutal summer of playing Galaxy container whack-a-mole as containers would saturate their CPU and become unresponsive. I’ll never forget hacking away at building a new microservice to relieve the load on the system so that we’d stop getting paged every 30-40 minutes. Luckily, we’ve never had to do that again! After stabilizing the system, we had to build out two more microservices to provide the necessary reactivity and authentication layers as we rebuilt our Meteor app from the ground up in Node. This also had the added benefit of being able to deploy the entire application in the same AWS VPCs. Thankfully, AWS had also released their ALB product so that we didn’t have to build and maintain our own websocket layer in EC2. All of our microservices, except for one special Go one, are now in Node with an Nginx frontend on each instance, all behind ELBs or ALBs running in Elastic Beanstalk.

Data storage at Mixmax

Originally, we had a single Mongo replica set that we stored everything on. As we scaled, we realized two things:

  • A single Mongo replica set wasn’t going to cut it for our many quickly growing collections
  • Analytics and rich searching don’t scale well in Mongo.

To solve for the first item, we now run multiple large scale Mongo deployments with a mix of replica sets and sharded replica sets (depends on the application activity for the given database). In solving for the second item, we now run multiple large Elasticsearch deployments to provide the majority of our rich searching functionality.

We also heavily use Redis across the entire platform for things like distributed locking, caching, and backing part of our job queuing layer. This has led to our most recent (and ongoing!) scaling challenge.

(here’s a screenshot of the tool that we use to administer to our worker queues that live on Redis)

Asynchronous processing at Mixmax

At Mixmax, we have multiple queueing systems running that all exhibit very different behaviours, due to all the different ways that our platform is used. We’ve gone through quite a few Redis-backed job queueing technologies before we arrived at our current place (from Kue to bull-queue to bee-queue to a mix of bee-queue and AWS Kinesis). Our current stack, a mix of bee-queue and AWS Kinesis, allows us to both seamlessly handle our steadily active queues (i.e. for sending emails) and weather the storm of work that powers our CRM syncing engines. This has been a really fun challenge, as part of this system handles in the high hundreds of millions of jobs a day with sporadic spikes of millions of jobs per minute. We’ve made huge progress here, and we still have a lot of progress to make as we continue to scale this asynchronous processing system.

How we ship

Our workflow centers around getting code live ASAP. Our CI pipeline is centered around GitHub as our VCS tied into TravisCI. Our CD pipeline then continues on from there using AWS Elastic Beanstalk to deploy new application versions.

All developers are able to work on a local copy of the entire infrastructure. Once a developer has their code ready, it goes through review on GitHub - side note, we’re loving all the work that they’re putting into their code review tooling. After code is reviewed and good to go, it lands on our staging environment, where we manually QA a few core flows before we’ll elevate the code to be released on our production environment. For running all of our services locally, we currently use a mix of supervisord and a tool built by one of our engineers named custody.

A huge part of our continuous deployment practices is to have granular alerting and monitoring across the platform. To do this, we run Sentry on-premise, inside our VPCs, for our event alerting, and we run an awesome observability and monitoring system consisting of Statsd, Graphite and Grafana. We have dashboards using this system to monitor our core subsystems so that we can know the health of any given subsystem at any moment. This system ties into our PagerDuty rotation, as well as alerts from some of our CloudWatch alarms (we’re looking to migrate all of these to our internal monitoring system soon).

(screenshot of our monitoring cluster monitoring our strongDM gateways)

Security hygiene at distributed scale

Being a distributed team is in our DNA. One challenge that we’ve faced as a part of being such a distributed team is providing auditible, available, secure and stable access to databases in our private networks for engineers that are authorized and need to have access to them. In a distributed world, auditing database access, credential management and rotation, and onboarding can be a nightmare. Someone running a query on a staging DB that’s taking down the test environment for every? Good luck hunting that down. Have a new engineer onboard and they need to run an audit query on the staging DB to see if their new code might break an old schema? Have fun configuring that. Need to run your periodic credential rotation, ...enjoy. This was not only a huge pain point for our team, but me personally, and then strongDM came into the picture.

strongDM acts as a control plane to manage access to every database and server. By centralizing all database credentials & ssh keys in strongDM, onboarding and offboarding becomes much faster. Simply add a user to a group and since the user never has access to the DB credentials (strongDM handles that) you never need to worry about rotating credentials purely due to employee offboarding. For auditing, since strongDM knows and can monitor each users’ connection, you have direct insight into every single query or access that a user makes - a godsend for auditing. When it comes to periodically rotating keys, it’s even simpler, as your rotating credential sets instead of credentials per user, without any action needed from a single other engineer - it simply works. Our engineers have enjoyed strongDM so much that some have even tweeted about it in moments of pure joy.

I seriously cannot imagine working without strongDM now. It’s one of those tools that seamlessly fits into your workflow and you can’t envision work without it.

(screenshots of strongDM in action)

What’s next? Processing all the things.

It’s an exciting time to be at Mixmax, our entire company is scaling quickly (we’ve grown 4x in the last two years, and this trend isn’t slowing down) along with our customer base. This means that we’re processing more data than ever before, and we’re having to get more and more creative to keep up with the amount of data coming in.

We’re currently prototyping our next generation processing systems, building them out in different languages, with different tech - it’s a fantastic time to join to come help us figure out our future direction as an engineering team all while working on a platform that our customers love!

Mixmax
Bringing email into the 21st century
Tools mentioned in article
Open jobs at Mixmax
Engineering Manager

The opportunity

Mixmax is the #1 product helping Revenue teams to become automatically proactive by eliminating busywork for a flawless customer experience.

We've just hit $10 million in annual revenue and we’re profitable! We just raised more money and are investing more to accelerate growth. We’re looking for great people to help us ship high-quality features to meet the needs of bigger customers. 

About Engineering

We can find two areas, EPD and Core teams. The EPD teams are cross-functional (Engineering + Product + Design) and work closely with Product to enhance our customer experience by following a product roadmap.

The Core teams manage the product platform i.e. AWS infrastructure, shared product services and Mixmax’s development framework following a technical roadmap

Teams must collaborate (besides Product) with all departments such as Customer Success, Sales or Marketing to:

  • Understand market and customer needs
  • Provide instruments and indicators to measure the impact of changes
  • Identify areas for improvement: functional, capabilities, security, scalability, etc.

Our product is hosted on AWS, following a micro-services architecture pattern with noSQL databases where NodeJS is our primary programming language.

This role can be based anywhere in Europe or LATAM.

What You Will Be Doing

  • Manage, mentor and lead a full-remote distributed team between 4-8 software engineers who could be located in USA, LATAM and EU.
  • Lead the efficient delivery of time-boxed software releases, ensuring high quality and following standard principles of security, scalability, maintainability, reusability, etc...
  • Contribute to product and technical roadmaps and lead prioritisation negotiating with different stakeholders
  • Work with Product Management to ensure requirements (what) and the strategy (why) are clear
  • Identify and continuously improve production metrics impacting your team, other teams and customers
  • Create a safe working environment, honest, fun, inclusive, fostering mutual respect and collaborative decision making
  • Identify staffing requirements and participate in the recruitment process of talented colleagues
  • Support the up-skilling of direct reports through training, development and regular performance evaluations and constant bi-directional feedback

What You Will Bring

  • Experience leading diverse teams of software engineers as a people manager
  • Experience leading complex engineering projects that needed cross engineering team collaborations
  • Experience with Cloud hosted systems and any of React, NodeJS and with both relational and NoSQL Databases
  • Experience as a hands-on software engineer and fundamentals of computer science to understand and recognise quality in the engineering work
  • Performance management experience
  • Ability to explain technical complexities to different audiences
  • Empathize with your team and have a genuine interest in the success and growth of the people you work with. Encourage, empower, and celebrate!

Desirable Skills

  • Experience managing distributed, multicultural teams
  • Experience in startups or bigger organizations with fast-paced delivery cycles
  • Functional knowledge of Revenue teams (SDR/AE/CSM)
  • Technical knowledge of Salesforce, G-Suite or similar customer-facing products

Our commitment to diversity and inclusion

At Mixmax, we know that nobody's perfect, and that no one ever matches 100% with a job description. That's okay–we're human after all! Diversity and inclusion are core to our culture, and we're actively committed to building a more inclusive and open workplace. No matter your background or how you identify, if you're excited about this role, please apply today!

The Mixmax story

At Mixmax, our vision is a world without busywork. Since our launch in 2015, Mixmax has become the product of choice for over 10,000 Sales and Customer Success teams to eliminate and automate repetitive work. This means more time to focus on what matters: engaging and serving the needs of customers.

We’re extraordinarily proud of the company we’ve built. We’re a driven, passionate, responsible group that values personal and professional growth equally. We take care of ourselves, our families, our customers, and one another. We believe in sustainable and diverse approaches to work and life, because optimizing for the long-term is the best path to success.

Our company is fully remote and distributed, with team members worldwide. We offer competitive salaries, meaningful equity, and generous benefits. And you get to work on a product people absolutely love!

Senior Backend Engineer

The Opportunity

Exciting news - Mixmax hit $10 million in annual revenue and we’re profitable! We just raised more money and are looking to invest more to accelerate growth.
We’re hiring product-focused software engineers with deep backend expertise and strong attention to detail. As a Backend Engineer, you’ll build the next generation of Mixmax features, scale distributed systems to handle millions of events, and drive quality across a large codebase, all as part of a globally distributed team. You'll also be responsible for improving and strengthening the foundation of our system, improving our team's developer experience, addressing our organizational security needs, and participating in collaborative decision-making about our architecture.

Within Three Months, You'll

  • Work with Node, MongoDB, Elasticsearch, Terraform, Redis, and AWS;
  • Make impactful improvements to our production infrastructure; and
  • Demo your first new feature to the entire company!

Within Six Months, You'll

  • Contribute to the development of critical projects while working closely with other engineers;
  • Work with your team to improve our architectural patterns, scalability, and developer experience; and
  • Contribute to the culture of a growing team.

 Preferred Skills and Background

  • We're looking for engineers who have experience with the core of our tech stack, which includes ES6 JavaScript and/or TypeScript, Node.js, AWS, and MongoDB.
  • Mixmax is still growing fast, so we love new teammates who strive to grow personally and professionally, beyond just expanding their technical abilities.
  • We take pride in the quality of our work, so we're looking for engineers with experience maintaining and supporting the systems they've built.
  • We're a distributed team working in multiple time zones, so excellent documentation and communication skills are paramount!

It’s Also Awesome (But 100% Not a Requirement!) For You To Have

  • Worked in a distributed codebase with microservices and shared modules;
  • Familiarity with secondary tools in our tech stack, like Elasticsearch or Redis;
  • Worked on a Salesforce integration; and/or
  • The ability to refer other great engineers to come work at Mixmax! 😉

Our Commitment to Diversity and Inclusion

At Mixmax, we know that nobody's perfect and that no one ever matches 100% with a job description. That's okay—we're human after all! Diversity and inclusion are core to our culture and we're actively committed to building a more inclusive and open workplace. No matter your background or how you identify, if you're excited about this role, we encourage you to apply.

The Mixmax Story

At Mixmax, our vision is a world without busywork. Since our launch in 2015, Mixmax has become the product of choice for over 10,000 Sales and Customer Success teams to eliminate and automate repetitive work. This means more time to focus on what matters: engaging and serving the needs of customers.

We’re extraordinarily proud of the company we’ve built. We’re a driven, passionate, responsible group that values personal and professional growth equally. We take care of ourselves, our families, our customers, and one another. We believe in sustainable and diverse approaches to work and life, because optimizing for the long-term is the best path to success.

Our company is globally distributed, with remote team members worldwide. We offer competitive salaries, meaningful equity, and generous benefits. And with Mixmax, you get to work on a product people love.

Solutions Engineer (EMEA)
EMEA

The opportunity

Exciting news! We just hit $11 million in annual revenue and are looking for an experienced Solutions Engineer to delight both our key customers and prospects by advising them how to best integrate Mixmax’s automation with their internal systems like Salesforce.  This role reports into our Head of Success and will be working with both Success and Sales to partner with our most strategic accounts and provide technical leadership around Mixmax’s Salesforce and helping them scale their Mixmax automation.

What you will be working on:

  • Participating in both pre and post-sales to become a trusted technical advisor helping customers define their Mixmax integration and automation strategy to hit their goals.
  • Guiding customers on the various Mixmax integration approaches across CRMs like Salesforce and Pipedrive as well as the Mixmax API and webhooks. 
  • Developing internal and external content like recorded demos, knowledge base articles and best practices documentation.
  • Reviewing customer systems and making recommendations to optimize.
  • Mentoring the CS and Sales teams around the more technical aspects of the Mixmax product and providing guidance.
  • Acting as a point of escalation for the triage and resolution of large and complex customer issues.
  • Collaborating with the product and engineering teams to give feedback and partner on customer issues.

Required skills and background

  • 2+ years experience as a Solutions Engineer/Architect, working directly with customers both pre and post sale to solve technical problems.
  • 2+ years of direct experience troubleshooting Salesforce or working with products that integrate with Salesforce 
  • Able to work 8am-5pm ET and occasionally shift earlier to support customers in EMEA
  • Ability to own technical inquiries related to Mixmax’s Salesforce sync, API and webhooks; comfortable advising new users on appropriate implementation of Mixmax’s technical products
  • Experience triaging technical tickets and bugs before they are sent to engineering teams
  • A proven record working closely with product and engineering teams to prioritize customer solutions
  • Comfortable resolving problems involving multiple pieces of software and integrations in complex customer environments.
  • Experience creating documentation and materials to help educate customers and internal teams on the technical aspects of the product
  • Experience with supporting other email products a plus
  • Experience supporting SaaS tools
  • Experience and past success in a remote position
  • Comfortable with ambiguity and pitching in with other projects as needed

Our commitment to diversity and inclusion

At Mixmax, we know that nobody's perfect, and that no one ever matches 100% with a job description. That's okay–we're human after all! Diversity and inclusion are core to our culture, and we're actively committed to building a more inclusive and open workplace. No matter your background or how you identify, if you're excited about this role, please apply today!

San Francisco Fair Chance Ordinance

Pursuant to the San Francisco Fair Chance Ordinance, and any other local ordinances, Mixmax will consider for employment qualified applicants with arrest and conviction records.

The Mixmax story

At Mixmax, our vision is a world without busywork. Since our launch in 2015, Mixmax has become the product of choice for over 10,000 Sales and Customer Success teams to eliminate and automate repetitive work. This means more time to focus on what matters: engaging and serving the needs of customers.

We’re extraordinarily proud of the company we’ve built. We’re a driven, passionate, responsible group that values personal and professional growth equally. We take care of ourselves, our families, our customers, and one another. We believe in sustainable and diverse approaches to work and life, because optimizing for the long-term is the best path to success.

Full-Stack Engineer ( React + NodeJS)
The opportunity

Exciting news - Mixmax hit $10 million in annual revenue and we’re profitable! We just raised more money and are investing more to accelerate growth. We’re looking for great people to help us ship high-quality features to meet the needs of bigger customers.
We’re hiring UX-focused Frontend Engineers of varying experience levels to own our client-side codebase across both our Chrome extension and our web apps. We need engineers skilled in modern web frameworks such as React, as well as highly technical engineers with deep client-side Javascript expertise and strong knowledge of fundamental web technologies and browser APIs. Regardless of experience level, our frontend engineers play an important role in making Mixmax the easiest-to-use solution for our customers, all as part of a globally distributed team.

The team
We’re hiring a full-stack engineer for team Rocket, a cross-functional product team that is in charge of the integration with Salesforce, as well as reporting and a bunch of other integrations and components. The current composition of the team is four senior developers, a product manager, an engineering manager, a designer, and a QA shared with other team. We need a fifth engineer to complement the team, who is experienced in modern web frameworks such as React, but who is also willing to work on the backend, which is based on a microservices (Node) architecture (the work distribution will be around 40-60% frontend-backend).

 Within three months, you’ll:
  • Work with React and Styled Components.
  • Understand our microservices architecture and become familiar with the codebase of the main services.
  • Be directly involved in the entire development lifecycle of Mixmax’s product.
  • Actively participate in our team Agile ceremonies.
  • Learn the basics about Salesforce and how the integration with Mixmax works.
  • Demo your first new feature to the entire company!
Within six months, you could:
  • Join one of our internal guilds and contribute to the engineering roadmap.
  • Be responsible for managing and improving our product quality and robustness by improving our alerts and logging (we use Sentry).
  • Ship work on critical projects and features while working with other engineers.
  • Contribute to improving the performance and robustness of our services and app.
Preferred skills and background:
  • We’re looking for engineers with expertise in React, Styled Components, and state management like Redux.
  • At the senior level, familiarity with modern browser APIs, and writing resilient, clean and tested code.
  • We’re a distributed team working in multiple time zones, so excellent documentation and communication skills are paramount!
It’s also awesome (and 100% not a requirement!) for you to:
  • Have some knowledge of Salesforce.
  • Be able to refer other great engineers to work at Mixmax!


Our commitment to diversity and inclusion

At Mixmax, we know that nobody's perfect and that no one ever matches 100% with a job description. That's okay–we're human after all! Diversity and inclusion are core to our culture, and we're actively committed to building a more inclusive and open workplace. No matter your background or how you identify, if you're excited about this role, please apply today!

The Mixmax story

At Mixmax, our vision is a world without busy work. Since our launch in 2015, Mixmax has become the product of choice for over 10,000 Sales and Customer Success teams to eliminate and automate repetitive work. This means more time to focus on what matters: engaging and serving the needs of customers.

We’re extraordinarily proud of the company we’ve built. We’re a driven, passionate, responsible group that values personal and professional growth equally. We take care of ourselves, our families, our customers, and one another. We believe in sustainable and diverse approaches to work and life, because optimizing for the long-term is the best path to success.

Our company is fully remote and distributed, with team members worldwide. We offer competitive salaries, meaningful equity, and generous benefits. And you get to work on a product people absolutely love!

Verified by
You may also like