Scaling Zapier to Automate Billions of Tasks

21,774
Zapier
Zapier is for busy people who know their time is better spent selling, marketing, or coding. Instead of wasting valuable time coming up with complicated systems - you can use Zapier to automate the web services you and your team are already using on a daily basis.

Editor's note: By Bryan Helmig, ‎Co-founder & CTO at Zapier



Zapier is a web service that automates data flow between over 500 web apps, including MailChimp, Salesforce, GitHub, Trello and many more.

Imagine building a workflow (or a "Zap" as we call it) that triggers when a user fills out your Typeform form, then automatically creates an event on your Google Calendar, sends a Slack notification and finishes up by adding a row to a Google Sheets spreadsheet. That's Zapier. Building Zaps like this is very easy, even for non-technical users, and is infinitely customizable.

As CTO and co-founder, I built much of the original core system, and today lead the engineering team. I'd like to take you on a journey through our stack, how we built it and how we're still improving it today!

The Teams Behind the Curtains

It takes a lot to make Zapier tick, so we have four distinct teams in engineering:

  • The frontend team, which works on the very powerful workflow editor.
  • The full stack team, which is cross-functional but focuses on the workflow engine.
  • The devops team, which keeps the engine humming.
  • The platform team, which helps with QA, and onboards partners to our developer platform.

All told, this involves about 15 engineers (and is growing!).

The Architecture

Our stack isn't going to win any novelty awards — we're using some pretty standard (but awesome) tools to power Zapier. More interesting are the ways we're using them to solve our particular brand of problems, but let's get the basics out of the way:

The Frontend

We're smack in the middle of transitioning from Backbone to React. We use Babel for ES6 and Webpack + Gulp to compile the frontend. We rely heavily on CodeMirror to do some of the more complex input widgets we need, and use React + Redux to do much of the heavy lifting for the uber-powerful Zap editor.

The Backend

Python powers a large majority of our backend. Django is the framework of choice for the HTTP side of things. Celery is a massive part of our distributed workflow engine. Most of the routine API work is done with the epic requests library (with a bunch of custom adapters and abstractions).

The Data

MySQL is our primary relational data store — you'll find our users, Zaps and more inside MySQL. Memcached and McRouter make an appearance as the ubiquitous caching layer. Other types of data go in other data stores that make more sense. For example, in-flight task counts for billing and throttling find themselves in Redis, and Elasticsearch stores historical activity feeds for Zaps. For data analysis we love us some AWS Redshift.

The Platform

Most of our platform is nestled inside our fairly monolithic core Python codebase, but there are lots of interesting offshoots that offer specialized functionality. The best example might be how we utilize AWS Lambda to run partner/user provided code to customize app behavior and API communication.

The Infrastructure

Since Zapier runs on AWS, we have quite a bit of power at our fingertips. EC2 and VPC are the centerpiece there, though we do use RDS where possible along with copious numbers of autoscaling groups to ensure the pool of servers are in tip-top shape. Jenkins, Terraform, Puppet and Ansible are all daily tools for the devops team. For monitoring, we can't rave enough about Statsd, Graylog, and Sentry (they're so good).

Some Rough Numbers

These numbers represent a rough minimum to help the reader guage the general size and dimensions of Zapier's architecture:

  • over ~8m tasks automated daily
  • over ~60m API calls daily
  • over ~10m inbound webhooks daily
  • ~12 c3.2xlarge boxes running HTTP behind ELB
  • ~100 m3.2xlarge background workers running Celery (split amongst polling, hooks, email, misc)
  • ~3 m3.medium RabbitMQ nodes in a cluster
  • ~4 r3.2xlarge Redis instances - one hot, two failover, one backup/imaging
  • ~12 m2.xlarge Memcached instances behind ~6 c3.xlarge McRouter instances
  • ~10 m3.xlarge ElasticSearch instances behind ~3 m3.xlarge no-data ElasticSearch instances
  • ~6 m3.xlarge ElasticSearch instances behind ~1 c3.2xlarge Graylog server
  • ~10 dc1.large Redshift nodes in a cluster
  • 1 master db.m2.2xlarge RDS MySQL instance w/ ~2 more replicas for both production reads and analysis
  • a handful of supporting RDS MySQL instance (more details below)
  • ...and tons of microservices and miscellaneous specialty services

Improving the Architecture

While the broad strokes of the architecture remain the same - we've only performed a few massive migrations - a lot of work has been done to grow the product in two categories:

  1. Supporting big new product features
  2. Scaling the application for more users

Let's dive into some examples of each, with as many nitty-gritty details as possible without getting bogged down!

Big Features like Multi-Step Zaps

When we started Zapier (fun fact: it was first called Snapier!) at a Startup weekend, we laid out the basic architecture in less than 54 hours (fueled by lots of coffee and more than a few beers). Overall, it was decent. We kept the design really, really simple, which was the right call at the time.

Except where it was too simple. Specifically, we made Zaps two-stepped: a trigger paired with an action, full stop.

It didn't take long for us to realize the missed opportunity, but the transition was going to be pretty complex. We had to implement a directed rooted tree with support for an arbitrary numbers of steps (nodes), but maintain 1-to-1 support for existing Zaps (of which there were already hundreds of thousands). And we had to do that while preserving support for hundreds of independent partner APIs.

Starting at the data model, we built a very simple directed rooted tree implementation in MySQL. Just imagine a table where every row has a self-referencing parent_id foreign key, plus an extra root_id foreign key to simplify queries, and you pretty much got it. We discussed switching to a proper graph database (like neo4j) but decided against it because the sorts of queries we make are simple and over isolated graphs of smaller sizes (roughly ~2-50 nodes).

A key aspect to make this work is inter-step independence. Every step has to consume some data (which folder to read from or which list ID to add to, for example), do some API magic, and return some data (say, the new file created or the new card added to a list), but otherwise be ignorant of its placement in the workflow. Each independent step is as dumb as a rock.

In the middle exists the omniscient workflow engine which coordinates independent Celery tasks by stringing together steps as tasks — one step feeding into the next as defined by the Zap's directed rooted tree. This omniscient engine also houses all the other goodies like error & retry handling, reporting, logging, throttling and more.

Even after we nailed the backend support, we had another huge problem: how do you build a UI for this thing?

First, you make sure you have some amazing designers and Javascript engineers on the team. Then you wrestle with nested Backbone views for a while before moving onto React. :-) In all seriousness: React is a godsend for the sorts of complex interfaces we are building.

One of the unique things about React is the performance characteristics are developer friendly, but only as long as you have your data figured out. If you aren't using immutable data structures, you should use some structural sharing library to do all mutations, along with deep Object.freeze() in development to catch spots where you attempt mutation directly.

There are tons of challenges in building such a complex UI, much of it around testing and feedback, but a huge amount of time was spent just getting the long-tailed data from different APIs to fit elegantly into the same places. Just about every weird shape of data has to be accounted for.

Finally, we were tasked with getting the new editor in front of users for alpha and beta testing. To do this we shipped both versions of the editor simultaneously and used feature switches to opt users in. We did months and months of testing and tweaking before we were happy with the result - you can check out the Multi-Step Zap launch page to get an idea of where it ended up.



Scaling the Application

It would all be for nought if the service isn't up and running reliably. As such, much of our attention is focused jointly on application design to support horizontal scalability and redundant infrastructure to ensure availability.

Some of the wiser decisions we've made so far is to double down on tech we're comfortable with and spin out isolated functionality when we hit a bottleneck. The key is to reuse the exact same solution and move it to another box where it is free to roam fresh pastures of CPU and RAM.

For example, over the last year or so we noticed session data had eaten up a ton of our primary database's IO and storage. Since session data is effectively a key/value arrangement with softer consistency requirements, we heavily debated options like Cassandra or Riak (or even Redis!), but ultimately decided to stand up a dedicated MySQL instance with a single sessions table.

Our instinct as engineers was to find the tool best suited to the job, but as a practical matter, the job didn't warrant additional operational complexity. We know MySQL, it can do simple key/value storage and our application already supports it. Talk about a no-brainer.

Further, careful design of the application can make horizontal scaling equally simple. Long running background tasks (like our Multi-Step Zaps) aren't bound by strict consistency requirements due to their light write pattern, so it is trivial (and safe!) to use MySQL read-only replicas as the primary touch point. Even if we occasionally get horrible replica lag measured in the minutes, 99.9% of Zaps aren't changing — and certainly aren't changing soon — so they continue to hum along.

Another good practice is to assume the worst. Design for failure from the beginning. While usually this is easier said than done, nowadays it is actually surprisingly easy to do. For starters: use auto scaling groups with auto-replacement. A common misconception is that ASGs are only for scaling to accommodate fluctuating load. Wrong! The ASG + ELB combo can be your backbone of reliability, one that enables you to randomly kill instances without worry since they get replaced in quick order.

Somehow we keep re-learning that the simpler the system is, the better you'll sleep.

The Day-To-Day

Locally, our engineers enjoy a fully functioning environment courtesy of Docker. docker-machine and docker-compose together stand up proper versions of MySQL, Memcached, Redis, Elasticsearch as well as all the web and background workers. We generally recommend running npm and even runserver locally, as file watching is kind of broken with VirtualBox.

The canonical GitHub "pull request model" drives most of our projects that are engineering focused, where day-to-day work is logged and final code review happens before merging. Hackpad houses the majority of our documentation, including copious onboarding documentation.

A big thing at Zapier is all hands support. Every four or five weeks, every engineer spends a full week in support helping debug and fix difficult customer issues. This is hugely important to us as it provides a baseline for understanding customers' pain (plus, you might have to deal with the bug you shipped!).

For CI and deployment, we use Jenkins to run tests on every commit in every PR as well as to provide a "one-click deploy" that anyone at the company can press. It's not uncommon for a new engineer to click the deploy button the first week on the job!

We have a full staging environment in a standalone VPC, as well as a handful of standalone web boxes perfect for testing long lived pull requests. Canary deploys to production are common — complete with full logs of any errors courtesy of Graylog.

You Can Zapier, Too!

Developers can use Zapier to do some pretty awesome stuff.

In addition to Multi-Step Zaps, we've also launched the ability to write Python and Javascript as Code steps in your workflow. No need to host and run scripts yourself — we take care of all of that. We also provide bindings to call out to the web (requests and fetch) and even store a bit of state between runs!

Our users are employing Code steps to build Slack bots (and games!), to replace one-off scripts and lots more. I personally use Code steps to write bots and tools to track code & bug metrics to a spreadsheet, transform oddly formatted data and replace a ton of crontabs.

Or, if you have an API that you want non-developers to be able to consume, we have a pretty epic Developer Platform, too. Simply define your triggers, searches and actions, and any user can mix your API into their workflows and integrate your app with over 500 apps like GitHub, Salesforce, Google Docs, and more.

And, we are often hiring, so keep an eye on our jobs page if you'd like to help us help people work faster and automate their most tedious tasks!


Zapier
Zapier is for busy people who know their time is better spent selling, marketing, or coding. Instead of wasting valuable time coming up with complicated systems - you can use Zapier to automate the web services you and your team are already using on a daily basis.
Tools mentioned in article
Open jobs at Zapier
General Interest Form- Future Enginee...
Flexible
<p><em><span style="font-weight: 400;">This post is not linked to a specific job. If you do not see any posting applicable to your skillset, please apply here to be notified about new roles that may be a</span><span style="font-weight: 400;"> good fit for you. </span>We accept applications on an ongoing basis.</em></p> <p><span style="font-weight: 400;">Want to be part of the team behind the product that is Making Automation Work for Everyone—all while advancing your career at a fast-growing, profitable, impact-driven company? Then read on…</span></p> <p><a href="https://zapier.com/jobs/our-commitment-to-applicants/"><span style="font-weight: 400;">Our Commitment to Applicants</span></a></p> <p><a href="https://zapier.com/jobs/culture-and-values-at-zapier/"><span style="font-weight: 400;">Culture and Values at Zapier</span></a></p> <p><a href="https://zapier.com/learn/remote-work/"><span style="font-weight: 400;">Zapier Guide to Remote Work</span></a></p> <p><a href="https://zapier.com/jobs/zapier-code-of-conduct/"><span style="font-weight: 400;">Zapier Code of Conduct</span></a></p> <p><a href="https://zapier.com/jobs/working-on-diversity-and-inclusivity/"><span style="font-weight: 400;">Diversity and Inclusivity at Zapier</span></a></p> <p>By filling out a general interest form, you become a member of our talent community here, which means you’ll be among the first candidates we review for each opening. You’ll also be invited to talent community workshops and receive quarterly newsletters to hear what’s new at Zapier.</p> <h2><strong>Zapier Compensation Guiding Principles</strong></h2> <p><span style="font-weight: 400;">We believe all Zapiens should be rewarded competitively and equitably, using practices that are simple and transparent. This philosophy ensures we’re able to find, grow, and retain exceptional people from a broad range of backgrounds. Here’s how we define our compensation principles:</span></p> <ul> <li>Competitive: Zapier pays well among the technology sector.</li> <li>Equitable: Consistent pay practices; Pay for impact.</li> <li>Simple: Pay is well understood, and pay practices are built for scale.</li> <li>Transparent: Zapiens know how pay works, including how their pay is determined.</li> </ul> <p>A Candidate's compensation package is finalized once the interview process is concluded and accounts for demonstrated experience, job knowledge, skills,&nbsp; abilities, and internal equity. We use a business impact approach to base pay, which means we set pay for all Zapier employees based on their demonstrated impact to Zapier’s success.</p> <p><span style="font-weight: 400;">For more information on Zapier’s Total Rewards please click</span><a href="https://zapier.com/l/jobs/total-rewards"><span style="font-weight: 400;"> </span><span style="font-weight: 400;">here</span></a><span style="font-weight: 400;">.</span></p> <p>&nbsp;</p> <h2><strong>This department includes teams such as:&nbsp;</strong></h2> <ul> <li>SRE</li> <li>Engineering</li> <li>Agile Practitioners</li> <li>Engineering Operations</li> <li>Support Engineering</li> <li>Security</li> <li>Team App Experiences</li> </ul>
Engineering Manager, New Products
NAMER
<p><strong>Job Posted: 1/9/24</strong></p> <p><strong>Location: Americas - North, Central and South America</strong></p> <p>Hi there!</p> <p>Zapier is on the lookout for talented&nbsp;<strong>Engineering Managers</strong> to lead some of our <strong>New Products</strong> teams. Our mission at Zapier is to empower everyone with the ability to increase their productivity through automation. We've already enabled millions to enhance their businesses, and now we're taking it a step further with Table, Interfaces, Canvas and many more.</p> <p><strong>Engineering Manager, Canvas and Growth</strong>: At the heart of Zapier's future, Canvas is designed to be a user-friendly and robust visual diagramming tool. It's crafted to simplify the process of planning automations, allowing for seamless integration of Zapier's diverse and unique features like Tables, Interfaces, and AI Chatbots. As the Engineering Manager on the Canvas and Growth teams, you'll play a crucial role in evolving Canvas from its current prototype phase to a fully-fledged product as well as develop and implement technical strategies to drive user acquisition, engagement, and retention across all of New Products.</p> <p><strong>Engineering Manager, Interfaces:</strong> Zapier Interfaces is at the forefront of customizable web solutions, offering a unique platform that empowers users to effortlessly create their own web apps with an intuitive drag-and-drop editor. This innovative tool provides essential building blocks, including text, media, forms, and more, seamlessly integrating with over 6,000 apps without requiring any coding skills. As the Engineering Manager for the Interfaces team, you will tackle significant challenges such as the introduction of new product features driving the evolution of Interfaces. Your leadership will be pivotal in enhancing performance, developing new components, and expanding enterprise functionalities, thereby accelerating growth and enabling users to craft tailored solutions that meet their specific business needs.</p> <p>If you’re interested in advancing your career at a fast-growing, profitable, impact-driven company, then read on…</p> <ul> <li><a href="https://zapier.com/jobs/our-commitment-to-applicants/">Our Commitment to Applicants</a></li> <li><a href="https://zapier.com/jobs/culture-and-values-at-zapier/">Culture and Values at Zapier</a></li> <li><a href="https://zapier.com/learn/remote-work/">Zapier Guide to Remote Work</a></li> <li><a href="https://zapier.com/jobs/zapier-code-of-conduct/">Zapier Code of Conduct</a></li> <li><a href="https://zapier.com/jobs/working-on-diversity-and-inclusivity/">Diversity and Inclusivity at Zapier<br><br></a></li> </ul> <h2>&nbsp;<strong>About You</strong></h2> <ul> <li><strong>You are a relationship builder and influencer.</strong> Your role involves not just leading your team but also collaborating with other teams and stakeholders. You possess excellent relationship-building skills and the ability to influence technical decisions across teams for the greater good of the company.</li> <li><strong>You are vocal and discerning about architecture and software design</strong>. Your opinions on architecture and software design are informed and assertive. You understand the delicate balance between meticulous planning and pragmatic decision-making. This insight enables you to identify which architectural decisions require careful deliberation and where agility and shortcuts are appropriate, ensuring that development is both strategic and adaptable to the dynamic needs of the project.</li> <li><strong>You're driven by results, not by process</strong>. In the ever-changing landscape of software development, you recognize that while processes are essential, they should never overshadow the end goal: delivering exceptional value. Your modus operandi involves light yet effective process and technical planning. You are adept at navigating through challenges with agility, ensuring that the emphasis remains on outcomes rather than getting entangled in procedures.</li> <li><strong>You care about the 'why' more than the 'what'</strong>. Your approach to engineering isn't just about ticking boxes or following directives. You deeply value clarity and purpose in your work. If priorities seem nebulous or not optimized, you're confident and proactive in seeking clarity, ensuring that the team's efforts align with the broader vision. Misalignments don't escape your keen eye, and you're quick to flag them, ensuring that the team remains on a coherent and impactful trajectory.</li> <li><strong>You live by the "always ship new code" mantra</strong>. You take immense pleasure in your team releasing code frequently, knowing that every deployment brings with it enhancements, new features, or solutions that directly impact users. This drive to constantly ship isn't just about quantity for you; it's rooted in your belief that iterative improvement, driven by real-world feedback, is the key to creating truly outstanding software. You're always on the lookout for opportunities to refine, innovate, and deliver, ensuring that users continually benefit from your relentless dedication to progress.</li> <li><strong>You embody the "let's make it work" ethos</strong>. Challenges, whether monumental or minuscule, don't deter you. You dive in with a can-do attitude, unfazed by the magnitude or nature of the task. You understand that in the broader scheme of things, not every task will be about building the next flashy feature. Sometimes, it's about fortifying the foundations, ensuring that the system is robust and resilient. You're discerning enough to know when to invest time in hardening the system versus when to prioritize speed. In essence, you're a true team player, understanding that every task, no matter how glamorous or mundane, plays a pivotal role in the team's collective success.</li> <li><strong>You have a user-centric approach</strong>. Emulating the engineer's ethos of advocating for the user, you ensure that your team's work aligns with user needs and enhances their experience with our products. Your leadership is grounded in empathy and a deep understanding of user-centric design.<br><br></li> </ul> <h2>&nbsp;<strong>Things You'll Do</strong></h2> <ul> <li>Work with the engineering team to make, evaluate, and advocate for important architectural decisions, ensuring they align with the long-term vision and technical standards of Zapier.</li> <li>Collaborate with other engineering teams to evaluate and inform their technical decisions, creating alignment and removing blockers for the execution of Canvas roadmap items.</li> <li>Lead initiatives to identify and address technical challenges, fostering a problem-solving mindset and driving continuous improvement in our engineering processes.</li> <li>Work closely with your engineering function and peer design function and product function to create a trio that tackles user problems and solutions.</li> <li>Develop effective ways to communicate, monitor, and lead your team through weekly one-on-one’s and team meetings.</li> <li>Keep the leadership team informed on your team’s progress in ways that are easy and enjoyable to receive, like one-on-one's, update posts, and regular team hangouts.</li> <li>Build rapport with each member of the Engineering Team and support them through coaching and mentorship to help level up their skills.</li> <li>Participate in code reviews, learning and spreading technical knowledge throughout Zapier -- moving knowledge to documentation where appropriate.</li> <li>Occasionally dive into the code; fixing bugs, improving developer experience, smoothing edges -- sometimes even spiking out small proofs of concept.</li> <li>Actively recruit, onboard, and train new engineers at Zapier. This might involve tweaking the skills portions of interviews, or writing better documentation.</li> <li>As part of our<a href="https://zapier.com/learn/customer-support/everyone-on-support/"> All Hands Support</a> initiative, help customers have the best experience with Zapier as possible.<br><br></li> </ul> <p>&nbsp;</p> <h2><strong>Zapier Compensation Guiding Principles</strong></h2> <p>We believe all Zapiens should be rewarded competitively and equitably, using practices that are simple and transparent. This philosophy ensures we’re able to find, grow, and retain exceptional people from a broad range of backgrounds. Here’s how we define our compensation principles:</p> <ul> <li>Competitive: Zapier pays well among the technology sector.</li> <li>Equitable: Consistent pay practices; Pay for impact</li> <li>Simple: Pay is well understood, and pay practices are built for scale.</li> <li>Transparent: Zapiens know how pay works, including how their pay is determined.<br><br></li> </ul> <p>The pay ranges for this role are (min - mid - max):</p> <ul> <li>United States 184,100 - 230,100 - 276,100 USD</li> <li>Canada 184,100 - 230,100 - 276,100 CAD<br><br></li> </ul> <p>A Candidate's compensation package is finalized once the interview process is concluded and accounts for demonstrated experience, job knowledge, skills,&nbsp; abilities, and internal equity. We use a business impact approach to base pay, which means we set pay for all Zapiens based on their demonstrated impact on Zapier’s success. In alignment with that philosophy, the upper half of a pay range is typically reserved for individuals who have consistently demonstrated a high impact in their current role and level while at Zapier.</p> <p>For more information on Zapier’s Total Rewards please click<a href="https://zapier.com/l/jobs/total-rewards"> here</a>.</p> <h2><strong>How to Apply</strong></h2> <p>At Zapier, we believe that diverse perspectives and experiences make us better, which is why we have a non-standard application process designed to promote inclusion and equity. We're looking for the best fit for each of our roles, regardless of the type of education or companies in your background, so we encourage you to apply even if your skills and experiences don’t exactly match the job description. All we ask is that you answer a few in-depth questions in our application that would typically be asked at the start of an interview process. This helps speed things up by letting us get to know you and your skillset a bit better right out of the gate. Please be sure to answer each question; the resume and CV fields are optional.</p> <p>After you apply, you are going to hear back from us—even if we don’t see an immediate fit with our team. In fact, throughout the process, we strive to never go more than seven days without letting you know the status of your application. We know we’ll make mistakes from time to time, so if you ever have questions about where you stand or about the process, just ask your recruiter!</p> <p>Zapier is an equal-opportunity employer and we're excited to work with talented and empathetic people of all identities. Zapier does not discriminate based on someone's identity in any aspect of hiring or employment as required by law and in line with our commitment to Diversity, Inclusion, Belonging and Equity. Our <a href="https://zapier.com/jobs/zapier-code-of-conduct">code of conduct</a> provides a beacon for the kind of company we strive to be, and we celebrate our differences because those differences are what allow us to make a product that serves a global user base. Zapier will consider all qualified applicants, including those with criminal histories, consistent with applicable laws.</p> <p>Zapier is committed to inclusion. As part of this commitment, Zapier welcomes applications from individuals with disabilities and will work to provide reasonable accommodations. If reasonable accommodations are needed to participate in the job application or interview process, please contact <a href="mailto:jobs@zapier.com">jobs@zapier.com</a>.</p> <p><strong>Application Deadline:</strong> The anticipated application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later, or if the position is filled.</p> <p>Even though we’re an all-remote company, we still need to be thoughtful about where we have Zapiens working. Check out<a href="https://docs.google.com/document/d/1VzesvLsgVG2RDRPtGvBF8CbcB9ZA-Hsddg0P2ildBb8/edit"> this resource</a> for a list of countries where we currently cannot have Zapiens permanently working.</p> <p>#LI-Remote</p>
Engineering Manager, Workflow Experie...
NAMER
<p><strong>Job Posted: 9/29/23</strong></p> <p><strong>L</strong><strong>ocation: New Zealand time zone to Pacific time zone</strong></p> <p>Hi there!</p> <p>We're looking for an <strong>Engineering Manager </strong>to join the team at Zapier. We’re on a mission to make everyone more productive at work. Zapier has helped millions of people build businesses through the power of automation.</p> <p>Part of Zapier’s Workflow Experience zone, the <strong>Observe</strong> team is on a mission to revolutionize admins’ interaction with our platform, focusing on enhancing visibility, insights, and trust. Our aim includes:</p> <ul> <li>Identifying opportunities to provide admins with real-time insights into task usage and performance, increasing value awareness across Zapier.</li> <li>Collaborating closely with Design, Product, and Engineering to create an intuitive, user-centric experience.</li> <li>Owning the entire process of notification management, performance reporting, and log content refinement.</li> <li>Continuously innovating and implementing new features such as a resolution center and alarm capabilities to proactively manage potential issues.</li> <li>Constantly sharing learnings and insights with other teams, promoting knowledge exchange within Zapier.</li> </ul> <p>We’re a globally distributed team, from New Zealand to the Americas, working together to make Zapier the trusted, go-to partner for business automation.</p> <p>If you’re interested in advancing your career at a fast-growing, profitable, impact-driven company, then read on…</p> <p><a href="https://zapier.com/jobs/our-commitment-to-applicants/">Our Commitment to Applicants</a></p> <p><a href="https://zapier.com/jobs/culture-and-values-at-zapier/">Culture and Values at Zapier</a></p> <p><a href="https://zapier.com/learn/remote-work/">Zapier Guide to Remote Work</a></p> <p><a href="https://zapier.com/jobs/zapier-code-of-conduct/">Zapier Code of Conduct</a></p> <p><a href="https://zapier.com/jobs/working-on-diversity-and-inclusivity/">Diversity and Inclusivity at Zapier</a></p> <h2>About You</h2> <ul> <li><strong>You are an effective team builder</strong>. This isn't your first rodeo. You have at least four years of experience managing teams of at least 4-5 engineers Backend &amp; Frontend Engineers. You know how to hire, train, and develop engineers from all backgrounds. You understand the benefits of building a diverse and inclusive engineering team.</li> <li><strong>You have experience with experimentation</strong>. Utilizing experiments to drive continuous improvements. &nbsp;These include a combination of bold changes (”shark bites”) and smaller improvements (”fish bites”). &nbsp;You understand how to optimize for rapid experimentation and follow-up iteration.</li> <li><strong>You’re a product thinker</strong>. As an Engineering Manager, you’ve worked to set goals, metrics, and a strategy for a product-focused Engineering team. You push your team to prioritize impactful work to meet the company’s goals. &nbsp;You can articulate how engineering-driven projects (aka addressing tech debt) help the team meet team and company goals, and can work with your product counterpart to prioritize them appropriately.</li> <li><strong>You can keep track of, prioritize, and lead multiple projects</strong>. There's no shortage of things you could be doing in a day. You'll carve out time for functional projects and make sure they solve real problems the team faces -- ideally making development even easier.</li> <li><strong>You have excellent communication skills</strong>. You regularly work with engineers and other stakeholders from various disciplines, balancing engineering concerns, such as technical debt, with product concerns. Ideally, you find solutions that address both the team and user needs, but if not, help build understanding around difficult decisions.</li> <li><strong>You are a skilled engineering leader</strong>. You've got a strong background in leading a team of Engineers working in our stack (Python, Django or React, AWS infrastructure, etc.). &nbsp;You understand how your team’s services work and can provide feedback on designs proposed by your engineers. You hold a high standard for operational excellence, monitoring service health and addressing issues.</li> <li><strong>You're adaptable</strong>. You've been in fast-growing companies and know how to build, change, and adapt to the needs of a company as it grows.</li> </ul> <h2>Things You'll Do</h2> <ul> <li>Work closely with your engineering function, peer design function, and product function to create a trio that tackles user problems and solutions.</li> <li>Develop effective ways to communicate, monitor, and lead your team through weekly one-on-ones and team meetings.</li> <li>Keep the leadership team informed on your team’s progress in ways that are easy and enjoyable to receive, like one-on-ones, update posts, and regular team hangouts.</li> <li>Build rapport with each member of the Engineering Team and support them through coaching and mentorship to help level up their skills.</li> <li>Drive continuous improvement through sprint retros, process improvements, and removal of technical blockers.</li> <li>Ensure your team defines and executes technical solutions in alignment with Zapier security requirements, architectural guidelines, system design principles, and engineering best practices.</li> <li>Collaborate across various disciplines (Product Managers, Designers, Researchers) to help set quarterly roadmaps that will achieve business goals through engineering best practices.</li> <li>Actively recruit, onboard, and train new engineers at Zapier. This might involve tweaking the skills portions of interviews or writing better documentation.</li> <li>Keep in touch with customer needs by reviewing or participating in customer interviews, sales calls, and user testing sessions.</li> </ul> <h2>Zapier Compensation Guiding Principles</h2> <p>We believe all Zapiens should be rewarded competitively and equitably, using practices that are simple and transparent. This philosophy ensures we’re able to find, grow, and retain exceptional people from a broad range of backgrounds. Here’s how we define our compensation principles:</p> <ul> <li>Competitive: Zapier pays well among the technology sector.</li> <li>Equitable: Consistent pay practices; Pay for impact.</li> <li>Simple: Pay is well understood, and pay practices are built for scale.</li> <li>Transparent: Zapiens know how pay works, including how their pay is determined.</li> </ul> <p><strong>The pay ranges for this role are (min - mid - max):</strong></p> <ul> <li>New Zealand 184,100 - 230,100 - 276,100 NZD</li> <li>United States 184,100 - 230,100 - 276,100 USD</li> <li>Canada CAD 184,100 - 230,100 - 276,100 CAD</li> </ul> <p>A Candidate's compensation package is finalized once the interview process is concluded and accounts for demonstrated experience, job knowledge, skills,&nbsp; abilities, and internal equity. We use a business impact approach to base pay, which means we set pay for all Zapier employees based on their demonstrated impact to Zapier’s success. In alignment with that philosophy, the upper half of a pay range is typically reserved for individuals who have consistently demonstrated a high impact in their current role and level while at Zapier.</p> <p>For more information on Zapier’s Total Rewards, please click<a href="https://zapier.com/l/jobs/total-rewards"> here</a>.</p> <h2>How to Apply</h2> <p>At Zapier, we believe that diverse perspectives and experiences make us better, which is why we have a non-standard application process designed to promote inclusion and equity. We're looking for the best fit for each of our roles, regardless of the type of education or companies in your background, so we encourage you to apply even if your skills and experiences don’t exactly match the job description. All we ask is that you answer a few in-depth questions in our application that would typically be asked at the start of an interview process. This helps speed things up by letting us get to know you and your skillset a bit better right out of the gate. Please be sure to answer each question; the resume and CV fields are optional.</p> <p>After you apply, you are going to hear back from us—even if we don’t see an immediate fit with our team. In fact, throughout the process, we strive to never go more than seven days without letting you know the status of your application. We know we’ll make mistakes from time to time, so if you ever have questions about where you stand or about the process, just ask your recruiter!</p> <p>Zapier is an equal-opportunity employer and we're excited to work with talented and empathetic people of all identities. Zapier does not discriminate based on someone's identity in any aspect of hiring or employment as required by law and in line with our commitment to Diversity, Inclusion, Belonging and Equity. Our<a href="https://zapier.com/jobs/zapier-code-of-conduct/"> code of conduct</a> provides a beacon for the kind of company we strive to be, and we celebrate our differences because those differences are what allow us to make a product that serves a global user base. Zapier will consider all qualified applicants, including those with criminal histories, consistent with applicable laws.</p> <p>Zapier is committed to inclusion. As part of this commitment, Zapier welcomes applications from individuals with disabilities and will work to provide reasonable accommodations. If reasonable accommodations are needed to participate in the job application or interview process, please contact <a href="mailto:jobs@zapier.com">jobs@zapier.com</a>.</p> <p><strong>Application Deadline:</strong> The anticipated application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later, or if the position is filled.</p> <p>Even though we’re an all-remote company, we still need to be thoughtful about where we have Zapiens working. Check out<a href="https://docs.google.com/document/d/1VzesvLsgVG2RDRPtGvBF8CbcB9ZA-Hsddg0P2ildBb8/edit"> this resource</a> for a list of countries where we currently cannot have Zapiens permanently working.</p> <p>#LI-Remote</p>
Sr. Engineer, Backend
NAMER
<p><strong>Job Posted: 2/9/24&nbsp;</strong></p> <p><strong>Location: Americas&nbsp;</strong></p> <p>Hi there!</p> <p>As Zapier continues to scale our mission to democratize automation, we’re hiring a <strong>Senior Backend Engineer</strong> to the Edge team.&nbsp;&nbsp;</p> <p>The <strong>Edge team</strong> is focused on building internal automation services that serve as a basis to power fantastic products. Our services are critical, run at scale, and under the hood power all of Zapier. We run at the <em>edge </em>of Zapier’s system boundary, handling both incoming traffic and outgoing requests to our partners.</p> <p>Zapier’s on a mission to make everyone more productive at work. Zapier has helped millions of people build businesses through the power of automation.</p> <p>If you’re interested in advancing your career at a fast-growing, profitable, impact-driven company, then read on…</p> <p><em>We know applying for and taking on a new job at any company requires a leap of faith. We want you to feel comfortable and excited to apply at Zapier. To help share a bit more about life at Zapier, here are a few resources in addition to the job description that can give you an inside look at what life is like at Zapier. Hopefully, you'll take the leap of faith and apply.</em></p> <p><a href="https://zapier.com/jobs/our-commitment-to-applicants/">Our Commitment to Applicants</a></p> <p><a href="https://zapier.com/jobs/culture-and-values-at-zapier/">Culture and Values at Zapier</a></p> <p><a href="https://zapier.com/learn/remote-work/">Zapier Guide to Remote Work</a></p> <p><a href="https://zapier.com/jobs/zapier-code-of-conduct/">Zapier Code of Conduct</a></p> <p><a href="https://zapier.com/jobs/working-on-diversity-and-inclusivity/">Diversity and Inclusivity at Zapier</a></p> <p><em>Zapier is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.</em></p> <p>Even though our job description may seem like we're looking for a specific candidate, the role inevitably ends up tailored to the person who applies and joins. Regardless of how well you feel you fit our description, we encourage you to apply if you meet these criteria:</p> <h2>&nbsp;</h2> <h2><strong>About You</strong></h2> <p><strong>You have 7+ years of experience.</strong> You've dedicated at least 4 of those years to building distributed,&nbsp; scalable web applications.</p> <p><strong>Required Technical Skills/Experience:</strong></p> <ol> <li>Strong expertise in using <strong>Python</strong> for building web services.</li> <li>Strong experience with <strong>AWS </strong>and <strong>cloud technologies, </strong>especially <strong>Lambda&nbsp;</strong></li> <li>Experience working with <strong>Kafka</strong>, <strong>SQS</strong>, or other event based technologies</li> <li>Experience in working with <strong>MySQL</strong>, with skills in database performance optimization and query tuning.</li> <li>Experience working with <strong>Memcached</strong> or <strong>Redis</strong></li> <li>Using tools like <strong>Grafana, Prometheus, </strong>and<strong> Datadog </strong>and for monitoring and performance optimization, and to identify and address performance bottlenecks.</li> </ol> <p><strong>Nice-to-Have Technical Skills/Experience:</strong></p> <ol> <li>Knowledge of CI/CD pipelines and infrastructure as code (e.g., using a tool like GitLab and Terraform).</li> <li>Proficiency in implementing and securing authentication and authorization mechanisms, such as OAuth, JWT, role-based access control (RBAC), and session management.</li> </ol> <p><strong>You value collaboration.</strong> You understand that building modern software is a team sport, and you enjoy working as part of a tight-knit team. You’re happy to pitch in and help the team, whether by reviewing code, pairing on a tricky problem, or just thinking about how to solve the challenges we’re facing.</p> <p><strong>You value exploration and versatility.</strong> You are comfortable working in other teams’ codebases to achieve the Edge team’s goals. This will involve embedding yourself in the other team, becoming familiar with their developer workflow, and collaborating with stakeholders on the other team to implement your solutions.</p> <p><strong>You can balance lots of concerns. </strong>Our team manages a wide variety of services catering to a broad spectrum of internal clients. You'll be expected to adeptly balance incoming work to address these services, prioritizing them ruthlessly yet judiciously. Your ability to manage this diversity without being overwhelmed is crucial, ensuring not only that we meet our varied demands but also consistently deliver high-quality, impactful results in our dynamic, fast-paced environment.</p> <p><strong>You advocate for the user.</strong> You have a keen eye for great design, and you’re empathetic to the needs of the end-user. When you see users struggling to succeed you take it as a personal challenge to understand why and help the team build a better product.</p> <p><strong>You love learning new things. </strong>You love researching new tech and driving forward with the implementation details.&nbsp;</p> <h2>&nbsp;</h2> <h2><strong>Things You Might Do</strong></h2> <p>Zapier is a small, fast-growing, and remote-first company, so you'll likely get experience on many different projects across the organization. That said, here are some things you'll probably do:</p> <ul> <li>Directly impact how our customers can be more productive by building and supporting the foundational building blocks of Zapier.</li> <li>Work with AWS services like SQS, Redis, S3, Lambda and MSK to build scalable solutions that process millions of requests. Use Terraform to maintain and build our infrastructure.</li> <li>Refactor or improve existing code as languages, frameworks, or techniques evolve. Help the team pick appropriate tools to solve new problems as they arise.</li> <li>Work closely with our product, design, and frontend teams to create amazing and intuitive experiences that make it effortless to connect different apps together.</li> <li>Help put tools, processes, and documentation in place to help us become a better, more effective organization.</li> <li>Help teach your colleagues new skills, through code review, discussions and mentoring. Help us all become better engineers and humans.</li> <li>Ship to hundreds of thousands of users every day while having lots of autonomy in terms of code and feature ownership.</li> <li>As a part of Zapier's<a href="https://workable.com/nr?l=https%3A%2F%2Fzapier.com%2Flearn%2Fcustomer-support%2Feveryone-on-support%2F"> all-hands philosophy</a>, help customers via support to ensure they have the best experience possible.</li> </ul> <p>You’ll also have the opportunity to specialize in a variety of areas of the Zapier codebase. Focusing on a specialization will not limit your growth at Zapier as we believe that each engineer brings a unique perspective and can contribute in all areas. We encourage collaboration and will frequently have engineers contribute across teams to assist with projects.</p> <h2>&nbsp;</h2> <h2><strong>Zapier Compensation Guiding Principles</strong></h2> <p>We believe all Zapiens should be rewarded competitively and equitably, using practices that are simple and transparent. This philosophy ensures we’re able to find, grow, and retain exceptional people from a broad range of backgrounds. Here’s how we define our compensation principles:</p> <ul> <li>Competitive: Zapier pays well among the technology sector.</li> <li>Equitable: Consistent pay practices; competency-based pay.</li> <li>Simple: Pay is well understood, and pay practices are built for scale.</li> <li>Transparent: Zapiens know how pay works, including how their pay is determined.</li> </ul> <p><strong>The pay ranges for this role are:</strong></p> <p>United States: $167,000 - $251,000 USD</p> <p>Canada: $167,000 - $251,000 CAD</p> <p>A Candidate's compensation package is finalized once the interview process is concluded and accounts for experience, competencies (job knowledge, skills and abilities) and internal equity. We use a competency-based approach to base pay, which means we set pay for all Zapier employees based on their competency and skills demonstrated in their role. In alignment with that philosophy, the upper half of a pay range is typically reserved for individuals who have consistently demonstrated a high level of job knowledge and skills for their current role and level while at Zapier.</p> <p>For more information on Zapier’s Total Rewards please click<a href="https://zapier.com/l/jobs/total-rewards"> here</a>.</p> <h3>&nbsp;</h3> <h2><strong>How to Apply</strong></h2> <p>At Zapier, we believe that diverse perspectives and experiences make us better, which is why we have a non-standard application process designed to promote inclusion and equity. We're looking for the best fit for each of our roles, regardless of the type of education or companies in your background, so we encourage you to apply even if your skills and experiences don’t exactly match the job description. All we ask is that you answer a few in-depth questions in our application that would typically be asked at the start of an interview process. This helps speed things up by letting us get to know you and your skillset a bit better right out of the gate. Please be sure to answer each question; the resume and CV fields are optional.</p> <p>After you apply, you are going to hear back from us—even if we don’t see an immediate fit with our team. In fact, throughout the process, we strive to never go more than seven days without letting you know the status of your application. We know we’ll make mistakes from time to time, so if you ever have questions about where you stand or about the process, just ask your recruiter!</p> <p>Zapier is an equal-opportunity employer and we're excited to work with talented and empathetic people of all identities. Zapier does not discriminate based on someone's identity in any aspect of hiring or employment as required by law and in line with our commitment to Diversity, Inclusion, Belonging and Equity. Our<a href="https://zapier.com/jobs/zapier-code-of-conduct/"> code of conduct</a> provides a beacon for the kind of company we strive to be, and we celebrate our differences because those differences are what allow us to make a product that serves a global user base. Zapier will consider all qualified applicants, including those with criminal histories, consistent with applicable laws.</p> <p>Zapier is committed to inclusion. As part of this commitment, Zapier welcomes applications from individuals with disabilities and will work to provide reasonable accommodations. If reasonable accommodations are needed to participate in the job application or interview process, please contact <a href="mailto:jobs@zapier.com">jobs@zapier.com</a>.</p> <p><strong>Application Deadline: </strong>The anticipated application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later, or if the position is filled.</p> <p><em>Even though we’re an all-remote company, we still need to be thoughtful about where we have Zapiens working. Check out</em><a href="https://docs.google.com/document/d/1VzesvLsgVG2RDRPtGvBF8CbcB9ZA-Hsddg0P2ildBb8/edit"> this resource</a><em> for a list of countries where we currently cannot have Zapiens permanently working.</em></p> <p>#LI-Remote</p>
Verified by
Co-founder
You may also like