Rust at OneSignal

4,655
OneSignal
OneSignal is a high volume mobile push, web push, email, and in-app messaging service.

This post is by Joe Wilm of OneSignal

Earlier last year, we announced OnePush, our notification delivery system written in Rust.

In this post, we will cover improvements in our delivery capabilities since then, an interactive tour of OnePush’s subsystems and reflections of our experience shipping production Rust code. We hope you'll find it insightful!

Delivery Stats

OnePush was built to scale deliveries at OneSignal. To know whether this endeavor was a success, we collect metrics such as historical delivery counts and delivery throughput. Here's how OnePush is performing:

  • OneSignal had ~10,000 users at the start of 2016 and now has over 110,000 at the time of publishing this post. (Over 10x growth!)
  • We've increased the number of daily notifications sent by 20x in the same period.
  • OnePush delivers over 2 billion notifications per week.
  • OnePush is fast - We've observed sustained deliveries up to 125,000/second and spikes up to 175,000/second.

The title image on this post is a screenshot from our live delivery monitoring. Each bar represents deliveries occurring in that second, and each vertical division denotes 5,000 deliveries. The colors represent different platforms like iOS, Android, Chrome WebPush, etc. Every single one of them was delivered by OnePush.

OnePush

OnePush is comprised of several subsystems for loading notifications, delivering notifications across HTTP/1.1 and HTTP/2, and for processing events and results.

Choosing Rust

Choosing the programming language for a core system is a big decision. If not careful, one could end up with months of time invested and get stuck writing library code instead of the application itself. This is less of a concern with programming languages that have a mature ecosystem, but that's not exactly Rust just yet. On the other hand, Rust enables one to build robust, complex systems quickly and fearlessly thanks to its powerful type system and ownership rules.

Given that we now have a production system written in Rust, it's obvious which side of this trade we landed on. Our experience has been positive overall and indeed we have had fantastic results. The following sections discuss the specific pros and cons we considered for building OnePush in Rust, what risks we accepted on the outset, the successes we had, and issues we ran into.

Reasons to not use Rust

The Rust ecosystem is young. Even if there exists a library for your purpose, it's not guaranteed to be robust enough for a production deployment. Additionally, many libraries today have a "truck factor" of 1. If the library's developer gets hit by a truck, it's going to be on you to maintain it.

Next, Rust's tooling story is weak. You can use tools like Racer and YCM to get pretty far, but they fail in a lot of cases. Good tooling is a necessity, especially for developers that are getting up-to-speed.

Having team members (who may be unfamiliar with Rust) contribute to the project may take a lot of "ramp-up" time. This risk has turned out to be quite real, but it hasn't stopped other members of our team from contributing patches to the project. Mentoring from team members more proficient with the language and familiar with the code base helped a lot here.

Finally, iteration times can be long. This wasn't something we anticipated up front, but build times have become onerous for us. A build from scratch now falls into the category of "go make coffee and play some ping-pong." Recompiling a couple of changes isn't exactly quick either.

Before settling on Rust, we considered writing OnePush in Go. Go has a lot going for it for this sort of application - its concurrency model is perfectly suited for managing many async TCP connections, and the ecosystem has good libraries for HTTP requests, Redis and PostgreSQL clients, and serialization. Go is also more approachable for someone unfamiliar with the language; this makes the code base more accessible to the rest of your team. Go's developer tools have also had more time to mature than Rust's.

Why choose Rust

Despite the negatives and the presence of a good alternative, Rust has a lot going for it that makes it a good choice for us. As mentioned earlier,

Rust enables one to build robust, complex systems quickly and fearlessly thanks to its powerful type system and ownership rules

This is huge. Being able to encode constraints of your application in the type system makes it possible to refactor, modify, or replace large swaths of code with confidence. The type system is our ultimate "move quickly and don't break things" secret weapon.

Rust's error handling model forces developers to handle every corner case. Even if there is a system with the potential to panic, it can be moved into its own thread for isolation. More recently, it has become possible to catch panics within a thread instead of only at the boundary. Languages like Go make it too easy to ignore errors.

Next, OnePush needed to be fast. Rust makes writing multithreaded programs quite easy. The Send and Sync traits work together to ensure such programs are free from data races.

At the end of the day, our OnePush service is just a program optimized for sending a lot of HTTP requests. The library ecosystem offered everything we needed to build this system: An async HTTP/2 client, an async HTTP/1.1 client, a Redis client library and a PostgreSQL client library. We are fortunate that the Rust community is full of talented and ambitious developers who have already published a great deal of quality libraries that suit our specific needs.

Finally, the developer leading the effort had experience and a strong preference for Rust. There are plenty of technologies that would have met our requirements, but deferring to a personal preference made a lot of sense. Having engineers excited about what they are working on is incredibly valuable. Such intrinsic motivation increases developer happiness and reduces burnout. Imagine going to work every day and getting to work on something you're excited about! Developer happiness is important to us as a company. Being able to provide so much by going with one technology versus another was a no-brainer.

Risks

Aside from risks associated with not choosing Rust, we had a few additional concerns for this particular project.

As a glorified HTTP client, OnePush needed to be able to send lots of HTTP/1.1 requests very quickly. In the beginning, this wasn't quite as true because of our scale and because Android notifications could be batched into single requests. Going forward, we expected a huge increase in HTTP/1.1 outgoing request volume due to growth and the new WebPush specification with encrypted payloads. Hyper (Rust's HTTP library), had an async branch that was just a prototype when we started. We hoped that, by the time we truly needed an async client, it would be ready.

As it turned out, the initial async Rotor-based branch of Hyper never stabilized since tokio and futures were announced in August 2016. By the time we really needed the async branch, we ended up having to spend a week or two debugging, stress-testing and fixing the Rotor-based hyper::Client. This turned out to be ok since it was a chance to give back to the Rust community.

Since we would be on the nightly channel for serde derive and clippy lints, another risk was spending a lot of time doing rustc upgrades. We avoided this situation by pinning to specific versions of the compiler and upgrading infrequently. When we did upgrade, the process required finding a recent rustc that was supported by both libraries. This will become less of an issue very soon with the advent of Macros 1.1.

Finally, Solicit (Rust's HTTP/2 library) uses three threads per connection. Although this is fine in isolation, having 20,000 connections quickly becomes expensive. We've mitigated this issue by using a short keep-alive to limit the number of active connections and by taking advantage of the Apple's HTTP/2 provider API (APNs), which allows 500 requests in-flight per connection.

Unexpected Issues

For the most part, we knew what we were getting into building such a system in Rust. However, one thorn in our side that we didn't anticipate was rust-openssl upgrades. We are stuck on an earlier version of rust-openssl since the Solicit library depends on an API that has been removed since v0.8.0. This means that we are unable to upgrade other dependencies which rely on rust-openssl until we fix the Solicit issue.

Another minor issue at one time was the limited test framework. A common feature for test frameworks is to have some setup and teardown steps that run before and after a test. We say this issue was minor because we were able to work around its absence by generating many tests declaratively with macros (discussed below).

Successes

Writing OnePush in Rust has been hugely successful for us. We've been able to easily meet our performance and scaling goals with the application. OnePush is capable of delivering over 100k notifications per second and efficiently maximizes the use of system resources. Despite being highly multithreaded, race conditions have not been an issue for us. Even better, OnePush needs very little attention. We were able to leave it running without any issues through the holiday break.

Regressions are very infrequent. There's a huge class of bugs in languages like Ruby that just aren't possible in Rust. When combined with good test coverage, it becomes difficult to break things - all thanks to Rust's fantastic type system. This isn't just about regressions either. The compiler and type system make refactoring basically fool-proof. We like to say that Rust enables belligerent refactoring - making dramatic changes and then working with the compiler to bring your project back to a working state.

The macro system has been another big win. Our favorite example of how this saves us engineering time is using macros for writing tests declaratively. For example, a large set of tests we have are for the Terminal. Each test takes some Events as input, and then the state of Redis and Postgres are checked to be correct after processing the event. The macro system enabled us to remove all of the boilerplate for these tests and declaratively say what the event is and what the expected outcome should be. Writing a test for this system today looks like this:

// Invoking terminal test-writing macro
push_test! {
    // The part before the arrow ends up being the test name.
    // The `response` describes an `Event`, and the rest describes the system
    // state after processing it. There are more parameters that can be
    // specified, but the default values are acceptable in this case.
    apns_success => {
        response: apns::Response::Success,
        success: 1,
        sending_done: true
    },
    // .. and so on
}

Writing a lot of similar tests in this fashion enables us to get a lot of coverage without a lot of work. It also helps us work around the lack of features in the Rust test system (such as before/after hooks).

The final thing we want to comment on here is serde. This library enables adding a #[derive(Deserialize)] attribute to a struct and getting a deserialize implementation. Combined with our serde-redis library, this makes it possible to load data out of Redis like so:

/// A person has a name and an ID.
///
/// This is just some data with a derived
/// Deserialize implementation
#[derive(Deserialize)]
struct Person {
    name: String,
    id: u64
}

// Gets a `Person` out of redis
let person: Person = redis.hgetall("person")?;

On the left hand side of the line fetching person, there's a binding name with a type annotation. On the right hand side, there's a call to Redis with HGETALL, and a ?. The ? is a bit of error handling; if the request is successful and deserialization works, person will be a valid Person, and the name and id fields can be used directly with knowledge that they were returned from Redis. If something goes wrong, like Redis is unreachable or there is data missing for the Person (such as a missing id), an error is returned from the current function.

This is really powerful! We can just describe our data, add this derive attribute and then safely load the data out of Redis. To get the same effect in a dynamic language, one would need to load this dictionary out of Redis and write a bunch of boilerplate to validate that the returned fields are correct. This sort of thing makes Rust more expressive than many high-level languages.

Open Source

Early adoption in an ecosystem means there are lots of opportunities for open source contributions. The most notable of our contributions is a project called serde-redis, a Redis deserialization backend for serde. We've also had the opportunity to contribute several patches to Hyper's Rotor-based async client. We use that client in OnePush and have made billions of HTTP requests with it.

What's next

We've come far with OnePush, but there's still more work to do! Here's just a few of our upcoming projects related to OnePush:

  • Upgrade to Hyper's Tokio-based async implementation. We probably won't be super early adopters here since we've got an HTTP client with a lot of production miles on it right now.
  • Rework result processing to use futures. The Terminal's concurrency from threads is limited, whereas something backed by mio could have much higher throughput. This would require futures compatible Redis and Postgres clients.
  • Replace Solicit's thread-based async client with a mio-based one. We've actually got a prototype of something from earlier in 2016.

We also have a new internal application written in Rust which we hope to blog about soon! It's a core piece of our monitoring which is responsible for collecting statistics from our production systems and storing them in InfluxDB.

Conclusion

We've had fantastic results building one of our core systems in Rust. It has delivered many billions of notifications, and it's delivering more and more each day. We hope that sharing our experience as early adopters in the Rust ecosystem will be helpful to others when making similar decisions. We've certainly found Rust to be a secret weapon for quickly building robust systems.

Like what we're doing? We're hiring!

OneSignal
OneSignal is a high volume mobile push, web push, email, and in-app messaging service.
Tools mentioned in article
Open jobs at OneSignal
Design Engineer
United States ()
<div class="content-intro"><p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">OneSignal is a leading omnichannel customer engagement solution, powering personalized customer journeys across mobile and web push notifications, in-app messaging, SMS, and email. On a mission to democratize engagement, we enable over a million businesses to keep their users - including readers, fans, players and shoppers - engaged and up to date by delivering 12 billion messages </span><em><span style="font-weight: 400;">daily.&nbsp;</span></em></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">1 in 5 new apps launches using OneSignal! We support companies in 140 countries, including Zynga, USA Today, Bitcoin.com, Eventbrite, Tribune, and many more - from startups and small businesses just getting off the ground to established companies communicating with millions of customers.</span></p> <p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">We’re Series C, venture-backed by SignalFire, Rakuten Ventures, Y Combinator, HubSpot, and BAM Elevate</span><span style="font-weight: 400;">. We offer</span><span style="font-weight: 400;"> remote work as the default option in the United States in California, New York, Pennsylvania, Texas, Utah and Washington. As well as in the UK and Singapore - with plans to expand the locations we support in the future. Some roles are hybrid roles and will be listed as such. We have offices in&nbsp;</span><span style="font-weight: 400;">San Mateo, CA and London, UK, and offer flex seating options for employees to work together in-person where we don't have offices. Hiring in Singapore is done in partnership with a local PEO.</span></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">OneSignal has a lot of the great tech startup qualities you'd expect, but we don't stop there. Our massive scale and small team, emphasis on healthy life balance and kindness in all our interactions, and focus on ownership and personal growth make OneSignal a uniquely great place to work.&nbsp;</span></p></div><h3><strong>About The Team:</strong></h3> <p><span style="font-weight: 400;">Our design team is a nimble group that spans user research, user experience, prototyping, visual design, brand design, design systems and frontend engineering. We believe design is equal parts form and function. We ideate a lot. We prototype a lot. We put our users first and continually aim to make our product more intuitive and delightful to use. We run cross functional workshops to influence design thinking across the company. We have a ‘ship it’ mentality and have a passion for always measuring and learning,&nbsp; care about building easy to use products, and have a knack for using data to make better design decisions.&nbsp;</span></p> <h3>What You'll Do:</h3> <ul> <li>Work directly on the UI and UX of our web products</li> <li>Partner with designers to develop OneSignal’s design system, “Beam”</li> <li>Partner with engineers to implement components and add design polish to our products</li> <li>Educate and be an evangelist for a11y, semantics and design system best practices&nbsp;</li> <li>Document our components and UX patterns, how they work, how designers can use them in Figma, how engineers can use them in code</li> <li>Create prototypes and tools to help us move faster at both discovery and execution</li> <li>Bring an engineering perspective to the design process to voice issues with feasibility and call out missed edge-cases</li> <li>Be a constant advocate for collaboration as well as act as a bridge between design and engineering</li> </ul> <h3>What You'll Bring:</h3> <ul> <li>3+ years experience working on a web platform</li> <li>Strong, provable design skills, specifically around accessibility and interaction design</li> <li>Intuitive mind for good user experience</li> <li>Effective communication skills with other team members (<a class="postings-link" href="https://onesignal.com/blog/design-team-meetings/">how we structure design team meetings</a>)</li> <li>Experience building design systems</li> <li>Obsessive attention to pixel perfection</li> <li>Portfolio of past work showcasing your process</li> <li>Mastery of design and prototyping tools (<a class="postings-link" href="https://onesignal.com/blog/designing-with-figma/">we use Figma</a>)</li> <li>Understanding of technical requirements and aspects of modern apps</li> <li>Experience designing and shipping projects from start to finish</li> <li>Fluency with HTML, CSS and JavaScript (we use React and TypeScript)</li> </ul> <p>&nbsp;</p> <p><span style="font-weight: 400;">The New York and California base salary for this full time position is between $104,000 to $122,000. Your exact starting salary is determined by a number of factors such as your experience, skills, and qualifications. In addition to base salary, we also offer a competitive equity program and comprehensive and inclusive benefits.</span></p><div class="content-conclusion"><h3>Qualities we look for:</h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Friendliness &amp; Empathy</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Accountability &amp; Collaboration</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Proactiveness &amp; Urgency</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Growth Mindset &amp; Love of Learning</span></li> </ul> <p><em><span style="font-weight: 400;">In keeping with our beliefs and goals, no employee or applicant will face discrimination/harassment based on: race, color, ancestry, national origin, religion, age, gender, marital domestic partner status, sexual orientation, gender identity, disability status, or veteran status. Above and beyond discrimination/harassment based on 'protected categories,' we also strive to prevent other, subtler forms of inappropriate behavior (e.g., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place in our workplace.</span></em></p> <p><em><span style="font-weight: 400;">Applicants with disabilities may be entitled to reasonable accommodation under the terms of the Americans with Disabilities Act and certain state or local laws. A reasonable accommodation is a change in the way things are normally done which will ensure an equal employment opportunity without imposing undue hardship on OneSignal. Please inform us if you need assistance completing any forms or to otherwise participate in the application and/or interview process.</span></em></p> <p><em><span style="font-weight: 400;">OneSignal collects and processes personal data submitted by job applicants in accordance with our<a href="https://onesignal.com/privacy"> Privacy Policy</a> - including GDPR and CCPA compliance.&nbsp;Please see our <a href="https://onesignal.com/job-privacy" target="_blank">privacy notice for job applicants</a>.</span></em></p></div>
Senior Software Engineer, User Data T...
United States ()
<div class="content-intro"><p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">OneSignal is a leading omnichannel customer engagement solution, powering personalized customer journeys across mobile and web push notifications, in-app messaging, SMS, and email. On a mission to democratize engagement, we enable over a million businesses to keep their users - including readers, fans, players and shoppers - engaged and up to date by delivering 12 billion messages </span><em><span style="font-weight: 400;">daily.&nbsp;</span></em></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">1 in 5 new apps launches using OneSignal! We support companies in 140 countries, including Zynga, USA Today, Bitcoin.com, Eventbrite, Tribune, and many more - from startups and small businesses just getting off the ground to established companies communicating with millions of customers.</span></p> <p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">We’re Series C, venture-backed by SignalFire, Rakuten Ventures, Y Combinator, HubSpot, and BAM Elevate</span><span style="font-weight: 400;">. We offer</span><span style="font-weight: 400;"> remote work as the default option in the United States in California, New York, Pennsylvania, Texas, Utah and Washington. As well as in the UK and Singapore - with plans to expand the locations we support in the future. Some roles are hybrid roles and will be listed as such. We have offices in&nbsp;</span><span style="font-weight: 400;">San Mateo, CA and London, UK, and offer flex seating options for employees to work together in-person where we don't have offices. Hiring in Singapore is done in partnership with a local PEO.</span></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">OneSignal has a lot of the great tech startup qualities you'd expect, but we don't stop there. Our massive scale and small team, emphasis on healthy life balance and kindness in all our interactions, and focus on ownership and personal growth make OneSignal a uniquely great place to work.&nbsp;</span></p></div><p>Our <a href="https://onesignal.com/blog/tag/development/">blog </a>contains more information about the OneSignal Engineering<a href="https://onesignal.com/blog/how-to-introduce-an-engineering-career-ladder-to-your-company/"> career ladder</a>, and our<a href="https://onesignal.com/blog/tag/development/"> diverse team.</a></p> <h3><strong>About The Team:</strong></h3> <p>Our User Data team empowers OneSignal customers with a Customer Data Platform that serves as a real-time system of record for user and audience data and provides timely and useful insights to our customers so that they can optimally understand and engage their users.&nbsp;</p> <p>As a Senior Software Engineer, you'll have the autonomy to take ownership of significant projects and directly impact our platform's performance and features. Your expertise will shape the way businesses engage with their users. Working remotely, you'll have the flexibility to create a schedule that works best for you, allowing you to excel in both your professional and personal life.</p> <h3><strong>What You'll Do:</strong></h3> <ul> <li>Collaborate closely with Product Managers, Designers, and fellow engineers to design and implement new full-stack features and functionalities for our Customer Data Platform, using languages such as React/TypeScript, Ruby, Golang and Rust</li> <li>Actively participate in peer code reviews and Technical Design Spec reviews, providing valuable technical insights to continuously improve our code base</li> <li>Work together with the team to efficiently resolve production issues and ensure the system scales smoothly to meet the growing demands of our customers.</li> <li>Conduct data analysis and performance monitoring to identify areas for optimization and enhancement</li> <li>Stay up-to-date with the latest industry trends and technologies, incorporating new ideas into our engineering processes</li> <li>Ability to work independently in uncertainty and drive multiple experiments to derive at a solution to unblock business and customer operations.</li> <li>Work on customer driven product development</li> </ul> <h3><strong>What You'll Bring:</strong></h3> <ul> <li>6+ years of professional software development experience</li> <li>Experience building backend frameworks at scale</li> <li>Experience with Rust and/or Golang, or a strong willingness to learn these two languages quickly&nbsp;</li> <li>Experience with distributed system event streaming framework such as Apache Kafka</li> <li>Experience with Docker and Kubernetes</li> <li>Experience designing RESTful API</li> </ul> <h3><strong>We value a variety of experiences, and these are not required. It would be an added bonus if you have experience in any of the following:</strong></h3> <ul> <li>Experience with ScyllaDB</li> <li>Experience with Ruby/Rails</li> <li>Experience building a robust React Web application</li> <li>Experience with continuous build in an Agile Environment</li> <li>Have a good understanding of clean software design principles</li> </ul> <p><em><span style="font-weight: 400;">The New York and California base salary for this full time position is between $160,000 to $180,000. Your exact starting salary is determined by a number of factors such as your experience, skills, and qualifications. In addition to base salary, we also offer a competitive equity program and comprehensive and inclusive benefits.</span></em></p><div class="content-conclusion"><h3>Qualities we look for:</h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Friendliness &amp; Empathy</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Accountability &amp; Collaboration</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Proactiveness &amp; Urgency</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Growth Mindset &amp; Love of Learning</span></li> </ul> <p><em><span style="font-weight: 400;">In keeping with our beliefs and goals, no employee or applicant will face discrimination/harassment based on: race, color, ancestry, national origin, religion, age, gender, marital domestic partner status, sexual orientation, gender identity, disability status, or veteran status. Above and beyond discrimination/harassment based on 'protected categories,' we also strive to prevent other, subtler forms of inappropriate behavior (e.g., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place in our workplace.</span></em></p> <p><em><span style="font-weight: 400;">Applicants with disabilities may be entitled to reasonable accommodation under the terms of the Americans with Disabilities Act and certain state or local laws. A reasonable accommodation is a change in the way things are normally done which will ensure an equal employment opportunity without imposing undue hardship on OneSignal. Please inform us if you need assistance completing any forms or to otherwise participate in the application and/or interview process.</span></em></p> <p><em><span style="font-weight: 400;">OneSignal collects and processes personal data submitted by job applicants in accordance with our<a href="https://onesignal.com/privacy"> Privacy Policy</a> - including GDPR and CCPA compliance.&nbsp;Please see our <a href="https://onesignal.com/job-privacy" target="_blank">privacy notice for job applicants</a>.</span></em></p></div>
Staff Software Engineer, User Data Team
United States ()
<div class="content-intro"><p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">OneSignal is a leading omnichannel customer engagement solution, powering personalized customer journeys across mobile and web push notifications, in-app messaging, SMS, and email. On a mission to democratize engagement, we enable over a million businesses to keep their users - including readers, fans, players and shoppers - engaged and up to date by delivering 12 billion messages </span><em><span style="font-weight: 400;">daily.&nbsp;</span></em></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">1 in 5 new apps launches using OneSignal! We support companies in 140 countries, including Zynga, USA Today, Bitcoin.com, Eventbrite, Tribune, and many more - from startups and small businesses just getting off the ground to established companies communicating with millions of customers.</span></p> <p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">We’re Series C, venture-backed by SignalFire, Rakuten Ventures, Y Combinator, HubSpot, and BAM Elevate</span><span style="font-weight: 400;">. We offer</span><span style="font-weight: 400;"> remote work as the default option in the United States in California, New York, Pennsylvania, Texas, Utah and Washington. As well as in the UK and Singapore - with plans to expand the locations we support in the future. Some roles are hybrid roles and will be listed as such. We have offices in&nbsp;</span><span style="font-weight: 400;">San Mateo, CA and London, UK, and offer flex seating options for employees to work together in-person where we don't have offices. Hiring in Singapore is done in partnership with a local PEO.</span></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">OneSignal has a lot of the great tech startup qualities you'd expect, but we don't stop there. Our massive scale and small team, emphasis on healthy life balance and kindness in all our interactions, and focus on ownership and personal growth make OneSignal a uniquely great place to work.&nbsp;</span></p></div><p>Our <a href="https://onesignal.com/blog/tag/development/">blog </a>contains more information about the OneSignal Engineering<a href="https://onesignal.com/blog/how-to-introduce-an-engineering-career-ladder-to-your-company/"> career ladder</a>, and our<a href="https://onesignal.com/blog/tag/development/"> diverse team.</a></p> <h3><strong>About The Team:</strong></h3> <p>Our User Data team empowers OneSignal customers with a Customer Data Platform that serves as a real-time system of record for user and audience data and provides timely and useful insights to our customers so that they can optimally understand and engage their users.&nbsp;</p> <p>As a Staff Software Engineer, you'll have the autonomy to take ownership of significant projects and directly impact our platform's performance and features. Your expertise will shape the way businesses engage with their users. Working remotely, you'll have the flexibility to create a schedule that works best for you, allowing you to excel in both your professional and personal life.</p> <h3><strong>What You'll Do:</strong></h3> <ul> <li>Collaborate closely with Product Managers, Designers, and fellow engineers to design and implement new full-stack features and functionalities for our Customer Data Platform, using languages such as React/TypeScript, Ruby, Golang and Rust</li> <li>Actively participate in peer code reviews and Technical Design Spec reviews, providing valuable technical insights to continuously improve our code base</li> <li>Work together with the team to efficiently resolve production issues and ensure the system scales smoothly to meet the growing demands of our customers.</li> <li>Conduct data analysis and performance monitoring to identify areas for optimization and enhancement</li> <li>Stay up-to-date with the latest industry trends and technologies, incorporating new ideas into our engineering processes</li> <li>Ability to work independently in uncertainty and drive multiple experiments to derive at a solution to unblock business and customer operations.</li> <li>Work on customer driven product development</li> </ul> <h3><strong>What You'll Bring:</strong></h3> <ul> <li>8+ years of professional software development experience</li> <li>Experience building backend frameworks at scale</li> <li>Experience with Rust and/or Golang, or a strong willingness to learn these two languages quickly&nbsp;</li> <li>Experience with distributed system event streaming framework such as Apache Kafka</li> <li>Experience with Docker and Kubernetes</li> <li>Experience designing RESTful API</li> </ul> <h3><strong>We value a variety of experiences, and these are not required. It would be an added bonus if you have experience in any of the following:</strong></h3> <ul> <li>Experience with ScyllaDB</li> <li>Experience with Ruby/Rails</li> <li>Experience building a robust React Web application</li> <li>Experience with continuous build in an Agile Environment</li> <li>Have a good understanding of clean software design principles</li> </ul> <p><em><span style="font-weight: 400;">The New York and California base salary for this full time position is between $180,000 to $210,000. Your exact starting salary is determined by a number of factors such as your experience, skills, and qualifications. In addition to base salary, we also offer a competitive equity program and comprehensive and inclusive benefits.</span></em></p><div class="content-conclusion"><h3>Qualities we look for:</h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Friendliness &amp; Empathy</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Accountability &amp; Collaboration</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Proactiveness &amp; Urgency</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Growth Mindset &amp; Love of Learning</span></li> </ul> <p><em><span style="font-weight: 400;">In keeping with our beliefs and goals, no employee or applicant will face discrimination/harassment based on: race, color, ancestry, national origin, religion, age, gender, marital domestic partner status, sexual orientation, gender identity, disability status, or veteran status. Above and beyond discrimination/harassment based on 'protected categories,' we also strive to prevent other, subtler forms of inappropriate behavior (e.g., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place in our workplace.</span></em></p> <p><em><span style="font-weight: 400;">Applicants with disabilities may be entitled to reasonable accommodation under the terms of the Americans with Disabilities Act and certain state or local laws. A reasonable accommodation is a change in the way things are normally done which will ensure an equal employment opportunity without imposing undue hardship on OneSignal. Please inform us if you need assistance completing any forms or to otherwise participate in the application and/or interview process.</span></em></p> <p><em><span style="font-weight: 400;">OneSignal collects and processes personal data submitted by job applicants in accordance with our<a href="https://onesignal.com/privacy"> Privacy Policy</a> - including GDPR and CCPA compliance.&nbsp;Please see our <a href="https://onesignal.com/job-privacy" target="_blank">privacy notice for job applicants</a>.</span></em></p></div>
Senior Software Engineer, Core Servic...
United States ()
<div class="content-intro"><p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">OneSignal is a leading omnichannel customer engagement solution, powering personalized customer journeys across mobile and web push notifications, in-app messaging, SMS, and email. On a mission to democratize engagement, we enable over a million businesses to keep their users - including readers, fans, players and shoppers - engaged and up to date by delivering 12 billion messages </span><em><span style="font-weight: 400;">daily.&nbsp;</span></em></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">1 in 5 new apps launches using OneSignal! We support companies in 140 countries, including Zynga, USA Today, Bitcoin.com, Eventbrite, Tribune, and many more - from startups and small businesses just getting off the ground to established companies communicating with millions of customers.</span></p> <p><span style="font-family: arial, helvetica, sans-serif;"><span style="font-weight: 400;">We’re Series C, venture-backed by SignalFire, Rakuten Ventures, Y Combinator, HubSpot, and BAM Elevate</span><span style="font-weight: 400;">. We offer</span><span style="font-weight: 400;"> remote work as the default option in the United States in California, New York, Pennsylvania, Texas, Utah and Washington. As well as in the UK and Singapore - with plans to expand the locations we support in the future. Some roles are hybrid roles and will be listed as such. We have offices in&nbsp;</span><span style="font-weight: 400;">San Mateo, CA and London, UK, and offer flex seating options for employees to work together in-person where we don't have offices. Hiring in Singapore is done in partnership with a local PEO.</span></span></p> <p><span style="font-weight: 400; font-family: arial, helvetica, sans-serif;">OneSignal has a lot of the great tech startup qualities you'd expect, but we don't stop there. Our massive scale and small team, emphasis on healthy life balance and kindness in all our interactions, and focus on ownership and personal growth make OneSignal a uniquely great place to work.&nbsp;</span></p></div><h3><strong>About The Team:</strong></h3> <div> <p>OneSignal has grown rapidly. Today we are serving billions of HTTP requests and sending over 12 billion messages daily. We achieved this scale by leveraging bare metal cloud and writing scale-sensitive components in languages like Rust and Go. This potent combination of high-performance, low-cost hardware with efficient resource utilization has given us an incredible competitive edge.</p> <p>The Core Services team enables product development at OneSignal through the design, development, and operations of cross-cutting horizontal services which form the backbone of our technical stack. This includes the delivery pipeline, responsible for delivering over 12 billion push notifications, SMS, and emails every day, gRPC services written in Go and Rust, and Kafka consumers written in Rust. You will work with Rust, Go, Ruby, gRPC, Redis, Kafka, Scylla, and PostgreSQL</p> <p>As a Senior Software Engineer, you'll have the autonomy to take ownership of significant projects and directly impact our platform's performance and features. Your expertise will shape the way businesses engage with their users. Working remotely, you'll have the flexibility to create a schedule that works best for you, allowing you to excel in both your professional and personal life.</p> <h3><strong>What You'll Do</strong></h3> <ul> <li>Collaborate closely with fellow engineers to architect and implement services required to back OneSignal’s product offerings</li> <li>Actively participate in peer code reviews and Technical Design Spec reviews, providing valuable technical insights to continuously improve our code base</li> <li>Work together with the team to efficiently resolve production issues and ensure the system scales smoothly to meet the growing demands of our customers.</li> <li>Conduct data analysis and performance monitoring to identify areas for optimization and enhancement</li> <li>Stay up-to-date with the latest industry trends and technologies, incorporating new ideas into our engineering processes</li> <li>Participate in production on-call rotation</li> <li>Ability to work independently in uncertainty and drive multiple experiments to arrive at a solution to unblock business and customer operations</li> <li>Debug production issues by utilizing metrics, logs, and distributed traces</li> <li>Design synchronous and asynchronous APIs for communicating between services in a large distributed system</li> <li>Design storage schemas for relational and non-relational databases</li> <li>Improve performance of systems through benchmarking and profiling&nbsp;</li> </ul> <h3><strong>What You'll Bring</strong></h3> <ul> <li>At least 5 years experience working as a software engineer</li> <li>Experience operating reliable production systems at scale</li> <li>Easily bored running tasks by hand and the ability to automate such tasks</li> <li>Experience with relational databases</li> <li>Experience with distributed system event streaming framework such as Apache Kafka</li> <li>Experience with Rust and/or Golang, or a strong willingness to learn these two languages quickly&nbsp;</li> <li>Ability to reason about how data flows through distributed systems</li> <li>Experience with Docker and Kubernetes</li> </ul> <h3><strong>We value a variety of experiences, and these are not required. It would be an added bonus if you have experience in any of the following</strong></h3> <ul> <li>Experience with any of Redis, Kafka, Scylla, Apache Cassandra, gRPC</li> <li>Experience profiling applications to improve performance in terms of time/CPU/memory</li> <li>Experience debugging issues in distributed systems</li> </ul> </div> <p><span style="font-weight: 400;">The New York and California base salary for this full time position is between $160,000 to $180,000. Your exact starting salary is determined by a number of factors such as your experience, skills, and qualifications. In addition to base salary, we also offer a competitive equity program and comprehensive and inclusive benefits.</span></p><div class="content-conclusion"><h3>Qualities we look for:</h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Friendliness &amp; Empathy</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Accountability &amp; Collaboration</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Proactiveness &amp; Urgency</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Growth Mindset &amp; Love of Learning</span></li> </ul> <p><em><span style="font-weight: 400;">In keeping with our beliefs and goals, no employee or applicant will face discrimination/harassment based on: race, color, ancestry, national origin, religion, age, gender, marital domestic partner status, sexual orientation, gender identity, disability status, or veteran status. Above and beyond discrimination/harassment based on 'protected categories,' we also strive to prevent other, subtler forms of inappropriate behavior (e.g., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place in our workplace.</span></em></p> <p><em><span style="font-weight: 400;">Applicants with disabilities may be entitled to reasonable accommodation under the terms of the Americans with Disabilities Act and certain state or local laws. A reasonable accommodation is a change in the way things are normally done which will ensure an equal employment opportunity without imposing undue hardship on OneSignal. Please inform us if you need assistance completing any forms or to otherwise participate in the application and/or interview process.</span></em></p> <p><em><span style="font-weight: 400;">OneSignal collects and processes personal data submitted by job applicants in accordance with our<a href="https://onesignal.com/privacy"> Privacy Policy</a> - including GDPR and CCPA compliance.&nbsp;Please see our <a href="https://onesignal.com/job-privacy" target="_blank">privacy notice for job applicants</a>.</span></em></p></div>
Verified by
Cofounder & CEO, OneSignal
COO
You may also like