Improving Distributed Caching Performance and Efficiency at Pinterest

853
Pinterest
Pinterest is a social bookmarking site where users collect and share photos of their favorite events, interests and hobbies. One of the fastest growing social networks online, Pinterest is the third-largest such network behind only Facebook and Twitter.

Kevin Lin | Software Engineer, Storage and Caching


Introduction

Pinterest’s distributed caching system, built on top of open source technologies memcached and mcrouter, is a critical component of the production infrastructure stack. Pinterest’s cache-as-a-service platform is responsible for driving down application latency across the board, reducing the overall cloud cost footprint, and ensuring adherence to strict sitewide availability targets.

Today, Pinterest’s memcached fleet spans over 5000 EC2 instances across a variety of instance types optimized along compute, memory, and storage dimensions. Collectively, the fleet serves up to ~180 million requests per second and ~220 GB/s of network throughput over a ~460 TB active in-memory and on-disk dataset, partitioned among ~70 distinct clusters.

As a core driver of reduced sitewide latency, the distributed caching tier is subject to stringent performance and latency requirements. Additionally, a key consequence of the sheer size of the fleet is that even small efficiency optimizations have an outsized impact on the total service cost footprint. Several years of operational experience running memcached at scale in production have provided unique insight into practical optimizations for driving improved performance and efficiency across the entire caching stack.

In this article, we will share some context on the observability and performance testing tools that enable optimization exploration work, followed by a deep dive into practical optimizations currently running in our production environment along dimensions of hardware selection strategy, compute efficiency, and networking performance.

High-level description of the available surface area for performance optimization for memcached running on virtual machines in public cloud environments

Monitoring, observability, and evaluation

All performance optimization efforts start with precise quantitative measurement and a structured, reproducible mechanism for generating workloads for isolated evaluation.

Critical monitoring prerequisites for all the performance evaluation conducted over the years include:

  • Server-side metrics for request throughput, network throughput, resource utilization, and hardware-level parameters (NIC statistics like per-queue packet throughput and EC2 allowance exhaustion, disk response times and in-flight I/O requests, etc.)
  • Client-side metrics for cache request percentile latency, timeout and error rates, and per-server availability (SLIs), as well as top-level application performance indicators like service RPC P99 response time

Pinterest leverages both synthetic load generation and production shadow traffic to evaluate the impact of tuning and optimizations. Historically, synthetic benchmarking has been useful for detecting performance regressions or improvements under maximum load, while shadow traffic evaluation has been more reflective of server resource utilization and overall performance under a real workload at scale.

  • Synthetic load generation: memtier_benchmark is an open source tool capable of generating a synthetic load against a memcached cluster with configurable parameters for the number of concurrent clients and threads, read/write ratio, data sizes, and transport mechanism (plaintext or TLS).
  • Production shadow traffic: mcrouter is an open source memcache-protocol routing proxy deployed as a client-side sidecar in the Pinterest fleet. It provides building blocks to design transparent shadow traffic routing policies with configurable traffic percentages and source/target cluster(s), allowing for flexible dark traffic experimentation across a variety of workload classes.

Together, these tools permit high-signal performance evaluation with zero or minimal impact to critical-path production traffic.

Performance and efficiency

Cloud hardware

Distributed caching at Pinterest serves a diverse array of workloads. In general, each class of workload can be categorized along the following high-level dimensions:

  • Throughput (compute)
  • Data volume (memory and/or disk capacity)
  • Data bandwidth (network and compute)
  • Latency requirement (compute)

While memcached can be arbitrarily horizontally scaled in and out to address a particular cluster’s bottleneck, vertically scaling individual hardware dimensions allows for greater cost efficiency for specific workloads. In practice, this entails standardization on a fixed pool of EC2 instance types optimized for each workload class.

Workload profile: Moderate throughput, moderate data volume

EC2 instance family: r5

Rationale: r5 family instances offer a vCPU-DRAM ratio that works well for most vanilla cache use cases at Pinterest. This instance type is considered the “baseline” against which others are evaluated.

Workload profile: High throughput, low data volume

EC2 instance family: c5

Rationale: c5 family instances are more cost efficient for use cases that would otherwise slot into the r5 type but hold significantly less memory. Maintaining the same vCPU count as its r5 counterpart allows it to serve the same throughput volume at a lower overall cost.

Workload profile: High data volume, relaxed latency requirement

EC2 instance family: r5d

Rationale: r5d family instances are functionally equivalent to r5 family instances but with an instance-colocated NVMe SSD used by extstore for secondary storage. r5d is cost efficient for clusters with high data volume such that there are tangible improvements to hit rate as data is written to disk. Due to the slower disk (relative to i3 family instances), higher tail latency is expected.

Workload profile: Massive data volume, relaxed latency requirement

EC2 instance family: i3 and i3en

Rationale: i3 and i3en family instances ship with a fast and sizable instance-colocated disk, which tangibly increases extstore performance for workloads with a very high ratio of working data on disk relative to DRAM. Additionally, they offer comparable memory capacity to r5 series instances, which reduces extstore thrashing by maintaining a reasonable DRAM to disk usage ratio.

EC2 instance type distribution for the Pinterest memcached fleet

In particular, using extstore to expand storage capacity beyond DRAM into a local NVMe flash disk tier increases per-instance storage efficiency by up to several orders of magnitude, and it reduces the associated cluster cost footprint proportionally. EC2’s storage-optimized instance types provide locally attached solid state drives capable of high random IOPS and R/W throughput, allowing onboarding of extstore use cases with massive data volumes and high request throughput without compromising tail latency.

The introduction of different shapes of storage-optimized EC2 instance types to the fleet (in particular, the lower-tier variants of the i3en instance family containing multiple independent disks per instance) further drives down costs while offering improvements in I/O performance and cost efficiency. Pinterest configures these instances with Linux software RAID at level RAID0 to combine multiple hardware block devices into a single, logical disk for userspace consumption. By striping reads and writes fairly across two disks, RAID0 doubles maximum theoretical I/O throughput with a best-case two-fold reduction in effective disk response time at the cost of a doubled MTTF. This increased hardware performance for extstore at the expense of an increased theoretical failure rate is a highly worthwhile tradeoff. Operating workloads on a public cloud necessitates designing infrastructure to be ephemeral cattle, capable of self-healing in the event of instance failures. A topology control plane for mcrouter automatically and gracefully responds to unexpected changes in server capacity; instance loss is a non-issue.

Compute

Approximately half of all caching workloads at Pinterest are compute-bound (i.e. purely request throughput-bound). Successful optimizations in compute efficiency translate into the ability to downsize clusters without compromising serving capacity.

More precisely, compute efficiency for memcached is defined as the additional rate of requests that can be serviced by a single instance for each percentage point increase in instance CPU usage, without increasing request latency. In simpler terms, an optimization that improves compute efficiency is one that allows memcached to serve a higher request rate at lower CPU usage, without changing request latency characteristics.

At Pinterest, most workloads (including the distributed cache fleet) run on dedicated EC2 virtual machines. Many historical efficiency improvements stem from optimizations in the hardware layer itself, like migrating to different instance families or upgrading to newer generations of existing instance types. However, operating workloads on dedicated (virtualized) machines offers unique opportunities for optimizations at the hardware-software boundary.

Memcached is somewhat unique among stateful data systems at Pinterest in that it is the exclusive primary workload, with a static set of long-lived worker threads, on every EC2 instance on which it is deployed. This is in contrast to database workloads which might have, for example, multiple colocated processes for decoupled storage and serving layers. To this end, one simple but highly effective optimization is tuning process scheduling in order to request the kernel prioritize CPU time for memcached at the expense of deliberately withholding CPU time from other processes on the host, like monitoring daemons. This involves running memcached under a real-time scheduling policy, SCHED_FIFO, with a high priority — instructing the kernel to, effectively, allow memcached to monopolize the CPU by preempting (and thus deliberately starving) all non-realtime processes whenever a memcached thread becomes runnable.

$ sudo chrt — — fifo <priority> memcached …

Example invocation of memcached under a SCHED_FIFO real-time scheduling policy

This one-line change, after rollout to all compute-bound clusters, drove client-side P99 latency down by anywhere between 10% and 40%, in addition to eliminating spurious spikes in P99 and P999 latency across the board. Additionally, it afforded the ability to raise the steady-state operation CPU usage ceiling by 20% without introducing latency regressions. Ultimately, this shaved close to 10% off memcached’s total fleet-wide cost footprint.

Week-over-week comparison of client-side P99 cache latency for one service while real-time scheduling was rolled out to its corresponding dedicated memcached cluster

Ratio of time spent by memcached waiting for execution by the kernel versus wall clock time, before and after real-time scheduling was enabled (data is collected from schedstat in the /proc filesystem)

Stabilization of spurious latency spikes after real-time scheduling was enabled on a canary host (red-colored series)

Networking

There are a few key dimensions when considering networking performance:

  • Data bandwidth, packet throughput, and TCP connection limits. EC2 imposes hard limits on per-instance PPS, aggregate bandwidth, and TCP connections (though only when deployed in a security group with TCP ingress rules). Excess usage beyond these limits is reported by the Elastic Network Adapter (ENA) driver and accessible via ethtool. Confusingly, EC2 also expresses total NIC bandwidth capabilities in terms of burst loads rather than steady-state loads, thus requiring some degree of trial-and-error to determine the practical bandwidth ceiling for workloads like memcached with predictable network characteristics.
  • Connection latency and reliability. Is there a way to make initial TCP connections to memcached faster and more reliable, especially under burst scenarios where thousands of clients are simultaneously establishing connections?
  • Overhead associated with transport-layer features like TLS. Is there a way to reduce the encryption/decryption compute overhead of TLS? Additionally, is there a way to reduce the cost of the initial setup cost (i.e. TLS handshake)?

From a cloud consumer’s perspective, EC2-enforced network limits can and should effectively be considered inherent hardware limitations. Unfortunately, there is no mechanism to work around these limits other than horizontally scaling the fleet to reduce per-instance usage.

In Pinterest’s caching architecture, mcrouter is a universal routing proxy and the single application-facing entry point into the distributed caching tier. Each mcrouter instance (effectively, every individual host in a service cluster) creates a statically sized, long-lived TCP connection pool to every individual memcached server in a cluster. Connection pool sizes are deterministically derived from the number of logical cores available on the host system, typically ranging from 8 to 72 for canonical instance types. This results in upwards of tens of thousands of active established TCP connections per server host, and easily over a million total connections per server cluster — necessitating a strategy for maintaining minimal connection latency and connection reliability at scale.

TCP Fast Open (TFO) is a mechanism for reducing the latency overhead of establishing a TCP connection by optimizing away one RTT in an otherwise costly TCP 3WHS (3-way handshake), while also allowing eager transmission of early data during the handshake itself. While originally intended for end users on unreliable home and mobile networks connecting to remote edge servers, TFO has demonstrated tangible improvements in connection reliability in a closed cloud environment as well. Implementing TFO support in memcached reduced average TCP connection durations of successive sessions by ~10%, most prominently in connections established across an Availability Zone boundary.

Packets exchanged between client and server during TFO cookie setup and a subsequent TFO-initiated session with early data

Separately, raising the sysctl parameter value for net.core.somaxconn and associated listen backlog size in the userspace listen(2) callsite in memcached improved burst connection availability for high-throughput clusters. Previously, deploying a new memcached binary would cause spikes in ECONNREFUSED server errors caused by exhausted server-side TCP accept queues driven by thundering herds of simultaneous inbound connections from thousands of client mcrouter instances. A more generous listen backlog threshold reduced per-server downtime and fixed the brief but frequent SLO violations whenever a shared tenancy cluster was deployed.

Lastly, TLS plays an important role for in-transit data encryption between memcached and mcrouter, and it is enabled for 100% of cache traffic within Pinterest in order to comply with sitewide authentication, authorization, and auditing policies. Even with hardware-accelerated cryptography, TLS adds non-trivial initial and steady-state overhead, due to a post-connect TLS handshake and application-layer encryption/decryption during network I/O, respectively. TLS session resumption, after implementation in memcached, reduced fleet-wide client-side connection timeout rates by allowing reuse of previously cached TLS sessions. One avenue for tackling steady-state overhead is kernel TLS (kTLS) — a mechanism to offload the TLS record layer from userspace to the kernel, implemented either in software or offloaded to supported dedicated NIC hardware for completely transparent inline data encryption/decryption. TLS session resumption was upstreamed by Pinterest to memcached and is available in version 1.6.3 onward; kTLS is an ongoing and relatively experimental optimization area.

Future work

Infrastructure optimization is a critical objective for Pinterest that ultimately drives a more delightful experience for Pinners while reducing our own cloud cost footprint. We look forward to continuing to explore avenues for improving cache performance and efficiency at all layers of the stack, from application clients and routing proxies to the servers themselves. In the near term, we intend to continue evaluation of software kernel TLS, explore compatibility of memcached with newer generations of EC2 instance types for improved price-to-performance characteristics, and application/proxy-side software optimizations like in-flight compression for improved storage efficiency. We hope to additionally build an end-to-end automated performance regression testing framework to track the impact of these optimizations over time.

Thanks to the entire Storage and Caching team at Pinterest for supporting this work, especially Ankita Girish Wagh and Lianghong Xu.

Pinterest
Pinterest is a social bookmarking site where users collect and share photos of their favorite events, interests and hobbies. One of the fastest growing social networks online, Pinterest is the third-largest such network behind only Facebook and Twitter.
Tools mentioned in article
Open jobs at Pinterest
Full Stack Engineer - Identity
San Francisco, CA, US; , CA, US
<div class="content-intro"><p><strong>About Pinterest</strong><span style="font-weight: 400;">:&nbsp;&nbsp;</span></p> <p>Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love.&nbsp;In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping&nbsp;Pinners&nbsp;make their lives better in the positive corner of the internet.</p> <p><em>Our new progressive work model is called PinFlex, a term that’s uniquely Pinterest to describe our flexible approach to living and working. Visit our </em><a href="https://www.pinterestcareers.com/pinflex/" target="_blank"><em><u>PinFlex</u></em></a><em> landing page to learn more.&nbsp;</em></p></div><p><span style="font-weight: 400;">We are looking for a Fullstack Engineer to join the Identity &amp; Compliance team, with a focus on product privacy, authentication and identity data management, to help us build the application that allows everyone to seamlessly and securely access Pinterest. You will specialize in building elegant product features with cutting-edge technology.</span></p> <p><strong>What you’ll do:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Build out end-to-end features to protect user privacy and ensure privacy compliance&nbsp; such as minor safety features, blocking functionality</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Build out end-to-end features for Identity products </span><span style="font-weight: 400;">to make things easy, seamless, and secure for our pinners</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Knit across teams by collaborating with product managers and designers and other functions to build a holistic experience that’s right for all pinners, creators, merchants and advertisers</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Contribute to each step of the product development process, from ideation to implementation to release; from rapidly prototyping, running A/B tests, to architecting and building solutions that can scale to support millions of users</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Analyze and visualize data to drive product insights and to inform our decisions</span></li> </ul> <p><strong>What we’re looking for:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">3+ years of full stack application development experience</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Proficiency in a dynamic programming language such as Javascript or Python</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong collaboration and communication skills</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience partnering with designers to implement design</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with A/B experiments and data analysis</span></li> </ul> <p><strong>US Applicants:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">The minimum and maximum salary for this position is $155,300 to $258,700 in the US;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">This position is eligible for equity; and</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Information regarding the culture at Pinterest and benefits available for this position can be found at </span><a href="https://www.pinterestcareers.com/pinterest-life/"><span style="font-weight: 400;">https://www.pinterestcareers.com/pinterest-life/</span></a><span style="font-weight: 400;">.</span></li> </ul> <p><em><span style="font-weight: 400;">* This compensation and benefits information is based on Pinterest’s good faith estimate as of the date of publication and may be modified in the future. The level of pay within the range will depend on a variety of job-related factors that may include location, travel, shift requirements, relevant prior experience and/or education, or particular skills and expertise.</span></em></p> <p><span style="font-weight: 400;">#LI-REMOTE</span></p> <p><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;#LI-KK6&quot;}" data-sheets-userformat="{&quot;2&quot;:8705,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0,&quot;16&quot;:12}">#LI-KK6</span></p> <p>&nbsp;</p><div class="content-conclusion"><p><strong>Our Commitment to Diversity:</strong></p> <p>At Pinterest, our mission is to bring everyone the inspiration to create a life they love—and that includes our employees. We’re taking on the most exciting challenges of our working lives, and we succeed with a team that represents an inclusive and diverse set of identities and backgrounds.</p></div>
IT Enterprise Architect
San Francisco, CA, US; , CA, US
<div class="content-intro"><p><strong>About Pinterest</strong><span style="font-weight: 400;">:&nbsp;&nbsp;</span></p> <p>Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love.&nbsp;In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping&nbsp;Pinners&nbsp;make their lives better in the positive corner of the internet.</p> <p><em>Our new progressive work model is called PinFlex, a term that’s uniquely Pinterest to describe our flexible approach to living and working. Visit our </em><a href="https://www.pinterestcareers.com/pinflex/" target="_blank"><em><u>PinFlex</u></em></a><em> landing page to learn more.&nbsp;</em></p></div><p>&nbsp;</p> <p><span style="font-weight: 400;">The IT Enterprise Architect will provide digital thought leadership for Pinterest’s IT organization. This individual will be the authoritative expert for business systems supporting Finance, HR, Sales, Marketing, Learning and Employee Enablement solutions. You will have a collaborative nature that facilitates strong partnering with Pinterest business units, particularly with the Engineering org. The IT Enterprise Architect will have a vast knowledge of and experience delivering technical capabilities across multiple technologies/platforms - you’ve walked a mile in the shoes of other technologists. You will use your vast knowledge and experience to drive excellence through standards in agile delivery practices. You will indirectly lead teams of technologists to advance Pinterest’s solutions delivery roadmap, providing guidance and oversight. You will solve complex technical challenges by rolling up your sleeves and taking part in the delivery, always focusing on the end user experience.&nbsp; You will partner with the IT leadership community to communicate the solutions and roadmap to business stakeholders. In short, you are an inspiring leader with an insatiable need to learn and grow, and an engineer at heart.</span></p> <p>&nbsp;</p> <p><strong>What you’ll do:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Evaluate projects from a technical perspective, ensuring alignment with business needs, IT roadmap, architecture and security standards, internal controls, and IT methodology</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ensure solution designs and implementation meet architecture quality standards</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ensure solutions improve overall information flow, reduce cost of service whenever possible, and do not duplicate existing functionality or diverge from business and IT strategies. Your focus will be on delivering high quality solutions and high reuse of solutions</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Foster innovation and implement creative out-of-the-box ideas by conducting brainstorming sessions, hackathons and other innovation events and elevate the thinking of Pinterest’s technologists</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Be the champion for a security-first mindset and elevate the overall security competency within IT</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Assist in the development of policies, standards, and guidelines that direct the selection, development, implementation, and use of Information Technology within the enterprise</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Guide Solutions Architects / Tech Leads to ensure that application and integration architecture aligns with system design guidelines and enterprise architecture standards.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Participate in business, data, application and infrastructure planning, providing insight into the future of the technology footprint within the assigned functional area</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Evaluate new technologies and create proposals for their adoption based on business goals, investments, benefits and risks</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Coach, mentor and guide solution delivery and operations teams to effectively leverage the technology footprint</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Build and maintain effective working relationships with business stakeholders as well as technical leaders within IT and other engineers throughout the organization (product engineering, data platforms, etc.)</span></li> </ul> <p>&nbsp;</p> <p><strong>What we’re looking for:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">10+ years of experience in IT and stakeholder relationship management</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">7+ years of experience leading organizations through major change initiatives</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">10+ years product management and engineering experience utilizing agile development methodologies</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Track record of successfully establishing and implementing transformation programs in an Agile and DevOps environment</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Demonstrated knowledge of SaaS, infrastructure, and cloud computing architectures</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Demonstrated knowledge of data security and compliance controls</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience designing/engineering Identity Access Management solutions a plus</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Demonstrated knowledge of advanced automation (bots, AI, low-code platforms)</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ability to respond effective to ambiguity and change, and keeps teams and strategic initiatives moving forward even when things are not certain or the path forward is not clear</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Track record of evaluating plans to identify risk, ensure alignment of goals, plans, and priorities across teams and functions, and meet commitments; ability to create clarity about what success looks like and inspire velocity in self and others to achieve results that accelerate functional impact</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Capable of stepping up to address difficult issues and saying what needs to be said</span></li> </ul> <p>&nbsp;</p> <p><strong>Colorado based applicants only:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">The minimum and maximum salary for this position is $222,300 to $344,500 in Colorado;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">This position is eligible for equity; and</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Information regarding the culture at Pinterest and benefits available for this position can be found at </span><a href="https://www.pinterestcareers.com/pinterest-life/"><span style="font-weight: 400;">https://www.pinterestcareers.com/pinterest-life/</span></a><span style="font-weight: 400;">.</span></li> </ul> <p><em><span style="font-weight: 400;">* This compensation and benefits information is based on Pinterest’s good faith estimate as of the date of publication and may be modified in the future. The level of pay within the range will depend on a variety of job-related factors that may include location, travel, shift requirements, relevant prior experience and/or education, or particular skills and expertise.</span></em></p> <p>&nbsp;</p> <p><span style="font-weight: 400;">#LI-REMOTE</span></p> <p><span style="font-weight: 400;">#LI-AJ2</span></p><div class="content-conclusion"><p><strong>Our Commitment to Diversity:</strong></p> <p>At Pinterest, our mission is to bring everyone the inspiration to create a life they love—and that includes our employees. We’re taking on the most exciting challenges of our working lives, and we succeed with a team that represents an inclusive and diverse set of identities and backgrounds.</p></div>
Machine Learning Engineer, Core Engin...
San Francisco, CA, US; , CA, US
<div class="content-intro"><p><strong>About Pinterest</strong><span style="font-weight: 400;">:&nbsp;&nbsp;</span></p> <p>Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love.&nbsp;In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping&nbsp;Pinners&nbsp;make their lives better in the positive corner of the internet.</p> <p><em>Our new progressive work model is called PinFlex, a term that’s uniquely Pinterest to describe our flexible approach to living and working. Visit our </em><a href="https://www.pinterestcareers.com/pinflex/" target="_blank"><em><u>PinFlex</u></em></a><em> landing page to learn more.&nbsp;</em></p></div><p><span style="font-weight: 400;">With more than 400 million users around the world and 300 billion ideas saved, Pinterest Machine Learning engineers build personalized experiences to help Pinners create a life they love. With just over 3,000 global employees, our teams are small, mighty, and still growing. At Pinterest, you’ll experience hands-on access to an incredible vault of data and contribute large-scale recommendation systems in ways you won’t find anywhere else.</span></p> <p><strong>What you’ll do:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Build cutting edge technology using the latest advances in deep learning and machine learning to personalize Pinterest</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Partner closely with teams across Pinterest to experiment and improve ML models for various product surfaces (Homefeed, Ads, Growth, Shopping, and Search), while gaining knowledge of how ML works in different areas</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Use data driven methods and leverage the unique properties of our data to improve candidates retrieval</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Work in a high-impact environment with quick experimentation and product launches</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Keeping up with industry trends in recommendation systems&nbsp;</span></li> </ul> <p><strong>What we’re looking for:</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">2+ years of industry experience applying machine learning methods (e.g., user modeling, personalization, recommender systems, search, ranking, natural language processing, reinforcement learning, and graph representation learning)</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">End-to-end hands-on experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark)</span></li> <li><strong>Nice to have:</strong> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">M.S. or PhD in Machine Learning or related areas</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Publications at top ML conferences</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Expertise in scalable realtime systems that process stream data</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Passion for applied ML and the Pinterest product</span></li> </ul> </li> </ul> <p>&nbsp;</p> <p><span style="font-weight: 400;">#LI-HYBRID<br>#LI-LA1</span></p><div class="content-conclusion"><p><strong>Our Commitment to Diversity:</strong></p> <p>At Pinterest, our mission is to bring everyone the inspiration to create a life they love—and that includes our employees. We’re taking on the most exciting challenges of our working lives, and we succeed with a team that represents an inclusive and diverse set of identities and backgrounds.</p></div>
Salesforce BSA, IT Sales Systems (Con...
San Francisco, CA, US; , CA, US
<div class="content-intro"><p><strong>About Pinterest</strong><span style="font-weight: 400;">:&nbsp;&nbsp;</span></p> <p>Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love.&nbsp;In your role, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping&nbsp;Pinners&nbsp;make their lives better in the positive corner of the internet.</p> <p><em>Our new progressive work model is called PinFlex, a term that’s uniquely Pinterest to describe our flexible approach to living and working. Visit our </em><a href="https://www.pinterestcareers.com/pinflex/" target="_blank"><em><u>PinFlex</u></em></a><em> landing page to learn more.&nbsp;</em></p></div><p>Do you like using your skills to improve internal business tools? Pinterest is looking for a Business Systems Analyst (BSA) to help accelerate the use of technology on our Salesforce implementation. You will work with a fast-paced team of engineers, portfolio leads, and business partners where you will continue to develop your skills while helping ensure that Pinterest has an effective and scalable platform for our business-critical departments. This role will have a direct influence on the company by helping drive productivity, efficiency and insights on behalf of our Sales organization.&nbsp; You will provide valuable contribution as a bridge between Sales teams and IT delivery.&nbsp; You should be able to utilize your time by testing key features &amp; functionality before they are production ready. This is an exciting opportunity to join a collaborative, inclusive team that manages a growing Salesforce implementation.</p> <p><strong>What you'll do:</strong></p> <ul> <li>Provide strong analytical, problem solving, and technical skills to effectively collect requirements, identify alternatives, and design and deploy quality solutions.</li> <li>Own functional tasks and deliverables, including: process definition and reengineering, high and detailed-level requirements gathering, functional design, testing, and continuous improvement of business applications.</li> <li>Collaborate with business stakeholders to gather requirements, deliver solutions with engineers, conduct Quality Assurance Testing (QA), facilitate User Acceptance Testing (UAT), and provide post-production support.</li> <li>Use and recommend best practices while delivering scalable high quality solutions.</li> <li>Perform Salesforce configurations and guide developers on advanced customization.</li> </ul> <section class="skills-section"> <p><strong>What we're looking for:</strong></p> <ul> <li>3+ years of hands-on work as a SFDC Admin and have Business Systems Analyst experience.</li> <li>5+ years of professional work experience.</li> <li>You consider yourself detail-oriented, organized, self-motivated, hard working, and a great team player with excellent interpersonal skills.</li> <li>You have strong written/verbal communication skills and can translate business requirements and engage directly with business stakeholders.</li> <li>You have extensive experience analyzing, designing, configuring and supporting Salesforce implementations.</li> <li>You are a certified SalesforceAdministrator and you have experience with Sales Cloud, Service Cloud, and SFDC Lightning framework.</li> <li>You can manage and implement advanced configurations such as custom objects, fields, formulas, validation rules, and workflow rules.</li> <li>You have experience with SFDC data management tools, SOQL queries, Microsoft Excel, JIRA, Visual Diagrams, and Google Suite.</li> <li>Preferred, but not required: You have experience with sales processes and tools ideally in the advertising/media/e-commerce industries.</li> </ul> <p class="p2"><span class="s1"><strong>More about contract roles at Pinterest:</strong></span></p> <ul class="ul1"> <li class="li3"><span class="s1">This is a temporary contract position at Pinterest. As such, the contractor who fills this role will be employed either by our staffing partner (PRO Unlimited) or by an agency supplier partner, and not an employee of Pinterest.</span></li> <li class="li3"><span class="s1">All interviews will be scheduled and/or conducted by the Pinterest assignment manager. When a finalist has been selected, PRO Unlimited or the agency supplier partner will extend the offer and provide assignment details including duration, benefits options and onboarding details.</span></li> <li class="li1"><span class="s1">The estimated salary range for this contract position is $88 - $108/hour based on experience and location. To learn more about the benefits offered to contractors through Pro Unlimited please visit: <a href="http://www.hrconnection.com/"><span class="s3">www.hrconnection.com</span></a>&nbsp;(username: generalinformation)</span></li> </ul> <p><span class="s1">#LI-REMOTE</span></p> </section><div class="content-conclusion"><p><strong>Our Commitment to Diversity:</strong></p> <p>At Pinterest, our mission is to bring everyone the inspiration to create a life they love—and that includes our employees. We’re taking on the most exciting challenges of our working lives, and we succeed with a team that represents an inclusive and diverse set of identities and backgrounds.</p></div>
Verified by
Software Engineer
Sourcer
Software Engineer
Talent Brand Manager
Tech Lead, Big Data Platform
Security Software Engineer
You may also like