Member of Technical Staff (Production Services)

Category-defining tech. Career-defining work.

Lots of tech companies disrupt. But, many fail when they try to scale. We're different. CockroachDB makes it easier for companies to build and scale apps. This is how and why we're helping some of the most innovative companies on the planet. We tackle problems head-on and focus on solutions that create lasting impact.

Because when our customers win, we all win.

The Role

Production Services owns the customer-facing capabilities of CockroachDB Cloud. The team manages and builds the platform features customers use directly such as implementing customer facing AI features, backup and restore, disaster recovery, and private connectivity.

We don’t build tooling for other engineering teams to consume. The features we ship go straight to customers. That shortens the feedback loop between what ships and how it performs with customers, and there's real distributed systems depth in making those features work reliably across clouds and regions at scale.

We're looking for a Software Engineer based out of Toronto, to join the Production Services team. The work is hands-on and production-facing. You'll work directly on the systems that protect customer data, handle cluster connectivity and disaster recovery, and power the operational platform CockroachDB Cloud is built on.

You'll report to Steve Tidwell, Senior Manager Engineering, as a part of the Production Engineering organization.

You Will

• Build and maintain the managed backup and restore systems that protect customer data across CockroachDB Cloud, with a focus on pipeline reliability, observability, and SLA compliance.
• Build and improve the customer-facing platform capabilities that shape the CockroachDB Cloud experience.
• Instrument systems with metrics, alerts, and tooling that surface issues early and reduce mean time to resolution.
• Reduce operational toil by building automation and tooling that improve reliability without scaling headcount.
• Help identify ways to apply AI to our operational workflows, from alert triage and automated diagnostics, to coding new features.
• Participate in on-call for our production systems.
• Partner with Production Infrastructure, Production Orchestration, and other teams on cross-cutting reliability and architectural work.
• Learn a lot about distributed systems, cloud infrastructure, and database systems.

The Expectations

In your first 30 days, you'll onboard and be oriented on the Production Services systems: AI based migration tools, backup and restore pipelines, private connectivity, and the job infrastructure that runs them. You’ll build context on CockroachDB and CockroachDB Cloud. We want you to take the time to learn before you start building.

After 3 months, you'll shadow on-call, review recent incidents and help reduce toil on the team. You'll understand our infrastructure, architecture, observability stack, and operational tooling. You will have contributed to multiple reliability efforts, and at least one major project.

After 6 months, you'll own meaningful parts of the platform. That might be backup reliability, scaling automation, or a new customer-facing capability. You'll be identifying opportunities to improve the system and driving that work forward.

You Have

• A solid understanding of software engineering. You understand reliability principles, SLOs and SLAs, error budgets, and how to engineer for operational outcomes.
• Experience building and running production-grade systems. Bonus if that includes data pipelines, backup and restore, or other customer-facing infrastructure.
• Comfort with Go or Python. We use Go, and if you don't know it, you'll pick it up here.
• Good observability instincts. You know how to instrument systems so that problems are visible before they become incidents.
• Experience running Kubernetes in production.
• Familiarity with at least one major cloud provider (AWS, Azure, or GCP) and its APIs.
• A working knowledge of networking fundamentals: HTTP, TLS, DNS.
• Previous on-call experience. You can stay calm under pressure and make good decisions quickly.
• Ideally 5+ years of experience and a CS degree or equivalent background.
• A collaborative approach to engineering. You do code reviews, you pair on hard problems, and you help the people around you get better.
• Comfortable working in a geographically distributed team. Most of your teammates will be in different time zones.

Bonus (You Have)

• Experience across multiple cloud providers.
• Built or contributed to observability tooling for your team or org.
• Applied AI or ML to operational workflows, such as intelligent alerting or automated remediation.
• Familiar with CockroachDB or distributed SQL.
• Experience on globally distributed teams.

The Team

Steve Tidwell - Senior Manager, Engineering

Steve has been in the tech industry for over two decades, working in global IT operations and management, corporate IT, networking, data center, and cloud-based platforms. Prior to joining Cockroach Labs a little over a year ago, he worked at Crunchyroll, Venturebeat, and Conviva, among others. His experience runs the gamut from building on-prem installations, to migrating those to the cloud, to his current primary focus on large-scale cloud-based distributed systems. During his free time he enjoys writing technical blog posts, reading science fiction, cooking, and gardening.

Isaac Wong - EVP, Research & Development

Isaac is responsible for the health of the engineering organization at Cockroach Labs. He partners closely with teams to ensure we have a balanced culture that promotes quality and innovation in pursuit of our goals. Before joining Cockroach Labs Isaac was in life sciences for 16 years with Medidata Solutions where he had a front row seat on the exciting ride from a 30 person startup to more than 2000 people worldwide. But the lure of distributed, resilient, and consistent SQL databases, along with the amazing technology and culture at Cockroach Labs proved too much. When not working he likes to draw, play the piano, and search NYC for cannolis with his wife and kids.