Principal Engineer Monitoring

Engineering San Francisco, California San Jose, California

Join us as we pursue our disruptive new vision to make machine data accessible, usable and valuable to everyone. We are a company filled with people who are passionate about our product and seek to deliver the best experience for our customers. At Splunk, we’re committed to our work, customers, having fun and most importantly to each other’s success. Learn more about Splunk careers and how you can become a part of our journey!


Splunk's Cloud Monitoring team is looking for an expert Principal Engineer in Monitoring to help craft and deliver the next evolution of the Splunk solution that we use to operate Splunk Cloud. You will be working with partners across the company and across the globe to find solutions to business needs at scale.


You will:

  • Work across the organization to define and deliver quality solutions that delight Splunk's passionate internal users.
  • Guide a team of tight-knit, super-smart engineers who are building the largest Splunk Cloud solution in the world.
  • Mentor and help new engineers to achieve more than they thought possible.
  • Establish and drive key metrics to measure our success and guide our focus.
  • Work closely with our partners in compliance and legal to maintain and evolve our policies around data stewardship.


  • You are passionate about building and running a very large scale data infrastructure. You understand the challenges and trade-offs to be made when building and deploying systems to production.
  • Expertise in working with Splunk at scale along with expertise in optimizing data models and summarization.
  • Experience working with internal customers and crystallizing their needs. Working with colleagues to agree prioritization in an open and collaborative manner.
  • Knowledge of standard methodologies related to security, performance, and disaster recovery.
  • Experience in responding collaboratively to operational issues and applying findings to prevent them happening again.
  • Highly skilled in identifying performance bottlenecks, identifying anomalous system behavior, and determining the root cause of incidents.
  • You are enthusiastic about making the many users of your product happier every day.

Preferred skills:

  • Experience with running a large Splunk instance, either on-prem or in the cloud.
  • Experience with development and deployment in a hosted cloud environment, preferably AWS and GCP.


We value diversity at our company. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or any other applicable legally protected characteristics in the location in which the candidate is applying.

For job positions in San Francisco, CA, and other locations where required, we will consider for employment qualified applicants with arrest and conviction records.


Thank you for your interest in Splunk!