Principal DevOps Engineer - Remote OK

Engineering San Francisco, California Austin, Texas Atlanta, Georgia Seattle, Washington Charlotte, NC, North Carolina Phoenix, United States Houston, Texas Portland, Oregon New York, New York Boston, Massachusetts Chicago, Illinois Los Angeles, CA, California Raleigh, North Carolina San Diego, California Newark, New Jersey Orlando, Florida Miami, Florida San Jose, California Morrisville, North Carolina

Join us as we pursue our disruptive new vision to make machine data accessible, usable and valuable to everyone. We are a company filled with people who are passionate about our product and seek to deliver the best experience for our customers. At Splunk, we’re committed to our work, customers, having fun and most importantly to each other’s success. Learn more about Splunk careers and how you can become a part of our journey!


As a Principal DevOps Engineer, you will be responsible for supporting internal Splunk solution delivery related to IT operations and security use cases. The innovations built by our team will be widely shared and you should be eager to drive these efforts and share your knowledge with the Splunk community.

The Splunk at Splunk team is renowned for excellence in engineering, both inside and outside of Splunk. Last year, we presented five highly-rated sessions at .conf20, published multiple blog articles, and open-sourced several of our innovations via Splunk’s GitHub project, including but not limited to: ansible-role-for-splunk, docker-swarm-splunk-hf, and TA-VaultSync.

Splunk at Splunk is also a strong partner with the Products and Engineering teams, where we’ve contributed to everything from Splunk-built addons (e.g. Zoom, Gmail, Falcon Data Replicator), to the health checks and alerts that ship with Splunk’s Distributed Management Console (DMC), to the feature roadmap for Data Stream Processor (DSP). If you’re looking for a team where you can make a big impact, this is it!


  • Collaborate with the team to develop cutting-edge solutions for corporate data collection to Splunk Cloud. You will have the opportunity to work on projects that involve the latest technologies, such as serverless data collection using AWS Lambda and the development of microservices running on Kubernetes.
  • Seize opportunities to automate Splunk administration and deployment workflows using your knowledge of DevOps tools (Ansible, Terraform, GitLab CI/CD pipelines), Python, REST APIs, and Splunk's product portfolio.
  • Develop, test, and deploy a variety of Splunk configurations, Splunk technology add-ons, modular inputs, external lookups, and custom search commands using Python and Splunk SDKs.
  • Implement automated testing, continuous integration, and continuous deployment to streamline how we operate Splunk and data collection services internally.
  • Instrument infrastructure monitoring, application performance monitoring (APM), and tracing using Splunk Infrastructure Monitoring (SignalFx) combined with Ansible roles and Terraform providers for Splunk’s corporate IT services.
  • Implement security best practices related to our team’s Linux-based infrastructure, Python code, and Docker services.


  • 12+ years working with distributed systems at scale in IT
  • 8+ years of software development experience in Python (expert level)
  • 5+ years DevOps experience in a technical lead role
  • 5+ years experience with configuration management (Ansible, Puppet) and infrastructure provisioning tools (Terraform, CloudFormation)
  • 5+ years experience with AWS, Docker containers, and container orchestration (Kubernetes, EKS, etc.)
  • 5+ years experience in cloud platform administration (preferably AWS)
  • Demonstrated ability to develop advanced CI/CD pipelines in GitLab (e.g. using dynamic child pipelines, automated unit testing, performance testing, etc.)
  • Strong knowledge of IT Security (encryption, firewalls, secure systems design, vulnerability management, secure software design, credential management, proxies, WAFs, etc.)
  • Deep understanding of IT fundamentals, including Linux and Microsoft operating systems, networking, security, cloud (AWS), and database technologies (MySQL, Postgres)
  • Excellent communication skills, both verbal and written; able to explain complex technical topics to both internal and external customers with ease
  • Fully proficient in git and version control systems (GitLab, BitBucket, GitHub)
  • Passionate about IT, technology, and following the latest industry trends

Preferred knowledge and experience: These are a huge plus.

  • Experience developing in Golang or writing custom Terraform providers
  • Experience developing custom Ansible modules
  • Certifications in AWS, Docker, Ansible, and/or Security (e.g. CISSP, CEH, Security+, etc.)
  • Published open-source projects on GitHub
  • Splunk experience a plus, but not required


  • Bachelor’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field
  • Master’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field preferred but not required

What We Offer You: 

  • An opportunity to innovate cutting-edge solutions using Splunk software for Splunk itself. Plus, the ability to share those innovations with our customers! How cool is that?
  • A rare opportunity to join one of the best Splunk engineering teams in the world with a stable, collaborative, and supportive work environment that encourages work-life balance.
  • A constant stream of new things for you to learn. We're always expanding into new areas, bringing in open source projects and contributing back, and exploring new technologies.
  • A set of extraordinarily hardworking, innovative, open, fun and dedicated peers, all the way from engineering and QA to product management and customer support.
  • Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation.

We don't expect people to work 12-hour days. We want you to have a successful time outside of work too. Want to work from home sometimes? No problem. We trust our colleagues to be responsible with their time and dedication and believe that balance helps cultivate an extraordinary environment

This isn’t a job – it’s a life changer – are you ready?

Splunk has been named one of San Francisco Bay Area’s “Best Places to Work” by the San Francisco Business Times, ten years in a row. We offer a highly competitive compensation package and a plethora of benefits.

We value diversity at our company. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or any other applicable legally protected characteristics in the location in which the candidate is applying.

For job positions in San Francisco, CA, and other locations where required, we will consider for employment qualified applicants with arrest and conviction records.

Thank you for your interest in Splunk!