Principal Data Engineer
Description
The Principal Data Engineer will spearhead rearchitecting and modernizing SnapOne Enterprise Data Warehouse (EDW) ecosystem using the Azure Databricks Platform. This entails leveraging services such as Azure Data Factory, Azure Fabric, and Data Lakes to ensure a seamless transition. This challenging role requires senior knowledge and experience across many facets of cloud technology: application development, hybrid environments / connectivity, and modern data services, while understanding the governance, security, monitoring and cost management aspects of the entire ecosystem.
The successful candidate will be responsible for the owning the modernization of the EDW platform engineering and sustainment of the SQL Server based solutions, ensuring its operational readiness (security, health and performance), implementing and owning master data processes, and performing data modeling in support of Snap One’s various application development and data management teams.
Specific Responsibilities
- Spearhead the migration from SQL Server to Azure Databricks, with a focus on driving business outcomes through advanced analytics, AI/ML, and modern cloud-based data engineering practices.
- Ensure the health and reliability of data pipelines by monitoring key metrics, logging, and automating alerting processes to detect anomalies and maintain data quality.
- Establish and maintain governance frameworks, leveraging tools like Unity Catalog for data lineage, access control, and regulatory compliance, ensuring proper data handling and security.
- Experienced in crafting and refining comprehensive architecture diagrams, data flows, technical designs, and technical requirements, ensuring clarity and effectiveness in communicating complex concepts to stakeholders and facilitating seamless implementation of data solutions.
- Develop and implement data integration strategies and reusable patterns to facilitate efficient sharing of data assets across the organization, while both documenting and adhering to best practices, guidelines, and industry standards, ensuring optimal utilization and consistency in data management practices.
- Act as a mentor, committed to training and developing less experienced data professionals to enhance their technical skills, foster collaboration with the business, and cultivate a deeper understanding of how to interact effectively with stakeholders to develop robust data solutions aligned with business objectives.
- Collaborate closely with product owners, business stakeholders, and leadership to gather and understand information, analytics, and business intelligence requirements, ensuring alignment with organizational goals and fostering effective communication channels to deliver actionable insights and solutions.
Required Qualifications:
- Bachelor’s degree and 8+ years of proven experience in designing and optimizing modern data architecture solutions with a focus on scalability, performance, and availability; OR 12+ years of expertise in architecting and managing complex, large-scale data infrastructure with an emphasis on delivering high-performance, reliable data systems.
- Extensive proficiency in leveraging Databricks platform, demonstrating adeptness in core functionalities including Delta Live, Unity Catalog, delta sharing, query federation and data integration to streamline data management processes.
- Proven track record in implementing AI models and orchestrating efficient data ingestion pipelines within the Databricks environment, showcasing a deep understanding of its advanced features for enhanced data processing and analytics.
- Comprehensive grasp of contemporary data architecture principles and utilization of cutting-edge toolsets including Databricks, Power BI Fabric, Data Lakes, and Azure Data Factory, ensuring efficient data management and analytics workflows.
- Firsthand experience rearchitecting and rebuilding an existing Data Warehouse, combined with a thorough understanding of EDW methodologies such as Kimball, OWT, and Data Mesh.
- Experience in defining system architectures, evaluating technical feasibility, and prototyping applications and interaction methodologies to ensure effective and scalable solutions
- Written and verbal technical communication skills with an ability to present sophisticated technical information in a clear and concise manner to a variety of audiences.
- Strong understanding of Data Warehouse and third-party application integration, including token management, service principals, OAuth, and Databricks-specific security features. This includes implementing Databricks VPC peering, Private Link, and managed identity integration to ensure seamless, secure service-to-service communication across platforms.
- Proficiency with SQL Server “Stack,” with a focus on understanding the limitations of traditional SQL Server environments and the benefits of transitioning to modern cloud-based solutions.
Preferred Qualifications:
- Experience with in-memory solutions like Power BI or Azure Analysis Services.
- Hands-on experience with core Azure technologies, including Databricks, ADF, Azure Analysis Services, SQL, Logic Apps, Data Lakes, Storage, and Log Analytics.
- Proficient in working with APIs and other data ingestion tools like Fivetran or HVR.
- Experience with cloud platforms such as Google Big Query or Snowflake is a plus.