Senior Data Engineering Lead – Regulatory Reporting
Description
Why Ness
We know that people are our greatest asset. Our staff’s professionalism, innovation, teamwork, and dedication to excellence have helped us become one of the world’s leading technology companies. It is these qualities that are vital to our continued success. As a Ness employee, you will be working on products and platforms for some of the most innovative software companies in the world.
You’ll gain knowledge working alongside other highly skilled professionals that will help accelerate your career progression.
You’ll also benefit from an array of advantages like access to trainings and certifications, bonuses, and aids, socializing activities and attractive compensation.
Requirements and responsibilities
We are looking for a Senior Data Engineering Lead to join our team and take ownership of critical regulatory reporting flows and data pipelines. This role involves working with large-scale financial data, optimizing AWS-based infrastructure, and ensuring compliance with FINRA and other regulatory bodies. You will collaborate with cross-functional teams to design, implement, and monitor robust data solutions that drive business-critical reporting.
What You’ll Do
- Lead the design, development, and deployment of regulatory and financial data solutions on Big Data platforms.
- Lead the development and optimization of ETL processes for Flow Enablement and Trade Report Implementation.
- Design schemas and implement data ingestion pipelines using Spark, S3, and Redshift.
- Ensure high-volume data processing efficiency through EC2 configuration tuning and performance testing.
- Architect and implement EMR Spark jobs, ensuring flexibility and scalability.
- Troubleshoot and resolve EMR job failures in collaboration with AWS Support.
- Ensure timely and accurate submission of regulatory reports to meet FINRA SLAs.
- Develop reconciliation jobs using Spark to validate data across multiple sources (ETL).
- Develop and maintain microservices using Spring Boot, gRPC, and REST APIs.
- Collaborate with cross-functional teams to gather requirements, define architecture, and ensure timely delivery.
- Mentor and guide junior developers, fostering a culture of technical excellence and continuous learning.
- Participate in Agile/SCRUM ceremonies and contribute to project planning and estimation.
What You’ll Bring
- 10+ years of experience in software development, with at least 3 years in a technical leadership role.
- Strong hands-on experience with Apache Spark, Hadoop, and Kafka.
- Proficiency in Java.
- Experience with Spring Boot, gRPC, RESTful APIs, and Microservices architecture.
- Familiarity with Oracle, Redshift and S3.
- Solid understanding of DevOps tools and CI/CD pipelines.
- Experience working in financial services or regulatory reporting is a strong plus.
- Excellent communication, leadership, and mentoring skills.
Not checking every single requirement?
If this role sounds good to you, even if you don’t meet every single bullet point in the job description, we encourage you to apply anyway. For most of the candidates that applied, we found a role that was a very good fit with their skills.
Let’s meet and you may just be the right candidate for one of our roles.
At Ness Digital Engineering we are willing to build a work culture that is based on diversification, inclusion, and authenticity.
Let’s meet and you may just be the right candidate for one of our roles.
At Ness Digital Engineering we are willing to build a work culture that is based on diversification, inclusion, and authenticity.