ETL QA Lead
Description
We are seeking a highly skilled QA Lead with a strong background in data engineering and ETL testing to define and execute comprehensive test strategies. This role requires deep expertise in validating data ingestion pipelines, ensuring data integrity, transformation accuracy, and completeness at every stage of the pipeline.
The ideal candidate will design and implement robust testing methodologies that extend beyond basic source-data validation. They should incorporate advanced techniques such as schema validation, data profiling, reconciliation strategies, and automation frameworks to enhance test coverage, reliability, and efficiency.
Key Responsibilities:
- Define and execute test strategies for validating data pipelines, transformations, and data quality.
- Develop and implement automation frameworks to improve test efficiency and reliability.
- Perform data validation and reconciliation between cloud storage and databases.
- Validate structured and semi-structured data formats, including JSON, AVRO, and Parquet.
- Design and implement schema validation, data profiling, and integrity checks across multiple stages of data processing.
- Collaborate with data engineers, analysts, and stakeholders to understand data flow and ensure high-quality data delivery.
- Troubleshoot and resolve data discrepancies and quality issues using a systematic approach.
- Optimize and automate test processes to support continuous integration and deployment (CI/CD) in data environments.
Required Skills & Qualifications:
- Strong experience in ETL Testing and Data Quality Assurance.
- Proficiency in Python for automation and data validation.
- Hands-on experience with various data formats (JSON, AVRO, Parquet) and cloud storage solutions.
- Expertise in data validation techniques, including schema checks, data profiling, and reconciliation methods.
- Familiarity with cloud databases, data lakes, and big data ecosystems.
- Experience in designing and implementing test automation frameworks for data pipelines.
- Strong analytical and problem-solving skills to identify and resolve data issues.
- Knowledge of CI/CD practices for data testing and automation.