Ness Digital Engineering Careers

Description

Senior Data Engineer; Must Have: Strong hands-on AWS, Snowflake, SQL and dbt experience. Nice to have – Python, PySpark, Airflow, Informatica

Requirements and Responsibilities

What You’ll Do

• Design, develop, and maintain scalable data pipelines and data transformation frameworks using AWS, Snowflake, and dbt.
• Develop robust ETL/ELT pipelines to ingest, transform, and integrate data from multiple source systems into Snowflake.
• Build and maintain data models in Snowflake supporting analytics, reporting, operational applications, and downstream data consumers.
• Develop reusable dbt models, macros, tests, snapshots, and incremental pipelines following established data engineering standards.
• Write complex and optimized SQL for data transformation, aggregation, validation, and analytical workloads.
• Design and implement scalable data ingestion solutions using AWS services such as S3, Glue, Lambda, and related data services.
• Develop data pipelines for both structured and semi-structured data and work with formats such as JSON, CSV, Parquet, and Avro.
• Implement incremental loading, CDC, partitioning, clustering, and other optimization techniques to efficiently process large datasets.
• Design and maintain appropriate data warehouse schemas and dimensional models, including fact tables, dimension tables, star schemas, and data marts.
• Implement data quality and validation frameworks using dbt tests, SQL-based validation, reconciliation, and data profiling.
• Monitor data pipelines and proactively identify failures, performance issues, data quality problems, and processing bottlenecks.
• Troubleshoot pipeline failures and data discrepancies and perform detailed root-cause analysis.
• Optimize Snowflake workloads through appropriate warehouse sizing, query optimization, clustering, partition pruning, and workload management.
• Implement appropriate logging, monitoring, error handling, alerting, and recovery mechanisms for production data pipelines.
• Develop automated data pipelines and workflows using orchestration tools such as AWS Glue, Airflow, or similar technologies.
• Implement version control, CI/CD, and deployment processes for dbt models, SQL code, and data pipelines.
• Work closely with Data Architects, Business Analysts, ETL Developers, SQL Developers, QA Engineers, and application teams to understand requirements and deliver scalable data solutions.
• Participate in data architecture and technical design discussions and contribute to data platform modernization initiatives.
• Support migration of legacy ETL processes to cloud-based AWS/Snowflake ELT architectures.
• Develop and maintain technical documentation covering data pipelines, data models, source-to-target mappings, dependencies, and operational procedures.
• Conduct code reviews and ensure adherence to development, security, performance, and data engineering standards.
• Support production releases, incident management, troubleshooting, and post-production validation.
• Work in an Agile/Scrum environment and actively participate in sprint planning, daily stand-ups, reviews, retrospectives, and technical discussions.
• Mentor junior data engineers and contribute to knowledge sharing and continuous improvement.

What You’ll Bring

• Bachelor’s degree or higher in Computer Science, Information Technology, Engineering, or a related field.
• 5+ years of relevant experience in data engineering, data integration, data warehousing, or cloud data platforms.
• Strong hands-on experience with Snowflake as an enterprise cloud data warehouse.
• Strong hands-on experience with AWS data services, particularly:
  • Amazon S3
  • AWS Glue
  • AWS Lambda
  • IAM and security concepts
  • Other AWS data/analytics services as applicable
• Strong hands-on experience with dbt (data build tool), including:
  • dbt models
  • Incremental models
  • Macros
  • Sources
  • Seeds
  • Snapshots
  • Tests
  • Documentation
  • Model dependencies
• Advanced proficiency in SQL, including complex joins, CTEs, subqueries, window functions, aggregations, and performance optimization.
• Strong understanding of ETL/ELT architecture and data pipeline development.
• Strong understanding of data warehousing and dimensional modeling concepts.
• Experience designing and implementing fact and dimension models, star schemas, snowflake schemas, and data marts.
• Experience working with large-volume datasets and optimizing data processing workloads.
• Experience with structured and semi-structured data such as JSON, CSV, Parquet, and Avro.
• Strong understanding of data quality, validation, reconciliation, and data governance practices.
• Experience implementing incremental loads and CDC processes.
• Experience with Git and CI/CD practices for data engineering projects.
• Experience with pipeline orchestration and scheduling tools such as Airflow, AWS Glue Workflows, or similar technologies.
• Strong troubleshooting and root-cause-analysis skills.
• Experience working in Agile/Scrum environments.
• Banking or financial-services domain experience is preferred.
• Experience with Python for data engineering, automation, or scripting is a plus.
• Experience with PySpark / Apache Spark is a plus.
• Experience with Informatica or other enterprise ETL tools is a plus, particularly for legacy ETL modernization.
• Experience with data migration from traditional data warehouses to AWS/Snowflake is a plus.
• Knowledge of Snowflake security, RBAC, data sharing, streams/tasks, and stored procedures is a plus.
• Strong analytical and problem-solving skills.
• Ability to work independently and collaborate effectively with cross-functional teams.
• Strong communication skills in English, both written and verbal.

Recommended External Screening Criteria

For Senior Data Engineer – AWS/Snowflake/dbt, I would keep the following as the core mandatory screening criteria:

Must Have

1. 5+ years of Data Engineering / Data Warehousing experience
2. Strong hands-on Snowflake
3. Strong hands-on SQL
4. Hands-on dbt development
5. AWS data services experience, particularly S3 and Glue
6. ETL/ELT pipeline development
7. Data Warehousing & Dimensional Modeling
8. Experience with large-volume data
9. Data quality and validation
10. Git / CI/CD experience

Preferred

• Python
• PySpark / Spark
• Airflow
• Snowflake Streams & Tasks
• Snowflake performance tuning
• AWS Lambda
• Informatica
• CDC
• Legacy ETL modernization
• Banking / Financial Services experience