Big Data Engineer

Engineering Pune, India

Description

Description:

We are seeking a talented Big Data Engineers to work on Qualys’ next-generation big data challenge. Working with a team of engineers and architects, you will be responsible for the creation of data lakes by extracting and transforming data from various data sources. As a Big Data Engineering team member, you would be responsible for the development, implementation, and support of critical data intelligence ETL solutions. This is a great opportunity to be an integral part of Qualys’ next-generation data technology platform by leveraging open source technologies and work on challenging and business-impacting projects.

Responsibilities:

  • Develop, deploy and maintain Big Data stack applications and data services, work on collecting, storing, processing, validating and analyzing huge sets of data.
  • Develop and support data flows, ETLs, UDFs and machine learning processes using Java, Scala, Python, R within Hadoop/Spark/Flink/Hive/Presto/Drill programming API.
  • Work with application developers and analytic teams to build data marts and snapshot tables for Data Warehousing and Data Lake management.
  • Monitor data transformations performance and define data retention policies.
  • Develop scalable and high-performance data services for high-speed querying.
  • Maintain security and data privacy in the distributed environment.
  • Test data processing prototypes and oversee handover to operational teams.

Requirements:

  • Must have experience with Big Data querying tools, such as Hive, Presto, Drill, Impala, BigQuery, Athena and strong knowledge of SQL (including DDL) and relational data structures.
  • Must have experience with developing of Spark, Flink applications and/or Hadoop MapReduce batch processing jobs.
  • Experience with NoSQL databases, such as Cassandra and NoSQL data structures.
  • Knowledge of various ETL techniques and frameworks.
  • Experience with various messaging systems, such as Kafka or RabbitMQ.
  • Ability to develop code in Java (Scala, Python, and R are all beneficial including experiences with Numpy/Pandas/Sci-kit).
  • Ability to leverage and partner business intelligence tools such as Power BI, Tableau, MicroStrategy.

EEO Employer/Vet/Disabled