Big Data Engineer
We are seeking a talented Big Data Engineers to work on Qualys’ next-generation big data challenge. Working with a team of engineers and architects, you will be responsible for the creation of data lakes by extracting and transforming data from various data sources. As a Big Data Engineering team member, you would be responsible for the development, implementation, and support of critical data intelligence ETL solutions. This is a great opportunity to be an integral part of Qualys’ next-generation data technology platform by leveraging open source technologies and work on challenging and business-impacting projects.
- Develop, deploy and maintain Big Data stack applications and data services, work on collecting, storing, processing, validating and analyzing huge sets of data.
- Develop and support data flows, ETLs, UDFs and machine learning processes using Java, Scala, Python, R within Hadoop/Spark/Flink/Hive/Presto/Drill programming API.
- Work with application developers and analytic teams to build data marts and snapshot tables for Data Warehousing and Data Lake management.
- Monitor data transformations performance and define data retention policies.
- Develop scalable and high-performance data services for high-speed querying.
- Maintain security and data privacy in the distributed environment.
- Test data processing prototypes and oversee handover to operational teams.
- Must have experience with Big Data querying tools, such as Hive, Presto, Drill, Impala, BigQuery, Athena and strong knowledge of SQL (including DDL) and relational data structures.
- Must have experience with developing of Spark, Flink applications and/or Hadoop MapReduce batch processing jobs.
- Experience with NoSQL databases, such as Cassandra and NoSQL data structures.
- Knowledge of various ETL techniques and frameworks.
- Experience with various messaging systems, such as Kafka or RabbitMQ.
- Ability to develop code in Java (Scala, Python, and R are all beneficial including experiences with Numpy/Pandas/Sci-kit).
- Ability to leverage and partner business intelligence tools such as Power BI, Tableau, MicroStrategy.