Data Engineer (SO2)
Description
Knowledge, Skills & Experience Industry Experience
Pathways
- DBMS administration and SQL development -> ETL specialist -> Cloud and Data Engineering.
- Software Engineer specialist in back-office systems and data sets -> Data Engineering. Data Engineering Team lead experience is an advantage.
Software Engineering
- A solid understanding of fundamental programming techniques and concepts such as:
- object oriented programming patterns.
- functional programming concepts.
- distributed computing problems and solutions.
- algorithm and data structure selection and optimization.
- test-driven development.
- A polyglot developer - experienced to expert, in at least two of the following languages:
Ideal
→ Go
→ Scala
→ Python
Advantageous
→ Java
→ C / C++
→ PHP
- Comfortable working with Linux operating systems, scripts and packages.
Data Engineering
- General Data Architecture knowledge such as Data Lakes, Data Warehouses and related technologies and concepts.
- Experience in, and an ability to express the comparative merits of SQL, NoSQL and Graph database systems.
- Solid experience in at least two RDBMS’s with MySQL and PostgreSQL being highly preferred.
- A high level of demonstrable SQL capability including query optimisation and tuning.
- An understanding of the technology, use cases, and choices available, in the following technological groups:
- Data Processing (e.g. Spark, Beam
- Data Workflows (e.g. Airflow, Luigi)
- Data Formats (e.g. Avro, Parquet, protobuf)
- Messaging/Streaming Systems (e.g. Kafka, RabbitMQ, NATS)
DevOps and Cloud Engineering
- Experience with infrastructure-as-code tooling: Ansible and Terraform.
- Provisioning AWS services to support product deployment.
- Metrics, monitoring, and alarms. Especially useful if you know Sumo Logic and PagerDuty.
- AWS services related to data engineering
Advantageous Skills and Experience
- Docker and related container-orchestration technologies (e.g. Kubernetes/AWS EKS).
- Microservice architectures.
- Data visualisation tools such as Tableau or Data Studio.
- Finance industry domain knowledge, trading fundamentals and fintech systems (e.g. FIX protocol).