Senior Backend Engineer
Description
Company Overview:
Lean Tech is a rapidly expanding organization situated in Medellín, Colombia. We pride ourselves on possessing one of the most influential networks within software development and IT services for the entertainment, financial, and logistics sectors. Our corporate projections present numerous opportunities for professionals to advance their careers and experience substantial growth. Joining our team means engaging with expansive engineering teams across Latin America and the United States, contributing to cutting-edge developments in multiple industries.
Position Title: Senior Software Engineer (Python+Spark+AWS)
Location: Remote - Colombia
What you will be doing:
We are seeking a Senior Software Engineer to take ownership of a large-scale, distributed data processing platform. In this role, you will design, build, and optimize backend software systems that process data at a significant scale. You will work with Python-based services and distributed computing frameworks running on a scalable AWS infrastructure.
This role goes beyond implementing data workflows. You will solve complex distributed systems problems, focusing on performance, reliability, and cost efficiency. You understand how to manage memory and execution in JVM-based environments through PySpark, address data skew, and design efficient compute architectures on AWS. Your responsibilities will include:
- Distributed Systems Architecture: Design and build scalable batch and streaming processing systems using Apache Spark (PySpark) on AWS EMR or AWS Glue.
- Performance Engineering: Optimize distributed jobs for performance and cost, including tuning memory allocation, parallelism, partitioning strategies, and diagnosing runtime failures such as out-of-memory errors and long-running shuffles.
- Storage & Data Platform Design: Architect and maintain the data storage layer on AWS S3, applying best practices for partitioning, compression, and columnar file formats such as Parquet, as well as modern table formats like Delta Lake or Apache Iceberg.
- Software Engineering in Python: Develop production-grade, modular, and testable Python code, transforming experimental or notebook-based logic into robust, maintainable software packages.
- Workflow Orchestration: Design and manage complex execution workflows and dependencies using orchestration tools such as Apache Airflow (MWAA) or AWS Step Functions.
- Code Quality & Mentorship: Lead code reviews, enforce engineering standards, and mentor junior engineers on software design and distributed systems concepts.
Requirements & Qualifications
To excel in this role, you should possess:
- Experience: 5+ years of professional experience in software or data-focused engineering roles.
- Distributed Computing: 4+ years of hands-on experience building systems with Apache Spark.
- Education: B.S. or M.S. in Computer Science, Mathematics, or a related engineering discipline.
- Orchestration: Proven experience working with Airflow or similar scheduling and orchestration tools.
- SQL Expertise: Advanced SQL skills for system analysis, validation, and debugging.
- PySpark Proficiency: Deep experience with the Spark DataFrame API and Spark SQL.
- System Internals: Solid understanding of Spark internals, including the Catalyst Optimizer, Tungsten execution engine, DAGs, and execution stages.
- Optimization Techniques: Hands-on experience resolving data skew, implementing broadcast joins, and managing driver and executor memory overhead.
- Data Formats: Strong knowledge of columnar storage formats (Parquet/ORC) and modern table formats such as Delta Lake or Apache Iceberg.
- Advanced Python: 5+ years of experience writing Python with a strong grasp of object-oriented and functional programming concepts.
- Cloud Compute: Experience configuring, scaling, and optimizing EMR clusters or managing AWS Glue jobs.
- Cloud Storage: Advanced understanding of AWS S3 access patterns and lifecycle management.
- Security: Experience configuring IAM roles and policies to ensure secure system access.
- Event-Driven Systems: Familiarity with serverless components such as AWS Lambda for triggering and automation.
Why you will love Lean Tech:
- Join a powerful tech workforce and help us change the world through technology
- Professional development opportunities with international customers
- Collaborative work environment
- Career path and mentorship programs that will lead to new levels.
Join Lean Tech and contribute to shaping the data landscape within a dynamic and growing organization. Your skills will be honed, and your contributions will be vital to our continued success. Lean Tech is an equal-opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.