Mid Level Microsoft Fabric Data Engineer
Description
Company Overview:
Lean Tech is a rapidly expanding organization situated in Medellín, Colombia. We pride ourselves on possessing one of the most influential networks within software development and IT services for the entertainment, financial, and logistics sectors. Our corporate projections offer many opportunities for professionals to elevate their careers and experience substantial growth. Joining our team means engaging with expansive engineering teams across Latin America and the United States, contributing to cutting-edge developments in multiple industries.
Position Title: Mid Level Microsoft Fabric Data Engineer
Location: Remote (LATAM)
What you will be doing:
Seeking for an experienced Data Engineer responsible for designing, building, and maintaining end-to-end analytics pipelines in Microsoft Fabric and Azure. Key tasks include creating performant ETL/ELT workflows with PySpark and SQL, managing telemetry and time-series data in ADX with KQL, and implementing hot/cold storage strategies. The engineer will own production pipelines, ensure data quality, apply CI/CD and infrastructure-as-code, and establish monitoring. Collaboration with product owners, analysts, and BI engineers is essential, along with mentoring juniors and driving platform improvements. Candidates need 3–6+ years of data engineering experience, strong SQL and KQL/ADX expertise, and recent Fabric/Azure experience; Power BI knowledge is a plus but not the focus.
- Design, implement, and maintain end-to-end ETL/ELT data pipelines using Microsoft Fabric components (Data Factory, Synapse Data Engineering), Fabric lakehouses/warehouses and OneLake to deliver analytics-ready datasets for downstream reporting and analysis.
- Own data modeling and transformations across Spark (PySpark) and SQL workloads, producing performant, cost‑efficient schemas and semantic models optimized for large-scale analytics.
- Develop and apply advanced SQL and Kusto Query Language (KQL) techniques to tune query performance, optimize aggregations and time‑series analyses, and ensure efficient access to multi‑GB/TB datasets.
- Build and tune Azure Data Explorer (Kusto) solutions: define ingestion patterns (e.g., queued/batched ingestion), retention and hot/cold storage strategies, partitioning and materialized views for gold-layer analytics.
- Implement storage and file-layout best practices (Parquet/Delta formats, effective partitioning and compaction strategies) in OneLake to support high-throughput time‑series data and reduce small‑file overhead.
- Implement development best practices for data engineering artifacts: author IaC/CI‑CD pipelines, apply Git-based workflows, automate testing and automate deployments for repeatable, auditable releases.
- Establish observability and monitoring for data workloads: configure metrics, alerts, capacity planning and cost‑monitoring to maintain pipeline reliability and resource efficiency (excluding run‑book/on‑call commitments where not in scope).
- Ensure data quality, lineage, security and compliance by implementing validation checks, access controls, documentation and collaborating with data governance stakeholders on cataloging and classification.
- Collaborate with product owners, analysts and BI developers to translate business requirements into data solutions and deliver data‑ready semantic/data models for reporting and analytics consumption.
- Mentor and guide junior engineers, contribute to team best practices, and drive architectural decisions and technical improvements across Fabric and Azure data platforms.
Requirements & Qualifications
To excel in this role, you should possess:
- 3–6 years of professional data engineering experience, including at least 2 years hands‑on with Microsoft Fabric Data Engineering (pipelines, notebooks, lakehouse/warehouse, OneLake)
- Advanced, hands‑on experience with Microsoft Fabric for data engineering workloads and OneLake integration
- Proven ability to design, build and maintain end‑to‑end ETL/ELT pipelines using Fabric, Azure Data Factory and/or Synapse Data Engineering
- Strong Spark experience (PySpark and/or Scala) and practical familiarity with Delta/Parquet formats for large‑scale transformations and storage
- Advanced SQL/T‑SQL development skills with demonstrated performance tuning and optimization for multi‑GB/TB datasets
- Intermediate data modeling and transformation skills focused on delivering analytics‑ready datasets and optimized query performance for reporting workloads
- Advanced experience with Azure Data Explorer (Kusto/ADX) and Kusto Query Language (KQL), including complex queries, aggregations, time‑series analysis and query performance tuning
- Practical experience with ADX ingestion and telemetry patterns (e.g., queued/batched ingestion), retention strategies and use of materialized views for gold‑layer outputs
- Knowledge of storage partitioning and file layout best practices for time‑series data (partitioning, compaction and small‑file handling) when using Parquet/Delta in OneLake
- Familiarity with Synapse Data Engineering, Azure Data Factory, and related Fabric runtimes (lakehouses and data warehouses) as part of the platform architecture
- Proven track record with large‑scale data movement and ETL/ELT pipeline reliability and performance
- Practical experience integrating data outputs with BI tools and semantic models; Power BI integration experience is beneficial, though the primary focus is data engineering
- Experience applying development best practices: Git, CI/CD for data pipelines, infrastructure‑as‑code patterns, automated testing and deployment for data artifacts
- Skills in monitoring and operating production pipelines—observability, alerting, incident response, capacity planning and cost optimization—while on‑call rotation is not expected
- Strong English communication skills, experience working in Agile teams, and the ability to collaborate with product owners, analysts and BI developers
- Demonstrated ability to mentor junior engineers, contribute to team best practices, and influence architectural and technical improvements across Fabric/Azure data platforms
- Working familiarity with adjacent Azure services (Azure Data Lake, Databricks, Logic Apps) and an understanding of data governance/catalog concepts (OneLake/Purview) is desirable
Nice to have:
- Practical exposure to telemetry ingestion technologies such as Azure Event Hubs and the Data Collector API, and experience integrating these sources with ADX or Fabric ingestion patterns
- Experience designing and executing performance benchmarking and load‑testing for large ETL/ELT workloads to validate throughput, latency and cost trade‑offs
- Familiarity with data‑testing practices and frameworks for pipelines and notebooks (unit and integration testing) to strengthen CI/CD reliability
- Proven ability to produce clear technical documentation and runbooks, and to present technical designs and trade‑offs to non‑technical stakeholders
- Relevant Microsoft/Azure certifications or formal training in Fabric/Azure data engineering, Spark or ADX (desirable but not required)
- Experience with performance and cost‑profiling methodologies for storage and computation to inform optimization and capacity planning decisions.
- Manage end-to-end pipelines, ensure performance, cost optimization, and production readiness.
- Partner with product owners, analysts, and BI teams to deliver analytics-ready datasets and models.
- Apply structured thinking to data modeling, performance tuning, and large-scale data movement.
- Implement monitoring, observability, alerts, and incident response for reliable pipelines.
- Enforce validation, security, access controls, and compliance standards.
- Balance performance, cost, and complexity when choosing storage, partitioning, and ingestion strategies.
- Contribute within Agile teams, actively supporting planning, delivery, and production deployment.
Soft Skills:
- Strong written and verbal English skills; able to document and present to technical and non-technical stakeholders.
- Excellent problem-solving and analytical skills.
- Demonstrates a willingness to learn and grow.
- Highly motivated to build upon existing work and set new benchmarks.
- Takes responsibility and drives projects forward with a strong sense of accountability.
Why you will love Lean Tech:
- Join a powerful tech workforce and help us change the world through technology.
- Professional development opportunities with international customers.
- Collaborative work environment.
- Career path and mentorship programs that will lead to new levels.
Join Lean Tech and contribute to shaping the data landscape within a dynamic and growing organization. Your skills will be honed, and your contributions will be vital to our continued success. Lean Tech is an equal-opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.