Data Engineer, Data Warehouse
Description
Company Overview
Shamrock Trading Corporation is the parent company for a family of brands in transportation services, finance and technology. Headquartered in Overland Park, KS, Shamrock is frequently recognized among the “Best Places to Work” in Kansas City and Chicago and was most recently recognized as one of America’s top 100 “Most Loved Workplaces” by Newsweek. We also have offices in Atlanta, Chicago, Dallas, Ft. Lauderdale, Houston, Laredo, Nashville and Phoenix.
With an average annual revenue growth of 25% over several decades, Shamrock’s success is attributed to three key factors: hiring the best people, cultivating long-term relationships with our customers and continually evolving in the marketplace.
Responsibilities
Shamrock Trading Corporation is looking for a Data Engineer who wants to utilize their expertise in data warehousing, data pipeline creation/support and analytical skills by joining our Data Services team. This role is responsible for gathering and profiling data from many internal and external sources, building pipelines and designing a reliable cloud-focused data platform for analytics and business intelligence teams. This role requires significant understanding of data mining and analytical techniques. An ideal candidate will have strong technical capabilities, business acumen, and the ability to work effectively with cross-functional teams. Responsibilities include but are not limited to:
- Work with Architects to understand current data models, to build pipelines for data ingestion and transformation
- Design and implement pipelines and/or ETL/ELT jobs in order to centralize data from internal and external sources, focusing on reliability and performance of jobs and ensuring the quality and consistency of data is maintained at all times
- Follow framework and code patterns established by Architects and Data Engineers for reliable job execution
- Gather, understand and fulfill technical requirements from other units and provide necessary updates throughout the project lifecycle
Qualifications
- Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience
- Experience extracting and manipulating data from a variety of sources, primarily cloud-based
- Experience building and maintaining data pipelines and ETL/ELT jobs: currently utilizing Matillion, Databricks, and Airflow
- Expertise in optimizing performance when loading into and extracting from relational and other data structures, including SQL tuning, indexing, etc.
- 5+ years of development experience with the following, and Subject Matter Expert (SME) with at least one:
- Strong SQL writing and tuning skills
- Designing and deploying jobs within Matillion or similar GUI-based ETL/ELT tool
- Expertise in designing, developing, deploying and/or supporting data pipelines using Databricks (pySpark, pySQL, DLT)
- Relational Databases (MS SQL, PostgreSQL, Redshift)
- Demonstrated analytical and problem-solving skills
- Strong communication and listening skills, ability to work directly with customer teams (Business units and internal IT)
Preferred Qualifications
- Understanding the frameworks and principles of traditional data warehousing
- Experience with data processing and analytics within AWS eco-system
- Experience data processing using Parquet and Avro files
- Working with event based / streaming technologies to ingest and process data
- Development experience delivering analytics processes using Python
- Experience with front-end BI and data visualization tools (such as Power BI, Grafana, and QuickSight)
Benefits Package
At Shamrock we hire bright, ambitious people and give them the tools they need to be successful. By investing in training and development, we hope to become a long-term career for employees, where there are always opportunities for advancement. Shamrock also offers a premier set of benefits for employees and their families:
- Medical: Fully paid healthcare, dental and vision premiums for employees and eligible dependents
- Work-Life Balance: Competitive PTO and paid leave policies
- Financial: Generous company 401(k) contributions and employee stock ownership after one year
- Wellness: Onsite gym and discounted membership to select fitness centers. Jogging trails available at Overland Park offices
#LI-NB1 #LI-Hybrid