Data Engineer Level 3
Description
JOB TITLE: | Data Engineer, Level 3 |
DEPT/DIV: | Chief Strategic Initiatives Officer/ Data & Analytics |
WORK LOCATION: | 2 Broadway |
FULL/PART-TIME | FULL |
SALARY RANGE: | $110,000 - $130,511 |
DEADLINE: | Until filled |
This position is eligible for teleworking, which is currently one day per week. New hires are eligible to apply 30 days after their effective hire date.
Opening:
The Metropolitan Transportation Authority is North America's largest transportation network, serving a population of 15.3 million people across a 5,000-square-mile travel area surrounding New York City, Long Island, southeastern New York State, and Connecticut. The MTA network comprises the nation’s largest bus fleet and more subway and commuter rail cars than all other U.S. transit systems combined. MTA strives to provide a safe and reliable commute, excellent customer service, and rewarding opportunities.
Position Objective:
The incumbent will help lead the team that designs, builds, tests, and delivers end-to-end, automated data pipelines over complex on-premises and off-premises platforms. They will work to extract data from multiple source systems containing structured, semi-structured, and unstructured data to make it consistent, reliable, available, and usable to colleagues across the MTA and, in support of the agency’s and New York State’s Open Data goals, to external stakeholders and the general public.
They will use languages such as SQL, Python, and R and relational database tools such as Oracle, Postgres, and SQL Server to analyze large datasets, build new ones, and design overall data architectures. They will carefully document all work and work closely with colleagues to define needs, problem-solve, support the overall team agenda, and build relationships throughout and at all levels of the agency. They will have to be able to quickly learn the unique features, data constraints, and business needs of any part of the MTA. In addition, unlike other data engineering roles, they will support the entire downstream pipeline process and, occasionally, end-users of the data products. In general, they will have to support the MTA’s strategic goals to build data systems and processes that are well-structured and sustainable.
Responsibilities:
- Lead projects to develop data pipelines, data warehouses, data marts, multi-dimensional cubes, and data lakes to collect, structure, and integrate a wide range of data sources; create data system assets that are efficient, timely, reliable, and accurate, robust, and scalable; and make information available to staff and decision-makers for analysis and consumption.
- Identify, propose, support, and carry out the development of new data sets, data access, extraction methodologies, algorithms, and other tools that support the agency's strategic goals, service quality, and efficiency, while advancing data-driven decision-making across the company through stakeholder partnering, tool design, and effective communications.
- Test data sets and pipelines, conducting root cause analyses to resolve issues, incorporating quality assurance functionality in work products, and finding enterprise-class solutions to project challenges where possible.
- Evaluate existing legacy data, algorithmic, and process solutions; redesign and implement modern data infrastructure to improve efficiency, timeliness, quality, robustness, and scalability.
- Play a lead role in developing standards for data architecture diagrams, system documentation, data models, business decision-making maps, and other design related artifacts to ensure consistency and improve understanding of both technical and non-technical users of Data & Analytics products.
- Provide input into data governance initiatives and influence the development of sustainable data management and governance practices across the agency while ensuring data governance and sustainability needs are built into project designs.
- Research and recommend the best pipeline technologies, toolsets, and applications to support the data science and reporting teams, keeping skills current and maintaining data systems in the latest technologies and cloud platforms.
- Lead project planning and support manager in the design of workstreams, updating project plans as needed, and communicating with clients and all relevant parties on project status.
- Enhance team performance by: supporting the recruitment of new teammates; supporting a professional environment that respects individual differences, enables all colleagues to develop and contribute to their full potential at the MTA and beyond; and mentoring less-experienced members of the team to build their business and coding/analytic knowledge and skills.
- Perform other duties as assigned.
Required Knowledge/Skills/Abilities:
- Experience developing front end data apps, from scratch, not just working on existing ones.
- Strong design skills, who can make calls about how things should be visually communicated.
- Experience actually hosting and deploying apps like this, in production, ideally where they have owned a significant part of the deployment/hosting.
- Strong skills in programming, database design, and data lake architectures for data engineering.
- Ability to play a lead role on the Data & Analytics team to set priorities and overall program planning in support of business goals.
- Ability to work with data of different types — structured, semi-structured, unstructured — as well as from distinct disciplines — transportation, finance, HR, asset maintenance.
- Knowledge of transit/ transportation systems and excellent judgment on how to manage or overcome technical, organizational, or governance constraints to meet business goals.
- Familiarity with common algorithms used to calculate KPIs and extensive experience with data visualization and business intelligence tools such as Power BI.
- Experience with data engineering orchestration tools (e.g., Airflow, Dagster, etc.) and DevOps tools (e.g., ADO, Git, Jenkins, Docker, etc.) and familiarity with related concepts (e.g., CI/CD).
- Extensive experience and ability to design and implement quality assurance and automated testing systems.
- Exceptional ability to read code and understand technical issues, keep up with technical innovation and trends in data engineering.
- Ability to collaborate and provide support to all levels of MTA, both technical and non-technical.
- Ability to deconstruct difficult problems into smaller and simpler pieces.
- Extensive experience in project design, strong project management skills, and the ability to lead successful team-based projects.
- Proactive interest in identifying strategic, policy, and business issues in all proposed and ongoing projects.
- Strong written communication skills for both non-technical and technical documents.
- Ability to present and engage on complex work products with executive audiences and to review and make decisions on project design options with clients and stakeholders.
Required Education and Experience:
- Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or related field. An equivalent combination of education and experience may be substituted in lieu of a degree.
- A minimum of (4) years of experience working in a front-end development or another position with similar programing and data visualization content.
- A minimum of (3) years of experience working in a data engineering or another position with similar programming and dataset design content.
- A minimum of (3) years of experience building pipelines, automating tasks through scripts, writing database queries, and debugging/ maintaining code.
- A minimum of 4 years) experience with both Python and SQL programming.
- A minimum of (4 years) experience with relational databases (e.g., Oracle, Postgres, SQL Server, NoSQL), including writing queries (generally with PL/SQL) to obtain and manipulate data.
The Following is/are preferred:
- Master’s degree in Computer Science, Information Technology, Engineering, Mathematics, or a related field.
Other Information
May need to work outside of normal work hours (i.e., evenings and weekends)
Travel may be required to other MTA locations or other external sites.
According to the New York State Public Officers Law & the MTA Code of Ethics, all employees who hold a policymaking position must file an Annual Statement of Financial Disclosure (FDS) with the NYS Commission on Ethics and Lobbying in Government (the “Commission”).
Equal Employment Opportunity
MTA and its subsidiary and affiliated agencies are Equal Opportunity Employers, including those concerning veteran status and individuals with disabilities.
The MTA encourages qualified applicants from diverse backgrounds, experiences, and abilities, including military service members, to apply.