PySpark Senior Developer (R5699)

Engineering Delhi, India


Job Duties


  • PySpark senior developer's responsibilities include the ability to design, build and unit test the application
  • Translate business and functional requirements into robust, scalable, operable solutions that work well within the overall data architecture at Avalara
  • Design and develop data pipelines, data ingestion and ETL processes that are scalable, repeatable and secure for stakeholder needs
  • Participating in peer reviews
  • Participating in Agile development activities including sprint planning, stand-ups, retrospectives and demos
  • Excellent verbal and written communication skills to collaborate effectively with partner teams globally
  • Qualification


  • Bachelor of Engineering in Computer Science or equivalent
  • 5+ years experience designing and developing in pyspark/ Python
  • Atleast 2+ years of experience in AWS ETL ecosystem
  • Primary Skills


  • Deep working experience in developing data processing tasks using pySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations
  • Working experience in AWS ecosystem services like S3/Athena AWS Athena/ Redshift/ Lambda/ S3/ Step functions, SQS, SNS
  • Experience with data modelling and building ETL pipelines
  • Experience in data processing, aggregation and transformation with unit tests and design data processing pipeline, exception handling and performance optimisation technique
  • Strong knowledge of working with different data file formats like Parquet, JSON, CSV, etc
  • Define and capture metadata and rules associated with ETL processes
  • Organise, cleanse, validate, and format data for storage
  • Working experience in SQL, NoSQL databases like – DynamoDB, Postgres, SQL
  • Experience in GitLab deployment using Terraforms and operationalising the code is added advantage
  • Good to have knowledge and skills in Devops/version control and containerisation
  • Added Advantage: AWS Certification

    About Avalara

    We're building cloud-based tax compliance solutions to handle every transaction in the world.

    Imagine every transaction you make — every tank of gas, cup of coffee, or pair of sneakers, every movie ticket, meal kit, or streamed song, every sensor-to-sensor ping. Nearly every time you make a purchase, physical or digital, there's an accompanying unique and nuanced tax compliance calculation.

    The logic behind calculating taxes — the rules, rates, and boundaries — is a global, layered, three-dimensional mess of complexity, with compliance dictated by governments and applied by every business, every day.

    Avalara works with businesses of all sizes, all over the world — from corner stores to gigantic global retailers — to calculate tax accurately and automatically, at speeds measured in milliseconds.

    That's a massive technical challenge, in terms of scale, reliability, and complexity, and we do it better than anyone. That's why we're growing fast.

    Headquartered in Seattle, Avalara has offices across the U.S. and around the world, in Belgium, Brazil, Canada, India, and the U.K.

    Avalara is an Equal Opportunity Employer. All qualified candidates will receive consideration for employment without regard to race, color, creed, religion, age, gender, national orientation, disability, sexual orientation, or any other factor protected by law.

    Avalara is an Equal Opportunity Employer. All qualified candidates will receive consideration for employment without regard to race, color, creed, religion, age, gender, national orientation, disability, sexual orientation, US Veteran status, or any other factor protected by law.