Job Summary:

We are seeking a highly skilled Senior Software Developer with expertise in AWS Glue and ETL to join our team. The ideal candidate will be responsible for designing, implementing, and optimizing ETL pipelines while ensuring seamless data integration and transformation. This role requires strong proficiency in Apache Spark, AWS Glue, and schema-on-read architectures to enable efficient data processing.

Key Responsibilities:

AWS Glue Integration & Enablement

• Introduce AWS Glue as the primary ETL tool in the technology stack.
• Configure AWS Glue Crawlers, Data Catalogs, and ETL jobs to automate schema discovery, normalization, and metadata management.
• Establish best practices and guidelines for AWS Glue adoption.

Data Catalogs and Schema-on-Read

• Design and implement schema-on-read architectures for dynamic schema interpretation.
• Create and maintain centralized data catalogs using AWS Glue for metadata management.

ETL Process Design & Optimization

• Develop agile ETL workflows for processing diverse data formats (JSON, CSV, text files).
• Optimize ETL processes for scalability, performance, and error handling.

Spark-Based Reusable Code Development

• Build modular, reusable Spark code for data processing, transformation, and cleaning.
• Ensure code is optimized for performance and scalability across multiple data sources.

Configuration-Driven Pipelines

• Design low-code ETL workflows using configuration-based setups.
• Collaborate with non-technical stakeholders to develop user-friendly interfaces for data pipeline configuration.

Data Quality and Validation

• Implement robust QA mechanisms to validate data integrity across the ETL pipeline.
• Develop automated validation scripts to ensure data accuracy and compliance with standards.

Collaboration and Leadership

• Work closely with cross-functional teams, including data engineers, software developers, and product managers, to align ETL pipelines with business requirements.
• Mentor junior team members in AWS Glue best practices and advanced ETL development techniques.

Monitoring and Maintenance

• Integrate monitoring tools such as AWS CloudWatch to provide visibility into ETL job performance and ensure timely issue resolution.
• Establish documentation for ETL processes, AWS Glue configurations, and reusable code components.

To apply, share your resume at hr@msezy.com