Job Summary:

We are seeking a highly skilled Senior Software Developer with expertise in AWS Glue and ETL to join our team. The ideal candidate will be responsible for designing, implementing, and optimizing ETL pipelines while ensuring seamless data integration and transformation. This role requires strong proficiency in Apache Spark, AWS Glue, and schema-on-read architectures to enable efficient data processing.

Key Responsibilities:

AWS Glue Integration & Enablement:

• Introduce AWS Glue as the primary ETL tool in the technology stack.
• Configure AWS Glue Crawlers, Data Catalogs, and ETL jobs to automate schema discovery, normalization, and metadata management.
• Establish best practices and guidelines for AWS Glue adoption.

Data Catalogs and Schema-on-Read:

• Design and implement schema-on-read architectures for dynamic schema interpretation.
• Create and maintain centralized data catalogs using AWS Glue for metadata management.

ETL Process Design & Optimization:

• Develop agile ETL workflows for processing diverse data formats (JSON, CSV, text files).
• Optimize ETL processes for scalability, performance, and error handling.

Spark-Based Reusable Code Development:

• Build modular, reusable Spark code for data processing, transformation, and cleaning.
• Ensure code is optimized for performance and scalability across multiple data sources.

Configuration-Driven Pipelines:

• Design low-code ETL workflows using configuration-based setups.
• Collaborate with non-technical stakeholders to develop user-friendly interfaces for data pipeline configuration.

Data Quality and Validation:

• Implement robust QA mechanisms to validate data integrity across the ETL pipeline.
• Develop automated validation scripts to ensure data accuracy and compliance with standards.

Collaboration and Leadership:

• Work closely with cross-functional teams, including data engineers, software developers, and product managers, to align ETL pipelines with business requirements.
• Mentor junior team members in AWS Glue best practices and advanced ETL development techniques.

Monitoring and Maintenance:

• Integrate monitoring tools such as AWS CloudWatch to provide visibility into ETL job performance and ensure timely issue resolution.
• Establish documentation for ETL processes, AWS Glue configurations, and reusable code components.

Qualifications & Skills:

• Proven experience with AWS Glue, including Data Catalogs, Crawlers, and ETL job configuration.
• Strong expertise in Apache Spark for data processing and transformation.
• Hands-on experience with schema-on-read architectures and managing evolving data schemas.
• Proficiency in ingesting and processing heterogeneous data sources (e.g., JSON, CSV, text).
• Experience developing modular, reusable code for data wrangling and transformation.
• Deep understanding of data quality and validation strategies in ETL pipelines.
• Proficiency in Python or Scala for Spark and AWS Glue development.
• Familiarity with AWS services such as S3, Athena, and Redshift.
• Strong knowledge of configuration-driven ETL design principles.
• Experience with monitoring and logging tools such as AWS CloudWatch.
• Knowledge of performance tuning in Spark-based ETL workflows.
• Ability to document workflows and provide technical training.
• Excellent problem-solving skills and attention to detail.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

Preferred Qualifications:

• Experience in big data processing and real-time data pipelines.
• Knowledge of DevOps and CI/CD practices for data engineering workflows.
• Familiarity with data security and compliance standards.

To apply, share your resume at hr@msezy.com