AWS Data Engineer

Experience: 5 + years

Work mode: Hybrid preferred els Remote

AWS Data Pipeline & Cost Optimization Engineer

Role Overview

We are looking for a hands-on engineer with strong expertise in AWS-based data pipelines to help enhance alerting systems, optimize compaction jobs, strengthen data sanity frameworks, and drive cost optimization initiatives. The ideal candidate should have experience working with large-scale data systems, observability frameworks, and AWS-native services.

Key Responsibilities

1. Alerts & Monitoring Refinement

• Design and implement batch alerts for data processing pipelines.
• Refine existing alerting mechanisms to improve signal-to-noise ratio and operational efficiency.
• Integrate new job dashboards and alerting systems with existing monitoring frameworks.
• Migrate monitoring workloads away from Amazon CloudWatch to more cost-efficient alternatives while maintaining observability standards.

2. Compaction Job Optimization

• Review and re-evaluate compaction job changes previously implemented.
• Implement improvements to reduce storage and compute costs.
• Optimize data lifecycle, partitioning, and storage formats (e.g., Parquet, ORC).
• Improve job scheduling and execution efficiency in AWS data pipelines.

3. Data Sanity Framework Enhancement

• Revisit the existing data sanity/validation framework.
• Identify gaps in data validation, quality checks, and anomaly detection.
• Implement automated data quality checks across ingestion and transformation layers.
• Establish measurable SLAs and reporting for data reliability.

4. Cost Optimization

• Identify cost drivers across AWS data infrastructure.
• Optimize compute (e.g., Amazon EC2), storage (e.g., Amazon S3), and analytics services (e.g., Amazon EMR).
• Improve resource utilization, implement right-sizing strategies, and introduce lifecycle policies.
• Provide periodic cost analysis and recommendations with measurable impact.

Required Skills & Qualifications

• 5+ years of experience in AWS-based data engineering.
• Strong hands-on experience with AWS data services such as:
  • Amazon S3
  • Amazon EMR
  • AWS Glue
  • Amazon EC2
  • Amazon CloudWatch
• Experience building and maintaining batch and streaming data pipelines.
• Strong understanding of data compaction strategies and storage optimization.
• Experience in observability, monitoring, and alert tuning.
• Proven track record in AWS cost optimization initiatives.
• Strong SQL and scripting skills (Python/Scala preferred).

Nice to Have

• Experience with large-scale time-series or telemetry data systems.
• Exposure to data validation frameworks and automated quality checks.
• Familiarity with ClickHouse or columnar data stores.
• Experience working in high-scale, production-grade environments.