Lead Data Engineer
Tasks
- Build data ingestion and transformation pipelines
- Conduct code and architecture reviews
- Design batch data pipelines
- Design real-time data pipelines
- Develop ETL/ELT frameworks
- Drive CI CD automated testing and observability
- Drive cost-optimization strategies
- Ensure data quality lineage observability governance
- Ensure high availability fault tolerance disaster recovery
- Implement infrastructure-as-code
- Implement secure data access controls
- Implement stream processing solutions
- Lead engineering mentorship and design guidance
- Operate cloud native data platforms
- Optimize data jobs performance scalability cost
Perks/Benefits
- N/A
Skills/Tech-stack
ARM Templates | AWS | Airflow | Amazon EMR | Amazon Redshift | Amazon S3 | Apache Flink | Apache Kafka | Apache Spark | Azure | Azure Data | Azure Data Lake | Azure Data Lake Storage | BigQuery | CI/CD | Cloud Storage | CloudFormation | Data Governance | Data Lake Storage | Data Lineage | Data Modeling | Data Warehousing | Databricks | Dimensional Modeling | Distributed Systems | Docker | Dynatrace | ELT | ETL | GDPR | GitHub Actions | Go | Google Cloud | Google Cloud Storage | IAM | JUnit | Java | Jenkins | Kubernetes | Observability | PCI DSS | Prometheus | Python | SQL | Snowflake | Snowflake Schema | Splunk | Star Schema | Terraform
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Senior-level Full TimeBengaluru, India7h ago
-
Lead Infrastructure Engineering INR 3000K-5000KAAA | AI Assisted Development | Access Control | Ansible | Audit managementSenior-level Full TimeIndia11h ago
-
Software Engineer II - Entity Intelligence INR 1200K-1680KAirflow | Algorithms | DNS | Data Structures | DatabricksOn-call rotation compensation | Remote/distributed teamMid-level Full TimeHybrid - Bangalore, India R15h ago
-
DevOps Engineer / Site Reliability Engineer (SRE) INR 2200K-3500KBash | CI/CD | Capacity Planning | Cloud Monitoring | Cloud loggingMid-level Full TimeIndia15h ago
-
Data minimization | Differential Privacy | Distributed Systems | GDPR | GoAnnual meetups | Co-working allowance | Flexible schedule | Fully remote | Home office setup allowanceSenior-level Full TimeIndia R16h ago
-
API | Access Governance | Access Management | Access reviews | Active DirectoryCareer development resources | Hybrid work environment | Wellbeing programsSenior-level Full TimeHyderabad, TG, India17h ago
-
API | Access Governance | Access Management | Access Review | Active DirectoryCareer development resources | Innovation practices | Wellbeing programsSenior-level Full TimeHyderabad, TG, India17h ago
-
Senior-level Full TimeJohnson Controls India COEE121h ago
-
Senior Staff Developer - IN - IDR - Integrations INR 1567K-4600KAWS | Azure | Distributed Systems | Docker | DynamoDBEmployee assistance program | Equity | Flexible leave | Life insurance | Medical insuranceSenior-level Full TimeOffice - IND - Karnataka, Bengaluru, …21h ago
-
Senior-level Full TimeJohnson Controls India COEE121h ago
-
Mid-level Full TimePune, India21h ago
-
Mid-level Full TimePune, India21h ago
-
Senior-level Full TimeNavi Mumbai, India (Finicity)21h ago
-
Senior-level Full TimeIndia-Mumbai (Kalpataru Square)21h ago
-
ServiceNow Platform Data Security Engineering- Manager INR 800K-1026KACL | Automation engine | Data masking | EDR | EncryptionMid-level Full Time11071 Ameriprise India - Hyderabad21h ago
-
Mid-level Full TimePune, India21h ago
-
Entry-level Full TimePune, India21h ago
-
Sr. Engineer | - VMS |Onsite, Bangalore INR 2500K-3500KAWS | Agentless Scanning | Asset Criticality | Attack surface | Attack surface managementProfessional training resources | Remote work support | Volunteer opportunities | Work-life balanceSenior-level Full TimeBangalore, India R21h ago
-
AVP - Infrastructure Security Services INR 1500K-4000KAAA | Access Control | Access Management | Account lifecycle management | AnsibleOn call 24x7 supportExecutive-level Full TimeBCIT Bengaluru Office (MGS), India21h ago
-
Senior-level Full TimeBengaluru, India1d ago
-
Cybersecurity Engineer INR 1500K-4600KApache Kafka | Apache Spark | Automation workflows | Compliance | Correlation logicSenior-level Full TimeHybrid (Bengaluru, Karnataka, IN); Hybrid (Pune, … R1d ago
-
Principal Engineer (Java/PAM), Software Engineering INR 3000K-5000KAWS | Access Management | Ant | Azure | CI/CDSenior-level Full TimeBengaluru1d ago
-
Software Engineer - Security INR 2000K-3487KAccess Management | Active Directory | Azure Active Directory | Bash | BigFixSenior-level Full TimeMumbai, Maharashtra, India1d ago
-
Consultant - Consulting Implementation INR 800K-1200KAWS | Access Control | Authentication | Cloud Security | DLPMid-level Full TimeGurgaon, Haryana, India2d ago
-
Sr. Software Development Engineer (IAM) INR 3000K-4000KAI anomaly detection | Anomaly Detection | CI/CD | Docker | HTTPEducation reimbursement | Health plans | Hybrid work | Parental leave | Retirement optionsSenior-level Full TimeBangalore, IND2d ago