Lead Data Engineer

Project description

We are hiring for projects in the Middle East. There are many opportunities in the region. Our team consists of frontend and backend developers, data analysts and data scientists, architects, analysts and project managers, and Databricks Data engineers. As a Lead Data Engineer, you will manage the development, implementation and maintenance of data infrastructure for modern data platform.

Responsibilities

1. Platform Design: Collaborate with cross-functional teams to refine and design the architecture of our data platform.
2. Platform Optimization: Continuously enhance the performance, scalability, and reliability of the data platform to meet evolving business requirements.
3. Platform Maintenance: Implement monitoring, alerting, and maintenance processes to ensure the availability and health of the data platform.
4. Data Governance: Establish and enforce data governance best practices to uphold data quality, security, and compliance standards.
5. Platform Integration: Integrate new technologies seamlessly into the data platform ecosystem, ensuring compatibility and interoperability.
6. Collaboration: Work closely with data engineers, data scientists, analysts, and business stakeholders to comprehend requirements and offer technical expertise and support.
7. Documentation: Create and maintain comprehensive documentation for the data platform architecture, processes, and workflows.
8. Mentorship: Provide guidance and mentorship to junior members of the data engineering team, nurturing their growth and development.
9. Design, develop, and maintain data pipelines to ingest and transform raw data into Silver and Gold tables using Databricks.
10. Build reusable data pipelines that can be utilized by the application development team.
11. Collaborate with data consumers and other stakeholders to understand data requirements and deliver high-quality data solutions.
12. Implement declarative ETL pipelines using Databricks, Delta Lake, and other relevant tools to ensure data is accurate, timely, and accessible.
13. Optimize data models and pipelines for performance, scalability, and reliability.
14. Model data from various sources, ensuring consistency and quality in the Silver layer.
15. Curate data assets in the Silver layer to support data science and machine learning applications.
16. Build and maintain the Gold layer to provide clean, aggregated, and business-ready data for reporting and analytics.
17. Ensure data governance and security best practices are followed.
18. Well versed with Databricks fine-tuning techniques for huge data sets.
19. Document data processes, models, and pipelines for transparency and knowledge sharing.
20. Well versed with DevOps practices for continuous integration and deployment.
21. Writing code, reviewing pull requests, providing constructive feedback and maintaining high code quality standards.
22. Maintain a deep understanding of data engineering principles, best practices, and emerging technologies to drive engineering initiatives.

SKILLS

Must have

1. Overall 5-7 years of experience in data engineering and transformation on Cloud.
2. 3+ Years of Very Strong Experience in Azure Data Engineering, Databricks.
3. Expertise in supporting/developing lakehouse workloads at the enterprise level.
4. Experience in PySpark is required - developing and deploying the workloads to run on the Spark distributed computing.
5. Candidate must possess at least a Graduate or bachelor's degree in Computer Science/Information Technology, Engineering (Computer/Telecommunication) or equivalent.
6. Cloud deployment: Preferably Microsoft Azure.
7. Experience in implementing the platform and application monitoring using Cloud native tools.

Nice to have

-