Senior Data Engineer

Job Title: Senior Data Engineer Job Location: Abu Dhabi Overview Data Engineering Lead is responsible for overseeing the development and support of internally built or supported ETL/ELT processes and Azure Data Services, including Azure Data Factory, Azure Databricks, Big Data and streaming solutions. This role involves gathering business requirements, designing data models, and developing end-to-end data solutions. ResponsibilitiesData Engineering (ETL/ELT & Data Processing): understand, analyse, and document business requirements to design data-driven solutions aligned with organizational goals. Deep expertise in ETL methodologies, Big Data, and Data Warehousing principles, including architecture design, data modelling, and performance optimization. Design, develop, test, optimize, and deploy ETL pipelines using Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage. Hands-on experience with leading ETL tools such as Talend, Informatica, and Pentaho for building scalable ETL/ELT workflows. Translate source-to-target mapping documents into efficient ETL processes to support data integration and transformation. Write, implement, and maintain robust ETL code and stored procedures using SQL, Python, and Spark.Cloud & Data WarehousingDesign, implement, and manage ETL processes on Microsoft Azure Cloud, with proficiency in ADF, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake. Administer and maintain cloud-based data platforms, including Azure Cloud and Snowflake Enterprise Data Warehouse, ensuring data security, performance, and availability. Develop and manage Spark jobs within Azure Databricks for processing large-scale datasets.CI/CD & DevOpsWork in an Agile development environment, applying best practices for CI/CD pipelines using Azure DevOps, GitHub, and automation tools. Automate deployment workflows, manage version control, and ensure continuous integration and delivery of data solutions. Implement DevOps practices for monitoring, testing, and optimizing data pipelines, ensuring high system reliability and performance.Monitoring & MaintenanceProactively monitor, troubleshoot, and maintain all ETL components to ensure data quality, system stability, and timely delivery of data. Optimize data workflows for performance, scalability, and cost-efficiency in cloud environments. Machine Learning & API Integration: Deploy Machine Learning (ML) models in the cloud using Azure Machine Learning and integrate with data pipelines. Design and integrate APIs to facilitate seamless data flow between systems and applications.Leadership & CollaborationLead, mentor, and support team members engaged in data engineering activities, promoting knowledge sharing and best practices. Collaborate with business stakeholders, data scientists, and IT teams to align technical solutions with business needs.Qualifications Essential:Bachelor's Degree in Computer Science, Engineering, or a related fieldDesirable:Advanced certification in Azure Data Engineering (Azure DF, Azure ML) Certification in Azure Databricks or any cloud data warehouse Any certification in data science is a plusExperience:5+ years of experience in Data Engineering 5+ years' experience of ETL development using Talend and Azure Data Services 5+ years' experience developing SQL queries, stored procedures, and views 3+ years' experience coding in Python 3+ years' experience with Azure or any cloud solutions3+ years' experience of Azure Databricks or any cloud data warehouse such as Snowflake or Redshift 2+ years' experience of Azure Data FactorySkillsProficient in Python and SQL for data engineering tasks Strong expertise in Azure Data Services, including Azure Data Factory, Azure Synapse Analytics, and Azure Databricks Knowledge of data warehousing concepts, ETL/ELT development, and optimization in Azure Experience deploying ML models using Azure Machine Learning Experience with Snowflake/Azure cloud data warehouses Strong SQL skills: stored procedures, views, and data modeling Experience with ERwin or similar tools for data modeling Experience with Azure DevOps and GitHub for version control and CI/CD Understanding of Master Data Management (MDM) and data profiling Excellent communication and collaboration skillsDesirable Skills:Experience with ML algorithms using Python, Azure Databricks, and Azure MLTechnology Stack Azure Data, Power BI, Tableau #J-18808-Ljbffr