Lead Data Engineer

6 days ago

Abu Dhabi, Abu Dhabi, United Arab Emirates Inception Full time

We are seeking a highly skilled Lead Data Engineer to architect and build scalable, cloud-native data and AI pipelines that power enterprise LLM, RAG, and retrieval systems.

Key Responsibilities


• Design, build, and optimize scalable data pipelines for AI/LLM workloads, including vectorization and embedding processing.


• Develop and maintain ETL/ELT workflows for structured, unstructured, and streaming data.


• Create and manage vector database indexing and similarity search pipelines using tools like FAISS, Pinecone, Weaviate, Qdrant, Chroma.


• Build retrieval systems for RAG, semantic search, and enterprise knowledge retrieval.


• Develop robust, reusable data orchestration pipelines using Airflow, Spark, or similar tools.


• Architect and manage data pipelines across Azure (primary), AWS, and GCP environments.


• Integrate and optimize storage and processing across SQL, NoSQL, and vector databases.


• Contribute to the design and implementation of event-driven architectures.


• Collaborate with AI teams to enable embedding generation, LLM integration, and model-serving pipelines.


• Ensure end-to-end data quality, monitoring, reliability, and observability.


• Lead or participate in system design for large-scale, distributed data and AI systems.

Required Skills

Programming & Data


• Strong expertise in Python for data processing, APIs, automation, or distributed workloads.


• Strong proficiency in SQL and knowledge of NoSQL databases (MongoDB, DynamoDB, Cosmos DB, etc.).


• Experience with vector databases, such as: FAISS, Pinecone, Weaviate, Qdrant, Chroma.


• Strong knowledge of data modeling, pipeline development, and ETL/ELT frameworks.

AI/LLM Infrastructure


• Solid understanding of vectorization, embeddings, and similarity search techniques.


• Familiarity with LLMs, embedding models, and RAG pipeline concepts.


• Experience integrating embedding-generation pipelines via Hugging Face, OpenAI, or other model providers.

Cloud & Distributed Systems


• Proficiency with Azure (primary), and familiarity with AWS and GCP.


• Experience with Docker and containerized development.


• Understanding of Kubernetes is a strong plus.

Orchestration & Big Data


• Expertise in Apache Airflow for scheduling and orchestration.


• Experience with Apache Spark or equivalent distributed processing frameworks.

Architecture & Engineering Fundamentals


• Strong system design fundamentals for scalable and distributed systems.


• Knowledge of event-driven architecture and modern data platforms.


• Strong understanding of DevOps, CI/CD, version control, and observability best practices.

Qualifications

  • 8+ years of progressive experience in data engineering, distributed systems, or AI/ML data infrastructure
  • Experience building RAG pipelines in production.
  • Knowledge of graph databases or hybrid search systems.
  • Understanding of model deployment, inference optimization, and caching techniques for LLM workloads.
  • Familiarity with data governance, IAM, and security patterns across cloud ecosystems

  • Lead Data and BI Engineer

    4 days ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Visionary Tech Services LLC Full time

    The Lead Data and BI Engineer makes trusted, timely data and insight delivery for AI. Reporting to the Head of AI and Data Engineering, this role owns the end‑to‑end data and BI stack from ingestion and transformation to semantic layers, metrics, and dashboards and line‑manages Associate and Senior Data Engineers and BI Engineers to ship secure,...

  • Lead Data and BI Engineer

    4 days ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Visionary Tech Services LLC Full time

    The Lead Data and BI Engineer makes trusted, timely data and insight delivery for AI. Reporting to the Head of AI and Data Engineering, this role owns the end‑to‑end data and BI stack from ingestion and transformation to semantic layers, metrics, and dashboards and line‑manages Associate and Senior Data Engineers and BI Engineers to ship secure,...

  • Data Engineer

    1 week ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Durlston Partners Full time

    Role: Data EngineerClient: Quant Trading FirmSalary: Up to $300,000 (tax-free)Location: Abu Dhabi, UAE (full relocation assistance provided)Durlston Partners is partnering with a leading Quant Trading firm that leverages advanced data and technology to execute market-beating strategies. They are seeking a Data Engineer to design, build, and scale the...

  • Head of Data Engineering

    4 days ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Arcus Search Full time

    Arcus Search have partnered with a leading Financial Services organisation in Abu Dhabi, who are looking for a Head of Data Engineering (Emirati) to join the team. More details are below:Role: Head of Data EngineeringLocation: Abu Dhabi - possibly could be based in Dubai tooBudget: 45, ,000 AEDThe role:This role owns the organisation's data platform and...

  • Data Engineer

    2 weeks ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Digital Corner for Network Solutions Full time

    AboutThis opportunity is for a leading Energy & Gas giant in the UAE. We are looking for Data Engineers to build the pipelines that power our analytics. You will be responsible for the extraction, transformation, and loading (ETL) of data, creating the reliable infrastructure needed for data science and business intelligence.ResponsibilitiesDesign, build,...

  • Data Engineer

    6 days ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates ClearPeaks Full time

    ClearPeaks is a specialist consulting firm delivering services and solutions in "Everything Data" - Business Intelligence, Advanced Analytics, Big Data & Cloud, and Web & Mobile Applications. Founded in 2000, we have been a trusted partner to our customers in over 15 industry verticals and functional areas, with operations spanning Europe, Middle East, the...

  • Senior Data Engineer

    2 weeks ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Durlston Partners Full time

    Role: Senior Data EngineerClient: Quantitative Trading FirmSalary: $300,000 (tax-free)Location: Abu Dhabi, UAE (full relocation assistance provided)Durlston Partners is partnering with a leading Quant Trading firm that leverages advanced data and technology to execute market-beating strategies. They are seeking a Data Engineer to design, build, and scale...

  • Senior Data Engineer

    14 hours ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Auxo Talent Full time

    We are seeking aSenior Robotics Data Engineer (Humanoid)to design, build, and operate large-scale data pipelines that power humanoid robots and embodied AI systems. This role sits at the intersection of robotics, data engineering, and machine learning, with direct impact on real-world robotic intelligence.You will own end-to-end robotics data...

  • hr data lead

    1 week ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Smart Research System Full time

    ROLE & RESPONSIBILITY:The HR Data Lead is accountable for the quality, integrity, governance, and consistency of HR data across enterprise platforms, with primary ownership of Oracle Fusion HCM. The role serves as the central authority for HR data standards, definitions, validation controls, and remediation, ensuring data is reliable for reporting,...

  • Data Engineer

    6 days ago

    Abu Dhabi, Abu Dhabi, United Arab Emirates Adecco Full time

    Role OverviewWe are seeking a highly skilledData Engineerto design, build, and optimize scalable data platforms that power analytics, regulatory reporting, and AI-driven innovation. The ideal candidate has strong cloud experience, deep understanding of distributed data systems, and a passion for building reliable, high-performance data...