Data Engineer

3 weeks ago


dubai, United Arab Emirates JobItUs Full time
Work Experience: 5.0 Year(s) To 12.0 Year(s)
Function: IT Software - System Programming Notice Period : Early joiners preferred (Maximum 30 days)Job Description:We are looking for a highly skilled and motivated Spark Data Engineer to join our team. The ideal candidate will have a strong background in Apache Spark, data ingestion, data processing, and data integration, and will be responsible for developing and maintaining our dynamic data ingestion framework using the Spark framework. The candidate should have expertise in building scalable, high-performance, and fault-tolerant data processing pipelines using Spark, and be able to optimize Spark jobs for performance and scalability. The candidate should also have experience in designing and implementing data models, handling data errors, implementing data quality and validation processes, and integrating Spark applications with other big data technologies in the Hadoop ecosystem.Responsibilities:Develop and maintain a dynamic data ingestion framework using Apache SparkImplement data ingestion pipelines for batch processing and real-time streaming using Spark's data ingestion APIsDesign and implement data models using Spark's DataFrame and Dataset APIsOptimize Spark jobs for performance and scalability, including caching, broadcasting, and data partitioning techniquesImplement error handling and fault tolerance mechanisms to handle data errors, processing failures, and system failures in Spark applicationsImplement data quality and validation processes, including data profiling, data cleansing, and data validation rules using Spark's data processing and data validation APIsIntegrate Spark applications with other big data technologies in the Hadoop ecosystem, such as Hadoop, Hive, HBase, Kafka, and othersEnsure data security by implementing data encryption, data masking, and data access controls in Spark applicationsUse version control systems, such as Git, for source code management, and implement DevOps practices, such as continuous integration, continuous delivery, and automated deployments, in Spark application development workflowsQualifications:Bachelor's or Master's degree in Computer Science, Data Engineering, or a related fieldStrong proficiency in Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib, with multiple production developments and deployment experience.Proficiency in either Scala or Python programming languages, with knowledge of functional programming conceptsExperience in developing and maintaining dynamic data ingestion frameworks using SparkExperience in data processing, data integration, and data modeling using Spark's DataFrame and Dataset APIsKnowledge of performance optimization techniques in Spark, including caching, broadcasting, and data partitioningExperience in implementing error handling and fault tolerance mechanisms in Spark applicationsKnowledge of data quality and validation techniques using Spark's data processing and data validation APIsFamiliarity with other big data technologies in the Hadoop ecosystem, such as Hadoop, Hive, HBase, Kafka, etc.Experience in implementing data security measures in Spark applications, such as data encryption, data masking, and data access controlsStrong problem-solving skills and ability to troubleshoot and resolve issues related to Spark applicationsProficiency in using version control systems, such as Git, and implementing DevOps practices in Spark application development workflowsExcellent communication and collaboration skills, with the ability to work effectively in a team-oriented environmentIs a leading IT Solutions and Consulting Company Apply NowInterested candidates are requested to apply for this job.Recruiters will evaluate your candidature and will get in touch with you.#J-18808-Ljbffr

  • dubai, United Arab Emirates The Chemical Engineer Full time

    £65 to £110 K + housing allowanceInsurance riskCompanyOur client is a leading player in the International high hazard Insurance sector where it is involvedIn cover associated with a variety of sectors including oil and gas, refining, petrochemicals, LNG, metals, and mining, maritime, mass- transport and so on. The organisation’s Gulf hub is enjoying...


  • dubai, United Arab Emirates Alpha Data Full time

    Site Engineer Jobs In Dubai UAE 2023 | Alpha DataFor More Top Organization Jobs Click HereLooking for a Site Engineer in Dubai, United Arab Emirates (UAE)Job Qualifications and Experience:The suitable candidate must have a Bachelor’s Degree in Civil Engineering or a related fieldMinimum 5+ years of proven experience as a site engineer, preferably within...

  • Data Engineer

    1 week ago


    dubai, United Arab Emirates Tyde Digital Full time

    Tyde Digital Data Engineer PositionTyde Digital, a reputable digital transformation consultancy, is seeking a highly analytical and detail-oriented Data Engineer to join our fast-growing team. In this role, you will be responsible for developing, implementing, and optimizing data pipelines with multiple endpoints, while also serving as a data analyst to...

  • Data Engineer

    4 weeks ago


    Dubai, United Arab Emirates SimplyApply Full time

    Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are...

  • Data Engineer

    6 days ago


    dubai, United Arab Emirates SimplyApply Full time

    Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking...

  • Data Engineer

    2 weeks ago


    Dubai, United Arab Emirates ClickJobs.io Full time

    Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data...

  • Data Engineer

    2 weeks ago


    Dubai, United Arab Emirates ClickJobs.io Full time

    Do you love building andpioneering in the technology space? Do you enjoy solving complexbusiness problems in a fast-paced, collaborative, inclusive, anditerative delivery environment? At Capital One, you'll bepart of a big group of makers, breakers, doers and disruptors, whosolve real problems and meet real customer needs. We are seekingData Engineers...

  • Data Engineer

    4 weeks ago


    Dubai, United Arab Emirates SimplyApply Full time

    DataEngineerDo you love building and pioneeringin the technology space? Do you enjoy solving complex businessproblems in a fast-paced, collaborative, inclusive, and iterativedelivery environment? At Capital One, you'll be part of abig group of makers, breakers, doers and disruptors, who solve realproblems and meet real customer needs. We are seeking...

  • Data Engineer

    2 weeks ago


    Dubai, United Arab Emirates NTT Ltd Full time

    Academic qualifications and certifications: Relevant qualification (Bachelor’s in computer science, information technology or applied math)Relevant certification, e.g. Certified Data engineer, Certified Professional, etc) Experience required: Proven experience working with data engineering, data mining within a fast-paced environmentExperience in...

  • Data Engineering

    7 days ago


    Dubai, United Arab Emirates PepsiCo Full time

    Qualifications 9+ years of overall technology experience that includes at least 3+ years of hands-on software development, data engineering, and systems architecture.3+ years of experience with Salesforce Cloud Technologies is must.3+ years of experience with Sales force Customer data modeling, data warehousing, and building high-volume ETL/ELT pipelines.3+...

  • Data Engineer

    5 days ago


    Dubai, United Arab Emirates Salt Full time

    REQUIREMENTS:5+ years mastering Azure Data Factory, Databricks and Power BI.Degree in Computer Science, IT, or a related field.Python expert, especially in data manipulation and notebook-based analytics.A deep understanding of cloud data security protocols and best practices.Experience optimizing cloud-based data platforms for peak performance.A track record...


  • dubai, United Arab Emirates Michael Page International (UAE) Full time

    We have been made aware of a global phishing campaign where employees from companies are being impersonated. We are confident that no PageGroup system has been breached. Find out how to protect yourself Must have extensive experience within the Banking or Financial Services sector.Opportunity to lead and develop a skilled data engineering teamAbout Our...

  • Data Engineer

    4 weeks ago


    dubai, United Arab Emirates DuneSoft.ae Full time

    A data engineer is an IT worker whose primary job is to prepare data for analytical or operational uses. These software engineers are typically responsible for building data pipelines to bring together information from different source systems.OverviewYou will be responsible for: Designing and implementing Big Data systems (e.g. data warehouses, data lakes,...

  • Data Engineer

    3 weeks ago


    Dubai, United Arab Emirates Dautom Full time

    Client Introduction:In this role you will have the opportunity to work closely with one of our esteemed clients. This client is a global leader in the Real Estate industry known for its commitment to quality and innovation. They have chosen Dautom as their trusted partner for their upcoming projects.Job Title: Data EngineerJob Purpose:As a Data Engineer...

  • Senior Data Engineer

    3 weeks ago


    Dubai, Dubai, United Arab Emirates Charterhouse UAE Full time

    ​Charterhouse is working with an IT Consultancy, who is looking to hire a Senior Data Engineer within their international team.About the roleThe Senior Data Engineer will play a crucial part within the company working closely with the Head of Data, with the focus of building robust, real-time, world-class platform. The Engineer will be responsible for...

  • Data Engineer

    3 weeks ago


    dubai, United Arab Emirates Vngrs Full time

    ABOUT OneTeam:OneTeam, headquartered in Dubai, is a company that emerged as a spinoff after 15 years of VNGRS’s domain experience, provides software consultancy services to Gulf Region countries, and consists of independent consultants from all over the world.At OneTeam;We love what we’re doing, we don’t pick our favorite “clients/works”.We...

  • Big Data Engineer

    3 weeks ago


    dubai, United Arab Emirates AI Staffing Ninja Full time

    Job Description:We are seeking a talented and experienced Big Data Engineer to join our team. As a Big Data Engineer, you will be responsible for designing, implementing, and maintaining our company's big data infrastructure. You will collaborate closely with data scientists, analysts, and other stakeholders to ensure that our data pipelines are efficient,...

  • Data Engineer

    2 weeks ago


    Dubai, United Arab Emirates RemoteWorker US Full time

    JOB TITLE: Data EngineerLOCATION: RemoteTHE ROLE: As a member of the Data Engineering team, you will create tools, pipelines, and systems that enable the business to reliably operate at scale, gain mission critical insight, and power engaging data products for our customers. You will be building important, large-scale observability into problems that are...


  • Dubai, United Arab Emirates Moove Full time

    **About Moove** **Moove** is on a mission to make the world more productive and successful by redefining access to auto finance and vehicle ownership. Our starting point is a technology-enabled lending model to radically transform the availability of auto finance and vehicles for on-demand mobility marketplaces. By doing so, Moove is creating sustainable...

  • Data Engineer

    6 days ago


    dubai, United Arab Emirates Dautom Full time

    In this role you will have the opportunity to work closely with one of our esteemed clients. This client is a global leader in the Real Estate industry known for its commitment to quality and innovation. They have chosen Dautom as their trusted partner for their upcoming projects.Job Title: Data EngineerJob Purpose:As a Data Engineer specializing in Power BI...