Data Engineer

1 week ago


Dubai, Dubai, United Arab Emirates JobItUs Full time
Work Experience: 5.0 Year(s) To 12.0 Year(s)

Function:
IT Software - System Programming Notice Period : Early joiners preferred (Maximum 30 days)

Job Description:
We are looking for a highly skilled and motivated Spark Data Engineer to join our team.

The ideal candidate will have a strong background in Apache Spark, data ingestion, data processing, and data integration, and will be responsible for developing and maintaining our dynamic data ingestion framework using the Spark framework.

The candidate should have expertise in building scalable, high-performance, and fault-tolerant data processing pipelines using Spark, and be able to optimize Spark jobs for performance and scalability.

The candidate should also have experience in designing and implementing data models, handling data errors, implementing data quality and validation processes, and integrating Spark applications with other big data technologies in the Hadoop ecosystem.


Responsibilities:


Develop and maintain a dynamic data ingestion framework using Apache SparkImplement data ingestion pipelines for batch processing and real-time streaming using Spark's data ingestion APIsDesign and implement data models using Spark's DataFrame and Dataset APIsOptimize Spark jobs for performance and scalability, including caching, broadcasting, and data partitioning techniquesImplement error handling and fault tolerance mechanisms to handle data errors, processing failures, and system failures in Spark applicationsImplement data quality and validation processes, including data profiling, data cleansing, and data validation rules using Spark's data processing and data validation APIsIntegrate Spark applications with other big data technologies in the Hadoop ecosystem, such as Hadoop, Hive, HBase, Kafka, and othersEnsure data security by implementing data encryption, data masking, and data access controls in Spark applicationsUse version control systems, such as Git, for source code management, and implement DevOps practices, such as continuous integration, continuous delivery, and automated deployments, in Spark application development workflowsQualifications:Bachelor's or Master's degree in Computer Science, Data Engineering, or a related fieldStrong proficiency in Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib, with multiple production developments and deployment experience.

Proficiency in either Scala or Python programming languages, with knowledge of functional programming conceptsExperience in developing and maintaining dynamic data ingestion frameworks using SparkExperience in data processing, data integration, and data modeling using Spark's DataFrame and Dataset APIsKnowledge of performance optimization techniques in Spark, including caching, broadcasting, and data partitioningExperience in implementing error handling and fault tolerance mechanisms in Spark applicationsKnowledge of data quality and validation techniques using Spark's data processing and data validation APIsFamiliarity with other big data technologies in the Hadoop ecosystem, such as Hadoop, Hive, HBase, Kafka, etc.

Experience in implementing data security measures in Spark applications, such as data encryption, data masking, and data access controlsStrong problem-solving skills and ability to troubleshoot and resolve issues related to Spark applicationsProficiency in using version control systems, such as Git, and implementing DevOps practices in Spark application development workflowsExcellent communication and collaboration skills, with the ability to work effectively in a team-oriented environmentIs a leading IT Solutions and Consulting Company Apply NowInterested candidates are requested to apply for this job.

Recruiters will evaluate your candidature and will get in touch with you.#J-18808-Ljbffr
  • Data Scientist

    1 week ago


    Dubai, Dubai, United Arab Emirates Open Data Science Full time

    Tabby is a company that transforms the way people manage their finances, offering them the opportunity to have greater control over their spending and optimize their savings. With more than 4,500,000 active users trusting Tabby, the platform has become a popular choice for individuals looking to reshape their financial habits.Collaborating with over 15,000...

  • Data Scientist

    1 week ago


    Dubai, Dubai, United Arab Emirates Open Data Science Full time

    Tabby creates financial freedom in the way people shop, earn and save by reshaping their relationship with money. Over 4,500,000 active users choose Tabby to stay in control of their spending and make the most out of their money.Over 15,000 global brands and small businesses, including H&M, Adidas, IKEA, SHEIN, noon, and Bloomingdale's use Tabby's technology...


  • Dubai, Dubai, United Arab Emirates The Chemical Engineer Full time

    £65 to £110 K + housing allowanceInsurance riskCompanyOur client is a leading player in the International high hazard Insurance sector where it is involvedIn cover associated with a variety of sectors including oil and gas, refining, petrochemicals, LNG, metals, and mining, maritime, mass- transport and so on. The organisation's Gulf hub is enjoying...


  • Dubai, Dubai, United Arab Emirates Crayon Data Full time

    The job posting is outdated and position may be filled Crayon Data is a fast-growing big data and AI company, founded in 2012 with a vision to simplify the world s choices. Crayons is the world s first AI engine that engages customers through taste-led customization and increases conversion rates for enterprises.Who are we looking for?In this role, you ll...


  • Dubai, Dubai, United Arab Emirates Value Driven Data Full time

    Job description - Confirmed Data-ML EngineerVDD is an AI-powered Dubai and US-based startup that specializes in data science and digital technology projects and products. Our goal is to democratize technology by translating technical language into user-friendly resources and solutions. VDD employs a unique approach, enhancing enterprise profitability and...


  • Dubai, Dubai, United Arab Emirates Alpha Data Full time

    Site Engineer Jobs In Dubai UAE 2023 | Alpha DataFor More Top Organization Jobs Click HereLooking for a Site Engineer in Dubai, United Arab Emirates (UAE)Job Qualifications and Experience:The suitable candidate must have a Bachelor's Degree in Civil Engineering or a related fieldMinimum 5+ years of proven experience as a site engineer, preferably within the...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates SimplyApply Full time

    Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates SimplyApply Full time

    Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates SimplyApply Full time

    Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates UAE Limited Full time

    We're looking for a experienced Data Engineer to join our team at Farcana, on a full-time basisResponsibilities:Design, implement, and maintain vector and graph databases to support advanced data structures and retrieval systems.Develop and optimize RAG (Retriever-Answer Generator) models for efficient data querying and AI-driven answers.Ensure database...

  • Data Engineering

    1 week ago


    Dubai, Dubai, United Arab Emirates PepsiCo Full time

    Qualifications 9+ years of overall technology experience that includes at least 3+ years of hands-on software development, data engineering, and systems architecture. 3+ years of experience with Salesforce Cloud Technologies is a must. 3+ years of experience with Sales force Customer data modeling, data warehousing, and building high-volume ETL/ELT...

  • Data Engineering

    1 week ago


    Dubai, Dubai, United Arab Emirates PepsiCo Full time

    Qualifications 9+ years of overall technology experience that includes at least 3+ years of hands-on software development, data engineering, and systems architecture.3+ years of experience with Salesforce Cloud Technologies is must.3+ years of experience with Sales force Customer data modeling, data warehousing, and building high-volume ETL/ELT pipelines.3+...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates dubizzle Full time

    Data Engineer | Intern Dubizzle Dubaidubizzle is the UAE'S favorite marketplace to buy, sell and find anything. If you can search for it, you can find it on dubizzle From Batmobiles to Dinosaur bones to a Boeing 747Bayut & dubizzle – The Arab World's only Homegrown Unicorn Business is now inviting applications for internships of 6 months duration in our...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates Salt Full time

    REQUIREMENTS:5+ years mastering Azure Data Factory, Databricks, and Power BI.Degree in Computer Science, IT, or a related field.Python expert, especially in data manipulation and notebook-based analytics.A deep understanding of cloud data security protocols and best practices.Experience optimizing cloud-based data platforms for peak performance.A track...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates RemoteWorker US Full time

    JOB TITLE: Data EngineerLOCATION: RemoteTHE ROLE:As a member of the Data Engineering team, you will create tools, pipelines, and systems that enable the business to reliably operate at scale, gain mission critical insight, and power engaging data products for our customers. You will be building important, large-scale observability into problems that are...

  • Lead Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates PrecisionHire Solutions Full time

    Information Technology & Other, 5+ Years of experience, Degree or Honours (12+3 or equivalent). Degree in a relevant field such as Computer Science, Computational Mathematics, Computer Engineering, or Software Engineering. Specialization or electives in a Data & Analytics field (e.g. Data Warehousing, Data Science, Business Intelligence) a nice-to-have 5+...


  • Dubai, Dubai, United Arab Emirates Michael Page International (UAE) Full time

    We have been made aware of a global phishing campaign where employees from companies are being impersonated. We are confident that no PageGroup system has been breached. Find out how to protect yourself Must have extensive experience within the Banking or Financial Services sector.Opportunity to lead and develop a skilled data engineering teamAbout Our...

  • Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates RemoteWorker US Full time

    JOB TITLE: Data EngineerLOCATION: RemoteTHE ROLE: As a member of the Data Engineering team, you will create tools, pipelines, and systems that enable the business to reliably operate at scale, gain mission critical insight, and power engaging data products for our customers. You will be building important, large-scale observability into problems that are...

  • Lead Data Engineer

    1 week ago


    Dubai, Dubai, United Arab Emirates Precision Hire Solution Full time

    Job Purpose: Connect the world through our hub in Dubai by joining the Emirates Group IT Team. As pioneers in aviation innovation, we are looking for exceptional professionals to strengthen our position in the industry as a Lead Data Engineer.In this role you will be responsible for:Lead technical design, development, and deployment for solutions within a...

  • Data Engineering Lead

    3 weeks ago


    Dubai, Dubai, United Arab Emirates Al Futtaim Private Company LLC Full time

    Job Requisition ID: 160139Established in the 1930s as a trading business, Al-Futtaim Group is a diversified and progressive regional company headquartered in Dubai, United Arab Emirates. Structured into five divisions: automotive, financial services, real estate, retail, and healthcare; with over 35,000 employees across 20+ countries in the Middle East,...