Senior Data Engineer

Specific Accountability :-Lead the design, build, and optimization of batch and real-time data pipelines. Own data model design, ingestion frameworks, transformation logic, and platform reliability. Ensure high standards for data quality, lineage, governance, security, and compliance. Drive performance tuning, cost optimization, monitoring, and incident resolution across data platforms. Translate business and analytics needs into scalable technical solutions and delivery plans. Guide code reviews, technical standards, and release practices for data engineering work. Mentor team members, allocate work, and provide technical leadership on complex initiatives. Collaborate with value delivery owners, architects, analysts, scientists, and application teams to enable trusted data productsDesired Candidate ProfileExperience level: 10+ years hands-on data engineering with production pipeline ownership; at least one platform or system owned end to end. Data transformation and modelingdbt (data build tool): models, tests, macros, incremental models, project structureData modeling: dimensional modeling, star schema, medallion architecture (raw / curated / presentation, or bronze / silver / gold)Data warehousing and lakehouse architecture conceptsData warehousing and lakehouse architecture conceptsOrchestrationApache Airflow: DAG design, task dependencies, triggers, sensors, parameterized runs, failure handling, schedulingWorkflow orchestration and monitoring of production pipelinesWorkflow orchestration and monitoring of production pipelinesData ingestion and loadingBatch data ingestion and ETL / ELT pipeline developmentLoad patterns: full load, incremental load, upsert / merge, change data capture (CDC)Multi-source ingestion: SFTP, REST APIs, relational databases, file-based ingestionData quality, data validation, schema evolution handlingQuery engine, storage, and table formatsDremio (or equivalent lakehouse query engine: Trino, Presto, Starburst, Athena)Apache Iceberg (or equivalent open table format: Delta Lake, Apache Hudi)S3 / S3-compatible object storage (AWS S3, MinIO, Dell ECS)Partitioning, table layout, and query performance tuningProgramming and databasesPython (pipeline development, automation, scripting)Advanced SQL, query optimization, performance tuningPostgreSQL and/or SQL Server (relational databases)Non-relational / NoSQL databases (must-have, not just exposure)The platform ingests from and operates alongside several non-relational stores. The candidate must be able to work with them as a data engineer, meaning read from them, model their data into the lakehouse, and reason about their consistency and performance characteristics. Screen for hands-on experience with at least two of the following families: Key-value / in-memory stores: Redis (caching, queues, state; TTL and eviction behaviour; using Redis as a pipeline coordination or lookup layer)Document databases: MongoDB, Couchbase, or DocumentDB (nested JSON documents, schema-on-read, flattening documents into relational / lakehouse tables)Wide-column stores: Cassandra, ScyllaDB, or HBase (partition-key design, read patterns, extracting at scale)Search / analytics engines: Elasticsearch or OpenSearch (index design, querying, extracting data out of an index for analytics)Data formats and semi-structured dataColumnar formats: Apache Parquet (required), ORC; row formats: AvroText and semi-structured formats: JSON, JSON Lines, CSV / delimited files, XML; encoding, delimiter, and quoting edge casesCompression codecs (Snappy, gzip, zstd) and their effect on storage and queryHandling nested and evolving schemas end to end (schema inference, drift detection, schema evolution: column add / delete / type change)Distributed processingApache Spark / PySpark (DataFrame API, partitioning, joins, skew handling, writing Iceberg / Parquet); or equivalent distributed engineUnderstanding of parallelism, shuffles, memory tuning, and cost of distributed jobsScripting, Linux, and connectivityBash / shell scripting and Linux command line (the deployment and operational tooling is script-based)SFTP / SSH: key management, automated transfers, file-based handoffs (verify / result files, archiving, retries)TLS certificates and secure connections (understanding certificate verification; never disabling it in production)Connectivity troubleshooting: DNS, firewalls, proxies, connection timeouts, service-to-service networking inside Kubernetes / OpenShiftYAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests