Big Data Engineer (Spark + Scala/Python)

We are seeking an experienced Big Data Engineer to design, develop, and optimize large-scale data platforms. The ideal candidate should possess strong expertise in Big Data technologies, distributed computing, real-time data processing, and modern data engineering practices. The role requires close collaboration with business stakeholders, data architects, data scientists, and engineering teams to build scalable and high-performance data solutions.

• Key Responsibilities
• Design, develop, and maintain enterprise-scale data ingestion and processing pipelines.
• Develop batch and real-time streaming applications using Apache Spark, Scala, Python, Kafka, Hadoop, and NiFi.
• Build scalable ETL/ELT frameworks to process structured, semi-structured, and unstructured data.
• Design and implement real-time data streaming solutions using Kafka and Spark Streaming.
• Develop and manage data integration workflows using Apache NiFi.
• Optimize data processing jobs for performance, scalability, and reliability.
• Collaborate with Data Architects and Data Modelers to implement data models and platform architecture.
• Implement data quality, reconciliation, monitoring, and governance frameworks.
• Troubleshoot production issues and perform root cause analysis.
• Ensure compliance with security, governance, and data management standards.
• Participate in code reviews, architecture discussions, and technical design sessions.