Senior Specialist - Data Engineering (Azure Databricks)

Job description
Data Pipeline Development Building scalable ETLELT workflows using Azure Data Factory Databricks Notebooks and Apache Spark for ingestion transformation and loading of structured and unstructured data

Data Storage Management Designing and managing storage solutions like Azure Data Lake Storage Azure SQL Database and Azure Blob Storage to ensure performance scalability and cost-effectiveness

Big Data Processing Leveraging Azure Databricks with Spark for distributed data processing machine learning model training and advanced analytics

Integration Collaboration Working closely with data scientists to prepare datasets for AIML models and integrating outputs into business applications

• Optimization Monitoring Tuning Spark jobs optimizing cluster configurations and monitoring pipelines for reliability and cost control
• Security Compliance Implementing data governance role based access control and compliance with organizational and regulatory standards

As per Company