Azure Databricks Lead

Exciting Opportunity Alert! HTC Global Services is hiring Azure Data bricks - Lead (Airline Project) for an 1-year extendable contract in Abu Dhabi, UAE (Onsite).

HTC Global Services - a leading CMM level 5 global provider of innovative IT and Business Process Services and Solutions since 1990 with headquarters in Troy, Michigan, USA.

• Lead and manage a team of Data Engineers, providing technical guidance, mentorship, and performance coaching.
• Design, develop, and optimize large-scale data pipelines using Azure Databricks and PySpark.
• Architect and implement modern data lake and lakehouse solutions using Azure Data Lake Storage (ADLS).
• Build and maintain batch and real-time streaming data pipelines using technologies such as Azure Event Hubs, Kafka, Spark Streaming, or Structured Streaming.
• Collaborate with Solution Architects, Data Scientists, Product Owners, and business stakeholders to define data requirements and technical solutions.
• Establish data engineering best practices, coding standards, CI/CD processes, and governance frameworks.
• Ensure high availability, scalability, security, and performance of data platforms.
• Drive Agile delivery, sprint planning, effort estimation, and technical reviews.
• Troubleshoot complex data engineering challenges and optimize platform performance.
• Evaluate emerging technologies and recommend improvements to the organization's data architecture.

Required Skills & Experience

Technical Skills

• Strong experience with Azure Data Engineering ecosystem.
• Expert-level proficiency in Azure Databricks.
• Strong hands-on experience with PySpark and Apache Spark.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Expertise in designing and implementing streaming data solutions using:
• Spark Structured Streaming
• Azure Event Hubs
• Apache Kafka
• Real-time data processing frameworks
• Experience with Azure Data Factory (ADF) for orchestration and ETL workflows.
• Strong SQL and data modeling skills.
• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of CI/CD pipelines, Azure DevOps, Git, and Infrastructure as Code.
• Experience working with large-scale enterprise data platforms.
• Strong experience with Databricks Unity Catalog for enterprise data governance, access control, metadata management, and secure data sharing across multiple workspaces.
• Hands-on experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables) for building scalable, reliable, and maintainable data ingestion and transformation pipelines.
• Expertise in implementing data quality checks, monitoring, lineage, and orchestration using Databricks-native pipeline capabilities.
• Experience working with Databricks Genie and AI-powered conversational analytics to enable business users to interact with enterprise data through natural language interfaces.
• Knowledge of integrating Generative AI capabilities within the Databricks ecosystem, including leveraging Databricks AI/BI, Genie Spaces, and Lakehouse AI features.
• Strong understanding of Delta Lake optimization techniques, including partitioning, clustering, performance tuning, and data lifecycle management.
• Experience in designing and implementing modern Lakehouse architectures using Databricks, Unity Catalog, Delta Lake, and Azure Data Lake Storage (ADLS).