Sr. Data Engineer - PySpark, Python & Cloudera (CDP)
GSS Group
We are looking for an experienced Senior Data Engineer with strong expertise in PySpark, Python, and Cloudera Data Platform (CDP) to join a high-performing Data Engineering team supporting enterprise-scale Digital Products & Transaction Banking initiatives.
The ideal candidate will have extensive experience designing, building, and optimizing large-scale data pipelines within modern Big Data ecosystems. This role requires strong technical expertise in distributed data processing, cloud-native data platforms, data quality, and enterprise data engineering best practices.
The successful candidate will work closely with Data Architects, Data Scientists, Analytics teams, Product Owners, and Business Stakeholders to deliver scalable, secure, and high-performance data solutions that power business intelligence, analytics, and machine learning initiatives.
Key Responsibilities
- Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python .
- Build robust batch and distributed data processing solutions using Cloudera Data Platform (CDP) .
- Develop and optimize ETL/ELT pipelines for structured and unstructured enterprise datasets.
- Design scalable data ingestion frameworks from multiple enterprise data sources.
- Ensure high data quality, integrity, governance, and availability across enterprise platforms.
- Perform data profiling, cleansing, transformation, and validation activities.
- Optimize Spark jobs for performance, scalability, and resource utilization.
- Work closely with Data Scientists to prepare datasets for analytics and machine learning use cases.
- Collaborate with Product Owners, Business Analysts, Architects, and cross-functional engineering teams.
- Monitor, troubleshoot, and resolve production data pipeline issues.
- Participate in code reviews and implement engineering best practices.
- Create technical documentation and maintain data engineering standards.
- Support continuous improvement of enterprise data platforms and engineering processes.
- Participate in Agile/Scrum ceremonies including sprint planning, backlog grooming, stand-ups, and retrospectives.
Required Technical Skills
- 8–10+ years of experience in Data Engineering.
- Strong hands-on expertise in Python .
- Strong hands-on expertise in PySpark .
- Extensive experience with Cloudera Data Platform (CDP) .
- Strong understanding of distributed data processing frameworks.
- Experience building enterprise-scale ETL/ELT pipelines.
- Strong knowledge of Big Data technologies.
- Experience with Hadoop ecosystem technologies.
- Strong SQL programming and query optimization skills.
- Experience with data modelling and data transformation techniques.
- Knowledge of data quality, validation, and governance principles.
- Experience working with Git and version control systems.
- Understanding of CI/CD practices for data engineering.
- Strong understanding of REST APIs and data integration patterns.
- Experience working with structured, semi-structured, and unstructured datasets.
Nice to Have
- Experience working with cloud-native data platforms.
- Exposure to machine learning data pipelines.
- Knowledge of feature engineering and data preparation for AI/ML workloads.
- Experience with workflow orchestration tools.
- Exposure to containerization technologies such as Docker and Kubernetes.
- Experience with DevOps practices for Data Engineering.
Required Competencies
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management skills.
- Ability to work in fast-paced Agile delivery environments.
- Strong ownership mindset with focus on quality and delivery.
- Ability to collaborate effectively with business and technical stakeholders.
- Strong debugging and performance optimization capabilities.
- Ability to manage multiple priorities and deliver within tight timelines.
Preferred Domain Experience
- Banking
- Financial Services
- Digital Products
- Transaction Banking
- Enterprise Data Platforms
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Sr. Data Engineer - PySpark, Python & Cloudera (CDP) in Dubai vacancy
- ...understands and serves our users. As a Sr. Data Scientist (AI & ML) on the global AI hub... ...of data scientists and machine learning engineers. You will own the full ML lifecycle,... ...with reproducible analysis and modeling in Python. Solid statistical foundations,...
AED15 - 18 per month
...has partnered with TalentOne to find a Data Engineer (Software Engineer, Data) to join their... ...Kafka, Spark (Data bricks), Airflow, and Python to deliver scalable, production-ready data... ...). ~ Strong Python skills, including PySpark. ~ Hands-on experience with Kafka and...AED26000 - 28000 per month
...region. They are building out their backend engineering team to develop secure, high-performance,... ...backend solutions. Create scalable data models and contribute to defining backend... ...backend development experience, ideally with Python. Experience with Go or Node.js is also...- ...We are hiring a Senior Sales Engineer to join our team, driving technical sales and client relationships in the oil & gas or construction sector. Requirements: Bachelor’s degree in Mechanical Engineering 3–4 years of relevant experience in an oil & gas or construction...
- ...Job Title – Python Backend Developer Location: Dubai, UAE About the Role... ...best practices . Experience working with data-intensive platforms, product information... ...well-documented code following software engineering best practices. Translate business and...
- ...trade execution and API functionalities. Required Skills: Proven experience in Node.js or Python development. Strong understanding of SQL databases and Redis for data management. Experience with decentralized exchange systems and cross-chain protocols is a...
- ...Management to influence product development and roadmap priorities. Required Qualifications ~5–8+ years of experience in Systems Engineering, Network Engineering, Pro AV Engineering, or a similar customer-facing technical role. ~ Proven experience working with system...
- ...Job Title: Chef de Partie – Sushi Department: Kitchen Operations Reports To: Sous Chef / Head Chef Job Purpose: The CDP – Sushi is responsible for preparing high-quality sushi dishes, maintaining food safety standards, leading the sushi section, and ensuring...
- ...What you will do The Senior Sales Engineer, Security systems will be reporting directly to the Sales Manager, Security Systems - GCC. As part of a growing team, you will be responsible for delivering a sales target in UAE across the Security systems portfolio....
- ...Role Overview We are seeking a Senior Data Analyst to join our Quantitative Dealing... ...sustainable long-term solutions. Use Python and SQL to extract, transform, validate,... ...trading data structures. Experience with PySpark and large-scale data processing. Familiarity...
- ...enterprises require. Sitting above data platforms, cloud... ...as a true business performance engine delivering measurable value. For... ...tools, our team primarily uses Python and SQL, with occasional work in other languages (e.g., R, PySpark, JavaScript). An ideal candidate...
- ...individual’s growth through shared perspectives, open discussions, and a drive for curiosity. What would make you the best candidate? Python Developer with 4+ years exp to build functional and efficient server-side applications. Python Developer responsibilities...
- ...markets in the Middle East. As a Sr. Specialist in the Integrity... ...team's detection and enforcement engine running as one system. You will... ...re-tuning based on what the data shows. Document how detection... ...dashboards and writing lightweight Python for automation. ~ Analytical...
- .... You'll manage a team of roughly 5–7 engineers and QAs, growing them technically and professionally... ...of ML and AI work — turning data scientists' experimentation into reliable... ...systems exposure Deep technical command of Python and asynchronous programming Strong...
- ...mission and contribute meaningfully to our data-driven journey. Your Mission... ...related field at a leading business school or engineering school. You bring A strong... ...Excel (advanced) and ideally in Power BI, Python, or SQL A genuine interest in the...
- ...Position: Data Engineer Date Posted: July 7, 2026 Industry: Information Technology / Data Engineering Employment Type: Full Time... ...with workflow orchestration tools, database technologies, and Python-based data engineering solutions. You will collaborate with technical...
- ...logistics network. Role Overview As a Data Analyst, you will be responsible for... ...Requirements • Bachelor's Degree in Computer Engineering, Computer Science, Information Technology... ...creation. • Basic knowledge of Python for data analysis or process automation....
- ...About the Role Explore data, engineer features, and build machine learning models for real business challenges. Design, run, and evaluate... ...projects, and academic work are welcomed). ~ Strong Python skills with experience using pandas, NumPy, and scikit-learn....
- ...We're recruiting a Data Analyst for one of our clients, a globally recognized e-commerce organization. About Our Client:... ...process automation or workflow optimization using tools such as VBA, Python, or SQL Prior experience in corporate Learning & Development...
- ...About the Role We're looking for a Data Scientist to build the Murphy’s brain —... ...level outcomes. Collaborate with product, engineering and commercial teams to define solutions... ...ship as code, not notebooks. ~ Expert Python. Comfortable in a real production codebase...
- ...Contract (extendable to 3yrs) About the Role We are seeking an experienced Cohesity NetBackup Engineer with 10+ years of hands-on expertise in enterprise data protection, backup, and recovery solutions, along with strong working knowledge of Azure cloud...
$80000 - $120000 per month
...predictive and generative models, and turns data into decisions clients can act on.... ...classification, and forecasting ~ Partner with engineering to move models from notebook to... ...degree or equivalent experience ~ Strong Python (pandas, scikit-learn) and SQL ~...- ...will design, develop, and apply advanced data science, statistical, and machine learning... .... Working closely with customers, engineers, and product teams, you will build production... ...Applied AI with excellent proficiency in Python and SQL. Experience with machine learning...
AED7000 - 11000 per month
...Position: Data Network Engineer Date Posted: August 12, 2026 Industry: Information Technology / Networking Employment Type: Full Time Experience: Not Specified (Relevant hands-on experience in Data Networking required) Qualification: Bachelor’s Degree...- ...Description We are looking for a driven and commercially minded Sr Specialist Partnerships to secure, execute, and grow key regional... ...operational efficiencies Synthesize partnership performance data into clear insights and recommendations for leadership...
- ...platforms across the globe. Our network is truly international with engineering teams in 13 countries and operations in 40+ countries.... ...~Work within a cross-functional team of Product, Engineering, Data Science and Product Analytics. ~Build strong relationships with...
- ...stock market index. Job Description We are looking for a Sr. Specialist Finance Systems with deep expertise in Microsoft... ...technical teams to define and validate customizations, integrations, and data migration requirements, providing functional sign-off....
- ...office. These include frontend and backend engineers, AI research scientists, and others from... ...We're looking to hire for our Data side of our AI team at Speechify. This role... ...software development. ~ Proficiency with bash/Python scripting in Linux environments ~ Proficiency...
- ...join us on our success journey. Job Description As a Sr. Coordinator Campaigns, you’ll play an important role in supporting... ..., marketing, or similar roles Basic analytical skills and a data-driven approach to support campaign performance Proficiency in...
$155300 - $213500 per year
...Position: Sr. Executive Assistant Date Posted: August 16, 2026 Industry: Information Technology / Data & Artificial Intelligence Employment Type: Full-Time Experience... ...and track multiple workstreams across Engineering and Go-To-Market teams. • Plan executive...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Sr. Data Engineer - PySpark, Python & Cloudera (CDP). Be the first to apply!
