Senior Data Engineer (Spark & Python Specialist)

Role:
Our client is seeking a Senior Cloud Data Engineer to join their Engineering team at the company. In this role, you will be a key technical contributor responsible for building, optimizing, and maintaining high-performance data processing engines.

You will apply your deep expertise in Spark Best Practices to develop modular, portable PySpark applications, ensuring the companies platform remains provider-agnostic while operating within modern environments like Microsoft Fabric.

As an experienced member of the Data Engineering team, you will focus on refactoring legacy logic into scalable Python-centric solutions, optimizing the companies data lakehouse architecture, and contributing to the overall technical excellence of their data platform.

Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure their Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines using Python, PySpark, and Delta Lake/Parquet, adhering to their strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of the companies platform by refactoring complex SQL-based ETL into modular, testable, and maintainable Python libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture (Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centric orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers in PySpark best practices, and contribute to the development of automated testing frameworks (Pytest) to ensure high-quality, reliable data delivery.

Required Skills and Experience

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics &a
  
  https://www.executiveplacements.com/Jobs/S/Senior-Data-Engineer-Spark--Python-Specialist-1271980-Job-Search-03-16-2026-04-10-42-AM.asp?sid=gumtree