Senior Data Engineer (Spark & Python Specialist)

Location: Remote (South Africa)
Employment Type: Full-Time
Industry: Data Engineering | Cloud Platforms | Financial Services Technology

WatersEdge Solutions is partnering with a client to recruit a highly skilled Senior Data Engineer (Spark & Python Specialist). This is a strong opportunity for a technically advanced engineer who enjoys building scalable, high-performance data solutions in a modern cloud environment. The role is ideal for someone who thrives on optimisation, code-first engineering, and modernising legacy data logic into clean, portable, Python-centric solutions.

About the Role

As a Senior Data Engineer, you’ll serve as a key technical contributor within the engineering team, focused on building, maintaining, and optimising large-scale data processing engines. You’ll work extensively with Spark, PySpark, Delta Lake, and cloud-based lakehouse environments, helping shape a provider-agnostic platform with strong engineering standards, portability, and performance at its core.

Key Responsibilities

• Optimise Spark-based processing through best practices in memory management, shuffle tuning, and partitioning

• Build and maintain data pipelines using Python, PySpark, Delta Lake, and Parquet

• Refactor legacy SQL-based ETL logic into modular, testable, maintainable Python libraries

• Build and optimise medallion architecture layers across Bronze, Silver, and Gold

• Support code-first orchestration approaches using tools such as Airflow, Dagster, or Python-based wrappers

• Participate in code reviews and mentor junior engineers in PySpark best practices

• Contribute to automated testing frameworks using Pytest

• Work closely with data scientists, analysts, a
  

  https://www.executiveplacements.com/Jobs/S/Senior-Data-Engineer-Spark--Python-Specialist-1272159-Job-Search-03-16-2026-07-00-14-AM.asp?sid=gumtree