Data Engineer

A large Retail and Consumer Finance Concern in Joburg seeks the technical expertise of a Data Engineer whose core role will be to drive, design & build scalable ETL systems. These systems will be for a Big Data warehouse where you will implement robust & trustworthy data to support high performing ML algorithms, predictive models and support real-time data visualisation requirements across the organisation to enable self-help analytics. The ideal candidate will require a 3-year IT related Degree, have 5-10 years' experience designing and developing data warehouses according to the Kimball methodology, is adept at design and development of ETL processes, SQL development experience and preferably SAS Data Studio and AWS experience. You must also be able to ingest/output CSV, JSON and other flat file types and any related data sources, have Retail/ Financial Services and Logistics experience and proficiency in Python or R or a willingness to learn.
Duties:

Systematic solution design of the ETL and data pipeline in line with business user specifications -



Ensure highest data quality assurance, data accuracy and data completeness through regular and in-depth review and testing of work.


Create easily understandable technical documentation that are kept up to date.


Conduct data design, database architecture, metadata and repository creation activities and tasks as required by business stakeholder.


Translate business needs into long-term architecture solutions.


Define, design and build dimensional databases.


Design the ETL pipelines.


Responsible for developing data warehousing blueprints, evaluating hardware and software platforms and integrating systems.


Evaluate reusability of current data for additional analyses.


Conduct data cleaning to rid the system of old, unused, or duplicate data.


Review object and data models and the metadata repository to structure the data for better management and quicker access.


Determine processes to ensure execution of relevant data application requirements for various business needs.


Utilise relevant templates that outlines the requirements for each step within the data modelling journey.


Conduct testing and quality control of databases to ensure accurate and appropriate use of data.


Initiate and successfully motivate improved ways of operating.



Develop and implement ETL pipelines aligned to the approved solution design -



Enhance and maintain existing ETL frameworks in line with agreed design patterns and internal governance standards to improve the EDW product offering and to remain scalable.


Implement the ETL pipeline in a timely manner.


Utilise most accurate data source to remodel into a set of data that is understandab