Data Engineer (JHB)
IT – Software Development
Johannesburg – Gauteng
ENVIRONMENT:
A large Retail and Consumer Finance Concern in Joburg seeks the technical expertise of a Data Engineer whose core role will be to drive, design & build scalable ETL systems. These systems will be for a Big Data warehouse where you will implement robust & trustworthy data to support high performing ML algorithms, predictive models and support real-time data visualisation requirements across the organisation to enable self-help analytics. The ideal candidate will require a 3-year IT related Degree, have 5-10 years’ experience designing and developing data warehouses according to the Kimball methodology, is adept at design and development of ETL processes, SQL development experience & preferably SAS Data Studio and AWS experience. You must also be able to ingest/output CSV, JSON and other flat file types and any related data sources, have Retail/ Financial Services and Logistics experience and proficiency in Python or R or a willingness to learn.
DUTIES:
Systematic solution design of the ETL and data pipeline in line with business user specifications –
- Ensure highest data quality assurance, data accuracy and data completeness through regular and in-depth review and testing of work.
- Create easily understandable technical documentation that are kept up to date.
- Conduct data design, database architecture, metadata and repository creation activities and tasks as required by business stakeholder.
- Translate business needs into long-term architecture solutions.
- Define, design and build dimensional databases.
- Design the ETL pipelines.
- Responsible for developing data warehousing blueprints, evaluating hardware and software platforms and integrating systems.
- Evaluate reusability of current data for additional analyses.
- Conduct data cleaning to rid the system of old, unused, or duplicate data.
- Review object and data models and the metadata repository to structure the data for better management and quicker access.
- Determine processes to ensure execution of relevant data application requirements for various business needs.
- Utilise relevant templates that outlines the requirements for each step within the data modelling journey.
- Conduct testing and quality control of databases to ensure accurate and appropriate use of data.
- Initiate and successfully motivate improved ways of operating.
Develop and implement ETL pipelines aligned to the approved solution design –
- Enhance and maintain existing ETL frameworks in line with agreed design patterns and internal governance standards to improve the EDW product offering and to remain scalable.
- Implement the ETL pipeline in a timely manner.
- Utilise most accurate data source to remodel into a set of data that is understandable to the end user.
- Understand data structures to deliver data sets that are deliver to exact requirements of end user brief.
- Ensure data is precise and is benchmarked and validated against financial records.
- Utilise consistent data sources which result in one version of the truth.
- Deliver on standard data marts that can be utilised for reporting and analysis which is well documented and understood by business users.
- Translate Meta data into explanatory reports and visuals for easy understanding to end user.
- Perform data pre-processing which includes data manipulation, transformation, normalisation, standardisation, visualisation and derivation of new variables/features, as applicable to developing specific algorithms or models.
Ensure data governance and data quality assurance standards are upheld –
- Facilitate an understanding of data sources to ensure governance, procedures and standards are upheld.
- Build data quality metrics and conduct data validation testing.
- Follow the IT governance process when implementing a change to ensure governance standards and protocols are followed.
- Work close with business to understand business processes and standards in order to develop data quality assurance metrics.
- Build exception reports to help identify data quality problems.
- Provide feedback to business owners on identified problems to ensure quality of data is rectified.
Deal with customers in a customer centric manner –
- Utilise specialist knowledge to explain the data and transfer the understanding to business end user.
- Conduct training and upskilling on new reports and or self-service analytics platforms to relevant stakeholders.
- Communication to stakeholders to keep them abreast of current developments within the function and to manage expectations.
- Apply the standards set out in all legislation, policy and procedure that effects the customer.
- Deal effectively and timeously with customer complaints and ensure that complaints are resolved or escalated in line with agreed standards.
- Deliver services and products to the customer within the parameters of the agreed SLA.
- Recognise and celebrate customer centric behaviour within others.
- Manage colleagues and customers’ expectations and communicate appropriately within the parameters of agreed SLAs.
- Act in a customer centric manner that is in line with the service code and core ideology in order to meet and exceed the requirements of internal and external customers.
Effective Self-Management and Teamwork –
- Actively and consistently maintain high standards of professionalism in all aspects of personal presentation and delivery.
- Apply knowledge of the organizational systems, structures, policies and procedures to achieve results.
- Demonstrate initiative in follow through to ensure that personal quality and productivity standards are consistently and accurately maintained.
- Provide appropriate resolution for tasks or deadlines not met.
- Support and drive the business’ core values.
- Maintain a positive attitude and respond openly to feedback.
- Take ownership for driving own career development.
- Show commitment to teamwork and a willingness to go the extra mile to achieve team objectives.
REQUIREMENTS:
Qualifications –
- 3-Year IT related Degree.
- Post-graduate qualification (advantageous).
Experience/Skills –
- 5-10 Years’ experience and understanding in designing and developing data warehouses according to the Kimball methodology.
- Adept at design and development of ETL processes. SQL development experience, preferably SAS Data Studio and AWS experience T
- he ability to ingest/output CSV, JSON and other flat file types and any related data sources.
- Proficient in Python or R or willingness to learn.
- Experience within Retail, Financial Services and Logistics environments.
- Data Architecture, Data Modelling and Data Pipelining.
- Solutions Architecture.
ATTRIBUTES:
- Adapting and responding to change.
- Presenting and communicating information.
- Stakeholder Management.
- Excellent written and verbal communication skills.
- Analysis and Judgement.
- Personal resilience.
- Achieving Personal Work Goals and Objectives.
- Customer orientation.
- Team player.
- Excellence orientation.
- Responsibility and Accountability.
- Innovative.
- Learning and Researching.