Menu Close

Data Engineer

IT – Analyst, Data Management
Cape Town – Western Cape

ENVIRONMENT:
A Specialist IT Service Provider in Durbanville seeks the technical expertise of a highly meticulous, analytical & solutions-driven Data Engineer to deliver accurate data reliably within the required processing time, ready for processing by analytics applications. This role will suit an all-round Data Engineer experienced in every step of data flow, starting from configuring data sources to integrating analytical tools. You will be expected to contribute to architecting, building, testing, and maintaining the data platform as a whole. You must possess a Bachelor’s Degree in Computer Science/Engineering or similar discipline, have 5-10 years’ suitable work experience, Intermediate to Advanced SQL optimization and developing ETL strategies, Docker, Kubernetes, API integration and development using Python – Fast API, Flask, Bash, Confluent Kafka/Kinesis, AWS, Git, knowledge of database and data warehousing principles (e.g., OLAP, Data Marts, Star Schema, lambda/kappa architectures), Agile, Kanban/Scrum and be able to conduct Systems Analysis and prepare requirement specifications concerning data-related business processes and systems.
 
DUTIES:
You will work across multiple business teams to implement new and improve existing systems for –
    • Extracting data from current sources.
    • Data storing/transition for all data gathered for analytical purposes.
    • Transformation: Cleaning, structuring, and formatting the data sets to make data consumable for processing or analysis.
 
  • Lead/Contribute to designing the architecture of a data platform.
  • Develop data systems tools, customize and manage integration tools, databases, warehouses, and analytical systems.
  • Data pipeline maintenance/testing.
  • Machine Learning algorithm deployment. Machine Learning models designed by their Data Scientists, deployed into production environments, managing computing resources and setting up monitoring tools.
  • Manage data and meta-data storage, structed for efficiency, quality and performance.
  • Track pipeline stability. Monitor the overall performance and stability of the systems.
  • Keep track of related infrastructure costs and manage these as efficiently as possible, continuously find the balance between performance and cost.
 
REQUIREMENTS:
Qualifications –
  • A Bachelor’s Degree in Computer Science or Engineering related field.
 
Experience/Skills –
  • 5-10 Years relevant experience.
  • Intermediate to advanced SQL optimization, developing ETL strategies.
  • Intermediate to advanced knowledge of database and data warehousing principles (e.g., OLAP, Data Marts, Star Schema, lambda/kappa architectures).
  • Experience with Agile or other development methodologies – Agile, Kanban or Scrum.
  • Ability to conduct Systems Analysis and prepare requirement specifications concerning data-related business processes and systems. 
 
Preferred Technical Experience (not essential) –
  • Implementing data pipelines using cloud infrastructure and services.
  • Implementing event-based systems using tools like Confluent Kafka or Kinesis.
  • Knowledge of CDC pipelines (E.g., Bin logs, Debezium, AWS Database Migration Service (DMS)).
  • API integration and development using Python – Fast API, Flask.
  • DevOps: Docker (ECS), Kubernetes (EKS), Spark clusters, etc.
  • Database analysis, design & administration.
  • SQL Query optimization and data architecture improvements.
  • DB: PostgreSQL, MS-SQL.
  • ETL: Python, Bash.
  • Infrastructure: AWS (Kinesis, API Gateway, S3, DMS).
  • Dev Tools: Git, Docker, Elastic Container Service (ECS), Elastic Kubernetes Service (EKS).
  • OS: Ubuntu, Windows Server.
  • AWS user experience, AWS cloud, Redshift, Glue, Athena
 
Advantageous –
  • Knowledge of working with stream data pipeline frameworks or solutions, e.g., Apache Flink, Beam, Dataflow, Databricks etc.
  • Knowledge of cloud data platforms.
  • Experience within the Automotive or manufacturing industry.
 
ATTRIBUTES:
  • Dynamic, driven by results. Enjoy getting things done.
  • A Can-do attitude is highly desirable.
  • Self-starter with the ability to successfully plan, organise, and execute assigned initiatives with minimal guidance and direction.
  • Strong listening, problem-solving and analytical skills.
  • Willing to take the initiative.
  • Excellent written and verbal communication skills.
  • Exhibits close attention to detail and is a champion for accuracy.
  • High level of Integrity.
  • Proven ability to meet deadlines, ability to multi-task effectively.
The ability and willingness to do a variety of different tasks.