Category: Development and implementation

Subcategory: Data and analytics

Data engineering - DENG

Designing, building, operationalising, securing and monitoring data pipelines, stores and real-time processing systems for scalable and reliable data management.

Levels:

Level 2:
Description: Assists in developing and implementing data pipelines and data stores. Performs administrative tasks to provide data accessibility, retrievability, security and protection. Supports the monitoring of data pipeline operations, identifying issues and escalating as needed. Participates in data migration and conversion tasks under routine supervision.

Level 3:
Description: Follows standard approaches and established design patterns to create and implement simple data pipelines and data stores to acquire and prepare data. Applies data engineering standards and tools to create and maintain data pipelines and perform extract, transform and load (ETL) processes, incorporating security and data integrity practices. Contributes to data migration and conversion projects, ensuring data integrity and consistency. Conducts routine data quality checks and remediation.

Level 4:
Description: Designs, implements and maintains complex data engineering solutions to acquire and prepare data. Creates and maintains data pipelines to connect data across data stores, applications and organisations. Builds in compliance with data governance and security standards. Supports the development of continuous integration and deployment practices. Monitors and optimises pipeline performance and scalability. Conducts complex data quality checking and remediation. Leads data migration and data conversion activities.

Level 5:
Description: Plans and drives the development of data engineering solutions, balancing functional and non-functional requirements. Monitors application of data standards, architectures and security, ensuring compliance and scalability. Develops and promotes continuous integration, deployment and monitoring practices. Contributes to organisational policies, standards and guidelines for data engineering.

Level 6:
Description: Leads the selection and development of data engineering methods, tools and techniques. Develops organisational policies, standards and guidelines for the development and secure operation of data services and products. Ensures adherence to technical strategies and architectures. Plans and leads data engineering for strategic, high-impact, large and complex programmes ensuring alignment with organisational objectives and industry practices.