This role works closely with the Head of Data Engineering to lead a team of Data Engineers who develop maintain and support our Data Lake and Data Warehouses. This is a hands-on role with about 60% hands-on work looking after our ETL frameworks and pipelines as well as new development and operations. 40% of the work will be mentoring and leading Data Engineers to ensure high quality of deliverables within the given standards and procedures.
What You Can Look Forward To:
Requirements
- Liaises with Key Stakeholders within the business.
- Facilitates Design workshops and meetings.
- Breaks work into sizable chunks and assigns to Data Engineers.
- Reviews completed requirements and ensures high quality standards of documentation.
- Facilitates Sign Off by technical stakeholders.
Development
- Provides peer review of Non Functional Requirements and ensures requirements can be met within the budget constraints.
- Has deep understanding of the Pickels Enterprise Data Platform and architecture.
- Develops good relationships with technical stakeholders and Datium peers.
- Designs Airflow and other frameworks.
- Documents the framework and framework components ready for implementation.
- Designs and documents pipelines in the context of the overall architecture.
- Develop and maintain Pipeline Development Standards.
- Develops ETL pipelines and complex transformation logic.
- Peer review Pipeline code and ensure standards are met.
- Provide guidance and coaching to Data Engineers.
- Develops complex components within the ETL framework.
- Assigns new features to a release cycle and ensures the development has been successfully completed.
- Manage the code deployment process and the promotion of code from one environment to the next (Code Merge)
- Provide Peer Review and approve Pull Requests
- Develops, documents and improves the coding standards.
- Ensures coding standards are followed.
Testing
- Ensures test cases are created and working as part of the sign off and review process.
- Organises release testing cycles.
- Reviews test results and provides updates and reports on any issues.
Technologies
- Develops complex SQL scripts, stored procedures.
- Enforces standards are met via reviews and other measures.
- Understands SSIS ETL pipelines and can update and maintain the processes.
- Ensures all ETL processes and data is correct and up to date.”
- Maintains security and access to the warehouse.
- Ensures data quality is in accordance to the SLAs.
- Expert skills in development of python scripts and use of standard libraries.
- Develops and maintains coding standards.
- Ensures all code is of high quality and adheres to standards and best practices.
- Expands the Airflow framework and facilitates changes.
- Reviews code changes and ensures all code is of high quality.
Operational
- Ensures system health checks are performed and any issues identified.
- Acts as an escalation point for any issue in the system and facilities error resolution.
- Ensures Regression Test Cases are developed and available in Operation Monitoring Dashboard.
- Ensures data quality is maintained at all times.
- Ensures excellent Security processes and standards are in place and well maintained.
- Develops and constantly improves Security policies.
- Ensures team members are following the standards and policies.”
- Ensures pipelines and ETL processes are up and running within the agreed SLAs.
- Triages System Errors and ensures any issues are resolved with the SLAs.
What We’d Like You To Have:
- 7+ years of working experience in Data Engineering and/or Software Engineering,
- 3+ years of experience leading small to medium teams (4-10 people)
- 3+ years of recent Python development experience or equivalent programming language (Java, C#, Scala)
- Advanced knowledge of Standard Query Language (SQL)
- Good written and verbal communication