Senior Data Engineer - Datium Insights

Senior Data Engineer

This role works closely with the Head of Data Engineering to lead a team of Data Engineers who develop maintain and support our Data Lake and Data Warehouses. This is a hands-on role with about 60% hands-on work looking after our ETL frameworks and pipelines as well as new development and operations. 40% of the work will be mentoring and leading Data Engineers to ensure high quality of deliverables within the given standards and procedures.

What You Can Look Forward To:

Requirements

  • Liaises with Key Stakeholders within the business.
  • Facilitates Design workshops and meetings.
  • Breaks work into sizable chunks and assigns to Data Engineers.
  • Reviews completed requirements and ensures high quality standards of documentation.
  • Facilitates Sign Off by technical stakeholders.

Development

  • Provides peer review of Non Functional Requirements and ensures requirements can be met within the budget constraints.
  • Has deep understanding of the Pickels Enterprise Data Platform and architecture.
  • Develops good relationships with technical stakeholders and Datium peers.
  • Designs Airflow and other frameworks.
  • Documents the framework and framework components ready for implementation.
  • Designs and documents pipelines in the context of the overall architecture.
  • Develop and maintain Pipeline Development Standards.
  • Develops ETL pipelines and complex transformation logic.
  • Peer review Pipeline code and ensure standards are met.
  • Provide guidance and coaching to Data Engineers.
  • Develops complex components within the ETL framework.
  • Assigns new features to a release cycle and ensures the development has been successfully completed.
  • Manage the code deployment process and the promotion of code from one environment to the next (Code Merge)
  • Provide Peer Review and approve Pull Requests
  • Develops, documents and improves the coding standards.
  • Ensures coding standards are followed.

Testing

  • Ensures test cases are created and working as part of the sign off and review process.
  • Organises release testing cycles.
  • Reviews test results and provides updates and reports on any issues.

Technologies

  • Develops complex SQL scripts, stored procedures.
  • Enforces standards are met via reviews and other measures.
  • Understands SSIS ETL pipelines and can update and maintain the processes.
  • Ensures all ETL processes and data is correct and up to date.”
  • Maintains security and access to the warehouse.
  • Ensures data quality is in accordance to the SLAs.
  • Expert skills in development of python scripts and use of standard libraries.
  • Develops and maintains coding standards.
  • Ensures all code is of high quality and adheres to standards and best practices.
  • Expands the Airflow framework and facilitates changes.
  • Reviews code changes and ensures all code is of high quality.

Operational

  • Ensures system health checks are performed and any issues identified.
  • Acts as an escalation point for any issue in the system and facilities error resolution.
  • Ensures Regression Test Cases are developed and available in Operation Monitoring Dashboard.
  • Ensures data quality is maintained at all times.
  • Ensures excellent Security processes and standards are in place and well maintained.
  • Develops and constantly improves Security policies.
  • Ensures team members are following the standards and policies.”
  • Ensures pipelines and ETL processes are up and running within the agreed SLAs.
  • Triages System Errors and ensures any issues are resolved with the SLAs.

What We’d Like You To Have:

  • 7+ years of working experience in Data Engineering and/or Software Engineering,
  • 3+ years of experience leading small to medium teams (4-10 people)
  • 3+ years of recent Python development experience or equivalent programming language (Java, C#, Scala)
  • Advanced knowledge of Standard Query Language (SQL)
  • Good written and verbal communication

How Our Application Process Works:

  • Include a cover letter with your application highlighting your relevant experiences and why you might be interested to join Datium.
  • Short phone screening interview.
  • Take-home technical tests.
  • Final interviews.

Application Details


    (Max upload file size 2 MB)
    Datium Insights

    Subscribe to Datium Insights

    Receive notifications when we release market updates and feature articles on the automotive industry