jobapplier

Senior Data Engineer

Optima Partners

Within our Data practice, you’ll work in mixed-capability pods alongside data engineers, analysts, data scientists and ML/AI engineers. You’ll bring technical depth, guide colleagues and turn data into practical solutions, clear insights and better customer experiences.

About Optima Partners

  • Who we are Optima Partners is a VC-backed, scaling AI and data consultancy headquartered in Edinburgh.
  • We bring together expertise in strategy, design, data science and engineering to help leading brands solve real business challenges.
  • We turn data into better customer experiences, smarter decisions and lasting business value.
  • Our clients include Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Eon Next, OVO and GSK.
  • We work alongside their teams to deliver practical solutions and share the skills to build on them.

Responsibilities

  • As a Senior Data Engineer within our Data practice, reporting to a Technical Lead, you will lead the design, build, testing and maintenance of reliable, scalable data pipelines and platforms for clients across diverse sectors.
  • Working in multi-disciplinary teams, you’ll take ownership of technical delivery and help shape solutions from requirements through to production.
  • This role suits an experienced data engineer who combines strong hands-on delivery with sound technical judgement, client-facing confidence and a commitment to developing others.
  • You will work alongside engineers, data scientists, architects and product specialists to make pragmatic design decisions, resolve complex challenges and improve engineering standards.
  • You’ll mentor colleagues, share knowledge and continue to develop your technical and consulting expertise.
  • Lead the design, build, testing and operation of scalable data pipelines using platforms such as Databricks, taking ownership of delivery quality and production readiness.
  • Design and develop data warehouses, data lakes and lakehouse solutions, making pragmatic architecture and data modelling decisions with architects and technical leads.
  • Build and optimise robust ETL/ELT workflows, embedding data quality checks, monitoring and controls to deliver reliable, accessible data.
  • Work directly with clients and multi-disciplinary colleagues to translate business requirements into technical designs, estimate work and communicate trade-offs, risks and progress.
  • Deliver well-modelled, documented datasets that support analytics, dashboards and visualisations in tools such as Power BI, Looker and QlikView, collaborating with analysts to meet reporting needs.
  • Provide hands-on technical leadership through design and code reviews, mentoring and knowledge sharing, helping colleagues adopt consistent engineering practices.
  • Troubleshoot complex production issues and improve performance, reliability and cost efficiency through automation, testing and CI/CD.
  • Maintain clear documentation and support effective client handover.
  • What skills we would like you to have Strong hands-on Python and SQL skills, with experience writing maintainable, tested code for production data workloads.
  • Practical experience building and optimising distributed data processing workloads using Spark and Databricks, or comparable platforms.
  • Strong understanding of relational databases, advanced SQL and query optimisation, with the ability to diagnose and resolve performance bottlenecks.
  • Experience implementing orchestration, scheduling, dependencies and failure handling using tools such as Airflow, Apache NiFi, Talend or platform-native workflows.
  • Strong data warehousing, modelling and schema design skills, with experience creating curated datasets for analytics and an understanding of how tools such as Power BI, QlikView and Looker consume data.
  • Experience using Git, automated testing and CI/CD to support collaborative development, controlled releases and maintainable data solutions.
  • Proven experience delivering production data engineering solutions end to end, including technical design, implementation, deployment and ongoing support.
  • Experience working with clients or business stakeholders to clarify requirements, explain technical choices and manage delivery expectations, alongside mentoring engineers and reviewing their work.
  • Experience delivering iteratively within Agile or Scrum teams, breaking down work, estimating effort and identifying dependencies and technical risks.

Requirements

  • Experience with Delta Lake or comparable lakehouse technologies, including notebooks, workflows and reliable production deployment.
  • Experience delivering and operating data solutions on at least one major cloud platform, such as AWS, Azure or GCP, including appropriate access and security controls.
  • Strong analytical and problem-solving skills, with sound judgement when balancing technical quality, delivery timelines and client priorities.
  • An inquisitive mindset and a commitment to continuous improvement, with the ability to evaluate new approaches and share practical learning across the Data practice.
  • Clear, confident communication, with the ability to explain complex technical ideas, challenge assumptions constructively and build trusted client relationships.
  • A collaborative, accountable approach, with the initiative to lead technical work, support colleagues’ development and give and receive constructive feedback.

Benefits

  • As technology partners with leading companies such as AWS and Databricks, we’ll invest in your training and development, alongside hands-on project experience with some of the UK’s leading organisations.
  • What we offer Competitive base salary.
  • Inclusion in our annual discretionary bonus plan with an on-target performance bonus of up to 15%.
  • Up to 37 days holiday inclusive of personal and public allocations.
  • Seven are fixed days for Christmas and New Year, and 30 are floating days taken at your discretion, subject to client scheduling and line manager approval.
  • Private medical insurance, single cover.
  • Group life and income protection insurance.
  • Salary sacrifice pension scheme after three months’ employment.
  • Access to over 1,000 perks and discounts via an employee discount portal.
  • Dedicated development time plus tools and funding to support personal and professional development.
Easy Apply