Senior Data Engineer
Optima Partners
- London, United Kingdom
- Hybrid
- Senior Level
- Posted
Within our Data practice, you’ll work in mixed-capability pods alongside data engineers, analysts, data scientists and ML/AI engineers. You’ll bring technical depth, guide colleagues and turn data into practical solutions, clear insights and better customer experiences.
About Optima Partners
- Who we are Optima Partners is a VC-backed, scaling AI and data consultancy headquartered in Edinburgh.
- We bring together expertise in strategy, design, data science and engineering to help leading brands solve real business challenges.
- We turn data into better customer experiences, smarter decisions and lasting business value.
- Our clients include Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Eon Next, OVO and GSK.
- We work alongside their teams to deliver practical solutions and share the skills to build on them.
Responsibilities
- As a Senior Data Engineer within our Data practice, reporting to a Technical Lead, you will lead the design, build, testing and maintenance of reliable, scalable data pipelines and platforms for clients across diverse sectors.
- Working in multi-disciplinary teams, you’ll take ownership of technical delivery and help shape solutions from requirements through to production.
- This role suits an experienced data engineer who combines strong hands-on delivery with sound technical judgement, client-facing confidence and a commitment to developing others.
- You will work alongside engineers, data scientists, architects and product specialists to make pragmatic design decisions, resolve complex challenges and improve engineering standards.
- You’ll mentor colleagues, share knowledge and continue to develop your technical and consulting expertise.
- Lead the design, build, testing and operation of scalable data pipelines using platforms such as Databricks, taking ownership of delivery quality and production readiness.
- Design and develop data warehouses, data lakes and lakehouse solutions, making pragmatic architecture and data modelling decisions with architects and technical leads.
- Build and optimise robust ETL/ELT workflows, embedding data quality checks, monitoring and controls to deliver reliable, accessible data.
- Work directly with clients and multi-disciplinary colleagues to translate business requirements into technical designs, estimate work and communicate trade-offs, risks and progress.
- Deliver well-modelled, documented datasets that support analytics, dashboards and visualisations in tools such as Power BI, Looker and QlikView, collaborating with analysts to meet reporting needs.
- Provide hands-on technical leadership through design and code reviews, mentoring and knowledge sharing, helping colleagues adopt consistent engineering practices.
- Troubleshoot complex production issues and improve performance, reliability and cost efficiency through automation, testing and CI/CD.
- Maintain clear documentation and support effective client handover.
- What skills we would like you to have Strong hands-on Python and SQL skills, with experience writing maintainable, tested code for production data workloads.
- Practical experience building and optimising distributed data processing workloads using Spark and Databricks, or comparable platforms.
- Strong understanding of relational databases, advanced SQL and query optimisation, with the ability to diagnose and resolve performance bottlenecks.
- Experience implementing orchestration, scheduling, dependencies and failure handling using tools such as Airflow, Apache NiFi, Talend or platform-native workflows.
- Strong data warehousing, modelling and schema design skills, with experience creating curated datasets for analytics and an understanding of how tools such as Power BI, QlikView and Looker consume data.
- Experience using Git, automated testing and CI/CD to support collaborative development, controlled releases and maintainable data solutions.
- Proven experience delivering production data engineering solutions end to end, including technical design, implementation, deployment and ongoing support.
- Experience working with clients or business stakeholders to clarify requirements, explain technical choices and manage delivery expectations, alongside mentoring engineers and reviewing their work.
- Experience delivering iteratively within Agile or Scrum teams, breaking down work, estimating effort and identifying dependencies and technical risks.
Requirements
- Experience with Delta Lake or comparable lakehouse technologies, including notebooks, workflows and reliable production deployment.
- Experience delivering and operating data solutions on at least one major cloud platform, such as AWS, Azure or GCP, including appropriate access and security controls.
- Strong analytical and problem-solving skills, with sound judgement when balancing technical quality, delivery timelines and client priorities.
- An inquisitive mindset and a commitment to continuous improvement, with the ability to evaluate new approaches and share practical learning across the Data practice.
- Clear, confident communication, with the ability to explain complex technical ideas, challenge assumptions constructively and build trusted client relationships.
- A collaborative, accountable approach, with the initiative to lead technical work, support colleagues’ development and give and receive constructive feedback.
Benefits
- As technology partners with leading companies such as AWS and Databricks, we’ll invest in your training and development, alongside hands-on project experience with some of the UK’s leading organisations.
- What we offer Competitive base salary.
- Inclusion in our annual discretionary bonus plan with an on-target performance bonus of up to 15%.
- Up to 37 days holiday inclusive of personal and public allocations.
- Seven are fixed days for Christmas and New Year, and 30 are floating days taken at your discretion, subject to client scheduling and line manager approval.
- Private medical insurance, single cover.
- Group life and income protection insurance.
- Salary sacrifice pension scheme after three months’ employment.
- Access to over 1,000 perks and discounts via an employee discount portal.
- Dedicated development time plus tools and funding to support personal and professional development.
