LLM Inference Engineer (Mid, Senior, Staff)

Join HAI as an LLM Inference Engineer, where you'll optimize the serving infrastructure for our groundbreaking healthcare AI. Your work will directly impact latency, cost, and availability for millions of patient conversations. You'll collaborate with systems engineers, ML researchers, and infrastructure experts to design and implement advanced serving architectures, optimize multi-LoRA serving systems, and continuously benchmark and improve system performance.

About Hippocratic AI

• Optimizing the systems that translate raw model capability into sub-100ms responses, making the difference between conversational experiences that feel natural and those that feel broken.
• Designing and deploying advanced serving architectures (disaggregated inference, optimized caching, speculative decoding) that meaningfully improve patient experience and operational efficiency.
• Experience optimizing LLM inference systems at scale

Responsibilities

• Continuously benchmarking and improving system performance across various deployment scenarios and GPU types, and implementing advanced quantization techniques to reduce model footprint while preserving quality.
• Experience with CUDA programming and GPU optimization

Requirements

• Proficiency in Python and C++
• Hands-on experience implementing quantization techniques for transformer models
• Speculative decoding techniques with draft models
• Proven expertise with distributed serving architectures for large language models

Eagle speculative decoding approaches

• Strong understanding of modern inference optimization methods, including:
• Track record of deploying inference systems in production environments
• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
• Deep understanding of performance optimization systems
• Experience with custom CUDA kernels