jobapplier

Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Falls Technology

Skills

Preferred

  • RAG testing
  • Text classification
  • scikit-learn / deep learning
  • AI
  • Python
  • PyTorch / TensorFlow / Keras
  • SQL
  • GitHub
  • Jenkins
  • Flask

Prove which LLMs actually work for the mission. You'll design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships AI systems they can trust.

About Falls Technology

We're a small business that's been delivering since 2019, so your work is visible and your voice counts.

Responsibilities

  • What you'll do Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)
  • Design and run experiments that benchmark LLMs against defined measures
  • Tune neural networks / LLMs on custom datasets and apply results to concrete use cases
  • AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)
  • Use deep learning frameworks such as PyTorch, TensorFlow or Keras Leverage GPUs for accelerated training and evaluation
  • Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets
  • Work in GitHub and Jenkins-style CI for reproducible experiments
  • Communicate methodological choices and model results to technical and non-technical stakeholders
  • What you bring (required) Active TS/SCI Hands-on RAG testing experience (must-have) Tuning neural nets / LLMs on custom datasets for real use cases
  • Building AI/ML models and ensembles in Python or R PyTorch, TensorFlow or Keras (key) GPU-accelerated computing (key) Testing, evaluating or validating LLM-based AI systems (key) Experiment design to benchmark LLMs (key) SQL including CTEs, set ops, aggregates and nested subqueries GitHub and Jenkins (or equivalent) version control / CI Nice to have Hugging Face Transformers library and hub
  • Building benchmarks that measure LLM performance
  • Cloud computing development and architecture
  • Flask or similar front-end/web frameworks

Requirements

  • Active TS/SCI required.
  • U.S. citizenship required.

Preferred

  • Text classification and topic modeling (scikit-learn or deep learning)

Location

On-site in McLean, VA (Full-Time).

Semantic search applications Tableau dashboards Verification/validation test benches, Explainable AI (XAI), ONNX Why Falls A 9% retirement contribution, 100% of family medical premiums paid, and 4 weeks of PTO.

Falls Technology is an Equal Opportunity Employer. All qualified applicants will receive consideration without regard to race, color, religion, sex, national origin, disability, or protected veteran status.