jobapplier

Senior QA Automation Engineer (AI-First)

Dow Jones

Skills

Required

  • automation
  • pipelines
  • automation frameworks
  • APIs
  • test strategies
  • software engineering
  • databases
  • test data validation
  • SQL
  • REST / GraphQL
  • CI/CD

Preferred

  • AI / LLM-powered
  • LLM
  • observability
  • Playwright
  • TypeScript
  • AI coding tools
  • Claude
  • Code
  • experimentation
  • Langfuse

Storyful, a company focused on building AI-native products. As a Senior QA Automation Engineer, you will play a crucial role in raising the quality bar and increasing delivery speed through AI-augmented engineering. You will design and build automation, evaluation, and quality foundations, working closely with software engineers, data scientists, product managers, and SRE. Your responsibilities will include building scalable automation frameworks, using AI coding agents, defining reusable testing patterns, and embedding testing early in the development lifecycle.

Responsibilities

  • You will also design evaluation pipelines for LLM-driven features and investigate complex issues across distributed systems.
  • Design and build the automation, evaluation, and quality foundations that enable engineers to own quality, while ensuring our products remain reliable, explainable, and trusted.
  • Build and evolve scalable automation frameworks across web UI, APIs, and data pipelines, and use AI coding agents to generate, refactor, and maintain test suites at scale.
  • Define pragmatic, risk-based test strategies across unit, integration, contract, and E2E layers, and establish clear definitions of done, quality gates, and release readiness standards.
  • Strong debugging and observability skills across distributed systems

Requirements

  • Strong software engineering fundamentals and ability to write maintainable production-grade code
  • Solid understanding of databases, APIs, and test data validation (SQL, REST/GraphQL)
  • Proven experience as a Senior QA Automation Engineer / SDET in production environments
  • Deep understanding of CI/CD and integrating automation into delivery pipelines

Security-aware mindset (OWASP, secure test design)

  • Experience designing automation frameworks and test strategies end-to-end
  • Experience driving shift-left practices and enabling engineers to own quality
  • Experience testing agentic systems and multi-step workflows
  • Performance and load testing at scale
  • Cloud experience (AWS/GCP) and test infrastructure design

Preferred

  • Strong hands-on experience with Playwright (or equivalent) using TypeScript
  • Demonstrated use of AI coding tools (e.g. Claude Code) as part of daily workflow
  • Familiarity with LLM observability and experimentation tools (e.g. Langfuse, MLflow)
  • Testing AI/LLM-powered systems (evaluation frameworks, prompt regression, drift detection)
  • Experience with LLM evaluation patterns (LLM-as-a-judge, semantic scoring, golden sets)
  • Exposure to RAG pipelines, vector search, or knowledge graph systems
  • Experience in high-trust domains (media, intelligence, risk, compliance)