jobapplier

Senior QA Automation Engineer - SIP/VOIP

IDT

This job is closed: the employer no longer lists it.

Skills

Required

  • SIP
  • QA
  • call flows / SIP signaling / media handling / softphones / SIP trunking
  • JsSIP / WebRTC
  • Python
  • Jira
  • SIP/VoIP systems
  • test design
  • automation frameworks
  • probabilistic
  • formal TMS tools
  • test automation
  • CI/CD pipelines
  • LLM APIs

Preferred

  • ASR / transcription
  • jitter / packet
  • LLM-based / conversational AI
  • evaluation frameworks
  • LLM-as-a-judge
  • speech technologies
  • WER
  • Jenkins

We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace.

Responsibilities

  • You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway.
  • You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue.
  • LATAM.
  • Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.
  • Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
  • Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery.
  • Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes.

Adversarially test the agent

  • against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data.
  • Prepare and provision SIP devices, endpoints, and call scenarios to automate call origination end-to-end.
  • Diagnose failures across the entire stack, isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer.

Extend automated coverage

  • and integrate test suites into the CI/CD pipeline to serve as release gates.
  • Define and track core quality metrics, including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness.
  • Maintain documented test coverage in the TMS with full traceability to business requirements.
  • Communicate and escalate quality risks to stakeholders backed by data and evidence.
  • for programmatic call origination and media control.
  • Allure TestOps
  • as our TMS, linking automated runs to documented test cases.
  • Jira
  • for project management and issue tracking.
  • Jenkins & GitHub Actions
  • for quality and delivery metrics.

Requirements

  • CV IN ENGLISH.
  • IS REQUIRED.
  • 5+ years in QA, with recent hands-on ownership of a complex, integrated product.
  • Practical experience testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures.
  • Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted).
  • Proficiency in Python, sufficient to independently build and maintain automation frameworks.
  • Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs.
  • Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions.
  • Hands-on experience with formal TMS tools and Jira
  • (spoken and written), with reliable overlap with US Eastern business hours.
  • Python with pytest
  • for test automation.
  • Playwright
  • for browser and WebRTC-based call driving.
  • JsSIP / WebRTC

SIP softphones and devices

  • for manual and exploratory testing.
  • for CI/CD pipelines.
  • Grafana
  • LLM APIs and evaluation tooling
  • currently in active development, and we are open to any effective approach.

Preferred

  • Direct experience testing LLM-based or conversational AI products, including evaluation frameworks and LLM-as-a-judge approaches.
  • Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection.
  • Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval.
  • WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss.
  • Background in testing IVR, contact center, or conversational IVR platforms.
  • Multilingual testing capabilities, with emphasis on Spanish/English code-switching.
  • Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes

Benefits

  • Remote work opportunity!
  • B2B Employment ($, gross).
  • Stable job with long-term growth perspective.
  • Competitive salary with annual performance review.
  • Really good hardware.
  • An exciting and challenging job with talented people around.
  • Continuous learning and career growth opportunities.
  • Compensation for professional training, seminars, and conferences.
  • Referral program: get rewarded for helping us grow the team with talented people.
  • Company-supported English classes to enhance your professional growth.