Senior QA Automation Engineer - SIP/VOIP
IDT
- Bogotá
- Remote
- Quality Assurance
- 5+ years
- Posted
This job is closed: the employer no longer lists it.
Skills
Required
- SIP
- QA
- call flows / SIP signaling / media handling / softphones / SIP trunking
- JsSIP / WebRTC
- Python
- Jira
- SIP/VoIP systems
- test design
- automation frameworks
- probabilistic
- formal TMS tools
- test automation
- CI/CD pipelines
- LLM APIs
Preferred
- ASR / transcription
- jitter / packet
- LLM-based / conversational AI
- evaluation frameworks
- LLM-as-a-judge
- speech technologies
- WER
- Jenkins
We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace.
Responsibilities
- You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway.
- You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue.
- LATAM.
- Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.
- Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
- Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery.
- Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes.
Adversarially test the agent
- against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data.
- Prepare and provision SIP devices, endpoints, and call scenarios to automate call origination end-to-end.
- Diagnose failures across the entire stack, isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer.
Extend automated coverage
- and integrate test suites into the CI/CD pipeline to serve as release gates.
- Define and track core quality metrics, including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness.
- Maintain documented test coverage in the TMS with full traceability to business requirements.
- Communicate and escalate quality risks to stakeholders backed by data and evidence.
- for programmatic call origination and media control.
- Allure TestOps
- as our TMS, linking automated runs to documented test cases.
- Jira
- for project management and issue tracking.
- Jenkins & GitHub Actions
- for quality and delivery metrics.
Requirements
- CV IN ENGLISH.
- IS REQUIRED.
- 5+ years in QA, with recent hands-on ownership of a complex, integrated product.
- Practical experience testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures.
- Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted).
- Proficiency in Python, sufficient to independently build and maintain automation frameworks.
- Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs.
- Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions.
- Hands-on experience with formal TMS tools and Jira
- (spoken and written), with reliable overlap with US Eastern business hours.
- Python with pytest
- for test automation.
- Playwright
- for browser and WebRTC-based call driving.
- JsSIP / WebRTC
SIP softphones and devices
- for manual and exploratory testing.
- for CI/CD pipelines.
- Grafana
- LLM APIs and evaluation tooling
- currently in active development, and we are open to any effective approach.
Preferred
- Direct experience testing LLM-based or conversational AI products, including evaluation frameworks and LLM-as-a-judge approaches.
- Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection.
- Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval.
- WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss.
- Background in testing IVR, contact center, or conversational IVR platforms.
- Multilingual testing capabilities, with emphasis on Spanish/English code-switching.
- Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes
Benefits
- Remote work opportunity!
- B2B Employment ($, gross).
- Stable job with long-term growth perspective.
- Competitive salary with annual performance review.
- Really good hardware.
- An exciting and challenging job with talented people around.
- Continuous learning and career growth opportunities.
- Compensation for professional training, seminars, and conferences.
- Referral program: get rewarded for helping us grow the team with talented people.
- Company-supported English classes to enhance your professional growth.
