Automation QA Engineer

About the company

• Partner company ensuring reliability, security, and performance of advanced AI-powered platforms.

Responsibilities

• Design, build, and maintain scalable automated test frameworks using Playwright and TypeScript.
• Develop test suites covering frontend, backend, APIs, and contract compatibility.
• Create testing strategies for nondeterministic AI systems: statistical assertions, semantic similarity, confidence-threshold validation.
• Validate asynchronous and event-driven workflows through failure injection, retries, idempotency checks, eventual-consistency assertions.
• Develop automated tests for multi-tenant security and isolation.
• Build adversarial test datasets for prompt injection, tool-call manipulation, data exfiltration; align with OWASP Top 10 for LLM Applications.
• Verify human-in-the-loop escalation procedures.
• Build golden-dataset-driven validation and synthetic-data generation pipelines.
• Automate performance and non-functional testing: latency, concurrency, queue backpressure, failover.
• Use Langfuse and OpenTelemetry to validate trace completeness, token usage, cost accounting.
• Integrate test suites into CI/CD with GitHub Actions.
• Mentor mid-level QA engineers.

Requirements

• At least 8 years of experience in test automation, API testing, and reusable test frameworks.
• Strong proficiency in Playwright and TypeScript (mandatory).
• Hands-on experience testing LLM-powered or nondeterministic systems, including RAG failure modes.
• Experience testing event-driven architectures, distributed workflows, background processing; knowledge of failure injection, idempotency, retries, eventual consistency; experience with Temporal is an advantage.