About the role
Our engineering team ships through orchestrated AI agents rather than hand-typed code. That only works if verification is stronger than generation, and this role owns verification. You build the test systems, evals, and quality gates that let a fleet of agents ship to production every day without breaking things.
You are the professional skeptic of the team. Agents write the code; you build the machinery that proves it is correct, secure, and fast, automatically and at scale. When the team moves faster, it is because your harnesses made it safe to.
This is an on-site position. We move fast, pair tightly, and think best in the same room.
A note on experience: the years-of-experience bar is a guideline, not a gate. If you have built serious test automation or eval harnesses for LLM systems, we want to talk, even if you are earlier in your career. Demonstrated verification machinery beats a long resume.
What you'll actually do
- Build and own the automated test infrastructure: unit, integration, and end-to-end suites that run on every agent-generated change.
- Design evals for LLM-powered features: output quality, correctness, regression, cost, and latency across models and prompts.
- Automate model and prompt regression testing, so a model swap or a prompt change is caught before a customer ever sees it.
- Build agents that test agents: adversarial reviewers, fuzzers, and verification loops wired straight into CI.
- Own the quality gates in CI/CD and keep the ship-daily pipeline fast, parallel, and trustworthy.
- Chase flaky tests, performance regressions, and security holes to their root cause and kill them there.
Must-haves
Behavioral
- Quality from the customer's shoes. A bug that reaches a user is a broken promise. You test the journey the customer actually takes, not just the code paths that are easy to cover.
- Runs toward the frontier. Models change monthly and tooling changes weekly, and that is the fun part. You treat every new model and framework as new surface to master, not new ground to defend.
- Trusts nothing, verifies everything. Generated code, glowing demos, green checkmarks: you assume broken until proven working, and you build the proof.
- Automates on the second repeat. If you tested something by hand twice, the third time is a script. Manual regression passes offend you.
- Output measured in shipped confidence. Your win condition is the team merging faster because your gates catch what matters and stay quiet about what does not.
- High agency. Finds the riskiest untested surface without being told, and closes it.
Technical
- Strong TypeScript or JavaScript, typically 2 to 5 years in SDET, QA automation, or software engineering with a testing focus.
- Deep with modern test frameworks: Playwright for end-to-end, Vitest or Jest for unit and integration.
- Has tested or built LLM-powered systems: understands non-determinism, prompt regressions, and how to design evals that still mean something. Experience with promptfoo, Braintrust, LangSmith, or a homegrown harness all count.
- CI/CD fluency: GitHub Actions or similar, parallelized suites, caching, and quality gates that do not become the bottleneck.
- API and contract testing fundamentals, plus comfort seeding databases and managing test data.
- Security and performance testing instincts: you probe auth boundaries and load paths without being asked.
Good-to-haves
- Built LLM-as-judge pipelines or custom eval harnesses at scale.
- Load and chaos testing: k6, Artillery, or similar.
- Daily use of agentic coding tools (Claude Code, Cursor) and experience testing what they produce.
- Observability chops: traces, metrics, and alerts with OpenTelemetry, Sentry, or similar.
- Contributions to open-source testing or eval tooling.
How we work
Ship-daily culture, tight feedback loops, and AI-orchestrated review. We give you the most capable models, generous token budgets, and the autonomy to use them. Bring the judgment.
How to apply
Email careers@vmdsolution.com with the subject line "Application: AI Agent and Model Automation Engineer (SDET)". Skip the cover letter. Send the best verification machinery you have built: a test harness, an eval suite, a CI pipeline, or a write-up of a bug class you eliminated for good.