Principal Software Engineer

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites, including:
• Unit, integration, and end-to-end (E2E)
• Functional, regression, performance, and safety testing
• Validate AI system behavior, including:
• Non-deterministic LLM outputs
• Hallucinations and edge cases
• Multi-step agent decision-making
• Design and manage evaluation systems:
• Golden datasets
• Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs, including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

• Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

• AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
• Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
• Monitoring: Prometheus, Grafana, OpenTelemetrySoft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus on quality KPIs
• Detail-oriented with a strong bias toward automation and scalabilityExperience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications

• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readinessPreferred

• Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to:
• Shift-left testing practices
• Production observability and monitoring
• Chaos or resilience testing

Location: Bangalore India

Department: AI Engineering / Quality Assurance Experience Level: Mid to Senior

We are looking for a highly motivated SeniorAI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

• Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
• RAG architectures and vector databases (Pinecone, Weaviate)
• Agent frameworks (LangChain, LlamaIndex, AutoGen)

• Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge of evaluation methods:
• LLM-as-a-judge
• BLEU, ROUGE, semantic similarity scoring
• Experience with prompt and agent regression testing
• Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related fieldNice-to-have:

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation