• Job Description

  Req#: 26-00265
  Job Title
  
  AI Testing Specialist
  
  Location
  
  Hybrid / Remote
  
  Employment Type
  
  Full-time
  
  Job Summary
  
  We are seeking an AI Testing Specialist to ensure the quality, reliability, security, and performance of AI-powered applications, machine learning models, and generative AI solutions. The ideal candidate will develop and execute comprehensive testing strategies for AI systems, validate model outputs, assess AI-specific risks, and collaborate with cross-functional teams to deliver high-quality AI products.
  
  Key Responsibilities
  
  • Design and execute test strategies for AI, machine learning, and generative AI applications.
  • Create and maintain test plans, test cases, and test data for AI features and workflows.
  • Validate AI model outputs for accuracy, consistency, relevance, factuality, and reliability.
  • Evaluate AI systems for hallucinations, bias, toxicity, fairness, and robustness.
  • Perform functional, regression, integration, API, end-to-end, performance, usability, and security testing.
  • Test prompt-based applications and optimize prompts for consistent results.
  • Develop automated testing frameworks for AI applications and APIs.
  • Verify data quality, preprocessing pipelines, and model inputs.
  • Conduct stress, load, and scalability testing for AI services.
  • Identify, document, prioritize, and track defects using bug management tools.
  • Collaborate with AI engineers, data scientists, software developers, product managers, and UX teams.
  • Monitor production AI systems and support continuous quality improvement.
  • Prepare test reports, quality metrics, and release recommendations.
  • Ensure compliance with organizational AI governance, security, privacy, and regulatory requirements.
  
  Required Qualifications
  
  • Bachelor's degree in Computer Science, Information Technology, Software Engineering, Data Science, or a related field.
  • 3-5+ years of experience in software testing, QA, or AI testing.
  • Strong understanding of software testing methodologies and quality assurance principles.
  • Experience testing APIs, web applications, and cloud-based systems.
  • Familiarity with AI, machine learning, and generative AI concepts.
  • Experience working in Agile or Scrum environments.
  
  Preferred Qualifications
  
  • Experience testing Large Language Model (LLM) applications.
  • Knowledge of prompt engineering and AI evaluation methodologies.
  • Experience with Responsible AI practices and AI governance.
  • AI, cloud, or software testing certifications.
  • Experience with MLOps workflows and model lifecycle management.
  
  Technical Skills
  
  • Manual and automated testing
  • Test planning and execution
  • API testing (Postman, REST Assured)
  • Automation frameworks (Selenium, Playwright, Cypress)
  • Programming (Python, Java, JavaScript, or C#)
  • SQL and database validation
  • Git and CI/CD tools
  • Test management tools (Jira, TestRail, Zephyr)
  • Performance testing (JMeter, k6, LoadRunner)
  • AI model evaluation techniques
  • Prompt engineering
  • LLM testing and validation
  • AI safety testing (hallucinations, bias, toxicity, prompt injection, jailbreak resistance)
  • Data validation and preprocessing verification
  • JSON, REST APIs, and cloud platforms (AWS, Azure, Google Cloud)
  
  Soft Skills
  
  • Analytical and critical thinking
  • Strong attention to detail
  • Problem-solving
  • Effective communication
  • Collaboration across multidisciplinary teams
  • Documentation and reporting
  • Adaptability
  • Time management
  • Continuous learning
  
  Preferred Experience
  
  • AI-powered enterprise applications
  • Conversational AI and chatbots
  • Generative AI products
  • Machine learning platforms
  • Retrieval-Augmented Generation (RAG) systems
  • SaaS and cloud-native applications
  • Healthcare, finance, retail, or other regulated industries
  
  Success Metrics
  
  • Test coverage and automation coverage
  • Defect detection and prevention rate
  • AI response quality and reliability
  • Reduction in production defects
  • Model evaluation accuracy
  • Compliance with AI quality and governance standards
  • Release readiness and stability
  • Customer satisfaction and user experience
  • Test execution efficiency
  
  Nice-to-Have Skills
  
  • AI evaluation frameworks (DeepEval, Ragas, LangSmith, Promptfoo)
  • LangChain or similar AI orchestration frameworks
  • Vector databases
  • Docker and Kubernetes
  • MLOps tools (MLflow, Kubeflow, SageMaker)
  • Explainable AI (XAI) concepts
  • Data annotation and synthetic data generation
  • Accessibility testing
  • Security testing for AI systems

• About the company

  The best remote jobs for you

Notice