Remote Jobs
Evaluation Scenario Writer - AI Agent Testing Specialist
What's your preference?
Job Description
- Req#: 8BE0FDABA1
Employer Industry: Artificial Intelligence and Technology Consulting
Why consider this job opportunity:
- Get paid for your expertise, with rates that can go up to $80/hour depending on your skills, experience, and project needs.
- Flexible, remote, freelance project that fits around your primary professional or academic commitments.
- Contribute to advanced AI projects and gain valuable experience to enhance your portfolio.
- Opportunity to influence how future AI models understand and communicate in your field of expertise.
- Work on innovative projects that shape the future of Generative AI.
What to Expect (Job Responsibilities):
- Design realistic and structured evaluation scenarios for LLM-based agents.
- Create test cases that simulate complex human workflows and define gold-standard behavior.
- Analyze agent logs, failure modes, and decision paths to improve AI agent performance.
- Collaborate with code repositories and test frameworks to validate evaluation scenarios.
- Iterate on prompts, instructions, and test cases to enhance clarity and difficulty.
What is Required (Qualifications):
- Bachelor's and/or Master's Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Computational Linguistics, or related fields.
- Background in QA, software testing, data analysis, or NLP annotation.
- Good understanding of test design principles, including reproducibility and coverage.
- Strong written communication skills in English.
- Basic experience with Python and JavaScript.
How to Stand Out (Preferred Qualifications):
- Experience in writing manual or automated test cases.
- Familiarity with LLM capabilities and typical failure modes.
- Understanding of scoring metrics such as precision, recall, and reward functions.
#ArtificialIntelligence #RemoteWork #FreelanceOpportunity #GenerativeAI #TechInnovationAbout the company
The best remote jobs for you
Notice
Talentify is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.
Talentify provides reasonable accommodations to qualified applicants with disabilities, including disabled veterans. Request assistance at accessibility@talentify.io or 407-000-0000.
Federal law requires every new hire to complete Form I-9 and present proof of identity and U.S. work eligibility.
An Automated Employment Decision Tool (AEDT) will score your job-related skills and responses. Bias-audit & data-use details: www.talentify.io/bias-audit-report. NYC applicants may request an alternative process or accommodation at aedt@talentify.io or 407-000-0000.