# Generative AI Engineer at StreetID
Source: https://true-hire-ten.vercel.app/jobs/python-developer-streetid-l536qrid · Company profile: https://true-hire-ten.vercel.app/companies/streetid · Apply on the employer's site: https://www.whatjobs.com/pub_api__cpl__3188517579__7065?utm_campaign=publisher&utm_medium=api&utm_source=freehire.me
## Job details
- Location: United States
- Seniority tag: Senior
- Category: AI Engineering
- Countries: United States
- Posted: 2026-10-21 (29 days ago, stale)
- Skills named: ab-testing, agentic-ai, ai, analytics, api, automation, aws, aws-bedrock, ci-cd, cloudwatch, cobol, crewai, data-governance, db2, generative-ai, iam, java, lambda, langgraph, llm, machine-learning, observability, python, rag, s3, spring, sybase, test-automation
## Requirements

- Expert-level Generative AI engineering experience, including hands-on development and deployment of production-grade LLM and agentic AI applications.
- Advanced Python programming expertise, including building scalable backend services, APIs, automated testing frameworks, and production evaluation pipelines.
- Demonstrated experience developing LLM evaluation frameworks, including offline/on-demand regression testing and online/continuous evaluation of production AI systems.
- Hands-on experience with LLM-as-Judge, evaluation rubric development, scoring methodologies, and automated quality assessment.
- Experience evaluating agent performance across task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
- Strong experience with LangGraph, Strands, CrewAI, or comparable agentic AI frameworks.
- Hands-on knowledge of Amazon Bedrock, Bedrock Guardrails, AWS Lambda, S3, IAM, and CloudWatch.
- Experience designing and operating scalable, production-grade applications and AI infrastructure.
- Familiarity with CI/CD automation, application monitoring, logging, and production troubleshooting.
- Strong communication skills and experience collaborating with enterprise engineering and architecture teams.
- Active AWS Certification (one required):
- AWS Certified Generative AI Developer – Professional
- AWS Certified Machine Learning Engineer – Associate

## Nice to have

- Experience with Amazon Bedrock AgentCore Evaluations, including on-demand evaluation runners, online monitoring, and A/B testing.
- Advanced knowledge of Agent-as-Judge frameworks, human-in-the-loop evaluation, and automated AI quality assurance.
- Experience with AI benchmarking frameworks such as tau-bench, SWE-bench, GAIA, and custom evaluation datasets.
- Experience evaluating RAG systems using faithfulness, context precision, context recall, and response quality metrics.
- Familiarity with distributed tracing, structured logging, observability platforms, and automated evaluation within CI/CD pipelines.
- Experience with enterprise legacy modernization, including applications built using Java/Spring, Sybase, DB2, or COBOL.
- Background delivering AI solutions within financial services, banking, capital markets, or other regulated industries.
- Knowledge of financial services compliance requirements, data governance, audit controls, and security standards.
- Experience designing reusable AI evaluation architectures and components across large enterprise application portfolios, including environments supporting 50+ applications.

## About the role
We are seeking a Senior AI Evaluations Engineer to work alongside AWS Professional Services (ProServe) teams, designing, developing, and operating enterprise-grade AI evaluation frameworks for large enterprise customers, primarily within financial services.

This role focuses on ensuring that Generative AI and agentic AI applications are accurate, reliable, secure, and production-ready. The ideal candidate brings advanced Python engineering expertise, hands-on experience with Amazon Bedrock, and a strong background in developing automated evaluation systems for LLM-powered applications and multi-agent workflows.

The engineer will lead the design of LLM-as-Judge and Agent-as-Judge frameworks, implement quality rubrics and scoring architectures, and develop offline regression testing and continuous live-traffic evaluation capabilities.

Key Responsibilities

- AI Evaluation Frameworks: Design, develop, and operate end-to-end evaluation frameworks for Generative AI and agentic AI applications, including offline regression testing, online monitoring, continuous evaluation, and A/B testing.
- Python Engineering: Develop scalable, production-grade evaluation services, automation frameworks, APIs, and testing pipelines using advanced Python.
- LLM-as-Judge / Agent-as-Judge: Build automated evaluation pipelines to assess agent behavior, task completion, reasoning trajectories, tool-call accuracy, groundedness, and safety.
- Evaluation Rubrics and Scoring: Define structured scoring methodologies and quality standards at the session, response, and individual tool-call levels.
- Agentic AI Development: Develop and evaluate multi-agent systems using LangGraph, Strands, CrewAI, or similar agent orchestration frameworks.
- AWS AI Infrastructure: Build and integrate evaluation solutions using Amazon Bedrock, Bedrock Guardrails, AWS Lambda, Amazon S3, IAM, and CloudWatch.
- Evaluation Infrastructure: Design reusable evaluation components, testing frameworks, and automated benchmarking solutions supporting enterprise AI deployments.
- CI/CD Integration: Integrate AI evaluation pipelines into software delivery workflows, enabling automated regression testing, quality gates, and production release validation.
- AI Observability: Implement monitoring, distributed tracing, structured logging, and performance analytics to assess production AI behavior and reliability.
- RAG Evaluation: Evaluate retrieval-augmented generation applications using metrics such as faithfulness, groundedness, context precision, and recall.
- AI Governance and Security: Implement evaluation processes that address hallucinations, data security, safety, compliance, and auditability within regulated enterprise environments.
- Technical Leadership: Collaborate with AWS architects, software engineers, data scientists, and enterprise stakeholders to establish evaluation standards, reusable frameworks, and AI engineering best practices.

Required Qualifications

- Expert-level Generative AI engineering experience, including hands-on development and deployment of production-grade LLM and agentic AI applications.
- Advanced Python programming expertise, including building scalable backend services, APIs, automated testing frameworks, and production evaluation pipelines.
- Demonstrated experience developing LLM evaluation frameworks, including offline/on-demand regression testing and online/continuous evaluation of production AI systems.
- Hands-on experience with LLM-as-Judge, evaluation rubric development, scoring methodologies, and automated quality assessment.
- Experience evaluating agent performance across task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
- Strong experience with LangGraph, Strands, CrewAI, or comparable agentic AI frameworks.
- Hands-on knowledge of Amazon Bedrock, Bedrock Guardrails, AWS Lambda, S3, IAM, and CloudWatch.
- Experience designing and operating scalable, production-grade applications and AI infrastructure.
- Familiarity with CI/CD automation, application monitoring, logging, and production troubleshooting.
- Strong communication skills and experience collaborating with enterprise engineering and architecture teams.
- Active AWS Certification (one required):
- AWS Certified Generative AI Developer – Professional
- AWS Certified Machine Learning Engineer – Associate

Preferred Qualifications

- Experience with Amazon Bedrock AgentCore Evaluations, including on-demand evaluation runners, online monitoring, and A/B testing.
- Advanced knowledge of Agent-as-Judge frameworks, human-in-the-loop evaluation, and automated AI quality assurance.
- Experience with AI benchmarking frameworks such as tau-bench, SWE-bench, GAIA, and custom evaluation datasets.
- Experience evaluating RAG systems using faithfulness, context precision, context recall, and response quality metrics.
- Familiarity with distributed tracing, structured logging, observability platforms, and automated evaluation within CI/CD pipelines.
- Experience with enterprise legacy modernization, including applications built using Java/Spring, Sybase, DB2, or COBOL.
- Background delivering AI solutions within financial services, banking, capital markets, or other regulated industries.
- Knowledge of financial services compliance requirements, data governance, audit controls, and security standards.
- Experience designing reusable AI evaluation architectures and components across large enterprise application portfolios, including environments supporting 50+ applications.
© TrueHire. Listing data from freehire; applications go to the employer's own page.