{"type":"video","version":"1.0","html":"<iframe src=\"https://www.loom.com/embed/45c2f50d7ea44a9094bce4d0f6062d67\" frameborder=\"0\" width=\"1170\" height=\"877\" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>","height":877,"width":1170,"provider_name":"Loom","provider_url":"https://www.loom.com","thumbnail_height":877,"thumbnail_width":1170,"thumbnail_url":"https://cdn.loom.com/sessions/thumbnails/45c2f50d7ea44a9094bce4d0f6062d67-4fce5074d6f24090.gif","duration":301.387,"title":"Advanced AI Configs","description":"In this video, I walk you through the capabilities of our AI Configs product, which allows us to decouple prompts and models from hard coding, enabling rapid iteration and performance metric collection in live production. I demonstrate how I implemented production evaluations using a Retrieval-Augmented Generation (RAG) approach, testing various models like SONNET 4 and Nova Pro, while leveraging custom parameters for enhanced accuracy and guardrails. The results show that while SONNET 4 excelled in accuracy, it came at a significantly higher cost compared to Nova Pro. My goal is to refine our RAG retrieval and prompts to make Nova Pro more competitive, balancing performance with cost efficiency. I encourage you to think about how we can apply these insights to improve our models and processes moving forward."}