{"type":"video","version":"1.0","html":"<iframe src=\"https://www.loom.com/embed/64d407301eca41faaa2ac2c0ae4f8a12\" frameborder=\"0\" width=\"1920\" height=\"1440\" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>","height":1440,"width":1920,"provider_name":"Loom","provider_url":"https://www.loom.com","thumbnail_height":1440,"thumbnail_width":1920,"thumbnail_url":"https://cdn.loom.com/sessions/thumbnails/64d407301eca41faaa2ac2c0ae4f8a12-1098adaec8167708.gif","duration":515.423,"title":"Ingesting and Embedding Data Efficiently for AI Systems 🚀","description":"In this video, I walk through the process of ingesting and embedding a large dataset of 75,000 markdown files, which is crucial for our Lore Manager system. I discuss the steps involved, including data cleaning, generating embeddings, and creating an index for efficient querying. I also highlight the importance of using an agent to refine search results and avoid hallucinations in responses. I encourage you to check the progress of the embedding job and ensure everything functions properly. Please proceed with the actions autonomously, but remember to avoid any data destruction."}