video1.0<iframe src="https://www.loom.com/embed/64d407301eca41faaa2ac2c0ae4f8a12" frameborder="0" width="1920" height="1440" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>14401920Loomhttps://www.loom.com14401920https://cdn.loom.com/sessions/thumbnails/64d407301eca41faaa2ac2c0ae4f8a12-1098adaec8167708.gif515.423In this video, I walk through the process of ingesting and embedding a large dataset of 75,000 markdown files, which is crucial for our Lore Manager system. I discuss the steps involved, including data cleaning, generating embeddings, and creating an index for efficient querying. I also highlight the importance of using an agent to refine search results and avoid hallucinations in responses. I encourage you to check the progress of the embedding job and ensure everything functions properly. Please proceed with the actions autonomously, but remember to avoid any data destruction.