Creating SQL Migration:
    A migration file is created using the command npx supabase migration new embed.
    An embed() trigger function is defined which will be used to asynchronously generate embeddings on arbitrary tables using a new embed Edge Function.

Setting Up Trigger:
    A trigger named embed_document_sections is created to execute the embed() function after an insert operation on the document_sections table.

Applying Migration:
    The migration is applied to the local database using the command npx supabase migration up.

Creating Edge Function:
    An Edge function file is created using the command npx supabase functions new embed.
    In the embed/index.ts file, an embedding pipeline is created using Transformers.js, and a function to generate embeddings, generateEmbedding, is defined.

Configuring Supabase Client:
    Environment variables for Supabase are checked, and a Supabase client is created.
    The Supabase client is configured to inherit the user’s permissions.

Fetching, Generating, and Updating Embeddings:
    Text content is fetched from the specified table and column.
    Embeddings are generated for each piece of text.
    The respective rows in the table are updated with the generated embeddings.

Returning Success Response:
    A success response with status 204 is returned upon successful execution of the embedding generation and updating process.

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter

from langchain.schema import Document
from langchain.vectorstores import SupabaseVectorStore
from supabase.client import create_client


import os
from dotenv import load_dotenv
from pathlib import Path

env_path = Path('.') / '../../.env.local'
load_dotenv(dotenv_path=env_path)

Deno's Advantages:
    Security: Deno is designed with security in mind. It operates in a secure sandbox environment, which means scripts can't access the network, file system, or environment variables without explicit permission. This feature aligns with Supabase's focus on building secure applications​1​.
    Modern JavaScript Features: Deno supports modern JavaScript features out of the box. This makes it easier for developers to write and maintain their code, and it’s particularly useful when working with modern libraries and frameworks like Transformers.js​2​.
    Built-in Tooling: Deno comes with built-in tooling like a test runner, code formatter, and bundler, which can simplify the development process, making it an attractive choice for building web applications quickly and efficiently​3​.

Transformers.js and Supabase/gte-small Model:
    Transformers.js Compatibility: The Supabase/gte-small model is designed to be compatible with Transformers.js, which is a library for using transformer models in the browser. This compatibility allows for the easy generation of embeddings using the model directly within a web application environment​4​​5​.
    Efficient Text Embedding Generation: The Supabase/gte-small model is utilized for generating text embeddings efficiently and is used in the workshop to facilitate the creation of a chat interface that can interact with documentations​6​.

General Text Embeddings (GTE): The thenlper/gte-small model, developed by Alibaba DAMO Academy, is based on the BERT framework and is trained on a large-scale corpus of relevant text pairs. This training enables the model to be highly effective in various text embedding tasks such as information retrieval, semantic textual similarity, and text reranking, which could be instrumental in LangChain's use case for text embeddings​7​.

Performance and Usage: In the Hugging Face description, it's noted that the thenlper/gte-small model has decent performance metrics on text embedding tasks when compared to other popular text embedding models. The provided code example also showcases how to utilize the model to generate text embeddings which can be beneficial in various natural language processing applications​7​.

model_name = "thenlper/gte-small"
model_kwargs = {'device': 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
hf = HuggingFaceEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

SUPABASE_URL = os.getenv("NEXT_PUBLIC_SUPABASE_URL")
SUPABASE_KEY = os.getenv("NEXT_PUBLIC_SUPABASE_ANON_KEY")

supabase_client = create_client(SUPABASE_URL, SUPABASE_KEY)

# Load the text document from the specified path
raw_documents = TextLoader('../../sample-files/roman-empire-1.md').load()

# Initialize a CharacterTextSplitter with a chunk size of 1000 and overlap of 0
# This will split the text into chunks of 1000 characters each, with no overlap between chunks
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)

# Split the loaded documents into chunks using the text splitter
documents = text_splitter.split_documents(raw_documents)

vector_store = SupabaseVectorStore.from_documents(
    documents, 
    hf, 
    client=supabase_client, 
    table_name="langchain_documents", 
    query_name="match_langchain_documents_mmr"
)

query = "Who was the first emperor of the Roman Empire?"
retriever = vector_store.as_retriever(search_type="mmr")
matched_docs = retriever.get_relevant_documents(query)
for i, d in enumerate(matched_docs):
    print(f"\n## Document {i}\n")
    print(d.page_content)


from langchain.agents.agent_toolkits import create_retriever_tool

tool = create_retriever_tool(
    retriever, 
    "search_about_roman_empire",
    "Searches and returns documents regarding the roman empire."
)
tools = [tool]


from langchain.agents.agent_toolkits import create_conversational_retrieval_agent

from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(temperature = 0)

agent_executor = create_conversational_retrieval_agent(llm, tools, verbose=True)

result = agent_executor({"input": "Hello, I would like to know about the roman empire."})