add examples

Signed-off-by: Matt Williams <m@technovangelist.com>

examples/langchain-typescript-selfqueryingretrieval/readme.md

@@ -0,0 +1,109 @@
+# Self Query Retrieval
+
+Filtering your vector database results to get better answers from your LLM.
+
+![sqr 2023-11-05 14_30_50](https://github.com/jmorganca/ollama/assets/633681/55afb7f5-ebd8-4c58-86ba-284594fd1ec8)
+
+## TLDR
+
+1. Install and run ChromaDB
+   1. Run `git clone https://github.com/chroma-core/chroma.git`
+   2. `cd chroma`
+   3. `docker-compose up -d --build`
+2. Navigate to this example's directory
+3. `npm install`
+4. `tsx ./GenerateSource.ts`
+5. `tsx ./FindArt.ts "are there any paintings from the artist Pablo Picasso"`
+
+Other questions to try:
+
+- Are there any paintings painted in 1881
+- Are there any paintings painted by Vincent van Gogh
+
+Note: If you haven't used `tsx`, it's a more modern alternate to `ts-node` and works especially well when you have libraries that use different module types. You can find it at [https://github.com/esbuild-kit/tsx](https://github.com/esbuild-kit/tsx).
+
+## Introduction
+
+Retrieval Augmented Generation (RAG) is what developers usually reach for when they want to ask questions to all of their notes. But often it doesn't give the results you need. And that's because there is still too much information. And frequently it's the wrong information. When you ask a question, RAG will retrieve a set of documents that it thinks are relevant to the question and then hand them off to the LLM. If you ask "what is a transformer", it may grab excerpts from the Transformers paper you read recently, along with sections of your Intro to Electronics book. Even if you ask a better question, such as "what is a transformer in the context of electrical engineering", it may still grab excerpts from the Transformers paper. And that's because the Transformers paper is a very good match for the question. It's just not the right match.
+
+Ideally, the Transformers paper and the Electronics book would be added to the database with some metadata, such as the topics or keywords. But RAG typically doesn't look at those metadata fields. And that's where Self Query Retrieval comes in. It's a way to use traditional database queries to narrow down the set of documents that RAG will use and thus get better results.
+
+## How it works
+
+There are a few things you need to do to enable Self Query Retrieval. First, there needs to be additional metadata about your content in the database. The examples in the Langchain documentation are based on movies, and the metadata includes the year, the director's name, the genre, etc. And then you need to pass the schema to the query to help it get the right documents.
+
+## The code
+
+There are two main parts to the code. First there is a GenerateSource.ts file and then there is a FindArt.ts file. Let's look at GenerateSource first.
+
+### GenerateSource
+
+The purpose of Generate Source is to create our data source. For this example, we are using the [Chicago Institute of Art API,](https://api.artic.edu/docs/#introduction) which is incredible. This will be loaded into a vector database, which for this example is ChromaDB.
+
+This could be any CSV file or other data source you have access to. The file would have a single descriptive column and then metadata columns. All the relevant columns from our dataset are being added to a Document object. Then that array of Documents is being loaded into ChromaDB. Finally, at the end, I verify that documents were created by outputting a count to the screen.
+
+```typescript
+  await new ChromaClient().deleteCollection({
+    name: "artcollection",
+  });
+
+  const vectorStore = await Chroma.fromDocuments(allartworkdocs, 
+        embedding, { collectionName: "artcollection" });
+  console.log(`Created vector store with 
+        ${await vectorStore.collection?.count()} documents`);
+```
+
+### FindArt
+
+To actually find the art, we need to start by loading the database:
+
+```typescript
+  const vectorStore = await Chroma.fromExistingCollection(embeddings, {
+    collectionName: "artcollection",
+  });
+```
+
+Now we can create our Self Query Retriever. This needs to be created referring to the LLM, the database, the description of the document and the description of all the attributes in the metadata, and finally a structured query translator which will take the query generated by the LLM and turn it into something useable by the database.
+
+```typescript
+const llm = new Ollama({
+  model: modelName
+})
+const documentContents = "Description of the art";
+
+const attributeInfo: AttributeInfo[] = [
+  {
+    name: "title",
+    type: "string",
+    description: "The title of the painting"
+  },
+  {
+    name: "date",
+    type: "integer",
+    description: "The four digit year when the painting was created"
+  },
+  {
+    name: "artistName",
+    type: "string",
+    description: "The first name and last name of the artist who created the painting. Always use the full name in the filter, even if it isn't included. If the query is 'van Gogh', the filter should be 'Vincent van Gogh'. Use Pierre-Auguste Renoir instead of just Renoir."
+  }
+]
+
+const retriever = SelfQueryRetriever.fromLLM({
+  llm, vectorStore, documentContents, attributeInfo, verbose: false, useOriginalQuery: true, structuredQueryTranslator: new ChromaTranslator()
+});
+```
+
+Now we can ask a question and get the results:
+
+```typescript
+const newquery = await retriever.getRelevantDocuments(query)
+```
+
+## Next Steps
+
+When you run this example, you will get a set of documents from the database that may be a bit more relevant to your question. Now you could feed those to the LLM and get the actual answer to the question based on these documents.
+
+To take this further, you could work on getting more out of the dataset. It turns out that this works best if there is only a single possible value for any given field. Our artists are often referred to by their last name, but sometimes using their full name. It may be Vincent van Gogh, or just van Gogh. Another way to get around this is to build a better query translator that knows that the search could be for a substring of the full name. But that also requires looking into the metadata searching capabilities of the database.
+
+Maybe it makes more sense to move the artist name and title of the work into the document itself. Then add some more metadata (there are at least 100 other attributes in the raw API that aren't used in this example.)