RAG pipeline with up-to-date knowledge: get answers based on increasing number of documents

This example implements a pipeline that answers questions based on documents in a given folder. To get the answer it sends increasingly more documents to the LLM chat until it can find an answer. You can read more about the reasoning behind this approach here.

Each query text is first turned into a vector using OpenAI embedding service, then relevant documentation pages are found using a Nearest Neighbor index computed for documents in the corpus. A prompt is built from the relevant documentations pages and sent to the OpenAI chat service for processing.

To optimize use of tokens per query, this pipeline asks a question with a small number of documents embedded in the prompt. If OpenAI chat fails to answer based on these documents, the number of documents is increased by factor given as an argument, and continues to do so until either question is answered or a limit of iterations is reached.

How to run the project

Setup environment:

Set your env variables in the .env file placed in this directory.

OPENAI_API_KEY=sk-...
PATHWAY_DATA_DIR= # If unset, defaults to ./data/. If running with Docker, when you change this variable you may need to change the volume mount.
PATHWAY_PERSISTENT_STORAGE= # Set this variable if you want to use caching

Run with Docker

To run jointly the Alert pipeline and a simple UI execute:

docker compose up --build

Then, the UI will run at http://0.0.0.0:8501 by default. You can access it by following this URL in your web browser.

The docker-compose.yml file declares a volume bind mount that makes changes to files under data/ made on your host computer visible inside the docker container. The files in data/live are indexed by the pipeline - you can paste new files there and they will impact the computations.

Run manually

Alternatively, you can run each service separately.

Make sure you have installed poetry dependencies with --extras unstructured.

poetry install --with examples --extras unstructured

Then run:

poetry run python app.py

If all dependencies are managed manually rather than using poetry, you can alternatively use:

python app.py

To run the Streamlit UI, run:

streamlit run ui/server.py --server.port 8501 --server.address 0.0.0.0

Querying the pipeline

To query the pipeline, you can call the REST API:

curl --data '{
  "user": "user",
  "query": "How to connect to Kafka in Pathway?"
}' http://localhost:8080/ | jq

or access the Streamlit UI at 0.0.0.0:8501.