Welcome to Pathway Developer Documentation!
Pathway is a Python data processing framework for analytics and AI pipelines over data streams. It’s the ideal solution for real-time processing use cases like streaming ETL or RAG pipelines for unstructured data.
Pathway comes with an easy-to-use Python API, allowing you to seamlessly integrate your favorite Python ML libraries. Pathway syntax is simple and intuitive, and you can use the same code for both batch and streaming processing.
Pathway is powered by a scalable Rust engine based on Differential Dataflow and performing incremental computation. Your Pathway code, despite being written in Python, is run by the engine, enabling multithreading, multiprocessing, and distributed computations. All the pipeline is kept in memory and can be easily deployed with Docker and Kubernetes.
Quick Install
To quickly get started with Pathway, you can install it via pip with the following command:
pip install -U pathway
⚠️ Pathway is available on MacOS and Linux. Pathway is currently not supported on Windows. Windows users may want to use Windows Subsystem for Linux (WSL), docker, or a VM.
To jumpstart a Pathway project, quickly check our cookiecutter template.
Try Pathway in just a few clicks
Ready to see what Pathway can do? Try one of our easy-to-run examples! Available in both notebook and docker formats, these ready-to-launch examples can be launched in just a few clicks. Pick one and start your hands-on experience with Pathway today!
Optional packages
| Package | Installation Command | Description | Notes |
|---|---|---|---|
| Basic LLM Tooling | pip install "pathway[xpack-llm]" | Install common LLM libraries (OpenAI, Langchain, LlamaIndex) | Learn more / Examples |
| Local LLM Deployment | pip install "pathway[xpack-llm-local]" | Libraries for local deployment | |
| Parsing Documents | pip install "pathway[xpack-llm-docs]" | Tools for working with documents (PDFs, Microsoft Word) | Contextful Parsing Pipeline |
| Airbyte Connector | pip install "pathway[airbyte]" | Support for Airbyte | Example |
| SharePoint Connector | pip install "pathway[xpack-sharepoint]" | Support for SharePoint | Requires a (free) license key |
| All | pip install "pathway[all]" | Install all the optional packages |
Docker
You can also use Pathway with Docker. The official Pathway Docker image is available on Docker Hub. You can pull and run the image using the following command:
docker pull pathwaycom/pathway
For more detailed instructions on how to run Pathway with Docker, please refer to our dedicated article.
License
Pathway is distributed on a BSL 1.1 License which allows for unlimited non-commercial use, as well as use of the Pathway package for most commercial purposes, free of charge. The code in the associated repository automatically converts to Open Source (Apache 2.0 License) after 4 years. Some public repos which are complementary to this one (examples, libraries, connectors, etc.) are licensed as Open Source, under the MIT license.
Some features of Pathway such as monitoring or advanced connectors (e.g., SharePoint) require a free license key. To obtain a free license key, you need to register here.
Learn more
Not sure how to use a specific feature of Pathway? The answer to your question is likely in the API docs.
See the API docsPick your use case. Connect data and run pipelines in minutes with Pathway App Templates.
Try Pathway App Templates!Curious about how Pathway works? Don't hesitate to take a look at the sources and clone the repo.
Go to GithubSelf-host your Pathway service with Docker, Kubernetes, or quickly launch a hosted container.
Deploy in one click