Photo by Matthew Dockery on Unsplash
5th part of a series
building the foundation -part 1 &adding your own data -part 2 &
adding a chat history - part 3 &
adding channels and taking actions on the results part 4were published before.
Motivation
When working with your own data and your ownGPT, you can enhance the results and improve the user experience by taking a few small steps in data preparation.
Prepare Data Locally
To prepare your data locally, follow the steps in this section. This method is suitable for small data sets.
For much larger sets, use Azure machine learning (see below).
Setup
To prepare your local machine, you need to install the necessary packages. You can find them in the requirements.txt file in the repository of part 1 of this series.
Configuration
To create a config file, name it "config.json" and use the following format. The file should contain a list of JSON objects. Each object should have two keys: one for the local data path and another for the target search service and index.
[
{
"data_path": "<local path or blob URL>",
"location": "<azure region, e.g. 'westus2'>",
"subscription_id": "<subscription id>",
"resource_group": "<resource group name>",
"search_service_name": "<search service name to use or create>",
"index_name": "<index name to use or create>",
"chunk_size": 1024, // set to null to disable chunking before ingestion
"token_overlap": 128 // number of tokens to overlap between chunks
"semantic_config_name": "default",
"language": "en" // setting to set language of your documents. Change if your documents are not in English. Look in data_preparation.py for SUPPORTED_LANGUAGE_CODES,
"vector_config_name": "default" // used if adding vectors to index
}
]
Note:
data_path
can be either a local path to files on your machine or an Azure Blob URL. An Azure Blob URL has the formathttps://<storage account name>.
blob.core.windows.net/<container
name>/<path>/
. If you use a blob URL, the data will be downloaded from Blob Storage to a temporary directory on your machine before the data preparation starts.
Create Indexes and Ingest Data
Disclaimer:
To avoid compromising the quality of the responses, please ensure that your data does not contain any duplicate pages.
- Run the data preparation script, passing in your config file. You can set njobs for parallel parsing of your files.
Use Azure Machine Learning to Prepare Data
Setup
- Install the Azure ML CLI v2
Prerequisites
Azure Machine Learning (AML) Workspace with associated Keyvault
Azure Cognitive Search (ACS) resource
(Optional if processing PDF) Azure AI Document Intelligence resource
(Optional if adding embeddings for vector search) Azure OpenAI resource with Ada (text-embedding-ada-002) deployment
(Optional) Azure Blob Storage account
Configure
Create secrets in the AML Keyvault for the Azure Cognitive Search resource admin key, Document Intelligence access key, and Azure OpenAI API key (if applicable).
Create a configuration file, such as
aml_config.json
. The format can be a single JSON object or a list of JSON objects, with each object specifying a combination of Key Vault secrets, chunking settings, and index configurations.
{
"chunk_size": 1024,
"token_overlap": 128,
"keyvault_url": "https://<keyvault name>.vault.azure.net/",
"document_intelligence_secret_name": "myDocIntelligenceKey",
"document_intelligence_endpoint": "https://<document intelligence resource name>.cognitiveservices.azure.com/",
"embedding_key_secret_name": "myAzureOpenAIKey",
"embedding_endpoint": "https:/<azure openai resource name>.openai.azure.com/openai/deployments/<Ada deployment name>/embeddings?api-version=2023-06-01-preview",
"index_name": "<new index name>",
"search_service_name": "<search service name>",
"search_key_secret_name": "mySearchServiceKey"
}
Create an Azure Machine Learning Datastore.
If your data is stored in Azure Blob Storage, you can initially create an AML Datastore that will be utilized to connect to your data during the ingestion process. Update the datastore.yml
file with your storage account information, including the account key. Afterward, execute the following command, incorporating the resource group and workspace name of your AML workspace:
az ml datastore create --resource-group <workspace resource group> --workspace-name <workspace name> --file datastore.yml
Submit the data processing pipeline job
In pipeline.yml
, update the inputs to reference your config file name and the datastore you created. If you are using data stored locally, comment out the datastore path and uncomment the local path, updating it to point to your local data location. Afterward, submit the pipeline job to your AML workspace:
az ml job create --resource-group <workspace resource group> --workspace-name <workspace name> --file pipeline.yml
Convert PDFs to Text
If your data is in PDF format, you must convert it from PDF to .txt format. You can use your own script for this task or utilize the provided conversion code (see repository in part 1) here.
Setup
Create a Form Recognizer resource in your subscription
Ensure that you have the Form Recognizer SDK.
Run the following command to obtain an access key for your Form Recognizer resource.
Create Indexes and Ingest Data from PDFs using Form Recognizer.
Provide your Form Recognizer resource name and key when executing the data preparation script:
python data_preparation.py --config config.json --njobs=4 --form-rec-resource <form-rec-resource-name> --form-rec-key <form-rec-key>
By default, this will utilize the Form Recognizer Read model.
Consider using the Form Recognizer Layout model if your documents contain numerous tables and significant layout information. Although it is more expensive and slower to run, it preserves higher-quality table information. Additionally, the Layout model helps maintain formatting information in your document, such as titles and subheadings, making citations more readable. Pass in the argument to use the Layout model instead of the default Read model. --form-rec-use-layout
.
python data_preparation.py --config config.json --njobs=4 --form-rec-resource <form-rec-resource-name> --form-rec-key <form-rec-key> --form-rec-use-layout
Conclusion
Refining and enhancing your ownGPT with your own data can significantly improve the user experience and the quality of generated responses. By preparing data locally or using Azure Machine Learning, you can efficiently process and index your data, incorporating additional features like PDF conversion and table recognition. This approach allows you to unlock the full potential of your ownGPT model, ultimately creating a more robust and personalized AI tool.
Top comments (0)