DEV Community

Manuel Sidler
Manuel Sidler

Posted on

Setup your Azure Search with PowerShell

#azure #powershell #microsoftazure

Microsoft provides cmdlets through Azure PowerShell and a REST API to manage your Azure Search resource.

In this scenario we need a Azure Search service which will:

  • extract storage metadata from an Azure Blob container
  • store the metadata inside an index
  • provide a custom key to query our index

To fullfil these requirements we're going to use a combination of the Azure PowerShell cmdlets and REST API endpoints to setup an Azure Search with the following components:

  • Azure Search service
  • Search query key
  • Search index connected to an Azure Blob storage

In order to achieve this, we're going to take these steps:

  1. Define common variables
  2. Login to Azure
  3. Create search service
  4. Create search query key
  5. Get search primary admin key
  6. Create index
  7. Create data source
  8. Create indexer

Prerequisites

Variables

First of all we need to define some variables which we'll use throughout the upcoming steps.

$tenantId = "<tenantId>" # eg. 12133905-3bce-414e-983f-79013911deee
$subscriptionId = "<subscriptionId>" # eg. 510f6c5b-b136-45e7-b239-dccb600331ea
$resourceGroup = "<resourceGroup>" # eg. my-project
$location = "<location>" # eg. WestEurope
$storageAccountName = "<storageaccount>" # eg. myprojectstorage
$searchName = "<searchName>" # eg. my-project-search
Enter fullscreen mode Exit fullscreen mode

Login

To get access to our Azure resources we need to connect to our Azure account:

Connect-AzAccount -Tenant $tenantId -Subscription $subscriptionId
Enter fullscreen mode Exit fullscreen mode

This cmdlet will open a popup where you can enter your credentials for your Azure account.

Create search service

Assuming the login was successful we're now able to create the search service:

$sku = "Free" # Free|Basic|S1|S2|S3|S3-HD
New-AzSearchService -Name $searchName -ResourceGroupName $resourceGroup -Sku $sku -Location $location
Enter fullscreen mode Exit fullscreen mode

Create search query key

We also need an additional search query key in our scenario:

$queryKey = "MyQueryKey"
New-AzSearchQueryKey -Name $queryKey -ServiceName $searchName -ResourceGroupName $resourceGroup
Enter fullscreen mode Exit fullscreen mode

Get search primary admin key

Unfortunately Azure PowerShell doesn't provide cmdlets to manage your search indexes, indexers and data sources. To create these resources we'll put the Azure REST API into action. But first we need to fetch the search primary admin key which we're going to use as an API key:

$adminKeyPair = Get-AzSearchAdminKeyPair -ResourceGroupName $resourceGroup -ServiceName $searchName
$primaryAdminKey = $adminKeyPair.Primary
Enter fullscreen mode Exit fullscreen mode

Create index

Instead of using a variable for the index definition, I would like to store it as JSON inside a folder. Once we stored the index definition, we can iterate through them when actually calling the Create Index endpoint. We're going to keep this index as simple as possible. Fields:

  • ID (Primary Key)
  • Title
  • Tags (Filterable Collection)
  • metadata_storage_path (Path of the Blob)

/indexes/myindex.json

{
    "name": "my-index",  
    "fields": [
        {"name": "ID", "type": "Edm.String", "key": true, "filterable": false, "sortable": false, "facetable": false, "searchable": false},
        {"name": "TITLE", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
        {"name": "TAGS", "type": "Collection(Edm.String)", "filterable": true, "sortable": false, "facetable": false, "searchable": false},
        {"name": "metadata_storage_path", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false}
    ]
}
Enter fullscreen mode Exit fullscreen mode

With our definition we're now able to create our index:

Get-ChildItem "indexes" -Filter *.json |
        ForEach-Object {
            $indexDefinition = Get-Content $_.FullName -Raw

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

            $url = "https://$searchName.search.windows.net/indexes?api-version=2017-11-11"

            Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $indexDefinition | ConvertTo-Json
        }
Enter fullscreen mode Exit fullscreen mode

Create data source

We take the same approach for creating the data source. But in this case we need to set the connection string from our storage account. As we're good developers we won't store the string inside the data source JSON definition ;-)

/datasources/mydatasource.json

{   
    "name" : "my-datasource",
    "type" : "azureblob",
    "credentials" : { 
        "connectionString" : "[STORAGECONNECTION]" 
    },  
    "container" : { 
        "name" : "myblobcontainer" 
    }
}
Enter fullscreen mode Exit fullscreen mode

To replace the STORAGECONNECTION placeholder we first need to fetch the connection string:

$storageKey = (Get-AzStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroup)[0].Value

$storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=$storageAccountName;AccountKey=$storageKey;EndpointSuffix=core.windows.net"
Enter fullscreen mode Exit fullscreen mode

With the storage connection string available we now can iterate through the data source folder and use the Create Data Source endpoint to create the data source:

Get-ChildItem "datasources" -Filter *.json |
        ForEach-Object {
            $datasourceDefinition = (Get-Content $_.FullName -Raw).replace("[STORAGECONNECTION]", $storageConnectionString)

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

             $url = "https://$searchName.search.windows.net/datasources?api-version=2017-11-11"

             Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $dataSourceDefinition | ConvertTo-Json
        }
Enter fullscreen mode Exit fullscreen mode

Create indexer

Last but not least we have to create an indexer. Again we use a JSON file to define the indexer. What's important here is that in our scenario we just want to index the storage metadata. You could also choose to extract all metadata or the file content.

/indexers/myindexer.json

{
    "name" : "my-indexer",
    "dataSourceName" : "my-datasource",
    "targetIndexName" : "my-index",
    "schedule" : {
        "interval" : "PT1H"
    },
    "parameters" : {
        "configuration" : {
            "dataToExtract" : "storageMetadata"
        }
     }
}
Enter fullscreen mode Exit fullscreen mode

To create the indexer we're going to use the Create Indexer endpoint:

Get-ChildItem "indexers" -Filter *.json |
        ForEach-Object {
            $indexerDefinition = Get-Content $_.FullName -Raw

            $headers = @{
                'api-key' = $primaryAdminKey
                'Content-Type' = 'application/json' 
                'Accept' = 'application/json' }

            $url = "https://$searchName.search.windows.net/indexers?api-version=2017-11-11"

            Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $indexerDefinition | ConvertTo-Json
        }
Enter fullscreen mode Exit fullscreen mode

Conclusion

We now have an Azure Search service which:

  • provides a custom key to query our index
  • extracts storage metadata from an Azure Blob container
  • stores the metadata inside an index

search query

Top comments (0)

Read next

bishopryant profile image

Brainstorming with GenAI

Ryan Bishop -

0x3d_site profile image

Launching Python.0x3D.site – Your Go-To Resource for All Things Python!

0x3d Site -

william_angani_569608dc03 profile image

SEMANTIC HTML

William Angani -

jsha profile image

Practical application of grid

Julia Shlykova -