The Use Case

Managing secrets is hard. Everything needs to run under its own username/password, and apparently keeping plaintext passwords in scripts is really bad.

To manage secrets better at our company we already implemented the following practices:
1) Avoiding hardcoding credentials (or any other configuration data) in the scripts, and commit scripts and configs separately.

2) Not actually storing real values in the config files: Instead, use placeholders like #{sqlserverPassword}# as values, and use a CI Task to replace the tokens with the actual values when deploying the script (many CI platforms have this feature out-of-the-box).

So even though we don't store passwords in Source Control anymore, when the scripts are deployed, the files may still contain passwords in plaintext.

You could use Integrated Security to avoid using passwords altogether, but this is only an option if your workers and resources are in the same Windows Domain.
You can generate SecureStrings and have your script users use those instead. It's pretty secure because only users can decrypt the SecureStrings that the same user had created on the same machine. But the downside is only users can decrypt the SecureStrings that the same user had created on the same machine 😅

So how to solve this?

First, we need a method for adding and retrieving secrets: Use Microsofts new SecretsManagement Module.

Note: The Module is still in PreRelease, but it can Add Secrets and Get Secrets, so good enough for our use case for now

So what is this Module?

The Secrets Management module helps users manage secrets by providing a set of cmdlets that let you store secrets locally, using a local vault provider, and access secrets from remote vaults. This module supports an extensible model where local and remote vaults can be registered and unregistered on the local machine, per user, for use in accessing and retrieving secrets.

Notice that it says per user: Users can only access their own local vault (assuming we are not using external vaults like Azure KeyVault, which probably would make this whole tutorial unnessecary).
To have the password available for the user to use, we must perform an Add-Secret command under the user context, so that the user can do Get-Secret to retrieve the password stored in their own vault.
The workflow would be something like: Invoke a command with the User Credentials passed as -Credential, executing a Scriptblock { Add-Secret -Name MySecret -Secret SuperSecretPassword } .

🚦 If there is a better/different way to provide secrets under different user contexts, please let me know.
For now, we continue this route. 🚦

The PowerShell Script to Add Secrets

I tried many methods to perform commands under a different user context:

• use Invoke-Command -ScriptBlock $Scriptblock -Credential $creds
  • why not? This requires WinRM to be available for the given user, even if the command is run on localhost, so I looked for something else.
• the Start-Job -ScriptBlock $Scriptblock -Credential $creds | Wait-Job | Receive-Job Combo
  • why not? This worked pretty consistently, up until to point I tried to integrate this in a CI Task. I got stuck getting ❌2100,PSSessionStateBroken errors, and it seems to be a common issue in CI systems. Apparently it has something to do with credentials not being passed while 'double hopping', resolving this would be again setting up WinRM in conjunction with something called CrepSSP.

So finally I ended up using the Invoke-CommandAs Module, which under the hood creates a Scheduled Task as the user, carries out your commands you specify in a $ScriptBlock, and finally removes the Task.
An example snippet for adding 1 secret for 1 user on 1 machine would be:

$userName = "domain\user_account"
$userPassword = "userPassword" # ✋Warning If the password contains $ signs, use single quotes!        
[pscredential]$credObject = New-Object System.Management.Automation.PSCredential ($userName,         $userPassword)            
$Credentials = Get-Credential $credObject 
$ScriptBlock = {          
   If ( -not(Get-Module Microsoft.PowerShell.SecretsManagement -listAvailable) ) {
      Write-Host "Secret Module not found, installing.."
      [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
      Install-Module -Name Microsoft.PowerShell.SecretsManagement -RequiredVersion 0.2.0-alpha1 -AllowPrerelease -Repository psgallery -Force -Scope CurrentUser
   }
   # proof that it worked:
   Write-Host $env:computername
   Write-Host $env:username
   Get-SecretInfo
}
Invoke-CommandAs -Scriptblock $ScriptBlock -AsUser $Credentials

Note: Notice the Tls Securtity protocol, I almost flipped my desk when I couldn't figure out why the Module wouldn't download, until I saw this

Problem: The user needs the right amount of user rights permissions to pull this off. I haven't fully figured out the exact set of permissions, I thought the user only needs the local log on as a batch job permission, but couldn't get this to work consistenly.
Solution: So I decided to throw this hack in for now: I ended up temporaly adding the user to the local Administrators group, add the secrets, and then remove it again from the group.

NET LOCALGROUP "Administrators" $userName /ADD | Out-Null
Invoke-CommandAs -Scriptblock $ScriptBlock -AsUser $Credentials
NET LOCALGROUP "Administrators" $userName /remove| Out-Null
##📣 If anyone has a better idea, please let me know :halp: 📣

🎉 Great, now that we have cobbled together a working script, we can now use this for every secret, for each user, on every machine that the user operates on, right? Also, apparantly current security conventions suggest that we should run each service as a different user, therefore increasing the amount users we have to manage 🤯. You might see how this can be a bit tedious to manage manually.
But here comes Ansible.

The Ansible Playbook

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

If you haven't played around with Ansible yet, I suggest watching the live-streams of this Jeff Geerling guy or check out his Ansible book (which is currently free)

NOTE: If there are more 'ansible' ways to achieve this use-case, please let me know.

The highlevel workflow would be:

• hosts: list of servers

• variables:

  • accounts: a dictionary where key = accountname, value = password
  • account_mapped_secrets: a dictionary where key = accountname, value= list of secret-keys
  • secrets: a dictionary where key = secret-key, value=secret-value

• playbook:

  • Get list of accounts that have secrets mapped
  • Loop over accounts, and for each account
  • construct a secret key/value disctionary for each account_secret of the account
  • use the above Invoke-CommandAs powershell snippet under the account user context to add each secret-key/value

A play would look something like this:

• the main.yml file

- name: Loop over Accounts that has Secrets to deployed
  include_tasks: add_secrets.yml  
  with_items: "{{ account_mapped_secrets }}"  
  loop_control:  
    loop_var: account  
    extended: yes

• the included tasks add_secrets.yml file

---  
- name: "${{ account }}$ -- Create temp Vault dict"  
  set_fact:  
    account_secrets_{{ ansible_loop.index }}: {}  

- name: "${{ account }}$ -- Populate Secrets"  
  set_fact:  
   account_secrets_{{ ansible_loop.index }}: "{{ lookup('vars', 'account_secrets_' ~ ansible_loop.index) |default({}) | combine( {item: secrets[item]} ) }}"  
  with_items: "{{ account_mapped_secrets[account] }}"  

- name: "${{ account }}$ -- Add Secrets to Local Vault"  
  win_shell: |  
   $userName = '{{ account }}'  
   $userPassword = '{{ accounts[account] }}'  
   [securestring]$secStringPassword = ConvertTo-SecureString $userPassword -AsPlainText -Force  
   [pscredential]$credObject = New-Object System.Management.Automation.PSCredential ($userName, $secStringPassword)  

   $ScriptBlock = {  

   Get-SecretInfo | Remove-Secret -Vault BuiltInLocalVault  #✋for demo purposes we delete any existing Secrets.
   $json = @"  
   {{ lookup('vars', 'account_secrets_' ~ ansible_loop.index) | to_nice_json }}  
   "@  
   ($json | ConvertFrom-Json).psobject.properties | ForEach-Object { Add-Secret -Name $_.Name -  Secret $_.Value }  
   Get-SecretInfo  

   }  
   try {  
     NET LOCALGROUP "Administrators" $userName /ADD | Out-Null  
     Invoke-CommandAs -ScriptBlock $ScriptBlock -AsUser $credObject -verbose  
     NET LOCALGROUP "Administrators" $userName /DELETE | Out-Null  
   } catch {  
     Write-Error $_  
   }  
  register: shellresult  

- name: "${{ account }}$ -- Fail if output is not exptected"  
  fail:  
    msg: shellresult.stdout_lines  
  when: shellresult.stdout.find("Vault") == -1  

- name: "${{ account }}$ -- Show Errors"  
  fail:  
    msg: "{{ shellresult.stderr }}"  
  when: shellresult.stderr != ""  

- name: "${{ account }}$ -- Print Result"  
  debug:  
    msg: "{{ shellresult.stdout_lines }}"

✋ You need to setup an Ansible Role to make this snippet work as it is.
My directory looks someting like this:

/etc/ansible  
├── ansible.cfg  
├── environs  
│ ├── dev                     # copy this subfolder for each environment (dev,test,prod...) you might have  
│ │ ├── group_vars  
│ │ │ └── all  
│ │ │      ├── main.yaml      # contains account_mapped_secrets variable
│ │ │      └── accounts.yaml  # contains accounts variable
| | |      └── secrets.yaml   # contains secrets variable
│ │ ├── hosts  
....
├── roles  
│ ├── secrets  
│ │ ├── tasks  
│ │ │ ├── add_secrets.yml     # the core of the play
│ │ │ └── main.yml            # the main tasks file that includes the above file in a loop
├── play_secrets_role.yaml    # a play that calls the secrets role

To run this play for the dev environment, run: ansible-playbook -i environs/dev play_secrets_role.yaml

Explaining the playbook:

• You can notice that I'm construcing an unique dictionary in each loop account_secrets_{{ ansible_loop.index }}. In the first versions of my playbook I just used account_secrets as the variable, and apparantly in Ansible, once you set a fact, you can't unset it. So the secrets dictionary would just keep appending between each loop, and the last user would end up having all the secrets.

• ansible_loop.index is the current iteration in the loop, but is only available when you have the extended: yes option when looping.

• We use a jinja combine() expression to dynamically create the account_secrets_x dictionary. Note that I refer to the above variable with the lookup('vars', 'account_secrets_' ~ ansible_loop.index) syntax instead of the account_secrets_{{ ansible_loop.index }}.
  This is because the following won't work: "{{ account_secrets_{{ ansible_loop.index }}) |defaul t({}) | combine( {item: secrets[item]} ) }}" as you can't have double interpolation inside a jinja expression.

Wait a minute, you're still storing plaintext passwords in the variable files!

Here comes Ansible again. By separate the accounts and secrets from the account_mapped_secrets into different variable files we can encrypt the first two with Ansible Vault
Then when running your ansible-playbook, pass in the extra argument --ask-vault-pass