Azure: Azure Resource Manager (ARM) templates for creating Virtual Machines for Standard Window's SKU's

Creating Virtual Machines (VMs) with the Azure portal is convenient but it can become tedious if numerous VMs are needed and it can be an error prone process. The New-AzVM PowerShell cmdlet can automate the creation of virtual machines. For rudimentary VMs, New-AzVM is straightforward to use from PowerShell. When this cmdlet is used to create virtual machines with complex restrictions that utilize the numerous parameter combinations, coding with New-AzVM can be a daunting task (see "Azure/PowerShell: Cmdlet Parameter and Result Complexity (Get-AzHost, Get-AzVM, New-AzVM)" for an overview of the parameter sets associated with New-AzVM and an example of the elaborate parameters that can be passed to New-AzVM).

One way to simplify the programmatic creation of a virtual machine is to use Azure Resource Manager (ARM) templates. Using PowerShell in conjunction with the New-AzVM cmdlet or ARM templates is an example of Infrastructure as Code (IAC). Instead of using portal and manual object creation, the code  used to create the infrastructure (Azure objects) can be checked into a source code control repository such as Git and treated like a first class Git citizen.

ARM templates are JSON format and describe how Azure entities are to be created. ARM templates can be created using Azure portal where instead of creating an object such as a VM, the configuration specified in Azure portal can be saved as a template. The templates saved do not magically create infrastructure objects. The template and its corresponding parameters are passed as command-line options to the New-AzResourceGroupDeployment cmdlet. Potentially the templates and their parameters are modified such as to create ten virtual machines named VM00 to VM09 which is simply using PowerShell to update the JSON attribute associated with virtual machine's name.

To demonstrate ARM templates, the steps to create virtual machine with the Azure portal will be demonstrated but instead of creating a VM, a template and parameter file will be generated for later use by New-AzResourceGroupDeployment.

The standard Azure portal appears as follows with a familar Virtual machines icon which when clicked on displays the virtual machine blade:

The virtual machine blade (screen) has an option to add a virtual machine which to no surprise is labeled by the word "Add":

Clicking on the Add option displays the "Create a virtual machine" page:

The desired parameters for a virtual machine can be filled in using the "Create a virtual machine page". For this scenario a VM is created the uses the image Windows Server 2012 R2 Datacenter - Gen 1. The image name "Windows Server 2012 R2 Datacenter - Gen 1" is a particular Windows SKU available to newly created Azure VMs.  Once all the parameters have been specified for the VM to be created the "Review + create" button can be clicked on:

After "Review + create" is clicked on, a set of validations are performed for the virtual machine configuration specified. An example validation might be that a virtual machine for a given Resource Group might not be permitted to expose a public IP address. Not permitting a public IP address  is standard-operation-procedure in environments that use VPNs to access VMs via private IP address. Another validation is the enforcement of password complexity for the default administrator account created.

The following screen is displayed after "Review + create" is clicked and the specified configuration has passed validation:

The command actions on the page above include a link (lower right) labeled as "Download a template for automation" (see below):

Clicking "Download a template for automation" link displays the following:

The Template tab allows the template to be selected in the right pane of the screen (see above). Notice above the options to Download, Add to library (preview), and Deploy are provided. Clicking on the Parameters tab displays the parameters associated with the template (see below):

Clicking on the Scripts tab displays the following:

Clicking on "Start" on the PowerShell tile displays the following (Manage Azure resources by using Azure PowerShell):

The New-AzResourceGroupDeployment cmdlet takes as command-line options a template file and a parameters file and deploys object specified by the template.

PowerShell: Two Interview Questions

 During a recent interview I was asked the following questions.

How do you remove duplicates from an array?

My answer was to perform a Select-Object on the array as there must be an command-line option to return only unique values. The Select-Object cmdlet behaves like C#'s LINQ and removing duplicates is something I would do in LINQ.

The answer is Select-Object -Unique:

[int[]] $arrayWithDuplicates = 1, 2, 3, 3, 3, 2, 1, 1, 3, 1, 2, 3

[int[]] $arrayWithUniqueValues = 

    $arrayWithDuplicates | Select-Object -Unique

$arrayWithUniqueValues -join ' '

The output of the above script is:

1 2 3

How do you read the first five lines of a file?

My answer was to perform a Get-Content on the file as there must be an command-line option to return only finite number of lines from the file. 

The answer is Get-Content <filename> -TotalCount 5 where an example is as follows:

Get-Content 'InterviewQuestions.ps1' -TotalCount 5

Azure/PowerShell: Cmdlet Parameter and Result Complexity (Get-AzHost, Get-AzVM, New-AzVM)

Overview

The concept of cmdlet parameter sets was discussed in "PowerShell: Cmdlet Parameter Sets". The motivation for introducing this topic was to show the complexity of certain PowerShell cmdlets and in a future posts how to escape this complexity (foreshadow: Azure Resource Manager templates). An example of a further complexity is that a cmdlet can have different return values and can return single values or arrays. Azure cmdlets can take dozens of parameters and contain multiple parameter sets. Making cmdlets more complicated is that certain cmdlets uses parameters that are created by invoking several different cmdlets which extend the values associated with the parameter. For example the New-AzVM can create an Azure VM using a parameter of type PSVirtualMachine. The PSVirtualMachine parameters could have been setup by invoking multiple cmdlets (New-AzVMConfig, Set-AzVMOperatingSystem, Add-AzVMNetworkInterface, Set-AzVMOSDisk). Debugging an invalid parameter passed to a cmdlet such as New-AzVm can be a nightmare.

In order to better understand cmdlet parameter sets and there complexity consider the documentation of several Azure related cmdlets:

• Get-AzHost: retrieves a host or a list of all hosts if no host name is specified
• Get-AzVM: retrieves the properties of an Azure VM
• New-AzVM: creates an Azure VM

The documentation for each of these cmdlets is broken into sections where each shaded rectangle corresponds to a different parameter set. I am color blind so I have no idea if the shaded rectangles contain a color or are simply shades of gray. 

PowerShell cmdlets such as those referenced above return inconsistent types. For example, the Get-AzVM cmdlet can return either of the following types depending on the parameters specified:

PSVirtualMachine

PSVirtualMachineInstanceView

Even when a PowerShell cmdlet is documented to return a specific type or types this may not be the case. Cmdlets can return a single value or an array of values. 

Get-AzHost

The documentation for Get-AzHost shows two parameter sets each identified by shaded rectangles:

The output for each parameter set appears to be consistent as the return value is documented as follows:

Microsoft.Azure.Commands.Compute.Automation.Models.PSHost

If Get-AzHost returns one value the return type is PSHost. If Get-AzHost returns multiple types the return value is of type array.

The first parameter set contains the following unique parameters:

• ResourceGroupName
• HostGroupName

The second parameter set contains a unique parameter:

• ResourceId

The first parameter set contains a -Name parameter which when excludes retrieves a list of all hosts.

Get-AzVM

The documentation for the Get-AzVM cmdlet identifies four parameter sets as follows:

As mentioned above the Get-AzVM will return different types depending on the parameters specified to the cmdlet:

PSVirtualMachine

PSVirtualMachineInstanceView

The return values can be a single value or an array of values.

The first parameter set identifies all parameters as optional which means no parameter is an option and hence unique. When passed no parameters, Get-AzVM (the first parameter set) returns all virtual machines for a subscription. The second parameter set has positional parameters ResourceGroupName and Name or their named counterparts as required. DisplayHint is also a unique parameter for the second parameter set. The third parameter set has a unique parameter, Location. The fourth parameter set has a lone unique parameter, NextLink.

New-AzVM

The New-AzVM cmdlet defines the following multiple parameters sets and takes over a dozen potential parameters including complex types built up by invoking multiple setup cmdlets. 

The unique parameter for this New-AzVm parameter set is -Credential:

The unique parameter for this New-AzVM parameter set is -VM:

The data type passed to the -VM parameter is PSVirtualMachine. Setting up a PSVirtualMachine can involve invoking multiple cmdlets in order to assign additional settings to the PSVirtualMachine instance.

The unique parameter for this new-AzVM parameter set is -DiskFile:

New-AzVM and PSVirtualMachine Configuration

The following example from New-AzVM demonstrates the level of complexity that can be associated with setting up a cmdlet's parameters. The script marked in boldface shows how many PowerShell cmdlets have to be invoked simply (or not so simply) to set up the New-AzVM's -VM parameter value:

PowerShell: Cmdlet Parameter Sets

PowerShell cmdlet parameter sets are documented at Cmdlet parameter sets where they are defined as follows:

Each parameter set must contain a unique  parameter that allows the PowerShell runtime to identify which parameter set is being used. In order to demonstrate parameter sets consider the Get-Depth function from the post PowerShell: Converting Json-Formatted Strings to/from Json-Objects and Json-Object Depth. The Get-Depth function has been modified as follows, GetDepthEx, to take two parameters where each parameter belongs to a separate parameter set (see the attribute property, ParameterSetName, below):

function Get-DepthEx()
{
  param (
    [Parameter(
      Mandatory=$true,
      ParameterSetName='FormattedJsonString')]
      [string] $json,
    [Parameter(
      Mandatory=$true,
      ParameterSetName='JsonObject')]
      [PSCustomObject] $jsonObject
  )

The code for GetDepthEx is the original GetDepth code save it has been modified (see text demarcated by boldface) to work with a Json-formatted string parameter (parameter set name, FormattedJsonString) or with  Json-Object (parameter set name, JsonObject):

function Get-DepthEx()
{
    param (
        [Parameter(
            Mandatory=$true,
            ParameterSetName='FormattedJsonString')]
        [string] $json,
        [Parameter(
            Mandatory=$true,
            ParameterSetName='JsonObject')]
        [PSCustomObject] $jsonObject
    )

    if (($null -ne $json) -and ($json.Length -gt 0))
    {
        # This step verifies that $json is a valid Json object
        $jsonObject = ConvertFrom-Json $json   
        if ($null -eq $jsonObject)
        {
            return 0
        }
    }

    if ($null -ne $jsonObject)
    {
        # As of PowerShell 7.0 the max -Depth is 100
        $json = ConvertTo-Json $jsonObject -Depth 100
    }   

    [int] $maximumDepth = -1
    [int] $depth = 0
    [char[]] $startingBrackets = '[', '{'
    [char[]] $endingBrackets = @(']', '}')

    foreach ($c in $json.ToCharArray())
    {
        if ($c -in $startingBrackets)
        {
            ++$depth
            $maximumDepth = if ($maximumDepth -ge $depth)
                { $maximumDepth } else { $depth }
        }

        elseif ($c -in $endingBrackets)
        {
            --$depth
        }
    }

    return $maximumDepth
}

The GetDepthEx method is invoked as follows where each invocation uses a different parameter set as is show using the code from the previous post "PowerShell: Reading, Modifying, and Saving Json Files":

[string] $sourceWindowsVmTemplateFilename =
             'Template2019.json'
[string] $destinationWindowsVmTemplateFilename =
             'TemplateAnyWindowsVM.json'
[string] $content =
             Get-Content -Raw -Path $sourceWindowsVmTemplateFilename
[PSCustomObject] $jsonObject =
    $content |
    ConvertFrom-Json

[int] $depthParameterSetFormattedJsonString =
        Get-DepthEx -Json $content
[int] $depthParameterSetJsonObject =
        Get-DepthEx -JsonObject $jsonObject

There are many permutations to parameter sets. For example it possible for a parameter to be included in multiple parameter sets. As a feature of PowerShell, parameter sets are not part of the day-to-day coding performed by most developers.  It is more likely that a developer will have to read the documentation associated with a cmdlet that supports multiple parameter sets. Cmdlets that contain multiple parameters sets include;

• Add-AzVMNetworkInterface
• Get-AzGallery
• Get-AzHost
• Get-AzHostGroup
• New-AzVM
• New-AzVMConfig
• New-AzVMSqlServerAutoBackupConfig
• Publish-AzVMDscConfiguration
• Remove-AzGallery
• Remove-AzHost
• Remove-AzHostGroup

PowerShell: Reading, Modifying, and Saving Json Files

In the previous blog entry, PowerShell: Converting Json-Formatted Strings to/from Json-Objects and Json-Object Depth, it was shown how to correctly create Json-objects from Json-formatted strings and how to create Json-formatted strings from Json-objects by taking into account the depth of the underlying Json-object being manipulated. This post expands on the manipulation of Json-object with PowerShell by demonstrating how to:

• read a Json-formatted string from a file
• convert the Json-formatted string to a Json-object
• modify the Json-object
• covert the Json-object to a Json-formatted string
• save the Json-formatted string to a file

The code used to perform the above tasks is as follows:

[string] $sourceWindowsVmTemplateFilename = 'Template2019.json'
[string] $destinationWindowsVmTemplateFilename =
    'TemplateAnyWindowsVM.json'
[string] $content =
     Get-Content -Raw -Path $sourceWindowsVmTemplateFilename
[int] $depth = Get-Depth $content
[PSCustomObject] $jsonObject = $content | 

                                 ConvertFrom-Json

if ($jsonObject.resources.properties.
      storageProfile.imageReference.sku -ne $windowsSku)
{
    $jsonObject.resources.properties.
        storageProfile.imageReference.sku = $windowsSku
    $jsonObject |
        ConvertTo-Json -Depth $depth |
        Set-Content $destinationWindowsVmTemplateFilename
}

The Json file is read by the Get-Content cmdlet where the -Raw command-line option causes the entire file to be read as a single string:

[string] $content = 
  Get-Content -Raw -Path $sourceWindowsVmTemplateFilename

A culled version of the Json identifies the Windows Sku is as follows:

<#
  "resources": [
  {
    {
      "properties": {
        "storageProfile": {
          "imageReference": {
             "sku": "2012-R2-Datacenter",
#>

Based on the pseudo-Json above the sku is accessible as follows:

resources.properties.storageProfile.imageReference.sku

The Json object's sku property is checked with an if statement and assigned as follows:

if ($jsonObject.resources.properties.
      storageProfile.imageReference.sku -ne $windowsSku)
{
  $jsonObject.resources.properties.
      storageProfile.imageReference.sku = $windowsSku

The Json-object's depth is computed as follows by using the Get-Depth method and the conversion from Json-object to Json-string is performed using the ConvertTo-Json cmdlet and its -Depth parameter:

[int] $depth = Get-Depth $content
$jsonObject | ConvertTo-Json -Depth $depth

The Json-formatted string is committed to a file using the Set-Content cmdlet:

    Set-Content $destinationWindowsVmTemplateFilename

The Json data being manipulated is based off of a real world situation where Azure virtual machines were being created as the host for specific versions of SharePoint on-prem. The Json-formatted string read from a file will be an Azure Resource Manager (ARM) template file used in the creating of a new virtual machine image for a standard Azure Windows SKU. Assigning the SKU will be the modification made to the Json-object which the supported SKUs are:

• 2012-R2-Datacenter
• 2016-Datacenter
• 2019-Datacenter

The Window's SKU returned is simply a string that depends on the version of SharePoint to be installed on the virtual machine (SharePoint 2013, SharePoint 2016 or SharePoint 2019). The function used to return the appropriate Windows SKU is Get-WindowsSku:

function Get-WindowsSku()
{
  param (
    [ValidateSet(2013, 2016, 2019)]
    [Parameter(Mandatory)]
    [int] $sharePointVersionYear
  )

  [string] $sku

  switch($sharePointVersionYear){
    2013 { $sku = '2012-R2-Datacenter'}
    2016 { $sku = '2016-Datacenter'}
    2019 { $sku = '2019-Datacenter'}
  }

  return $sku
}

PowerShell: Converting Json-Formatted Strings to/from Json-Objects and Json-Object Depth

PowerShell has extremely versatile cmdlets for converting Json-formatted strings to Json-objects (ConvertFrom-Json) and converting Json-objects to Json-formatted strings (ConvertTo-Json). A real world case where such cmdlets come in handy is when working with Azure Resource Manger (ARM) templates (JSON formatted files) in order to create virtual machines from standard SKU's and from images. The ConvertTo-Json cmdlet has a default behavior that needs to be accounted for when using it in real world applications. There are too many online blogs that fail to demonstrate that by default ConvertTo-Json only works on Jason objects with a maximum depth of 2. This blog post shows the correct use of ConvertTo-Json. 

The PowerShell cmdlet that converts a Json-formatted string to a Json object or, when the -AsHashtable command-line option is specified, to a HashTable, is ConvertFrom-Json:

The PowerShell cmdlet that converts an object to a Json-formatted string is ConvertTo-Json:

The CovertFrom-Json cmdlet has a -Depth command-line option that defaults to a value of 1024 while the ConvertTo-Json cmdlet has a -Depth command-line option that defaults to a value of two. The depth of a Json object is defined as the maximum nested level of each property (starting with a { bracket and ending in a } bracket) and each nested level of an array (starting with a [ bracket and ending in a ] bracket).

Consider a case where a Json-formatted string with a depth of nine is converted to Json-object. There would be no need to set the -Depth parameter when invoking CovertFrom-Json as the default depth is 1024. If the Json-object is edited and then converted to a string using the CovertTo-Json cmdlet, there would be a problem unless the -Depth command-line option was set to at least a value of nine (as computed by the Get-Depth method below). Since CovertTo-Json default -Depth defaults to two, then the Json-formatted string would be malformed since the Json-object converted has a depth of nine.

The Get-Depth function takes as a parameter a Json-formatted string and returns the Json's depth:

function Get-Depth()
{
  param (
    [Parameter(Mandatory=$true)]
    [string] $json
  )

  # This step verifies that $json is a valid Json object
  $jsonObject = ConvertFrom-Json $json
  if ($null -eq $jsonObject)
  {
    return 0
  }

  [int] $maximumDepth = -1
  [int] $depth = 0
  [char[]] $startingBrackets = '[', '{'
  [char[]] $endingBrackets = @(']', '}')

  foreach ($c in $json.ToCharArray())
  {
    if ($c -in $startingBrackets)
    {
      ++$depth
      $maximumDepth = if ($maximumDepth -ge $depth)
        { $maximumDepth } else { $depth }
    }

    elseif ($c -in $endingBrackets)
    {
      --$depth
    }
  }

  return $maximumDepth
}

The following snippet of PowerShell shows CovertFrom-Json being used to convert a Json-formatted string to a Json-object. ConvertTo-Json is used with the -Depth command-line parameter assigned using a value return by the Get-Depth function and ConvertTo-Json is used without the -Depth command-line parameter assigned:

[string] $content = <getting content shown in future blog>
[int] $jsonDepth = Get-Depth $content
[PSCustomObject] $jsonObject = $content | ConvertFrom-Json
[string] $updatedContent = $jsonObject |
  ConvertTo-Json -Depth $jsonDepth
[string] $updatedContentDefaultDepth = $jsonObject | ConvertTo-Json

Below is an image of the Json-formatted string created by ConvertTo-Json using the default value for -Depth, two: 

Note that the properties hardwareProfile, storageProfile, networkProfile, osProfile, and diagnosticsProfile are unassigned as these properties exceed the default depth.

Below is an image of the Json-formatted string created by ConvertTo-Json using the -Deptth determined by the Get-Depth function and notice that hardwareProfile, storageProfile, networkProfile, osProfile, and diagnosticsProfile are all unassigned values:

Git: Reverting git commit

Every developer making use of git from the command-line has mistakenly invoked "git commit". The "git commit" command commits staged changes to the local repository. In a previous post it was demonstrated how to rollback a git add (Git: Reverting git add).

A typical git commit takes the following form of a git add (with some command-line) followed by a git commit -m:

git add .

git commit -m "some message related to the commit" 

The following undoes the last commit and leaves the state of your files on disk unchanged (meaning your working tree is unchanged). The changes previously staged by the commit are reverted to unstaged:

git reset HEAD~ 

After invoking "git reset HEAD~" the files on disk can be modified as needed and the git add and git commit commands can be invoked where "git commit -c" will commit using the previous commit message text (as in "some message related to the commit" from the example message above):

git add .

git commit -c ORIG_HEAD

To edit the original commit message use the following sequence of commands:

git add .

git commit -c ORIG_HEAD

When "git commit -c ORIG_HEAD" is invoked an editor will display the log message from the previous commit and the message associated with the previous commit can be edited.

Git: Reverting git add

Every developer making use of git from the command-line has mistakenly invoked "git add". The "git add" command commits a change or changes from the current working tree to a staging area where this change can be later committed to the repository.

To rollback an add associated with a single file, invoke git reset specifying the file to be rolled back (un-stage the file):

git reset <file> 

To rollback all files added but not committed, invoke git reset without the <file> parameter specified:

git reset

Visual Studio: C# projects changed to target .NET Framework 4.7.2 won't build

Upgrading an existing set of C# projects the following error was encountered while trying to build:

your project does not reference ".NETFramework,Version=v4.7.2" framework. Add a reference to ".NETFramework,Version=v4.7.2" in the "TargetFrameworks" property of your project file and then re-run NuGet restore

The solution is to delete the obj and bin folders of each project which fails to build with the above message.