Why YAML you might wonder? A YAML file is very easy to use, and many infrastructure-as-code tools (like Ansible, Terraform, etc.) use YAML for configuration, so people that are usually working on infrastructure will be pleased to see something like this if a DBA gives them a such YAML file. Additionally, it gives you the possibility to touch only the configuration, but run the same code again for a different deployment.

And for you, as a SQL developer or DBA, you will gain time, as you won't have to click through Azure pages to configure everything, but just quickly edit the YAML file and run the script. Because most of the settings will be similar for you, you don't have to touch every configuration you've made before, either.

You can change the VM size, region, SQL Server version, and more by simply editing the YAML file without touching the script itself. The best part: the entire deployment now takes about 30 minutes from start to finish, including database restores (sample databases by Microsoft; your databases may take longer if they are bigger) and installing tools.

In this post, I'll walk through a PowerShell script that deploys a fully configured SQL Server 2022 VM on Azure. It sets up Azure Bastion for secure access, Key Vault for secrets management, Azure Files for persistent storage, and installs development tools automatically. The entire deployment is driven by a single YAML configuration file.

The Solution: Config-Driven Deployment

The solution is split into two files:

• config.yaml: a declarative configuration file that defines what to deploy

• vm_creation_with_bastion.ps1: the deployment script that makes it happen

The Configuration File

Everything that might change between deployments lives in config.yaml:

resourceGroup:
  name: "YourResourceGroupName"
  location: "Switzerland North"
  tags:
    Purpose: "Demo"

vm:
  size: "Standard_DS13_V2"
  image:
    publisherName: "MicrosoftSQLServer"
    offer: "sql2022-ws2022"
    skus: "sqldev-gen2"
    version: "latest"

storage:
  resourceGroup: "yourstorageRG"
  accountName: "yourstorageaccount"
  fileShareName: "yoursqlbackupsharename"
  driveLetter: "Z"

Want a different VM size? Change one line. Different region? One line. Different SQL Server version? One line. No script modifications needed.

Architecture Overview

The deployment creates resources across two resource groups by design:

Azure Subscription
|
+-- Resource Group: YourResourceGroupName (disposable)
|   +-- Azure Bastion
|   +-- SQL Server VM (Managed Identity)
|
+-- Resource Group: yourstorageRG (persistent)
    +-- Key Vault (secrets: vm-admin-password, storage-account-key)
    +-- Storage Account
        +-- File Share (mounted as Z: on VM)
            +-- restore-databases.ps1 (uploaded by script)

Why two resource groups? The VM resource group is disposable. You can tear it down and rebuild without losing your Key Vault, storage account, or database backups. The persistent resource group survives VM rebuilds, which matters a lot when you frequently spin up and destroy environments during development.

Choosing an Azure Bastion SKU

Azure Bastion comes in different SKU tiers. Check https://learn.microsoft.com/en-us/azure/bastion/bastion-sku-comparison for more information.

Key Design Decisions

1. Idempotent Resource Creation

Every resource is checked before creation. Run the script twice and it won't fail or create duplicates. This is crucial for development environments where you might iterate on the config and re-run the script multiple times.

2. Key Vault Soft-Delete Handling

Azure Key Vault has a soft-delete feature that retains deleted vaults for 90 days. If you delete a Key Vault and try to create one with the same name, you will get an error: "The vault name is already in use."

The script handles this by automatically incrementing the name.

At the end of the deployment, the script notifies you of the name change so you can update your config for future runs.

3. Cryptographically Secure Password Storage

The password is stored in Key Vault immediately and never written to disk. Instead of relying on Get-Random (which uses a PRNG), the script generates VM admin passwords using RNGCryptoServiceProvider and guarantees complexity requirements

4. Managed Identity for Secrets Access

The VM uses a system-assigned managed identity to retrieve secrets from Key Vault. No credentials are hardcoded or stored on the VM. The file share mount script, which runs on the VM, fetches the storage account key at runtime.

Even if someone gains access to the VM, they can not extract long-lived credentials. The managed identity token is short-lived and scoped.

5. Persistent File Share Mount

The Azure Files share is mounted as a drive letter (Z:) using New-SmbGlobalMapping, which makes it available to all users and services on the VM. A scheduled task re-mounts it on every reboot.

6. Automated Software Installation

The script installs development tools on the VM via Invoke-AzVMRunCommand, defined entirely in the YAML config:

softwareInstalls:
  installScript: |
    choco install vscode git powershell-core tabular-editor -y
    Install-Module -Name dbatools -Force
  logonScript: |
    code --install-extension ms-mssql.mssql

Some tools (like VS Code extensions) need a user session, so the script registers a one-time logon task that runs at first login and then cleans itself up.

7. Database Restore Script

A restore-databases.ps1 script is automatically uploaded to the file share. After logging into the VM, you can restore all .bak files from the share with a single command:

Z:\restore-databases.ps1

It uses dbatools to iterate over all backup files and restore them to the local SQL Server instance.

Running the Deployment

Prerequisites:

• PowerShell 7+ with the Az module installed

• An active Azure session (Connect-AzAccount)

Then simply:

.\vm_creation_with_bastion.ps1

Or with a custom config:

.\vm_creation_with_bastion.ps1 -ConfigFile "my-environment.yaml"

At the end, the script prints the VM credentials and total deployment time:

Deployment completed in 00:18:42.

VM Login Credentials:
  Username: youradminusername
  Password: <generated-password>

I am going to publish a YAML writing helper for SQLVMs so that the picking of a SQLVM image is easier and faster, and it'll write the YAML file for you. Stay tuned for that!

The full source code is available on GitHub.

About a year ago, my original script from my blog post “Deploy Azure Sql Server Vm With PoSh” (blog, substack) from 2021 broke due to a change in the underlying model. In last August, Microsoft updated its documentation regarding deploying Azure SQL Server VMs on Azure. Therefore, I was able to redevelop the code, and below, you’ll find the new code along with some new updates that Microsoft made. There are some changes around the model of Azure VMs and the model for Azure Bastion.

In the next chapter, you’ll find a script that deploys an Azure SQL Server VM together with Azure Bastion. Azure Bastion is a service by Microsoft that could be seen as something like “Security as Service” by Microsoft. If you want to use RDP via open internet, make sure you protect your RDP connection. Azure Bastion covers your RDP connection with a TLS tunnel and it is very easy to use. Another way is to use VPN or a private endpoint, but then you have to manage the network security yourself. If you are just a database engineer / DBA like myself or similar, then possibility don’t know how to do this, and Azure Bastion is an excellent option as it won’t expose your VMs IP-Address to the internet.

For who is this script?

If you are a DBA or DB dev at a smaller company or a smaller IT team or just want to have a proof of concept, but your IT team is not able to provide you something like this, this may be for you. Also, if you want to learn Azure Powershell and are a DBA, this is a great start. If you however want to have a solution on a bigger scale, be or adaptions more frequently, you probably will find either Bicep by Microsoft or something like Terraform by Hashicorp more helpful. I personally started to develop this script a during the pandemic years when I was a DB developer and wanted to have a platform on which a small team could develop new things and showcase it to customers, even give them the VM as a means to try the data warehouse solution we’ve built for a core banking system provider.

The execution of the whole script, including the installation of dbatools takes about 25 mins in total, even though not the whole setup is fully automated yet; I’m working on it to bring it to the fullest automation though. I’ll post a bit more in-depth information about how to adapt this script in future. That post will give you an understanding of the underlying structure that ARM (Azure Resource Manager) gives you to work with PowerShell, but actually also for anything like Bicep and Terraform, as they also have to work with ARM in the end.

The new script

I’m not going to give much more details on the script here, but I think I’ve put comments within the script relatively extensively:

# Existing code
$ResourceGroupName = "sqlvm1"
$Location = "East US"
$ResourceGroupParams = @{
    Name     = $ResourceGroupName
    Location = $Location
    Tag      = @{"Purpose" = "Demo" }
}

New-AzResourceGroup @ResourceGroupParams

$SubnetName = $ResourceGroupName + "subnet"
$VnetName = $ResourceGroupName + "vnet"
$PipName = $ResourceGroupName + $(Get-Random)

# Create a subnet configuration
$SubnetConfig = New-AzVirtualNetworkSubnetConfig -Name $SubnetName `
    -AddressPrefix 192.168.1.0/24

# Create a virtual network
$Vnet = New-AzVirtualNetwork -ResourceGroupName $ResourceGroupName `
    -Location $Location `
    -Name $VnetName -AddressPrefix 192.168.0.0/16 `
    -Subnet $SubnetConfig

# Create a public IP address and specify a DNS name
$Pip = New-AzPublicIpAddress -ResourceGroupName $ResourceGroupName -Location $Location `
    -AllocationMethod Static -IdleTimeoutInMinutes 4 -Name $PipName

# Rule to allow remote desktop (RDP)
$NsgRuleRDP = New-AzNetworkSecurityRuleConfig -Name "RDPRule" -Protocol Tcp `
    -Direction Inbound -Priority 1000 -SourceAddressPrefix * -SourcePortRange * `
    -DestinationAddressPrefix * -DestinationPortRange 3389 -Access Allow

# Rule to allow SQL Server connections on port 1433
$NsgRuleSQL = New-AzNetworkSecurityRuleConfig -Name "MSSQLRule"  -Protocol Tcp `
    -Direction Inbound -Priority 1001 -SourceAddressPrefix * -SourcePortRange * `
    -DestinationAddressPrefix * -DestinationPortRange 1433 -Access Allow

# Create the network security group
$NsgName = $ResourceGroupName + "nsg"
$Nsg = New-AzNetworkSecurityGroup -ResourceGroupName $ResourceGroupName `
    -Location $Location -Name $NsgName -SecurityRules $NsgRuleRDP, $NsgRuleSQL

$InterfaceName = $ResourceGroupName + "int"
$Interface = New-AzNetworkInterface -Name $InterfaceName `
    -ResourceGroupName $ResourceGroupName -Location $Location `
    -SubnetId $VNet.Subnets[0].Id -PublicIpAddressId $Pip.Id `
    -NetworkSecurityGroupId $Nsg.Id

# Define a credential object
# For a proof of concept or demo, this is fine, but you should really use
# a key vault to store your secrets otherwise.
$userName = "azureadmin"
$SecurePassword = ConvertTo-SecureString 'YourSecretPassword' `
    -AsPlainText -Force
$Cred = New-Object System.Management.Automation.PSCredential ($userName, $SecurePassword)

# Create a virtual machine configuration
# Note that you can change eg. SKU to an SQL Server SKU of your choice, eg
# standard or enterprise edition. 
$VMName = $ResourceGroupName + "VM"
$VMConfig = New-AzVMConfig -VMName $VMName -VMSize Standard_DS13_V2 |
Set-AzVMOperatingSystem -Windows -ComputerName $VMName `
    -Credential $Cred -ProvisionVMAgent -EnableAutoUpdate |
    Set-AzVMSourceImage -PublisherName "MicrosoftSQLServer" `
    -Offer "sql2022-ws2022" -Skus "sqldev-gen2" -Version "latest" | 
    Add-AzVMNetworkInterface -Id $Interface.Id

# Create the VM
New-AzVM -ResourceGroupName $ResourceGroupName -Location $Location -VM $VMConfig

# New code to deploy Azure Bastion
$BastionSubnetName = "AzureBastionSubnet"
$BastionSubnetConfig = New-AzVirtualNetworkSubnetConfig `
    -Name $BastionSubnetName -AddressPrefix 192.168.2.0/24

# Update the virtual network with the Bastion subnet
$vnet = Get-AzVirtualNetwork -Name $VnetName -ResourceGroupName $ResourceGroupName
Add-AzVirtualNetworkSubnetConfig -Name $BastionSubnetName `
    -VirtualNetwork $vnet -AddressPrefix "192.168.2.0/24" | Set-AzVirtualNetwork

# Create a public IP address for Bastion (standard sku)
$BastionPipName = $ResourceGroupName + "bastionpip"
$BastionPip = New-AzPublicIpAddress -ResourceGroupName $ResourceGroupName -Name $BastionPipName -Location $Location `
    -AllocationMethod Static -IdleTimeoutInMinutes 4 -Sku Standard

# Create the Bastion host
$BastionName = $ResourceGroupName + "bastion"
New-AzBastion -ResourceGroupName $ResourceGroupName -Name $BastionName `
    -PublicIpAddressRgName $ResourceGroupName -PublicIpAddressName $BastionPipName `
    -VirtualNetworkRgName $ResourceGroupName -VirtualNetworkName $VnetName -Sku "Basic"

$StorageAccountName = "kaysbackstorageacc"  # Use lowercase letters and numbers only
$FileShareName = "sqlbackupshare"

# Check if the storage account exists
$storageAccount = Get-AzStorageAccount -ResourceGroupName $ResourceGroupName `
    -Name $StorageAccountName -ErrorAction SilentlyContinue
if (-not $storageAccount) {
    # Create the storage account if it doesn't exist
    New-AzStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageAccountName `
    -Location $Location -SkuName "Standard_LRS"
    Write-Host "Storage account '$StorageAccountName' created."
}
else {
    Write-Host "Storage account '$StorageAccountName' already exists."
}

# Get storage account context
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $ResourceGroupName `
    -Name $StorageAccountName)[0].Value
$Context = New-AzStorageContext -StorageAccountName $StorageAccountName `
    -StorageAccountKey $StorageAccountKey

# Check if the file share exists
$fileShare = Get-AzStorageShare -Context $Context -Name $FileShareName `
    -ErrorAction SilentlyContinue
if (-not $fileShare) {
    # Create the file share if it doesn't exist
    New-AzStorageShare -Context $Context -Name $FileShareName
    Write-Host "File share '$FileShareName' created."
}
else {
    Write-Host "File share '$FileShareName' already exists."
}

# install the stuff to the vm
$script1 = @"
# Install NuGet and dbatools
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
Install-Module -Name dbatools -Force
"@

Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroupName -Name $VMName -CommandId 'RunPowerShellScript' -ScriptString $script1

More on Azure Bastion

So Azure Bastion now comes in some different flavors, which is a relatively new offering. Bastion now has some different SKUs respectively tiers: Basic, Standard and Premium. Furthermore, you can choose from a private-only and developer offer. As for a better understanding of the SKUs, I recommend reading the quite long feature overview table. The main takeaway is though that if you want to use to an azure sql vm and connect to it via RDP, don’t pick the basic SKU, but a higher tier. The basic SKU won’t support SSL protected RDP connection.

How to use the script

I recommend to use Azures Cloud Shell as this is the easiest way to use and you can share the script with your team, too. I like to create a folder with mkdir where you can store such kind of scripts. In the following picture, you’ll see how to get to the Azure Editor that is heavily inspired by vscode.

Create a *.ps1 file with the name you like (see screenshot above, you can use touch to create a file). Open it with the editor (mouse click no. 4 in above image and click no. 5 in below image) and copy & paste the script from above inside. You can save the script with [ctrl]+[s] or via the menu of the editor.

More to come for full automation

Currently, my script that executes a PowerShell script against the created VM via Invoke-AzVMRunCommand, unfortunately does not create a working attached network storage yet.

There is an issue that I am still investigating and hoping that I can resolve it in the next few weeks. Once I’ll figure that one out (or if you have a tip for me - please let me know!), then I’ll post it to complete this script. Until then you can use the following post deployment script on the vm directly as a workaround:

# Map the network drive
$connectTestResult = Test-NetConnection -ComputerName sqldatabasebaks.file.core.windows.net -Port 445
if ($connectTestResult.TcpTestSucceeded) {
    # Save the password so the drive will persist on reboot
    # For a proof of concept / demo, this approach is fine, 
    # but normally, you should want a key vault to store your secrets!
    cmd.exe /C "cmdkey /add:`"sqldatabasebaks.file.core.windows.net`" /user:`"localhost\sqldatabasebaks`" /pass:`"YourPassword`""
    # Mount the drive
    New-PSDrive -Name Z -PSProvider FileSystem -Root "\\sqldatabasebaks.file.core.windows.net\sqlbaks" -Persist
}
else {
    Write-Error -Message "Unable to reach the Azure storage account via port 445. Check to make sure your organization or ISP is not blocking port 445, or use Azure P2S VPN, Azure S2S VPN, or Express Route to tunnel SMB traffic over a different port."
}


# copy to mssql folder for restore with dbatools
# you may need to adapt the folder path according to your own file storage system
Copy-Item -Path "Z:\*.bak" `
    -Destination "C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\Backup" `
    -Recurse


# use dbatools for restore dbs
Set-DbatoolsConfig -FullName sql.connection.trustcert `
    -Value $true # to ensure that dbatools trusts the certificate
Restore-DbaDatabase -SqlInstance sqlvm1VM -Path "C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\Backup"

How to connect to the VM via Bastion

First, go to your resource group and click on the VM so you get into its information window:

After this, you have 2 ways to connect to the VM via Azure Bastion, both ways are marked with the click no 7 in the screenshot below. Either you use the left side vertical menu or the horizontal menu.

After this you get to the menu where you enter the credentials you already used in the script (that is hopefully stored in a key vault). Click on connect.

After this, a new browser tab opens and on the top left, you’ll be asked if you allow to transfer text data to the VM, which you want for this script. In the next screenshot, you’ll see “Zulassen” which is German for “Allow”. Click on allow.

For the backups, I like to use Azure File Storage which is a relatively cheap storage. I use this for SQL Server backups. That storage can be mapped to a network drive on the VM. So in order to retrieve the password and as well the script to do so, lets go to its resource group. I recommend to use a different resource group for the storage than the VM. That allows you to delete the VM from time to time to save costs or for a fresh setup. To do so, you simply can delete its resource group, and because your file storage is in another, your backup resource group will stay. If you want to upload backups to your file storage, you can do this very easy this way: Go to your storage account, and here you’ll have the possibility to click on a button that opens Azure Storage Explorer for you.

In there, you need to go to your file storage and drag and drop your files to upload. The next screenshots show you how to retrieve the code for your own environment, for your password and your own storage:

Use this script and overwrite this part in my deployment script, so you’ll have exactly what you’ll need for your own file storage. After you executed the deployment script (takes about 20 mins) copy the post deployment script (the last code block above) and copy it into PowerShell ISE, then run it:

After you’ve run, open SSMS and make sure that you’ll tick “Trust server certificate” as we don’t have a server certificate on this VM.

Now, finally, you’ll see that, in my example case, AdventureWorks2022 and WideWorldImporters got restored. The restore part was magically done by dbatools, with just a few lines of code!

Do you have any questions? Any feedback? Could I have done anything better? You want to give input to improve my code? Let me know in the comments or send me a message!

Firstly, we want to get a nice overview over all csv files that failed to load. This can be done in a lot of different ways, but I like to do it this way:

# check local log
from pyspark.sql.functions import when
df_log = df_log.withColumn("Check", when(df_log["row_amount_diff"] == 0, "✅").otherwise("❌"))
# these special characters from [Win Key] + [.] or another smiley picker

With this, you’ll get this output:

If you remember the script in the previous article, the script opened the files first, just to check how many lines it sees in there. Later, the script checks how many lines were loaded and the row amount diff gives back if there was a difference. If yes, it would not be 0, obviously. The last column gives a very nice indication of whether or not a csv file needs attention. If there’s ❌, you probably want to check the line. Note that the columns of a such table are sortable, so you can of course also sort the table after the ❌ signs, and you’ll see all failed loads instantly.

So next, in the same notebook, I’d like to see if the meta lakehouse actually received the data, too. I use the meta lakehouse for such kinds of events. This serves a lot of purposes: One being able to measure problems in the future. Do always the same files fail? Is there a pattern, maybe they only ever fail on certain dates like every leap year and nobody would know if we wouldn’t have a such log? Or every easter?

# check log in meta_lakehouse
df = spark.sql(f"SELECT * FROM meta_lakehouse.load_log_bronze order by load_timestamp_utc_bronze desc LIMIT {csv_count}")
display(df)

So here, it is clear that the lakehouse did indeed work as intended, too.

Now, lets go back to the code that gave us warning in the first place, I actually have another small trick here:

# Print the number of tables that gave warnings
if warning_count == 0:
    print(f"\n\033[92m Number of tables with warnings: NONE - SUCCESS \033[0m")
else:
    print(f"\n\033[91m Number of tables with warnings: {warning_count} → GO CHECK LOG! \033[0m")
    displayHTML('<a href = "https://app.fabric.microsoft.com/groups/[redacted]" target="_blank">Click here to get to the notebook to investigate this issue</a>')

The last line, the one I added for this post, adds a link to another notebook (i redacted the exact link to my notebook).

If I click on this link, I get to the next notebook with wich I can investigate the issue. In other words, notebooks are a great way to give me hints on how I can fix an issue! This new notebook has some similar code, but designed to show you which lines are responsible for the failure:

from pyspark.sql import SparkSession
from pyspark.sql.functions import monotonically_increasing_id

# Path to your CSV file
csv_file = "Files/Person.Person.csv"

# Get the number of lines in the CSV file
lines = spark.read.text(csv_file)
line_count = lines.count() - 1 # -1 to exclude the header
print(f"{csv_file} should have this amount of lines: {line_count:,}".replace(",", "'"))

# Read the CSV file with PERMISSIVE mode to get all records
df_all = spark.read.options(
    delimiter=",",
    header=True,
    encoding="UTF-8",
    inferSchema=True,
    multiLine=True,
    escapeQuotes=True,
    mode="PERMISSIVE"
).csv(csv_file)

# Read the CSV file with DROPMALFORMED mode to get only valid records
df_well = spark.read.options(
    delimiter=",",
    header=True,
    encoding="UTF-8",
    inferSchema=True,
    multiLine=True,
    escapeQuotes=True,
    mode="DROPMALFORMED"
).csv(csv_file)

# Identify malformed records by finding the difference
df_mal = df_all.subtract(df_well)

# Add a record number to the malformed records DataFrame
df_mal = df_mal.withColumn("record_number", monotonically_increasing_id())
columns = ['record_number'] + [col for col in df_mal.columns if col != 'record_number']
df_mal = df_mal.select(columns)

# Show the malformed records with their record numbers
print(f"The amount of read well formed lines: {df_well.count():,}".replace(",","'"))
print(df_mal.count())
display(df_mal)

This returns me this:

So the print output tells me that there was a load failure because of 4 lines. Those 4 lines are visible in the table. Upon investigation of this table, you can notice that something in the column “Demographics” doesn’t add up, hence the failure must have happened in the column before. Most likely, a character is responsible for this, as we see in “AdditionalContactInfo” XML data which is very prone to failures like this. So either a character needs to be taken care of or XML data in general should be attacked differently, eg. in an own table. Another possible solution is to avoid XML altogether and just scrap everything in there in a different way so that XML never gets loaded. I’m sure there could be other ideas for this, too. If you want, let me know in the comments!

I think this notebook also should give the person who is investigating this issue an idea of other incidents. A such overview is achieved very easily:

df = spark.sql("SELECT * FROM meta_lakehouse.bad_records LIMIT 1000")
display(df)

So this is another great example of make use of a meta lakehouse that may support you to investigate such issues. If you click on “entire_record”, you actually indeed can see the entire record:

So I think I've shown with this that notebooks are a really great way to support people on investigating issues. Meta lakehouses are obviously not meant to be accessible for everyone, but they may have data that business users may find helpful, eg. if you have a Power BI report that you want give an indication that the data is not fresh for this specific report yet. Make use of your own comments and your own failure reports so that you find your issue easier and quicker instead of drowning in error logs. Notebooks do have error logs, and they may be helpful too, but they have log a lot of information that may make it a bit convoluted to point you to your solution to the problem. However, I will cover this in a future post.

If you have any questions, I’m looking forward to get them in the comments below! Or if you have any suggestions on what I could improve in them: I am all ears!

Cover image credit: created with Microsoft Bings CoPilot.

The move was quite easy, especially with the great support by Hashnode! I received the answers to my questions within a few hours in most cases, but never more a half day!

So if you are looking for a possibility for a newsletter outside of my LinkedIn Newsletter, you can now use Hashnode to subscribe to my posts. Additionally, you can use RSS here, too.

I hope you will subscribe to both my newsletters, on kayondata.com and on LinkedIn.

Credit: Cover picture generated by CoPilot.

Certainly, it is best to avoid CSV files today. However, CSV files are still very common, so I am using this approach first, but I plan on writing other posts on other ways to do this. Another reason why I am using CSV files is the fact that this is also a very nice starting point to learn some basics and to exercise some data engineering stuff. Nevertheless, if you find yourself in a situation of choosing loading directly from a database without files, usually, this is the better route. If you have the possibility to use parquet instead of CSV, usually parquet should be the winner in the decision.

That said, let’s start with some code that I’ve written some months back on runtime 1.2 (Spark 3.4, Delta 2.4), but it should work on runtime 1.3 (Spark 3.5, Delta 3.2) as well, according to my test today. The code does quite some stuff, and it expects you to have an existing lakehouse that your notebook is attached to. Moreover it expects you to have a lakehouse called meta_lakehouse which is used to log the data loads and errors. You’ll see an output that returns some simple plausibility checks:

import os, time, datetime 
from pyspark.sql.functions import monotonically_increasing_id, lit, current_timestamp, when, col, regexp_replace, array, concat_ws
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, TimestampType
from notebookutils import mssparkutils


# Specify the folder containing CSV files
csv_folder = "Files"

# Get a list of all files in the folder
files = notebookutils.fs.ls(csv_folder)

# Filter for CSV files
csv_files = [file.path for file in files if file.name.endswith(".csv")] # to test only one file, simply write the name before .csv

# Define the schema for the log DataFrame so it has have good column names
log_schema = StructType([
    StructField("file_name", StringType(), True),
    StructField("row_amount_file", IntegerType(), True),
    StructField("row_amount_table", IntegerType(), True),
    StructField("row_amount_diff", IntegerType(), True),
    StructField("load_timestamp_utc_bronze", TimestampType(), True)
])

# Define the schema for the malformed records DataFrame so it has good column names
malformed_schema = StructType([
    StructField("file_name", StringType(), True),
    StructField("record_number", IntegerType(), True),
    StructField("entire_record", StringType(), True),
    StructField("explanation", StringType(), True),
    StructField("timestamp", TimestampType(), True)
])

# Initialize an empty DataFrame for log entries with the defined schema
df_log = spark.createDataFrame([], schema=log_schema)
df_mal = spark.createDataFrame([], schema=malformed_schema)
df_mal_log = spark.createDataFrame([], schema=malformed_schema)

# Initialize a counter for warnings
warning_count = 0

# count files
csv_files_count = [file for file in files if file.name.endswith('.csv')]
csv_count = len(csv_files_count)
file_processed_count = 0

# Process each CSV file
for csv_file in csv_files:
    file_processed_count += 1
    table_name = os.path.splitext(os.path.basename(csv_file))[0] # Extract table name from the file name (remove ".csv" extension)
    table_name = table_name.replace(".","_") # remove "." for database schema and replace it with "_"

    print(f"{table_name}.csv is being processed. Remaining files to process: {csv_count - file_processed_count}") 

    # Get the number of lines in the CSV file
    lines = spark.read.text(csv_file)
    line_count = lines.count() - 1 # -1 to exclude the header
    print(f"{table_name} has this amount of lines: {line_count:,}".replace(",", "'"))

    # Read the CSV file with PERMISSIVE mode to get all records
    df_all = spark.read.options(
        delimiter=",", 
        header=True,
        encoding="UTF-8", 
        inferSchema=True, 
        multiLine=True,
        escapeQuotes=True, 
        mode="PERMISSIVE"
    ).csv(csv_file)

    # Read the CSV file with DROPMALFORMED mode to get only valid records
    df_well = spark.read.options(
        delimiter=",", 
        header=True,
        encoding="UTF-8", 
        inferSchema=True, 
        multiLine=True,
        escapeQuotes=True, 
        mode="DROPMALFORMED"
    ).csv(csv_file)

    # Identify malformed records by diff it
    df_mal = df_all.subtract(df_well)

    # Add a record number to the malformed records dataFrame
    df_mal = df_mal.withColumn("record_number", monotonically_increasing_id())
    columns = ['record_number'] + [col for col in df_mal.columns if col != 'record_number']
    df_mal = df_mal.select(columns)

    # Show the malformed records with their record numbers
    print(f"The amount of read well formed lines: {df_well.count():,}".replace(",","'"))
    if df_mal.count() == 0:
        print(f"\033[92mThe amount of read mal formed lines:  {df_mal.count():,}\033[0m".replace(",","'"))
    else:    
        print(f"\033[91mThe amount of read mal formed lines:  {df_mal.count():,}\033[0m".replace(",","'")) 
        # display(df_mal)

    # Add id column and put it into the front 
    df_well = df_well.withColumn("id", monotonically_increasing_id())  
    columns_ordered = ["id"] + [col for col in df_well.columns if col != "id"]
    df_well = df_well.select(*columns_ordered)

    # Add load_timestamp
    df_well = df_well.withColumn('load_timestamp_utc_bronze', current_timestamp())

    # Write to Delta table
    df_well.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable(table_name)

    # Check the number of rows in the Delta table
    delta_table = spark.table(table_name)
    delta_count = delta_table.count()
    print(f"{table_name} has this amount of lines in the delta table: {delta_count:,}".replace(",", "'"))

    # Create a dataframe for the current log entry
    df_current_log = spark.createDataFrame([(
        table_name,
        line_count,
        delta_count,
        line_count - delta_count,
        datetime.datetime.now()  # Use current timestamp
    )], schema=log_schema)

    # Append the current log entry to the log DataFrame
    df_log = df_log.union(df_current_log)

    # Append the current malformed dataframe to a log dataframe
    # display(df_mal)
    # df_mal_log = df_mal_log.union(df_mal)

    # Prepare the malformed records for logging
    df_mal_log_entry = df_mal.withColumn("file_name", lit(table_name)) \
                             .withColumn("entire_record", concat_ws("|||", *df_mal.columns)) \
                             .withColumn("explanation", lit("malformed")) \
                             .withColumn("timestamp", current_timestamp()) \
                             .withColumn("record_number", monotonically_increasing_id())

    # Select the columns in the desired order
    df_mal_log_entry = df_mal_log_entry.select("file_name", "record_number", "entire_record", "explanation", "timestamp")

    # Append the current malformed records to the log DataFrame
    df_mal_log = df_mal_log.union(df_mal_log_entry)

    # Compare the counts and print messages
    if line_count == delta_count:
        print(f"\033[92mSUCCESS: {table_name}.csv loaded into Delta table {table_name} with matching line counts\033[0m")
        print(f"\n")
    else:
        print(f"\033[91mERROR: {table_name}.csv line count {line_count:,}".replace(",", "'") + f" does not match Delta table line count {delta_count:,}".replace(",", "'") + f" \033[0m")
        print(f"\n")
        warning_count += 1

# Write the log DataFrame to the lakehouse log table in the meta_lakehouse database
if not df_log.filter(df_log["row_amount_diff"] > 0).rdd.isEmpty():
    df_log.write.format("delta").mode("append").saveAsTable("meta_lakehouse.load_log_bronze")
    print(f"\n\033[93m ALL CSV files LOADED with WARNING into meta_lakehouse.load_log_bronze table.\033[0m")
else:
    print(f"\n\033[92m ALL CSV files SUCCESSFULLY LOADED into meta_lakehouse.load_log_bronze table.\033[0m")

# Write the malformed records DataFrame to a new Delta table in the meta_lakehouse database
if not df_mal_log.rdd.isEmpty():
    df_mal_log.write.format("delta").mode("append").saveAsTable("meta_lakehouse.bad_records")
    print(f"\n\033[93m Malformed records have been written to the meta_lakehouse.bad_records table.\033[0m")

# Print the number of tables that gave warnings
if warning_count == 0:
    print(f"\n\033[92m Number of tables with warnings: NONE - SUCCESS \033[0m")
else:
    print(f"\n\033[91m Number of tables with warnings: {warning_count} → GO CHECK LOG! \033[0m")

So if you run this, you’ll see a lot outputs like this:

As you can see, it outputs some table names and csv names. The csv-names are not actually very correct here, because the actual csv name is HumanResources.Departement.csv, so there is a dot that derives from the database schema. In this case, I am using a lakehouse without schema (it is still in preview) and the lakehouse does not allow points in a table name, hence I replaced the point with an underscore in line 53 via table_name = table_name.replace(".","_"), as also the comment after the very code explains.

Now, you could ask why I didn’t want to use shortcuts to load the data - I could have done this and it is a neat trick for many occasions. In this case, I not only wanted to showcase the possibility to use a meta_lakehouse, but also the possibility to have more control in the data that gets loaded here. For example, I get the possiblity to have some simple plausibility checks that certainly could be done much more sophisticated. This is going to be another blog in future.

An interesting code block is here (line 62):

    # Read the CSV file with PERMISSIVE mode to get all records
    df_all = spark.read.options(
        delimiter=",", 
        header=True,
        encoding="UTF-8", 
        inferSchema=True, 
        multiLine=True,
        escapeQuotes=True, 
        mode="PERMISSIVE"
    ).csv(csv_file)

    # Read the CSV file with DROPMALFORMED mode to get only valid records
    df_well = spark.read.options(
        delimiter=",", 
        header=True,
        encoding="UTF-8", 
        inferSchema=True, 
        multiLine=True,
        escapeQuotes=True, 
        mode="DROPMALFORMED"
    ).csv(csv_file)

Firstly, we get to give pySpark the possiblity to load all data and to try to figure out what datatypes the fields are. delimiter=”,” tells pySpark that it should use commatas, but you could use semicolon if needed, or another character. inferSchema=True is actually superfluous here as the spark.read.options uses this per default, but I think it is always a good idea to be explicit. In this case, the code reads the whole file, and for other file formats other than csv, you can tweak this with samplingRatio. There are many other options you can use here, and there’s a quite comprehensive documentation here. Be aware that inferring a schema based on a whole file isn’t what you want in all cases, especially if your file is too big. If you are aiming for speedy loads, then you should not using inferring schema, but define the targeted schema from the start. In this case, with relatively small amount of data of an AdventureWorks2022 sample data, I think this is a good showcase. By the way, with this you will also be able to get the schema pretty easy.

Another trick I am showcasing here is the possiblity to get the data that were malformed reads with a diff (line 84):

    # Identify malformed records by diff it
    df_mal = df_all.subtract(df_well)

Last thing I want to point out for today is the possiblity to write out statements in colors (line 154):

# Write the log DataFrame to the lakehouse log table in the meta_lakehouse database
if not df_log.filter(df_log["row_amount_diff"] > 0).rdd.isEmpty():
    df_log.write.format("delta").mode("append").saveAsTable("meta_lakehouse.load_log_bronze")
    print(f"\n\033[93m ALL CSV files LOADED with WARNING into meta_lakehouse.load_log_bronze table.\033[0m")
else:
    print(f"\n\033[92m ALL CSV files SUCCESSFULLY LOADED into meta_lakehouse.load_log_bronze table.\033[0m")

The strange code within print(f” ) is a code for coloring text. The difference we see in the warning output and the success output is [93m (yellow) and [92m (green)

At the very end, there’s an output like this:

The last line hints that the next blog post will be about how to find the issues easily and how to fix them. Notebooks are great to instruct people on what to do, especially if they have links to the next notebook!

Do you have any improvements I could have made in my code? Or do you have a question you want to discuss? Even better, found any mistake? Let me know in the comments!

This post was first published on Fabric community blogs for data engineering. Credit: Cover image was created by DALL-E by Microsoft CoPilot.

If you don't know the AdventureWorks database, it is a sample database by Microsoft. It is used to demonstrate the capabilities of SQL Server. The database is used in many examples, tutorials and documentation by Microsoft. You can download the database here.

There are many versions of it, and i'll be using the AdventureWorks2022 database. I've got my database on a local SQL Server instance. The first step is to export the data to csv files. If you're using a different database, please make sure that it isn't a productive database as exporting all data to csv files can be expensive process. This is not to say that it can't be used in a productive environment, but test it on a non-productive environment first. This way you'll be able to see how long it takes and how much cpu and memory it uses on your system.

So, let me walk you through the Python code by the comments within it:

# importing some libraries
import pathlib as pl
import pyodbc
import pandas as pd

# define the folder path to where the csv files will be exported to
folder_path = pl.Path(
    r"C:\Users\userName\OneDrive\csv"
)

# build connection string to the database
conn = pyodbc.connect(
    "DRIVER={ODBC Driver 17 for SQL Server};"
    "SERVER=localhost;"
    "DATABASE=AdventureWorks2022;"
    "Trusted_Connection=yes;"
)

# create cursor
cursor = conn.cursor()

# get all sql server table names
qAllTables = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE'"
table_names = pd.read_sql_query(qAllTables, conn)
print(f"table_names:{table_names}")

# Loop over all tables, get the data with a SELECT *, put it into a df and
# then save the data to csv

print(f"Number of tables: {len(table_names)}")

for table in table_names["TABLE_NAME"]:
    if table == "sysdiagrams" or table == "DatabaseLog":  # Skip system tables
        continue
    query = f"SELECT * FROM {table}"
    df = pd.read_sql(query, conn)
    file_path = folder_path / f"{table}.csv"
    print(f"Saving to: {file_path}")
    df.to_csv(file_path, index=False)

conn.close() # close the connection, always a good move.

I am assuming that you already have a Fabric capacity. If you don't you may be eligible for a free trial. In this case, you may want to follow the steps as described on Microsofts documentation website here.

Next, go into your Fabric capacity. Here, you should create a new workspace and a new lakehouse:

Click on the "Create Workspace" button in the left menu bar (1) which opens a fold-out window. After this, click on the green "+ Create Workspace" button (2) in the bottom left corner.

• Next, on the right side, another flap-out window is opening. Here, you can enter your workspace name. The description is optional. If you can, I advise to use the trial capacity that may be available to you. For this click ont he radio button a bit further down in that very window. At the bottom of this very window, you can click on the green "Apply" button to create the workspace. The other options are optional and are not needed for this tutorial.

Go into your workspace you just created. Click on "Workspaces" and you'll find the workspace that you've just created in the flap-out window on the left.

Next, click on the [+ New] button on the left top menu bar. This will open a drop-down window from where you'll be able to choose a new lakehouse. Click on the "Lakehouse" option.

Next, you'll want to enter your lakehouse name into the pop-up window that appeared. I recommend to use a name that is something like adventureworks_bronze. This makes it easier to understand what this lakehouse contains and that it is a bronze layer of the lakehouse. I'll be explaining the layer concept in a later blog post, but for now, we don't have to worry about that. If you have the option to click on "Lakehouse schemas" that is currently in public preview, I recommend not to choose this. The reason is that it can make the concept a bit more complicated which we don't need for this tutorial. In a later post, I'll cover this, but for now, let's skip this. Click on the green "Create" button to create the lakehouse.

Now that we've created it, we can upload our files. In the middle of the screen (1), we have the possibility to upload the files, but we also could make use of a right click on the folder "Files" (2) that you see on the left side of the screen. The advantage the second "method" is that you actually see the folder where your files will be uploaded to. You can create a folder in that folder if you want to. For this tutorial, i am going to keep this simple and will upload the files directly into this root folder. In the flap-out window on the right (3) you can go to your local folder and select all your files you've just created with the Python script above. If you choose the option "Overwrite if files already exist" you can upload the files again and overwrite the existing ones.

So this is for today, I'll be back with the next blog post on this soon!

Credit: Cover image was created by DALL-E via Microsofts CoPilot.

• Work colleagues or employers
• Mentors
• Friends
• Family Your post does not have to be personal, it can also be just a story you anonymize people so that the reader who it actually matters, will know that you appreciate them. If you even can incorporate some code, please do! Happy blogging and happy SQLTuesday! Please publish your blog article on December 12 2023.

Sorry, my title indeed said "OneDrive" and that is actually by intention. It is really easy to mix them up, just look at this screenshot I did on my Laptop. It shows the file explorer and it very much looks the same.

OneLakes logo is also shown in the taskbar in the bottom right corner next to the clock in Windows, just like OneDrive.

At this point, I want to mention it, since I've fogotten to mention it in the last post. Currently, Microsoft Fabric is still in preview.

So what is the big deal with OneLake? Well, if you are using Fabric, you'll notice that OneLake is pretty much the data center for any of Fabrics data experiences:

• Data Factory

• Synapse Data Engineering

• Synapse Data Warehouse

• Synapse Data Science

• Synapse Real Time Analytics

• Power BI

You could imagine this as a centre for all data. You want to provide data for Data Science? OneLake can provide you this. You also can use it for your DataLake to source your DWH in Fabric. Actually, you could imagine it as the center of Fabric.

OneLake Shortcuts

Microsoft also provides OneDrive Shortcuts. This is a Feature that allows you to unify your data in one place, hence avoiding duplication of your data. It is a virtualization of the data and you can access it from different experiences and places. In other words, you could imagine them as links that point to the actual places of data. I don't want to go much deeper in this, because the previous link to Microsofts documentation is excellent.

OneLake as data driver

A lot of companies strive to be a data driven company. Understandably so, because it means that decisions that need to be taken, be it as an analyst, data scientist, data engineer or anyone of the C-level management, will be based on data. OneLake gives transparency to everyone to the structure that data may provide and any data worker can incorporate them in their work.

So in my opinion, OneLake helps to centralize data to any organization. Architects and Security Engineers need to define together with the business and other data workers, how to organize the data system in the organization. Once this is done, and security permissions are set, I believe, this will propel the mutual understanding of an organizations existing data across departements.

So this was about OneLake. In my next post, I will go deeper in Shortcuts to show you how to use them and to glide in to the next topic.

Caption for cover picture: OneLakes Logo by Microsoft

So finally I've found time to dig deeper into Microsoft Fabric, the newcomer in Microsofts data platform family. I've played a bit with it and I thought I should blog about some distinct differences in between Azure Synapse Analytics and SQL Server. In the beginning, I have to admit, that MS Fabric quite confused me in terms of what it could mean for Azure Synapse.

Now that I've got more knowledge through reading into it and more importantly, playing around with it, I can draw some conclusions. But first and foremost, I think anyone should understand that MS Fabric is a SaaS offering, while Azure Synapse is a PaaS offering. This means that inherently, there are different approaches and use cases for both.

Fabric vs Synapse?

Anyways, I do not think that MS Fabric is going to replace Azure Synapse. I guess both will co-exist. Just like the way for example, Azure SQL and Azure SQL Managed instance co-exist. They both do not mean that they are going to replace SQL Server, neither will any of them replace each other. There are some hints for this, one being this video: {{< youtube 6BI89Y2S-Oo >}} This video deserves much more attention than it has (shout-out to Ginger Grant for pointing me to this video!). Also the channel needs much more attention than it does have!

As I am intending to start a small series about Fabric, I want to begin with some differences, as I mentioned in the beginning of the article.

Roundup

• Fabric has in its core, center, and is functioning around one concept: OneLake. I believe this is the thing that makes the magic of all of it possible in the first place. All services around it, they work incredibly smoothly and well together. I really admire that concept.
• OneLake inherently improves the concept of having a better convergence of people working with data, one and non-duplicated data. Think about this: Data Scientists, Data Engineers, and heck, Business Analysts can share their data in one place. One place to rule them all (data, not the people 😉). I think that is quite revolutionary if you think about the fact that most data today are pretty much scattered in data silos. Fabric is all about breaking up data silos.
• While Fabric as it is today, does integrate many services existing today in one SaaS offering, it does not cover all abilities of each service. Nevertheless, it is a strong, powerful and very enjoyable tool to work with.

So this is the start of the mini series in which I will blog about Fabric. I will try to keep it succinct as possible and blog what I've am learning about Fabric or Azure Synapse Analytics, to help you to understand the differences and similarities of them.

For this month, Rie Merritt called for advices for running a user group. I know, I'm a quite late to join this game. But since I was busy for my university just until this week, I was not able to write for the call more timely - apologies. I hope nobody minds me joining this late.

So I've founded an online-only Azure Data Community user group, Data TGIF, in June last year and have led it ever since. It is still fairly small and only has around 170 members so far, so it is up to you decide whether or not this group is successful.

In my opinion, an important factor for a user group is to set yourself apart from others. There are currently 140 Azure Data Tech Groups in 40 countries worldwide, so especially if your user group are online events, you've got some competition. Get a catchy name and a logo that is unique. My group is called Data TGIF and unlike most other groups, it does not just offer a platform for speakers to speak, but it is also a place to meet others. So it is also an networking event. I'm very happy to see that that way, that this group is able to spark talks amongst the attendees and speakers. Oftentimes, this breaks the ice for everyone. And during those talks, oftentimes new questions pop up and get asked to the speakers. Another thing this user group is set apart from others with is the format: it is 30 mins for talks and 30 mins for the networking part. That way, so I like to believe, there is a good balance in between just listening and interaction of every attendee.

I am vehemently believing that user groups are a great way to start speaking. So if anyone wants to try to speak, I am more than happy to help and if the speaker wishes, I will try to find a mentor for them. User Groups should be a speaking platform for everyone, from the freshest speakers to the most experienced speakers.

Furthermore, try to find channels you're communicating and stick to them. For Data TGIF, I am using 3 channels: MeetUp.com, Twitter and LinkedIn. Be consistent in using them and try to get as many connections as you can. Get in touch with the speakers and provide them marketing materials so that they can do marketing themselves, too.

Last not least, try to get in touch with the community. So far, I haven't seen anyone of the other community founders or leaders biting me. In fact, I can say that I've grown to be friends with some of them and with some of them, I even have plans for future events. Some other user group organizers will even help you out with growing your user group - a shoutout to Cloud Data Driven for this!

Good luck in growing your user group and of course, don't forget the most important part in this: have fun!

So the year 2021 is history. For me, it was a quite eventful year although exactly one year ago, I hoped that the pandemic situation would improve. It kind of did, true. However I hoped it'd be better than it turned out to be. Nevertheless, I hope on January 1st 2023, I'll have a better lookback to the year 2022 regarding this.

In March 2021, I've submitted my thesis for CAS Machine Learning (German) at my University of Applied Sciences in Business Administration Zurich, HWZ. It got accepted, so I received that first CAS diploma of three.

After I've decided to volunteer to moderate some sessions for the both DataWeekenders in 2020, I wanted more. Originally, I planned to start to speak in 2022 after I've finished my studies at University for Applied Sciences HWZ in Zurich where I am pursuing a MAS in Business Intelligence (planning to finish in December 2022). But I got hooked, badly. So I was looking around what it takes to be speaking and also looked for a mentor who I found in Sander Stad. Thank you a lot again, Sander, for your tips and encouragements you gave to me! One of my highlights was truly my speaking debut on DataWeekender where I also volunteered to be a moderator again. I really am looking forward to be moderating and hopefully speaking on this event again! Since, I got the opportunity to speak on some other conferences and user groups so that actually became a new hobby of mine. Instead of listing all opportunities to speak here, you'll find on this site.

In June, I've created the Azure Data Community User Group, Data TGIF. This User Group has an event every month where someone is presenting a session for 30 mins. After the session, the attendees and the speakers have the opportunity to talk to each other and just to do some networking. I'm always excited when we have attendees to meet and talk each others - from around the world. This small event really makes the "SQL Family" closer to a "world village community" and closes a gap many of us feel due to the pandemic.

In November, I've submitted my thesis for CAS Customer Intelligence (German) at my university, received my second CAS diploma in December and shortly after the submission in November, I was given another big opportunity: actually co-organize a whole 24-hours long conference, the Power BI Fest 2021. It was quite a lot of work and I had fun while working on it.

Outlook

If you know me, then you probably know that I'm always having some new projects up my sleeve. I'll tell about one or another soon! I am also working on some new sessions I hope to be select to talk about in this year.

For March 2022, I am looking forward to submit my last CAS thesis in CAS Applied Data Analytics (German) on which I'm currently working on.

In December 2022, I'm hoping to finish my studies at my University.

Thanks

I just want thank all people from the SQL Community here who I've met, befriended, had nice conversations or just gave encouragements to me in the last year. Thank you all for your support and I really hope to see you all not just online, but also in person someday! I am optimistic that I will meet some other, new awesome people of the SQL Family and am looking forward to do so!

Image credit: Image by Moritz Knöringer via Unsplash.

September marks the end of this summer but we've got some exciting news about new events that we'll be joining this month. This post is giving you an overview over these!

We hope you've enjoyed your summer break and are ready for the new season for many new events!

This September '21, I'll have two very interesting events that I will attend as a speaker, panel member or a moderator apart from organizing Data TGIF this Friday:

Data TGIF This month, we'll have a full hour talk given by the outstanding speaker Tracy Boggiano and I'm very much looking forward to listen to her about Mental Health and Wellness in IT: Safeguarding our most precious resource at the Data TGIF user group I'm organizing! The event is this Friday at 15:30 CEST.

Data Platform Summit 2021 This is one of the major SQL Server / Azure SQL / Power BI events worldwide that will be held from September 13-16 around the clock to serve all timezones with having more than 100 sessions. It also features a pre- (8+9 Sept) and post-conference (20+21 Sept). I am going to speak about Automated provisioning of SQL Server VMs on Azure on Sept 13 at 18:15-19:30 CEST where you will be able to chat with me live. You can sign up for this event here. I'm very honored to be a member of the highlighted panel discussion about Diversity, Equity Inclusion on Sept 15 17:30-18:45 CEST. Finally, I'll also be available for a chat-session on Sept 16 at 14:30 CEST.

Future Data Driven Summit is a new event that will be held for the first time. It will be a full day event with different types of topics related to Data & AI, DevOps, PowerBI and Data Visualization, Integration & Automation and cloud infrastructure. I am honored to be holding a keynote and moderating for the Future Data Driven Summit which will be on September 29 at 16:00 CEST and has a very intriguing speaker lineup! You can join this event for free here. At the moment, we have more than 1200 people that have signed up!

I hope you'll be joining in one or even all of my sessions I'll be organizing, giving or moderating! Like always, if you have any questions, feel free to contact me via my contact page!

dbatools really has the ability to change a DBAs work life. It is a free, very powerful and feature rich open source PowerShell module that offers you 550+ commands that allows you to take backups, restore them or just do a lot in SQL Server administration that you would need SSMS and some work for. With dbatools, you can do many tasks more and that with just a few lines of code. This post gives you an introduction to dbatools. dbatools is often called the SQL Server Management Studio (SSMS) for PowerShell. So in short, you can manage a SQL Server instance and its databases with PowerShell which also means that you can use scripts to manage them. Just to give some examples on how dbatools can be used so you’ll get a picture how helpful it can be.

# Simple query with a parameter which could be passed from a variable in the PowerShell code. It also prevents SQL injection.
Invoke-DbaQuery -SqlInstance "serverName" -Database "dbName" -Query 'SELECT * FROM Employees WHERE Name = @name' -SqlParameters @{ Name = "Marc Linder" }

# Another example with multiple parameters, which could be assigned from variables in PowerShell code before, too.
$queryBetween = @{
    "StartDate" = $startdate;
    "EndDate"   = $enddate;
};
Invoke-DbaQuery -SqlInstance "serverName" -Database "dbName" -Query "StoredProcedure_with_in_between_search" -SqlParameters $QueryParameters -CommandType StoredProcedure

# Creates full backups of the two databases dbName1 and dbName2. If you don't specify any database names, it backups all databases on the instance!
# Backups will stored in the default backup folder. You can specify the target folder with -Path though. With -Type Diff you also could choose a differential backup 
Backup-DbaDatabase -SqlInstance "ServerName" -Database dbName1, dbName2 -Path C:\backups\

Very simple, right? And there are 550+ commands like this in total that dbatools provides. Dbatools was invented by Chrissy LeMaire who is a MVP for Cloud and Datacenter Management & also MVP for Data Platform. She develops and maintains it now with Jess Pomfret (MVP Cloud and Datacenter Management) and Shawn Melton (MVP Cloud and Datacenter Management) and with many other contributors.

In my experience, dbatools does not only an awesome job for a lot of tasks around SQL Server but is also relatively easy and fast to develop with. It is ideal for almost any problem you may encounter as a DBA. dbatools will truly change your DBA life! With dbatools you don’t develop your hardest parts of the scripts for a DBA problem alone anymore as dbatools is open source (see code on github here) and thus tested within the community. A few lines of codes will boost your scripts that you would not be able to write alone.

If you’re worried about security, Chrissy also wrote a very insightful post on dbatools.io on how secure dbatools is – spoiler: its very secure.

So if you want to use dbatools, you can for example, just use SQL Server Job Agent which allows you to start your script according your needs, e.g. after a load or taking backups at a past midnight time very easily. Of course, you also can commit the code into a git repo and use it with SQL Server Job.

If you like to know more about dbatools, I recommend to buy and read the book “Learn dbatools in a Month of Lunches” by Chrissy LeMaire, Rob Sewell, Jess Pomfret and Cláudio Silva.

# How to install dbatools:
# Via PowerShell Gallery (Win 10+ / Win Server 2016+)
Install-Module dbatools
# Via Choclatey:
choco install dbatools -y

More install options are described here and there is also an option with downloading it if your server does not have any internet connection.

If you want to update dbatools, it of course also has a very neat tool to update itself:

Update-dbatools

If you need to choose the download option, it may happen that after you’ve installed a newer version of dbatools, you’ll end up having more than just one version of dbatools. In that case, you may want to remove older versions. To do that, you can use the following code as documented here:

& "$((Get-InstalledModule dbatools).InstalledLocation)\cleanup.ps1"

If you have any questions about dbatools, please feel free contact me. I’m happy to answer you!

I just started to organize a free to join MeetUp called Data TGIF that will be scheduled once a month on a Friday afternoon. This MeetUp organizes free data events for Friday afternoons to celebrate the start of the weekend by meeting people to a Microsoft Data Platform talk. If you don’t know the term TGIF: According to Wikipedia, TGIF stands for Thank God It's Friday and is a common expression in English speaking Western countries. It expresses gratification that the working week is nearly over, and a weekend of leisure will soon be here.

The goal of this group is to spread knowledge in Microsoft Data Platform and to give opportunities for national and international networking to the Swiss community. Primarily, this MeetUp will be held online so that everyone can participate from their own place. As soon as the pandemic situation will improve, we will probably start to organize in-person meetings. Meetings are held in English so that we may cover all Swiss language regions.

The speaker usually will have 30 mins for their presentation and there will be 30-45 mins for networking afterwards although there may be exceptions to this rule. If we host an in-person event, it may have a different format than this.

The first speaker will be Andy Cutler who is an Azure Data Platform professional working predominantly with Azure Synapse Analytics (SQL Pools), Data Factory, SQL Server and Power BI. He holds an MSc in Business Intelligence and Data Mining.

Andy will be speaking about Creating a Logical Data Warehouse using Azure Synapse Analytics Serverless SQL Pools.

We are looking forward to this event and would be happy to see you there! Please come and sign up for Andys session for free!

Introduction

ADS is a free, open source and cross-platform SQL code editor with built-in IntelliSense that is also able to manage SQL Server in Azure, so just this description alone sets it quite apart from SSMS. And yes indeed, it is cross platform which means you can use it on Mac OS and on Linux.

Features that SSMS does not have

ADS boasts some features that SSMS doesn’t have. Most important examples are:

Extensions ecosystem similar to that one like Visual Studio Code Source control for git out of the box Jupyter Notebooks for SQL, PowerShell, Python and more Dashboard for an excellent overview over your Instance Integrated terminal Excellent in connecting to Azure Dark mode ADS is builds upon the same system as Visual Studio Code (VSCode). Don’t mix it up with Visual Studio – VSCode is an very extensible and powerful editor whereas Visual Studio is a fully fledged IDE. However, if you want, you can use extensions to make VSCode to a very powerful system that is close to an IDE or even meets it. If you’re familiar with VSCode, you’ll be able to adapt to ADS very quickly.

So why still use SSMS?

Well in short, SSMS still has some capabilities that ADS does not have, especially for administering SQL Server Instance. However, for most if not all SQL developments, ADS can do the same as SSMS or even better. When it comes to administering SQL Server though, SSMS is still more powerful. For example, if you want to configure Always On Groups, using Maintenance Plans, PolyBase, or Replication, SSMS is still the winner. Microsoft says that ADS does not replace SSMS. In my opinion, perhaps it will one day, but for the foreseeable future, it certainly does not yet.

But can I avoid installing ADS when installing SSMS?

Firstly, I personally advise to install ADS as well in most DEV environments because it does have some very interesting and useful features. But to answer this question, if you’re installing it via graphical user interface, then you are unfortunately forced to install ADS, too. If you must avoid the installation of ADS, for example because you want to have a hardened PRD, you can avoid it if you install it via command line this way:

SSMS-Setup-ENU.exe /Passive DoNotInstallAzureDataStudio=1

If I uninstall SSMS, will ADS be uninstalled too? No. If you want to uninstall both, you’ll have to uninstall them separately.

This was a short introduction in this topic. If you have more questions, we will be happy to answer hem!

At the end of March this year, I’ve decided that I want to start to speak this year. Originally, I’ve planned to start to do so after my studies for Master of Advanced Studies in Business Intelligence at the University of Applied Sciences in Business Administration Zurich. However, I’ve been wanting to start speaking for a longer time, so I’ve just gave it a go after I would finish my first part of this studies for which I received the Certificate of Advanced Studies in Machine Learning just recently. If everything goes as planned, I’ll be finishing my studies in spring 2022.

Originally, I’ve planned to start at a New Stars Of Data event in 2022 and this spring, it actually fell in the time when I had to write the thesis for the CAS Machine Learning so that was not an option. But then there was DataWeekender for which I decided to apply to speak for.

So I went to the website for looking a mentor to get me some insights on how one should prepare and deliver a speaking. Through speakingmentors.com, I quickly found out that Sander Stad specializes more or less the same topic I’ve decided to talk about and contacted him. Sander is a very experienced speaker and did give me more than a few tips and tricks on how to improve a talk. Angela was right: speakingmentors.com is a very helpful resource you should turn to if you’re deciding to speak for the first time.

I would like to thank some people who gave me their support in this little journey:

• My employer OneDigit AG who supported me to start speaking on events
• My wife Vivien who supported me by supporting me emotionally and giving me the time to prepare my speaking.
• Sander Stad for his mentorship, advices and encouragements from his enormous experience he gave me. It was my honor to be your mentee, Sander! And I might add, we actually had some very nice conversations together! I'll owe you more than just a beer!
• Angela Henry for casually pointing me to speakingmentors.com and sparked the idea in me in the first place. Thank you a lot for that!
• Alex Yates for having the idea and actually creating this website with which you may find your future mentor. A nice and funny guy whom I've had the chance to meet over MS Teams once. Thank you!
• The guys behind DataWeekender: Kevin Chant, Gethyn Ellis, Mark Hayes and Damir Matešić who gave my the first opportunity to speak publicly. That was a great experience and your event will always be specially remembered!
• Nikola Illic who was encouraging me for speaking almost until I had to switch my microphone on. Thanks!
• A huge thanks to the audience I’ve had - I was very nervous, but I hope that wasn't too noticeable!

Thank you so much to everybody! Without you, my journey wouldn’t had been half fun! I am still learning and will be continuing my journey. And Alex, should I once feel that I’ve got enough experience, I’ll walk on Sanders path and sign up to your site to be a mentor for a next new speaker.

PS: If you’re looking for my DataWeekender session files, they’re over here.

Photo by Howie R on Unsplash

Especially in this time of COVID-19 caused lockdowns, working from home is widespread. Therefore, working together on a developing VM in the cloud may be an attractive solution. In the past few months, I developed a solution to use VMs for such business cases. We also had in our minds that this VM could be a showcase for customers that prefer having their solution on their premises without Azure. Therefore, a VM in Azure in a Infrastructure-as-a-Service (IaaS) solution can make a lot of sense. This can also be a cost-efficient solution for those who want to try out things for a couple of hours. Even if you need this VM for a longer time, the costs are fairly limited, depending on the hardware specifications you are choosing.

The following article is split in three parts. In the first part, a script sets up a VM from scratch. The second walks you through the code and will setup Azure Bastion. And the last part will walk you through the installation of some software including Azure Data Studio via Chocolatey and the restore of a sample database with dbatools.

The full setup takes a bit more than 1 hour and is set up almost completely automatically.

Pre-requisites

Editor and connection to Azure I am using Visual Studio Code with some extensions for Azure. In order to connect to your Azure Account, you need to connect your editor of your choice with Connect-AzAccount.

Files in Azure File Store

You will need to upload a backup file to your file Azure File Store. Azure File Store is a quite cheap storage for storing files like database backups. To upload a backup file, I recommend using Azure Storage Explorer with which you can upload your files in a very convenient and secure way.

Azure Bastion

Azure Bastion will be set up this script automatically. However, I at this point, want to give you some resources first why and when you should use Azure Bastion. If you’re using a VM via Internet, it is always a bad idea to use a RDP connection without an encrypted tunnel.

https://docs.microsoft.com/en-us/azure/bastion/bastion-overview https://jussiroine.com/2019/09/building-a-secure-remote-access-solution-using-azure-bastion-host/

dbatools

We are going to install dbatools on the VM after its deployment. At this point, you don’t need to know much about this awesome collection of PowerShell commandlets. It contains 500+ easy and efficient to use scripts that are free and open source.

Chocolatey

We will also install Chocolatey on the VM. Again, at this point, you don’t really need to know much about it as we will just install some software with Chocolatey. It enables you to automatize, download and install many software with few lines of code, making it very quickly for anybody who has to set up a VM (or computers) from scratch. There is also a paid service by Chocolatey, but we will be using the free service offer.

Lets start with the script

As an overview the following script does the following tasks:

• Creates the VM with SQL Server
• Creates Azure Bastion

Following you’ll see the entire script and after this script in the second part of this article, I’ll walk you through it.

# Variables
## Global Settings
$Location = "switzerlandnorth"
$ResourceGroupName = "bloglab"

## Settings for Storage
$StorageName = $ResourceGroupName + "storage"
$StorageSku = "Standard_LRS"

## Settings for Network
$InterfaceName = $ResourceGroupName + "ServerInterface"
$NsgName = $ResourceGroupName + "nsg"
$VNetName = $ResourceGroupName + "VNet"
$SubnetName = "Default"
$VNetAddressPrefix = "10.0.0.0/16"
$VNetSubnetAddressPrefix = "10.0.0.0/24"
$TCPIPAllocationMethod = "Dynamic"
$DomainName = $ResourceGroupName

## Settings for Bastion
$publicIpName = "pip" + $ResourceGroupName
$BastionName = "Bastion" + $ResourceGroupName
$BastionSubnetName = "AzureBastionSubnet"

## Settings for hardware
$VMName = $ResourceGroupName + "VM"
$ComputerName = $ResourceGroupName + "Server"
$VMSize = "Standard_D4s_v3"
$OSDiskName = $VMName + "OSDisk"

## Settings for SQL Server licence
$PublisherName = "MicrosoftSQLServer"
$OfferName = "sql2019-ws2019"
$Sku = "SQLDEV"
$Version = "latest"

# Set credentials for logging into VM
$Credential = Get-Credential -Message "Type the name and password of the local administrator account." 

# Creation of Objects
## Resource Group
New-AzResourceGroup -Name $ResourceGroupName -Location $Location

## Storage
$StorageAccount = New-AzStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageName -SkuName $StorageSku -Kind "Storage" -Location $Location

## Network
$SubnetConfig = New-AzVirtualNetworkSubnetConfig -Name $SubnetName -AddressPrefix $VNetSubnetAddressPrefix
$VNet = New-AzVirtualNetwork -Name $VNetName -ResourceGroupName $ResourceGroupName -Location $Location -AddressPrefix $VNetAddressPrefix -Subnet $SubnetConfig
$PublicIp = New-AzPublicIpAddress -Name $InterfaceName -ResourceGroupName $ResourceGroupName -Location $Location -AllocationMethod $TCPIPAllocationMethod -DomainNameLabel $DomainName
$NsgRuleRDP = New-AzNetworkSecurityRuleConfig -Name "RDPRule" -Protocol Tcp -Direction Inbound -Priority 1000 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 3389 -Access Allow
$NsgRuleSQL = New-AzNetworkSecurityRuleConfig -Name "MSSQLRule"  -Protocol Tcp -Direction Inbound -Priority 1001 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 1433 -Access Allow
$Nsg = New-AzNetworkSecurityGroup -ResourceGroupName $ResourceGroupName -Location $Location -Name $NsgName -SecurityRules $NsgRuleRDP, $NsgRuleSQL
$Interface = New-AzNetworkInterface -Name $InterfaceName -ResourceGroupName $ResourceGroupName -Location $Location -SubnetId $VNet.Subnets[0].Id -PublicIpAddressId $PublicIp.Id -NetworkSecurityGroupId $Nsg.Id

## Compute
$VirtualMachine = New-AzVMConfig -VMName $VMName -VMSize $VMSize
$VirtualMachine = Set-AzVMOperatingSystem -VM $VirtualMachine -Windows -ComputerName $ComputerName -Credential $Credential -ProvisionVMAgent -EnableAutoUpdate #-TimeZone = $TimeZone
$VirtualMachine = Add-AzVMNetworkInterface -VM $VirtualMachine -Id $Interface.Id
$OSDiskUri = $StorageAccount.PrimaryEndpoints.Blob.ToString() + "vhds/" + $OSDiskName + ".vhd"
$VirtualMachine = Set-AzVMOSDisk -VM $VirtualMachine -Name $OSDiskName -VhdUri $OSDiskUri -Caching ReadOnly -CreateOption FromImage

## Image
$VirtualMachine = Set-AzVMSourceImage -VM $VirtualMachine -PublisherName $PublisherName -Offer $OfferName -Skus $Sku -Version $Version

## Create the VM in Azure
New-AzVM -ResourceGroupName $ResourceGroupName -Location $Location -VM $VirtualMachine

## Add the SQL IaaS Extension with chosen license type
New-AzSqlVM -ResourceGroupName $ResourceGroupName -Name $VMName -Location $Location -LicenseType PAYG

## Create Azure Bastion Host
Add-AzVirtualNetworkSubnetConfig -VirtualNetwork $VNet -Name $BastionSubnetName -AddressPrefix 10.0.1.0/27
$VNet | Set-AzVirtualNetwork
$publicip = New-AzPublicIpAddress -ResourceGroupName $ResourceGroupName -name $publicIpName -location $Location -AllocationMethod Static -Sku Standard
New-AzBastion -ResourceGroupName $ResourceGroupName -Name $BastionName -PublicIpAddress $publicip -VirtualNetworkId $VNet.id

I will walk you through some major settings that the code above has so you may alter them accordingly to your needs.

$Location = "switzerlandnorth"
$ResourceGroupName = "bloglab"

Since I live and work in Greater Zurich Area, I’ve picked switzerlandnorth as my hosted location for Azure. If you want to pick a different one but don’t know your choices, you can get a list with Get-AzLocation.

$ResourceGroupName

Defines the name of your Resource Group. You can think of it as a container that holds all objects you've got within it. This makes configurations for your objects easier. For example, in order to copy or move your objects from "switzerlandnorth" (Zurich) to "switzerlandwest" (Geneva) would mean that you only need to address the container and every single object within this group.

$StorageSku = "Standard_LRS"

This defines what kind of storage you will use. For a better decision what choices you have, check its documentation. In our example, I’ve picked the standard type. Please note that there may be pricing variations amongst the different types which you may want to find out about here.

# Set credentials for logging into VM
$Credential = Get-Credential -Message "Type the name and password of the local administrator account."

Shortly after you start the script, you’ll notice that you need to enter some credentials. This is actually this line that cast this question. In order to access the VM, you need to pass exactly this credentials. In other words, during the creation of the VM, your initial credentials are set so you may access it.

## Settings for Network

Next, you’ll see some networking configurations. These configurations have to be made because you’ll have a virtual net around your virtual machine. You need it also for Azure Bastion which basically lets you access your VM via a secure tunnel via SSL. Your browser will provide the actual access. I’ll show you more about this later, but for now, it is sufficient to say that Azure Bastion makes it easy for you to tunnel your RDP connection. It is not exactly free but it allows you to create a secure RDP connection if you don’t have the means to create an own VPN tunnel. You can find out more about the pricing for Azure Bastion here.

## Settings for Bastion
$BastionAddressPrefix = "10.0.1.0/27"

In the settings for Azure Bastion, we define some names. I’ve used the pretty standard naming conventions that Microsoft suggests here. The last line in these settings is what I want to emphasize here. It is important to note that $BastionAddressPrefix needs to be at least /27 as Microsoft states in its documentation in the bullet point “Subnet” here. This can be a source of confusion if you don’t pay attention to this.

## Settings for hardware
With $VMSize = "Standard_D4s_v3"
$OSDiskName = $VMName + "OSDisk"

With this we define some VM specific “hardware”. Again, with what you decide to go for, the pricing may change. To find out more about this, Microsoft provides a calculator for pricing where you may get your estimates. Some more information about the right sizing for your VM provides Thomas Maurer, a Senior Cloud Advocate at Microsoft. If you want to list which options you have for your preferred location, you can use

Get-AzVMSize -Location "switzerlandnorth"

SQL Server License Since we actually want to have a SQL Server on the VM, we also need to decide what kind of licensing we want. In this installation, we are going to use. If you want to find out what options you have for your preferred location you can type:

## Settings for SQL Server licence
Get-AzVMImageOffer -Location switzerlandnorth -Publisher 'MicrosoftSQLServer'

For more information about the costs and licensing can be found on Microsofts documentation about Pricing guidance for SQL Server on Azure VMs.

So with the code we have so far, we now configured almost all variables. The rest of the code actually creates the needed objects.

If you run the code, you’ll notice that it’ll take about 5 minutes until it continues at some point. The creation of Azure Bastion host takes some while. After the code has completed, after about 20 minutes, you’ll have a fair amount of objects in your resource group, being the VM and the Bastion host amongst them.

I will show you how to set up the generated VM with a SQL Database and various software in an automated way.

Starting and setting up the VM

Now the VM is deployed and already started. Now is also the moment you’ve been waiting for – you want to access to your VM. In order to do so, go to your Azure portal and find your resource and your VM in there. Once you’re clicked on your VM, you’ll see a screen like below. Here, click on “Connect” as below and choose the option Bastion.

After this, you’ll see a site with a button “Use Bastion” on which you click. In the next window, you need to enter the credentials you’ve set for this VM during the running of the script. Your browser starts to connect to your VM. During that time, you will see a small window asking if you want to allow copy & paste. If you allow it, you can copy & paste text onto the VM. It is not possible to copy files into the VM on this way. However, like I’ve already mentioned in the prerequisites, you can use the Azure File Storage Explorer with which you can use drag and drop to upload files to this File Storage. You can map this storage as a network drive so you can access even larger files very conveniently that way. The next few steps will allow you do so.

After your VM started and you see Windows, you might want to have a full screen view on your VM. To do so, click on this >> on the left side

This will slide open a small window like this. In here, you can choose full screen:

And add a File Share:

# Enabling execution of PowerShell scripts:
set-executionpolicy remotesigned -Force

# Mounting Azure File Store as cheap data storage 
$connectTestResult = Test-NetConnection -ComputerName $YourDataStorageName.file.core.windows.net -Port 445
if ($connectTestResult.TcpTestSucceeded) {
    # Save the password so the drive will persist on reboot
    cmd.exe /C "cmdkey /add:`"$YourDataStorageName.file.core.windows.net`" /user:`"Azure\sqlsrvbaks`" /pass:`"$YourPassword`""
    # Mount the drive
    New-PSDrive -Name Z -PSProvider FileSystem -Root "\\YourDataStorageName.file.core.windows.net\wideworldimporters" -Persist
}
else {
    Write-Error -Message "Unable to reach the Azure storage account via port 445. Check to make sure your organization or ISP is not blocking port 445, or use Azure P2S VPN, Azure S2S VPN, or Express Route to tunnel SMB traffic over a different port."
}

With this, we are now able to install chocolatey:

# Install Chocolatey:
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
Chocolatey makes it extremely easy to install a lot of software with just one PowerShell line as following:

# Install Chocolatey GUI and other software:
choco install chocolateygui microsoft-edge tabular-editor daxstudio azure-data-studio git dbatools vscode -y

This installs:

• Chocolatey GUI
• Microsoft Edge
• Tabular Editor
• Dax Studio
• Git
• Dbatools
• Visual Studio Code

Of course, you could add / remove the software as you like in that line. The last -y means that you agree to the installs of all listed software. dbatools takes quite some time to be installed (ca. 10 minutes) so don’t be surprised if it seems to be stuck – it is not.

With this Tabular Editor, .Net Framework 4.8 gets installed. Therefore we need to reboot the VM:

# Rebooting Computer to complete installs
Restart-Computer

You just can keep the browser tab for the VM open until the reboot is done and you can continue.

With dbatools being installed, we would like to restore our databases. But, remember – we need to have the backup files first which we need to bring to our VM first? Now it is the time to map Azure File Store to our VM. If you go to Azure Portal, go to

And add a File Share

And click on the created file share next. In the next window, click on Connect which will open a side window. In here, you can define a mapping letter (eg. Z) and it shows a some code with which you can map your file store.

Copy this code (attention: this has a password in it, keep this secret). Paste this into PowerShell ISE or put it into a file and run it on your VM. As a result, you should see something like this:

If you’ve uploaded a file via Microsoft Storage Explorer, you’ll see the file here. In order to restore this file or even more files, you can use the following code:

# Restore Parameters
$SQLBackupSource = "Z:\*.bak"
# Restore databases
Copy-Item $SQLBackupSource -Destination "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Backup" -Force -Verbose
Restore-DbaDatabase -SqlInstance localhost -Path "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Backup\"

In order to shut down your VM via Code or removing it or even deleting the entire resource group with all its objects in it, those few lines may help you out:

# For stopping this VM: 
Stop-AzVM -Name $VMName -ResourceGroupName $ResourceGroupName

# # For removing this VM: 
Remove-AzVM -ResourceGroupName $ResourceGroupName -Name $VMName

# For removing resource group (and all its objects, including this VM):
$ResourceGroupName = "bloglab"
Remove-AzResourceGroup -Name $ResourceGroupName -Force

This was just a small dive into this topic. Of course it would be possible to automatize this even more in a wholesome DevOps way. It also might be interesting to know that Microsoft offers some tools that may support this idea with:

• Azure Automation
• Azure Lab
• Azure DevOps

If you have a question, feel free to contact me via LinkedIn!