Disk space full running your Sitecore Docker instances

Hello, dear Sitecorians. I hope you are well out there 🙂

I recently had a weird issue where my disk space was suddenly full. This was really strange, I looked everywhere and guess who the culprit was? windowsFilter in the Docker(in ProgramData) folder. It used a lot of disk space.

So why is this?

From time to time when the container/image is removed it leaves orphan layers to C:\ProgramData\docker\windowsfilter without adding -removing suffix to them.
Cleanup() -function is called when Docker daemon is stopping but by default, shutdown timeout is only 15 seconds which is not enough to clean up these images.

*source https://github.com/docker/for-win/issues/745#issuecomment-445243539

NOTE! This is not a Sitecore issue. This is a docker issue

So what to do? Doing the good old docker system prune, did not help.

So after some googling, I found some suggestions:
1.Delete the windowsfilter folder using the docker-ci-zap.exe, https://github.com/moby/docker-ci-zap
But!! Not the best approach according to the comments in https://github.com/moby/moby/issues/26873#issuecomment-249338936
i dont recommend deleting ‘windowsfilter’ manually. it will most likely break the operating system. i posted this problem over 1 1/2 years ago. Amazing that its still not fixed.

2.Adds a -removing -suffix to the orphan layers, and let docker remove them(when closing down the service),
https://github.com/docker/for-win/issues/745#issuecomment-445243539
This is the winner, feels safe and it works 😉

So let’s follow the instructions from the https://github.com/docker/for-win/issues/745#issuecomment-445243539 comment.

Grab/download the Find-OrphanDockerLayers.ps1 – https://gist.github.com/olljanat/340b4033eb24d8d33ec75f2c3c3b6b3d

param (
    [switch]$RenameOrphanLayers
)

If ($RenameOrphanLayers) {
	Write-Warning "$($env:COMPUTERNAME) -RenameOrphanLayers option enabled, will rename all orphan layers"
}

# Get known layers on Docker images
[array]$ImageDetails += docker images -q | ForEach { docker inspect $_ | ConvertFrom-Json }
ForEach ($Image in $ImageDetails) {
	$ImageLayer = $Image.GraphDriver.Data.dir
	
	[array]$ImageLayers += $ImageLayer
	$LayerChain = Get-Content "$ImageLayer\layerchain.json"
	If ($LayerChainFileContent -ne "null") {
		[array]$ImageParentLayers += $LayerChain | ConvertFrom-Json
	}
}

# Get known layes on Docker containers
[array]$ContainerDetails = docker ps -a -q | ForEach { docker inspect $_ | ConvertFrom-Json}
ForEach ($Container in $ContainerDetails) {
	[array]$ContainerLayers += $Container.GraphDriver.Data.dir
}

# Get layers on disk
$LayersOnDisk = (Get-ChildItem -Path C:\ProgramData\Docker\windowsfilter -Directory).FullName
$ImageLayers += $ImageParentLayers
$UniqueImageLayers = $ImageLayers | Select-Object -Unique
[array]$KnownLayers = $UniqueImageLayers
$KnownLayers += $ContainerLayers

# Find orphan layers
$OrphanLayersTotal = 0
ForEach ($Layer in $LayersOnDisk) {
	If ($KnownLayers -notcontains $Layer) {
		[array]$OrphanLayer += $Layer
		$LayerSize = (Get-ChildItem -Path $Layer -Recurse -ErrorAction:SilentlyContinue | Measure-Object -Property Length -Sum -ErrorAction Stop).Sum
		$OrphanLayersTotal += $LayerSize
		Write-Warning "$($env:COMPUTERNAME) - Found orphan layer: $($Layer -Replace '\r\n','') with size: $(($LayerSize -Replace '\r\n','') / 1MB) MB"
		
		If (($RenameOrphanLayers) -and ($Layer -notlike "*-removing")) {
			$LayerNewPath = $Layer + "-removing"
			Rename-Item -Path $Layer -NewName $LayerNewPath
		}
	}
}

Write-Host "$($env:COMPUTERNAME) - Layers on disk: $($LayersOnDisk.count)"
Write-Host "$($env:COMPUTERNAME) - Image layers: $($UniqueImageLayers.count)"
Write-Host "$($env:COMPUTERNAME) - Container layers: $($ContainerLayers.count)"
$OrphanLayersTotalMB = $OrphanLayersTotal / 1MB
Write-Warning "$($env:COMPUTERNAME) - Found $($OrphanLayer.count) orphan layers with total size $OrphanLayersTotalMB MB"

Let’s run the script:

.\Find-OrphanDockerLayers.ps1 -RenameOrphanLayers

Hua, look at this:

Over 400 GB, that is just crazy…

Next will be to set the shutdown-timeout in the C:\ProgramData\Docker\config\daemon.json or in Docker Desktop:

Next will be to restart the Docker Service, you can do this “manually” from the command prompt or using a bat/powershell script.
Here is the “manual” approach. You can always put it in a bat file, name it RestartDocker.bat

net stop docker
net stop com.docker.service
taskkill /IM "dockerd.exe" /F
taskkill /IM "Docker for Windows.exe" /F
net start docker
net start com.docker.service

Here is a powershell script, name it RestartDocker.ps1

Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Restarting docker"

foreach($svc in (Get-Service | Where-Object {$_.name -ilike "*docker*" -and $_.Status -ieq "Running"}))
{
    $svc | Stop-Service -ErrorAction Continue -Confirm:$false -Force
    $svc.WaitForStatus('Stopped','00:00:20')
}

Get-Process | Where-Object {$_.Name -ilike "*docker*"} | Stop-Process -ErrorAction Continue -Confirm:$false -Force

foreach($svc in (Get-Service | Where-Object {$_.name -ilike "*docker*" -and $_.Status -ieq "Stopped"} ))
{
    $svc | Start-Service 
    $svc.WaitForStatus('Running','00:00:20')
}

Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Starting Docker Desktop"
& "C:\Program Files\Docker\Docker\Docker Desktop.exe"
$startTimeout = [DateTime]::Now.AddSeconds(90)
$timeoutHit = $true
while ((Get-Date) -le $startTimeout)
{

    Start-Sleep -Seconds 10
    $ErrorActionPreference = 'Continue'
    try
    {
        $info = (docker info)
        Write-Verbose "$((Get-Date).ToString("HH:mm:ss")) - `tDocker info executed. Is Error?: $($info -ilike "*error*"). Result was: $info"

        if ($info -ilike "*error*")
        {
            Write-Verbose "$((Get-Date).ToString("HH:mm:ss")) - `tDocker info had an error. throwing..."
            throw "Error running info command $info"
        }
        $timeoutHit = $false
        break
    }
    catch 
    {

        if (($_ -ilike "*error during connect*") -or ($_ -ilike "*errors pretty printing info*")  -or ($_ -ilike "*Error running info command*"))
        {
            Write-Output "$((Get-Date).ToString("HH:mm:ss")) -`t Docker Desktop startup not yet completed, waiting and checking again"
        }
        else
        {
            Write-Output "Unexpected Error: `n $_"
            return
        }
    }
    $ErrorActionPreference = 'Stop'
}
if ($timeoutHit -eq $true)
{
    throw "Timeout hit waiting for docker to startup"
}

Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Docker restarted"

I had to do this a number of times, because of the size. It doesn’t remove them all, it will take 5-10 Gb on each “restart”.

.\RestartDocker.bat

.\RestartDocker.ps1

And that did it, it worked like a charm 🙂

I hope this will help anyone out there having disk space issues running docker.

That’s all for now folks 🙂

Visions In Code

Never look back