Hello, dear Sitecorians. I hope you are well out there 🙂
I recently had a weird issue where my disk space was suddenly full. This was really strange, I looked everywhere and guess who the culprit was? windowsFilter in the Docker(in ProgramData) folder. It used a lot of disk space.
So why is this?
From time to time when the container/image is removed it leaves orphan layers to C:\ProgramData\docker\windowsfilter without adding -removing suffix to them.
Cleanup() -function is called when Docker daemon is stopping but by default, shutdown timeout is only 15 seconds which is not enough to clean up these images.
*source https://github.com/docker/for-win/issues/745#issuecomment-445243539
NOTE! This is not a Sitecore issue. This is a docker issue
So what to do? Doing the good old docker system prune, did not help.
So after some googling, I found some suggestions:
1.Delete the windowsfilter folder using the docker-ci-zap.exe, https://github.com/moby/docker-ci-zap
But!! Not the best approach according to the comments in https://github.com/moby/moby/issues/26873#issuecomment-249338936
i dont recommend deleting ‘windowsfilter’ manually. it will most likely break the operating system. i posted this problem over 1 1/2 years ago. Amazing that its still not fixed.
2.Adds a -removing -suffix to the orphan layers, and let docker remove them(when closing down the service),
https://github.com/docker/for-win/issues/745#issuecomment-445243539
This is the winner, feels safe and it works 😉
So let’s follow the instructions from the https://github.com/docker/for-win/issues/745#issuecomment-445243539 comment.
Grab/download the Find-OrphanDockerLayers.ps1 – https://gist.github.com/olljanat/340b4033eb24d8d33ec75f2c3c3b6b3d
param ( [switch]$RenameOrphanLayers ) If ($RenameOrphanLayers) { Write-Warning "$($env:COMPUTERNAME) -RenameOrphanLayers option enabled, will rename all orphan layers" } # Get known layers on Docker images [array]$ImageDetails += docker images -q | ForEach { docker inspect $_ | ConvertFrom-Json } ForEach ($Image in $ImageDetails) { $ImageLayer = $Image.GraphDriver.Data.dir [array]$ImageLayers += $ImageLayer $LayerChain = Get-Content "$ImageLayer\layerchain.json" If ($LayerChainFileContent -ne "null") { [array]$ImageParentLayers += $LayerChain | ConvertFrom-Json } } # Get known layes on Docker containers [array]$ContainerDetails = docker ps -a -q | ForEach { docker inspect $_ | ConvertFrom-Json} ForEach ($Container in $ContainerDetails) { [array]$ContainerLayers += $Container.GraphDriver.Data.dir } # Get layers on disk $LayersOnDisk = (Get-ChildItem -Path C:\ProgramData\Docker\windowsfilter -Directory).FullName $ImageLayers += $ImageParentLayers $UniqueImageLayers = $ImageLayers | Select-Object -Unique [array]$KnownLayers = $UniqueImageLayers $KnownLayers += $ContainerLayers # Find orphan layers $OrphanLayersTotal = 0 ForEach ($Layer in $LayersOnDisk) { If ($KnownLayers -notcontains $Layer) { [array]$OrphanLayer += $Layer $LayerSize = (Get-ChildItem -Path $Layer -Recurse -ErrorAction:SilentlyContinue | Measure-Object -Property Length -Sum -ErrorAction Stop).Sum $OrphanLayersTotal += $LayerSize Write-Warning "$($env:COMPUTERNAME) - Found orphan layer: $($Layer -Replace '\r\n','') with size: $(($LayerSize -Replace '\r\n','') / 1MB) MB" If (($RenameOrphanLayers) -and ($Layer -notlike "*-removing")) { $LayerNewPath = $Layer + "-removing" Rename-Item -Path $Layer -NewName $LayerNewPath } } } Write-Host "$($env:COMPUTERNAME) - Layers on disk: $($LayersOnDisk.count)" Write-Host "$($env:COMPUTERNAME) - Image layers: $($UniqueImageLayers.count)" Write-Host "$($env:COMPUTERNAME) - Container layers: $($ContainerLayers.count)" $OrphanLayersTotalMB = $OrphanLayersTotal / 1MB Write-Warning "$($env:COMPUTERNAME) - Found $($OrphanLayer.count) orphan layers with total size $OrphanLayersTotalMB MB"
Let’s run the script:
.\Find-OrphanDockerLayers.ps1 -RenameOrphanLayers
Hua, look at this:
Over 400 GB, that is just crazy…
Next will be to set the shutdown-timeout in the C:\ProgramData\Docker\config\daemon.json or in Docker Desktop:
Next will be to restart the Docker Service, you can do this “manually” from the command prompt or using a bat/powershell script.
Here is the “manual” approach. You can always put it in a bat file, name it RestartDocker.bat
net stop docker net stop com.docker.service taskkill /IM "dockerd.exe" /F taskkill /IM "Docker for Windows.exe" /F net start docker net start com.docker.service
Here is a powershell script, name it RestartDocker.ps1
Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Restarting docker" foreach($svc in (Get-Service | Where-Object {$_.name -ilike "*docker*" -and $_.Status -ieq "Running"})) { $svc | Stop-Service -ErrorAction Continue -Confirm:$false -Force $svc.WaitForStatus('Stopped','00:00:20') } Get-Process | Where-Object {$_.Name -ilike "*docker*"} | Stop-Process -ErrorAction Continue -Confirm:$false -Force foreach($svc in (Get-Service | Where-Object {$_.name -ilike "*docker*" -and $_.Status -ieq "Stopped"} )) { $svc | Start-Service $svc.WaitForStatus('Running','00:00:20') } Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Starting Docker Desktop" & "C:\Program Files\Docker\Docker\Docker Desktop.exe" $startTimeout = [DateTime]::Now.AddSeconds(90) $timeoutHit = $true while ((Get-Date) -le $startTimeout) { Start-Sleep -Seconds 10 $ErrorActionPreference = 'Continue' try { $info = (docker info) Write-Verbose "$((Get-Date).ToString("HH:mm:ss")) - `tDocker info executed. Is Error?: $($info -ilike "*error*"). Result was: $info" if ($info -ilike "*error*") { Write-Verbose "$((Get-Date).ToString("HH:mm:ss")) - `tDocker info had an error. throwing..." throw "Error running info command $info" } $timeoutHit = $false break } catch { if (($_ -ilike "*error during connect*") -or ($_ -ilike "*errors pretty printing info*") -or ($_ -ilike "*Error running info command*")) { Write-Output "$((Get-Date).ToString("HH:mm:ss")) -`t Docker Desktop startup not yet completed, waiting and checking again" } else { Write-Output "Unexpected Error: `n $_" return } } $ErrorActionPreference = 'Stop' } if ($timeoutHit -eq $true) { throw "Timeout hit waiting for docker to startup" } Write-Output "$((Get-Date).ToString("HH:mm:ss")) - Docker restarted"
I had to do this a number of times, because of the size. It doesn’t remove them all, it will take 5-10 Gb on each “restart”.
.\RestartDocker.bat
or
.\RestartDocker.ps1
And that did it, it worked like a charm 🙂
I hope this will help anyone out there having disk space issues running docker.
That’s all for now folks 🙂