We use Remote Desktop Services to provide our clients with SaaS access to our Windows desktop application. An issue on one of our cloud servers running Windows Server 2019 has been plaguing me for a while. Server performance was sluggish, and I noticed in Task Manager that wsappx was using high CPU. That’s the AppX Deployment Service, which is responsible for installing UWP/Windows Store apps for each user. Even though we don’t have Windows Store installed on the server, there are still certain system AppX packages that are installed for every user. These are in the C:\Windows\SystemApps folder, and configuration of these apps is stored in the %localappdata%\Packages folder for each user. I noticed that the Packages folder was empty for users added after a certain date. Another symptom was that the Start menu did not function for these users. Installation was failing and Windows continually retried the installation for these users. Hence, the high CPU usage. Running Get-AppXLog in Powershell revealed the following error:
Error 0x800705AA: While processing the request, the system failed to register the windows.stateExtension extension due to the following error: Insufficient system resources exist to complete the requested service.Uh… okay. It would be REALLY handy to know what that “insufficient resource” is! The most insufficient resource is this error message! My ISP was stumped as well. So, I started trying the usual things: DISM and SFC repairs, chkdsk, disabled firewall, etc. I eventually copied an image of the server to my local machine and ran it in Hyper-V, so I could try more invasive things without affecting users on the live server. I started uninstalling programs and even ran an in-place upgrade to Server 2022. Nothing helped. By now, many would have spun up a new server and called it a day. In this case, that would disrupt the work of 200+ users, not to mention cause a bunch of support calls while these users tried to migrate to the new server.
One of the behaviors was that a folder for each app would be created in the Packages folder during installation, and then quickly deleted. I needed to know if this was a symptom of the failure or the cause of it (maybe something else was deleting the folders during installation). So, I loaded up Process Monitor (SysInternals) to do some forensics. I filtered for events on the Packages folder and tried to install one of the AppX packages from Powershell. I could see that wsappx was deleting the folder (not some other process), meaning the deletion was a symptom of the failure, not the cause. Then, I changed the filtered on Command Line to capture all events of the wsappx service, which included all directories and registry access. This was thousands of entries, so I found where the package folder was being deleted and started looking above it for clues to what caused the failure. Lo and behold! One nearby entry had the result INSUFFICIENT RESOURCES. It occurred when attempting to add a value to the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Notifications. I found that key in RegEdit and it took a while to open… a good sign I was on the right track. I then tried to manually add a value to that key and got an error. Bingo! Basically, that registry key was full and could not accept any more values. That led me to the following article:
That article points to a hotfix and a utility to clean up the registry bloat causing the problem. Since it was designed for Windows 8.1, I did not feel comfortable installing it on my Windows Server 2019. Fortunately, I found another utility that addresses the issue:
https://github.com/Lazy-256/clnotifications
I downloaded/unzipped clnotifications.zip then ran the fix. It reduced the Notifications registry section from 256K entries to 500. AppX package installations are now succeeding and wsappx is no longer hogging CPU.
Maybe this article will help someone else that encounters the same error. If this isn’t your exact problem, RegistrySizeLimit or excessive firewall entries could also be the culprit. And if you get stuck, Process Monitor can help you find where the issue is.