We use Remote Desktop Services to provide our clients with SaaS access to our Windows desktop application. An issue on one of our cloud servers running Windows Server 2019 has been plaguing me for a while. Server performance was sluggish, and I noticed in Task Manager that wsappx was using high CPU. That’s the AppX Deployment Service, which is responsible for installing UWP/Windows Store apps for each user. Even though we don’t have Windows Store installed on the server, there are still certain system AppX packages that are installed for every user. These are in the C:\Windows\SystemApps folder, and configuration of these apps is stored in the %localappdata%\Packages folder for each user. I noticed that the Packages folder was empty for users added after a certain date. Another symptom was that the Start menu did not function for these users. Installation was failing and Windows continually retried the installation for these users. Hence, the high CPU usage. Running Get-AppXLog in Powershell revealed the following error:
Error 0x800705AA: While processing the request, the system failed to register the windows.stateExtension extension due to the following error: Insufficient system resources exist to complete the requested service.Uh… okay. It would be REALLY handy to know what that “insufficient resource” is! The most insufficient resource is this error message! My ISP was stumped as well. So, I started trying the usual things: DISM and SFC repairs, chkdsk, disabled firewall, etc. I eventually copied an image of the server to my local machine and ran it in Hyper-V, so I could try more invasive things without affecting users on the live server. I started uninstalling programs and even ran an in-place upgrade to Server 2022. Nothing helped. By now, many would have spun up a new server and called it a day. In this case, that would disrupt the work of 200+ users, not to mention cause a bunch of support calls while these users tried to migrate to the new server.
One of the behaviors was that a folder for each app would be created in the Packages folder during installation, and then quickly deleted. I needed to know if this was a symptom of the failure or the cause of it (maybe something else was deleting the folders during installation). So, I loaded up Process Monitor (SysInternals) to do some forensics. I filtered for events on the Packages folder and tried to install one of the AppX packages from Powershell. I could see that wsappx was deleting the folder (not some other process), meaning the deletion was a symptom of the failure, not the cause. Then, I changed the filtered on Command Line to capture all events of the wsappx service, which included all directories and registry access. This was thousands of entries, so I found where the package folder was being deleted and started looking above it for clues to what caused the failure. Lo and behold! One nearby entry had the result INSUFFICIENT RESOURCES. It occurred when attempting to add a value to the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Notifications. I found that key in RegEdit and it took a while to open… a good sign I was on the right track. I then tried to manually add a value to that key and got an error. Bingo! Basically, that registry key was full and could not accept any more values. That led me to the following article:
That article points to a hotfix and a utility to clean up the registry bloat causing the problem. Since it was designed for Windows 8.1, I did not feel comfortable installing it on my Windows Server 2019. Fortunately, I found another utility that addresses the issue:
https://github.com/Lazy-256/clnotifications
I downloaded/unzipped clnotifications.zip then ran the fix. It reduced the Notifications registry section from 256K entries to 500. AppX package installations are now succeeding and wsappx is no longer hogging CPU.
Maybe this article will help someone else that encounters the same error. If this isn’t your exact problem, RegistrySizeLimit or excessive firewall entries could also be the culprit. And if you get stuck, Process Monitor can help you find where the issue is.
Thank you very much.
Got same issue on Windows server 2016.
Utility you’ve given worked like a charm.
Hello Joel,
Thank you for your post that help me to find and understand the same issue on my customers RDS Farm 2016/2019/2022.
On one server we have 260’000 entries…. in Notifications Keys
After deleted all done and all work
Thanks Joel, you’re a lifesaver. I encountered the same issue on a RemoteApp-Server on Windows Server 2016. Start menu, notification center and the settings app wouldn’t open anymore. After running the tool, it reduced the registry entries from 262000 to 500. After a reboot, everything is working again 🙂
Te agradezco Joel, llegue al mismo punto de recursos insuficientes y en el proceso de filtrar los eventos eran tantas entradas que estuve por darme por vencido, hasta que encontré tu blog, corrobore la clave y Bingo, llevaba casi 3 semanas con el error entendiendo la lógica del funcionamiento del UWP en los tiempos libres y la falta que habia de archivos en el proceso de creación de usuarios identificando las carpetas que se borraban, los eventos de Appx me ayudaron mucho. Muy agradecido con toda tu labor.
English translation: I appreciate it, Joel. I reached the same point of insufficient resources, and while filtering the events, there were so many entries that I was about to give up until I found your blog. I verified the key and Bingo! I had been dealing with the error for almost 3 weeks, trying to understand the logic of UWP functionality in my free time and identifying the missing files in the user creation process, recognizing the folders that were being deleted. The Appx events helped me a lot. Very grateful for all your work.