Thanks for the info. I took a look at those links. We already do most of the suggestions; each app has its own app pool that runs under a custom Identity (a service account we created just for the IIS processes).
I’ve spent the day trying to isolate the issue and I think I see what’s going on. The underlying MachineKey file is being created with the wrong ACL permissions, which prevents the file from ever being modified or removed. My guess is that this is due to the combination of connecting to Postgres with SSL client certificates and using an ASP.NET application running under a custom service account identity (instead of the default ApplicationPoolIdentity).
To help debug, I created a simple ASP.NET web app that runs a basic query:
Code: Select all
string myConnStr = ConfigurationManager.ConnectionStrings["MyConnectionString"].ConnectionString;
using (var pgSqlConnection = new PgSqlConnection(myConnStr))
{
pgSqlConnection.Open();
var cmd = pgSqlConnection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM pg_stat_activity;";
var ret = Convert.ToInt32(cmd.ExecuteScalar());
return "There are currently " + ret + " active database connections.";
}
I then created a new web site and a new application pool in IIS, leaving all the defaults except for the Application Pool Identity which I changed from ApplicationPoolIdentity to our custom service account (MYDOMAIN\IISServiceAccount).
When I hit the web app and it connects to the database, a new key file is created in the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder. If I right-click and view the Properties, then under Security I see that the owner is my service account (as expected), but the OWNER RIGHTS is only set to “Read permissions”. However, there is another ACL entry that grants Full Control to the IIS ApplicationPoolIdentity account ("IIS AppPool\{App pool name}") -- despite the fact that I specifically configured the application pool to use a custom Identity instead.
You can see this a little more clearly in the last entry of the PowerShell Get-Acl output:
Code: Select all
PS> Get-Acl "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\2043c0bc8afc723bafb54183f657bf07_
aae40b3a-ad44-409e-88c5-c23c5125b58a" | format-list
Path : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\2043c0bc8afc723bafb54183
f657bf07_aae40b3a-ad44-409e-88c5-c23c5125b58a
Owner : MYDOMAIN\IISServiceAccount
Group : MYDOMAIN\Domain Users
Access : OWNER RIGHTS Allow ReadPermissions
NT AUTHORITY\SYSTEM Allow FullControl
BUILTIN\Administrators Allow FullControl
IIS APPPOOL\CrytoWebAppPool Allow FullControl
I’ve noticed that if I change the app pool Identity back to ApplicationPoolIdentity, then the machine key file in the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder is cleaned up after stopping the app pool. If I leave the Identity set to my custom service account, the file is never removed and a new one is created each time the app pool recycles. As the number of these files grows, the probability of the app accessing one of the existing key containers increases. When that happens, it gets an Access Denied, and throws the “Couldn't acquire crypto service provider context” exception.
I’ve managed to get the exception to happen almost every time (instead of randomly once per week) by manually creating a bunch of machine key files with the same Key Container naming scheme, which I reverse engineered from the ASCII string within the existing files.
I ran this code from within the web app so that the resulting keys would have the same owner (IONHARRIS\IISServiceAccount) as the real keys created automatically.
Code: Select all
// Causes a ton of private keys to be written to the machine key folder. Use the key container
// naming scheme “{GUID}{DOMAIN}{ACCOUNT NAME}{5-DIGIT INTEGER}” to ensure conflicts.
for (int i = 1; i < 30000; i++)
{
string baseContainerName = "{48959A69-B181-4cdd-B135-7565701307C5}MYDOMAINServiceAccountName";
var provider = new RSACryptoServiceProvider(new CspParameters
{
KeyContainerName = baseContainerName + i,
Flags = CspProviderFlags.UseMachineKeyStore
});
provider.Dispose();
}
Afterwards every database connection attempt failed with the "Couldn't acquire crypto service provider context" exception.
So it now seems pretty clear that the keys are being created with too few permissions to be disposed of later, and eventually build up to the point that they begin to prevent future connections. Trouble is, I can’t tell why the underlying machine key is being created with an ACL that uses the name of the Application Pool instead of using the service account Identity.
It doesn’t seem like the file should be owned by one user account but give full control to another (in this case, unused) user account. Of course, changing the app pool Identity back to ApplicationPoolIdentity fixes the issue in my sample app, but not in the production system where we do want the apps to run under a managed service account.
Thanks for your help so far. Any additional guidance would be greatly appreciated.
- Steven
