Otdfbt

I am attempting to execute the following Powershell command:

Enter-PSSession -ComputerName localhost

The server in use is running Windows Server 2008 R2 SP1 64-bit. The server is on a domain. I am logged in under my domain administrator account. The powershell session was started as Administrator.

I'm getting the following error message from powershell itself:

PS C:UsersDaniel> Enter-PSSession -Computername localhost
Enter-PSSession : Connecting to remote server localhost failed with the following error message : The client cannot
connect to the destination specified in the request. Verify that the service on the destination is running and is
accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most
commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to
analyze and configure the WinRM service: "winrm quickconfig". For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -Computername localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : InvalidArgument: (localhost:String) [Enter-PSSession], PSRemotingTransportException
 + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

PS C:UsersDaniel>

Using Event Viewer, I am able to hunt down the following two errors under Applications and Services Logs > Microsoft > Windows > Windows Remote Management > Operational

General:
 The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig".
Detail:
 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft-Windows-WinRM" Guid="A7975C8F-AC13-49F1-87DA-5A984A4AB417" /> 
 <EventID>161</EventID> 
 <Version>0</Version> 
 <Level>2</Level> 
 <Task>7</Task> 
 <Opcode>0</Opcode> 
 <Keywords>0x400000000000000a</Keywords> 
 <TimeCreated SystemTime="2016-08-17T23:10:40.766446000Z" /> 
 <EventRecordID>56814</EventRecordID> 
 <Correlation ActivityID="0190DC40-F800-0000-3291-5DB0DAF8D101" /> 
 <Execution ProcessID="7888" ThreadID="7912" /> 
 <Channel>Microsoft-Windows-WinRM/Operational</Channel> 
 <Computer>FNZAS2.flow.net.nz</Computer> 
 <Security UserID="S-1-5-21-2875926586-1071052228-4104636349-1151" /> 
 </System>
 <EventData>
 <Data Name="authFailureMessage">The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig".</Data> 
 </EventData>
 </Event> 

General:
 WSMan operation CreateShell failed, error code 2150858770
Detail:
 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft-Windows-WinRM" Guid="A7975C8F-AC13-49F1-87DA-5A984A4AB417" /> 
 <EventID>142</EventID> 
 <Version>0</Version> 
 <Level>2</Level> 
 <Task>10</Task> 
 <Opcode>2</Opcode> 
 <Keywords>0x4000000000000002</Keywords> 
 <TimeCreated SystemTime="2016-08-17T23:10:40.766446000Z" /> 
 <EventRecordID>56816</EventRecordID> 
 <Correlation ActivityID="0190DC40-F800-0000-2F91-5DB0DAF8D101" /> 
 <Execution ProcessID="7888" ThreadID="7912" /> 
 <Channel>Microsoft-Windows-WinRM/Operational</Channel> 
 <Computer>FNZAS2.flow.net.nz</Computer> 
 <Security UserID="S-1-5-21-2875926586-1071052228-4104636349-1151" /> 
 </System>
 <EventData>
 <Data Name="operationName">CreateShell</Data> 
 <Data Name="errorCode">2150858770</Data> 
 </EventData>
 </Event>

I've been trying quite a few things to verify everything. Here's some more longform powershell output to show some of my working so far.

PS C:UsersDaniel> $PSVersionTable.PSVersion

Major Minor Build Revision
----- ----- ----- --------
4 0 -1 -1


PS C:UsersDaniel> winrm quickconfig
WinRM service is already running on this machine.
WinRM is already set up for remote management on this computer.

PS C:UsersDaniel> Enable-PSRemoting
WinRM Quick Configuration
Running command "Set-WSManQuickConfig" to enable remote management of this computer by using the Windows Remote
Management (WinRM) service.
 This includes:
 1. Starting or restarting (if already started) the WinRM service
 2. Setting the WinRM service startup type to Automatic
 3. Creating a listener to accept requests on any IP address
 4. Enabling Windows Firewall inbound rule exceptions for WS-Management traffic (for http only).

Do you want to continue?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): A
WinRM is already set up to receive requests on this computer.
WinRM is already set up for remote management on this computer.

Confirm
Are you sure you want to perform this action?
Performing the operation "Set-PSSessionConfiguration" on target "Name: microsoft.powershell SDDL:
O:NSG:BAD:P(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD). This lets selected users remotely run Windows PowerShell
commands on this computer.".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): A

PS C:UsersDaniel> Enable-PSRemoting -force
WinRM is already set up to receive requests on this computer.
WinRM is already set up for remote management on this computer.

PS C:UsersDaniel> winrm get winrm/config
Config
 MaxEnvelopeSizekb = 500
 MaxTimeoutms = 60000
 MaxBatchItems = 32000
 MaxProviderRequests = 4294967295
 Client
 NetworkDelayms = 5000
 URLPrefix = wsman
 AllowUnencrypted = true [Source="GPO"]
 Auth
 Basic = true [Source="GPO"]
 Digest = true
 Kerberos = true
 Negotiate = true
 Certificate = true
 CredSSP = true [Source="GPO"]
 DefaultPorts
 HTTP = 5985
 HTTPS = 5986
 TrustedHosts = *
 Service
 RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
 MaxConcurrentOperations = 4294967295
 MaxConcurrentOperationsPerUser = 1500
 EnumerationTimeoutms = 240000
 MaxConnections = 300
 MaxPacketRetrievalTimeSeconds = 120
 AllowUnencrypted = false
 Auth
 Basic = true [Source="GPO"]
 Kerberos = true
 Negotiate = true
 Certificate = false
 CredSSP = true [Source="GPO"]
 CbtHardeningLevel = Relaxed
 DefaultPorts
 HTTP = 5985
 HTTPS = 5986
 IPv4Filter [Source="GPO"]
 IPv6Filter [Source="GPO"]
 EnableCompatibilityHttpListener = false
 EnableCompatibilityHttpsListener = false
 CertificateThumbprint
 AllowRemoteAccess = true [Source="GPO"]
 Winrs
 AllowRemoteShellAccess = true [Source="GPO"]
 IdleTimeout = 7200000
 MaxConcurrentUsers = 10
 MaxShellRunTime = 2147483647
 MaxProcessesPerShell = 25
 MaxMemoryPerShellMB = 1000
 MaxShellsPerUser = 30

PS C:UsersDaniel> winrm e winrm/config/listener
Listener [Source="GPO"]
 Address = *
 Transport = HTTP
 Port = 5985
 Hostname
 Enabled = true
 URLPrefix = wsman
 CertificateThumbprint
 ListeningOn = null

PS C:UsersDaniel> get-service WinRM

Status Name DisplayName
------ ---- -----------
Running WinRM Windows Remote Management (WS-Manag...

PS C:UsersDaniel> winrm get wmicimv2/Win32_Service?Name=WinRM
Win32_Service
 AcceptPause = false
 AcceptStop = true
 Caption = Windows Remote Management (WS-Management)
 CheckPoint = 0
 CreationClassName = Win32_Service
 Description = Windows Remote Management (WinRM) service implements the WS-Management protocol for remote management.
 WS-Management is a standard web services protocol used for remote software and hardware management. The WinRM service l
istens on the network for WS-Management requests and processes them. The WinRM Service needs to be configured with a lis
tener using winrm.cmd command line tool or through Group Policy in order for it to listen over the network. The WinRM se
rvice provides access to WMI data and enables event collection. Event collection and subscription to events require that
 the service is running. WinRM messages use HTTP and HTTPS as transports. The WinRM service does not depend on IIS but i
s preconfigured to share a port with IIS on the same machine. The WinRM service reserves the /wsman URL prefix. To prev
ent conflicts with IIS, administrators should ensure that any websites hosted on IIS do not use the /wsman URL prefix.
 DesktopInteract = false
 DisplayName = Windows Remote Management (WS-Management)
 ErrorControl = Normal
 ExitCode = 0
 InstallDate = null
 Name = WinRM
 PathName = C:WindowsSystem32svchost.exe -k NetworkService
 ProcessId = 936
 ServiceSpecificExitCode = 0
 ServiceType = Share Process
 Started = true
 StartMode = Auto
 StartName = NT AUTHORITYNetworkService
 State = Running
 Status = OK
 SystemCreationClassName = Win32_ComputerSystem
 SystemName = FNZAS2
 TagId = 0
 WaitHint = 0

PS C:UsersDaniel> winrm id
IdentifyResponse
 ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
 ProductVendor = Microsoft Corporation
 ProductVersion = OS: 6.1.7601 SP: 1.0 Stack: 3.0
 SecurityProfiles
 SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/basic, http://schemas.dmtf.org/
wbem/wsman/1/wsman/secprofile/http/spnego-kerberos

PS C:UsersDaniel> Enter-PSSession -ComputerName localhost
Enter-PSSession : Connecting to remote server localhost failed with the following error message : The client cannot
connect to the destination specified in the request. Verify that the service on the destination is running and is
accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most
commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to
analyze and configure the WinRM service: "winrm quickconfig". For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : InvalidArgument: (localhost:String) [Enter-PSSession], PSRemotingTransportException
 + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

PS C:UsersDaniel> 

I've also tried restarting the WinRM service, as well as restarting the whole server. Still getting the same errors.

It's easy to miss. To my (inexpert) eye, the second error message in the Event Viewer seems like it could be meaningful:

WSMan operation CreateShell failed, error code 2150858770

I've found this error code on another question on Server Fault, but there's no answers.

I've managed to find a similar issue here. I have tried the MaxFieldLength and MaxRequestBytes suggested by Arthur_Li, but this didn't resolve the problem for me.

That error code looks like it might be in decimal, so I've tried converting that to hex and searching for the hex code instead, and didn't find anything much that the base error code didn't already turn up.

I'm completely stumped at this point. I've set up PowerShell Remoting on other servers before without issues like this.

Once piece of advice I have received is: "Stop using 2008 R2. Upgrade to something more recent." We were planning to do that sometime in the next six months anyway. But it's not something we're going to be able to act on until, probably, the end of September at the earliest.

I can work around this by logging into the machines, uploading the deployment scripts and package myself, and running them manually. But that kind of defeats the point of having an automated deployment process in the first place.

Any assistance would be greatly appreciated.

UPDATE #1

Attempting to delete and then restore a default listener for WinRM.

PS C:UsersDaniel> winrm delete winrm/config/listener?address=*+transport=HTTP
WSManFault
 Message
 ProviderFault
 WSManFault
 Message = WS-Management does not allow changes to a listener created automatically by the group policy.
The policy "Allow Auto Configuration of listeners on WinRm service" would need to be set to "Not Configured" in order to
 create a new listener for same Address and Transport or to modify an already existing listener.

Error number: -2144108406 0x8033808A
Cannot change GPO controlled setting.

I went in here to gpedit.msc. Turns out that the "Allow Auto Configuration of listeners on WinRm service" has been unhelpfully renamed to "Allow remote server management through WinRM". I set this to "Not configured" and tried again.

PS C:UsersDaniel> winrm delete winrm/config/listener?address=*+transport=HTTP
PS C:UsersDaniel> winrm create winrm/config/Listener?Address=*+Transport=HTTP
ResourceCreated
 Address = http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous
 ReferenceParameters
 ResourceURI = http://schemas.microsoft.com/wbem/wsman/1/config/listener
 SelectorSet
 Selector: Address = *, Transport = HTTP

PS C:UsersDaniel> winrm e winrm/config/listener
Listener
 Address = *
 Transport = HTTP
 Port = 5985
 Hostname
 Enabled = true
 URLPrefix = wsman
 CertificateThumbprint
 ListeningOn = 10.10.90.6, 127.0.0.1, ::1, fe80::100:7f:fffe%11, fe80::5efe:10.10.90.6%13

PS C:UsersDaniel> Enter-PSSession -ComputerName localhost
Enter-PSSession : Connecting to remote server localhost failed with the following error message : WinRM cannot process
the request. The following error with errorcode 0x80090322 occurred while using Negotiate authentication: An unknown
security error occurred.
 Possible causes are:
 -The user name or password specified are invalid.
 -Kerberos is used when no authentication method and no user name are specified.
 -Kerberos accepts domain user names, but not local user names.
 -The Service Principal Name (SPN) for the remote computer name and port does not exist.
 -The client and remote computers are in different domains and there is no trust between the two domains.
 After checking for the above issues, try the following:
 -Check the Event Viewer for events related to authentication.
 -Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or
use HTTPS transport.
 Note that computers in the TrustedHosts list might not be authenticated.
 -For more information about WinRM configuration, run the following command: winrm help config. For more
information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : InvalidArgument: (localhost:String) [Enter-PSSession], PSRemotingTransportException
 + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

PS C:UsersDaniel>

On that topic, here's the current configuration of my GPO for WinRM

Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Client

• Allow Basic authentication: Enabled


• Allow CredSSP authentication: Enabled


• Allow unencrypted traffic: Enabled


• Disallow Digest authentication: Not configured


• Disallow Kerberos authentication: Not configured


• Disallow Negotiate authentication: Not configured


• Trusted Hosts: Not configured

Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Server

• Allow remote server management through WinRM: Not configured (Note: This was set to 'Enabled' in the examples prior to this update)


• Allow Basic authentication: Enabled


• Allow CredSSP authentication: Enabled


• Allow unencrypted traffic: Enabled


• Specify channel binding token hardening level: Not configured


• Disallow WinRM from storing RunAs credentials: Not configured


• Disallow Kerberos authentication: Not configured


• Disallow Negotiate authentication: Not configured


• Turn On Compatibility HTTP Listener: Not configured


• Turn On Compatibility HTTPS Listener: Not configured

The error message has changed. When I jump into Event Viewer, I now get the following two errors. Note that they've both changed. The first changed dramatically, the second less dramatically.

General:
 Omitted for brevity. Same as per the "authFailureMessage" in the details below.
Detail:
 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft-Windows-WinRM" Guid="A7975C8F-AC13-49F1-87DA-5A984A4AB417" /> 
 <EventID>161</EventID> 
 <Version>0</Version> 
 <Level>2</Level> 
 <Task>7</Task> 
 <Opcode>0</Opcode> 
 <Keywords>0x400000000000000a</Keywords> 
 <TimeCreated SystemTime="2016-08-18T00:37:41.784323600Z" /> 
 <EventRecordID>61452</EventRecordID> 
 <Correlation ActivityID="0190DC40-F800-0000-79D1-5DB0DAF8D101" /> 
 <Execution ProcessID="7888" ThreadID="8116" /> 
 <Channel>Microsoft-Windows-WinRM/Operational</Channel> 
 <Computer>FNZAS2.flow.net.nz</Computer> 
 <Security UserID="S-1-5-21-2875926586-1071052228-4104636349-1151" /> 
 </System>
 <EventData>
 <Data Name="authFailureMessage">WinRM cannot process the request. The following error with errorcode 0x80090322 occurred while using Negotiate authentication: An unknown security error occurred. Possible causes are: -The user name or password specified are invalid. -Kerberos is used when no authentication method and no user name are specified. -Kerberos accepts domain user names, but not local user names. -The Service Principal Name (SPN) for the remote computer name and port does not exist. -The client and remote computers are in different domains and there is no trust between the two domains. After checking for the above issues, try the following: -Check the Event Viewer for events related to authentication. -Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport. Note that computers in the TrustedHosts list might not be authenticated. -For more information about WinRM configuration, run the following command: winrm help config.</Data> 
 </EventData>
 </Event>

General:
 WSMan operation CreateShell failed, error code 2150858909
Details:
 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft-Windows-WinRM" Guid="A7975C8F-AC13-49F1-87DA-5A984A4AB417" /> 
 <EventID>142</EventID> 
 <Version>0</Version> 
 <Level>2</Level> 
 <Task>10</Task> 
 <Opcode>2</Opcode> 
 <Keywords>0x4000000000000002</Keywords> 
 <TimeCreated SystemTime="2016-08-18T00:37:41.784323600Z" /> 
 <EventRecordID>61454</EventRecordID> 
 <Correlation ActivityID="0190DC40-F800-0000-7CD1-5DB0DAF8D101" /> 
 <Execution ProcessID="7888" ThreadID="8116" /> 
 <Channel>Microsoft-Windows-WinRM/Operational</Channel> 
 <Computer>FNZAS2.flow.net.nz</Computer> 
 <Security UserID="S-1-5-21-2875926586-1071052228-4104636349-1151" /> 
 </System>
 <EventData>
 <Data Name="operationName">CreateShell</Data> 
 <Data Name="errorCode">2150858909</Data> 
 </EventData>
 </Event>

UPDATE #2

Attempting to clear out WinRM settings and then restore the defaults.

Powershell output at: pastebin.com/E5wgXE1q

Underlying Windows Event logs are the same as those generated in Update #1.

UPDATE #3

Using Mer's winrm/config output as a guide, I've gone through my local machine group policy objects and reset everything back to be 'Not Configured' This gives me a winrm/config output that matches Mer's.

I still wasn't able to get through, however. Tried the same clear-out/reset steps followed in Update #2 just to be safe, and that didn't work either.

Powershell output at pastebin.com/EuzyDR6d

Output in Event Log is the same as for Update 2.

Will try a server restart to see if that makes a difference.

UPDATE #4

Server restart didn't fix. Still getting the same error message as per Update #2.

UPDATE #5

Okay. This is nuts.

All of the problems above are happing on a server we'll call AS2.

I just jumped over to the AS1 server, and set up remote powershell. Just to make sure I'm not going insane.

• AS1: Enter-PSSession localhost > Successful


• AS1: Enter-PSSession AS2 > Successful


• AS2: Enter-PSSession localhost > Failure


• AS2: Enter-PSSession AS1 > Successful

Earlier, I was having problems getting from AS2 into any server. But somewhere along the line I fixed that. Now it's just the localhost on AS2 that's the problem.

This feels completely nuts. Why can't AS2 remote to itself, when it is clearly happy to accept incoming connections, and it can make outgoing connections fine?

UPDATE #6

Okay, new information: CredSSP authentication does work. It seems to be specifically something to do with Negotiate authentication on this server that is broken.

I may be able to use this as the basis of a workaround for what I'm trying to do. That still wouldn't explain why Negotiate seems to be broken on this server though.

Can you remove the existing listener with:

winrm delete winrm/config/listener?address=*+transport=HTTP

And add a new one with:

winrm create winrm/config/Listener?Address=*+Transport=HTTP

And, check again with:

winrm e winrm/config/listener

ListeningOn should be listing your IP addresses, not null.