Recently I was in the midst of setting up a simple two node Cisco ISE 2.3.0.298 cluster and got to the stage when I registered the secondary node through the GUI of the primary. At that point, things wouldn’t work.
After spinning up new ISE VMs and getting certificates sorted out, you should create your ISE cluster by registering secondary, tertiary, etc nodes via the Primary node (see below).
When registering the secondary ISE server, I eventually got a Failed message after almost an hour of waiting. Because there’s so little in the database to replicate, this should have taken only a few minutes. The logs weren’t very helpful, either.
Just prior to this, I changed ISEV02 from Secondary to Standalone and then back again to Standalone. I didn’t suspect this would hurt anything since the secondary server wasn’t in a cluster yet.
After a few minutes of digging, I discovered that the application service, which handles the GUI among other things, would not start anymore. I reset all services, rebooted the server, and looked at some debugs. There was nothing there that helped, and restarting everything got the service only to initialize but not fully start.
The command show application status ise shows you all the processes related to ISE and is very helpful when configuring things that restart services. The third process from the top, Application Server, takes a little longer to start than the others, so I like to monitor it when working on initial ISE configurations.
To reset the application service, use application stop ise followed by application start ise. To reset the entire config, use application reset-config ise.
After making my changes to ISEV02, the process was stuck in initializing. After a couple hours it went to not running. I tried the above commands, but nothing worked. Because this was a brand new installation, I had very little time into this process. Rather than troubleshoot it further, I chose to re-deploy the virtual machine and start over.
This took literally minutes, so it was probably the best way to get moving on this project and make better use of my time. I hate doing that, but really it made no sense to troubleshoot a virtually blank ISE server when I could get a fresh one up and running so quickly.
I already knew that after getting the secondary (and tertiary, etc) ISE servers up and running, all the configuration should happen from the Primary – including registering other servers in the cluster. In this case, because the secondary ISE server was not in a cluster yet, I don’t know why changing from standalone and back again should have broken anything.
In conclusion, after getting the secondary ISE servers online, I won’t be touching anything on them directly anymore even if I think it won’t hurt anything.
{networkphil} 
Isn’t it great how Cisco software makes you afraid to maintain it?
LikeLike