Restart all Jenkins NodesConsiderations for building SysV or Upstart compatible Bash scriptsUse Master/Slave Jenkins Server vs maintaining 3 separate servers?Jenkins CI Master-Slave setup with docker for slave serverHow to use automatic Restart with Jenkins on SystemdJenkins Kubernetes plugin: pending All nodes of label are offlineHow do I set up HAProxy Behind Cloudfront for JenkinsEnsure a Jenkins job runs once on or after the specified cron time - even if Jenkins were not running at the specified cron timeAutomate jenkins slave node cleanupJenkins build happening on master instead of slaveWhy are psql commands in my script suddenly being killed by jenkins / hudson?
Resize before convert or convert before resize?
Filter a data-frame and add a new column according to the given condition
How was this character able to keep fighting effectively in S8E5 of Game of Thrones?
Help understanding this line - usage of くれる
Substring join or additional table, which is faster?
Is there any way to adjust the damage type of the Eldritch Blast cantrip so that it does fire damage?
As programmers say: Strive to be lazy
Why was my Canon Speedlite 600EX triggering other flashes?
Will the volt, ampere, ohm or other electrical units change on May 20th, 2019?
Meaning of "work with shame"
Is this a group? If so, what group is it?
Given 0s on Assignments with suspected and dismissed cheating?
Should generated documentation be stored in a Git repository?
Mark command as obsolete
Do we have C++20 ranges library in GCC 9?
Find the unknown area, x
Adding labels and comments to a matrix
Can I say: "When was your train leaving?" if the train leaves in the future?
Do not cross the line!
Was the dragon prowess intentionally downplayed in S08E04?
Were any toxic metals used in the International Space Station?
Polynomial division: Is this trick obvious?
Why does SSL Labs now consider CBC suites weak?
How to continually let my readers know what time it is in my story, in an organic way?
Restart all Jenkins Nodes
Considerations for building SysV or Upstart compatible Bash scriptsUse Master/Slave Jenkins Server vs maintaining 3 separate servers?Jenkins CI Master-Slave setup with docker for slave serverHow to use automatic Restart with Jenkins on SystemdJenkins Kubernetes plugin: pending All nodes of label are offlineHow do I set up HAProxy Behind Cloudfront for JenkinsEnsure a Jenkins job runs once on or after the specified cron time - even if Jenkins were not running at the specified cron timeAutomate jenkins slave node cleanupJenkins build happening on master instead of slaveWhy are psql commands in my script suddenly being killed by jenkins / hudson?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I manage a Jenkins server (running 2.89.4) with about 40 nodes or so attached, each running either Centos7, OSX, or Windows (7, 8, 10). Recently, we suffered some power outages and many slaves lost their connection to the master. This required a lot of manual remoting and ssh'ing to reconnect them all. Since, I have made sure each slave process is running a server, however I'd like an additional safety net of being able to automatically restart every vm and/or box these slaves are running on. The problem is, since there are several different types of operating systems, there isn't a simple way to do it (that I know of yet).
I've tried using a Groovy script, but that can mostly just be used for restarting the slave service, not the machine itself. This was my reference:
https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves
I've also looked into an Ansible approach, but many machines are unreachable, and Ansible requires different types of scripts for Windows vs. Unix.
The best approach I've seen so far is here:
https://stackoverflow.com/questions/29165080/restart-jenkins-slave-from-master
But it doesn't explain how to run this on every single slave, it just looks like it executes on one slave. Has anyone else done anything like this before? I'd love to create a single job, or single script than can be executed and it restarts every single node (using a command like shutdown -r -f) connected to Jenkins regardless of platform.
bash ansible jenkins
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
add a comment |
I manage a Jenkins server (running 2.89.4) with about 40 nodes or so attached, each running either Centos7, OSX, or Windows (7, 8, 10). Recently, we suffered some power outages and many slaves lost their connection to the master. This required a lot of manual remoting and ssh'ing to reconnect them all. Since, I have made sure each slave process is running a server, however I'd like an additional safety net of being able to automatically restart every vm and/or box these slaves are running on. The problem is, since there are several different types of operating systems, there isn't a simple way to do it (that I know of yet).
I've tried using a Groovy script, but that can mostly just be used for restarting the slave service, not the machine itself. This was my reference:
https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves
I've also looked into an Ansible approach, but many machines are unreachable, and Ansible requires different types of scripts for Windows vs. Unix.
The best approach I've seen so far is here:
https://stackoverflow.com/questions/29165080/restart-jenkins-slave-from-master
But it doesn't explain how to run this on every single slave, it just looks like it executes on one slave. Has anyone else done anything like this before? I'd love to create a single job, or single script than can be executed and it restarts every single node (using a command like shutdown -r -f) connected to Jenkins regardless of platform.
bash ansible jenkins
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
add a comment |
I manage a Jenkins server (running 2.89.4) with about 40 nodes or so attached, each running either Centos7, OSX, or Windows (7, 8, 10). Recently, we suffered some power outages and many slaves lost their connection to the master. This required a lot of manual remoting and ssh'ing to reconnect them all. Since, I have made sure each slave process is running a server, however I'd like an additional safety net of being able to automatically restart every vm and/or box these slaves are running on. The problem is, since there are several different types of operating systems, there isn't a simple way to do it (that I know of yet).
I've tried using a Groovy script, but that can mostly just be used for restarting the slave service, not the machine itself. This was my reference:
https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves
I've also looked into an Ansible approach, but many machines are unreachable, and Ansible requires different types of scripts for Windows vs. Unix.
The best approach I've seen so far is here:
https://stackoverflow.com/questions/29165080/restart-jenkins-slave-from-master
But it doesn't explain how to run this on every single slave, it just looks like it executes on one slave. Has anyone else done anything like this before? I'd love to create a single job, or single script than can be executed and it restarts every single node (using a command like shutdown -r -f) connected to Jenkins regardless of platform.
bash ansible jenkins
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
I manage a Jenkins server (running 2.89.4) with about 40 nodes or so attached, each running either Centos7, OSX, or Windows (7, 8, 10). Recently, we suffered some power outages and many slaves lost their connection to the master. This required a lot of manual remoting and ssh'ing to reconnect them all. Since, I have made sure each slave process is running a server, however I'd like an additional safety net of being able to automatically restart every vm and/or box these slaves are running on. The problem is, since there are several different types of operating systems, there isn't a simple way to do it (that I know of yet).
I've tried using a Groovy script, but that can mostly just be used for restarting the slave service, not the machine itself. This was my reference:
https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves
I've also looked into an Ansible approach, but many machines are unreachable, and Ansible requires different types of scripts for Windows vs. Unix.
The best approach I've seen so far is here:
https://stackoverflow.com/questions/29165080/restart-jenkins-slave-from-master
But it doesn't explain how to run this on every single slave, it just looks like it executes on one slave. Has anyone else done anything like this before? I'd love to create a single job, or single script than can be executed and it restarts every single node (using a command like shutdown -r -f) connected to Jenkins regardless of platform.
bash ansible jenkins
bash ansible jenkins
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
asked Feb 23 '18 at 16:45
user3270760user3270760
11327
11327
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:
Batch:
echo "Restarting Windows Machine...."
hostname
shutdown -r -f
Shell:
echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now
I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f898614%2frestart-all-jenkins-nodes%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:
Batch:
echo "Restarting Windows Machine...."
hostname
shutdown -r -f
Shell:
echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now
I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
add a comment |
I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:
Batch:
echo "Restarting Windows Machine...."
hostname
shutdown -r -f
Shell:
echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now
I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
add a comment |
I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:
Batch:
echo "Restarting Windows Machine...."
hostname
shutdown -r -f
Shell:
echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now
I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:
Batch:
echo "Restarting Windows Machine...."
hostname
shutdown -r -f
Shell:
echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now
I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
answered Feb 23 '18 at 17:50
user3270760user3270760
11327
11327
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
add a comment |
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
The only thing I haven't solved yet is how to skip a node if it is offline for some reason.
– user3270760
Feb 23 '18 at 17:51
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f898614%2frestart-all-jenkins-nodes%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
