Otdfbt

I am responsible for managing both our production server (mail, web, database are all on one server) and our test server. Both are built on Debian. However as I am very new to system administration, I have only been installing updates as I come across things that have to be updated so that I can have newer features and get bug fixes. Its a pretty ad hoc process right now, and I'd like to make it less so.

So I am wondering how people who know what they're doing handle this? How often do you perform upgrades on your servers? Is the upgrade process different between test and production? Do you always upgrade any test servers first? And do you do a full update of all software, or do you just install selected updates?

I run apt-get update -qq; apt-get upgrade -duyq daily. This will check for updates, but not do them automatically.

Then I can run the upgrades manually while I am watching, and can correct anything that might go wrong.

Besides the security concerns of maintaining a patched system, I find that if I leave it too long between patches, I end up with a whole bunch of packages that want to be upgraded, and that scares me a-lot more than just upgrading one or two every week or so. Therefore I tend to run my upgrades weekly, or if they are high priority, daily. This has the added advantage of knowing which package broke your system (ie. if you're only upgrading a couple at a time)

I always upgrade less critical systems first. I also have a "rollback plan" in place in case I can't fix the system. (since most of our servers are virtual, this rollback plan usually consists of taking a snapshot before the upgrade that I can revert to if necessary)

That being said, I think an upgrade has broken something only once or twice in the past 4 years, and that was on a highly customized system - so you don't have to be TOO paranoid :)

On top of previous answers - a couple more specifically Debian things: you should Subscribe to debian-security-announce and debian-announce and / or check out the Debian Security page.

Assuming you are running the stable release of Debian, most patches will be security or bug related which should mean that there won't be too many major changes between versions of any given package. According to the debian patch policy patches should also have been in testing for some time before being moved to the stable branch by the maintainer. Obviously This won't stop breakages when patching but should prevent them in most cases.

It would be prudent to ensure that your testing sever is kept upto date and any packages that have bugs affecting you and your servers should be kept up to date. All packages that have security advisories against them should be updated as soon as you know the patch is stable.

Debian is usually a very stable OS and not one you should be overly concerned with breakages on however always read what is going to be updated before it get updated and keep an eye out for anything that seems strange. I use VCS on my /etc/ dir as well to ensure that any config file changes can be seen with a 'git diff' command.

I do a dry run (first) to see what is going to be updated. Sometimes, libraries (lets call it libfoo for this example) change their API, which breaks programs that we wrote / installed ourselves. If some critical library is updated, I grab the source and try rebuilding our stuff against it prior to updating.

I also check to see that we're not jumping to an intermediate version of some public service, ie apache, etc. I'd rather stay a year behind and not encounter random breakage, unless the update is critical.

If you are a system administrator, you should be pulling RSS feeds from sites like Secunia, which should let you know way ahead of time if your distro is going to be pushing some patches.

Do not ever, ever just blindly upgrade / update. Unfortunately, the task of knowing what is broken falls on you, not your distro package manager, especially if your systems support programmers.

Where I work we have a pretty extensive process that involves using software called PatchLink to notify us of the most important security related updates, and we apply them after testing, on a package by package basis. We have thousands of servers though.

If you only have two servers the process should be much simpler. Although I do not think doing an "apt-get update/upgrade" is your best bet.

I would monitor patches for the software you are running, and make decisions based on the fixes in those releases on when to upgrade.

Since you have a test server, obviously, always test the update before applying them.

Manual updates are best as mentioned here in the sense that you can see what's happening. However, for very large numbers of servers that might become impractical. Dry run is a standard practice, in fact, most package managers will ask you before proceeding.

Updating regularly tends to be best though it can be a bit of a balancing act. Frequent updates means less in one go and less to go wrong at once. If things do go wrong there are fewer candidates to inspect. Packages are also slightly better at updating in smaller steps, as generally when the programmer updates they're looking at going from the last version to the next, whether they'll give any attention beyond the last version can vary, though this tends to matter mostly for software that's rapidly evolving.

Not all updates are non-disruptive. You'll want to watch out for this. Some will restart services leading to down time.

In an ideal setup you might have the following:

• A means of seemlesly switching servers (A/B or tick tock). This means you update one while it's on the bench, then simply swap the traffic from the current one to the new one. This may be more complicated for services such as databases.


• The ability to test updates. You should have test servers that are practically clones of production (but without connecting to any production services). These would allow you to test updates first.


• A good backup strategy, incremental is ideal. You never know. It's always better to be safe than sorry.


• Be aware of which times have the most activity and what level of downtime is tolerable.


• Know how to rollback an update or a specific package.


• Have your own package mirrors so updates are consistent and predictable across servers. This is the first step towards a decent unattended system that you can trust. It means you can update the mirror, run update on one or more test machine then if that's good let it go out automatically. I had an excellent time with aptly managing around 800 EPOS machines.


• A good level of consistency so that you can know that if something will work here, it'll work there.

Some of these can be overkill to varying degrees for small setups but should be kept in mind.

Generally speaking, updates are usually relatively painless for server distros. This is because they nearly always only stick to bug fixes and security updates. You may however have problems if people have done odd things to the system or you add additional package sources.

Although it's moderately rare, they do occasionally make mistakes and break compatibility between minor package versions.

I like cron-apt to automate this process, but as @dinomite pointed out on another question regarding updates, configuring it specifically to automate security-related updates is a very smart idea -- you can then manually update what you need. I had been using cron-apt for all updates but actually changed this based on his answer. If you like it, you should probably vote his answer up rather than this one.

On debian i install cron-apt and edit its configuration file to mail me if ther are any changes. this way i get notified if there are updates for my systems and do the updates by hand

Along the same lines as cron-apt you should take a look at the unattended-upgrades package http://packages.debian.org/lenny/unattended-upgrades.

Its very easy to configure and will enable you to have security updates downloaded and applied automatically but leave other updates for manual upgrading (or at your discretion upgrade everything!).

The Official Ubuntu Server Guide, has a reasonably detailed section covering the use of the unattended-upgrades package https://help.ubuntu.com/9.04/serverguide/C/automatic-updates.html

Note: depending on your level of caution/paranoia you can do a rolling upgrade on a group of test servers first, then if there are not issues, allow your production boxes to update, although I personally have not come across any trouble with security updates wrecking havoc so far (knock on wood)...

There is also a config option to mail you the results of each security update as well, once its applied. Also, if there were any dialog or interactive prompts that were presented during the update, ones that will need manual tweaking by a sysadmin, it will mention these as well.