10 Mistakes Systems Administrators Shouldn’t Be Making
In the 21st century, you’d assume that hacking into an official network or infecting it with a virus would be quite hard. So, if someone was to tell you that some of the biggest network security breaches of our time happened over the last 5 years, would you believe them?
From huge data leaks where millions of private accounts were compromised to sensitive details of a multi-million-dollar company’s top-level management being published on the Internet, we’ve seen all kinds of ridiculous data breaches between over the last decade. And every major breach was the result of poor network security practices being employed by incompetent system administrators.
Some of these mistakes system administrators make were so bad that it would be unfair to not give them a special mention. Take Ashley Madison for example, the world’s leading ‘married dating service for discreet encounters’.
Someone was careless enough to leave authentication tokens and credentials in the website’s source code! From a hacker’s perspective, this was like breaking a tiny bit of cheese off a large block in a mousetrap.
Hackers cracked more than 11 million passwords in 10 days and published private details online for everyone to see.
The point is, without the right systems administrator, every single packet of data that belongs to your organization is susceptible to being poached and manipulated.
So, before you find someone to fill in positions for systems administrator jobs, take a look at these dumb mistakes that systems administrators should be hyper-aware of.
Blunder No. 1: Using sudo for everything
When you log in as root, you have full control over the box. This can be extremely dangerous because if your credentials get stolen, an attacker can do whatever he or she wants.
In Windows parlance, there’s no need to log in as Administrator if there are no administrator-level tasks to perform. Instead of directly logging into systems as root, log in through your personal account and use sudo for specific commands as needed. It’s easy to backslide if you’re not careful. A script fails because one of the commands needed sudo — and now everything must be restarted. If you fail to keep track of which commands require elevated privileges and which don’t, you may go back to running everything as sudo.
Blunder 2: Running scripts of unknown origin
Installing third-party Linux applications is another area where sudo can be abused. All you have to do is copy and paste the command — which is already set up to use sudo — directly into the terminal to kick off the install script. Every single command in that script will be executed with elevated privileges.
Here’s an example, copied right off the Web (with the URL hidden):
sudo -v && wget -nv -O- https://xxx/xxx/linux-installer.py | sudo python -c “import sys; main=lambda:sys.stderr.write(‘Download failed\n’); exec(sys.stdin.read()); main()”
This gives sudo privileges to an item hosted elsewhere on the Web, as well as running Python locally. Not recommended! Windows admins face similar potential catastrophes running downloaded PowerShell scripts.
Even if you trust the source, never assume a script downloaded from the Internet is safe. Always vet the contents of the script first and verify that executing the commands will not result in nefarious actions.
Blunder No. 3: Running privileged services as root
Applications should never run as root. Create unique service accounts with very specific privileges for each application and service running on the machine. Service accounts typically lack home directories and are restricted in what they can do on the file system if someone tries to log in using the account. If an attacker compromises a service account, he or she still must get a local exploit working to get more privileges to execute code.
Each application should use a customized account to access the database instead of root or the administrator’s personal account. Web applications should be owned by the appropriate group and user. When assigning domain privileges to Windows applications, don’t give the application administrator-level access.
Major Linux distributions handle service accounts by default, but if the administrator manually configures third-party packages, it’s easy to make a mistake. Remember to switch permissions after all the installation and configuration is complete to make sure root or the administrator’s personal account is no longer the owner of the application.
Blunder 4: Reusing passwords
Go ahead, roll your eyes. We’ve all heard about the evils of reusing passwords across sites, systems, and applications. But the fact remains that it’s a big problem, and sys admins are not immune. Recently, Mozilla said an unknown attacker broke into a privileged user’s account for its Bugzilla bug tracking database and stole information about 53 critical vulnerabilities. It turned out the “privileged user” had reused the Bugzilla password on another website, and the password had been exposed in that site’s breach.
Many times, servers are set up with weak administrator passwords or with the same password as other machines on the network. Brute-force attacks using common passwords and dictionary words work because enough people still make this elementary mistake. When multiple machines have the same password, the problem is compounded. Instead of setting up the same root password on all machines, sys admins should opt to use a key file. Each server should have a public key file and the sys admin’s workstation would have the private key associated with the public key. This way, the sys admin can access all the machines that have been deployed on the network, but an attacker moving laterally through the network will not be able to log in without a valid key. And there is no password to intercept.
Blunder 5: Sharing admin accounts
Administrator accounts — such as access to the database and administrator portals — are often shared around the network. Instead of setting up the environment so that administrators request elevated privileges when needed, these admin accounts are shared willy-nilly. That’s asking for trouble.
Ideally, there should be separate accounts: one for root and one for each administrator. The administrator accounts shouldn’t start off with the highest levels of access — the administrator can ask for special access rights when working on specialized tasks. The Intermedia report found that 32 percent of IT professionals have given out their login and password credentials to other employees.
It’s bad enough not knowing exactly who is using the administrator accounts, but even worse, the passwords are rarely reset when an administrator leaves the company. Because the passwords aren’t regularly cycled, ex-colleagues can come right back in and cause damage with impunity. The Intermedia survey found that one in five IT professionals said they would access company information after they leave their current job. Password change policies aren’t for end-users only. Periodically change passwords, particularly administrator and service accounts. And whenever an administrator leaves, reset the passwords.
Blunder 6: Leaving troubleshooting tasks in place
When troubleshooting, you perform various tricks and experiments to find and fix the problem. As you make these attempts, you tend to bypass the usual processes. The problem comes when you fix the problem and move on to the next fire. Admins in a hurry may forget and leave things in disarray — and open to potential abuse.
You may have opened ports in the firewall, for example, as you tried to figure out why an application wasn’t responding. Once the fix is in place, you need to go back and close those ports before they can be used by attackers. By the same token, if you turned off SELinux because it was interfering with troubleshooting, remember to turn it back on again after you are done. When troubleshooting, keep track of what you do as you go, so that afterward you can restore configurations to their original settings — except for the changes you really needed to make.
Blunder 7: Failing to keep track of log files
Log files are handy, especially when troubleshooting, because they let you see what’s happening at the most granular possible level. When you don’t need those files anymore, turn off the process generating them. The last thing you want to do is leave debugging on and generate log files containing information that may be useful to attackers.
As a best practice, always keep track of what logs are created and know what kind of information is in them.
Blunder 8: Storing passwords in plain-text files
When there are so many passwords to track, it’s tempting to write them down in a text file. That’s a gift for attackers snooping around as they get access to various systems. It sounds obvious, but everyone knows of at least one instance where someone saved all important passwords in a text file.
If the passwords must be saved in plain text in a file — such as database credentials for an application — set up file permissions to restrict who can view the contents of the file. Also, make sure the database account is a service account stripped to the bare privileges.
Blunder 9: Leaving unused accounts lying around
Old, unused accounts are liabilities. Perhaps software was installed for evaluation, then removed — and the accounts that were added as part of the installation are still on the system. Don’t leave them there. Attackers can exploit forgotten accounts like these, especially if they retain their default passwords.
For accounts that need to remain on the system but will not being used going forward, disable the account by editing the password file and replacing the account password with a string of characters. Obviously, when employees leave your organization, a process should be in place to deprovision their accounts immediately.
Blunder 10: Being lax about patches
The golden rule: Install security updates as soon as they are available (backing up the affected systems first, of course). Too many servers are compromised not because of a zero-day exploit, but because a year-old patch was never installed.
Even if it’s a critical server, a little downtime as part of a scheduled maintenance window is far better than losing hours and days because attackers successfully compromised the box. Promptly test patches as they are released and create a schedule for rolling out updates.
Unfortunately, you may be thwarted in your efforts to patch right away — usually because the patch will break a legacy app. In that case, don’t simply shrug and say “too bad.” Highlight the problem to the appropriate stakeholders. Escalate the issue. Maybe there are ways to quarantine the servers to minimize risk or to adopt new technologies and reduce dependencies on legacy products.
In real life, patching can be a political quagmire. If a manager who outranks you prevents a system from being patched, make sure everyone understands the risks of failing to do so.