(   )
                          (    )
                           (    )
                          (    )
                            )  )
                           (  (                  /\
                            (_)                 /  \  /\
                    ________[_]________      /\/    \/  \
           /\      /\        ______    \    /   /\/\  /\/\
          /  \    //_\       \    /\    \  /\/\/    \/    \
   /\    / /\/\  //___\       \__/  \    \/
  /  \  /\/    \//_____\       \ |[]|     \
 /\/\/\/       //_______\       \|__|      \
/      \      /XXXXXXXXXX\                  \
        \    /_I_II  I__I_\__________________\
               I_I|  I__I_____[]_|_[]_____I
               I_II  I__I_____[]_|_[]_____I
               I II__I  I     XXXXXXX     I
            ~~~~~"   "~~~~~~~~~~~~~~~~~~~~~~~~

Wed, 12 Mar 2014

Adventures of a Sysadmin, Fun With cPanel Edition

Oh, how I loathe cPanel. It completely takes over your linux servers. Once installed, there is no uninstalling it, it has hooks into every part of the OS. Once infected by the cPanel virus, you must make all changes to the server configuration through cPanel, or cPanel's highly intuitive collection of command line utilities. Everything.


I have a client for whom I assist in maintaining a cPanel-infected server. He has about 25 hosted accounts, and since cPanel has made it oh-so-easy for him to perform common administrative tasks himself, he felt the need to buy a shell domain just to use as a DNS zone for a handful of other domains - all on the same server (right, it doesn't make sense to me, either). In effect he's only using this dummy zone's NS records to get traffic to the server itself.


So anyway, the other day he asks me to transfer one of his websites to a new server. No problem, cPanel provides a web-gui just for that! It first backs up the entire account, then, instead of just transferring it, it helpfully compresses it all and transfers the tarball to the new server. Of course, this is his biggest account, with 4GB of website. And holy fuck, this thing is the slowest compression utility I have ever seen, somehow it is gzipping at like 1MB every 20 seconds. It must be using some native perl interface to zlib. I'll let it run, why not? I've got time to kill. Two hours later...sigh, kill the web page, which is spinning aimlessly, kill the backup process. Delete the 270MB partial backup tarball (270MB in 2 hours...). Delve into the cPanel docs for that obscure command line switch not exposed to the web gui that will fix things.

Aha! --skiphomedir. Re-run backup from command line, which now takes 10 seconds. Install it on the new server, another 10 seconds. Rsync the home directory to the new server - a minute or two via a fast LAN connection. Phew.

Now my client says we MUST use the exact same DNS servers for this domain. "Problem," I say. "You're using the nameservers for this domain in six other domains. If we move the nameserver IPs, those other six sites will break. I'll just change the host records for this website to point to the new server...it will take a few seconds." This works, but I guess is not what he wants as it is far too easy. I should have intuited that moving a website meant also moving the DNS server. I change the IPs back, he buys a new domain name, I create a new dummy domain on the new server, and give its zone two NS records that point to...you guessed it, itself. Now he has to change the delegation at the registrar. And create the glue records. And this just became a whole lot harder to undo if there is a problem.


Ah, cPanel's so-called easyapache. Re-compile apache and PHP, and choose all the addon modules you need. It's easy! Just don't think of using older versions of PHP. Or perhaps the older versions using deprecated functions still in use on a website you just transferred to a new server. Shit. . "Hmm, cPanel has old versions of PHP I can install, they are unsupported, but who cares! Sweet!" . Website now showing a blank page. . Apparently, enabling an old version of PHP as an addon module to easyapache helpfully ignores all other addon modules. I mean, it's old and unsupported, why would you need that pesky GD API?


posted at: 16:50 | path: / | permalink | cpanel, linux, sysadmin

Mon, 10 Sep 2012

Musings on Network Security

As a sysadmin, I have always thought simplicity should be a key guideline when securing Linux or Unix servers. That sounds rather meaningless by itself, so an example is in order. Anyone who spends time looking at the log files on an internet-facing server or firewall will notice the almost constant barrage of SSH brute-force attacks. SSH is indispensable as a remote administration tool, so it is likely to be installed on every such Linux or Unix system. Some admins like to install automatic analysis and blocking tools (e.g., fail2ban), but I dislike such tools because they are just another way of "enumerating badness" [1]. So I secure SSH with a set of simple changes:

Many admins balk at only allowing SSH from static IP addresses, especially with the prevalence of 'pseudo-static' IP addresses assigned to home cable or DSL modems. But it's not as limiting as you may imagine. Cheap VPS (SDF [2], Linode or AWS) systems routinely come with static IPs, and shell services like SDF offer login servers with static IPs. Agent-forwarding [3] can help make login through intermediate hosts convenient. If you must allow login from anywhere, configure a default-drop firewall and use single-packet authorization (SPA) [4] instead.

Each of these in isolation might not be very effective against a determined attack. But taken together, they provide a very secure environment for SSH. That doesn't preclude a server being compromised through some other network-accessible application, but with these changes SSH itself is quite secure. The idea, of course, is to secure all of your internet facing applications in similar, simple ways and if possible with a default-drop mindset. Web or Internet applications meant for public consumption are the one exception where default drop just isn't possible. Particularly in those cases, I add outbound filtering to host-based firewall rules. That way, if your shiny new wordpress install is ever compromised (when, not if), you can at least contain the damage.

Speaking of disabling PAM authentication, this quote from Patrick Volkerding, the creator of Slackware Linux is a great example of choosing simplicity:

If you see a security problem reported which depends on PAM, you can be glad you run Slackware. I think a better name for PAM might be SCAM, for Swiss Cheese Authentication Modules, and have never felt that the small amount of convenience it provides is worth the great loss of system security. We miss out on half a dozen security problems a year by not using PAM, but you can always install it yourself if you feel that you're missing out on the fun. (No, don't do that) [5]

It is notable that even today, PAM is not used in Slackware.

posted at: 18:25 | path: / | permalink | linux, networking, pam, security, slackware, ssh, sysadmin, tips

Sun, 19 Feb 2012

Alt.sysadmin.recovery Manpages

More sysadmin humor, the alt.sysadmin.recovery manpages.

posted at: 11:06 | path: / | permalink | humor, sysadmin

Don't Mess With the Sysadmin

A funny reminder not to mess with your sysadmin. Reminds me of the BOFH stories.

posted at: 10:28 | path: / | permalink | humor, sysadmin

Sun, 22 May 2011

Remote Access

I setup VNC access to a desktop for a client recently, which they promptly b0rked by replacing their router and with it all the firewall/port forwarding settings - without telling me. In trying to get access to try and fix it, I explained how I would first need the IP address for the new router. I received this helpful email in response:

I think we were able to set up remote desktop.  I have the following
info. Let me know if this works.

IP Address bob-24f763ed307
Ext/Int Port 3389 for Remote Desktop

posted at: 20:03 | path: / | permalink | sysadmin, wtf

Wed, 06 Apr 2011

Using Old OSes On Servers

Of all the linux distros or BSD's to choose from, I would say Fedora ranks at the bottom for me as far as production server use. It's really meant as a testing OS, to test new ideas before they get incorporated into RHEL. While there are issues with any old operating system as far as community or vendor support, Fedora releases in particular have a very short lifespan (Fedora Legacy, which had been providing support for old Fedora releases, was shut down in 2007). I mention this because I have a client that contacts me every few months for help with some intractable server issue. From just a security perspective, this is scary, FC5 was released in 2006:

[root@www log]# uname -a Linux hostname 2.6.9-023stab051.3-enterprise #1 SMP Wed Nov 4 19:28:06 MSK 2009 i686 i686 i386 GNU/Linux [root@www log]# cat /etc/redhat-release Fedora Core release 5 (Bordeaux)

posted at: 16:39 | path: / | permalink | sysadmin, wtf

Mon, 02 Aug 2010

This Server is a Tad Overloaded...

A server I do development work on...yikes:

15:32:42 up 259 days, 19:17, 72 users, load average: 300.82, 272.70, 190.05

posted at: 21:24 | path: / | permalink | sysadmin, wtf

Tue, 20 Apr 2010

Partitioning Woes

Who partitions servers like this? A braindead hosting provider, that's who.

Filesystem Size Used Avail Use% Mounted on /dev/sd1 16G 1.5G 13G 10% / /dev/sd3 4.8G 275M 4.3G 6% /var /dev/sd2 246G 564M 233G 1% /home /dev/sd0 99M 18M 77M 19% /boot

I mean think about it.../tmp, /usr/local, and /usr are all lumped into the root partiton. /boot is tiny, given that this is a 300GB disk, and on CentOS (this is a CentOS 5.3 dedicated server) that tiny /boot will fill up after a few kernel upgrades - since the old kernels and related files hang around unless you delete them. And 5GB for /var? Again, it's a 300GB disk...meant to be used as a web/database server - you could be a bit more generous here. While we're at it, LVM would be nice. Sheesh.

posted at: 12:46 | path: / | permalink | WTF, sysadmin

Fri, 31 Jul 2009

Happy Sysadmin Day!

Happy Sysadmin Day [0], and thanks, smj! That is all.

posted at: 15:08 | path: / | permalink | sysadmin