Archive for the 'IT' category

No “Bandwidth limit exceeded”: How to manage bandwidth limits in cPanel hosting courteously

Monday, July 26th, 2010

We just had a small hosting accident yesterday.

One of our clients had his weblog cut off with the dreaded Bandwidth Limit Exceeded notice:

Bandwidth Limit Exceeded
The server is temporarily unable to service your request due to the site owner reaching his/her bandwidth limit. Please try again later.

Richard’s visitors had pumped 80 GB out already this month on his FreeTheAnimal weblog primal living weblog. Not bad for a single writer not in a formal network. Every month his traffic is growing. Congratulations Richard!

These 80 GB of bandwidth are the real thing, with just a few slightly overweight images, not a single big file accidentally uploaded.

Richard was surprised and upset to see his weblog cut off as were we. While most hosts cut clients off as a routine matter of business, we do not. We treat our clients as we would like to be treated ourselves.

Two of Foliovision’s core hosting policies are:

1. never cut off a client’s site for bandwidth
2. no bandwidth overages

Clearly this is a clear violation of our hosting and customer service commitment. We recently moved from the dreadful hSphere control panel to the comparative friendliness of cPanel. We knew how to fix this issue in hSphere but had missed adding it in our default cPanel server setup. Here's how you make cPanel behave much more courteously with your customers.

Start by turning off bandwidth cutoffs. It’s in WHM under Server Configuration, Tweak Settings:

whm tweak settings bandwidth limits

A very big page of options come up. Search for: “Disable Suspending accounts that exceed their bandwidth limit”:

disable suspending accounts bandwidth limit

While you are at it, you may as well hit up your users with email warnings at 80 and 90 and 95% bandwidth so they can ask for an upgrade or at least are ready for their upgrade when it comes. Unfortunately, these notifications only go to the client and not to you.

To go without bandwidth overage cutoff this, we need to know when clients are exceeding bandwidth so we can upgrade them. There doesn’t seem to be any simple notification system for admins. A workaround would be to create an account notification email for each client on your own hosting email account which is forwarded to them and then to you. Advantage is that you’d see what snarky emails your automated server software is sending out on your behalf to alienate clients. Still that’s lots of extra work when setting up an account and another moving part to break.

But there is a screen available in WHM: Account Information: View Bandwidth Usage.

It’s a lot of trouble to open up to WHM find that and click it. So I created direct browser bookmarks to our two cPanel servers which look like this:

http://91.162.85.74:2086/scripts/showbw

(change the numbers above for your own server’s WHM IP)

This gives you this very helpful chart to see who is in the yellow and red zone. I’ve already adjusted everyone’s plans to get them back into the white so sadly there’s no yellow or red on this chart.

disable suspending accounts bandwidth limit

If anyone knows how to get automated notices sent to server admin as well, please share the wealth.

By Alec

IT | No comments

Fighting Spam: SpamAssassin vs MailFoundry vs SpamSieve

Thursday, July 8th, 2010

At Foliovision, we consider one of our primary tasks keeping not just our inboxes free of spam but those of our high end clients as well. Especially as those clients use Blackberries and don't want to get spam on the road or all night long. At the same time, our high end clients are in businesses like real estate and insurance so false positives are a real danger. Fighting spam with no false positives is one of our toughest IT jobs.

Here's how we do it and how we did it, comparing SpamAssassin, Mailfoundry and SpamSieve (Apple only unfortunately, but there are Windows alternatives available, including the built-in filter in Thunderbird).

cPanel comes with SpamAssassin by default. SpamAssassin is powerful software but can come up with false positives. The default setup is 5. Moving it up to 8 or even 6 will reduce your false positives substantially. Viagra type spam comes in at 12 plus. At WiredTree, they have configured email with Exim to extensively use RBL (real time blackhole). This is so very important as a huge amount of spam doesn’t even reach SpamAssassin or your inbox. Unlike spam filtering, RBL filtering is quite accurate. Not only that but your legitimate correspondents will get a bounce notice so they can try and contact you via other means. Given the profluence of social networks (LinkedIn, Facebook, Twitter), contact forms and free email services (Gmail, Yahoo, Hotmail just to name the big three) if they can’t be bothered contacting you otherwise, you probably didn’t need the email.

The other huge advantage of RBL filtering is the amount of spam which you have to check by hand, goes way, way down.

Previously we used MailFoundry at CartikaHosting which was reasonably accurate but suffered from the following problems:

• far too much spam gets through to the blocked list which means you have a lot of spam to comb by hand
• far too many false positives which means you have to go through those lists, as MailFoundry learns from experience (i.e. if you allow false positives, the situation will get worse, unlike most installs of SpamAssassin where the chances of a false positive will stay the same)
• a separate report for every mailbox, even when you redirect them to a single bulk mailbox. When you have a lot of emails to cover (my situation), getting 7 to 20 spam reports to check per day, leaves you with nearly 150 per week. I dreaded getting those MailFoundry reports. With SpamAssassin, all the spam gets filtered in the bulk mailbox and I have just a single mailbox full of spam to check occasionally and which can be sorted by from addresss, date or subject line.

After dumb SpamAssassin (non-learning), one can add a local learning spam filter on your primary computer (it won’t help with your secondary machines) to reduce spam still further. I use Michael Tsai’s SpamSieve and am very satisfied with it. Given the amount of email which goes through my computer, state of the art is worth the investment for me. Both Thunderbird and Apple Mail come with quite decent built-in Bayesian filters which would probably do for most people.

With your SpamAssassin settings up at 8, you won't have too many false positives (default 5 is dangerous) but with a local trainable filter (setting up and managing training filters in SpamAssassin is far too painful even on IMAP), you won't see much spam either and you can train your own ham (marginal spam) to pass your local filters.

The next nice trick that SpamAssassin has is the spam box. It will automatically file everything away where you don’t have to see it unless you want to. Spam will be removed from your inbox before it gets there. That’s very important for Blackberry and mobile mail users as the last thing you want on the road are notifications for spam. Moreover, on these mobile devices it’s not easy or inexpensive to run spam filtering. You can even route your spam to a secondary mailbox based on the subject line. In the secondary mailbox it will be directly routed to the spam folder based on the subject line, further reducing the number of spam boxes one has to check. The disadvantage is that any false positives will be much harder to find in one’s secondary mailbox (thousands of spam per week, in comparison to the tens of spam per week in my primary mailbox).

By leaving my bulk email in POP with SpamAssassin, smart filters and Bayesian filters, I end up with just a SpamAssassin spam box to check and my local SpamSieve filter. The best of both worlds.

This is a huge advantage over MailFoundry and its endless reports. I'm very happy to be away from Cartika Hosting's Hsphere/Mailfoundry combination and back on cPanel/SpamAssassin with RBL at Wiredtree.

By Alec

IT | 2 comments

Setting Up Email Securely on cPanel servers: example WiredTree

Thursday, July 8th, 2010

You do want to be using SSL. Unencrypted connections are far too easily eavesdropped. On the other hand, it’s worth remembering that SSL only gets your login and email encrypted between your computer and your smtp server. Once your email hits the big pipes, it’s unencrypted again, vulnerable to whomever can get access to the transit points. A rogue operative in any ISP or fiber optic supplier could still siphon off huge amounts of data. Even if such a person existed, s/he would be unlikely to be able to regularly get all of your email though. However, random emails, especially if they traverse exotic territories with loose security could be grabbed.

Email is not private. Don’t forget that ever. Email is not private.

Thunderbird, Apple Mail, Outlook Express (and the hideous blue Windows Live Writer replacement) all offer SSL. But be sure not to check “Secure Authentication” if your particular host does not support it. Secure Authentication just has to do with security certificates and does not actually increase your security once you have an appropriately made self-signed certificate. Just be careful about authorising changes to your security certificate.

SSL IMAP will go across port 993. SSL POP will go across port 995. Checking your port assignment is one way to be sure your email connection is encrypted. SMTP SSL will work across port 25 (and probably some others, but I've tested port 25).

SSL will get you most of the way there to reasonable privacy.

While you are at it, make sure that when you access webmail (with cPanel at http://yourdomain.com/webmail, the connection forwards to an https address.

Don’t forget that when you are visiting other login protected websites, unless the connection is https, you are handing over your login and password to the owner of the hotspot (if he cares to log it or take it). If you plan to spend a lot of time on hotspots as a traveller, you need more than SSL email. You need a VPN. Don’t go cheap or unbranded on your VPN supplier (setting up your own VPN is a big enough hassle that most small businesses should be outsourcing their VPN connections), as that organisation will have steady access to all of your communication and can keep really detailed logs. Unlike a single rogue hotspot, you will be using the VPN consistently over time and with a the same login.

These steps should not be considered security overkill, but just the basics.

By Alec

IT | 3 comments

How to move clients email accounts (or your own) from one server to another with no lost mail

Wednesday, July 7th, 2010

The most important thing which you must know before starting the move is whether the mail account to be moved is POP or IMAP.

If the account is POP, your task is fairly straightforward.

You want to make sure that you move any unread move (mail from between the time your client last collected email and the time of the move is picked up and put on the new mail server). The best way to do that is to log into the old mail server and the new mail server via IMAP simultaneously. You will see what has been read and what hasn't. Just move the unread messages.

If you move the read messages, when your client logs in again via POP, he or she will have to sort through a 1000 or even 3000 archived messages in the inbox. Not fun.

We recommend using Apple Mail as the IMAP client as it's very easy and quick to set up. Windows Live Mail hides the IMAP accounts and folders and is ugly as sin. Thunderbird is very fiddly and exposes too many options but could do in a pinch or if you don't have a Mac handy.

After the move, there may also be a small lag while your client is only seeing the old server for reasons of DNS cache and won't get his or her new mail. For that reason, it's better to shut down the old mail server immediately after transferring the mail so he/she can't be picking up mail from both places at once.

For that reason I recommend doing the move at night at 3 in the morning, as your client has probably turned off his or her computer so the DNS will be renewed in the morning. If not, you'll have to ask them to flush their DNS cache (it's easy enough via GUI without IPconfig: turn on and off networking in the network control panel in Windows, Macs will flush the DNS automatically by switching network configurations). In the case, that even that is too technical, a restart will do the trick.

For a truly seamless move, it's essential that your host is using a modern convention for the mail login and smtp. All of our good hosts are using mail.domainname.com for POP, IMAP and SMTP. You will probably also be so lucky. If that's not the case, then you definitely have to involve the client in the move so that they will have the new login information.

At Foliovision, we proceed on the principle that our clients don't want to know about the IT unless they absolutely have to. They have work to take care of. It's up to us to sweat the details. And when we say sweat, we mean it. Ideally, they'd never notice that anything ever was changed or went wrong.

In principle, after moving from one host to another you should change all your passwords (the old ones are compromised from the previous host). In practice, I recommend keeping the passwords the same initially so that all of a client's automated logins will keep working. One doesn't want to be troubleshooting passwords and account moves simultaneously. As long as the passwords are the same, the move should be seamless.

Of course in most cases your clients will have to authorise the new server for email. Most will do so as a matter of course. (Get asked enough about security, you stop caring.)

If your client is already using IMAP, your task becomes much more sophisticated. If he or she is just using the standard IMAP mailboxes (Junk, Sent, Trash), your life is pretty simple. In this case, you simply move the contents of each mailbox (including Inbox) to the new account. When your client logs in, the messages will match and he/she will carry on work as before.

If your client has a sophisticated server side nest of mailboxes, you have a lot more work in front of you. First you have to duplicate the mailbox structure by hand and then drag the contents of each mailboxes over by hand. Dragging and dropping whole mailboxes won't work (at least in Apple Mail), as they are made into subfolders of the inbox.

In that case, you can look at zipping up the maildir folders, moving them over and resetting permissions. That leaves you no guarantee that the accounts are working properly. So if it's not a high volume mail move, I recommend a move by hand to be sure everything is working properly before you leave the job.

Here's some gotchas to look out for with a mail moves. Basically, POP ignores IMAP's read and unread flags.

• For IMAP moves, careful not to touch unread mail on the server as the client may never see it (his/her email client will ignore read mail)
• when POP picks up mail, they are marked as read in IMAP (bad).
• if mail is marked as read in IMAP, it still gets downloaded via POP as unread (bad).

STEP BY STEP GUIDES

First check the mail server logs to see which clients log in via POP and which login via IMAP. It's all there in black and white with usernames. You need root access for this or submit a support ticket for shared hosting.

Here's the step by step guide for POP moves:

1. set up new mail account on new server
2. set up IMAP account for old account in Apple Mail
3. set up IMAP account for new account in Apple Mail
4. drag only the unread contents to the new account
5. archive the old account for a few days (by disabling the mailbox: don't forget to delete all these accounts within a week for privacy reasons)

Here's the step by step guide for IMAP moves:

1. set up new mail account on new server
2. set up IMAP account for old account in Apple Mail
3. set up IMAP account for new account in Apple Mail
4. recreate the mailbox structure of the existing IMAP account
5. drag all the contents to the new mailboxes one by one
6. archive the old account for a few days (by disabling the mailbox: don't forget to delete all these accounts within a week for privacy reasons)

By Alec

IT | No comments

Apple Mail, IMAP, IDLE and Smart Mailboxes don’t mix well, spike CPU

Saturday, June 19th, 2010

As I've mentioned, I handle hundreds of messages per day for myself and my clients. I have separated my email from bulk email effectively now, but still found my computer sluggish.

The problem seemed to be around Apple Mail. I'm new to IMAP so I decided to dig deeper. I initially thought the issue was with Rules, as I have SpamSieve and many dozens of rules to deal with bulk email (things I might want to read but don't want in my InBox).

The problems turns out to be something else altogether: 

Smart Mailboxes. Every time you get new mail your Smart Mailboxes folders update their unread counts: "Updating Smart Mailbox Unread Counts" is the message you will see in Mail's Activity Monitor.

Apple Mail
Smart Mailboxes
Spiking CPU

I have a lot of Smart Mailboxes (great feature, btw) which allow me to check how many leads each of my clients received this week and this year. At a glance, I can see how business is going (don't forget to weed out automated tests occasionally). Some of these mailboxes have many thousands of messages. Updating them takes a few seconds each.

These Smart Mailboxes get updated every time you get even one new email.

Normally I check my email manually only which means these updates don't happen very often and happen at a time when I'm working in Mail and am not surprised or bothered by 15 seconds of sluggishness. On the other hand, with my primary account as IMAP on IDLE that meant every time I received a single email all my spam filters and Smart Mailboxes updates ran each time. Given that even my primary email gets at least a couple of hundred emails per day, that's more workstops than I'm willing to put up with.

The simplest solution then would be to close Mail altogether when I'm not using it. This option doesn't appeal to me at all as I use Mail for reference and for writing messages even when I'm not checking for new messages.

1. First issue: with IDLE checked (if it works, it often doesn't), you will get new message pushed to you like it or not. Turn off IDLE
2. Second issue: make sure your general preferences are set to update only manually.
3. Third, pray.

This will probably stop IMAP from updating automatically, making life much better again. At least when Mail is a background application.

Preferences checking for new mail manually in Apple Mail

If you click into Mail and start looking at IMAP messages for every unread message you read, the whole cycle of Smart Mailbox updates start again. There is the feeling of sluggishness.

That's the price of using IMAP in Apple Mail apparently. There is no solution I can find, apart from deleting all your Smart Mailboxes. Which makes Apple Mail no better than any other Mail client, albeit a little bit prettier.

I thought I had a solution here, but in the end but just found a problem. Not even prayer will help here.

What we need is a way to turn Smart Mailboxes on and off (I don't need them all the time, I just need them when I'm in the mood to do a bit of analysis). Deleting them all and recreating them is not really an option. It took me weeks to refine them.

One way of dealing with this would be to disable Spotlight (which would stop smart filtering) but that would mean no advanced search function, something I use everyday.

The only workable solution is an on and off option (probably in right click) for Smart Filters. Or even by each Smart Filter with right click. I don't expect that from Apple. But perhaps someone can find a hidden preference that we can use from the command line. Otherwise one has to go back to POP (not as much of an issue with POP as there seems to be a delay before the filters update).

Are there no heavy Mail users at Apple, who use Smart Filters in their work and use IMAP too?

By Alec

IT | 8 comments

Apple Mail: Migrating from POP to IMAP Smoothly for Power Users

Thursday, June 10th, 2010

If like me, you are an old Eudora hand, you probably used POP reliably for decades before moving to Apple Mail and the possibility of troublefree IMAP use.

Apple Mail Setup IMAP: To Take the Leap Or Not?

You’ve probably also heard horror stories of unsynced and lost email from those who took the jump to IMAP in the 90’s. You prefer the security of local mail on POP for the following reasons:

• your mail doesn’t spend much time on the cloud so there’s less possibility of it being read unless someone is actively tracking you
• what’s on your computer is getting backed up by you so you have physical control of the data

Now however you may have a Mac Mini, a Macbook, Macbook Pro 17” portable desktop, a Windows 7 netbook, a Nokia N97 mini smartphone with keyboard, an iPhone and an iPad as well as a photo studio Hackintosh. Or five of the above at any given time.

Making sure you always have local up to date email with you is a hassle (you may be doing it by carrying a FireWire boot drive with you everywhere, but that doesn’t work on the netbook or the Hackintosh or the iPhone or the iPad or your Nokia N97mini either.

Working with the web interface is getting old as it means going through the same email twice, once online and once locally to get everything put away in the right folders.

If like me, you have a lot of custom sorting routines built in to Apple Mail, you certainly don’t want to give up those two hundred odd handbuilt rules.

But have no fear. All is not lost, even for email power users. You can smoothly move to IMAP for your important mail. I receive at least 600 emails per day, not counting SPAM.  So if I can successfully make the POP to IMAP move, so anyone can. Fortunately, the vast majority just need to be filed for future reference.

The first thing to do is to separate essential email from non-essential email. If you’ve been using throwaway addresses and forwards (which you should have been doing all along), make sure all of those extra addresses end up at an address like Leave that non-essential mail in POP where it will be sorted automatically as it comes in and put in all your custom local folders.

With your primary address, you will have a lot less mail to deal with. You likely have just two goals for incoming mail: Archive or Delete. I recommend you keep your IMAP structure quite simple. That’s been enough for me.

What I’d like to see again, I archive after reading, what I don’t want to see again I delete and what I haven’t dealt with yet stays in the inbox for the moment for when I get back to my main machine.

Of course that means I can’t deal with all the less important email when on the road. But that’s no problem. When I’m on other assignments I’m just as happy not to see all those newsletters and can go after the bulk mail when I’m particularly bloody minded.

Now enjoy being able to really work with your email on every machine you own. I recommend Thunderbird (now has a widescreen viewing option) on Windows and Apple Mail on Mac. Come home with your work done.

Every once in a while, I’d recommend moving your archived email to a local folder. That way not too much email stays online and you also have the security of local backup.

local and imap folders archive

IMAP will change your life for the better. Enjoy mail on all your mobile devices. Maintain sorting for bulk mail.

Now that the essential prep for the move is done, I will post the nitty gritty technical details of POP to IMAP transitions.

By Alec

IT | No comments

How to create a network backup with Apple’s TimeMachine

Tuesday, May 11th, 2010

If you have tried to set up network backup on OS X and you ran into the message "the backup disk image could not be created", probably this article will help you.

OS X's TimeMachine software had native support for network backup until the OS X  Leopard 10.5.2 was released. Apple had its own reasons for the decision to remove network backup, but many advanced users including us at Foliovision would still like to be able to back up over the network.

We have a bunch of Mac Minis in a mixed network of Linux and Windows computers. We'd like to use all our Minis for work and not for backup and use one of our older Linux towers to store the backup.

Fortunately Apple left us an option to turn-on network support for TimeMachine by running the following command in terminal:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

We can finally see network hardrives in selection for backup places. The problem might seems to be solved. No dice: it is only the beginning.

If you select a network drive for backup, you will see a very familiar message:

the backup disk image could not be created

Google automatically completes the phrase as soon as you type "the backup d".

The real problem is in the file system which must be used for TimeMachine backup. TimeMachine only supports Mac OS Extended (Journaled) file system.

The solution is to create a place on the network, which will trick TimeMachine, into thinking that it holds data in Mac OS Extended (Journaled) file system. This trick is accomplished via copying a sparsebundle image (with special name) to a network share. You have to do following steps for to make TimeMachine successfully run a backup.

OVERVIEW

1. Enable network hardrive support in TimeMachine
   
2. Mount network shared place for backup data
   
3. Create a sparsebundle virtual image
4. Copy the sparsebundle virtual image to the network shared place
   
5. Set up TimeMachine for network backup
6. Optimisation and other information

1. Enable network hardrive support for the TimeMachine software

Network hardrive support for TimeMachine is turn off by default. For changing this fact we have to type following command to the terminal:
defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

After that, you should see mounted network harddrive in TimeMachine's locations for backup. If it is not this case, probably a restart is needed.

2. Mount Network shared place for backup data

If you have a shared location for backup data on Linux server, you can map the directory as a drive to Mac in Finder application. You have to do following steps:

1. Click GO and then “connect to server” in finder menu.
   

   

    

2. Write following path if your sharing is based on protocol samba (SMB):
   SMB://<IP_ADDRESS_or_COMPUTER_NAME>/<NAME_of_SHARED_PLACE> (e.g. SMB://192.168.1.25/Backup)
   

   

3. Create a sparsebundle virtual image

This step is very important, because the TimeMachine doesn’t allow to backup data to a network drive, which has any file system except “Mac OS Extended (Journaled)”. So we have to create a disk image in "Mac OS Extended (Journaled)” format.
We have two ways to do it:

1. Open disk utility (Disk Utility) and create new virtual disk with following atributes:
   • Save As: <computer hostname>_<hex mac address of en0 interface>.sparsebundle (e.g. MacAlec_00ef9a048c4f.sparsebundle, if you forget to add .sparsebundle, it will be added automatically)
   • Volume Name: Backup of <computer hostname> (e.g. Backup of MacAlec)
   • Volume Size: the max amount of space you're going to set aside for backups. (The volume size is depends on amounts of backup data. I have chosen 150 GB)
   • Volume Format: Mac OS Extended (Journaled)
   • Encryption: None
   • Partitions: No partition map
   • Image Format: Sparse bundle disk image
     

     

   • Note: Disk utility automatically mounts the image to system volumes. We won’t need the image mounted, so we can safely unmount it.
2. The previous step can be done by one command in terminal:
   hdiutil create -size 150g -fs HFS+J -volname "Backup of MacAlec" MacAlec_00ef9a048c4f.sparsebundle

3. Copy the sparsebundle virtual image to the network shared place

You can copy the sparsebundle image to the network shared place using one of the following steps:

• Copy the file to shared place by drag and drop operation in Finder application.
• If you prefer to work with the terminal, you can type this instead:
  cp -r <sparsebundle_image_disc_location>/<computer hostname>_<hex mac address of en0 interface>.sparsebundle /Volumes/<NAME_of_SHARED_PLACE>/(e.g. cp -r /Users/Alec/Documents/MacAlec_00ef9a048c4f.sparsebundle /Volumes/Backup/)

Now we can safely delete the sparsebundle image copy on local computer once we are sure that it has been copied to the shared location.

Set up the TimeMachine for network backup

1. Open TimeMachine preferences (in SystemPreferences).
2. Click on Change Disk button for select the network drive for backup. (We have to select the "Backup"in our case.)

3. 

4. The first backup will start in two minutes. TimeMachine supports incremental backup, thus the first backup can take very long time (it is depends on network bandwidth and amount of backup data).

If you want to eject the mapped shared volume (/Volumes/Backup in our case), you have to do it within the two minutes countdown before backup starts or after backup is finished. TimeMachine has its own mechanism for mapping network hardrives, so a backup process isn't interrupted by your hard drive mounting or unmounting. TimeMachine will automatically mount the virtual sparsebundle image, when the backup starts. You will see Backup of <computer_name> as connected device on your desktop, so you won't need to manually mount the network hard drive later.

4. Optimisation and other information

 in order to avoid long delays in backup process caused by Spotlight indexing, You should set the Spotlight application to not index the mapped network drive. Here's how to remove spotlight indexing for a hard drive:

• Open up the Spotlight software preferences in system preferences window
• Move to privacy options and add the network drive by plus symbol
  

  

If you'd like to back up a  little less often, you can modify the file com.apple.backupd-auto.plist which is located in /System/Library/LaunhDaemons/ to change backup time interval. Open up the file in text editor and find the section:

<key>StartInterval</key>
<integer>3600</integer>

You should change the number 3600 to the number of seconds of your backup interval.

Result

I deliberately deleted some files to test backup reliability. This worked just fine. I was able to restore data from backup using TimeMachine's restore feature. TimeMachine restore is easy to use and lets you choose data from any date and hour.  If you have followed the exact steps above, you should have a working network backup now on a non-Apple computer.

I would like to thank Nick Hilliard and MacCorner for useful information.

By Matej

IT | 2 comments

Keep your Apple OS X Computer Running Fast with MenuMeters

Friday, March 12th, 2010

MenuMeters is a superb tool for those who use their computers heavily.

While you are multitasking you are instantly alerted to issues in uploading, memory leaks and paging, caches being permanently to disk, core processes or user projects getting stuck.

Frankly the cost of the instant info is having a fair amount of your menu bar taken up with the four indicators.

OS X MenuMeters Raging Menace

So on my most recent two Snow Leopard installs I tried to get by without MenuMeters. Bliss in simplicity. Higher productivity.

No such luck. Without instant visual feedback, your computer will bog down on a broken Internet connection or a runaway process, slowing one down more than the milliseconds to see where the issues are.

When you notice a problem, MenuMeters helps you dig deeper very quickly. Under the connection section you can instantly open up Network Utility, Network Preferences or Internet Connect.

Under the memory pie chart, you can see full details of what is happening with your memory.

MenuMeters disk usage

Under the processor indicators, you have the option of opening Activity Monitor or Console. Console is your best friend for quickly finding malfunctioning software, often from the nominally most reliable sources i.e. HP Print and Scan drivers. Most issues are clearly logged there). So having a directly link to it is great.

Normal mortals may not need this but if you are typically running 15 plus apps with lots of open browser tabs and open browsers (you know who you are), you should have this.

MenuMeters is truly well thought out software which helps one do one's work better and faster. The kind of software which makes owning an Apple computer a better experience.

By Alec

IT | No comments

Apple iTunes Library: Move iTunes Successfully via the Finder

Tuesday, March 9th, 2010

 For reasons unknown, Apple makes it really difficult to move around one's iTunes library.

Just moving the library to another hard drive will result in all the files being disconnected. Unlike Aperture or Final Cut Pro, there is no automated path fix. If you want to correct the paths you have to do it file by file.

There's also a hidden function inside iTunes which is really deadly (I believe it comes turned on by default) to reorder your library. If you do that, compilation albums will often be broken into the individual song. Each in their individual artist folder.

Basically if you let iTunes loose on your library, you will entirely lose Finder organisation. Guess what? Then you will be fully dependent on iTunes as no finder based album play system (the excellent Vox for instance) will work well anymore. So there is method to the madness.

Even the songs which don't have correct metatags will all get dumped in a large virtual graveyard instead of being left in their date or album structure.

Assuming that you were clever or cautious enough to turn off allowing iTunes to organise your library, you don't want to let Apple get their grasping hands on your library now when you move it with consolidation or anything else.

There is a work around which requires getting your hands dirty but indicates just how easy it would be for Apple to get it right.

iTunes library files

1. Quit iTunes.
2. Go into your iTunes library (probably ~/Music/iTunes) Once you are there, backup the iTunes xml and database files.
3. Open up the iTunes XML (use a serious text editor like TextWrangler/BBedit or Smultron, we are talking about 10 to 20 MB of XML data for a six to twenty thousand song library)
4. Do a find and replace in the iTunes xml file to update all of the song locations. They are stored as simple paths like: file://localhost/Volumes/MusicHall/MusicMovies/Music/
5. Save and close the iTunes XML
6. Now open the iTunes Library file in a text editor, delete everything, and save. If you don't delete the data inside the library, iTunes will prefer the data from the database and erase your XML file (be careful with this one it happened to me the first time and deleted my special hand made XML file. I didn't have the right backup: in this case, delete the new iTunes Library and iTunes Music Library XML, put the original iTunes Library back in, regenerate the XML and start again and make sure to delete the data inside the iTunes Library but leaving the file in place).
7. Reopen iTunes: the library will be rebuild from the xml.

All your files should be in place now, including metadata. Apparently the date added will be changed to the current date but that's a small loss.

Why Apple must make basic maintenance so difficult, I have no idea. If it weren't for the internet (the base of this hint came from Metafilter), it wouldn't be much fun owning a Mac anymore, as it would be impossible to run it as a free machine.

One of my principle grips about iTunes is that it stores album art in a database, as well as all your song ratings, instead of in the original files as well. Earlier versions of Aperture were criticised for the same thing, but pro users were refusing to use Aperture because of the data lock-in. Apple can get away with a lot more with a free app which is obligatory for all those iPod and iPhone owners.

By Alec

IT | 7 comments

FreeDB or Musicbrainz: Why is there no software to upload album info in OS X?

Tuesday, March 2nd, 2010

Whatever happened to the CDDB and to FreeDB?

CDDB evolved into Gracenote. It looked like they were losing their stranglehold when Roxio moved to FreeDB in 2000. A closed settlement resulted in Roxio moving to Gracenote full time. I hope they were clever enough to get free stock in Gracenote for the pleasure.

The next death knell (although no one knew how important it was at the time) for FreeDB was that Apple went with Gracenote and then disabled any ability for users to submit to FreeDB (for a couple of years it was possible to use the FreeDB servers instead by monkeying around in one's hosts file, but it was a pretty techy solution). Without iTunes or Roxio's Toast, FreeDB was cut off from any oxygen in the Apple ecosphere.

Gracenote was recently sold to Sony for $260 million. The venture capitalists and the thugs at Gracenote managed to get something for their trouble.

In the meantime in about 2006, the FreeDB had a melt-down between the project owner and the lead developers. In the meantime, the horribly named Musicbrainz hit the scene with a music recognition algorithm. Terribly complicated, terribly slow. Apparently it works. But there is no easy way to submit data.

MusicBrainz

My inspiration here? There isn't a single tagging client I can find for OS X which will allow me to upload to either FreeDB or Musicbrainz!

There should be a client (free) which will grab the Gracenote/CDDB info which iTunes collects and resubmit it to both FreeDB and Musicbrainz. iTunes can't do something like that as part of its own license but the new client can.

If Gracenote wants to shut this client down, it begins as open-source and goes offshore. The client should include a manual option so that not all of the data is polluted. The client should allow itself to identify itself as alternative software (to make sure that the database recipient can't be faulted for accepting the external data).

Personally, I'd resubmit all my music info to Musicbrainz and FreeDB if this app existed. And I know a hundred more who would do so as well. Litigation is likely to drop off at this point, as the aggressive thieves at Gracenote have been paid out now.

I can't imagine Sony wants to go whacking through the bushes, snatching at end users.

The CDDB story is one of the best examples I've seen of how human beings can turn any act of grace (pardon the pun) into loathsome slavery.

This sad saga worries me as it suggests that Apple is more than prepared to turn our computers into corporate property. For the moment, OS X is very free and my data is my own, but frankly the rumours of DRM on the iPad for ebooks worry me.

If Apple thinks its core audience (hey remember us, we're the guys who kept you alive through that huge trough at the time of the clones) will put up with proprietary data formats and heavy DRM, they are very wrong.

In the meantime, I want a client to let me submit track and album info to FreeDB and Musicbrainz.

By Alec

Business, IT | 7 comments

Apple Mail: How to avoid being tracked yet still see images

Thursday, February 25th, 2010

Apple Mail is the email client I use as it looks good and has a great search function. But it doesn't matter if you use Apple Mail or something else, if you don’t want your every move tracked, you have to prevent your email client from loading external images. On OS X, I use Apple Mail and the way I block external images is with Little Snitch.

The way to do it is to block all connections except the ones you allow.

deny all connections Apple Mail Little Snitch

But for some services you do want to see the external images.

In my case one service for which I do want images though is Basecamp. 37signals finally upgraded their email notifications to html and they look a whole lot better and are arguably functionally better as well – as the presentation of information is clearer.

Basecamp html email notifications

So how do I get the thumbnails in Basecamp?

If you’re on OS X with Little Snitch and Basecamp, unblock:

• asset3.37img.com
• asset2.basecamphq.com

If these two don’t work, what you need to do is open the source of the email (in Mail command-option-U). Unfortunately 37signals chose to use Base 64 encoding for the html part (absolutely no need to do so).

You want to copy the part of the document that comes after this:

--mimepart_4b8295ce1ce1d_4f231be02701438a6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: Base64 Content-Disposition: inline

and before this:

--mimepart_4b8295ce1ce1d_4f231be02701438a6--

and then take it to ToastedSpam’s Base64 Decoder and plug it in.

You’ll now have plain html and you just have to search for asset or img in the html. Add those domains to Little Snitch to allow connections in Mail.

Allow Mail connections asset Basecamp HQ

If there are any other html emails for which you would like the images, you can use a similar technique to allow their images. Keep in mind most newsletters have a link to allow you to open the email in a browser which is a quicker and easier solution.

By Alec

IT | 5 comments

Backup Router-Firewall for a Small Office: D-Link DIR-100 vs ZyXEL Prestige 334

Friday, January 29th, 2010

At Foliovision, we still plan to build a dual load balancing open source router on one of our old computers someday soon. In the meantime, our trusty old D-Link DI-804HV router was misbehaving a couple of weeks ago with all kinds of routing errors and slowing down our work. Another consequence was lousy Skype communication. Happily enough after a full reset, I was able to get it back to normal.

D Link DI 804HV

Our router has to look after about eight to fifteen computers at a time so it's no longer the load of just a few computers. Our primary connection is 3 MB/sec download from Chello.

But we aren't doing much VPN work so our needs aren't extreme. It would be nice if Skype didn't drop off on us in the middle of work. Our main concern is redundancy. With 8 to 15 people working at any given time, even 15 minutes downtime is too much now. Four hours downtime would be nearly 40 hours work as most of our work requires Internet access at this point!

So we were shopping for a backup router, really as our workhorse was back in action. A router which would allow us to work for a few days in the meantime. We don't use any WIFI (or Bluetooth) at Foliovision as wirelesss is the biggest IT time sync and security risk. WiFi is also quite possibly unhealthy.

Most modern routers include the WiFi station whether you want it or not so there is a limited selection of basic routers without WiFi.

We bought two to compare and to keep one or both (I need one at home as well). We bought ZyXEL Prestige 334 and a D-Link DIR-100. I've had good experiences with D-Link and Asanté (not easily available in Europe) and horrid experiences with LinkSys and SMC (defective wireless cards sold in knowledge that they didn't work, useless return policy). We wanted something that would work properly out of the box.

ZyXEL Prestige 334

The ZyXEL cost about €30, the D-Link €25. In each case, less than $50. Not much for relatively full-featured boxes.

I prefer the classic D-Link look of the DI-804HV but the DIR-100 is very small, every elegant and very discreet. No complaints. The flashy silver ZyXEL looks like a refugee from a Star Trek set. Still, its silver shell is perfectly serviceable if a little bit more attention seeking.

D Link DIR 100

To test, we set each router up with its default setup with ports closed and ran SpeedNet tests.

I expected the D-Link DIR 100 to be not up to the test as it has just 2MB of RAM memory plus 8MB of additional flash. No problem. Handled our 3MB/sec cable connection perfectly, identical speeds to the D-Link 804HV.

D Link DIR 100 speeds

To my surprise the ZyXEL could only handle 1.4MB/sec download.

ZyXEL P 334 failure

The D-Link was very easy to configure with full features, so we didn't even bother wrestling with ZyXEL interface which has a reputation for being a little tougher to use as the other router/firewall manufacturers.

For an advanced router in the multiple hundreds of euros category (an area where ZyXEL has lots of candidates), the best way to go is with a custom built load-balanced solution. It will be more powerful and configurable and the software is all open source anyway. Based on what I've seen from ZyXEL's entry-level Prestige 334 unit, you are paying for brand and not performance.

For instance, the D-Link DIR-100 is open source. The operating system is MClinux. For the VPN section, the software is FreeSwan. D-Link is playing fair and releasing their source code. You can download the D-Link source code for their routers.

We run a Mac and PC mixed network now and the D-Link DIR-100 was easily configurable in Safari on Mac. No issues running it with both Mac and Windows running at the same time. In fairness, the ZyXEL didn't appear to have any issues either.

For a backup or basic router, the D-Link DIR-100 is the clear way to go. I might look up at picking up DIR-130 for its QoS (quality of service) priority filtering for our Skype calls. What's interesting about D-Link is the entry level DIR-100 is that it keeps up with their former mid-level 804HV. That's impressive and a wonderful migration of capability across the range over time.

By Alec

IT | No comments

Shareware awards just a scam? MacUpdate, Versiontracker and IUseThis.com vs the Windows world

Thursday, January 21st, 2010

We buy and use a lot of software here at Foliovision. We have all kinds of weird stuff running for checking web rankings and logging backlinks in our SEO business. We don't like Adobe much for price gouging so we buy all kinds of graphics bits and bobs to

Basically our rule is that if a software program can do it faster, then have a software program do it. This approach allows us to offer our clients more service within their budget. So we have contact with a lot of software. While out shopping online (how's that for a pleonasm) I've often seen sterling awards pages for what looks like really rubbish programming.

inventory builder bogus software awards

Where do these ugly little banners come from and how the software developers earn them?

It turns out just by submitting software. No, you say, impossible. Somebody's looked at the software. No, no one has ever looked at the software.

Prove it, you ask.

I don't have to. A gentleman by the name of Andy Brice already has.

He submitted a bogus piece of software to 1033 sites. His bogus soft included a screenshot like this:

bogus software submission screenshot

Andy's fake app was approved and listed on 218 software directories. Incredible. Even more shocking, he won 16 awards. Sixteen 5 star awards would be enough to give even the worst piece of malware a veneer of respectability.

So what is a shareware purchaser to do?

Buy a Mac. In the Mac universe, there are only three software download sites of any import with VersionTracker.com and Macupdate.com battling for top dog for the last three years, with IUseThis.com trailing.

The ratings and comments on all three are legitimate and minimally censored in favour of the developers. Even I'm aghast at some of the things I read on Macupdate in particular (Nate, for the most part, please keep them up, following the profiles of some of these curmudgeons is incredibly entertaining and keeps people coming back to Macupdate).

The sites are not perfect. One developer had friends inside Macupdate whom he tried to use to censor commentary (Misha, I believe). In the end, after some comments being pulled and the tempest in a teapot rising higher Nate finally intervened and put the review back online and warned off the developer.

In any case, these three sites don't hand out prizes. They allow developers to display user ratings badges on their sites like this:

If a developer posts an incorrect Versiontracker or MacUpdate badge he or she will be asked to pull the badge down immediately. In any case, it is standard practice for the badge to link directly to the developer's site. If a developer encourages sock puppet votes he's blacklisted. Unfortunately the list is only three developers long, so I'm not inclined to believe it's complete. On the other hand, perhaps Nate and team have caught and warned 200 developers but it didn't get to blacklist levels.

A couple of years ago, I had the feeling MacUpdate might become a bit smug as VersionTracker atrophied. IUseThis.com came along just in time and made MacUpdate pay more attention to the convenience of users of the site first. What kind of nuisance am I talking about? There was a period of six months where it was impossible to search MacUpdate from outside the site - you had to load their overly busy, distracting home page to do so. Finally they quit that about a year ago.

So in the Mac universe there are just three sites of any import. On any of them fake feedback is likely to be called quickly. There are no fake awards. If a Mac user sees any other award badges, s/he will ignore those banners.

Straightforward access to high quality shareware is another reason I decided to take Foliovision Mac-centric at the end of 2009. Previously we had only had one and then two Mac users in the company. Now we have moved to fifty-fifty. We do have some very bad copycat developers like Koingo Software (from beautiful BC just like me) who hawk their second class wares everywhere, but you can usually suss them out pretty quickly.

It would be nice if CNET would pull about three quarters of the javascript and half the ads off of Versiontracker so we could go back to a two horse race. I'm not quite sure what ails IUseThis.com but at least they are there as an insurance policy if MacUpdate starts to go off the same overly monetized rails as Versiontracker. But all in all, as Mac users we are quite fortunate in our developers and our shareware sites.

Especially in comparison to the Windows world.

By Alec

IT | 9 comments

Keyloggers for OS X – Why you should install one and which one to choose: Spellcatcher, BackTrack, logKext

Sunday, January 17th, 2010

Everyone who works on the web should have a keylogger. Browsers crash often enough when you are writing into a form or browsers have hot keys (especially forward or back) which will reload the page on you at an unexpected time, just when you are in the middle of a very long post.

 If you value your time, the question is not whether to use a keylogger
but which keylogger for Mac OS X to choose. Photo jgarber.

I've heard all the privacy arguments against keyloggers but I'm not sold. If you are typing into a computer, particularly one which is near constantly connected to the Internet, you need to accept that there is very limited privacy. For very private writing, it should be done on paper or on an old computer which is no longer capable of being hooked up to the Internet easily or at all (i.e. missing a network card and wifi and/or automatic DHCP).

By Alec

IT | 34 comments

Little Snitch shortcoming: Badly needs wildcards

Wednesday, January 13th, 2010

In this day and age, running a computer without some kind of an outgoing firewall is like driving your car with your eyes closed.

There are so many malfeasors - from phishers to corporate spies - trying to track you and place you and grab you every time you check your email or you browse the web, that everyone needs a firewall.

You can test this on OS X by installing Little Snitch and scrolling through your messages. A few of the outgoing calls are for innocent images, mainly they are for tracking tags and tracking images.

Little Snitch - a few of the Edgesuite calls on one week fresh install!

Little Snitch is free for 3 hour periods at a time so it can be installed to test and find out what domains you'd like to be blocking.

For long term blocking of nasty sites OS X, your solutions are threefold:

1. buy an outgoing firewall, i.e. Little Snitch. A bit pricey at $30 a license but it's Little Snitch or go hungry
2. configure your hosts file to block most of the major offenders (people do keep lists)
3. use GlimmerBlocker control panel to block the baddies via internal proxy (works on all browsers)

Of the three, GlimmerBlocker was the best and the simplest. Until after a year and a half GlimmerBlocker decided to seize up and prevent me from editing Wordpress sites. Apparently it's database got full or something. I lost eight hours trying to troubleshoot this mystery when I desperately needed to work so GlimmerBlocker is banned now. A pity as GlimmerBlocker is free.

A hosts file is also great. By adding bad sites to the hosts file and redirecting them to localhost (127.0.0.1) you stop them dead in their tracks.

One of the more complete lists of domains to block via hosts is kept online here: http://www.mvps.org/winhelp2002/hosts.txt

To edit your hosts file on OS X, the quickest way is to use Terminal to give TextEdit root access to the hosts file.

1. Open a terminal window and type the following: sudo /Applications/TextEdit.app/Contents/MacOS/TextEdit /etc/hosts
2. Hit return and enter your admin password when prompted and again hit return.
3. Your Hosts file should automatically open in TextEdit.
4. Copy the hosts from http://www.mvps.org/winhelp2002/hosts.txt or ssomewhere else (mvps wastes a lot of space on the x-rated zone for those of us not into: alternative suggestions to mvps welcome)
5. Paste it into your text hosts file.
6. Save.
7. Test that hosts is working by browsing to one of the domains listed in mvps.
8. Continue to edit and save at your leisure (at each save the hosts file is updated and live: well done Apple).

Unfortunately both LittleSnitch and the hosts file suffer from the same fatal flaw. They don't allow wildcards. So for instance, one of the worst bits of spyware around is Infusionsoft. Every time you get an email from a subscriber of infusionsoft they want to know if you opened it. Therefore every image is specific to you. But each user has his or her own subdomain, i.e. empowered.infusionsoft.com.

If you block infusionsoft.com in hosts with 127.0.0.1 infusionsoft.com, you will only block the homepage. You would have to block 127.0.0.1 empowered.infusionsoft.com for each and every Infusionsoft subscriber who sends you email.

Little Snitch automates this process and allows you to make the block application specific. I use Apple Mail exclusively as my email client (looks great, full feature set and very fast, btw) so Little Snitch and I pick off these offenders one by one.

What I really want though are wildcards in Little Snitch so that I can ban all infusionsoft.com subdomains with a single *.infusionsoft.com within Apple Mail. The same applies for edgesuite.net within Apple Mail (Edgesuite is for corporate email spies like Apple and eBay, Infusionsoft is for slippery marketers like Rich Schefren and friends like Mike Filsaime).

When is the Objective Development team going to get off their backsides and improve Little Snitch by adding wildcards? There is a three page thread in their forum which goes back to 2004 asking for subdomain blocking in Little Snitch. What's particularly galling is that ObDev have not even bothered to answer our concerns.

Until Objective Development add a subdomains/wild card feature to Little Snitch, you may want to hold off on the purchase. Managing outgoing requests one subdomain at a time is very tedious work.

In the meantime, if you'd like to block certain domains or you have doubts about a certain piece of software, there is a free solution. You can turn Little Snitch on for three hours while you deal with spam email or install new software, take note of the domains you'd like to block and then add those domains to your hosts file.

By Alec

IT | 1 comment