Mail, Problems and Solutions

10 Problems and 50 solutions for large mail systems

Introduction

If you run a mail site of over a hundred users then there are special problems you need to deal with. A large mail system consume computer resources like there is no tomorrow. In this short document we will document 10 key problems and solutions you might apply.

These issues apply pretty much regardless of the mail system you use, whether it is PC networked, or Unix hosted. Some solutions will rely on user training and user acceptance, but we must all recognise that not all users even understand the problem, let alone want to deal with it. The technical solutions you will need will be almost entirely system dependent and may need experts to implement them.

Centreline 2000 has been dealing with large mail sites for many years, including government and multi-national corporations. Our experience has lead to a development of skills, products and toolkits to keep mail under control.

1. PC Files are big

PC files, with all their "power" features, tend to be big files. A quick check on our file system of MS-Word documents showed files ranging in size from 3 kilobytes to a staggering 150K. The average size is around 7K and that is for single page letters. By comparison our Uniplex file size averages about 1.5K.

When you mail one of those PC documents to 10 users that's 70K gone just like that as the document is copied to the users local mail box. A hundred users in a few weeks are going to generate a lot of mail and consume a lot of disk space.

What can you do?

• Regular administration and clean-up of mail.
• Conversion of outgoing mail to small (maybe even text only) formats.
• Run a non-duplicating system in parallel, for example a notice board system
• Put "gates" on send mail that warn the user of a high volume mail (multiply the file size by the number of target recipients). You may even disallow mail over a certain size.
• If your mail and operating system allow it, put the mail store on an independent disk partition so that it can be expanded or even relocated over time.

2. Mail to "all"

If you support hundreds of users on distribution lists then a simple "mail to all" can have a major impact. Firstly, there is the immediate hit on CPU resource as the mail is processed into the individual mail boxes. Secondly, there is (depending upon your system) the impact of the mail being delivered over your LAN or WAN (watch that phone bill!). Thirdly, there is the loss of disk space as the mail is replicated for each user and other audit information goes in along side it.

Probably the worst site I have seen had over 50% of the mail store taken up by the days menu from the canteen - mailed to all religiously every morning! A legitimate use of mail certainly, useful to many of the mail recipients certainly, read and removed by all recipients - you must be joking!

In another instance I saw computer systems brought down simultaneously in the UK, the US and Germany as an ordinary user accidentally mailed a 700 page document to all.

What can you do?

• Remove the "all" distribution list.
• Distribute this information in other ways (Information boards, common file access...).
• Implement "junk" mail clean-up routines.
• Redirect "all" mail to a postmaster who reviews and accepts or cancels the item.
• Filter mail through "gates" to trap large volume messages.

3. Converting & viewing different file formats

A particular issue for sites moving from a single office automation system to a mixed PC environment is the multiplicity of document and file formats in the system. Even if your own site uses a single word processor, what happens when mail arrives from an external site?

What can you do?

• Add a "browser" to the mail client. A browser reads many file formats for display and printing.
• Add a convertor to the mail client, so that any user can use an incoming file regardless of format.
• Add automatic conversion to the mail server, so that during transmission it converts to the receiving users' word processor.
• Use "tagging" formats such as MIME that automatically assign file types to mail. You could then automatically add additional attached formats to the MIME message.

4. Removing junk mail

Now, you know that many of the messages on your mail system are junk. That is to say the messages fall into one of the following categories: unwanted, unsolicited, temporary or gossip.

What can you do?

• If your mail system supports user archiving or save-to-file then make sure users know how to use the facility. Then remove mail more than 7 days old from the incoming mail, unless archived.
• Remove unread mail after 7 days.
• Implement junk mail "tagging". Each message is tagged (e.g. today only, general, important). You can then remove different types at different ages, today only is removed each night, general stays for a week, important stays for ever.
• Move junk mail to other systems, notice boards or information sheets.
• Move gossip to special mail boxes
• Implement a user to user "intercom" system.

5. When to remove mail

You will have to remove it sometime, so when do you clear out the trash? Well, the biggest problem is knowing when and what is trash. If you can implement a "tagging" method (described above) then your battle is half way one. If you can't then it comes down to defining a policy, letting users know what the policy is, and then running the policy as a mail admin function.

The enforcement of the policy is important. When the system runs like this users will accept it. If the system is only run periodically then it will seem draconian every time it is run.

What does the policy say?

• Any unread mail is put in the trashcan after 7 days.
• Any read mail is put in the trashcan after 14 days, unless it has been archived or saved.
• Any mail message in the trashcan is deleted after 7 days.
• If you want to be lenient, give users more days grace, and save mail to archive media.

Incidentally, in some organisations (I have in mind government agencies, but it applies to other groups too) there may be a legal requirement for you to archive all mail for audit purposes.

6. Users who don't read mail

Some users never read their mail, it's that simple. But, they still consume resources - in fact they may consume more resource than most...

What can you do?

• Identify why they are not using mail. Do they know about it, do they need training, is there a technical problem, have they got a terminal? (I've seen it, don't laugh.)
• Check you've got the right mail address in your alias files (I've seen this too).
• Has the user been posted to another department or even left the organisation? How do you find out when people leave? You must setup contact with the personnel department if you haven't already. There are other system admin functions you need to run too when a user leaves.
• If the user will never use mail, then remove them from the mailing system.
• Does the user need something else, maybe a printed copy?

7. Mail viruses and trojan horses

New word processor documents can support built in programs, usually used benignly. But it is possible to "infect" a document with a destructive "virus". The user receiving the mail message calls it into his word processor and the next thing is a flashing message saying "Formatting all drives..."

This experience was amply demonstrated by Microsoft who managed to ship hundreds of thousands of infected disks with a (thankfully harmless) variant of this kind of virus.

What can you do?

• Implement a view only (no edit, no save) mail client.
• Only allow save to file on request to administrator.
• Audit any save to file actions.
• Filter all macros out of files before saving.
• Disable macros in the word processor (and spreadsheet, database etc)
• Implement virus checking policy - rigorously.

8. Archiving mail

Most large sites will archive mail rather than just delete it. Indeed there may be legal requirements to archive and keep for years. If you are operating a very tough clean-up policy (where mail is only available for a few days before archiving) then you must allow users to be able to request old files from the archive.

What can you do?

• Archive regularly - of course.
• Prepare indexes of the archive so users can identify messages.
• Consider free text or keyword searching to identify archived files.
• Maintain archives in a format that you can restore individual files simply from the archive.

9. Mail distribution versus duplication

Nearly all mail systems currently duplicate mail, that is to say a single message, sent to several users is copied into each users' mailbox. A few systems support distribution, that is a single copy is held and many people access it. In the middle are systems that generally duplicate, but at least have the intelligence to send only single copies over phone lines and let the receiving computer duplicate the files.

The thing about mail, is that from a resource viewpoint it is very effective at one to one communication, but inefficient at one to many communication.

What can you do?

• Look for implementations which support distribution.
• Make sure distribution is enabled (many systems don't do this out of the box even though they have the feature).
• Use alternatives for "broadcast" information, such as notice boards, shared files, user directories, etc.

10. Our users don't understand the problem

Well, of course not. In fact, why should they? This is essentially a computer resource problem, not a user problem. However, a little care and attention of your users can soothe a number of your aches and pains.

What can you do?

• Train your users.
• Watch out for "problem" users, maybe a quiet request about cleaning up, maybe mandatory training.
• Implement policies and stick to them.
• Recognise that less than 20% of your users will consume 80% of your resources, focus on those users.
• Your mail problem may represent a user need. For example, regular mailing of the telephone list means your users need a telephone directory utility.
• Remember it's actually your problem, not theirs.

Conclusion

Even a moderate sized site can soon find the mail system becomes a major resource hog. Implementing even just a few of the suggested solutions above will resolve a significant number of your problems.

For those sites that are coming from a Unix or VMS style background, beware the PC. Firstly it is an insecure device, users can load any old file onto it, either by floppy or from the Internet or almost anywhere. Secondly, PC programmers are used to having solely dedicated resource, their programs and data files look that way too, big and bloated and with many redundant features. Thirdly, PC systems promote new activities, mail was just text, but you will see text, graphics, even whole databases being sent round the mail system.

You cannot stop the PC tide, but you must protect yourself against it.