What do you do when you need to make a change to a system used by several thousand people? When making a change to any complex system, there is always the possibility of unintended consequences. The answer is this: Test, Test, Test. When you’re done testing and you’re ready to deploy your change be sure [...]]]>

What do you do when you need to make a change to a system used by several thousand people? When making a change to any complex system, there is always the possibility of unintended consequences. The answer is this: Test, Test, Test. When you’re done testing and you’re ready to deploy your change be sure to have a backup plan.

I found myself in this very situation last week. We found an issue with our email system which would affect only a few users in a very specific scenario; however, it did need to be fixed. After researching the problem and several possible solutions I settled on one. “Stop. What if this solution causes other problems?”  We host well over 2000 mailboxes on our email system, and while I consider myself to be a competent engineer no amount of staring at a potential solution is going to reveal all of the possible side-effects.

After consulting with more seasoned colleagues, the decision was to implement the solution on a limited number of mailboxes first and also have a safety net in case things go wrong. We would watch for a day, and if no one reported any issues we would start rolling out this fix to other mailboxes.

After half a day of testing our solution a report of undelivered mail came in, and after investigating it I found it was indeed a corner case I hadn’t considered. My stomach got that all-too-familiar feeling: the “what would have happened if I had done that with all 2000+ mailboxes we host” feeling. Luckily, I had my safety net to fall back on, and I was able to re-deliver the messages that had not gone through the first time.

Let’s take the example above and construct a list of general steps to solving a problem on a complex system:

1. Think about the problem. What caused it? What are the solutions? Which solution has the least impact on existing operations?
2. Do a small-scale test of the proposed solution. This test should not affect any person or process in production. Adjust the solution and repeat this step until no side-effects are found.
3. Test the proposed solution on a subset of the production system for an extended period. Have a backup plan in case something goes wrong. Adjust the solution and repeat this step until no side-effects are found.
4. Take the leap: deploy your change on a wider scale on the production system. This probably should be incremental as there is still a chance of side-effects not found before.
5. After the solution has been fully deployed continue to monitor the situation for unusual activity.

We all communicate in one or more forms today:  email, text messages, phone calls, even a face-to-face conversation — utilized less than ever today.  Communication continues to evolve no matter which medium you choose to use.  Every one of these mediums started out independent of the Internet, but today, each of them can be accomplished [...]]]>

We all communicate in one or more forms today:  email, text messages, phone calls, even a face-to-face conversation — utilized less than ever today.  Communication continues to evolve no matter which medium you choose to use.  Every one of these mediums started out independent of the Internet, but today, each of them can be accomplished (probably for free) via the Internet.  In my mind, this makes the Internet the universal medium for communication.

You may think that email has always depended on the Internet to function.  However, email was actually created in 1965 at MIT — the Internet didn’t come into existence until the late 1980s.  In its initial form, email was limited to a small group of people on a time-sharing computer system.  Only that group of people could email each other, so its usefulness on a global scale was non-existent.  When the Internet reached broad acceptance and use in the 1990s, email became an inseparable component that people grew to depend on.

While text messaging and phone calls are not dependent on the Internet like email is today, I have no doubt that they will become so in the not-so-distant future.  Currently, one can use services like Google Voice to do both.  However, while using Google Voice to make free voice calls to other phones independent of a for-pay service isn’t trivial just yet, Google’s recent purchase of Gizmo5 suggests that it soon might be.

Services like Skype and Google Talk have allowed video chat for quite a while now.  That’s about as close to face-to-face as you can get via the Internet with out current level of technology.  However, I look forward to the day when we have Star-Trek-style holograms that make it just as realistic to meet with someone on the other side of the planet as it is to drive to the other side of town to have coffee with them.  I have no doubt that when humanity creates that technology, it will communicate via the Internet (or some extension of it).

I’m also excited to see what other communication mediums will be created in the years to come.

I ordered some hardware so I could build my new PC earlier this week.  I wanted to take a moment to describe the anticipation that geeks nerds tech people go through when they order new computer hardware.  It usually involves checking the tracking numbers every few hours, starting only a few hours after making the [...]]]>

I ordered some hardware so I could build my new PC earlier this week.  I wanted to take a moment to describe the anticipation that geeks nerds tech people go through when they order new computer hardware.  It usually involves checking the tracking numbers every few hours, starting only a few hours after making the initial order.  Personally, I also read reviews about the specific components I ordered to convince myself that it was worth the money spent.

When the shipments finally arrived (only 2 days after ordering…thanks newegg.com), my anticipation of receiving the parts was shortly replaced by the anticipation of getting it all working.  I decided to utilize my lunch hour (or 2 hours on this particular day) to put it all together, and after it was finished, my anticipation continued when I pressed the power button to power it on for the first time.  My hopes were short-lived however, as it didn’t power on correctly the first time.  This problem only served to prolong my anticipation, only this time it was the anticipation of fixing the problem that was inevitably caused by me.

So, after having built this new computer, was it worth all the money spent and time consumed?  I can’t say I can legitimately say “yes” at this point, but I’ll reserve judgement until I finish my first LAN party with this new beast.

I’m going to take a break from my normally-technical blog posts.  Recently, I was reading through a few threads on one of the mailing lists I subscribe to, and I came across a wealth of information on a particular subject that I otherwise wouldn’t have been exposed to.  The subject of the discussion threads is [...]]]>

I’m going to take a break from my normally-technical blog posts. Recently, I was reading through a few threads on one of the mailing lists I subscribe to, and I came across a wealth of information on a particular subject that I otherwise wouldn’t have been exposed to. The subject of the discussion threads is unimportant to the topic at hand; the key lesson here is that community involvement can tremendously help people stay informed in whatever field they’re involved.

I’m specifically advocating local community groups that share expertise in areas of mutual interest. These groups can be something as simple as a mailing list on which people collaborate on issues they’re facing or topics they’d like to discuss with like-minded people. Some groups have formal meetings where people give presentations and have Q&A sessions. Some collaborative groups may even be sponsored by commercial entities that have a vested interest in ensuring that these groups continue to exist.

But don’t simply read the mailing list, or sit there passively in the meetings: get involved! Offer assistance to people when you see they’re having problems. If you have something you’ve found to be useful in your area of expertise, contribute to your local group(s). The great thing about this concept is that it has been so successful in so many different fields. Technical, business, artistic, and even hobby groups exist to help people collaborate. I might even call it a viral concept. I’ve personally benefited a great deal by several groups in my area, and I hope I can also contribute something to these groups in the coming days.

So, it’s my turn to blog again, and while Apple preventing the Palm Pre from syncing music with iTunes was a very tempting topic, I kept on reading my news feeds this morning to find something even more interesting and just as predictable: Twitter employee’s Gmail is “hacked”, and confidential information is compromised. I placed [...]]]>

So, it’s my turn to blog again, and while Apple preventing the Palm Pre from syncing music with iTunes was a very tempting topic, I kept on reading my news feeds this morning to find something even more interesting and just as predictable: Twitter employee’s Gmail is “hacked”, and confidential information is compromised. I placed the word “hacked” in quotes in order to point out that his Google account was not “hacked” in the traditional sense of the term; someone simply guessed his password.

Now would be the appropriate time for you, my dear reader, to think about your own Gmail/GoogleApps password, and decide whether or not it is easily-guessable by a would-be attacker.  Here’s a quick guide to ensuring password security:

1. Never EVER use a word from the dictionary as your password. I can’t tell you how many times I see people use words like “petunia” or “screwdriver” as their password. The convenience of remembering the password isn’t worth the risk of an attacker using a dictionary attack and (almost) instantly compromising your sensitive data.
2. Avoid sharing a single password on multiple websites. OK, I’ll admit it; I violate this one on some of my personal stuff. But, it’s really only with sites that I couldn’t care less if someone got into it. I have a standard I-don’t-really-care password for trivial sites like forums and news sites, and lots of no-one-will-ever-guess-this passwords for email, banking, etc.
3. You can find all sorts of sites telling you how to create a secure password, but I like this simple method: create a sentence you can remember, and then use the first letter from each word in the sentence to generate your password. You may even do some fun stuff like substituting numbers and symbols for letters in order to make it even more cryptic. Just make sure the resulting password is sufficiently long and not dictionary-based. Oh, and make sure the sentence is memorable enough that you don’t need to write it down on a sticky note.

Given the amounts of money that can be siphoned from even a single attack, criminals can afford to spend lots of time trying to guess your password. It’s definitely worth your time and effort to ensure the security of your sensative information.

Businesses such as Worthwhile are in a constant battle known as security.  This battle is taken up in multiple fronts: physical, interactive, and technological. Physical security is pretty straight-forward.  We put locks on our doors, we password-protect our computers and email, and we maintain regular backups of important data (usually multiple copies in more than [...]]]>

Businesses such as Worthwhile are in a constant battle known as security.  This battle is taken up in multiple fronts: physical, interactive, and technological. Physical security is pretty straight-forward.  We put locks on our doors, we password-protect our computers and email, and we maintain regular backups of important data (usually multiple copies in more than one physical location).

Interactive security is a term I just made up on the spot, but it describes a process that we’ve had in place at Worthwhile for some time.  I’m referring to our process of verifying that someone is who they say they are.  For example, if a customer calls us and requests that we make some change to their account (reset an email or FTP password, add a new domain, change a DNS record), we go through a process to verify that this voice on the other end of the phone is authorized to request these changes.  Like physical security, this is also fairly straight-forward, as we can clearly define a set of guidelines to help us make the right decision.

But here comes the kicker: technological security (Dan Wooster would call this “securing our bits” ).  Unlike both physical and interactive security which must only occasionally be adjusted, technological security is a constantly changing beast.  Like any other company, Worthwhile takes a layered approach to technological security.  We have firewalls, spam filters, and VPNs, and so on.  No one product can protect everything.  In addition, like many companies, Worthwhile relies on countless lines of third-party code.  We use popular content management systems such as Joomla and WordPress, as well as third-party and custom plug-ins to provide specific functionality.  All of these products contain known and unknown bugs.  So, what can one do to deal with this situation?  Here’s a few things that I consider to be essential:

1. Search for know vulnerabilities. I recently found secwatch.org which allows you to search several vulnerability databases at once.  This search will help you answer the question: “is this code known to be insecure?”  If so, it may be worth looking elsewhere for that functionality.

2. Personally perform a code review. Don’t simply download and install some WordPress plug-in that says it adds the best functionality since RSS feeds; review the code yourself to make sure that the developer took reasonable steps to protect his code from being exploited.

3. Contract with a security professional to perform a penetration test. We recently went through this process to test the security of our network infrastructure.  The analyst we contracted with wasn’t able to enter our network, but he did make some recommendations to our on-going security efforts.

I remember 5 or 6 years ago, there was a statistic that an unfirewalled Windows machine connected to the Internet could only last 30–60 seconds without being infected with some virus/worm/trojan.  Luckily for us all, Microsoft has since been taking a more active role in helping to protect Windows users.  However, as the maintainer of a complex network of different architectures and applications, I must take additional steps to ensure security in all forms.

Over the last several years, I’ve observed several trends in the online world that exemplify why it’s important to be careful who you trust to develop your online application. To support my opinion, I’ve included three examples. Evil 1. Not being careful with someone else’s data. I could site 1,000 examples of a company or [...]]]>

Over the last several years, I’ve observed several trends in the online world that exemplify why it’s important to be careful who you trust to develop your online application. To support my opinion, I’ve included three examples.

Evil 1. Not being careful with someone else’s data. I could site 1,000 examples of a company or government agency that screwed up and exposed personal and/or financial data. For the sake of your time (and mine) let’s look at the recent case of the FAA. In this instance the data breach exposed financial data for more than 45,000 FAA employees and retirees. Indeed, according to another article covering this incident [1], the FAA was actually warned about its data security issues, but ignored the warnings. Further, it took the agency a full week before they notified its employees of the breach. This example shows that other people pay the price whenever we — developers and technology professionals — don’t take our responsibilities seriously.

Evil 2. Intentionally and arbitrarily excluding a portion of your potential user-base in order to save yourself time or effort. Inutit’s Quickbooks software has implemented this practice. An error occurred that made linking to a financial institution unusable. Intuit gave us a “choice” — pay per incident to fix the issue, or pay for an upgrade and receive a year of support — both were the same amount. The new version is available online, but when the primary user attempted to login using Mozilla Firefox on Ubuntu Linux, he received a message stating logging in was not possible from an unsupported platform. This type of non-service is what we like to refer to as FAIL (yes, the all-caps is necessary for the proper effect!). I installed the User Agent Switcher [2] extension for Firefox, which fooled the site into thinking we were using IE7 on Windows XP. The user was then allowed to login and complete his transaction without a problem. The reason the “unsupportable platform” code was there was most likely to reduce the possibility of support calls for non-Windows platforms. But, let’s be honest. People who are using Linux on their desktop probably aren’t going to call customer support over an issue anyway. In addition, it probably took more time to add the code to refuse login than it saved in support time!

Evil 3. Vendor lock-in. Vendor lock-in is the practice of making it difficult or impossible for your customer’s to choose a different service provider. We’ve all been in this situation before — you use a product, you build up a chunk of information you’d like to carry to another program or device. However, the designer of the original product offers no way for you to export “your” data to a neutral format. Recently I ran into this when I was helping a friend. They have been paying monthly AOL fees for several years, when the only AOL service they were using was email. We tried to move their account to g-mail. Vendor lock-in became evident when attempting to get a contact list out of the AOL email software — no functionality is provided for this service. To solve this, I wrote a BASH script in Linux to parse an HTML copy of the address book, and converted that data to a CSV that could be imported into g-mail. Interestingly, AOL does provide a way to “import” contacts into their email software!

So, there you have it. My short list of things in this world that need fixing. They all could be fixed by focusing on the customer when writing software.

Email filtering is an important tool for safeguarding your email address against spam and viruses. Introduction Consider the following statistics from a previous Wire issue: Last year, the average email account received 2,200 Spam emails. This year, you can expect that number to increase by 63% Email filtering is an important tool for safeguarding your [...]]]>

Email filtering is an important tool for safeguarding your email address against spam and viruses.

• Last year, the average email account received 2,200 Spam emails.
• This year, you can expect that number to increase by 63%

Email filtering is an important tool for safeguarding your email address against this onslaught.

A filtered email address is one that accepts mail only if it passes certain tests. Filtering systems can automatically block most viruses and, more notably, irksome spam that would otherwise flood your inbox. Unfortunately they can’t filter out everything, but they help in significantly decreasing the amount of dangerous viruses and spam sent to your inbox.

Special software filters and organizes incoming messages by applying various tests to the emails, such as word tests, blacklist tests, and html validation. The filter can also be configured to detect email addresses or to detect images, expressions, and keywords within the actual message. Based on the results of the tests, the filter then allows a message through to the inbox or redirects it elsewhere.

Messages that fail the tests are sent to a “quarantine” area, where you can periodically review the contents to be sure there aren’t any false positives—legitimate emails flagged as spam. You can then choose to either deliver or permanently quarantine the messages.

Tip for avoiding false positives: Add important email addresses to your address book or safe list, and be sure to periodically check quarantined messages for false positives.

Filtering software is primarily helpful because it removes spam and virus-infected emails. However, you can also use email filtering to prioritize and sort messages by placing certain conditions on incoming messages. For example, you could set up your email so that all incoming messages with a certain sender or subject matter are automatically filtered to a specified mail folder.

You should have a couple levels of protection:

• Your email host should provide a virus filter.
• Most email programs include some degree of filtering capability.
• Computer anti-virus software should protect your inbox.

The bad news: You can expect an over 60% Spam increase this year.The good news: Using these Worthwhile tips, you can take a bite out of Spam in 2007. Introduction The bad news: You can expect an over 60% Spam increase this year. The good news: Using these Worthwhile tips, you can take a bite [...]]]>

The bad news: You can expect an over 60% Spam increase this year.The good news: Using these Worthwhile tips, you can take a bite out of Spam in 2007.

Introduction

The good news: Using these Worthwhile tips, you can take a bite out of Spam in 2007 and increase internet productivity on both the personal and corporate levels.

• An estimated 85 billion Spam e-mails were sent daily.
• The average user received 2,200 Spam e-mails.
• Spam accounted for 90% of all e-mail and over 80% of all corporate e-mail.

• In 2004, the California legislature found that spam cost United States organizations alone more than $10 billion.
• Last year, Spam cost all Internet users $255 million.

(1.) Protect your computer. If you have an Internet connection such as cable or DSL, have a properly functioning firewall, as well as up-to-date virus and spyware protection software.

• Don’t post your e-mail address in plain text on a web page. Instead, use a form to get people to contact you.
• Don’t give your e-mail address to organizations you don’t want spreading your address.
• Create an e-mail address for newsletter signups, etc., and only use your primary address for important contacts.
• Set up a temporary address for purchasing things online. A forward works nicely for this. For example, buystuff @ yourdomain.com could forward to you @ yourdomain.com. When you start getting spam at “buystuff,” delete it and create a new one.
• When you send out a mass e-mail, include your recipients as bcc: not as cc:
• Don’t forward spam to others.
• Set your e-mail client (Microsoft Outlook, Mozilla Thunderbird, etc.) to NOT automatically download images.

Deciding where to host your website is just as important as deciding where to live. Web hosting is similar to renting a home. Introduction Deciding where to host your website is just as important as deciding where to live. Web hosting is similar to renting a home. Instead of renting an actual home, you are [...]]]>

Deciding where to host your website is just as important as deciding where to live. Web hosting is similar to renting a home.

Introduction

Deciding where to host your website is just as important as deciding where to live. Web hosting is similar to renting a home. Instead of renting an actual home, you are renting virtual space from a Web host. Just like a landlord would take care of your home, a Web host maintains your virtual home.

What Makes a Hospitable Web Host?

Uptime: Your host provides access to your website. Choose a host with a history of solid uptime – not just one that claims it. Since you won’t sit there refreshing your web page 24/7, you’ll rarely know if it’s not operational.

Support: Great hosting hardware and software is useless without the expertise and willingness to support it. Make sure your host can do more than maintain your account. What if you want to make improvements to your software or web presence? Will your web host respond to your requests quickly?

Email: Your web host should also be able to provide your email. Make sure they have an excellent spam and virus filter. Ask to see their spam filter statistics. A good host will provide you access to your email from anywhere via webmail.

Security: A good host will proactively identify and protect against threats. Find out what your host is doing to stay alert to possible vulnerabilities. What happens when there are security breaches? Ask about their track record and see if they’ve been hit by the same type of attack twice. Security vulnerabilities should only be exploited once!

Want to learn more? Read previous Worthwhile Wire issues.