I spent most of today rerunning the same tests over and over and over and over – trying to get a fix through where each round of fixes broke something new. No, this wasn’t the developer’s fault. He’s dealing with a horribly “interesting” codebase with even more “interesting” legacy implementation choices.

Take a large web application with multiple customers. Users can be linked to one or more customers. The user has the login but they can’t to anything without picking which customer they’re working with. Naturally, there’s a linking table that connects users with customers. So far so good.

When you’re doing stuff with users, you can edit ones who are already set up with the current customers, you can create completely new users (who will link to the current customer) or you can link an existing user to the current customer. Still not so hard, right?

Right… Now, add the user identifier for the Evil Third Party Interface. Which must be linked to both the user and the customer. So, it gets added to the linking table because that’s the logical place for it. And now the fun begins. Because the person who is logged in can also have the identifier for the Evil Third Party Interface, and who gets to see what is controlled by privilege, to make sure this one new thing did what it was supposed to, I needed to check all sorts of things:

• I’ve got the Evil Third Party thingy, I’m a god user, and I change something else in my user record.
• I’ve got the Evil Third Party thingy, I’m a god user, and I change my Evil Third Party thingy.
• I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who has the Evil Third Party thingy.
• I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who doesn’t have the Evil Third Party thingy.
• and so one…

There ended up being rather a lot of these lovely combinatorial options just to check the addition of one simple field. The poor developer was as frazzled as I was by the time we finally got them all doing what they were supposed to do.

And this, Ye Who Do Not Test, is a pretty simple situation. Believe me, I’ve seen worse. I live worse. The last time someone asked me why I (the lone tester for a team of 10) don’t do much regression testing, I showed them my half-finished regression guideline wiki. Which is now somewhere in the order of 300 pages, all of them with multiple cross-links because one small change here can affect things in half a dozen other places, and the validation rules need to be seen to be believed.

Brainless checking is easy. Any monkey can do it. Testing? That’s hard. Figuring out the combinations in the first place is hard enough. Working out which ones you need is harder.