I spent most of today rerunning the same tests over and over and over and over – trying to get a fix through where each round of fixes broke something new. No, this wasn’t the developer’s fault. He’s dealing with a horribly “interesting” codebase with even more “interesting” legacy implementation choices.
Take a large web application with multiple customers. Users can be linked to one or more customers. The user has the login but they can’t to anything without picking which customer they’re working with. Naturally, there’s a linking table that connects users with customers. So far so good.
When you’re doing stuff with users, you can edit ones who are already set up with the current customers, you can create completely new users (who will link to the current customer) or you can link an existing user to the current customer. Still not so hard, right?
Right… Now, add the user identifier for the Evil Third Party Interface. Which must be linked to both the user and the customer. So, it gets added to the linking table because that’s the logical place for it. And now the fun begins. Because the person who is logged in can also have the identifier for the Evil Third Party Interface, and who gets to see what is controlled by privilege, to make sure this one new thing did what it was supposed to, I needed to check all sorts of things:
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something else in my user record.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change my Evil Third Party thingy.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who has the Evil Third Party thingy.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who doesn’t have the Evil Third Party thingy.
- and so one…
There ended up being rather a lot of these lovely combinatorial options just to check the addition of one simple field. The poor developer was as frazzled as I was by the time we finally got them all doing what they were supposed to do.
And this, Ye Who Do Not Test, is a pretty simple situation. Believe me, I’ve seen worse. I live worse. The last time someone asked me why I (the lone tester for a team of 10) don’t do much regression testing, I showed them my half-finished regression guideline wiki. Which is now somewhere in the order of 300 pages, all of them with multiple cross-links because one small change here can affect things in half a dozen other places, and the validation rules need to be seen to be believed.
Brainless checking is easy. Any monkey can do it. Testing? That’s hard. Figuring out the combinations in the first place is hard enough. Working out which ones you need is harder.
9 thoughts on “Why Testing is Hard”
Testing isn’t hard, fixing the crap that won’t pass the test is hard. True for hardware, software, or wetware
Oh, that part’s even harder. But figuring out what to test is a challenge all by itself.
Test everything, the one thing you don’t test WILL fail
Even if it has been rock solid for years
It’s almost too bad that you can’t test by having an idiot try and do something under careful supervision. Somehow I think most users would object to being referred to as idiots.
You have specs for what should happen in each case? I’ve seldom seen that.
The trick is figuring out how it will be broken. And then explaining when some random idiot comes up with a completely new way to break it.
They need to hire me to try to run one of their programs in a testing setting. Trust me – the gods of Error will show up to give autographs and Kate can catch them in action! LOL
Bruce Miller – we have something that kind of passes for specs. If you squint just the right way.