This afternoon was a classic example of missing the point of having a staging server. You know, the one that is mostly a mirror of the production environment which, being one of those web applications that thousands of customers consider mission critical, isn’t ever allowed to be down. The one that is supposed to have code and database changes intended for release migrated to it a week in advance so that any problems with integrating the several hundred bajillion lines of interesting code that have evolved over the years into a tentacly mess to rival Cthulhu himself can be found well before the targeted deployment date so that pushing changes to the live servers goes as smoothly as humanly possible.
What actually happened was this: integration started after lunch today. For a deployment tomorrow at oh god am. It finished shortly before I usually leave. So instead of several days to test the targeted items and maybe do some regression in the critical areas (trust me, full regression is the kind of dream that makes winning the lottery look like something that happens to you on a daily basis) I have in “official” work time maybe fifteen minutes. Naturally, I stayed until my balance started to go, and the developers were still arguing with the Wonderful Source Control that doesn’t understand the difference between a change and a file with a change so dragged in a bunch of other changes that aren’t going out at oh god am tomorrow morning.
Ladies, gentlemen, and others, the point of having a staging server is not so you can have two nervous breakdowns for each deployment instead of just the one when your integration crashes and burns on the mission critical live environment. The point is so you can do this strange thing called planning and have everything ready and tested against something that was just like the live environment until you did the integration. So you know (within reason – this is a black art we’re dealing with) that you won’t have any serious issues when you do deploy.
Oh, and when your one and only tester is narcoleptic? It’s a really, really bad idea to leave it all until the evening before you deploy. Just saying.
I spent most of today rerunning the same tests over and over and over and over – trying to get a fix through where each round of fixes broke something new. No, this wasn’t the developer’s fault. He’s dealing with a horribly “interesting” codebase with even more “interesting” legacy implementation choices.
Take a large web application with multiple customers. Users can be linked to one or more customers. The user has the login but they can’t to anything without picking which customer they’re working with. Naturally, there’s a linking table that connects users with customers. So far so good.
When you’re doing stuff with users, you can edit ones who are already set up with the current customers, you can create completely new users (who will link to the current customer) or you can link an existing user to the current customer. Still not so hard, right?
Right… Now, add the user identifier for the Evil Third Party Interface. Which must be linked to both the user and the customer. So, it gets added to the linking table because that’s the logical place for it. And now the fun begins. Because the person who is logged in can also have the identifier for the Evil Third Party Interface, and who gets to see what is controlled by privilege, to make sure this one new thing did what it was supposed to, I needed to check all sorts of things:
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something else in my user record.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change my Evil Third Party thingy.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who has the Evil Third Party thingy.
- I’ve got the Evil Third Party thingy, I’m a god user, and I change something for someone else who doesn’t have the Evil Third Party thingy.
- and so one…
There ended up being rather a lot of these lovely combinatorial options just to check the addition of one simple field. The poor developer was as frazzled as I was by the time we finally got them all doing what they were supposed to do.
And this, Ye Who Do Not Test, is a pretty simple situation. Believe me, I’ve seen worse. I live worse. The last time someone asked me why I (the lone tester for a team of 10) don’t do much regression testing, I showed them my half-finished regression guideline wiki. Which is now somewhere in the order of 300 pages, all of them with multiple cross-links because one small change here can affect things in half a dozen other places, and the validation rules need to be seen to be believed.
Brainless checking is easy. Any monkey can do it. Testing? That’s hard. Figuring out the combinations in the first place is hard enough. Working out which ones you need is harder.
There’s no way around this. Any software project where you have to interface with Someone Else’s Software you will run into untold suffering because Third Parties are Evil. They might be the nicest people in the multiverse. They might make the most wonderful software in several different realms of existence. But the moment your software has to talk to their software they immediately transform into eldritch horrors the likes of which H.P Lovecraft gibbered about before trailing off into the dreaded ellipsis…
This micro-rant brought to you by the discovery that customers using a third party that my company’s software talks to neglected to set a crucial item to be required on the third party’s software. As a result, it wasn’t there when our software pulled their changes. Because it wasn’t there it caused errors down the way because our software requires this one little item. And of course, since shit flows downhill, guess who gets the blame. Yup. Us. This, children, is why testers are such cynical people.
It’s also why I don’t write about the things I run into in my work life. No matter how I translated them into books, nobody – but nobody would believe it.
Well, not exactly. I still need to clean things up here, but at least my theme and plugins are all updated and backed up.
Next challenge: update the static pages.
One of the first things you learn after you start work in a corporate environment is that weekends are never – ever – bloody long enough. This revelation is often followed closely by the discovery that Dilbert really is funny. It’s a remarkable dividing line. People who have worked in a corporate environment laugh at Dilbert. People who haven’t don’t. And all of them agree, weekends are never long enough.
Which is why today was phenomenally unproductive. Some things got done, but none of them are things that actually matter in any way. Well, apart from the small and rather dismal amount of writing that happened.
Tomorrow I need to finish updating the site – the theme update needs to happen and isn’t simple because this particular theme has a nasty habit of overwriting the header images with its defaults. I still prefer it because it lets me set things up with the layout I like, but upgrading the thing is a pain. I also need to write a lengthy rant for my next guest post on According To Hoyt and do a few other bits of general maintenance around the house and computer. I should have done some of this today but somehow, it just didn’t happen.
Not that this is surprising or anything. I was falling over at work so I left early (thanks to yesterday’s uber-long day I only needed to be at work for 5 minutes to clock up a 40 hour week. I was there a bit longer than that), then slept for an hour or so when I got home. I’d probably have slept longer except the Bugger-cat was determined to have his good food and he wasn’t letting anything stop him.
Yes, I lead such an exciting life.
I go to work early. Partly because I start to fade around 2 or 3 even when I sleep disgustingly late, and it makes sense to get in as much that needs thinking and a functioning brain before my body decides “hell no”. Unfortunately that means if I’ve got to stay late – like today when a release went out after 5pm (gotta love live web stuff. Can’t take it down when someone will be using it. Even if the “someone” is the company’s internal people who could work around things so long as we did it in a quiet time) so I didn’t get home until nearly 7. Yay for 12 hour workdays. Not.
It’s also my Mad Genius day. This means I write a post a day or two before and set it up to post early in the morning. When I get home in the evening I respond to comments. Today’s effort was a bit of an expansion of my thoughts on three of the panels I was on at Ravencon. Next week I’ll probably ramble about one of the other panels I did, unless someone in the writing world does something spectacularly stupid that I can rant about.
And on that note, it’s time I went to bed.
Quiet day at work today – I spent a good chunk of it decommissioning my collection of Windows XP virtual machines and setting up a series of Windows 7 virtuals to replace them.
Why do I need so many, I hear you ask? Well, if you want to test a website properly, you need multiple versions of Internet Explorer, and that means multiple systems. So now I have a system with IE 8 (third largest slice of the employer’s customer base), one with IE 9, and one with IE 10. My machine is now on IE 11 (largest slice of the customer base), and also runs Firefox and Chrome. Then there’s the Macbook that lives on my desk to test Safari. Chrome/Safari is the second biggest slice of the customer base, so Chrome and Safari need to get tested as well.
It’s quite remarkable how many bugs are specific to one or two browsers. Even more fun are the “bugs” which are actually the browser working as designed, but our software exploits a fault in the IE family…
Oh, yes. Browser compatibility testing is fun. I am so grateful I don’t have to test for IE 7 compatibility as well. And I am terrified by the users who are still running with Windows 2000 and IE 6. If they ever report a bug they’re probably in for a very nasty shock.
Also known as “Fun with testing software, episode the infinity-somethingth”.
Today at work the helpdesk calls were coming in fast, all from the customers using the venerable (but stable and more importantly usually working) PC payroll program, complaining that their new FTP passwords (they’re reset periodically for security reasons) to our systems weren’t being accepted. The plot thickened with the discovery mid-morning that pasting the new passwords worked just fine. Keying them failed. By late afternoon, the truth was discovered (and with it, the list of who had not moved over to the newer fully-encrypted FTP module)… The venerable software, written to work with an equally venerable mainframe system that does everything in all caps naturally uppercased everything that went in. Including – ta-da! – the FTP password. Which is – of course – masked, so users couldn’t see that it was busily uppercasing itself.
Such a little thing, and such angst it caused. Yes, now everyone in the programming and test team knows why our passwords were always in all caps in the past.
Lesson learned: do not neglect the little things, for they shall trip you when you least expect it and you shall spend much time going “WTF?” before you figure it out.
The FTP problem is solved. For reasons known only to itself, the software that runs the backend of my hosting provider decided it would cache the error message. Once I cleared cache, I reset my password and all is happy.
This brought about a rather unwelcome realization. It is not the case that there are some software companies whose secret (or not so secret) goal is to tear the fabric of reality asunder and summon the Great Old Ones. This is in fact the stated or unstated goal of all software development. And I test it for a living – complete with the unstated goal that the software does not in fact warp reality to the point that Cthuloid tentacles curl around the straining edges of the space time continuum-thingy. Aside from anything else, it takes a boatload of LOX, several tons of brimstone, and a spit that would put skyscrapers to shame to properly cook the things, and then you need to find the brave souls willing to actually eat several tons of Cthulhumari.
No, much better to keep the gate between dimensions firmly locked. Padlocked. With extra thick chain and bonus spiky bits.
Now you know why software testers have a kind of graveyard humor about them. We only hope we’ll get to enjoy one when the time comes.