Who tests the testers?

Got an interesting situation at my current contract. The application I'm working on sources data via various ETL (extract, transform and load) processes, via various datamarts, and into our application database.

In the application, there are various calculations that drive off the data in the database. Some of these calculations can get rather hairy. As a tester, we have to prove the calculations on the UI are correct, based on what's in the database (as we can't go any further back to enter/source data), so we write rather large, ugly SQL scripts, that possibly make incorrect assumptions, or may be just plain wrong.

As the test lead, I need to come up with a method to minimise mistakes made by the testers, so we're not creating too much 'noise' in the defect list. I have some possible solutions, none of which are perfect, but should suffice in the short term. It will be interesting too see if the development team have made the same assumptions or conclusions re how to derive the answers, based on the same Use Cases as we have.

I guess this is a bit of a questions of 'who tests the testers' - how far down the line of testing the test scripts do we want to go? Should it be a case of the project team just biting the bullet, and sucking up the extra analysis time needed from all involved on the first few times these queries are run, to get them correct based on the requirements documentation?

Image courtesy of post406's photostream (creative commons licence)

Named and Shamed...

Today is British Gas - aspx stack traces in their Central Heating insurance app.

A number of wtf's here:

1. The most obvious - their app is broked :)
2. Should have a custom 500 error page instead of a nasty stack trace
3. They are using a crazy mix of cgi, jsp, java and aspx (check out the url on the home page: http://www.house.co.uk/cgi-bin/house/house/jsp/app/viewHomePageAction?BV_SessionID...)

Their site is back up again now. But then the layout on the 'HomeCare' page is a bit of a shocker... has a semi-liquid borders which resize the main part of the page to the browser width, but then static width content inside that, so gets chopped off.

I could go on for years finding faults in major companies websites :) And I probably will!

CruiseControl + the FIT/FITNesse Framework

Where I'm working, we're using FitNesse on some of the in-house java development projects to run automated 'business user acceptance testing'. I'm still not 100% convinced of the value of FitNesse - it takes a lot of setup, and anytime the archictecture of the underlying systems changes, the java fixtures need to be rewritten to take this into account. This means only test analysts who are also Java devs can modify the 'framework'...

I'm currently trying to get FitNesse and CruiseControl integrated - as the development team (supposedly) use CruiseControl as their continuous integration solution, and Fit/FitNesse currently lacks a lot of the reporting and scheduling features I'ld like to see. So we get some 'free' reporting/scheduling tools, and in the future can integrate the Fit Tests with the current build process as an extra layer of tests.

So the current solution is a lot of hacking of XML/XSLT stylesheets - I don't have a JDK installed on my PC yet, so can't get CruiseControl up and running, but I've got the FitNesse end running. There is a command line runner that can recursively run tests generated in spreadsheets, and spit out xml or html results, which some nice people have posted their xslt to turn into stuff that CruiseControl can display nicely :) Not sure if the command line test runner can run the FitNesse wiki pages or not...

For invoking builds on the fly, the users will have to use the CruiseControl JMX Control Panel, which is pretty ugly and rather hard to find... I might just hardcode a html page which invokes the build myself, or see if I can put a link up on the CruiseControl admin page.

Fingers crossed! I'll post my results here in case anyone ever wants to give it a try themselves :)
Would be great if there was a CruiseControl plugin for FitNesse all ready to go...

I don't get it...

Came across this summary of an article on StickyMinds today:

EBay's PayPal unit said a number of users had problems transferring money from their PayPal accounts to their bank accounts on Sunday and Monday. Amanda Pires, a PayPal spokeswoman, said a dedicated team was still trying to solve the problem on Monday afternoon. Pires said eBay was not sure what caused the glitch. "We are focused on fixing it, not diagnosing it," she said.

I really don't get that quote from the ebay spokesperson --> how can they fix an issue with their software, without diagnosing what the problem is? Sounds like a 'hack and slash' software development methodology - 'lets just mess around until it works like it should', rather than finding out what the problem actually is. I hope that quote was taken totally out of context :). And remind me never to be a spokesperson for any company!

http://www.stickyminds.com/s.asp?F=S11891_NEWS_6

New JIRA beta out

Oooh wish I was running JIRA here... keen to try out the new beta. Might
have to download it at home and put it on me laptop.

One feature that looks nice is the SVN/CVS commit functionality now can
force a 1-to-1 relationship between SVN/CVS checkins and JIRA issue
numbers in comments (The 'Commit Acceptance Plugin'). So JIRA can reject
changes if they don't contain JIRA issue refs via a pre-commit hook. Nice!
Great if you have developers who are guilty of sneakin in bug fixes and
changes without telling anyone :-) Tho if they are really sneaky, can
still sneak changes against other bugs/changes.

http://confluence.atlassian.com/display/JIRA/JIRA+3.7+Beta+2+Release+Notes

Kaizen

'Kaizen (Japanese for "change for the better" or "improvement", the
English translation is "continuous improvement", or "continual
improvement) is an approach to productivity improvement originating in
applications of the work of American experts such as Frederick Winslow
Taylor, Frank Bunker Gilbreth, Walter Shewhart, W. Edwards Deming and of
the War Department's Training Within Industry program by Japanese
manufacturers after World War II. The development of Kaizen went
hand-in-hand with that of quality control circles, but it was not limited
to quality assurance.. [snip] A closer definition of the Japanese usage of
Kaizen is "to take it apart and put back together in a better way." What
is taken apart is usually a process, system, product, or service.'

http://en.wikipedia.org/wiki/Kaizen

Just came across this term somewhere (can't remember where). Sounds like
what I like doing - tearing down processes/systems (whether it be
applications, protocols, or the general 'culture' of how things are done)
and putting faster, cleaner ones in. Might have to do some reading up on
this.

Test Management in JIRA?

From the i'm-frustrated-with-excel-spreadsheets-to-manage-testing department.

The lack of reasonably priced, extendable and usable (big feature for me!) Test Management tools is starting to annoy me... Test Director/Quality Center is too expensive for most places I've worked (you'ld think an Investment bank would be able to afford it but no! They are looking at some weird thing I've never heard of, so could be badly supported yikes!), and requires too much training/maintainence for a 'looseish' UAT process like we have here (ie there is a process, but too many people push outside it due to time restrictions and lack of testing knowledge). There are cheap/free alternatives around that I've tried, but they all seem lacking in lots of functionality, or just plain unsuable and inflexible.

I think JIRA (change/defect management tool) could probably be extended to include a Test management function --> if it had that, it would do everything that Test Director/Quality Center does excluding the linkages with automated testing (ability to schedule and fire off tests created with winrunner/loadrunner/QTP/Silk(?) and probably others), and would really help with the whole 'integrated' change management Suite if you don't care about running automated tests from the UI.

For sites that don't use automated testing, or automation platforms that don't play nice with Test Management apps, it could then provide a low(ish) cost 'total' change management system that hooks into scm also... mmmm dream on Rach!

A Test Case is not so different from the usual workflows - but it usually doesn't get 'closed' in the same manner as bugs/changes, and need to jump around in history against versions of test cycles (ie for regression testing). Then store results of testing and easily raise bugs/changes (mmm) on failures etc. From there, it's not a huge step to add requirements mangement also - as the linkages/relationships between requirements, Tests and Changes (defects etc) aren't all that different. Drop some reports into a dashboard specifically for Testing/Requirements, and bob's your uncle.

One day I'll do myself a el-cheapo Test & Requirements Management app (in Access if I'm desperate, or Java/JSP or something - might as well try to extend JIRA if I'm going that far, and if Atlassian still supply source code for your own tweaks :)) so I can work out the workflow/requirements of a good app myself, as I've never quite found one that does everything I want. Hmmmm wonder if Atlassian are lookin for a contract User Interface Designer/Business Analyst type person workin in London at London rates!

That's todays <rant/> for ya :-)

I heart JIRA

I am so missing JIRA, the issue management tool I implemented and championed at Virgin Travelstore a few years ago. I miss lots of things:

• Ability to schedule fixes into releases
• Sensible workflows
• Built in graphs/reports that work
• Assigning tasks to groups rather than people
• rss feeds (!)
• Exporting to csv(excel), pdf, can't remember other formats
• Usability! So easy to use compared to some other tools
• and lots more stuff that I can't even remember right now....

Haven't found another tool that works better yet for change management. Stuck with MKS currenlty (yeah I'ld never heard of it before!) and ClearQuest in my previous contract. Tried Test Director/Quality Center, but that still doesn't seem so powerful.

Fit/Fitnesse

Fit/Fitnesse (http://fitnesse.org/) has been adopted by some projects at my current job for 'automation' testing in the UAT process.

Now I can see automated testing is needed in UAT, esp useful for regression testing. However the Fit/Fitnesse framework only works for java based apps, and hooks in at the class level. A big problem with it, is it requires java developers to write the tests.... so there's stumbling block number one -- business users aren't java developers - so either grad students (who work a few days a month if we're lucky), or external consultancies are used to build these tests. How they are maintained, I have no idea!

IMHO (and in most of the software testing world!), UAT should be testing end-to-end business processes, not messin about with the java code. When using an automated tool, it should interact with the UI, as this is where the end user interacts with the software. Often a business process will jump into various different apps, or even paper/offline processes.

To me, this seems too much like low level testing that is done at unit/integration or system test phases. Note the 'U' in UAT stands for 'User'. If it's not simulating user interaction, it ain't a UAT test.

Random software development

So we keep getting new versions of code to run in UAT. Only problem is -- no release notes. Nadda. Nothing. Just a list of code modules that need updating, which means nothing to me!

Who has a development contract that doesnt include the supplier sending release notes? Or at least a napkin with some scribbed bug numbers on it that's been left on a pub table for much too long..? It means UAT testing (which is the only testing actually carried out... don't get me started on that...) has to have a random guess at what might or might not be fixed, and no idea on any impact on the rest of the system. So bye bye to any regression testing. Rather frustrating for me attempting to manage the UAT process, no?

[Edit] Turns out I can't find requirements to match half the bugs logged. Hmm, so there's issues from both ends at not supplying what's needed for a good development project.. Requirements and release notes.

Issue reporting styles in the real(?) world

My first real <rant/>...

So I've come across two major schools of thought when it comes to issue reporting...
School 1: for every bug/change/improvement, log a seperate issue
School 2: for related bugs/changes/improvements, log a single issue that contains all the related issues

My personal preference is for School 1. Why?

1. Bug Maintainence Time
Logging a issue with lots of issues in it is faster for administration - easier to close one bug, and resolve one bug to a developer.
Winner: School 2

2. Reporting
Logging seperate issues for each issue obviously means your reports will be accurate. If parts of the issue are fixed and not others in a version, its impossible to accuratley tally up the opened/closed counts against a version, or a person (ie bugs assigned per developer, bugs closed per developer etc). And even worse if the issue affects multiple 'components' or projects, then can't tell which areas have the most problems. Some systems enable time to be logged against tasks - estimates and actual time, and for future learning/planning, each problem/issue should have time logged seperatley allowing for better future planning. I've also come across sites where multiple issues that aren't even particularaly related are logged in a single issue (yeah, maybe they all happened during installation, but really some were installation bugs, others totally unrelated to installation, but a build issue instead).
Winner: School 1

3. Accuracy
Often if there's a large list of issues in one issue, some can get missed out (at spec, dev or test phase)
Winner: School 1

4. Parallel workflow
Assigning an issue containing lots of issues to one person means it wont show up in anyone elses work list until the first person is done/reassigned the issue. Seperating out each issue into it's own report means they can be worked on in parallel where possible, and not have to be passed from one to another.
Winner: School 1

Tools like JIRA are able to have 'sub tasks' - many small tasks logged seperatley, but grouped under a large task. Ie A group task called 'fix layout headers in all apps' - with a small subtasks for each application. This allows each subtask to be completed, reported etc seperatley, and also easily see the progress on the entire group of issues as a whole. Also tools that allow bulk editing make life easier for managers/developers etc to update multiple issues at a time, removing a lot of the administration time required.

Of course there needs to be some flexibility (ie if there are 5 spelling mistakes on a web page, then drop those into one issue, but spelling mistakes in different parts of the website need to be seperated out), but for the sake of simplicity and accuracy [speaking as someone who's been responsible for reporting and managing a change request system], logging seperate issues for each change is the way I would go. For a few extra minutes a day of administration time by your developers/managers, you'll be rewarded with a more accurate and realistic view of what's really going on in your change management process, and the overall quality of your products, assuming quality is something that is considered important in your overall product of course!

Another thing that's been bugging me is Issue ownership. I've worked at some sites where the Test Manager isn't particulary involved with issues after they've been logged. My philosophy is that the Test team are responsible for saying yay or nay to whether they consider a release is *OK*, so they own those issues raised, and should be involved with any planning around getting the issues fixed, changed, postponed, deleted etc. There are also sites where any old person (support, marketing, sales and other non-testers) have access to the issue/change management tools, so can raise issue, but then aren't considered responsible for ensuring those issues are dealt with, or confirming they are correctly resolved. If a bug is raised by a non-test team member, (i.e. it's usually been found outside a normal test cycle) it's my belief that someone from that team should be ultimatley responsible for verifying/confirming that issue is resolved to their satisfaction.