Clark,
Understood, but let's look closer at regression testing and test platforms. If we start with the test platforms and developed a list of all the hardware they would need to test a code revision on, how big is the list?
The list will easily number into the hundreds of thousands of computers and devices. But the real rub here is in the configuration of those computers and devices. Obviously these devices will be running absolutely 'clean' installs of the BIOS and operating systems (which are reset every time a new test is done).
No one in the world could possibly run regression testing on the billions and billions of 'combinations' of BIOS, operating systems, drivers, updates, and applications that actually exist and will be exercised when
ebay releases a code revision.
But let's consider just one computer model, say a Dell XPS 13. We would need a number of them since Dell made running changes in the hardware over the life of the XPS 13. I do not know the exact number but let's conservatively call it 10.
Moving on to the operating systems, in the field there are users who will be running all kinds operating systems (Windows, Linux, etc) and versions. But let's ignore all of them and just drill down on the latest release of Windows (Win10). There are currently seven versions; Windows 10 (1809) 10.0.17763, Windows 10 (1803) 10.0.17134, Windows 10 (1709) 10.0.16299, Windows 10 (1703) 10.0.15063, Windows 10 (1607) 10.0.14393, Windows 10 (1511) 10.0.10586, Windows 10 10.0.10240.
Now lets move on to browsers; there is a huge number of browsers but let's just use 5 of them; Chrome, IE, Firefox, Edge, Opera. Which version of these 5 browsers should we test against? IE 11 alone has 3 major releases; 11.0.10240.16384, 11.0.9600.18617, 11.0.9600.16384 so lets greatly restrict the browser testing to the three most recent builds of each browser (a total of 15).
So to summarize the testing plan for just the Dell XPS computer;
- 10 different XPS
- 7 different builds of Win10
- 15 different browsers
We will ignore all the possible combinations of OS hotfixes, updates and any impact that the installation of any of the thousands of application that will be found in the field. Just using the list above, there are over 1000 combinations. The next question is how long does it take to run the test? Computer has to boot up, log into the operating system, start the browser, test the new
ebay code, log the results. Assuming a test time of 2 minutes, that is a total of 33 hours of testing for just the Dell XPS 13. Of course some of the testing could be concurrent, but it would take a full day for a company to test just this one computer model.
The above is not how large companies test for obvious reasons, the testing cycle is highly automated and the only feasible way to regression test is to use 'representative' platforms. This confluence of hardware and software and the overwhelming huge number of combinations in the field it represents, guarantees that users will find problems that testing has 'missed'. So it is true to say that testing is inadequate but in my opinion 100% regression testing is impossible. Software testing is an incredibly complex and costly thing for all companies. At one time, Microsoft employed ten-thousand testers! In one MS meeting Bill Gates said, "I know I pay a lot of testers, but I really have no idea what they do".
So is the beef with
ebay or the fact that comprehensive testing is impossible given the computing diversity and permutations?
When a company is faced with this kind of testing problem, they do the 'best they can'. They certainly will not be able to help an angry user running Windows XP with 15 years of accumulated old applications and malware. Yet there are a number of posts on this and other forums where this is exactly the case, users blaming
ebay or other software companies for the issues they are seeing on their old crusty XP machines.
I sold my
ebay stock a while back, I do not like defending
ebay. I am highly critical of some of the things they do; for example I have been beating the drum about their badly flawed feedback system for years. But my opinion is that it is naïve to think that
ebay has some employee or manager just implement a new idea/code without significant marketing analysis, cost justification, and testing. Large companies ALWAYS group code revisions into larger releases for reasons that should be obvious to anyone (as outlined above). Yet I I have seen posts where people think that some coder has simply changed something because he/she thought it a good idea. I would agree that there are some code changes which get fast-tracked. When country regulations or legal conditions change there are times when a company like
ebay has to short-cut their own standard development procedures. But these kinds of situations are not by choice for a company.
My intent is to simply give some balance to the discussion. While it may be popular to hate on
ebay and other software companies, there are many aspects to technology development that may not be understood by stamp collectors. I welcome folks who offer their career and business experiences to our discussions whether it is printing, accounting, agriculture, retail, manufacturing, etc. For myself, I enjoy gardening but would not consider my opinion to be as significant as a person who farmed for a living. Hopefully offering a bit of the 'other side of the coin' on technology development helps balance this topic.
Don