programming management style term

[JeenLeen]

What is the term for when you have two programmers independently generate the same file, then compare the results to make they are identical, in order to validate the file?

I'm thinking of something similar to 'double data entry', but for writing a program to generate a file instead of just data entry.

EDIT: as noted in my reply further down, I think it might be called "dual processing".

[halfeye]

Unnecessary? Unhelpful? Wasteful?

If you intend to compare the code, it's almost certainly going to be different even if it's trivial. Comparing the results is fine, but how many edge cases must you test to know which version is better?

Pair programming where one guy keys and the other watches is probably better than this.

[tyckspoon]

You might be thinking of a hash or checksum comparison, where a kind of software fingerprint is generated from a particular piece of data - if the hashes of the file match, you can be reasonably certain that the two files are the same (it's not 100%, but it's close enough to be valid for testing, especially if you have control of the source files you're checking.) Wouldn't expect it to work for having people manually generate information, but it can be used as a way to make sure two different copies of a program are working as expected - you have a verified copy of the program generate the information, have the copy you want to test do the same thing with the same initial settings, and compare to confirm that the outputs match.

[Florian]

What is the term for when you have two programmers independently generate the same file, then compare the results to make they are identical, in order to validate the file?

I'm thinking of something similar to 'double data entry', but for writing a program to generate a file instead of just data entry.

Sorry, your question doesn't really make sense.

Comparing the output file is trivial. When the programmers have strict specifications how the output file should look like and be structured, than methods like comparing hash or checksum will lead to the same result.

That doesn't help when it comes to the code used in the actual program that will generate the output files. Modern programming languages are flexible enough that you can use multiple approaches and formats, as well as formatting and still get to the same end result.

[JeenLeen]

I think I must have mis-wrote.
I did not mean to talk about comparing the code. What I meant was validating the output by comparing the output. The idea being that, if the output is identical, then you are also validating whatever method the programmer used to generate the output. (Since it's unlikely that both of them made exactly the same mistake.) Comparing the outputs via a checksum would be one method of doing that.

I know this is a thing, since I've heard one of the big insurance companies around here has their entry-level programmers do that. I'm also pretty sure I heard a more generic form of this other than the more specific "hash or checksum comparison". Is it called "dual processing"?
I have read of Pair Programming, but that's not what I'm thinking of.

[halfeye]

I think I must have mis-wrote.
I did not mean to talk about comparing the code. What I meant was validating the output by comparing the output.

I sort of read it as that, but comparing the code made more sense to me (still not much).

The idea being that, if the output is identical, then you are also validating whatever method the programmer used to generate the output. (Since it's unlikely that both of them made exactly the same mistake.) Comparing the outputs via a checksum would be one method of doing that.

What would be the point? if it's the same file every time, then why does it need to be written more than once? If it is variable, then you will almost certainly get many errors from both programs. If both programs produce lots of errors, it becomes more likely that at some stage the same error will be made by both programs, and will not be detected because both programs made it (survival of the fittest replicated bug).

I know this is a thing, since I've heard one of the big insurance companies around here has their entry-level programmers do that.

As a teaching method it might well be fine, in small doses it sounds like a great way to teach programming to complete newbies, that's an entirely different thing than using it for production of files that are going to be large and complicated and used in situations which are critical to the survival of the business.

I'm also pretty sure I heard a more generic form of this other than the more specific "hash or checksum comparison". Is it called "dual processing"?

I have read of Pair Programming, but that's not what I'm thinking of.

Dual keying data is a fine and respected tradition, and can catch a lot of errors, some will still get through, but the reduction in the number of errors is well worth the extra expense.

Programmer time is much more expensive than data-entry time, and the programs will need extensive testing apart from comparing the output. You will need to run both programs over lots of different datasets, and even then you won't be sure that there isn't a shared bug waiting to bite you somewhere in some obscure combination of factors.

[Manga Shoggoth]

They aren't exactly the scenario you describe, but:

An old program and a new program being run at the same time to produce the same output - Parallel Running (a form of testing)

Two programs producing the same output to guard against failure of a single item - Redundant Processing (more common at the hardware level than at program level, although some systems do use it)

[Florian]

I think I must have mis-wrote.
I did not mean to talk about comparing the code. What I meant was validating the output by comparing the output. The idea being that, if the output is identical, then you are also validating whatever method the programmer used to generate the output. (Since it's unlikely that both of them made exactly the same mistake.) Comparing the outputs via a checksum would be one method of doing that.

I know this is a thing, since I've heard one of the big insurance companies around here has their entry-level programmers do that. I'm also pretty sure I heard a more generic form of this other than the more specific "hash or checksum comparison". Is it called "dual processing"?
I have read of Pair Programming, but that's not what I'm thinking of.

Sorry. I've spent quite some time as a freelancer in the ABAP/SAP environment, which is a rather complex environment and one of the few cases that require very strict coding. Output is output. You get it based on given specs no matter the initial code or performance of the code.

In short: This is bogus.

[veti]

An old program and a new program being run at the same time to produce the same output - Parallel Running (a form of testing)

This is what I was thinking.

But no matter what you call it, I don't see why you would want programmers to do it. Testers, that's what you need.

Programmers want the program to work. They will spend time figuring out, if necessary, what wierd levers and voodoo you have to do to make it work correctly. They may not even be conscious of it, but the result is that they give the product an easy ride in a hundred subtle ways.

Testers love to break things. They'll read the documentation and follow instructions with deliberately varying degrees of precision. If they can get an output that looks plausible but is completely wrong, they'll tell you about it. The programmer would probably figure out what they did wrong and try again, without ever mentioning it.

[DavidSh]

Wikipedia has an article on N-version programming, which looks like this. The idea is patterned after hardware redundancy systems, but replacing physical failures with software failures.

[halfeye]

Wikipedia has an article on N-version programming, which looks like this. The idea is patterned after hardware redundancy systems, but replacing physical failures with software failures.

That looks about what I'd expect, particularly:

In 1986, Knight & Leveson conducted an experiment to evaluate the assumption of independence in NVP, they found that the assumption of independence of failures in N-version programs failed statistically.[4][5][6]

If you are debugging by comparing the outputs, the bugs that are duplicated across programs are selected for survival, despite being bugs.

[JeenLeen]

I talked with some folk who used to work at that insurance company, and apparently they just called it "Quality Control". I did get some details in that it was one person was the main person assigned and the other a checker, but it was basically both doing the work independently and then comparing counts.

Wikipedia has an article on N-version programming, which looks like this. The idea is patterned after hardware redundancy systems, but replacing physical failures with software failures.

That does describe it pretty well, from what I can tell. Thanks.

If you are debugging by comparing the outputs, the bugs that are duplicated across programs are selected for survival, despite being bugs.

That is true. I admit a danger of this method is that it is possible both people make the same mistake, and this probably limits it to mistakes that are really hard to detect.

I don't think this idea is a great one. It does seem helpful, if you have the resources to have programmers work redundantly (which my place definitely doesn't as a standard policy, but some do), to decrease the amount of errors. I found I had to start doing this with the programmers I manage since one of them is basically incompetent and I can't trust his work--it's wrong about 50% of the time--yet due to bureaucracy we can't fire him or move him to another project. So I run the same data he runs (minus some formatting stuff that's time-consuming and another system automatically checks) and make sure we have the same counts. If I weren't double-checking him, we'd be submitting a lot of bad data and it's possible nobody would notice. But that's less evidence for this being a good method than evidence one should use competent programmers who check their own work. (I'm pretty sure he does no QC of his own processes & assumes everything works fine.)

[Manga Shoggoth]

This is what I was thinking.

But no matter what you call it, I don't see why you would want programmers to do it. Testers, that's what you need.

Programmers want the program to work. They will spend time figuring out, if necessary, what wierd levers and voodoo you have to do to make it work correctly. They may not even be conscious of it, but the result is that they give the product an easy ride in a hundred subtle ways.

Testers love to break things. They'll read the documentation and follow instructions with deliberately varying degrees of precision. If they can get an output that looks plausible but is completely wrong, they'll tell you about it. The programmer would probably figure out what they did wrong and try again, without ever mentioning it.

I know! The test team sits behind me...

I have also been known to do testing when it requires skills the test team lack.

[JeenLeen]

Sorry. I've spent quite some time as a freelancer in the ABAP/SAP environment, which is a rather complex environment and one of the few cases that require very strict coding. Output is output. You get it based on given specs no matter the initial code or performance of the code.

In short: This is bogus.

I apologize, for I missed your post and it seems a useful one to reply to.

The concept behind this method is in part (or maybe completely) to make sure that the given specs were actually followed accurately.

In my work, I have file specifications given out by the federal government. When someone on my team generates a file that seems accurate (e.g., the federal check system doesn't automatically bounce it as incorrectly formatted), I still check it in case they did something wrong. Sometimes they mis-processed the raw data and got the counts wrong, and sometimes they didn't exactly follow the specs and left out some category they should have reported. (We also don't have very clear specifications on how to process the raw data, so if the programmer makes some incorrect assumptions, that can also cause errors in the counts. And I'll admit sometimes it's my fault for assuming they know something or would ask if they don't know something.)

So I think that's why this can have merit.

[IthilanorStPete]

Reproducible builds and the tech surrounding those might be what you're thinking of.

[jayem]

That looks about what I'd expect, particularly:

If you are debugging by comparing the outputs, the bugs that are duplicated across programs are selected for survival, despite being bugs.

While that's kind of true. We are actually also interested in the bugs that aren't duplicated across programs.

The bugs entering the system are by definition selected for survival against whatever the small team used to find them.
The bugs leaving the system are by definition selected for survival against whatever all the small teams used to find them.

There are 4 options:
option 1,
Neither Team A or Team B make the mistake (or they correct it before submission)
In this case neither 2 Version software and 1b Version software both contain the bug.

option 2,
Team A make a mistake and Team B also make the same mistake, Team B also don't spot the mistake.
In this case the 2 Version software and 1b Version software both contain the bug.

option 3,
Team A make a mistake and Team B also make the same mistake, Team B however can spot the mistake (because many common mistakes are easy to spot, this will happen a lot).
In this case the 2 Version software contains the bug while the 1b Version software doesn't.
However when we introduce Team C,D,E this approach has diminishing returns. So for any interesting project we can redefine our new Team A as the team contain the appropriate number of verifiers and count these bugs as a special case of option 1.

option 4
Team A make a mistake and Team B don't the same mistake, Team B however wouldn't spot the mistake (because Team A's working was convincing enough to make Team B think like Team A).
In this case the 1b Version software contains the bug while the 2 Version software doesn't.

Now the question is does option 4 happen relatively frequently compared to option 2 (or 3). If it occurs a lot then we can cut a decent number of bugs out. If there are too many option 2 bugs the project is fubar anyway.

I can't read the papers (without paying). They it seem claim that option 4 doesn't occur regularly. But if they look at the bugs caught then they clearly are partially independent (I.E by definition they are option 4). And if they looked at the bugs that get through then of course these will be (much more) non-independent because the independent onces got caught by the process (Akin to the story about bombers and bullet holes).
Hopefully they either looked at both cases or managed to do some clever statistical wizardry to split a 4-version into 2 2-versions.

And if things are 'output is output', then things are a bit boring (the sort of area where I'd expect to find it useful would be in processing Cern data or simulating galaxies).
And in a sense any test-suite is kind of this for the special cases where you only run the other program on a few pieces of data, and the other program (presumably because of the selection of the data or at the cost of time) can be perfect.

[Telok]

Let me see if I understand what's going on.

1. You have a piece of software that, given a specific input produces a specific output.
2. You have a team writing a new piece of software to do the same job because the current software requires a web exposed WinNT server running a Jet database, using a WordPerfect version from the previous decade for output, and nobody has the source code any more.
3. The output of the new software must be equal to or better than the output of the old software.

Select
Case: You have a stamp or stickers that say 'Passed QC Inspector #8' or something like that
--> You're doing quality control
Case: You send it back with a note saying what's wrong and to try again, sigh heavily, and start a drinking problem
--> You're middle management
Case: You enter half an SQL statement into a text field labelled 'Last Name' and start laughing and/or sobbing when it nukes the test platform so hard that a wipe-reinstall is required.
--> You're a tester
Case: All of the above
--> You're a project manager. Double check that the client gave you the right specs and be happy that you know what the output is supposed to look like before you have to start development.

[jayem]

Let me see if I understand what's going on.

1. You have a piece of software that, given a specific input produces a specific output.
2. You have a team writing a new piece of software to do the same job because the current software requires a web exposed WinNT server running a Jet database, using a WordPerfect version from the previous decade for output, and nobody has the source code any more.
3. The output of the new software must be equal to or better than the output of the old software.

That seems backward to the situation

1. You have the software that ought to given a specific input produces a specific output. You do not know that it does
2. For interesting data you do not yet know what the output should be.

Spoiler

Show

A heavily templated report [A] goes down to [B] with ... (you can off course check the right field is in the right place against a specification. You know if it worked for Frankfurt there's nothing magical about Cologne)

For even something like count all the cases with X&Y you need to create a fullish set of data, and then evaluate the answer to create the one test case. At which point it's arguably taking longer to create and submit the fake data than to just do your count

2b. There is reason to believe that non-interesting data and unit testing won't catch the issue (or creating the unit tests will take considerable longer than the recreation)
3a. Using the initial program to validate the test case, and then the test case to validate the program is clearly flawed.
3. If you make a new piece of software (that presumably is inferior in other ways) and it gives the same output. Does this support the case that the initial program was right (and if so what are the limits to the claim).

So it's kind of like the quality control case, but for the prototype.

[JeenLeen]

Eh... I'm going to try to give a basic, real example. The above is really interesting, and I think more interesting than what I was asking about, but it seems like it's something else. Though jayem's #1 and 2 sound close, in that there is input that should yield a specific output, and we're writing a program to (we hope) generate that output.

One of our reports is counts of students in a district that fit a certain demographic. The input (our raw data) is a list of districts with counts. We have to format that in a special way (one row per district with various details) and also make a state total (by summing up the districts.)

I assign it to a guy on my team, and he generates the file. I have no idea what method he used. SAS, R, SQL Server, Microsoft Access, copying & pasting and manually doing stuff in Excel... all of those could work (not saying all are equally desirable methods).

But I know this guy sometimes messes up his reports, so I also generate the same counts. I then write a quick program to compare my counts to his. If our counts match, I assume his method was bug-free since it's unlikely we both made the same mistake. If I find something wrong, I ask him to double-check wherever I found the discrepancy, probably after double-checking my own work for a bug on my end. (I'm not actually checking formatting since the portal we submit the data into verifies format. I just need to verify the right counts are submitted.)

So something like that is what I'm talking about, except where it's not a supervisor checking their employee's work, but two employees assigned the same thing to sorta cross-check each other.

[jayem]

Eh... I'm going to try to give a basic, real example. The above is really interesting, and I think more interesting than what I was asking about, but it seems like it's something else. Though jayem's #1 and 2 sound close, in that there is input that should yield a specific output, and we're writing a program to (we hope) generate that output.

Both 1&2 kind of both need to be true.
If you have a program (procedure) you trust there's no need to check it. You might well want to check a new program against it (as in Telok's example, and it might be that the old one is the erroneous one) but the new program isn't being written to check the old one. If you have no program there's nothing to check.

If you already have (enough) validated data and output, then why are you not using that rather than re-validating it (rhet?).

Some of the other posts were I think about similar things, including the ones identifying the basic problems.
That is it definitely sounds like a mild version of N-versioning.
That the errors are possibly less likely to be independent than you think (although we really need someone to read the paper)
That if they can follow clear specs closely, then you really shouldn't have independent errors. (or in other words if the specs are sufficiently controlling and shared then you aren't really N-versioning the programming just the typing)*
That better experience at knowing what to test for is more likely to find things (in my opinion a slightly different set of things)

* Which (a) is potentially a problem (for you) and (b) the other 'solutions' may be worse than the problem. You don't actually want a program, you want the output. The program is just the means to the end. The expected XKCD comics

[Florian]

@Jeenleen:

It rather seems that you have an entirely different problem at hand. From what I gather, you have a fixed output format, but no fixed input method to get that output format.

[leafman]

An old program and a new program being run at the same time to produce the same output - Parallel Running (a form of testing)

We call it parallel processing at work, but it is the same thing. We use it when we have a manual process or automated process we trust and we are moving to a new way of getting the output that we don't trust to be correct everytime.

[JeenLeen]

@Jeenleen:

It rather seems that you have an entirely different problem at hand. From what I gather, you have a fixed output format, but no fixed input method to get that output format.

Correct. (To be slightly more precise, there is sometimes a fixed input our 'raw data' has, in that it is formatted however the person supplying it to me gives it to me and I can usually get them to be consistent from year to year. But there's no standardized method or specifications for how that raw input data gets made into the final output.)

We call it parallel processing at work, but it is the same thing. We use it when we have a manual process or automated process we trust and we are moving to a new way of getting the output that we don't trust to be correct everytime.

I have considered "parallel processing" as the right jargon, but I wasn't sure since Googling the term just brought up stuff on computer processors.
I plan to use the term in a presentation on quality control tips*, so I was hoping for some jargon-term that would make it clear what I meant. But I can probably explain it in just a sentence or two, stating it could be used to check work or to check a new method for doing something. I do really appreciate all this conversation since it has informed me that this practice is not common, and so I can phrase things more on validating new processes than how I was originally picturing things.

The more I think about it, the main reason we do it here is to compensate for potential mistakes or incompetency, and it's mostly due to one employee consistently making mistakes due to apparent incompetence. We don't use it systematically for most things.

*with the caveat that most would not have the staff or time to do this on everything

[aspi]

So something like that is what I'm talking about, except where it's not a supervisor checking their employee's work, but two employees assigned the same thing to sorta cross-check each other.

Since you put it this bluntly, I suppose I can do the same: That's your employer paying two people to do the same job because (at least) one of them is incompetent. Why would that need a name? That's not a state you want to have in any business. If you consistently assign work to people that they can't do by themselves and need to be double checked, those people need to be let go or given a different task that the can handle. I know this sounds harsh, but data integration on the level you describe is not a hard job (it's just annoying to do).

Moving beyond the question at hand, there is one advice I would give, especially since you have asked multiple database-, SQL- and data management-related questions in the past: what you need is a proper database infrastructure. And by infrastructure I mean nothing more than a database and some well defined schemata that everyone adheres to. Everything else is data integration when new data comes in. Your example data summaries can then be generated in a few lines of SQL code that can be verified manually in a few minutes. While is may seem daunting to set up, it really isn't, and this is simply what you have to do when you are curating data from multiple sources. That's what database are designed to do and there is literally decades of research into and business experience in this very topic that can help to avoid those problems.