Redressing Survivorship Bias: Giving voice to the voiceless

Survivorship Bias, or Survivor Bias is the name given to sampling errors that can skew the findings of research. In the Second World War the Americans were keen to research the damaging hits taken by their aeroplanes so they could reinforce those areas, and so they photographed the damage to planes returning from bombing missions over Germany. Abraham Wald pointed out the error of this sampling process as the planes that did not return provided more accurate data.

In education, and every other profession, we are complicit in Survivorship Bias each time we positively accepted many of the predetermined judgements made by those who have experienced idiosyncratic pathways to success. Survivorship bias is something that we all need to consider as we plan our next pathways in education, and we should pause and consider giving voice to the voiceless – those who will never attain the status of feted educational legends with the one best method for success.

Introduction: Sampling and Research Biases
All research attempts to manage the uncontrolled variables that influence any judgements. Such biases sit alongside Survivor Bias and they influence judgements that are being made. The literature lists confirmation bias and evaluation bias (Schwind & Buder, 2012; Munro & Stansbury, 2009) but every aspect of a researcher’s beliefs and personality has the capacity to bias subjective judgement.    

During the Second World War, the American aeroplanes operating over Germany suffered damage from anti-aircraft flack and German fighter aircraft interceptions. The American Bomber Command wanted to research damage to their planes so that they could scientifically identify the areas of greatest damage and then fortify the planes’ fuselages. Abraham Wald, a Hungarian mathematician who fled to the United States and worked at the Columbia University in the Statistical Research Group, became involved in this research into plane damage. Wald was able to point out that there was little point in reinforcing the damaged areas on surviving planes because the critical data were the damage to the planes that had been shot down.

Wald was reported as saying, “Gentlemen, you need to put more armour-plate where the holes aren’t because that’s where the holes were on the planes that didn’t return”. This sampling error he called Survivorship Bias.

Interestingly, in the modern financial context Survivorship Bias has become a key concept when making investment decisions. James Chen (2019) noted that “Survivorship bias or survivor bias is the tendency to view the performance of existing stocks or funds in the market as a representative comprehensive sample without regarding those that have gone bust. Survivorship bias can result in the overestimation of historical performance and general attributes of a fund or market index.” This error causes inflation of fund returns because loss of data from the companies that have been bankrupted, and it can mislead investors into making bad investment decisions.

Donohue (2018) drew attention to Survivorship Bias in epidemiology and entrepreneurship. A classic example of this bias is examining the success of university graduates and ignoring the likes of Mark Zuckerberg (Facebook), Bill Gates (Microsoft), Jack Dorsey (Twitter), and Travis Kalanick (Scour and Uber), all of whom quit university to follow their dreams. The Survivorship Bias error would not consider the entrepreneurial success of university dropouts in this case.  

Perhaps an extreme example of the problems with Survivorship Bias is the case of pedigreed dogs that have been bred for showing, not functionality, and as a result, genetic problems now underscore the dog breed viability. This situation came about by prize-winning dog breeders becoming dog show judges, and then promoting the more unique physical characteristics that made their own dogs so successful. The less successful dog breeders did not have a voice in determining what was considered “good”. The RSPCA (n.d.) warned of this inwardly controlling process:

'Unfortunately, some of the breed standards require exaggerated looks such as flat faces, abnormally large eyes, excessively wrinkled skin or disproportionately short legs. Exaggerated features can cause pain and distress to dogs and seriously compromise their welfare and their ability to lead a normal comfortable life. For some breeds over time, these physical features have become increasingly exaggerated, for example, breeders have selected for even flatter faces or even shorter legs, further compromising animal welfare.'

Furthermore, the pups that don’t fit the declared standards are often euthanised.

It is easy to find the phenomenon of successful people exaggerating the characteristics they think that made their dogs, or themselves successful survivors, and then promoting in that image. So much for the concept of Best in Show.

School Leadership Studies and Survivorship Bias
School leadership and administrative studies are replete with Survivorship Bias examples. School leadership by its very nature is based on survival, and retiring, successful school principals are often happy to give their verbal version of Sinatra’s song: “I did it my way”. The danger for those keen to follow these recipes for success is that the historical educational contexts are in a constant state of flux, and the uncontrolled variables often get in the way of promised success.

Example 1: Educational 'war stories' – the currency of credibility.
Relating information sharing among school leaders back to Wald’s wartime experiences a common currency in establishing status and credibility is the power of a school leader’s war stories. The Wallace Foundation’s (2013) paper noted:

'One former principal, in reflecting on his experiences, recalled a typical staff meeting years ago at an urban school where “morale never seemed to get out of the basement.” Discussion centered on “field trips, war stories about troubled students, and other management issues” rather than matters like “using student work and data to fine-tune teaching.” Almost inevitably, teacher pessimism was a significant barrier, with teachers regarding themselves as “hardworking martyrs in a hopeless cause”' (p. 8).

These 'war stories' fulfil human personal and professional needs, and promotional selection panels thrive on these recounts. When asked to describe solving a school-based problem the applicant who can tell the selection panel of success in a Covid 19 affected school that also had an outbreak of leprosy is always going to be marked higher because of the degree of difficulty loading. In comparison the leader who successfully ran a school without the overt trauma is always going to be comparatively disadvantaged despite the fact that he/she may have processes within the school that averted such a disaster in the first place and so, because of this, never actually had to deal with the ensuing escalation. Surviving awesome disaster always outranks stability and success in undiscerning minds.

Example 2: Principal selection: Ducks pick ducks
All selection criteria, or professional standards for principals are based on beliefs about what good leadership looks like across an educational jurisdiction, and they are impregnated with undeclared Survivorship Bias. Our starting point is that no two schools are identical, and rather than use a generic, normative, templated view of 'The Universal Principal', who will be outstanding in every situation, it is better to apply a 'horses for courses' approach. The ubiquitous, templated standards, which are common around the world, are based on the experiences of a few successful educational administrators who have followed their own unique career pathways – they are the equivalent of Wald’s returning planes, or Best in Show. These damaged, returning plane equivalents are then woven into narratives that are often elevated to Gospel status by the successful elite, while those who fail are simply dismissed from the discussion. An excellent analysis of the selection of American military flag officers reinforces Survivorship Bias when successful officers select aspirant officers in their own image: “However, it is well established that promotion boards tend to select officers in their own image, or as one interviewee put it, “ducks pick ducks” (Jackson, Kidder, Mann, Waggy, Lander & Zimmerman. 2020. p. 73).

Example 3: The dangers of stated 'best practice' 
The labelling and promotion of 'best' anything is often fraught with danger. The aphorism: 'Beauty is in the eye of the beholder' is so true for all judgements of best practice: Australian-of-the-year, the Brownlow medal, Membership of the High Court, Dally M Medal, and even the school netball team selection! The problem is always the subjectivity of the judgments, and obvious and hidden biases that influence those making the decisions. Among the hidden biases in every case, Survivorship Bias is a possible factor.

Conclusions
The Survivorship Bias sampling problem is with school leaders every time they get into the realm of evidence-based decisions, which is why all school leaders need a good understanding of research methodology, and an awareness of the dangers of Survivorship Bias.

In the practical domain of collegial leadership advice, particularly when successful school leaders are paraded before their peers to relate why they have been so successful, the narratives are often a prime examples of Survivorship Bias. With the thousands of variables influencing daily decisions in school leaders’ lives, there is no way that these events can be replicated or even adopted with certainty in another educational context. The real-life warning of a subjectivity myopia can be seen with the inline breeding for pedigreed show dogs, and the unethical consequences reinforce the need for the diverse functionality of 'horses for courses' and even hybrid vigour.  

A lack of consideration of the professional narratives of the unsuccessful members of the profession is not only an ethical issue, but an example of a lack of common sense that reinforces Survivorship Bias by an argumentum ad hominem – attacking the unsuccessful, non-survivors.

References
Chen, J. (2019) Survivorship bias. Investopedia. https://www.investopedia.com/terms/s/survivorshipbias.asp

Donohue, W. (2018, July 24). Seven lessons on survivorship bias that will help you make better decisions. I Done This Blog. http://blog.idonethis.com/7-lessons-survivorship-bias-will-help-make-better-decisions/

Jackson, K., Kidder, K.L., Mann, S., Waggy, W.H., Lander, N., & Zimmerman, S.R. (2020). Raising the flag: Implications of U.S. military approaches to general and flag officer development. Santa Monica, CA: RAND.

Munro, G.D., & Stansbury, J.A. (2009, September). The dark side of self-affirmation: Confirmation Bias and illusory correlation in response to threatening information. PSPB, 35(9), 1143-1153. doi: 10.1177/0146167209337163

RSPCA (n.d.). What animal welfare problems are associated with pedigree dog breeding? https://kb.rspca.org.au/knowledge-base/what-animal-welfare-problems-are-associated-with-pedigree-dog-breeding/

Schwind, C., & Buder, J. (2012).  Reducing confirmation bias and evaluation bias: When are preference-inconsistent recommendations effective – and when not? Computers in Human Behavior, 28, 2280-2290.

Wallace Foundation (2013, January). The school principal as leader: Guiding schools to better teaching and learning. Perspectives. https://www.wallacefoundation.org/knowledge-center/Documents/The-School-Principal-as-Leader-Guiding-Schools-to-Better-Teaching-and-Learning-2nd-Ed.pdf