How to make completely opposing claims using the same survey data (or: how to cherry-pick)

Sheffield Council put out a statement on the BBC recently, during an episode of Countryfile, defending their outsourced Streets Ahead contract against accusations of needlessly felling thousands of mature street trees:

We surveyed 27,000 households. Fewer than 7% said they disagreed with the plans. The contract will bring huge benefits to the city's infrastructure which is why the vast majority of Sheffielders support our plans and why the activists remain out of touch.

Only a small minority ('fewer than 7%') oppose their plans? And 'the vast majority of Sheffielders' are in support? Here's the thing. Using exactly the same data Sheffield Council have used, I could put out the following equally correct statement, in support of the tree protestors:

Fewer than 7% of households said they agreed with Sheffield Council's plans. The vast majority of Sheffielders oppose their plans. The Council remains out of touch.

Wait, what? Only a small minority support the plans? That's entirely the opposite message. How can the same numbers support both of them?

Well, you have to do two misleading things. First, you need to cherry-pick a number. Second, you use a dubious statistical choice that makes it look like a tiny minority oppose the plans, when in reality the data shows an even split of opinion.

Let's go through those two. First, the cherry-pick. The actual survey numbers are as follows. The total number of households they posted letters to is 26677 (round up and you get '27000 households'). 3574 households actually responded - that's 13.4% of total survey invites. Of those 3574, 1744 households opposed the plans and 1800 households supported them.

1744 opposed, 1800 in support? That sounds like something close to an even split of opinion - and indeed, it's not statistically distinguishable from half against, half in support. Not a tiny minority, not a vast majority.

If we cherry-pick just one of those and ignore the other, we're half way to making one of our two opposing statements. The next step: ignore that you should use the number of responses to your survey (3574) to work out the percentages and use the number of letters you posted instead (26677).

By doing that, you can get the 'fewer than 7%' number for both. So we can cherry-pick too: 1800 in support as a proportion of all the letters posted? Fewer than 7%. (1800 over 26677 then multiplied by a hundred to get the percent.)

If exactly the same numbers can be used to produce two completely opposed statements, I hope it's obvious that you're doing something wrong and the numbers are being misused.

The council have defended the statement saying it's factually correct. If you squint, you can just about see how 'we surveyed 27,000 households, fewer than 7% said they disagreed with the plans' is technically true. But I've just shown how the same 'technically true' method can be used to support entirely the opposite message. That's the power of cherry-picking.

And it's not a one-off either. Via the Streets Ahead twitter account, the same data was used to claim only a tiny minority on one street opposed the plans there:

Our household survey results show that of the 54 households on the road, 5% opposed our proposals for street tree replacement.

You won't be surprised to learn: there were only six actual responses on that street, 3 for and 3 against. So again, it's equally correct (but still inappropriate) to say "5% supported our proposals". (It was Rivelin Valley Road, so's you know - again, the numbers are in the document above.)

All of this is ignoring the 'vast majority of Sheffielders in support' statement. In a way, this is the most worrying part. It's just plain wrong, if we're going by this data. But in the context of the 'fewer than 7%' line, I can imagine how one might think, 'well, more than 93% must be in support then'. That's kind of implied, isn't it?

Yet as we've just seen, using the Council's (inappropriate) method, it would actually be 'fewer than 7%' opposed and 'fewer than 7%' in support. They not only omitted to mention this, they have added in a 'vast majority' claim that appears to be completely unfounded. So we're clear, there's nothing in these numbers that even remotely supports a 'vast majority' either for or against. It's an even split.

The ethics of numbers

If your idea of factually correct allows you to make entirely opposed claims with the same numbers, it means you are likely cherry-picking: "pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position". Though here, the cherry-pick wouldn't really work without also mangling how surveys are meant to be used.

I work with numbers in my job: it's a matter of professional ethics to make sure, as much as we can, that our work can be trusted. (Have a read of this code of practice from the UK Statistics Authority - it's a good take on the kind of integrity and honesty we're supposed to aim for.)

We don't know how Sheffield Council created this statement. I can imagine a single over-worked officer under great pressure to get a message out at short notice. But I don't think it's unreasonable to expect the same level of trust from our local councils when they use statistics.

As Ralf Little recently said to Jeremy Hunt (I paraphrase slightly): 'the good news is, now that you know that this statistic is total nonsense, you won’t feel the need to use it again'.

The actual numbers

Let's end on looking at what this survey actually does show - that there's a pretty even split for and against. I should start by saying, we shouldn't really be using the independent tree panel survey1 for this at all. Households were asked their views on trees on their own road. They were not asked, 'do you support or oppose the city-wide Streets Ahead plan for tree management?' They also surveyed households, not individuals. But I guess that's small potatoes compared to the above.

27000 households (rounded up) is the invite number and 3754 is the response number. Trying to maximise response number is central to any survey: the higher the response rate, the more your sample can be relied on to accurately capture what the larger group thinks.

This is hopefully obvious, but let's spell it out to be sure. We don't know what the households who didn't respond think. This is the entire point of surveys: get a sample of views so you can make deductions about everyone else.

So here, the actual split in the response numbers I gave above is 49.6% opposed, 50.4% in support. I may get round to another post explaining why this can't be statistically distinguished from an even 50/50 split - though the intuitive idea is just: how much could that split change as you get more responses? Here, we have a 16% sample - that's pretty big. It's very unlikely to change a lot, but because it's so close to 50%, it could likely shift either side of that 50/50 mark.

At any rate, it is astronomically unlikely that 'fewer than 7%' is the correct percent opposed. For that to be true, all the other households that didn't respond would have to be 100% in favour. The 16% sample would have had to have picked up on every single household opposed. Just... no.

So to end: whether or not the Council knew they were doing this, they have selected numbers to support their own message - as I've shown with a statement claiming exactly the opposite, using exactly the same data and method. This is some way before worrying about sampling rates and confidence intervals. And the 'vast majority' thing... whu?? So let's just end with a tip:

  • Check if you can put out two equally true but mutually exclusive statements using your method. If you can, your method is wrong. Try again.

  1. Sheffield Council surveyed households, one street at a time, to find out if residents wanted an independent tree panel to re-examine decisions about trees on their street. Again, the data is here. It collated all of those single street surveys into one document. 

Comments

INCOMPETENCE, MISREPRESENTATION & DECEIT

It should be borne in mind that even if a valid survey methodology had been used, and even if the results had been correctly analysed and reported, a survey such as this is inappropriate as a basis for making tree population management decisions. It is contrary to all current good practice in urban forestry and arboriculture and incompatible with a SUSTAINABLE approach. Using a survey such as this to inform decisions on whether or not to fell trees is unnecessary, and represents unnecessary waste of resources (in particular, public funds). In short, using a survey for this purpose represents malpractice.

Sheffield City Council did know what they were doing. It was a purely political tactic. By conducting a survey such as this, they could falsely claim to have "consulted" the public, albeit only residents on streets where trees are scheduled for felling. This was a bonus for SCC as they had neglected to take any steps to consult the public on plans prior to commencement of the £2.2bn city-wide highway maintenance project. The survey also created valuable media opportunities for SCC & Amey, and positive media coverage, at a time when they were under intense scrutiny for their incompetent acts, and omissions (reckless and wilful), not least of all misrepresentation of information and distribution of false or misleading information.

The positive media coverage temporarily distracted attention from and provided cover for SCCs/Amey's wrongful, incompetent acts and omissions. It successfully diverted the media spotlight - temporarily - away from their transgressions, reducing the likelihood of public accountability and distracting opposition campaigners, causing them to lose focus & divert limited resources. It bought time. With the passage of time, SCC seek to dismiss their wrongs by saying those are in the past and we must "move forward", or "it's great to have hindsight". When I met with Cllr Bryan Lodge - SCC Cabinet Member for Environment and Streetscene (Cllr Fox’s successor) - back in August 2016, and told him that the surveys and Independent Tree Panel were an unnecessary waste of public resources, he used the latter to excuse his continuing errors. I pointed out to him - correctly - that all SCC errors could have been avoided had SCC & Amey put in place steps to enable and ensure adequate fulfilment of existing policy commitments* and if they were to use and apply the range of good practice guidance and recommendations they claim to, and which Amey are contractually bound to apply, in order to fulfil their contractual commitments*. I also reminded Cllr Lodge that his hindsight comment was well & truly out of order, as I had advised him on the most appropriate ways of addressing all highway tree related matters soon after he accepted his position as Cabinet Member for Environment and Streetscene. I also reminded him that I had previously provided similar advice to Cllr Jack Scott in early 2014 (then Cabinet Member for Environment, Recycling and Streetscene) and all Cabinet Members as of 2015, on numerous occasions. I also reminded him of the very detailed communications that the Save Our Roadside Trees group (SORT) has distributed to every Councillor in the city (in June 2015 & January 2016):

https://www.stocksbridgecommunity.org/sites/default/files/files/SORT%20L...

https://www.stocksbridgecommunity.org/sites/default/files/files/SORT%20L...

I made it clear to Cllr Lodge that what the Council lacked and needed was foresight and that his wrongful acts and omissions, and those of all SCC Councillors and Officers, could not be defended by the excuse that they did not know any better, given that they had received so much sound advice from SORT & I. Cllr Lodge then promised to start identifying and correcting errors and start identifying and implementing steps to achieve adequate, positive, change. As we have all witnessed, he did not do a thing – it was just one of his many lies.

* See the following:

COST OF SUSTAINABILITY:
https://ianswalkonthewildside.wordpress.com/2017/09/10/ignorance-pig-hea...

TREES & HAZARD MANAGEMENT (published):
https://www.stocksbridgecommunity.org/comment/802#comment-802

TREE POPULATION MANAGEMENT BY NUMBERS (published):
https://www.stocksbridgecommunity.org/comment/779#comment-779

"STREET TREE MASSACRE" - a response to Cllr Peter Price (published):
https://www.stocksbridgecommunity.org/comment/732#comment-732

A couple of the most important points amongst a number missing from the above critique are that:

1) the survey was accompanied by an introductory letter that was full of misrepresentation, intended to drum up support for proposed felling;

2) many of the questions were biased, leading questions;

3) a household representative was invited to provide reasoning to support their decision. However, THE LONG-STANDING SCC POLICY FOR TREE WORKS WAS NOT PRESENTED TO RECIPIENTS OF THE SURVEY. For your benefit, here it is:

“WE ARE UNABLE TO CARRY OUT WORK WHERE:

• Trees belong to private properties
• Falling leaves or fruit are causing an annoyance
• Falling blossom, sap or bird droppings are causing an annoyance

• Trees are blocking light or causing shade
• Trees are obstructing telephone wires (contact your telephone service provider)
• Trees are obstructing TV or satellite reception
• We do not remove trees for construction or widening of driveways”

Source:
Sheffield City Council, 2015. Roadside Trees. [Online] Available at: https://www.sheffield.gov.uk/in-your-area/report_request/plants/trees.html [Accessed 15 August 2015].

4) Many people that completed the survey and supported felling did so solely for one or more of the above reasons. It was never made clear how such survey responses would be/were handled by SCC when assessing whether or not a sufficient number of households were opposed to proposals to trigger reference to the sham "Independent" Tree Panel.

I have commented on the household felling survey previously:
https://www.whatdotheyknow.com/request/foi_request_engineering_solution#...

Some of my other letters on the mass culling of Sheffield’s street trees:

SUSTAINABILITY Vs TREE MASSACRE (published):
https://www.stocksbridgecommunity.org/comment/813#comment-813

“HOW SCC & AMEY GET IT SO WRONG: FELLING: SCC/AMEY INCOMPETENCE AND DECEIT”:
https://www.stocksbridgecommunity.org/comment/698#comment-698

“THE GREAT SHEFFIELD CHAINSAW MASSACRE” – A Response to Louise Haigh MP (published):
https://www.stocksbridgecommunity.org/comment/756#comment-756

SUSTAINABILITY – LEGISLATION:
https://www.stocksbridgecommunity.org/comment/835#comment-835
&
https://ianswalkonthewildside.wordpress.com/2017/11/07/george-monbiot-ad...

On a final note, I’ll just point out that Sheffield City Council has been able to squander resources on media spin & smear campaigns, converting enquiries & requests to Freedom of Information requests, the Highway Trees Advisory Forum, the household felling survey, the “Independent” Tree Panel, various court cases & employment of private surveillance, because it has fined Amey MILLIONS OF POUNDS - over £2m in 2015, alone! – for neglect to apply the level of care expected of reasonably skilled professionals when undertaking highway maintenance works for the Streets Ahead project. See:

COUNCIL INCOMPETENCE:
https://www.stocksbridgecommunity.org/comment/728#comment-728

RECKLESS TREE FELLING: OPENNESS, TRANSPARENCY & JUSTICE:
https://www.stocksbridgecommunity.org/comment/714#comment-714