I intend to do a series of posts on how polling is done, how to interpret polls and what polls do and don't tell us. The numbers thrown out by the talking heads on television are basically useless and, since I used to do socio-economic research for a living, I thought this would be a useful contribution. I haven't prepared those posts yet. Partly because I'm very busy and partly because my overall level of contempt for our political class and its apologists has reached an all time high and I'm having a hard time getting into this election.
Before passing on some of the latest polling information, I will give you a couple of caveats. Anyone using current polling data to predict a winner in the election or a number of seats is, frankly, nuts. I say this for two reasons. First, there is a large amount of soft support for all parties. As much as 5% of any party's support could disappear almost over night. So far this already seems to be happening to the NDP. Second there is a huge pool of undecided voters. Two weeks ago undecideds were at 19 - 20%. Today that seems to have dropped to about 16% but, to put it into context, that is more than the total national support for either the NDP or the Bloc. When you combine these two factors, it is impossible to predict any outcome with any accuracy at the moment.
Furthermore, many of the polls are now coming out without undecideds being included in the numbers. This is blatantly misleading and any poll that reports decided voters only and does not include undecideds should be ignored. There have been many attempts to improve polling models to predict which way undecided voters will break. To date, none of them have been successful.
In addition, while 84% of voters are supporting one party or another, we also know that only about 50% of them will actually vote. Voter motivation is very hard to quantify. Yet another problem is with voter efficiency (aka voter concentration).
In other words, use with extreme caution.
CPAC-SES Nightly Tracking Poll
Tracking polls are quite different from your standard opinion poll as they involve continuous polling. I like to use SES data because they are usually the most user friendly poll and are much more forthcoming about their methodology than some other companies. They also tend to provide more detailed breakdowns.
Methodology: A national random telephone survey is conducted nightly by SES Research throughout the campaign. Each evening a new group of 400 eligible voters are interviewed. The daily tracking figures are based on a three-day rolling sample comprised of 1,200 interviews. To update the tracking a new day of interviewing is added and the oldest day dropped. The margin of accuracy is ±2.9%, 19 times out of 20. The respondent sample is stratified geographically and by gender. The data may be weighted by age according to data from the 2001 Canadian Census administered by Statistics Canada. Percentages reported may not add up to 100 due to rounding.Leadership Index Score Methodology:
Every day, SES will report on a daily leadership index score. The leadership index score is a summation of the three leadership indicators (trustworthy, competence, vision). For example, on November 30th (N13), Jim Harris received a Leadership Index Score of 5 because 2% of Canadians identified Jim Harris as the most
trustworthy leader, 1% said he was the most competent and 2% said he had the best vision for Canada. The Leadership Index Score will be tracked daily with updated results from the previous night of polling. Unlike the other measures tracked by SES, the leadership index score will not be based on a three-day rolling sample. It will be reported from results the evening before. This will allow SES to measure the performance of the federal party leaders and the key factors driving performance on a daily basis.CPAC Questions:
1. If a FEDERAL election were held today, could you please rank your top two current local voting preferences? (First ranked reported)
2. Are you currently leaning towards any particular FEDERAL party, and if you are, which party would that be?
Currently (rotate), Paul Martin is the leader of the federal Liberal Party, Stephen Harper is the leader of the Conservative Party of Canada, Jack Layton is the leader of the federal NDP, Gilles Duceppe is leader of the Bloc Quebecois and Jim Harris is leader of the federal Green Party.
3. Which of the federal leaders would you best describe as the most trustworthy leader?
4. Which of the federal leaders would you best describe as the most competent leader?
5. Which of the federal leaders would you best describe as the leader with the best vision for Canada?
6. Based on what you know about the federal party leaders and their performance during the election campaign up until today, who do you think would make the best Prime Minister?
That is about as much transparency as you are going to get from a polling organization. And here are the numbers.
Results: Polling November 29 to December 1, 2005 (Random Telephone Survey of 1,200 Canadians, MoE ± 2.9%, 19 times out of 20). Percentages may not add up to 100 due to rounding.Decided Voters (N=1,015, Change from November 13th)
LIB - 37% (+3)
CP - 29% (+1)
NDP - 15% (-5)
BQ - 14% (NC)
GP - 5% (+1)
*16% of Canadians were undecided (+2)Leadership Index (Daily composite of the Leaders' Trust, Competence
and Vision; Change from November 13th)
Martin - 84 (+8)
Harper - 58 (NC)
Duceppe - 39 (+2)
Layton - 25 (NC)
Harris - 8 (+3)Best PM (N=1,200, Change from November 13th)
Martin - 29% (NC)
Harper - 21 (-1)
Unsure - 17% (+4)
Layton - 14% (-2)
None - 12% (+1)
Duceppe - 6% (-1)
Harris - 2% (-2)
SES Analysis: The opening days of the campaign have seen the NDP slip in support with both the Liberals and the Conservatives picking up support...The Liberals have picked up in Quebec at the expense of the New Democrats and marginally picked up in Atlantic Canada at the expense of the Conservatives.
Our Leadership Performance Index shows an improvement for Paul Martin....BQ leader Duceppe does well on the trustworthy measure....
In regards to who Canadians think would make the best PM, unsure is up.
Strategic Counsel is also doing a tracking poll for CTV and the Globe and Mail. Unfortunately, the SC poll (at least as published) does not include undecideds. For decided voters the results are very similar to the SES poll.
When asked who voters would support "if the election were being held tomorrow," The Strategic Counsel found (change from Nov. 24-30 numbers in brackets): 35 per cent would support Liberals (no change) 30 per cent would vote for Conservatives (no change) 16 per cent would vote for the NDP (-1) 14 per cent would vote for the Bloc Quebecois (no change) 6 per cent would vote for the Green Party (no change) The poll results are based on a proportionate national sample of adult Canadians interviewed over a three-day period between Nov. 29 and Dec. 1.Overall, the results are considered accurate within +/-2.3 percentage points.
There are regional breakdowns in both polls and I have some thoughts on the results but my time is done for the day and the regional breakdowns carry additional caveats. I will include additional data from Strategic Council (and anyone else where I am able to determine undecided levels) in the future. Pogge has come up with a table structure that we can use to present information in a more readable form. However, it is unlikely that I will dive head first into polling results until I am able to prepare the series of posts I mentioned earlier. There is no point in me becoming just another talking head spouting numbers.
I do hope the additional information I have been able to provide is useful to you.
Excellent... thank you for this. All too often we see polls that leave people feeling vaguely uneasy and questioning, "Where are these numbers from?", "How did they reach these results?", "What were the questions?" etc.
One suggestion: we have a number of polls conducted by multiple entities. Could you do a breakdown of the various polling group and what their biases tend to be?
I posted an entry yesterday on the interpretation of polls and the questionability of representative random sampling.
Here is it:
http://frostyballot.blogspot.com/2005/12/crystal-balls-and-myth-of.html
Hatamoto- I assume your concern is that pollsters may be manipulating their data in accordance with their biases. In other words cooking the books to make one party or another look good.
Various polling organizations have been associated with certain political parties in the past. I don't have a listing of who has been involved with which party.
What I will say is that I consider it highly improbable that any of the mainstream pollsters would deliberately manipulate their data collection because 1)there is an organization called The Marketing Research and Intelligence Association most pollsters belong to that oversees a professional code of standards; 2) polls are under heavy scrutiny from other pollsters, academics and people like me which would make it likely that anyone manipulating their data would be caught; 3)getting caught would pretty much put the company out of business.
There is a little more room to tilt things in the analysis of the data because that is more subjective. No one is entitled to their own facts but everyone is entitled to their own conclusions.
If manipulation of the analysis is occuring, I expect it to be more likely at the media presentation level than at the polling organization level. In fact one media conglomerate who shall remain nameless (because they can afford more lawyers that I can) has been caught doing what many consider a deliberate misrepresentation of polling data. This is why I use the data directly from the pollster whenever possible and in future posts will tell you when I use something that is not "from the horse's mouth".
There will always be legitimate variations in polling results because companies do not draw their samples or word their questions exactly the same way nor are all interviewers equally well skilled and trained. Normally these variations will be small as they were in the two polls I used in the post. In some cases, one company will have consistently higher or lower results across the board than the others but as long as their results are consistent over time and the rankings and spread are similar, this is legitimate too. Of course , a sample occasionally just goes sideways.
There are many ways that data can be accidently (or deliberately) be messed up and I will tell you how that can happen in the series I will put together.
Orla - Your concern is somewhat relevant. Unfortunately the statistics you quote are not. The percentage of households with internet access has no relevance whatever to sample bias and the number of cell phones per 100 population tells us exactly nothing about the number of households that have only cell phones.
This issue was discussed at length during the 2004 US election and two things came out of that discussion. First, there is effectively no way to accurately determine the number of households that have only cell phones. The best result would be obtained from inserting a question in the census. The next partial census is next year and the next full census is in 2011.
Second, the group most likely to have only cell phones are younger people who are also among the least likely to vote. If 5% of the people have only cell phones (an extremely high estimate IMO) and only 20% of them vote, how much of a sample bias has actually been introduced?
Further, the samples we are dealing with in the polls I have quoted are not simple random samples. They are stratified random samples controlling for residence, sex and age. Unless there is some reason to assume that people with only cell phones are significantly different from those of the same sex and age living in the same region who have regular telephones (and I don't know of any reason to assume that), then any sample bias resulting from the fact that people who only have cell phones aren't getting calls from pollsters would be negligible.
I can assure you that, if this was the biggest problem that polling organizations had in getting accurate results, there would be a lot of pollsters running around with ear to ear grins.
I also find it interesting that, while all kinds of people are concerned that those who only have cell phones aren't being represented in samples, I have yet to hear anyone expressing a concern that those who have no phones at all (surely at least an equal number) aren't being represented in samples. I guess the opinions of technophiles are more important than the opinions of the poor.
My interest isn't exactly about deliberate skewing or non-deliberate biases as a result of methodology (or flaws thereof), but I do look forward to your exposition about the pitfalls. :D
I was specifically interested in what appears to be persistant biases, deliberate or not, in our various polling organizations. One could say, for instance, that polls done by ipsos seem to have a consistantly higher-than median polling result for one or two parties over the body of all polls taken (I don't know if this is true or not, I just happen to know the name ipsos ;).
In the last election, I noticed certain firms seemed to be more favorable to one party over others on a regular basis, eg. one firm would consistantly give an additional 2% deviation to the NDP, another would take a point from the LPC and give it to the Bloc, etc.
I'd be interested, if you're willing to tackle it, in the specifics of the defacto biases in each poller, and perhaps conjecture on why those biases exist. I'm probably asking for an awful lot, but I think, if you're doing up a large body of work anyways, that it could be a valuable link that ties the theory of why polling can be inconsistant to practical examples of how biases manifest in a live (and very pertinent) setting.
Im not all that sure about polls, but I do listen to my friends and other people interested in talking about politics : They all agree for the most part that Martin is "TOAST", maybe even burnt at that. I agree and I also believe that the CPC will form the next Govmt.