Statistics New Zealand’s Public Engagement on the 2018 Census

Tackling challenges together This case study is part of a series by CPI and Engage Britain examining innovations in citizen engagement that have used tech-enabled deliberative methods to enhance the policymaking process. There are seven assessed against the Public Impact Fundamentals and two briefings that describe processes for ideas gathering. The series demonstrates that though we are in the early days of using these methods, they offer much hope for more legitimate answers to the big public policy questions of our time. Such methods are also becoming more widespread but work best when used in combination with offline methods. Our paper pulls all the innovations together and offers a view on what we can learn so far.

Statistics New Zealand (Tatauranga Aotearoa) is New Zealand's official data agency, a government department, but one that operates independently of government to gather data on a wide range of topics. In preparation for the country’s 2018 census, the agency identified that a fifth of the questions from the previous census in 2013 might need to be revised to ensure they were up-to-date and relevant to the needs of the public, government, and other stakeholders. These topics included contentious and socially sensitive issues such as gender identity, sexual orientation and religion as well as issues subject to rapid change, such as technology. 

Statistics NZ, or Stats NZ, recognised the need for greater public input in its decision-making process to address gaps in its knowledge and to ensure its decisions were grounded in an understanding of the views of the public and affected populations. Stats NZ developed a public engagement initiative that included online engagement, offline workshops, and a formal submissions process of census questions, followed by two years of testing potential questions. This was the first time Stats NZ used an online approach as part of their public engagement, in which it was supported by Loomio, a cooperative social enterprise that had developed a platform designed for supporting discussion and decision-making. The Loomio platform was used to host online public discussion, while the organisation also assisted in the online recruitment process and in providing facilitation training for Stats NZ staff. 

The initiative was particularly successful at accessing marginalised and hard-to-reach groups, including young people, and enabling their views to inform the reworking of the questions for the 2018 Census.[1] Despite this, the eventual census has been criticised for its “digital first” approach to data gathering, its failure to include a Pākehā ethnic category, its questions on sexual orientation, the absence of a non-binary gender option, its low completion rates, and the delays in publishing results.[2][3]

The challenge

The New Zealand Census of Population and Dwellings was due to take place in 2018, five years after it was last held in 2013. Information from the official census helps determine how billions of dollars of government funding are spent across the country. It is also used by councils, community groups, iwi (Māori tribes or collectives) and businesses to plan for the future, and it helps the government make decisions about which services are needed and which locations should be prioritised. This would include the provision of hospitals, kohanga reo (kindergartens where lessons are conducted in Māori), schools, roads, public transport, and recreational facilities.[4]

In developing the census questions, Stats NZ was aware of the need to ensure the information gathered by the census was up-to-date and relevant to contemporary needs, while also ensuring sufficient compatibility for comparison across previous censuses. Therefore, Stats NZ engaged in internal testing of potential questions, identifying a range of areas potentially in need of revision. These included questions about technology, e.g. should people still be asked whether they use fax machines, gender identity, e.g. should there be a third gender option, and topics relating to smoking, drug use, and religion. Stats NZ recognised that the process of decision-making would benefit from further input from stakeholders, the public, and affected populations. 

The initiative

In late 2014, Stats NZ tested potential questions within their team. They identified the fact that up to a fifth of the questions might need to be changed. The questions covered a wide range of issues, of which the Stats NZ team had only partial direct experience, so they recognised the value of public input and of engaging a diversity of groups in the process of reworking questions. There was concern within the organisation that traditional approaches to public engagement, such as town hall meetings, would only attract a small, unrepresentative portion of the population. 

With these concerns in mind, the agency appointed a project manager and hired Loomio to support them in engaging the public online and in recruiting a wider and more diverse population of participants. Loomio provided Stats NZ staff with training in online facilitation,  advice on the delivery of the project, and management of the Loomio discussion platform itself, which offers online discussion and decision-making. It includes various features that nudge groups towards consensus-based decision-making and visual tools that support debate. It had previously been used by a variety of groups across the globe, including local government in New Zealand, Podemos in Spain, student organisations in the UK, and Belgium’s Pirate Party. 

Stats NZ initially tried to recruit participants through newspaper adverts. However, this strategy had very limited success. With the support of Loomio, they engaged in social media and online marketing and managed to recruit a much wider audience. Part of this strategy included a technique called “snack media”. This involved taking small pieces of content, typically concerning controversial issues, and placing them in various social media channels that would then draw people on to their site. Furthermore, much of the information that government documentation and Stats NZ provided regarding their decision-making was written in dense prose, often inaccessible to the public. The Loomio team therefore provided support in summarising and adapting this material to ensure it was accessible and engaging. Through the Stats NZ’s existing mailing lists and social media, the team reached out to groups within the LGBTQ+ community, Māori, religious organisations, and other civic groups that would be particularly affected by proposed changes. 

The Loomio discussions remained open for two months from 30 April to 30 June 2015. The discussions were organised around key themes and divided into the following 12 topic areas

  • Education and training
  • Ethnicity, culture and identity
  • Families and households
  • Health
  • Housing
  • Income
  • Location
  • Population structure
  • Second address/residence
  • Telecommunications
  • Transport
  • Work. [4]

Each topic area was further divided into individual discussion subtopics: for example, “Ethnicity, culture and identity” included 11 discussion subtopics, such as ethnicity, religious affiliation, sexual orientation, and gender identity. When visiting an individual discussion subtopic, participants could read a summary of the issue, preliminary recommendations produced by Stats NZ on changes to the questions, and a summary of key points and suggestions emerging from the public discussions. The discussions on Loomio were facilitated by members of Stats NZ. Participants were informed that the discussions would be considered as an input to the final decision-making process, and 260 people participated in these discussions. 

In addition to online deliberative engagement, Stats NZ ran a series of face-to-face seminars and stakeholder workshops in Auckland, Christchurch, Dunedin, Hamilton and Wellington in May 2015. These seminars involved small groups of experts, statisticians and academics.

Finally, a formal submission period ran between 18 May and 30 June, beginning two weeks after the online deliberative engagement process started, which allowed time for people to discuss with others and develop their thinking on the topics. This process was open to all citizens, who could make formal suggestions directly to Stats NZ, either as individuals or as organisations. The process involved filling in a form that could be submitted via the Stats NZ website, by email, or through paper submissions. The Loomio platform provided a link to the form, and advised participants that submitting a formal suggestion was the best way to influence decisions. 

Following the public engagement process, Stats NZ carried out survey development work and testing, including cognitive testing of questions (cognitive tests are used to understand how respondents interpret questions and instructions), mass completion tests, and two large-scale pilots of the census. The conclusions and final decisions were then made by Stats NZ and published in a final report.

The public impact

The final decisions regarding the 2018 census were made by Stats NZ, following the public engagement and formal submission period and subsequent testing. The process resulted in several changes and additional questions to the 2018 census. Many changes were not prompted specifically by public engagement, but rather by other considerations and at other stages of the testing process.

The newly-added topic areas included:

Location

  • Usual residence one year ago

Housing 

  • Access to basic amenities
  • Dwelling dampness indicator
  • Dwelling mould indicator

Transport

  • Educational institution address
  • Main means of travel to education

There were also major changes to questions in the following subtopic areas:

  • Disability
  • Types of heating 
  • Means of travel to work 
  • Place of residence five years ago.

The main topic areas where changes to questions had been considered but, in the end, were not included were:

  • Sexual orientation
  • Gender identity
  • Step families
  • Licence to occupy (a common tenure for people living in independent, self-care, townhouses and units in retirement villages, in which a person lives in a dwelling but does not own it)
  • Ownership of other dwellings
  • Second address/residence.[6]

The decision made by Stats NZ to exclude questions on sexual orientation and gender identity was a particularly contentious one, as it went against the findings of the public engagement process. Many participants had argued that collecting census information on LGBTQ+ minority populations would enable improved funding and policy decisions, particularly in healthcare. Consultation on gender identity also revealed a high level of interest in collecting relevant data. It was felt, particularly by NGOs and researchers, that this information would be useful to inform funding and policy decisions of relevance to these population groups. 

While Stats NZ conducted testing on questions around these issues, the final report commented with regard to the inclusion of questions regarding sexual orientation: “The results from the July 2016 test indicated that there would likely be some issues in producing high-quality data for this topic. In the data collected, the non-heterosexual populations were smaller than the number of respondents who did not answer the question or indicated they preferred not to answer it. As a result, the level of confidence in the data we would be able to produce for this topic would be of concern. We also received some negative feedback in our public testing of this question, indicating sensitivity to answering questions on this topic.”[6]

Thus, a key concern was that collecting information on sexual orientation  in a self-completed questionnaire like the census would not produce good quality data. Stats NZ decided instead to include relevant questions in other surveys, such as the New Zealand General Social Survey, a survey of approximately 8,000 people run every two years and which focuses on wellbeing.[6

On the decision to exclude questions regarding gender identity, the final report concluded: “Testing indicated that response behaviour to this question was variable across transgender respondents. Gender identity can change over time and be expressed in a number of ways and forms. It is difficult to create a question that captures all these aspects. Information collected from a census question on gender identity would not enable us to output any population estimates on the populations of interest. Therefore, we will not include this topic in the 2018 Census. However, we are committed to further investigating this important but complex topic within the wider OSS (New Zealand’s Official Statistics System).”[6] The decision was therefore to exclude any questions regarding sexual orientation or gender identity from the 2018 census. 

Finally, the categories of ethnicity remained unchanged, despite extensive debate online. Various terms for New Zealanders of European descent were considered. Stats NZ use the category “European New Zealand”, while many online commentators preferred  “New Zealander”, and others wanted “Pākehā” a Māori term for New Zealanders of European descent, while yet others suggested “Tauiwi”, a term for any non-Māori person born in New Zealand. Each term was contested.[7

As the final report explains, the majority of comments stated a dissatisfaction with the response options, claiming that they were insufficient and divisive. Many commenters felt that ‘New Zealand European’ was an inadequate description and wanted a ‘New Zealander’ response category. Others commented on the difference between national identity and ethnicity, and the value of the ethnicity data. In their view, less meaningful data would be the result if a ‘New Zealander’ response option was included.”[6] The implication of the last point was that it blurred the distinction between national and ethnic identity. 

The final report provided the following reasons for its decision not to change questions on ethnicity. “The ‘New Zealander’ responses decreased from 10.9 percent to 1.6 percent between the 2006 and 2013 censuses. It appears that the response rate is largely dependent on publicity around this topic. Yet, most submissions during the engagement process favoured retaining the question as doing this will provide the data quality benefits of continuity and comparability over time. Statistics NZ therefore decided that their question on ethnicity will be included with no change [that is, it retained the terminology of ‘New Zealand European’].”[6]

On 6 March 2018, the government census was carried out. It was a “digital first” census with citizens primarily expected to use the internet to answer the questions.[3] The release of the results has been delayed three times, and they are expected to be released in September 2019.[8] The data has been delayed in part due to a low response rate, falling from 94.5 to 90 percent. [9] Richard Arnold, a lecturer in Victoria University’s School of Mathematics and Statistics, said of the low completion rate that “it is disastrous… the problem is that whenever you have undercount you’ve got some risk of error, risk of bias”. [11] Arnold believes that large groups which might have had significant numbers of people missed out include people aged in their teens and early 20s, Māori and Pacific Islanders, and the elderly.[11] This is a particular issue in relation to data provided by Māori respondents, whose response rate is predicted to be as low as 70 or 80 percent. Inadequate operational resourcing and staffing levels have been blamed for the low completion rate.[3] This could have significant impact on representation and resources for Māori, especially for small iwi. One report quotes an expert as saying that Māori risk losing an electoral seat and more than 20 new iwi will not be counted correctly, due to unreliable census data.[3

The 2018 Census has also been criticised for its failure to include Pākehā as an ethnic identity option, any questions on sexual orientation, and any questions that capture non-binary gender identity.[2] [10] New Zealand’s minister of statistics, James Shaw, said that it was too late to change the questions, but he would push to include questions on gender identity and sexual orientation in the 2023 census. He further highlighted other ways of collecting data in the meantime, explaining that “we’ve got to gather this data because we make significant healthcare funding decisions and other public policy decisions in relation to this community, and we don’t have granular enough information at the moment”.[3]

Written by Martin King

What did and didn't work

All cases in our Public Impact Observatory have been evaluated for performance against the elements of our Public Impact Fundamentals.

Legitimacy

Public Confidence Fair

Evidence on public confidence is mixed, and it is helpful to make a distinction between initial attitudes to the public engagement process and attitudes to the final decisions of Stats NZ and its execution of the 2018 census. The public engagement process itself received positive feedback from the participants involved, one member of Loomio commenting that “individuals from the trans-community reported that although they recognised that the ultimate decisions were made by Stats NZ, they appreciated that efforts had been made to engage with them and that this represented the first time they felt the government were taking transgender issues seriously”. The agency’s retrospective analysis observes that one of the project’s strengths was that it created a “real buzz that the government was willing to engage in this way” and that the “gender diversity community (sic) were still talking about the project after the process”. Furthermore, the success of the project in engaging the public through digital methods was seen to have had “flow-on effects” with other organisations.[5] 

Nevertheless, there has subsequently been a sense of disappointment and some harsh criticism of the decision not to accept the results of the public engagement process and gather information on either sexual orientation or gender identity in the 2018 census.[2][10] Social activist Aych McArdle commented: “If you don’t count someone, you’re almost saying they don’t count”, and observed the importance of collecting this information for providing health and education services.[10]

The delivery of the 2018 census has also been criticised on other counts, with news reports highlighting problems of operational resourcing and subsequently low completion rates. In particular, there were problems in collecting data on the Māori population due to the primarily online execution of the census, which may have significant adverse effects on the representation of Māori and on resources for small and vulnerable groups.[3] Matthew Tukaki, Māori Council spokesperson has been quoted as saying that “this is a disgrace because the data runs the risk of telling a story that is neither accurate nor true. It gives public servants the ability to paint a picture that, because there was no or little response from some of these Māori communities, no one lives there and therefore it’s a perfect excuse to withdraw services.”[9

Stakeholder Engagement Fair

During the initial stages, three broad stakeholder groups were identified: government, users (for example, corporations that were interested in demographics change), and the general public. In addition to engaging the three broad stakeholders, Stats NZ also used mailing lists and social media to engage groups particularly affected by decisions on whether to include specific questions, for example on sexual orientation, religious affiliation, ethnicity, and Māori descent.[7

Stats NZ reported being very satisfied with the project and its capacity to access hard-to-reach groups and engage a much wider group of people than through traditional methods. Susan Riddle from Stats NZ observed that through Loomio they were able to “reach people who wouldn’t have otherwise contributed to the conversation, including marginalised populations and youth”.[1] While the use of a digital platform to perform public engagement reduced some barriers and enabled a larger and more diverse group of participants to be recruited than traditional town hall methods, it should be noted that the project recruited only 260 people, and the online method introduces its own barriers via a digital divide that typically excludes older people and those with limited access to the internet. Some communities with limited connectivity, such as many Māori communities, were therefore disproportionately excluded.[9]

In this sense, although the process was more open and inclusive than previous approaches, there were limitations to stakeholder engagement. Furthermore, on some key issues the final decisions taken went against the recommendations that had emerged from the public engagement process. In its final report, Stats NZ acknowledged the input from the public and provided an account of why they did not implement these recommendations.[6] Nevertheless, the decisions have been criticised in the media, and some stakeholder groups have ultimately felt excluded and let down by the outcome.[10]

Political Commitment Good

An individual from Loomio reported that political commitment was high for the public engagement process, and that –  despite the project’s limited funding – those members of Stats NZ who were involved in the project displayed strong commitment by their willingness to try new things and by their responsiveness to Loomio’s guidance. A retrospective analysis conducted by Stats NZ observed that having two members of the Stats NZ team as a dedicated resource was a particular strength of the project.[5]

Although Stats NZ was generally enthusiastic about the project and invested in it, one interviewee noted that the department’s communications team was much more cautious and was concerned that the public engagement was risky because it opened the government to criticism and potential embarrassment. Loomio led workshops addressing these fears and developed strategies for mitigating risk, but ultimately this shows that there were conflicting tendencies regarding political commitment within the government and its various agencies.

Policy

Clear Objectives Strong

The objectives of the public engagement initiative were clear: Stats NZ wanted to gather public opinion on potential changes to the 2018 census. Specifically, it wanted to engage a larger and more diverse public than it expected to achieve through traditional offline processes. The purpose and scope of the public engagement process were also clearly communicated to people through the Loomio platform – public input would be used to advise decision-makers and direct them on the potential revisions. In addition, Stats NZ presented summaries of the topics online, as well as the reasoning behind its preliminary judgements on what future questions should look like and summaries of the online conversations over the course of the public engagement process. 

Evidence Strong

Stats NZ’s retrospective analysis argues that one of the strengths of the project was the “high-quality information used to help decision-making”.[5] The public engagement process itself was delivered with the support of Loomio, a team with deep expertise and experience in online facilitation, open technology development, and collaborative decision-making. The process was piloted within the Stats NZ team before the public engagement process began. According to an internal assessment, Stats NZ and Loomio worked together to ensure the participants were well informed, and material about the subject matter and purpose of the project was clear, accessible and based on strong evidence.

Feasibility Fair

During the public engagement stage, the use of technology both in engaging the public and hosting discussions was considered to increase the efficiency and significantly reduce costs and barriers when compared with the traditional approaches of print media engagement and town hall meetings.[1] However, as mentioned above, relying heavily on online technology produces its own barriers, and while the process did use offline workshops, they were described by an internal assessment as “too little, too late”.[5]

One interviewee observed that, in their experience, governments frequently underestimate the costs and resources required for a public engagement process. In this case, they felt that while they had managed to engage 260 members of the public in the process, they could have engaged thousands with greater resources and more time dedicated to recruitment. An internal assessment observes that “a larger proactive targeted campaign would have greatly benefited the process”.[5]

Action

Management Strong

Stats NZ appointed a project manager to oversee and administer the process of public engagement, and one interviewee involved in the delivery of the project described how they employed younger and more tech-savvy people to support them. Loomio provided training for 20 people involved in the project to develop the skills necessary for hosting and facilitating public discussion processes. One interviewee observed that much time was dedicated to developing the skills of those involved in delivering the project and building capacity in relation to public engagement, and this was regarded as one of the project’s more successful elements. 

The analysis of the project identified a number of strengths in relation to the project management, including a clear structure and setup, and a flexibility within the system to adapt to challenges and improve the approach to public engagement. There was a strong relationship between Loomio and the Stats NZ team, which enabled a robust engagement process.[5]

Measurement Strong

After the initiative finished, Stats NZ investigated whether the information emerging from the public engagement process could have been acquired from other data sources. It also carried out extensive testing of the proposed changes, including cognitive tests of questions, mass completion tests, and two large-scale pilots in 2016 and 2017. These tests were designed to ensure that the questions produced high-quality data and helped determine the agency’s ultimate decisions about which questions to adapt or retain unchanged, and which new questions to include. 

Alignment Good

The analysis of the project indicated a strong commitment and alignment of values on the part of those delivering the project. It observed a strong relationship between Stats NZ and Loomio, characterised by a high level of trust, honesty and integrity, as well as both partners being very responsive to each other. The report also describes the Stats NZ team as having the courage and willingness to try different things, while Loomio was described as providing ongoing, detailed and thoughtful support.[5] One interviewee from Loomio also described the Stats NZ team as being very committed to the project and the aims of the public engagement process.