The data revolution and new data sources

The SDGs have underscored the need for a data revolution   to address increased diversity, complexity, and quality of data required to deliver results. The concept of the data revolution comes with renewed emphasis on and stronger advocacy for transparency and accountability in the production of, access to, and use of data. 

The data revolution has significantly raised interest in new data sources primarily Big Data  , from acknowledgment of their existence to their recognition as a parallel and wider universe of data that can support the SDGs. The use of new data sources is explicitly encouraged in FPOS “Principle 5: Cost-effectiveness: Data for statistical purposes may be drawn from all types of sources [...]. Statistical agencies are to choose the source with regard to quality, timeliness, costs and the burden on respondents.” (FPOS, UN, 2014).

Big Data has been around for as long as national statistical systems (NSSs) have been serving as the primary source of statistics for governance and development. This time, Big Data and other new data sources such as citizen-generated data   have become hugely relevant and important as they continue to expand and offer endless possibilities to create and improve statistics within and beyond the NSS and through evolving data-ecosystems    .

Data ecosystems pose a real challenge to NSSs but also present enormous opportunities for improving methods and adapting innovation and technology to improve quality of current data and produce new statistics. With lack of government resources still a significant bottleneck in many countries, NSSs need to step up not only to improve quality of existing data but to start developing and producing smart statistics   built on improved existing, conventional data and new statistics created by the data ecosystems. 

Developing smart statistics will require more than just improving systems and capacities. Producing smart statistics will necessitate transformation of the NSS into a modern NSS   guided and supported by a modern NSDS  . Appropriate reforms will be needed in several fronts such as legal basis, official statistical and open data policies, data quality assurance, data protection, statistical business processes, competencies and skills, innovation and technology, and financing. More than ever, NSOs will need to strengthen public-private partnerships with new data sources, from the traditional private and civil society organizations to data science and technology institutions and private citizens.

Concrete actions

  • Include data revolution concepts and principles in the assessment phase of the NSDS
    • Identify data ecosystem stakeholders and assess their data needs. Step 1.1 | Step 3.3
      • Analyze need for smart statistics, including Big Data, citizen-generated data, and other data from non-traditional sources.
    • Assess current statistical output in relation to smart statistics and the data ecosystem. Step 3.2
      • Compare current statistical outputs and desired smart statistics.
      • Assess viability of potential new data sources, including Big Data, citizen-generated data, and other data from non-traditional sources.
    • Assess the NSS in terms of 
      • readiness (institutional resources and capacities/competencies and skills) and 
      • openness (legal framework, policies, standards, systems, and technologies) to address the data ecosystem. Step 3.1
        • Prioritize the NSO and data producers in priority sectors such as education, health, agriculture, macroeconomy and finance, labour and employment, prices, income, and poverty, environment, among others, in the organizational assessment.
  • Consider the characteristics of a modern NSS in the visioning exercise, with consideration of NSS readiness and openness. Step 4.1
  • Identify appropriate strategic goals and key outputs to address smart statistics, including the use of Big Data, citizen-generated data, and other data from non-traditional sources, in key or priority sectors or subject-matter areas. Step 4.2
  • Identify concrete actions to address smart statistics in key or priority sectors or subject-matter areas. and corresponding costs as well as key risk factors and mitigating measures at the national, sectoral/subject-matter, and agency levels. Step 5.1 | Step 5.2 | Step 5.3  
    • Key considerations
      • Quick and easy
      • Wide range of partnerships (e.g., work with data scientists and software application developers on use of different data sets, data analytics, mapping, and infographics, etc.)
      • Well-managed reforms.
  • Establish institutional partnership mechanisms (policies, standards, and arrangements) between NSS and data ecosystem stakeholders, initially in priority sectors and/or subject-matter areas. Step 6.3
    • Coordinate with the relevant state authority on overall Open Data (or open government) policy and plans that may impact existing and new data sources.
    • Consider partnerships with Big Data sources (e.g., data science firms, telecommunication companies, and other service providers) for data and technology sharing. 
    • Study and consider existing systems of civil societies including citizen-generated data.
  • Monitor and evaluate milestones and outcomes of initiatives in developing and integrating new data sources and stakeholders. Step 6.4 | Step 7.2 | Step 7.3
    • Identify learnings and areas for improvement.