JOSM External Expert Peer Review of Scientific Integrity - Executive Summary
The oil sands in Alberta represent the third largest proven reserve of oil in the world. Sound environmental monitoring of air, water and ecosystem health is essential to assess the impacts of oil sands development over time. In 2012, the Governments of Canada and Alberta launched the Joint Oil Sands Monitoring (JOSM) to improve, consolidate and integrate the existing disparate monitoring arrangements into a single, transparent government-led approach with a strong scientific base. The Joint Canada-Alberta Implementation Plan for Oil Sands Monitoring (“the Implementation Plan”) describes a phased and adaptive approach to monitoring in the oil sands region, to be undertaken over three years (2012 to 2015).
The Panel’s mandate and approach
The Implementation Plan calls for the monitoring system to undergo external expert peer review after year three, to ensure that scientific integrity is maintained. An external panel of six experts (the Panel) was assembled to cover the four monitoring component areas: air, water, wildlife contaminants and toxicology, and biodiversity and land disturbance. The Panel was asked to address the following question:
What progress has JOSM implementation made in improving the scientific integrity of environmental monitoring of the impacts of oil sands development, and can it be demonstrated?
For the purpose of the review, “scientific integrity” was defined as rigour, the use of internationally recognized quality-assured and quality-controlled standards and protocols, and transparency. The Panel was also asked to recommend improvements as needed.
To conduct its review, the Panel reviewed a number of documents and interviewed key stakeholders responsible for the delivery and management of JOSM. The work of the Panel was made more challenging by the absence of an overarching document that clearly articulates the policy and scientific goals of the Governments of Canada and Alberta for oil sands monitoring. It was difficult for the Panel to fully assess the advancements in scientific integrity made by JOSM and the suitability of the monitoring approach without understanding these goals.
JOSM has made good progress since 2012 in improving the scientific integrity of oil sands monitoring but more work is needed going forward.
JOSM has made significant advances over the monitoring in place prior to 2012 by establishing more robust and comprehensive monitoring and improving the rigour, transparency and adherence to internationally recognized standards and protocols of the monitoring. The number of sampling sites and the sampling frequency have increased, geographical coverage has widened, more chemical contaminants are now being analyzed and more ecological studies have been added. Progress has been made in implementing unified and consistent monitoring approaches and making monitoring data more accessible to stakeholders. The shift in responsibility for monitoring and distribution of funds supporting monitoring activities from industry to the recently created Alberta Environmental Monitoring, Evaluation and Reporting Agency (AEMERA) reduces a potential conflict of interest on the part of monitoring organizations and may provide greater confidence in the impartiality of the monitoring results.
However, more work needs to be done to establish a fully integrated and harmoniously functioning long-term monitoring system. The next three sections outline the Panel’s findings on rigour, standards and protocols, and transparency, the key measures of scientific integrity for this review.
Monitoring efforts are largely addressing relevant questions, using sound scientific methods and appropriate and consistent methodologies.
The following examples illustrate this finding:
Air: Measurements undertaken to monitor pollutant transformation are well suited to gain a better understanding of local air quality and atmospheric transformation processes experienced by the pollutants emitted from multiple sources over the short to medium term.
Water: The reach-based and tributary water and sediment sampling is entirely appropriate for addressing the science questions and for the integrated assessment of downstream impacts.
Wildlife contaminants and toxicology: This component has successfully initiated a broad program to examine the potential effects of chemical contaminants from oil sands development on the native biota coupled with selected laboratory exposure studies.
Biodiversity and land disturbance: Data collection in this component is well thought-out. It was also prudent for core terrestrial biodiversity monitoring activities to piggyback on an existing province-wide monitoring program and to strengthen it to address the Implementation Plan objectives.
Limited comprehensive analysis of the data collected poses challenges for determining whether the current level of monitoring is appropriate for assessing the ecosystem impacts caused by oil sands development.
Despite the wealth of data collected from 2012 to 2015, the full dataset has not been regularly analyzed and interpreted or this analysis is not readily or consistently available. Thus, there is insufficient evidence to assess whether current monitoring activities are adequate for a full evaluation of the environmental impacts of the oil sands. For instance, in the case of work on sediment under the water component, only a detailed data analysis will indicate the appropriateness of the sampling frequency. The Implementation Plan cannot be truly adaptive without ongoing comprehensive analysis of results.
Although the Implementation Plan identifies analysis and incorporation of historical monitoring data as one of its goals, there is limited evidence of efforts to evaluate and use the data collected before 2012. Assessment of the pre-JOSM data is needed to provide a historical basis for comparison and extend the data time series to better understand long-term trends and changes in the ecosystem.
Integration of activities and reporting within and across the four monitoring components is limited.
The limited comprehensive data analysis so far means that JOSM has made limited progress on harmonizing and rationalizing pre-existing disparate monitoring approaches and activities. The four monitoring components also remain fairly separate from one another. For example, there is no definition of the level of information on air and water quality, including precipitation chemistry, needed to support the evaluation of the ecosystem changes in terms of wildlife and terrestrial biodiversity. Although it may be less important at this stage to integrate biodiversity and land disturbance monitoring with the monitoring of air and water, there are several opportunities to further the integration of activities and results across JOSM. One of the only substantive examples of integration is the work on deposition of pollutants, which brings together air, water and ecosystem health. However, this work is still in its infancy.
The Panel also found that monitoring efforts within a component area were not always integrated. For example, there remain inconsistencies in methods used to assess oil sands development on wetland vegetation and effects of potential contaminants on specific indicator plant species. In air quality monitoring, the ongoing work on the emissions inventory is not yet strongly integrated with modelling and remote sensing efforts.
Standards and protocols
Air, water, wildlife contaminants and biodiversity monitoring generally use internationally recognized standards and protocols.
For air and water monitoring in particular, the methodologies described in project plans, reports and scientific papers are robust and meet international standards. Methods for biodiversity assessment, however, tend to be less standardized than those for chemical analyses, for example. Standard operating procedures (SOPs) are not always available for many of the ecological studies (e.g., wildlife toxicology), perhaps because of their nature, which meant that the Panel could not always readily ascertain how these data were obtained.
JOSM lacks a fully documented QA process and a uniform QA approach.
The Panel found good evidence supporting quality assurance (QA) for the air and water components. However, across the four monitoring components, many monitoring activities are not documented with sufficient details and lack QA plans (with perhaps the exception of routine air monitoring). JOSM has not yet implemented a uniform QA approach across its activities and the Panel found limited evidence of external QA audits for the four components. There is also no single document that explains the various QA processes that are in place.
The JOSM Information Portal is an excellent step forward in making monitoring data available to a wide range of stakeholders.
The sharing of data through a single vehicle, the Canada-Alberta Oil Sands Environmental Information Portal (JOSM Information Portal), is intended specifically to improve transparency and increase public confidence. The portal currently contains background information on JOSM, data and associated information from the four monitoring components, documents and publications, and other news and information relevant to JOSM.
The data and information available to stakeholders are not always comprehensive or easy to use and understand.
The data presented in spreadsheets on the JOSM Information Portal can be interpreted only by trained individuals due to the portal’s limited search and exploration functions. Since, as noted above, limited comprehensive data analysis has been undertaken to date, it may be difficult for the general public and other stakeholders to understand and use the data. The portal is also not fully populated. For example, many protocols describing methodologies, data, maps and graphs provided to the Panel for this review are not yet available on the portal.
In contrast, the JOSM annual reports mostly present high-level information on monitoring activities without details of the measurement results and interpretation. Many of the analyses done were associated with scientific papers or presentations made at international meetings and conferences. However, there is no single document, released annually or at another appropriate frequency, that integrates all the pieces into a set of major findings that can be easily understood by a broad range of stakeholders. There is also no single document that regularly tracks progress towards achieving the larger goal of assessing the environmental impacts of oil sands development.
The Panel developed the following recommendations to improve the scientific integrity of JOSM.
Better define and document specific policy and scientific goals of the Governments of Canada and Alberta for the monitoring of the oil sands.
Development of a planning document that clearly articulates the policy and scientific goals of JOSM and drives the scientific basis of the monitoring would enhance the scientific integrity of the monitoring system. This document could also support future assessments and external peer reviews of scientific integrity. An analysis of progress against identified scientific and policy goals could also support the establishment of an adaptive monitoring system. Indeed, as the monitoring progresses and results are obtained, these goals should be periodically assessed and modified.
Conduct more comprehensive data analysis and interpretation.
Comprehensive data analysis could further improve the scientific integrity of JOSM and allow for better assessment of progress in the future:
- JOSM-collected data: After three years in operation, there likely is a long enough time series and enough data collected to do a comprehensive analysis, which should inform directions for the next five years.
- Pre-JOSM data: Analyzing and assessing the historical data gathered by other monitoring bodies before 2012, and incorporating them into a longer time series, would help detect slow changes over time. Although this work would require a substantive effort, the Panel pointed to proven methodologies, e.g., air quality monitoring in the European Union.
- Ongoing data analysis: To meet the scientific goals of JOSM going forward, comprehensive data analysis and interpretation should be performed at an appropriate frequency. The results should be made available to stakeholders.
The Panel recommended a flexible approach so that as analyses and interpretation are completed, monitoring can be adapted to enhance understanding and fill gaps.
Take the necessary steps to enhance the integration of the monitoring within and across the four components.
More frequent and comprehensive analysis of data to inform the development of new science and policy questions could drive better integration of monitoring within components and vice-versa. For example, for wildlife monitoring, better integration could improve sampling efficiency and interpretation of results.
Integration across components would help to maximize the value of the data collected and better quantify the ecosystem changes that are occurring. For example, as more is learned about levels of air and water contaminants and their spatial and temporal distribution, better alignment with monitoring of biodiversity and land disturbance could be planned. Integration could be supported further by establishing regular science meetings for individuals involved in oil sands monitoring across the four components.
Develop and document a uniform QA approach that is implemented and tracked across all monitoring activities.
A more rigorous approach to the QA process, including full independent auditing, could likely increase stakeholder confidence in the reported data. Monitoring in the four component areas should be the subject of applicable QA processes. A suite of QA documents could include an overall QA program plan, detailed SOPs and a QA annual report documenting the implementation of the program plan and the completion of any QA audit.
Make monitoring data and information more readily available and accessible to stakeholders.
Transparency would be enhanced if the JOSM Information Portal made available all relevant monitoring information, including data, maps, graphs, SOPs, QA information and project plans. Greater value could also be extracted from the data on the portal if users had access to better search capabilities and simple exploration or visualization tools. Environment and Climate Change Canada (ECCC) and AEMERA could work with key stakeholders to better understand their information needs and redesign the portal accordingly.
To make monitoring data and information more readily accessible, ECCC and AEMERA could prepare and publicly release a results report that interprets and integrates monitoring results for air, water, wildlife contaminants and biodiversity. Ideally this report would be released every year to complement the annual report or, at a minimum, every few years. This could also support better integration across the four components.
The way forward
JOSM was envisioned as an ongoing long-term monitoring effort. The Panel recognized that 2012 to 2015 was a period of transition during which those engaged in JOSM worked to integrate the previous disparate programs and to implement new monitoring to measure the environmental impacts of oil sands development.
Looking forward, implementing the Panel’s recommendations to improve the scientific integrity of monitoring would greatly enhance the value of JOSM to its stakeholders. The first step would be to clearly define JOSM’s scientific objectives for the next 5 years, and then for the next 10 years and 25 years. Operating plans could be effectively developed to provide the data necessary to achieve those objectives within a comprehensive quality-assured framework. Within this framework, there needs to be better integration of the components, more effective and timely analysis of all data, and provision of policy-relevant information that is easily and fully accessible to all interested parties. Quality-assured data and associated analyses should also be made available. JOSM would then be able to fulfil its promise to provide a sound basis for decision-making on the environmental impacts of oil sands development.
- Date Modified: