Ensuring The Quality of Data Collected By Observers: The Debriefing Process
Debriefer Marlon Concepcion and observer John Anderson review data.
The Fisheries Monitoring and Analysis (FMA) Division’s North Pacific Groundfish Observer Program supplies data needed for management of the groundfish fisheries in the Bering Sea and Gulf of Alaska. Our success is dependent on the quality of the data collected by observers. A major focus of FMA is to ensure quality data are collected by observers using scientifically valid methods. While observer training prior to deployments ensures that methodologies are standardized, the quality and reliability of the data collected cannot be fully assessed by FMA until observers complete a debriefing with staff at the conclusion of deployments. In 2007, FMA staff conducted 610 post deployment debriefings reviewing information collected from over 50,000 hauls.
Debriefings are always in person, and this provides important face-to-face interactions between FMA staff and observers. Given the remote working environment in Alaska, the in-person debriefing contact is an opportunity to review the at-sea work in depth and discuss any unusual incidents or sightings at sea. Staff gain good knowledge of the observer’s work performance and the observer receives feedback for future work.
Debriefing consists of five main components: 1) completion of an electronic survey for each vessel or processing plant assignment during the observer’s deployment, 2) an interview with staff, 3) a thorough data check, 4) documentation of the observer’s work and performance in a written evaluation, and 5) completion of a feedback questionnaire.
Completion of a survey systematically captures detailed descriptions of each deployment to a vessel or processing plant. Questions in the survey cover sampling methodologies, difficulties encountered during the cruise, potential violations, protected species interactions, and other details pertinent to the data collected. Once the survey is completed, observers submit their data and are scheduled for a staff interview.
During the interview, the observer’s surveys are reviewed, and any necessary corrections or clarifications are completed. All data collected by the observer are reviewed for completeness, accuracy, and format, and all samples are submitted. The methodologies used are discussed to ensure our sampling protocols were followed. The interview is intended to be educational with staff providing feedback to the observer regarding work accomplished and suggestions for future deployments.
Although observer data are quality control checked throughout each deployment, it is at the debriefing where a final data check is completed and corrections are made. The data checking process is extensive. For example, we check for unlikely geographic positions, specimens recorded that are outside of the expected length or weight ranges, mismatches between different forms, and missing data entries. Staff work with the observer to resolve each error.
At the close of the debriefing, the observer receives a written evaluation of his/her work performance. This feedback to the observer is essential to future deployments, and we identify any future training needs.
Last, observers have an opportunity to complete a final questionnaire where they can give anonymous feedback to NMFS with suggestions for improvement. Managers periodically review this survey to assess the comments.
Debriefings are labor intensive and are essential to maintain high quality data for fisheries management. The debriefing process lasts from a few hours to several days, depending on the number of vessel or processing plant assignments, the completeness of the surveys, and the performance of the observer. Unfortunately, the debriefing workload comes in large surges as observers complete their field work. FMA shifts staff to assist in the surge workload while continuing to maintain a quality process.
