Panel Peer Review of PHMSA Pipeline Safety Research Projects: 2018

R&D Menu


• Regulations
• Advisory Bulletins
• Interpretations

Since 2006, the Pipeline and Hazardous Materials Safety Administration's (PHMSA) Pipeline Safety Research and Development (R&D) Program has held annual structured peer reviews of active research projects to maintain research data quality, in accordance with mandates from the Office of Management and Budget (OMB) and the Office of the Secretary of Transportation (OST). PHMSA holds these reviews virtually via teleconference and the internet in order to save time and resources. Virtual teleconferences facilitate attendance from all U.S. time zones, Canada, and Europe, making it easier for panelists, researchers, project co-sponsors, and representatives of Agreement Officers to participate.

The annual peer review continues to build on an already strong, systematic evaluation process developed by PHMSA's Pipeline Safety R&D Program and certified by the Government Accountability Office. The 2018 peer-review panel, which consisted of three academic representatives, reviewed five projects using 11 evaluation criteria grouped into the following four categories:

  1. Project management;
  2. The method used to transfer results to end users;
  3. Project coordination with other closely related programs; and
  4. The quality of project results.

The potential ratings assigned by the peer-review panel are: Ineffective, Effective, More than Effective, and Very Effective. The average score for the five projects assessed during the May 2018 review was More than Effective; the average sub-criteria were also rated highly, underpinning these findings. All peered projects and the overall program retained the rating of More than Effective-the same rating received in 2017. At the time of the reviews, the majority of the projects were approximately 60 to 90 percent complete. The panelists made several recommendations associated with each project during the course of the review that were categorized into Strong and Weak points. However, none of these comments identified the critical actions required to salvage a project from failure, but instead recommended actions to further improve good performance.

Rating Scale
Very Effective4.5 to 5.0 (8 Projects)
More than Effective3.0 to 4.4 (16 Projects)
Effective1.9 to 2.9 (0 Projects)
Ineffective0.0 to 1.8 (0 Projects)
Average Program Score4.2

Program Averages - Review Categories and Sub-Criteria
Review Categories and Sub-Criteria Score Rating
1. Project Management. 4.3 More than Effective
  1.1. How well is the project being managed (on budget and schedule)? 4.2 More than Effective
  1.2. How well is the project making progress toward the work scope objectives? 4.5 Very Effective
2. Approach taken for transferring results to end users. 4.2 More than Effective
  2.1. Is there a plan for dissemination of results, including publications, and reporting? 4.1 More than Effective
  2.2. How much end user involvement is incorporated into the work scope? 4.3 More than Effective
  2.3. For results that may include marketable products and technologies, are commercialization or U.S. Patent plans established? 4.3 More than Effective
3. Project coordination with other related programs. 4.1 More than Effective
  3.1. Does the project build on, or make use of, related or prior work? 4.4 More than Effective
  3.2. Is the work of the project being communicated to other related research efforts? 4.1 More than Effective
  3.3. Has consideration been given to possible future work? 3.9 More than Effective
4. Quality of project results. 4.3 More than Effective
  4.1. Are the intended results supported by the work performed during the project? 4.4 More than Effective
  4.2. Are the intended results consistent with scientific knowledge and/or engineering principles? 4.3 More than Effective
  4.3. Are the intended results presented in such a manner as to be useful for identified end users? 4.2 More than Effective
Average Category Score and Rating: 4.2 More than Effective