Panel Peer Review of PHMSA Pipeline Safety Research Projects: 2017

R&D Menu

Since 2006, the Pipeline and Hazardous Materials Safety Administration’s (PHMSA) Pipeline Safety Research and Development (R&D) Program has held annual structured peer reviews of active research projects to maintain research data quality, in accordance with mandates from the Office of Management and Budget (OMB) and the Office of the Secretary of Transportation (OST). PHMSA holds these reviews virtually via teleconference and the internet in order to save time and resources. Virtual teleconferences facilitate attendance from all U.S. time zones, Canada, and Europe, making it easier for panelists, researchers, project co-sponsors, and representatives of Agreement Officers to participate.

The annual peer review continues to build on an already strong, systematic evaluation process developed by PHMSA’s Pipeline Safety R&D Program and certified by the Government Accountability Office. The 2017 peer-review panel, which consisted of six academic representatives, reviewed 18 projects using 11 evaluation criteria grouped into the following four categories:

Project management;
The method used to transfer results to end users;
Project coordination with other closely related programs;
Quality of project results.

The potential ratings assigned by the peer-review panel are: Ineffective, Effective, More than Effective, and Very Effective. The average rating for the 18 projects assessed during the May 2017 review was More than Effective; the average sub-criteria were also rated highly, underpinning these findings. Five projects were rated Very Effective and 13 were rated More than Effective. Panelists made several recommendations associated with each project during the review that were categorized into Strong and Weak points. Recommendations focused on improving what was deemed to be overall good performance. Common project weaknesses identified for projects rated More than Effective were in the categories of scientific knowledge and/or engineering principles and/or communication. The majority of the projects were approximately 70 to 90 percent complete, which means only some projects will be further studied in future reviews.

2017 Peer Review Report

Rating Scale
Very Effective	4.5 to 5.0 (8 Projects)
More than Effective	3.0 to 4.4 (16 Projects)
Effective	1.9 to 2.9 (0 Projects)
Ineffective	0.0 to 1.8 (0 Projects)
Average Program Score	4.2

Program Averages - Review Categories and Sub-Criteria
Review Categories and Sub-Criteria	Score	Rating
1. Project Management.	4.4	More than Effective
1.1. How well is the project being managed (on budget and schedule)?	4.3	More than Effective
1.2. How well is the project making progress toward the work scope objectives?	4.4	Very Effective
2. Approach taken for transferring results to end users.	4.3	More than Effective
2.1. Is there a plan for dissemination of results, including publications, and reporting?	4.1	More than Effective
2.2. How much end user involvement is incorporated into the work scope?	4.3	More than Effective
2.3. For results that may include marketable products and technologies, are commercialization or U.S. Patent plans established?	4.4	More than Effective
3. Project coordination with other related programs.	4.1	More than Effective
3.1. Does the project build on, or make use of, related or prior work?	4.3	More than Effective
3.2. Is the work of the project being communicated to other related research efforts?	3.9	More than Effective
3.3. Has consideration been given to possible future work?	4.0	More than Effective
4. Quality of project results.	4.0	More than Effective
4.1. Are the intended results supported by the work performed during the project?	4.0	More than Effective
4.2. Are the intended results consistent with scientific knowledge and/or engineering principles?	3.6	More than Effective
4.3. Are the intended results presented in such a manner as to be useful for identified end users?	4.1	More than Effective
Average Category Score and Rating:	4.2	More than Effective