Research & Development: Internal, External, Stakeholder, & Peer Reviews
Review for PHMSA Research and Development Program and project activities is an essential component for output quality assurance. There are many sources of guidance for conducting legitimate External, Stakeholder and Peer reviews. Guidance has come from the President's Office of Management and Budget (OMB), the Secretary of Transportation, the National Transportation Safety Board (NTSB), and the Government Accountability Office (GAO). A number of laws and regulations codify these requirements including the Pipeline Safety and Integrity Act of 2002 (PSIA-2002), the Performance Assessment Rating Tool (PART) evaluation process for the OMB, the President's Management Agenda R&D Investment Criteria and the Government Performance and Results Act (GPRA). Congress ultimately has a significant role in evaluating the budget and the performance of PHMSA. Interagency Agreements between agencies with complementary pipeline R&D programs must be factored into reviews.
Finally, there is oversight of the safety advisory committees, State pipeline safety organizations such as National Association of Pipeline Safety Representatives, national standards organizations such as American Society of Mechanical Engineers, industry trade and technology organizations, and the general public. Over any given year, PHMSA is involved with several items that assist in providing quality assurance for R&D activities and outputs. These items can be filed into an external, stakeholder, and peer review approach which applies to both program and project level activities.
For more information on any given area of program and project review, please select one of the following specific topics:
Figure 1. Illustrates the various items which address program and project reviews. Click any box for additional information
The Pipeline & Hazardous Materials Safety Administration (PHMSA), coordinates extensively with the Research and Innovative Technology Administration (RITA). RITA is dedicated solely to the advancement of DOT priorities for innovation and research in transportation technologies and concepts. Innovations that will improve our mobility, promote economic growth, and ultimately deliver a better integrated transportation system. Because PHMSA works closely with the RITA mission, greater collaboration, information sharing, coordination, support, and advocacy for research efforts are possible.
A specific annual programmatic review is presented to the Department's Operating Administrations which allows the Department and PHMSA to receive a "Green" from OMB with the President's Management Agenda R&D Investment Criteria.
The PHMSA R&D Program has been reviewed from the Government Accountability Office (GAO), the Inspector General (IG), the Office of Management and Budget (OMB) and Congress (Senate or House). These reviews have focused on various program areas and have occurred on a periodic basis since 2002. For example the GAO issues a report in 2003 with findings that the PHMSA R&D Program needs to develop a systematic approach to evaluate project outcomes. The IG issued a report in 2004 with findings that the PHMSA R&D Program should complete the development of its web-based system for monitoring the cost, schedule and performance of research and development projects. Congress has reviewed PHMSA activities (including R&D) during two hearings in 2004. These hearings which were the Senate Full Committee Oversight Hearing on Pipeline Safety and the House of Representatives Subcommittee on Highways, Transit & Pipelines identified no major issues on PHMSA R&D Program activities. Also in 2004, Congress reviewed PHMSA interagency R&D activities from the Pipeline Safety Improvement Act of 2002 mandated submittal of the Five-Year Interagency R&D Program Plan. Using the PART evaluation, the OMB evaluated PHMSA in 2004 where the R&D Program was one component of the overall office program structure. OMB reported that the PHMSA and the programs within were moderately effective with the requirements of the PART. Feedback from these reviews have directly adjusted management approaches to raise efficiency, align the program with processes which provide relevance to the mission and vision, high quality outputs and outcomes with effective performance. Since PHMSA has a rapidly growing R&D program and that much effort has been placed in developing a National Pipeline Research Agenda, future reviews from the GAO, IG, OMB and Congress can be expected.
PHMSA has a mission to insure the safe, reliable and environmentally sound operation of the nation's pipeline transportation system. This mission requires PHMSA to address feedback and interface with several stakeholders from the general public, state and local and other Federal regulators or agencies, pipeline operators, industry trade associations, and many others. Many of these stakeholders have reviewed the PHMSA R&D Program activities on an annual or periodic basis. The following sections identify the stakeholder group and discuss with detail how they provide a review of R&D program activities.
The PHMSA Blue Ribbon Panel
Joint Meeting of the Technical Hazardous Liquid Pipeline Safety Standards Committee & Technical Pipeline Safety Standards Committee (Congressionally Mandated)
Joint Government/Industry Research & Development Forum
Pipeline Safety Improvement Act (PSIA) of 2002 Interagency Coordination Meetings
Pipeline Safety Improvement Act (PSIA) of 2002 - Five-Year Interagency R&D Program Plan & Annual Report
PHMSA R&D Program Website
Both the Office of Management and Budget (OMB) and the Government Accountability Office (GAO) have issued reports with specific guidance on how to develop peer reviews for government R&D programs. The National Academy of Science and many other government and non-government boards and panels have addressed this issue and have provided guidance as well. There is currently no recognized definition of peer review which applies across the federal government. PHMSA defines "Peer Reviews" as a process of independent reviews and evaluations which provide feedback for adjusting program management to accomplish the following: raising program efficiency, funding projects with activities which provide relevance to the mission and vision, assuring high quality project outputs and outcomes with effective performance. PHMSA has divided project peer reviews into three parts, pre-solicitation, pre-award and post-award. Each part of the project peer review provides quality assurance that R&D projects funded are appropriate for the challenges faced, of high quality and assist the achievement of program goals. The following sections identify the process of how project peer reviews are completed.
Joint Reviews of PHMSA White Papers and Proposals
For all solicited R&D topics in BAAs, PHMSA utilizes several items to facilitate that white paper and proposal submissions originate from professional, experienced and effective researchers. The solicitation states that all submissions will be held to a 50% cost share between the government and industry. This requirement forces researchers to organize with credible groups increasing the likelihood of bringing project results to commercialization. The remaining strong criteria are used to evaluate each white paper and proposal to assure the best qualified researcher can conduct the proposed scope. These criteria are consistent with the Federal Acquisition Regulations and are as follows:
1. Offeror's understanding and description of the "state of the art" in the proposed research area. The degree to which the offeror identifies the current status of the technology in the research area it is proposing to address, including previous work done by the offeror and others in the proposed R&D area and appropriateness of how the prior work relates to the proposed application of the technology. Adequacy and feasibility of the applicant's approach to achieving stated objectives.
2. Scientific and technical merit of proposal to advance pipeline safety. The degree to which the proposed technology or methodology is based on sound scientific and engineering principles. The degree to which the proposal will advance pipeline safety, if the work is successful.
3. Adequacy and feasibility of technical approach and realism of cost estimate. Appropriateness, rationale, and completeness of the applicant's technical approach. The adequacy of the proposed project schedule, staffing plan and planned travel to complete the proposed work. The realism of the cost estimate to meet objectives and deliver products in the proposed timeline. The likelihood that the proposed work will be successful and develop a new successful technology.
4. Technical experience and capabilities of the offeror in Federal research programs. Credentials, capabilities and experience of key personnel. Demonstrated corporate experience of the applicant and participating organizations in managing similar projects that are similar to the proposal in size, complexity, and technology. The adequacy of the facilities and equipment to perform project tasks.
5. Timeline to implement proposed technologies or concepts into practice into the pipeline industry. Overall potential timeline of the proposed technology to become an accepted commercial product if work is successful.
6. State of the application and coordination and collaboration with end users to implement proposed technologies or concepts into the pipeline industry. (Criteria to be used during full proposal evaluation only) If applicable, the proposed research should identify whether the deliverable addresses technology or consensus standards. The offeror could include scope items that coordinate with standard organizations and or vendor technology organizations increasing the chance that project results will be applied or commercialized. Items such as invitations to project meetings, providing copies of milestone reports and drafting project outputs in formats recognized with that organization are suggested. The offeror could provide clarity, logic and likely effectiveness of project organization including subcontractors to successfully complete the project.
7. As appropriate, information on capabilities of the lead company and other collaborators expected to participate in consolidated R&D Programs. (Criterion to be used during full proposal evaluation of Consolidated Programs only) PHMSA is looking for the increased scientific and technical merit of the consolidated program itself versus that of its individual projects. We are looking not simply for project management assistance, but for the increased value added by adopting a more integrated approach to addressing a known technical challenge to pipeline safety. Therefore, successful proposals for consolidated programs will clearly articulate both how the individual technical projects interrelate and are sequenced to maximize the benefits. Such proposals will also clearly articulate how the proposed approach will deliver significantly greater improvements in either knowledge or technical solutions to clearly identified challenges than if these projects were pursued individually or in a different sequence; the adequacy and availability of the personnel, facilities and equipment to perform project tasks; and the demonstrated corporate experience of the applicant and participating organizations in managing similar projects.
PHMSA R&D projects funded or co-funded under Interagency Agreements (IA) do not fall under the pre-award reviews described above and are considered Intramural. Historically, PHMSA has had IAs with the U.S. Minerals Management Service (MMS) and the National Institute of Standards and Technology (NIST). Projects co-funded with the MMS follow requirements under the Federal Acquisition Regulations (FAR) and are mission pertinent. In 2004 an IA was signed with NIST that itemized specific projects covering mutual areas of interest. The NIST IA facilitates collaboration as proscribed in PSIA 2002.
Post-Award - Annual Panel Peer Review
Please visit our Annual Panel Peer Review page for more information and for the outputs from each review.
Selection of Reviewers
Scope of the Review
Peer Review Reports (Source: OFFICE OF MANAGEMENT AND BUDGET)
The Contracting Officer (CO)
The Contracting Officer's Technical Representative (COTR)
Submission of Research Project Results to Public Events
PHMSA R&D Logic Model