Research & Development: Internal, External, Stakeholder, & Peer Reviews
R&D Menu
For more information on any given area of program and project review, please select one of the following specific topics:
Internal Reviews
External Reviews
Stakeholder Reviews
Peer Reviews
Other
Figure 1. Illustrates the various items which address program and project reviews. Click any box for additional information
The Pipeline & Hazardous Materials Safety Administration (PHMSA), coordinates extensively with the Research and Innovative Technology Administration (RITA). RITA is dedicated solely to the advancement of DOT priorities for innovation and research in transportation technologies and concepts. Innovations that will improve our mobility, promote economic growth, and ultimately deliver a better integrated transportation system. Because PHMSA works closely with the RITA mission, greater collaboration, information sharing, coordination, support, and advocacy for research efforts are possible.
A specific annual programmatic review is presented to the Department's Operating Administrations which allows the Department and PHMSA to receive a "Green" from OMB with the President's Management Agenda R&D Investment Criteria.
Programmatic Presentations:
- RD&T Program Review - October 14, 2010
- RD&T Program Review - May 23, 2007
- RD&T Program Review - May 9, 2006
- RD&T Program Review - June 23, 2005
The PHMSA R&D Program has been reviewed from the Government Accountability Office (GAO), the Inspector General (IG), the Office of Management and Budget (OMB) and Congress (Senate or House). These reviews have focused on various program areas and have occurred on a periodic basis since 2002. For example the GAO issues a report in 2003 with findings that the PHMSA R&D Program needs to develop a systematic approach to evaluate project outcomes. The IG issued a report in 2004 with findings that the PHMSA R&D Program should complete the development of its web-based system for monitoring the cost, schedule and performance of research and development projects. Congress has reviewed PHMSA activities (including R&D) during two hearings in 2004. These hearings which were the Senate Full Committee Oversight Hearing on Pipeline Safety and the House of Representatives Subcommittee on Highways, Transit & Pipelines identified no major issues on PHMSA R&D Program activities. Also in 2004, Congress reviewed PHMSA interagency R&D activities from the Pipeline Safety Improvement Act of 2002 mandated submittal of the Five-Year Interagency R&D Program Plan. Using the PART evaluation, the OMB evaluated PHMSA in 2004 where the R&D Program was one component of the overall office program structure. OMB reported that the PHMSA and the programs within were moderately effective with the requirements of the PART. Feedback from these reviews have directly adjusted management approaches to raise efficiency, align the program with processes which provide relevance to the mission and vision, high quality outputs and outcomes with effective performance. Since PHMSA has a rapidly growing R&D program and that much effort has been placed in developing a National Pipeline Research Agenda, future reviews from the GAO, IG, OMB and Congress can be expected.
PHMSA has a mission to insure the safe, reliable and environmentally sound operation of the nation's pipeline transportation system. This mission requires PHMSA to address feedback and interface with several stakeholders from the general public, state and local and other Federal regulators or agencies, pipeline operators, industry trade associations, and many others. Many of these stakeholders have reviewed the PHMSA R&D Program activities on an annual or periodic basis. The following sections identify the stakeholder group and discuss with detail how they provide a review of R&D program activities.
The PHMSA Blue Ribbon Panel
The PHMSA Blue Ribbon Panel is called to order periodically when topics within the R&D Program are identified which require stakeholder discussion and feedback. Historically, these topics have been on research program direction, program approach and priorities, as well as specifically on how to better measure the effectiveness of its R&D investment. The Blue Ribbon Panel includes representatives from several Federal, State and Local governments, and various liquid and gas pipeline industry associations.
Joint Meeting of the Technical Hazardous Liquid Pipeline Safety Standards Committee & Technical Pipeline Safety Standards Committee (Congressionally Mandated)
This joint committee meeting is an annual public event where the PHMSA presents several current or planned actions which have implications to industry operations. R&D Program activities and direction are also presented and provide feedback to PHMSA management and decision makers. For instance, feedback from this event assisted the PHMSA in the development of the Interagency Five-Year R&D Plan and program direction. The joint committees include representatives from several Federal, State and Local governments, and various liquid and gas pipeline industry trade associations.
Joint Government/Industry Research & Development Forum
The R&D Forum is held on a periodic basis to set a national pipeline research agenda. The objective of the forum is to allow government and industry pipeline stakeholders to develop a consensus on the technical gaps & challenges for future R&D. It addresses both short and long term research objectives for liquid and gas and transmission and distribution pipelines, covering onshore, offshore and Arctic environments. In addition, details of the ultimate research goals, technology demonstrations, and transfer and commercialization are discussed. Feedback received from this event directly contributes planning guidance for program and project level direction. PHMSA utilizes these identified gaps and challenges as validation of which topics should be solicited in Broad Agency Announcements (BAA). This provides transparency on programmatic decisions to stakeholders and removes perceptions of an expert driven model.
Pipeline Safety Improvement Act (PSIA) of 2002 Interagency Coordination Meetings
The Pipeline Safety Improvement Act (PSIA) of 2002 directed the Departments of Transportation and Energy and the National Institute of Standards and Technology to carry out a program of research, development, demonstration and standardization to ensure the integrity of pipeline facilities. To coordinate on our joint requirements of the Act, quarterly meetings are held to discuss program and project level efforts. Though not named in PSIA 2002, the Minerals Management Service is involved with the coordination meetings because of their similar R&D program focus.
Pipeline Safety Improvement Act (PSIA) of 2002 - Five-Year Interagency R&D Program Plan & Annual Report
The Pipeline Safety Improvement Act (PSIA) of 2002 directed the Departments of Transportation and Energy and the National Institute of Standards and Technology to carry out a program of research, development, demonstration and standardization to ensure the integrity of pipeline facilities. An initial Five-Year Interagency R&D Program Plan was jointly developed and submitted to Congress. By December 17 of every year after, an update report of the plan will submitted to Congress. This report identifies the coordination, collaboration and co-funding efforts by the participating agencies (to include the Minerals Management Service) to meet the requirements of the Act. In addition to the mandated requirements, the participating agencies agree to work together on the development and application of performance measures to evaluate research effectiveness of pipeline facility research, development, and demonstration projects.
PHMSA R&D Program Website
PHMSA developed the R&D website to include information on all current and previous research program and project activities and to provide a method for feedback and comments for all stakeholders. This includes announcements, important meeting or event information and project summaries, and available public project deliverables. The website also provides a query feature to search for subject projects and a map illustrating where current efforts are being conducted.
Both the Office of Management and Budget (OMB) and the Government Accountability Office (GAO) have issued reports with specific guidance on how to develop peer reviews for government R&D programs. The National Academy of Science and many other government and non-government boards and panels have addressed this issue and have provided guidance as well. There is currently no recognized definition of peer review which applies across the federal government. PHMSA defines "Peer Reviews" as a process of independent reviews and evaluations which provide feedback for adjusting program management to accomplish the following: raising program efficiency, funding projects with activities which provide relevance to the mission and vision, assuring high quality project outputs and outcomes with effective performance. PHMSA has divided project peer reviews into three parts, pre-solicitation, pre-award and post-award. Each part of the project peer review provides quality assurance that R&D projects funded are appropriate for the challenges faced, of high quality and assist the achievement of program goals. The following sections identify the process of how project peer reviews are completed.
Pre-Solicitation
PHMSA identifies priorities and validates which research topics to solicit through sponsoring a periodic R&D Forum with its stakeholders. This allows government and industry pipeline stakeholders to develop a consensus on the technical gaps & challenges for future R&D. It also reduces duplication of programs, factors in ongoing research efforts, leverages funds and broadens synergies. Stakeholder involvement is very important so PHMSA formed a "Blue Ribbon Panel" comprised of high level representatives from federal & state agencies and industry operators and trade organizations. This panel met twice since June 2003 to ensure that the R&D program design is aligned with the needs of the pipeline safety mission, makes use of the best available knowledge and expertise, and considers stakeholder perspectives. Periodic future meetings will continue vetting program process and further refine the program.
Pre-Award
A time tested competitive merit review process assures PHMSA research awards are best suited to address identified priorities. A merit review panel comprised of representatives from federal & state agencies and industry operators and trade organizations use strong evaluation criteria to review research white papers and proposals. In addition, a 50% cost share between the government and industry is required which forces researchers to organize with credible groups increasing the credibility and applicability of the proposed work.
Joint Reviews of PHMSA White Papers and Proposals
For every BAA, an independent diverse merit review panel of 7 to 10 persons is assembled to review and score submitted white papers and proposals. The review team has representatives from the PHMSA, other Federal and State agencies and from several pipeline industry operators or trade associations. Each reviewer must sign and submit to the PHMSA Contracting Officer (CO) a non-disclosure and conflict of interest form. These steps insure that the review team is based on expertise, independence and the absence of conflict of interest. The combined team provides valuable insight to ongoing industry activities and on other R&D efforts within the pipeline industry. From these insights, consensus is obtained on which white papers should be requested as proposals and finally, which proposals are technically acceptable and recommended for award. This insures that an appropriate researcher for each solicited topic is selected for award.
For all solicited R&D topics in BAAs, PHMSA utilizes several items to facilitate that white paper and proposal submissions originate from professional, experienced and effective researchers. The solicitation states that all submissions will be held to a 50% cost share between the government and industry. This requirement forces researchers to organize with credible groups increasing the likelihood of bringing project results to commercialization. The remaining strong criteria are used to evaluate each white paper and proposal to assure the best qualified researcher can conduct the proposed scope. These criteria are consistent with the Federal Acquisition Regulations and are as follows:
1. Offeror's understanding and description of the "state of the art" in the proposed research area. The degree to which the offeror identifies the current status of the technology in the research area it is proposing to address, including previous work done by the offeror and others in the proposed R&D area and appropriateness of how the prior work relates to the proposed application of the technology. Adequacy and feasibility of the applicant's approach to achieving stated objectives.
2. Scientific and technical merit of proposal to advance pipeline safety. The degree to which the proposed technology or methodology is based on sound scientific and engineering principles. The degree to which the proposal will advance pipeline safety, if the work is successful.
3. Adequacy and feasibility of technical approach and realism of cost estimate. Appropriateness, rationale, and completeness of the applicant's technical approach. The adequacy of the proposed project schedule, staffing plan and planned travel to complete the proposed work. The realism of the cost estimate to meet objectives and deliver products in the proposed timeline. The likelihood that the proposed work will be successful and develop a new successful technology.
4. Technical experience and capabilities of the offeror in Federal research programs. Credentials, capabilities and experience of key personnel. Demonstrated corporate experience of the applicant and participating organizations in managing similar projects that are similar to the proposal in size, complexity, and technology. The adequacy of the facilities and equipment to perform project tasks.
5. Timeline to implement proposed technologies or concepts into practice into the pipeline industry. Overall potential timeline of the proposed technology to become an accepted commercial product if work is successful.
6. State of the application and coordination and collaboration with end users to implement proposed technologies or concepts into the pipeline industry. (Criteria to be used during full proposal evaluation only) If applicable, the proposed research should identify whether the deliverable addresses technology or consensus standards. The offeror could include scope items that coordinate with standard organizations and or vendor technology organizations increasing the chance that project results will be applied or commercialized. Items such as invitations to project meetings, providing copies of milestone reports and drafting project outputs in formats recognized with that organization are suggested. The offeror could provide clarity, logic and likely effectiveness of project organization including subcontractors to successfully complete the project.
7. As appropriate, information on capabilities of the lead company and other collaborators expected to participate in consolidated R&D Programs. (Criterion to be used during full proposal evaluation of Consolidated Programs only) PHMSA is looking for the increased scientific and technical merit of the consolidated program itself versus that of its individual projects. We are looking not simply for project management assistance, but for the increased value added by adopting a more integrated approach to addressing a known technical challenge to pipeline safety. Therefore, successful proposals for consolidated programs will clearly articulate both how the individual technical projects interrelate and are sequenced to maximize the benefits. Such proposals will also clearly articulate how the proposed approach will deliver significantly greater improvements in either knowledge or technical solutions to clearly identified challenges than if these projects were pursued individually or in a different sequence; the adequacy and availability of the personnel, facilities and equipment to perform project tasks; and the demonstrated corporate experience of the applicant and participating organizations in managing similar projects.
PHMSA R&D projects funded or co-funded under Interagency Agreements (IA) do not fall under the pre-award reviews described above and are considered Intramural. Historically, PHMSA has had IAs with the U.S. Minerals Management Service (MMS) and the National Institute of Standards and Technology (NIST). Projects co-funded with the MMS follow requirements under the Federal Acquisition Regulations (FAR) and are mission pertinent. In 2004 an IA was signed with NIST that itemized specific projects covering mutual areas of interest. The NIST IA facilitates collaboration as proscribed in PSIA 2002.
Post-Award - Annual Panel Peer Review
Over any given year, previously awarded projects come to completion. Unfortunately, projects each have their own unique timeline of when they will submit the project deliverables. This creates some difficulty of when a peer review of projects can occur. These reviews are consistently scheduled during the 1st quarter of each calendar year. The reviews are held via the internet and conference call with a previously formed panel of reviewers.
Please visit our Annual Panel Peer Review page for more information and for the outputs from each review.
Selection of Reviewers
PHMSA will solicit and select panel reviews on the basis of expertise, independence and the absence of conflict of interest. A signed and submitted form will be required from each reviewer to legitimize an unbiased panel. The review panel will be comprised of stakeholders representing academia, engineering societies, pipeline industry operators, and trade associations. A small honorarium will also be considered for academics and retired expert panelists.
Scope of the Review
Panel members shall be given a "charge" which will identify what is expected of them and what PHMSA is looking to achieve. During the annual peer review of projects, the members of the peer review panel will be provided with a scorecard on which to rate the performance of projects and contractors. The review criteria have evolved since the initial peer review in 2006. They are specific to the year they were used and found in the output report for that review.
Peer Review Reports (Source: OFFICE OF MANAGEMENT AND BUDGET)
PHMSA will instruct the panel to supply a report that describes the nature of their review and their findings and conclusions. The report shall also disclose the names, organizational affiliations, and a short paragraph on the credentials and relevant experiences of each peer reviewer. PHMSA will be given an advanced copy of the report in order to prepare a written response to the peer review panel explaining: agreement or disagreement; any actions PHMSA has undertaken or will undertake in response to the report; and (if applicable) the reasons PHMSA believes those actions satisfy any key concerns or recommendations in the report. This written response will be included in the report appendix and posted on the PHMSA R&D website.
Exemptions
PHMSA R&D projects funded or co-funded under Interagency Agreements (IA) do not fall under the panel peer reviews described above and are considered Intramural.
Other Items That Facilitate Relevance, Quality, & Performance
The Contracting Officer (CO)
The CO is trained and certified to perform their duties as described in the Federal Acquisition Regulations (FAR). The CO provides oversight during the solicitation process, pre-award review and post-award actions maintaining competitive and fair actions. The CO also protects the government and public interest since government R&D has been funded with taxpayer dollars.
The Contracting Officer's Technical Representative (COTR)
The COTR has been trained and certified to perform their duties as described in the FAR. Each research project funded has a COTR assigned to it. Three duties that provide quality assurance of the project outcomes are monitoring that the research meets contractual requirements, reviewing all submitted deliverables for technical competency, and approving accurate and efficient milestone payment.
Technology Demonstrations
In collaboration and coordination with the other federal and private research stakeholders, PHMSA will organize and hold technology demonstrations. These demonstrations are periodically planned with rotation of the subject technology to be used. Technology demonstrations are a means of evaluating the merit of technologies that are reaching the prototype stage. Demonstrations expose the technologies to the environment in which the technology must be operated successfully. Demonstrations also promote the deployment and utilization of new technologies through observations and participation by pipeline operators, equipment vendors, standards organizations, and pipeline safety officials. Demonstrations are just one stage in a technology transfer process but can be considered a major milestone for achieving an ultimate research goal. PHMSA will organize and hold future technology demonstrations when appropriate to assist road mapping and technology transfer efforts.
Submission of Research Project Results to Public Events
Starting in FY 2004, research projects awarded from Broad Agency Announcements now have contractual requirements stipulating researchers to submit the project results to at least one public event. These events can range from pipeline industry workshops, conferences, forums, and symposiums. For many of these events, a stringent peer review occurs on abstracts and papers submitted to be presented. The review is usually comprised of knowledgeable persons with the subject technology and provides quality assurance that the best paper will be presented and published in the event proceedings. This new contractual requirement will further assist the PHMSA in providing quality assurance for R&D project activities.
PHMSA R&D Logic Model
The R&D Logic Model assists PHMSA to illustrate how program elements, performance goals and project activities are integrated to achieve program goals. It also provides planning and tracking of R&D activities on how they contribute to outputs, customers reached, short-term outcomes and long-term impacts. The logic model helps quantify the effectiveness of program and project level efforts. It constitutes a living document that will evolve as PHMSA creates actions to achieve program goals.