Research & Development: Systematic Evaluation Process

1. Are the research priorities identified the right ones?

PHMSA identifies the right priorities and validates which research topics to solicit through sponsoring a periodic R&D forum with its stakeholders. This allows government and industry pipeline stakeholders to develop a consensus on the technical gaps & challenges for future R&D. It also reduces duplication of programs, factors in ongoing research efforts, leverages funds and broadens synergies. Stakeholder involvement is very important so PHMSA formed a “Blue Ribbon Panel” comprised of high level representatives from Federal & State agencies and industry operators and trade organizations. This panel met in June 2003 to ensure that the R&D program design is aligned with the needs of the pipeline safety mission, makes use of the best available knowledge and expertise, and considers stakeholder perspectives. Periodic future meetings will continue vetting program processes and further refine the program. PHMSA periodically coordinates with its pipeline safety inspectors and the National Association of Pipeline Safety Representatives (NAPSR) at the R&D forum to establish the right regulatory priorities which research can help solve.

2. Are the research contractors selected best suited for these priorities?

A time tested competitive merit review process assures PHMSA research awards are best suited to address identified priorities. A merit review panel comprised of representatives from federal & state agencies and industry operators and trade organizations use strong evaluation criteria to review research white papers and proposals. In addition, 30% resource sharing between the government and any non-federal source is required which forces researchers to organize with credible groups increasing the credibility and applicability of the proposed work.

3. Are the awarded projects performing well?

A Management Information System (MIS) was developed and utilized to assure awarded projects are performing well. The MIS electronically monitors and tracks contractor performance as the project moves toward completion. This system provides the necessary oversight so specific contractual milestones and accounting are systematically followed as prescribed in the award documents. The system was designed to improve and maintain program quality, efficiency, accounting and accountability. Additional oversight is provided by Agreement Officer's Technical Representatives (AOTR) who are trained, certified, and designated to each project in accordance to the Federal Acquisition Regulations. The AOTR also coordinates with research project co-funding organizations keeping lines of discussion open about project status and quality.

4. Are the program outputs of high quality?

A peer review process was designed to validate program outputs are of high quality. Guidance was reviewed and followed from the Government Accountability Office and Office of Management and Budget (OMB) to aid in the development of the process. PHMSA is also working with the Department's Research and Innovative Technology Administration (RITA) to insure the peer review requirements will allow PHMSA to receive a "Green" from OMB. Feedback from our stakeholders is also important so a specific barnstorming session on peer reviews was designed into the March 2005 R&D Forum’s agenda. Feedback noted that peer reviews are important to validate quality but also to keep research targeted at achieving goals and to facilitate knowledge transfer of results. Several perspectives were gathered and drove the annual peer review approach now implemented. Peer review of the program is managed through RITA for all Department modals to address the OMB R&D Investment Criteria within the Performance Assessment Rating tool (PART). A report documenting the project and program level peer review results will be available to the public on the program website. The R&D Forum also provides a level of peer review because projects are presented to large audiences of various technical backgrounds.

5. Are the program outputs applied to intended users?

Quantifying desired impacts on research program objectives validates how outputs are applied to intended users. Several specific systematic process features were put in place to generate quantifiable data. Within the MIS, this data is collected, tracked and will be reported on in an annual performance report available to the public on the program website. PHMSA is applying research outputs to pipeline safety inspectors by employing them as AOTRs on project awards. This practice has been expanded from just one AOTR in 1999 to over forty in 2005. As a result, the PHMSA's pipeline safety technical knowledge base was strengthened considerably in recent years.