Evaluating and monitoring for impact: developing a framework for risk prevention programs | Natural Hazards Research Australia

Evaluating and monitoring for impact: developing a framework for risk prevention programs

Photo: Matt Palmer, unsplash
Project type

Core research

Project status

Expressions of Interest

This research project - through the evaluation of two contrasting case studies, an existing risk prevention program and the piloting of a novel program - will establish a national monitoring and evaluation framework for risk prevention programs which can be used by practitioners to understand the efficacy of their programs. 

This project is currently open for Expressions of Interest.  

Project teams responding to this call for Expressions of Interest are required to submit their response using the Centre’s current EOI submission form, downloadable from the top right corner of this page.  

EOI proposals are due by 5pm AEDT, Thursday 7 December 2023 to research@naturalhazards.com.au. Successful applicants will be notified by the end of December 2023. 

An online project briefing was held on Monday 9 October to provide a more detailed briefing of the project and the opportunity for interested parties to pose specific questions. View the recording below. 

Project details

This project will establish a national framework for monitoring and evaluating risk prevention programs through: 

  • Work package 1: Evaluating the impact of the Queensland Fire and Emergency Services’ Fight Fire Fascination program, where fire services professionals supports parent and educators to teach children about fire safety through outreach. This package will seek to understand: 
    • the efficacy of the program against objectives, content, delivery and training 
    • the impact of the program on fire play and other behaviours 
    • how to monitor the efficacy of the program on an ongoing basis and build the capacity of staff to do so. 
  • Work package 2: Piloting and evaluating a novel program seeking to create targeted disaster preparedness videos delivered through wi-fi captive portals. The concept is designed to provide tourists and tourism workers, who are traditionally harder to reach with preparedness strategies, with targeted information on how to prepare for disasters in their area. Through the pilot, research will work closely with stakeholders at the sites selected for the pilot, to create buy-in and engagement. This package of work will create an understanding of: 
    • the effect of wi-fi captive portal videos have on disaster preparedness knowledge and perception of risk in tourists and tourism workers 
    • how tourists, tourism workers, and tourism businesses perceive and relate to disaster preparedness information delivered through Wi-Fi captive portal videos 
    • how to enhance wi-fi captive portal videos to improve disaster preparedness and user experience 
    • how the effectiveness and cost/benefits of wi-fi captive portal videos compare with disaster preparedness information delivered to transient populations through other channels 
    • how to pilot programs in a manner that can support future scaling to other applications and jurisdictions. 
  • Work package 3: The use of the two contrasting case studies, in combination with a literature review, will be used to create a recommended approach for the monitoring and evaluation of similar programs across jurisdictions. Researchers will disseminate their recommendations on monitoring and evaluation approaches to other agencies running risk prevention programs.

Frequently asked questions

Q) Can NHRA connect interested parties with other agencies or organisations involved in developing this project, to discuss it further and assist in developing an EOI submission? 

A) While an EOI is open for submissions, NHRA procurement processes don't allow NHRA staff to connect interested parties, as this can cause conflicts of interest. However, interested parties are able to contact whomever they wish, including those organisations listed on the EOI, to discuss the project and explore opportunities for collaboration or potential consortium submissions. If an online project briefing has been organised for an EOI, we encourage interested parties to attend the briefing to ask questions and meet other interested parties. 

Q) Can interested parties contact NHRA with questions about the project design before submitting?

A) ​While an EOI is open for submissions, NHRA procurement processes don't allow NHRA staff to share knowledge of the project design directly with interested parties. However, if you have questions, you can email research@naturalhazards.com.au and you will receive a timely response if appropriate, according to our procurement processes. Interested parties are also encouraged to contact organisations listed on the EOI to discuss the project design. 

Q) Is there a preference for a project team to be from a single research organisation, or from across multiple organisations? 

A) The Centre has no preference for either a single organisation or a multi-organisation project team. EOIs will be accepted from either and will be evaluated against the evaluation criteria in the same way. 

Q) Can the proposed project team include researchers from government agencies or research consultancies, as well as university-based researchers? 

A) EOIs will be accepted from multi-organisation project teams, and project teams can include researchers at government agencies and research consultancies. The proposed project team's capacity to undertake the project will be evaluated in the same way whether researchers are university-, consultancy- or agency-based. 

Q) Would academic salaries (excluding administrative overheads) be eligible to budget? 

A) Yes, academic salaries can be included within the project budget, but only when they are direct project costs and their salary is not already covered elsewhere. E.g. the salary costs of a contract researcher who is actively working on the project can be included up to the FTE component they are contributing. However, the salary costs of a tenured academic who is already paid through the university/research organisation would be included as an in-kind contribution up to the amount of their FTE contribution to the project. 

Q) What do you mean by "peer review"? 

A) This is a quality control process. The final report must be reviewed by someone who is an expert in the field and independent of the project.  

Q) At what rate can salary on-costs be charged? 

A) Administrative overheads are limited to direct salary-related on-costs. Indirect cost recoveries are not to be included. Other administrative overheads and indirect costs can be included as in-kind contributions. Where the salary on-cost rate exceeds 28% this will need to be justified.  

Q) Can equipment costs be included in the budget? 

A) Ordinarily, project funds are not to be used to purchase equipment. Where funds for equipment are requested, they will need to be justified 

Q) What is the budget for the project? 

A) The Centre has not specified a budget for this project intentionally. We request applicants to proposed a budget for what they think they project required to meet the outcomes. The research team should note and consider the value for money criteria when proposing their budget. NHRA envisages video production costs would be capped at $15k a video, this should be represented in EOI budgets.

Q) Can international research teams apply? Alternatively, can international research teams be part of a consortium bid? 

A) International research teams can be part of research projects when they are part of a consortium submission that is led by an Australian research organisation. The international team would need to be subcontracted by the lead Australian based research organisation. Please note that all budget submissions must be in Australian dollars and the lead organisation must bear the cost of funds transfer and responsibility for due diligence as required under Australian Foreign Interference regulations. 

Q) Is there an option of registering my interest as an independent researcher that could assist a larger team or are you only able to consider proposals that address the entire project? 

A) Yes, you can submit an EOI as an independent researcher addressing part of the project requirements. In the EOI submission form, please outline which aspects of the project you can address. Once the EOI closes, we can consider linking researchers together, although whether or not we can do that will depend on a range of factors, so we can’t guarantee that outcome. Of course, the other option is to reach out to potential collaborators now about putting in a joint EOI submission that addresses the entire project. 

Q) What are the key questions that the first WP: Fight Fire Fascination seeks? 

A) The FFF by the QFES would like to know if the program is working and having an impact on the participants. They would like to know if/how their program approach leads to attitudinal change.  

Q) Does Work Package 3 require visual material/presentation? 

A) We want this project to be accessible, having an end-user insight led training. Hence, developing a training utilising visual presentation is possible.  

Q) Is the program interested in producing simulation games and/or crisis manual as part of the review and monitoring aspect of the program? 

A) Simulation, which incorporates all potential scenarios of a given event, could be an output to creatively address the aims of this project. The end-product of the program should consider budget constraints. The solutions should be user driven. Pilot testing should be done before iteration and alternative delivery models explored.  

Q) Which aspect of the program will have the greatest importance and budget allocation: outcomes, framework and/or training component? 

A) It is entirely dependent on the researchers on which aspect they will put priority in the budget. Ultimately, the goal is to have a well distributed budget that can address the aims of the program. The submissions will be assessed on how the money will be spent realistically. 

Q) Can a consultancy firm be lead organisation for the project? 

A) NHRA acknowledges that the work required may be beyond the skill set of one organisation. Hence, a consortium is encouraged. NHRA will contract a lead organisation that would subcontract other parties.   

We are not able to put together different groups while the EOI is open. Thus, we encourage that interested parties who would like to work with others and share their email addresses in the chat during the briefing.