Conservation impact scores identify shortfalls in demonstrating the benefits of threatened wildlife displays in zoos and aquaria

Abstract Zoos and public aquaria globally display numerous wild harvested, threatened species. To validate conservation credentials, displays are often associated with research projects, educational interpretation, or conservation-related activities. However, accompanying conservation benefits are rarely assessed. In this study, an approach to evaluate conservation benefits of captive wildlife experiences is modelled by assessing four Australian aquarium displays of the Critically Endangered largetooth sawfish Pristis pristis. Conservation impact scores were calculated for research, education, and conservation-related activities. In a novel approach, sawfish-related education (gaining knowledge, changing attitudes, and intentions to change behaviours) was evaluated using a before and after study design (n = 2 229), and conservation impact scores were calculated using effect sizes. Although visitors to all aquariums demonstrated significant positive attitudinal changes, and at one site gained knowledge, no significant change in behavioural intentions were detected. Educational messages addressing attitudes and behaviours were mostly generalised and untargeted. Formative and ongoing evaluations are needed to develop and maintain targeted and relevant messages. With one exception, research projects and conservation activities were unlikely to contribute substantially to sawfish conservation due to limited support from the aquaria. We recommend that increased support is directed to projects that are targeted towards impactful conservation goals.


Introduction
Given the acceleration of biodiversity loss globally (Urban, 2015), both in situ and ex situ wildlife tourism experiences have become increasingly controversial (see Ballantyne, Packer, Hughes, & Dierking, 2007;D'Cruze et al., 2018;Moorhouse, D'Cruze, & Macdonald, 2017;Tribe & Booth, 2003). Although sustainable wildlife tourism goals incorporate the delivery of conservation, public education, and socio-economic benefits (Catibog-Sinha, 2010), and some activities potentially benefit wildlife or natural habitats (Ballantyne, Packer, & Falk, 2011, Miller et al., 2004, consequences can also be detrimental. Wild harvests for captive or semi-captive wildlife experiences directly impact populations, human interactions may cause wildlife behavioural changes or habitat modification, and the failure of re-introductions to the wild causes stress and/ or harm to released individuals Keulartz, 2015). An evaluation of both positive and negative consequences is therefore needed to determine the overall impact of tourism activities on wildlife .
Captive wildlife experiences such as zoos and aquaria clearly illustrate the potential for both substantial positive and/or negative conservation consequences. These institutions are ideally placed to promote conservation activities to the public as they attract up to 700 million people annually (Gusset & Dick, 2011), and most mission statements are indeed framed around conservation (Patrick & Caplow, 2018). Additionally, sustainable practices, research, field projects, and education programs, are promoted and encouraged by regional and global zoo and aquarium associations (Barongi, Fisken, Parker, & Gusset, 2015;Patrick & Caplow, 2018;ZAA, 2014). However, many visitors want to view and interact with animals infrequently seen in the wild (Packer & Ballantyne, 2012), and threatened species are often featured in displays. The scale of threatened species displays is significant: 15.6% of threatened bird, 23.0% of threatened mammal, and 35.6% of threatened elasmobranch (shark and ray) species are on display (Buckley, Crook, Pillans, Smith, & Kyne, 2018;Conde et al., 2013). Many of these captive populations are not self-maintaining, and displays are often supplemented by wild harvests (Murray & Watson, 2014). Although these harvests are usually far smaller in volume than harvests for other commercial purposes, it is possible that the cumulative impact of multiple sources of removal from the wild could impact the viability of some wild populations (Buckley et al., 2018).
In the context of zoos and aquaria as conservation hubs, it is crucial that the collection of wild animals for display does not contribute to the extinction risk of the species, and that education benefits of display counteract removal of individuals from the wild. To demonstrate conservation credentials, threatened species displays are often linked to conservation-related activities such as breeding or reintroduction programs, research projects, educational or communication programs, and fundraising. Such 'hybrid' conservation approaches, that integrate in situ and ex situ efforts, have become increasingly important as zoos and aquaria pursue conservation credibility (Tribe & Booth, 2003). However, the success of these efforts is rarely evaluated, even though the quantification of conservation benefits derived from zoo and aquarium displays is a major research priority (Tribe, 2001).
Multiple methodological and theoretical issues plague evaluations of zoo and aquarium conservation efforts. Conservation success would ideally be defined as changes in conservation status, population size, extent of habitat, or other benefits to wild populations; and an experimental or quasi-experimental evaluation would be undertaken on an appropriate time frame to demonstrate positive changes in these key metrics (Ferraro & Pattanayak, 2006). Unfortunately, long ecological response times, financial or logistical restraints, and confounding effects (such as unrelated population trends, human activities, and environmental changes) usually preclude this approach. Alternative methods are therefore required to evaluate conservation benefits (Ferraro & Pattanayak, 2006;Kapos et al., 2008).
Conservation education is a dominant theme in zoo and aquarium mission statements globally (Patrick & Caplow, 2018), and is used to promote positive conservation behaviour changes in visitors . It is argued that zoos and aquaria provide educational and emotional experiences that inspire changes in visitor attitudes and ultimately conservation behaviours (Ardoin, Wheaton, Bowers, Hunt, & Durham, 2015;Clayton, Fraser, & Burgess, 2011;Clayton, Fraser, & Saunders, 2009;Smith, Weiler, & Ham, 2010). The Theory of Planned Behaviour (Ajzen, 1991) is an influential theoretical framework of behaviour that is often used to evaluate the success of conservation education . According to this theory, knowledge is one of many background factors (including demography and personality) influencing personal beliefs and is indirectly related to behavioural change. Personal beliefs (which underpin constructs of attitudes, perceived norms, and perceived behavioural control) guide intentions to perform specific behaviours (Fishbein & Ajzen, 2010). Although personal beliefs are infrequently studied as measures of educational success, changes in knowledge, attitudes, and behavioural intentions are the most commonly used measures. These are also sometimes considered to be measures of conservation success.
Major criticisms of conservation education evaluations that examine changes in knowledge, attitudes, and behavioural intentions warn that: knowledge is a minor factor in predicting whether visitors undertake conservation actions ; changes in intentions do not always translate into actual conservation actions (Ballantyne & Packer, 2005;Smith, Broad, & Weiler, 2008); and that measures of changes in conservation actions are seldom informative about the benefits to wild species or habitats, which are the truer measures of conservation success (Ferraro & Pattanayak, 2006;Hughes, 2013;Smith et al., 2010). While research does repeatedly detect small positive changes in visitor knowledge and attitudes (Khalil & Ardoin, 2011), persistent behaviour changes due to this education, or direct benefits to conservation, are rarely evaluated or confirmed (Nygren & Ojalammi, 2017). For example, recent research has persuasively demonstrated that zoo and aquarium visitors leave with an increased understanding of biodiversity and knowledge of actions to help protect it (Moss, Jensen, & Gusset, 2015), but over a longer time frame it is difficult to attribute persistent changes to the visit (Smith et al., 2008).
New multi-disciplinary methods, evaluating all conservation-related activities associated with displays, are needed to examine the relationships between captive wildlife displays, conservation benefits, and sustainability. Although there are several approaches to assess conservation effectiveness, few can provide practical, robust, project level evaluations of impacts of captive wildlife conservation programs (Howe & Milner-Gulland, 2012). Two potential approaches are the Cambridge Conservation Forum (CCF) Evaluation Tool (Kapos et al., 2008) and 'conservation impact scores' (Mace et al., 2007). Conservation impact scores were developed specifically for the evaluation of a wide range of conservation projects associated with zoo and aquarium displays, and have proven to be a user-friendly method of project evaluation compared to the CCF Evaluation Tool (Washington, 2012). Scores reflect the overall conservation impact of projects, calculated as a function of the significance of the project for conservation, the volume of input into the project, and the success of the project (Mace et al., 2007). Quantitative evaluations of educational success can also be integrated with conservation impact scores to examine the benefits of public education. The scores can then be used to identify strategies to maximise project success, thereby facilitating zoo and aquarium managers to meet their conservation commitments (Mace et al., 2007). Furthermore, natural resource managers may be assisted with decisions surrounding the sustainability and licencing of wild harvests of threatened species by this quantifiable measure of the conservation benefits of displays (Beechener et al., 2010).
Globally, numerous captive and semi-captive wildlife tourism experiences are suited to multi-disciplinary evaluations of their conservation impact, to determine whether claims of positive conservation benefits are justified. These experiences may negatively impact wildlife while simultaneously promoting the positive consequences of the experience or associated conservation activities. For example, zoo displays of marine mammals (Jiang, L€ uck, & Parsons, 2007;Ventre & Jett, 2015), large cats, primates (Keulartz, 2015), and elephants (Riddle & Christopher, 2011), are often associated with a range of conservation-related activities, although animal welfare issues result in controversy about the value of displays. Many sanctuaries, orphanages or refuges for elephants, orangutans (Tisdell, 2010), cheetahs, lions, owls, andmonkeys (Van Tonder, Hoogendoorn, &Block, 2017) offer wildlife tourism opportunities such as animal interactions, ostensibly to support conservation; but neither conservation benefits or impacts on the animals are generally assessed. Despite the widespread need for conservation impact evaluations, the most critical assessments may apply to wildlife experiences that place the survival or viability of wild species or populations at risk. For example, threatened fish species are regularly harvested from wild populations for educational aquarium displays despite world-wide population declines (Buckley et al., 2018).
Sawfishes (Family Pristidae) are a group of threatened marine fish that illustrate the critical need for evaluations of captive wildlife displays, and can be used to model a multi-disciplinary evaluation of conservation benefits. Sawfishes are large shark-like rays (possibly reaching over 7 m in length) (Carpenter & Niem, 1999), with a distinctive saw-like appendage that is used to hunt prey (Wueringer, Squire, Kajiura, Hart, & Collin, 2012) (Figure 1). Sawfish populations are at greater extinction risk than most other sharks or rays, due to a high susceptibility to capture in fishing nets, strong associations with widely impacted habitats such as mangroves and seagrass beds, and slow intrinsic population growth rates (Dulvy et al., 2014(Dulvy et al., , 2016. All five species are threatened with extinction, and the recovery of sawfish populations is a global priority to maintain biodiversity and coastal ecosystem services (Dulvy et al., 2014).
Due to their impressive size, charismatic appearance, and ability to readily adjust to life in enclosed spaces, sawfishes have been highly prized and displayed by aquaria for more than 70 years (McDavitt, 1996;NMFS, 2009). The Critically Endangered largetooth sawfish Pristis pristis is the most commonly displayed species, with at least 34 wild harvested individuals displayed globally in 2017 (White, Duke, & Squire, 2017). In 2015, the Australian captive population represented a significant portion of the global captive population (>25%), with nine largetooth sawfish displayed by four Australian public aquaria. Given that some Australian aquaria release largetooth sawfish after time in captivity, the number of individuals affected by aquarium activities in this region is even higher than the captive population at any one time reflects (Buckley, Wedd, Johnson, & Cutter, 2014).
The Australian region remains one of the last remaining strongholds for wild largetooth sawfish, although populations have undergone significant declines (DSEWPaC, 2011b;Stevens, Pillans, & Salini, 2005). Due to uncertainty surrounding cumulative risks to wild populations of largetooth sawfish, it is unclear if wild harvests for aquaria are sustainable. The Australian federal government recognises the potential for aquarium displays to have educational benefits related to sawfish conservation (DSEWPaC, 2011a), and fisheries management agencies in one territory and one state allow harvests for the "continuation of public education" (NTDPI, 2012) and "visitor education purposes" (QPIF, 2009), respectively. The export of sawfish from Australia to international aquaria is however now banned (DSEWPaC, 2011a). An evaluation of conservation benefits associated with displays is essential for the satisfaction of permitting requirements, and to allow public aquaria to demonstrate their conservation commitments. We aimed to meet these critical needs by examining the conservation benefits of four Australian aquarium displays of Critically Endangered largetooth sawfish Pristis pristis. Conservation impact scores were generated for associated research, education, and conservation-related activities to evaluate the likely contribution to largetooth sawfish conservation. To generate effect sizes for sawfish-related educational benefits, changes in knowledge, attitudes, and behavioural intentions were evaluated using a before and after study design. Ultimately, this research is intended to demonstrate new methods to facilitate decision-making by natural resource managers, and zoo and aquarium managers, to improve the sustainability of captive wildlife displays.

Methods
This research was approved by the Charles Darwin University Human Ethics Committee (Approval H14079).

Study sites
There were four study sites at Australian public aquaria displaying largetooth sawfish: Territory Wildlife Park (TWP) (Northern Territory), Sea Life Melbourne Aquarium (Melbourne Aquarium) (Victoria), Sea Life Sydney Aquarium (Sydney Aquarium) (New South Wales), and Underwater World Sea Life Aquarium (UWW) (Queensland) (Figure 2). Each of these aquaria displayed between one and four sawfish in multi-species exhibits. Three of these exhibits included other species of shark or ray. Interpretive materials that mentioned sawfish were highly variable in size and shape, and included static signage, television programs, touch screens, and public presentations (Table 1).

Manager surveys
A single senior manager or curator at each aquarium completed an online manager's survey to identify research, conservation activities, and educational messages associated with each display. An example manager's survey can be found in the supplemental material (Supplementary material 1). Manager surveys were developed and distributed online using Qualtrics software (Qualtrics V. 61502, Provo, UT, USA). The survey elicited intended aims of research and conservation activities, and the volume of financial or in-kind contributions to these activities by the aquarium. These aims were used as proxy measures for research and conservation activity outcomes, as this could not be determined by experimental or quasi-experimental approaches (Kapos et al., 2008). To ensure participant confidentiality and anonymity, no demographic information relating to Australian public aquarium managers is reported in this study.

Conservation impact scores
The likelihood of a positive conservation impact by each research project, conservation activity, and educational effort, was assessed using 'conservation impact scores'. This scoring system was originally developed to measure the success of a diverse range of zoo conservation projects for reporting purposes and conservation program development (Mace et al., 2007). Although different project types cannot be directly compared (Mace et al., 2007), conservation impact scores are one of the most consistent and clear methods to measure the conservation influence of diverse zoo and aquarium projects (Beri, Tranent, & Abelson, 2010). It has also been found to be a practical and efficient approach when used to evaluate the contribution of zoos and aquaria to global biodiversity conservation (Gusset & Dick, 2010).
Conservation impact scores have been criticised due to the use of independent assessments as a proxy measure of the conservation outcomes of research and conservation activities (Kapos et al., 2008), with some inconsistencies in scoring between assessors (Howe & Milner-Gulland, 2012). However, other applications of the scoring system have found it to be a reliable and robust method, and particularly practical when limited data is available on outcomes (Gusset & Dick, 2010). In 2011, Wilkinson, Barton, Wilson, and Zimmerman examined the conservation impact of the Kinabatangan Orangutan Conservation Project, basing their methods on conservation impact scores. Three independent assessors were used to compensate for score subjectivity (Maciaszek, 2012), and we adopt this methodology to maximise consistency in scoring.
Conservation impact scores were calculated following methods developed and recommended by Mace et al. (2007). Scores were the product of three measures: the conservation significance of the targeted subject of investigation or activity (importance), the scale of the effort (volume), and the results of the effort (effect) ( Table 2). Each measure was assessed using survey information collected at each Australian aquarium housing largetooth sawfish. The detailed methods are presented below, but were not disclosed to the aquarium managers, independent assessors (see below), aquarium visitors, or any other participants in the research ahead of the surveys. Possible scores ranged from one to a maximum of 64.
Importance. An assessment of the importance of the target for each research and conservation activity was made by the lead author and four independent assessors, as per Mace et al. (2007). This measure considered how influential or significant the target of the research or conservation activity was for conservation, and was scored on a scale of one (low/negligible) to four (very high or international) ( Table 2). The total importance score was calculated as the mean score from all assessors. Although inconsistencies of importance scores due to independent assessments are possible (Walter, et al., 2005), this method has been found to be consistent in comparable applications (Gusset & Dick, 2010), and robust compared to other indices of success (Howe & Milner-Gulland, 2012). Similarly, the importance of educational messages was assessed by considering the likelihood that target audiences may influence relevant conservation policy or practice, ranking from one (low, untargeted audience), to four (very high, such as conservation or political advocates), as per Mace et al. (2007) ( Table 2). How influential/significant was the target of the effort for conservation?
How many/much of the target was addressed by the effort?
What is the relevance of the findings or output for conservation? Research and conservation activities Significance of the research or conservation activity target: 1. Low/negligible 2. Moderate/local 3. High/national or regional 4. Very high/international Target: subject of investigation or activity (species, activity, policy).
Significance: relative importance of the research target in relation to global priorities as assessed by independent assessors and lead author.
Overall scale of contribution to the research or conservation activity calculated as the mean of all contribution types: Financial contribution ($AU) 1: <1000 2: 1001-5000 3: 5001-10000 4: >10001 Staff contribution (total weeks over project duration): 1: <1 2: 1-5 3: 6-20 4: <20 Equipment and supplies contributed 1: none 2: little 3: some 4: a lot Effect of the research or conservation activity: will it help conserve wild P. pristis populations? Definitely will not 2. Probably will not 3. Probably will 4. Definitely will Effect: the potential relevance of the research project for conservation outcomes, compared with no project, assessed by aquarium managers and lead author.

Education
Influence of the targeted people: 1. Low (untargeted) 2. Moderate (children) 3. High (school teachers, media) 4. Very high (conservation or political advocates) Target: aquarium visitors. Influence: the extent to which the visitors influence relevant policy or practice, now or in the future.
Volume. The scales of contributions by aquaria to research and conservation activities (volume) was elicited by asking managers to identify the amount of contributions the aquarium had provided, in terms of financial, staff, and in-kind contributions. This allowed managers flexibility to record important non-financial contributions, rather than restricting the measure to the cost of the project in terms of annual funding. Volume was scored on the scale of one to four for each contribution type, with the total volume score being the mean of all contribution types (Table 2). For education, the scale was calculated relative to the annual visitation to the aquarium, scored from one (<1 000) to four (>100 000) ( Table 2).
Effect. The results of research and conservation activities (effects) were scored by aquarium managers, the lead author, and four independent assessors to determine whether achieving the targeted outcomes would help conserve wild P. pristis populations, as per Mace et al. (2007). The total effect score was calculated as the mean score from all assessors (Table 2). Like importance scores, the use of independent assessments for effect scores has been found to be a robust method (Gusset & Dick, 2010;Howe & Milner-Gulland, 2012). A more formal measure of the success (effect) of education is required (see Mace et al., 2007), and this was therefore measured as the standardised effect size of significant changes in visitor knowledge, attitudes, and behaviours, assessed with a quasi-experimental design (see below).

Visitor surveys
Educational success was quantified using visitor surveys with a quasi-experimental design. This design was necessary due to the one-way floor layout of all participating aquaria, which precluded an experimental control group that did not view the sawfish exhibit. The experimental design included a group surveyed both on entry to the aquarium and on exit (Group A), and an exit only control group to test for response bias and/or pre-test sensitization (Group B) ( Figure 3). A pilot study was undertaken at the TWP (n ¼ 65) to test the survey instrument and its' practical application. The required sample size was determined as per Bethlehem (2009), and a target of n ¼ 500 completed surveys was established for each participating aquarium. The most common measures of educational success were used, being changes in knowledge, attitudes, and behavioural intentions (Khalil & Ardoin, 2011). This maximised the applicability of methods to evaluate conservation success used in this study, to existing research examining the success of educational programs. Due to logistical and time constraints, it was not possible to measure actual visitor conservation behaviours, even though this is generally recognised as the ultimate goal of educational programs in zoos and aquaria (Clayton et al., 2009;Ogden & Heimlich, 2009). Behavioural intentions were measured instead, as they are correlated with actual behaviours, particularly on short time scales (Webb & Sheeran, 2006).
Four targeted educational messages in each of the categories of knowledge, attitudes, and behaviours, were elicited for each aquarium via the manager's survey (a total of twelve messages per aquarium). The specific number of messages were chosen to limit the survey to a single double-sided A4 page, thereby maximising response rates and minimising survey length. Surveys also included items relating to demographic information, including prior visitation, nationality, age, biological sex and education. A copy of a survey can be found in the supplemental data section available on the web-based version of this paper (Supplementary material 2).
Knowledge. Given that licencing conditions for the collection of sawfish for Australian public aquaria refer to public or visitor education in general (NTDPI, 2012;QPIF, 2009), we considered that a fair assessment of educational success should aggregate general visitor knowledge of sawfish biology, ecology, and conservation. Visitor knowledge was measured using questions with a true/false/don't know response format. These single-item knowledge responses were coded, with correct responses coded as '1' and incorrect responses as '0'. The measure of knowledge responses for each participant was a sum of their knowledge question scores, resulting in an overall knowledge score between 0 and 4.
Attitudes. Attitudes towards sawfish were quantified using a direct attitude measure with a 7-point semantic differential (SD) scale, using four items per scale. Semantic differential scales are particularly suited to measure attitudes of the public towards unfamiliar animals, as people may have strong emotional responses despite a lack of knowledge about the animal (Reimer et al., 2014). These scales may therefore be particularly suited to sawfish research. One of the adjectives used for each item was selected from specific terms used by aquarium managers to describe the targeted attitudes, and a thesaurus was used to find appropriate antonyms. The resulting four adjective anchor pairs were arranged based on an a priori expectation of whether they would represent a negative or positive attitude, with positive adjectives listed on one side of the scale and negative adjectives on the other. Data were entered as scores from 1-7, with higher scores representing more favourable responses. The mean score of all items in a scale was used as the attitude measure for multi-item scales. Reverse coding was used to control for consistency bias.
Behaviours. Changes in visitor behaviours were predicted by a measure of behavioural intentions using a 7-point scale. Levels of agreement for behavioural intention items varied, for example ranging from 'extremely likely' to 'extremely unlikely', and 'definitely do' to 'definitely do not'. Although Fishbein and Ajzen (2010) recommend four items to ensure reliability of behavioural intention scales, due to page space limitations only two (reverse coded) items were used for each scale. It was recognized that not all participants had, or will have, the opportunity to perform all behaviours items surveyed, and in these cases the target audience was identified in the survey by an opportunity question. For example, if the behaviour was releasing a sawfish unharmed, then the opportunity question was: "In the next five years are you likely to go fishing in a tropical river or coastal area?" (see Supplementary material 2). If participants would not have the opportunity to perform the targeted behaviour then they were not included in the analyses for that behaviour.
Participants. Visitor surveys were administered on-site at each aquarium between April and June 2015, by the first author and volunteer staff. Survey periods ranged from 6 to 14 days at each aquarium, for all opening hours until the target number of samples was reached. Surveys were undertaken mainly during week days, with 2-3 weekend days at each aquarium included in the survey effort. Surveys were undertaken during school terms, except at the Territory Wildlife park where half of the survey period occurred during the school holidays.
Potential adult (18þ years of age) participants were approached on a continual ask focal sampling basis as they reached a pre-determined point, once recruitment of the previous participant was finished (Diamond, Horn, & Uttal, 2016). This method, also called "next-to-pass", is a widely accepted method to select participants for similar research (Mellish, Pearson, McLeod, Tuckey, & Ryan, 2019;Moss et al., 2015). If it was uncertain if an individual was more than 18 years old, they were approached to participate, and demographic information recorded on the survey was later used to determine if the survey was retained. No incentives were provided for participation. Survey responses were excluded when entry and exit surveys were not matched and when survey responses were largely incomplete.
There was a high visitor participation rate in visitor surveys at each aquarium, with between 55% and 79% of approached individuals participating: an acceptable rate for social science research (Dillman, Smyth, & Christian, 2008). Reasons given for non-participation were largely due to child minding duties or English as a second language. Of the visitors that agreed to complete paired entry and exit surveys, between 6% and 14% did not complete the exit survey. Approximately 6% of surveys collected were not used in the final analysis (Table 3). At each aquarium, there was a relatively even spread of participants between Group A and Group B for all demographic information collected. Most participants were Australian, and slightly more than half were female. Between 42 to 59% of participants had visited the aquarium on a previous occasion (Table 4).
Data analysis. Data were analysed with the Statistical Package for the Social Sciences (IBM SPSS Statistics for Windows, Version 23.0. Armonk, NY). To identify possible demographic differences between Group A and Group B, Chi-Square Test of Independence were conducted. Significant educational effects were identified using t-tests, and Cohen's d effect sizes were used in calculation of conservation impact scores. These statistics are widely applied in conservation education evaluation (Mellish, Ryan, Pearson, & Tuckey, 2018), and can be used to compare results among different studies (Lakens, 2013) which is crucial to generate comparable conservation impact scores. As standardised group mean differences, Cohen's d can be meaningfully interpreted against benchmarks such as those suggested by Cohen (1988), unlike effect sizes used with ANOVA models such as partial eta squared that measure the strength of associations between variables (Lakens, 2013).
The reliability of scales and the presence of response bias and/or pre-test sensitisation were examined to maintain experimental validity. Independent samples t-tests for exit survey responses from Group A and Group B were used to detect pre-testing effects for all knowledge, attitude, and behavioural intention measures. The internal consistency of four item attitude scales was examined using Cronbach's a (Cho & Kim, 2015;Schmitt, 1996), and the inter-item reliability of two item scales was examined using Spearman-Brown Coefficients (Eisinga, Te Grotenhuis, & Pelzer, 2013).
To detect changes attributable to the aquarium visit, paired samples t-tests were used to examine changes in visitor knowledge, attitude, and behavioural intentions between reliable paired entry and exit surveys that did not show significant pre-testing effects. Bonferroni corrected critical values were used to control for an increased Type I error rate due to multiple  significance tests (Garc ıa, 2004). For reliable measures with significant pre-testing effects, independent samples t-tests were used to compare Group A entry scores and Group B exit scores to examine general differences between the groups. Cohen's d was used to measure effect sizes for educational messages with a significant result. Correlations of 0.10, 0.20, and 0.30 were considered as relatively small, medium, and large respectively (Gignac & Szodorai, 2016). For the purposes of calculating conservation impact scores, when there was not a significant improvement in knowledge, attitudes, or behavioural intentions detected, this was considered equivalent to no discernible effect. If the effect size of significant results were small, marginal improvements in education were detected; if the effect size was medium, improvements were detected; and if the effect size was large, substantial improvements were detected (see Table 2).

Manager surveys
Education was identified by all aquarium managers as an intended outcome of largetooth sawfish displays, and public recreation was an intended outcome for all displays except at Underwater World. The Territory Wildlife Park additionally identified research and conservation activities as intended outcomes. Conservation impact scores were calculated for all intended education, research, and conservation activities (Table 5). Mean conservation impact scores showed considerable consistency across all aquaria (ranging from 11.5 to 16.0), primarily due to similar importance and volume measures. The calculation of these conservation impact scores can be found in the supplemental data section (Supplementary materials 3 and 4).
Mean importance measures of the two research projects and three conservation activities associated with the Territory Wildlife Park sawfish display indicated that the significance of the targets ranged from moderate or local to very high or international. Mean volumes of contributions were moderate, except for one conservation activity which had a high volume of contributions. Effect scores indicated that a successful outcome for all projects probably would help conserve wild P. pristis populations. Conservation impact scores for four of the five research projects and conservation activities ranged from 8.7 to 13.1, indicating that substantial contributions to largetooth sawfish conservation were unlikely. One conservation activity (assisting National Geographic to film sawfish documentaries) scored 22.7 and was therefore more likely to contribute substantially to sawfish conservation.

Visitor surveys
Conservation impact scores for all 15 knowledge and attitude messages with a significant effect ranged from 12 to 16 out of a possible maximum of 64 (Supplementary material 4). This indicates that limited conservation benefits could be expected from these messages. No significant changes in behavioural intentions were detected. Four specific messages were identified from manager surveys in each of the educational categories of knowledge, attitudes, and behaviours, for each aquarium (Table 6). These messages were used to quantify educational success by way of visitor surveys, independent of the interpretation or other tools used to transmit the messages. The exception was for UWW who did not intend to promote any specific behaviours in their education effort, although knowledge and attitude messages were identified. Given this, behavioural intentions were not analysed for visitors to this aquarium. Significant between-group differences were found between age classes at Melbourne Aquarium, and education level at the Territory Wildlife Park (results of Chi-Square Tests of Independence are given in Supplementary material 5).
In total, there were 48 applications of SD (attitude) scales, being for Group A (entry) surveys, Group A (exit) surveys, and Group B (exit) surveys at each aquarium. These applications were found to be reliable in 42 instances (Cronbach's a > 0.70) (Supplementary material 6a-6d). Of the remaining six applications, three had a Cronbach's a ¼ 0.69, while the least reliable application had Cronbach's a ¼ 0.56 with no improvement if any of the items were removed. All SD scales were retained in the final analyses after considering the arbitrary nature of determining an acceptable alpha level; that scales with reduced numbers of items may be considered reliable with alphas >0.50; and that conclusions drawn from the results of these analyses would be minimally affected by some degree of error in these results (Carver, 1997;Cho & Kim, 2015;Schmitt, 1996). Following this reasoning, 30 of the 36 applications of behavioural intention scales in this study were found to be reliable (Spearman-Brown Coefficient >0.50) (Supplementary materials 7a-7c). The entry survey had a significant effect on 24 of 41 reliable knowledge, attitude, and behavioural intention measures; a result indicative of substantial response bias or pre-test sensitisation (independent samples t-tests; p < 0.05; Supplementary material 8a-8d). In every case, the direction of the effect was positive, in that survey scores at the aquarium exit were higher for visitors that had completed entry surveys than for those that had not.
When considering only reliable measures that did not display pre-test sensitisation, significant gains in sawfish-related knowledge and positive changes in attitudes after visiting aquaria were detected (paired samples t-tests; p 0.003; Table 7). At all aquaria, visitors left with significantly more positive attitudes towards sawfish and sawfish conservation; when leaving Melbourne Aquarium, they knew more about sawfish biology and threats in the wild. In most cases (64%), the effect sizes (Cohen's d) were large, and in all other cases were moderate (Gignac & Szodorai, 2016; Table 8).
Independent samples t-tests between Group A entry scores and Group B exit scores for reliable measures displaying pre-testing effects also revealed general differences in visitor attitudes towards sawfish and sawfish conservation at the entry and exit of TWP and Sydney aquariums (independent samples t-tests; p < 0.002; Table 8). In each case, the effect sizes were large and indicated substantially more positive attitudes towards sawfish and the protection of sawfish populations and habitats at the exit.

Discussion
This study integrated a conservation influence scoring method for in situ and ex situ research and conservation projects (conservation impact scores) with a quasi-experimental survey to quantify educational benefits, in a new approach that evaluated the conservation benefits of Critically Endangered largetooth sawfish displays in public aquaria. The potential for evaluations of many other captive and semi-captive wildlife experiences was demonstrated. Such evaluations have the potential to justify claims of conservation benefits and/or provide a better understanding of theoretical and practical measures that can be applied to maximise benefits.

Benefits of research and conservation activities
Research and conservation activities associated with largetooth sawfish displays showed highly project-dependent conservation benefits, with only one conservation activity likely to have a significant positive impact on wild sawfish populations. This activity, providing support to an international television network to film documentaries raising awareness of sawfish conservation, had a conservation impact score that indicated an 'appreciable contribution' to conservation (see Gusset & Dick, 2010). Higher levels of staff and in-kind support were contributed to this activity than any other. High levels of support for research or conservation efforts is an indication of an institution's commitment to conservation (Fa, Gusset, Flesness, & Conde, 2014;Miller et al., 2004), and can greatly increase the probability of success of the efforts (Miller et al., 2004). In the global setting, a high level of financial or in-kind support of research and conservation projects by zoos and aquaria has resulted in an increased contribution to global biodiversity conservation (Gusset & Dick, 2010).
None of the research or conservation activities identified in this study were targeted specifically to conserve wild largetooth sawfish populations, and effect measures illustrated the uncertainty that even completely successful outcomes would provide conservation benefits. It has previously been noted that zoos and aquaria often lack clear targets for conservation efforts , and that research and conservation activities targeting specific threats to individual species could substantially improve any expected benefits (McGowan, Traylor-Holzer, & Leus, 2017). This could be addressed by developing specific conservation missions for exhibits. In the  (Ferraro & Pattanayak, 2006). Geographic location, and consequently the opportunity to undertake in situ conservation activities, may influence research and conservation activities undertaken by aquaria. In this study, TWP was the only aquarium situated within the Australian range of largetooth sawfish and located near wild largetooth sawfish habitats (Figure 1), and the only aquarium to identify conservation activities and research projects associated with the sawfish display. Three of these projects involved in situ activities such as filming sawfish for documentaries raising sawfish awareness, partnering with Indigenous rangers and university researchers to relocate juveniles from a drying billabong to the adjacent river system, and surveying sawfish populations. The direct access to natural sawfish habitats provided this aquarium with enhanced opportunities to demonstrate substantial conservation efforts.

Educational benefits
Aquarium visitors showed substantial gains in sawfish-related knowledge, and repeatedly demonstrated a positive attitudinal change towards sawfish or sawfish conservation. This may reflect public unfamiliarity with the species, as providing information about rare and unfamiliar species can result in significantly more positive attitudes (Reimer et al., 2014). Research has variously demonstrated that such positive changes in visitor knowledge or attitudes may persist for as little as eight weeks (Adelman, Falk, & James, 2000) to more than two years after the visit . However, there is considerable debate whether changes last in the long term, or result in real conservation outcomes (Hughes, 2013;Smith et al., 2010).
No significant changes to behavioural intentions were detected in this study despite knowledge gains and positive changes in attitudes. Research has previously linked positive changes in knowledge or attitudes to self-reported pro-environmental actions (Miller et al., 2013;Pearson, Lowry, Dorrian, & Litchfield, 2014), but largely only weak or no links have been found (Hughes, 2013;Moss et al., 2017). Many factors may weaken links between knowledge, attitudes, and behaviours, including personal beliefs (Fishbein & Ajzen, 2010), variance in opportunity, the effort required to accomplish behaviours, the duration or impact of the educational experience, or personal psychological barriers (Hughes, 2013). Also, the intent to engage in conservation-related behaviours has also been positively related to the emotional connection that people form with large, charismatic wildlife (Skibins, Powell, & Hallo, 2013), but these strong connections are usually associated with highly active species that make eye contact (Powell & Bullock, 2014). On this level, the relatively sedentary sawfish may fail to engage visitors to the same extent as charismatic megafauna such as bears, great apes, big cats, or elephants.
Engaging visitors emotionally and providing supporting post-visit resources may promote positive changes in conservation behaviours (Ballantyne, Packer, Hughes, & Gill, 2018;Ballantyne et al., 2011;Hughes, 2013;Hughes, Packer, & Ballantyne, 2011;Monroe, 2003), and these principles may be usefully applied to sawfish displays. For example, recent research on cage diving with white sharks Carcharodon carcharias demonstrated that combining emotional responses, educational messages, and post-visit resources are most likely to promote conservation behaviours in the long term (Apps, Dimmock, & Huveneers, 2018). Exploring similar optionssuch as providing information or souvenirs promoting conservation actions that are retained after the aquarium visitmay result in persistent sawfish conservation awareness and/or conservation actions.
Conservation impact scores indicated that overall, public education by the aquaria was unlikely to contribute substantially to sawfish conservation. Although significant educational success was detected and large numbers of aquarium visitors were exposed to educational messages, messages were not targeted to a specific audience and were often very generalised (such as checking the sustainability of seafood). The benefits of educational efforts could be significantly increased with well-defined behavioural messages that incorporate action, goal, place, and time elements (Smith, 2009), specifically targeting visitors that are likely to support conservation, or that can influence policies or practices that relate to conservation (Curtin & Papworth, 2018;Mace et al., 2007). Appropriate and impactful messages (such as the live release of sawfish caught while fishing, or the protection of juvenile sawfish habitats) could be targeted to audiences, and would be more likely to make a substantial contribution to conversation. Finally, most messages were also passive, and reliant on visitors reading signs or watching television screens. Passively-directed messages are less successful than more active approaches (Miller et al., 2004). Given the content, delivery style, and lack of targeted audience, many messages examined in this study could not reasonably be expected to have a significant or direct impact on sawfish conservation.
The Theory of Planned Behaviour (Ajzen, 1991) provides a theoretical framework that has been commonly used in education evaluations , and this provided our research with relatively standardised and clear measures of educational changes. However, changes in knowledge, attitudes, and behavioural intentions may be flawed as measures of conservation education success, as they may not reflect true benefits to wild species or habitats (Ferraro & Pattanayak, 2006;Hughes, 2013;Smith et al., 2010). In this study we calibrated the measured size of education changes (effect size) by importance (the likelihood that target audiences actually influences relevant conservation policy or practice) and volume (number of visitors exposed to the education) when calculating conservation impact scores. Thus, measures of educational changes became more informative about the potential for real conservation benefits.
Although aquarium visitors in this study had gains in sawfish-related knowledge, and a positive attitudinal change towards sawfish or sawfish conservation, no significant changes to behavioural intentions were detected. As we looked through a Theory of Planned Behaviour lens, it became clear that the educational messages in this study were not directed towards changing personal beleifs, even though they may be a key driver of behavioural intentions. To be more effective at changing visitor's conservation behaviours, zoos and aquaria must implement exhibit interpretations that are founded in conceptual models of behaviour change. There are a number of conceptual models of behaviour change other than the Theory of Planned Behaviour that could be usefully applied in this context. The Elaboration Likelihood Model postulates that well considered messages will result in attitudinal changes and may ultimately influence behaviours (Petty & Wegener, 1999), and this model could be particularly suited to application in a zoo or aquarium setting (MacDonald, Milfont, & Gavin, 2016). Social marketing techniques (including audience analysis, and message design for realistic and appropriate objectives) have also be used to develop appropriate educational approaches for wildlife tourism settings (Bates, 2010;Iles, 2004;Smith, Weiler, Smith, & van Dijk, 2012). Exhibit interpretations developed with suitable conceptual frameworks in mind may ultimately contribute to sustainable wildlife tourism by generating visitor satisfaction and encouraging the protection of wildlife (Ham & Weiler, 2012).

Limitations and future directions
This research illustrated some of the known challenges associated with empirical evaluations of conservation benefits associated with wildlife tourism ventures. For example, the actual outcomes of research and conservation activities were not able to be determined experimentally, and so proxy measures were used to calculate conservation impact scores. Thus, there is a level of uncertainty surrounding the realised conservation benefits of the research and conservation activities. The use of paired samples instead of aggregate statistics in the evaluation of public education allowed the identification of changes on the individual level, thereby establishing that the aquarium-based education was a likely determining factor in any changes (Wagoner & Jensen, 2014). However, the detection of pre-testing effects indicated that the administration of an entry survey primed visitors to learn sawfish-related conservation messages. Although this is a methodological challenge, due to our conservative interpretation of results we consider that this was unlikely to have meaningfully influenced the outcomes or conclusions drawn. An additional implication of these results may be that some consideration could be given to pre-visit education, thereby increasing the learning outcomes of visits.
Four positive educational outcomes were not considered to be significant in this study due to the use of Bonferroni corrected critical values of p 0.003 and p 0.002. The use of Bonferroni corrected critical values to control for an increased Type I error rate has previously attracted some criticism due to the risk that potentially significant results are ignored (Garc ıa, 2004). Interestingly, the outcomes ignored in this study included two behavioural intention items (the intention to report illegal fishing activities, and to pick up discarded nets and lines). Changes in behavioural intentions were not otherwise detected in this study.
The methodological and theoretical challenges of multi-disciplinary evaluations exposes our research to criticism from both the biological sciences (for example, due to the use of proxy measures which is rare in empirical research) and the social sciences (for example, we did not use any actual measures of behaviour, whether qualitative or quantitative). The challenges of integrating these disciplines have led to growing calls for a more collaborative approach between conservation and biological sciences (Bennett et al., 2017;Mascia et al., 2003;Teel et al., 2018). It is crucial to continue the exploration of new ways to quantify the conservation benefits of zoo and aquarium displays as they relate to sustainable wildlife tourism goals, to support decision making for both the wildlife tourism sector and natural resource managers (Catibog-Sinha, 2010). To this end, the research presented above should be perceived as a foundation on which improved methods may be built.

Conclusions and recommendations
Research projects, conservation activities, and educational messages associated with largetooth sawfish displays in Australian aquaria generally indicated that most activities were unlikely to contribute substantial conservation benefits for largetooth sawfish. There were positives including significant educational successes, demonstrating that aquaria have the potential to contribute to sawfish conservation with modifications to research, conservation activity, and educational efforts.
We recommend that Australian aquaria develop specific and targeted educational messages for sawfish displays, with ongoing evaluations to ensure that messages remain realistic and appropriate over time. The importance of evaluations to maximise the effectiveness of conservation education has been increasingly recognised in the literature (e.g., de White & Jacobson, 1994;Heimlich, 2010;Khalil & Ardoin, 2011;Mellish et al., 2018), and environmental education programs supported by evaluations are more likely to succeed than other programs (Carleton-Hug & Hug, 2010). Research and conservation activities should address specific conservation goals and draw more financial and in-kind support.
The results of this study can be applied more broadly, illustrating that the positive conservation impact of captive and semi-captive wildlife tourism experiences can be maximised using conceptual models of behaviour change when experiences or exhibits are established. This ensures that educational messages are targeted, appropriate, and transmitted in ways that capture and hold visitors' attention. Other in situ or ex situ conservation efforts associated with captive wildlife displays should be carefully considered to ensure that real conservation outcomes are supportedwhether by financial or in-kind supportas much as possible. These steps will ensure that conservation mission statements can be substantiated by the quantification of conservation benefits, and will demonstrate a clear commitment to sustainable wildlife tourism Gusset et al., 2014;Gusset & Dick, 2010).