The effects of game-based financial education: New survey evidence from lower-secondary school students in Finland

Abstract The authors of this article studied the effects of game-based financial education approaches using a sample of lower-secondary school students in Finland. The sample consisted of 640 students from 42 schools in different areas of the country. The authors focused on three different game-based interventions using a pre- and post-intervention survey design. They compared the effects of the interventions (and their combinations) to a control group that received only traditional teaching. They found robust effects with respect to knowledge gained from game-based approaches. However, the effects on self-reported financial behaviors were weak.

principles-based approaches; Lusardi et al. (2017), who investigated the use of visual tools and narratives in financial education; Kaiser and Menkhoff (2018), who found that active learning approaches worked better than traditional lectures in a sample of entrepreneurs; and Carpena et al. (2019), who studied the effects of personalized counseling and goal-setting. However, none of these studies took place in a school setting. In contrast, the recent article by Iterbeke et al. (2020) studied, in a secondary school setting, the impact of ability matching and differentiated education in financial education.
However, the use of games in financial education has remained an understudied issue. Nevertheless, there is a perception, especially among practitioners, that games can be an important tool in teaching and learning financial literacy (e.g., Maynard et al. 2012). In the broader field of economic education, and especially in research concerning higher education, the potential of using games and simulations has been recognized for quite some time. 1 There has been an increased interest in game-based approaches in Finnish schools to increase the effectiveness of financial education. Appropriately for that context, the aim of this study is to evaluate the effects of game-based approaches on financial literacy compared to the situation where there are only traditional teaching methods. Therefore, we do not compare the effects of a stand-alone intervention vs. no intervention but rather, the effects of the delivery mode (game-based vs. traditional) and their influences on financial literacy. The methods that we investigate relate to the paradigm of active or experiential learning, which has been regarded as having promise in economic education (Amagir et al. 2018;Collins and Odders-White 2015).

Financial literacy teaching in Finnish schools
The Finnish educational system has received considerable positive attention related to achievements in the Programme for International Student Assessment (PISA) rankings since the early 2000s. More recently, the performance of Finland in the PISA has somewhat declined. As a response, there has been an increased emphasis on phenomenon-based learning and digitalization in teaching in the new curriculum plan for primary and secondary schools (Lonka et al. 2018). Finland also ranked highly in the PISA for financial literacy in 2018, in second place out of 20 countries (Organisation for Economic Cooperation and Development [OECD] 2020).
Economic and financial literacy can be regarded as examples of cross-disciplinary themes. Financial literacy is not taught as a separate subject. It is part of a module in "economic knowledge, " which in turn is part of the social studies curriculum. The learning goals of the module include elements both in personal financial management and in a broader macroeconomic context. Economic issues are taught in diverse fields of study, such as study counseling, home economics, and mathematics. Teaching typically takes place within social studies, where teachers mostly specialize in history and not economics. One outcome of this is that teachers actually are not very well informed in economic analysis. Moreover, the material in the school textbooks does not always adequately cover the voluntary K-12 content standards by the Council of Economic Analysis (Siegfried et al. 2010), although they are partly covered in the Finnish curriculum (Kalmi, Maliranta, and Alanko 2019).
The ninth year of mandatory schooling in Finland contains an economics curriculum that combines the subjects of financial literacy, which is understood to be personal financial management skills applicable at the level of individuals and households, and economic literacy (Hansen, Salemi, and Siegfried 2002;Walstad and Soper 1988), which is related to the application of key economic principles to economic analyses in various issues that are beyond personal financial management; for instance, issues related to informed voting behavior in economic policy contexts. According to the national curriculum for grades 7 through 9, after having the economic knowledge module within social studies, the students should be able to apply economic knowledge in the domains of personal financial management, entrepreneurship and informed career choice, planning for the future, and democratic participation. The curriculum also includes certain attitudinal elements, such as applying ethical thinking in economic choice. However, these normative and attitudinal issues are not graded.

Programs of game-based learning
We focused on three distinct game-based programs. The programs have rather different foci, as two of them are geared toward personal financial management and one is a business game. We summarize the characteristics of these games in table 1. Yrityskylä is a business game in a physical learning environment. It is operated through the Economic Information Office (EIO), which is owned by the Confederation of Finnish Industries. Since 2010, EIO has also run a similarly named educational program, Yrityskylä, for primary school students, which is well known in Finland (Kalmi 2018b). The program for the secondary school is more recent, with the pilot phase having started in 2016. Nevertheless, it has spread quite rapidly, and the aim is to increase its scope to a national level in a few years.
In the game, students represent a business organization and have several choices. For instance, they can make purchase decisions, try to optimize production processes, apply for loans, make investment decisions, and sell their products, all in a simulated environment. The skills that are required in the game include knowledge about some basic accounting decisions, mathematics, human interaction, and foreign language skills (as some sales pitches are made in English). 2 Students play this game in a designated physical environment using digital tools (tablets). They work in teams of five students, which allows for some functional specialization according to student capabilities. A visit to the physical learning environment is preceded by some general lessons on business management and on other decision skills that are required for gameplay. While the topic of the game is not personal finance, the learning goals are related, e.g., to work-life skills and budgeting, so it is quite possible that there is spillover to personal finance domains. Altogether, according to teachers, this game took 5 to 8 hours, including the visit to the physical learning environment that took several hours. Therefore, timewise, this is the most extensive of all the studied programs.
Oma Onni is a Web-based learning environment in which the material is produced by students in a vocational school (see Kalmi 2018a). It is sponsored by foundations that own Oma Säästöpankki, a bank with roots in local savings banks. It is produced by the SEDU vocational school in Seinäjoki, Western Finland. It has been in operation since 2010, and its learning contents have been continuously updated.
The material in Oma Onni includes games, quizzes, videos, and other interactive material. Most of the learning for secondary school students takes place over the Internet. The students typically go through this material individually in the classroom. Additionally, students from the vocational institute may also visit lower-secondary schools. In addition to being Web-based, another feature of the program is that it represents a form of peer learning, as the lower-secondary school students learn from vocational school students, who are only a few years older. The focus of the program is clearly on personal financial management. The learning goals of the program relate to work-life skills, knowledge that is relevant in managing personal finance, and financial attitudes oriented toward prudence and forward planning. Teachers reported that the use of the learning environment took approximately 4 to 6 hours. 3 Money Flow Challenge is a mobile game with five different levels. It is designed by a small personal finance consulting firm called Planago. The game includes topics such as consumption, savings, investment, and mortgage decisions. Students learn to make rational, utility-based decisions concerning personal financial management. This particular learning environment does not require much infrastructure compared to the other two environments and mainly requires a mobile device that can be used for playing the game. Students played this during lectures. There was also a short tutorial. Ideally, it would also include a feedback discussion, although that apparently did not take place in all schools. The hours used for the study of this game were lower than in the two other cases. Students were usually exposed to the Money Flow Challenge over two hours of classes. Money Flow Challenge is also the most recent of the programs, having started in 2014. Game-based teaching is regularly used in many disciplines in Finnish schools. Within economics, one limiting factor has surely been the lack of easy access to relevant games. There are some exceptions, like the monetary policy games developed by the European Central Bank and some games launched by the Finnish branch of Junior Achievement. These games differ from the above-mentioned programs because they are relatively short and easy to use, usually taking up only a fraction of the class hour. According to our teacher survey, the use of other games was somewhat more common in the control group than in the treatment group. In our empirical approach, we test whether this difference affects our results.

Allocation of programs
In our study, we focus on the development of the economic capabilities of 9th-grade students, who have a mandatory module of economics. The focus of the study is on whether game-based approaches result in better learning achievement than in the absence of gamified teaching. The classes are divided into an intervention group, where the classes participate in game-based teaching, and a control group that does not participate in the interventions. In this approach, it is notable that there exists considerable heterogeneity due to the different teaching styles of social science educators. Some teachers may well use games in their teaching even if part of the control group. Within the intervention group, the vigor at which teachers incorporate game-based approaches in their teaching is likely to differ among teachers. It is also possible that many teachers in the control group are likely to use active learning approaches, whereas the teaching of many teachers in the intervention group may be more traditional. To control for this, we asked the teachers about the teaching methods they use.
The ways in which the programs were allocated to different schools differed somewhat among programs. Yrityskylä and Oma Onni were allocated with direct agreements between the program providers and municipal school authorities. Yrityskylä aims to be a nationwide provider of financial education and is widely used in different municipalities, even though the program for secondary schools is relatively new and not offered in all parts of the country. Oma Onni is in turn tied to the operating area of the savings bank behind the program. In neither of these cases could the researchers affect the allocation of the programs, as the allocation depends on the geographical location of municipalities. In contrast, it was possible to randomize the use of the Money Flow Challenge. A constraint was that we did not want to offer it to schools that already had two interventions to prevent an overload of economic education programs in our groups. Thus, we randomly offered the Money Flow Challenge to 14 schools in the group of schools that either did not participate in the other two interventions or participated in only one of the other interventions. In the end, approximately one-half of the subjects to whom it was offered took it.
In summary, because of the practical constraints posed by the arrangements of the programs, even the intention to treat is not randomly allocated (except for the Money Flow Challenge intervention). The intention to treat is determined by external factors such as the availability of the programs that may or may not be related to student characteristics. However, because it is determined at the municipality level and not, e.g., by individual teachers, we have no strong reasons to believe the intervention and control samples differ in their unobservable characteristics.
We selected 12 towns for our work, with the idea that we would find a reasonable representation of schools with and without programs. Initially, approximately 60 schools from 12 different towns agreed to participate in the study, but some of them dropped out before the start of the year or during the year. Furthermore, because parental consent letters were required, the number of students was limited to 982 of over 3700 students who took the pre-intervention survey. Due to school, class, or student attrition, the final sample size was further limited to 640 students who also took the post-intervention survey. Thus, in the final data, we analyzed pupils from 42 schools in 11 towns. A total of 433 of the responding students received game-based teaching along with typical economics education; an additional 44 students should have received the treatment but did not, while 163 did not receive treatment and formed our control group. 4 Some students participated in two game-based interventions, while others participated in only one intervention. Table 2 presents the final data for the number of students in different groups.

Survey data and questionnaire
We conducted pre-and post-intervention surveys on economic knowledge, savings behavior, planning ahead, and impulse shopping among ninth year students. The survey was based partly on a previous questionnaire used in the Oma Onni program in analyzing student learning (Kalmi 2018a). However, the Oma Onni program focuses solely on personal finance, whereas some programs included in our study, as well as the learning goals of the Finnish curriculum, are broader, including issues of "economic literacy" (for instance, firms, macroeconomics, and economic policy). Therefore, we broadened these questions so that one-half of the knowledge questions covered questions related to personal finance, and the other half of the questions concerned firms and macroeconomics. 5 Moreover, the questionnaire was divided into knowledge questions 6 and into questions dealing with behavior and attitudes. The question sets in the two surveys were identical regarding financial behavior, but the financial knowledge questions were different. The two surveys and their questions were pretested with vocational education students.
The first survey was conducted in autumn 2017 with the assumption that no economics education took place before the survey. The second survey took place in spring 2018 after the educational interventions and other economics teaching in the curriculum. However, some classes had already started economics education before they took the first survey, while some classes had their final lessons in economics only after the second survey was taken. Therefore, we obtained the teaching schedules and controlled for these differences in our empirical setting.
We also conducted a survey among the parents in which we asked for background information, such as parental education, and tested for parents' own financial knowledge with the "three big" questions of financial literacy (Lusardi and Mitchell 2014). The additional parental survey was voluntary. We had a response rate of 76 percent of those who also gave a research permit.
Our pre-and post-intervention surveys contained questions on financial knowledge and attitudes and behavior, as well as students' background characteristics. Table 3 presents definitions and measurements of the variables used in the empirical analysis. 7 Our main outcome variable is the standardized test score, which measures the student's financial knowledge. The test scores were demeaned so that they had a mean of zero and a standard error of one.
Because some of the schools and classes dropped out between the first and second survey and parental consent was required, our final sample was shaped by a two-stage selection process and, thus, may not present the initial target group of the study. Furthermore, some parents selected not to respond to the parental survey. Table 4 analyzes these attrition processes by showing students' initial level of financial knowledge according to the group.   Table 4 shows that students in the treatment group performed slightly better in the pre-intervention survey. The table further illustrates that those students who also returned the second survey performed significantly better in the pre-intervention survey, which may reflect unobservable school or teacher characteristics. This selection process appears in both the treatment and the control groups. However, a smaller proportion of students in the control group answered the post-intervention survey than in the treatment group, and the attrited students in the control group appear to differ from non-attrited students somewhat more than in the treated group. We find no statistically significant differences in the financial knowledge between the treatment and control groups for students who took both surveys. Students for whom we obtained parental consent also performed significantly better in the pre-intervention survey. Receiving a response to the parental survey also appears to correlate with students' higher financial knowledge. The plausible reason for this is that answering the parental survey correlates with the student's socioeconomic background. Parents who take a more active interest in their children's school performance can be both more likely to respond to parental surveys and support their children to attain higher levels of academic performance. However, we do not believe that these selection processes endanger our treatment effect estimation because these processes appear in both groups. As attrition is more common in the control group, we explore the predictors of attrition and use inverse probability weighting in our regressions to take selected attrition into account. Finally, it should be noted that our final sample includes students with a higher initial level of financial knowledge than the whole population of 9th-graders, which may limit how far we can generalize our results. 8  Table 5 presents summary statistics for the treatment and control groups based on the first survey before any intervention had taken place. The initial level of financial knowledge does not statistically differ significantly between the two groups. The groups are also similar with respect to variables reflecting family background. However, the groups differ with respect to some observable background characteristics, e.g., compared with the treatment group, the control group had more Swedish-speaking students, lower average math grades, and the teaching of economics had already started in more classes. There are also some initial differences in students' attitudes toward saving and personal finance. These are observable characteristics that we can control in our empirical setting.

Estimation methods
Next, we specify a regression model to investigate the effect of game-based economics education on financial knowledge and behavior. As we explained before, the programs were not randomly placed in municipalities, so a difference-in-differences estimation strategy is appropriate. We compare the change in outcome between the first and second surveys across treatment and control groups while controlling for students' and classes' background characteristics. The basic specification is the following: where y refers to the outcome variable, which is either the level of financial knowledge or selected financial behavior outcome. The Intervention variable indicates whether student i is in the treatment or control group. In some of the specifications, we divide the treatment group further, depending on whether the student participated in one or two interventions. We also estimate the disaggregate results for each of the three interventions. The Post variable is a time variable for the separation between the pre-survey and post-survey. Our main interest is in the interaction of the Intervention and Post variables, which reveals the treatment effect of the game-based education interventions. Control variables include background variables related to student, class, and family characteristics. Prior studies have observed that students' numeracy and cognitive skills, as well as socioeconomic background, are related to financial knowledge and behavior (e.g., Frisancho 2020; Lührmann, Serra-Garcia, and Winter 2015); thus, we control for these characteristics. These include gender, mother tongue, math grade, and average grade, having a bank account, and receiving pocket money. We also include as family background characteristics the number of books at home, whether the student talks about money with parents, and parental financial literacy. The control variables are measured in the first survey to avoid any influence from the interventions on the control variables. The difference-in-differences estimation rests on the assumption that, absent intervention for both treatment and control groups, students will have similar development in their financial knowledge. All students follow the same curriculum in Finnish schools and, thus, the parallel trend assumption should hold. However, some of the classes had already started their economics education prior to the first survey, and some classes finished their economics lessons only after the second survey. These differences may cause the parallel trends assumption to not hold. However, for most classes, we have this information available, and we can thus add a control variable as well as a time interaction variable to allow for different time trends with those particular students. Therefore, observations where this information is not available are dropped in some of the regressions.
We identify the classes and students who, according to a preassignment, should have received the treatment. However, some classes did not follow through with the game-based interventions. For example, the Money Flow Challenge game was randomly offered to schools, but some teachers self-selected not to use it in their teaching. Participation in Yrityskylä was preassigned at the municipality level; however, some schools or classes did not participate, e.g., due to scheduling reasons. Thus, our empirical analysis estimates the average effect of intention to treat (ITT). We can also identify students who actually participated and received the treatment; i.e., we could estimate the average treatment effect on treated (ATT). If treatment participation reflects teachers' self-selection, participation can be correlated with the knowledge and behavioral outcomes that we study. Therefore, ATT results can be biased, but ITT results provide an unbiased picture.
However, as Ding and Lehrer (2010) discuss, selective attrition of students may bias even ITT estimates. Following their approach, we thus use inverse probability-weighted regressions to correct for potentially selective attrition. First, let's define L t+1 = 1 to indicate students who did not answer the post-intervention survey and L t+1 = 0 to indicate non-attrited students. Then, we may model the attrition process as follows: where L t i +1, * is a latent variable, Controls indicate variables from the pre-intervention survey that predict student attrition, and u is an error term. We estimate this equation with a probit model and then calculate the probability of each student's non-attrition, i.e., Pr(L t+1,I = 0) given the pre-intervention survey information. We then use the inverse of this predicted probability to weight the observations used to estimate equation 1.
In our data, the attrition may happen at the school, class, or student level. Of 982 students who answered the pre-intervention survey altogether, 342 did not answer the second survey. Of these students, 200 did not answer the second survey either because their school or class dropped out. One hundred forty-two students did not answer, even though some of their classmates did. Thus, it appears that school, class, or teacher characteristics are more important determinants of attrition than individual student characteristics.
This was confirmed when we analyzed which factors explained the attrition. Treatment group status, the language of the school, and whether teaching had started were the most significant drivers of student attrition. Lower pre-survey financial knowledge predicted the attrition in some regressions, but its effect varied depending on whether other controls were included. Other characteristics of the students did not explain the probability of attrition. Thus, we chose to use treatment status, language, and pre-intervention survey financial knowledge to calculate the probability of attrition. As teaching schedules and parental financial knowledge are used as controls in some regressions, we also use these variables to calculate the inverse probabilities for those ITT estimations, where these variables are used. Furthermore, when we separate between the intervention programs or one vs. two interventions in equation 1, we use these same treatment indicators to explain attrition. The results for the probit models explaining the attrition are available upon request.
As a robustness test to account for student dissimilarity between the intervention and control group, we also use propensity score matching and difference-in-differences estimation. However, because the theory developed by Abadie andImbens (2006, 2016) has not been extended to handle multivalued treatments, we cannot use matching techniques in our main analysis to compare one vs. two interventions or the effects of the three different programs. Therefore, we only use matching as a robustness test for the case of any treatment vs. no treatment. Table 6 presents the estimation results from difference-in-differences estimations using inverse probability weighting to explain students' financial knowledge. 9 First, we analyze students' overall performance on the financial knowledge questions and add more control variables in the second specification. In specifications 3 and 4, we separate whether students participated in one or two interventions. Then, we separately analyze how their knowledge developed in the two subfields of our survey: personal finance (specifications 5 and 6) and firms and macroeconomics (specifications 7 and 8). In table 6, we present only the estimates for the main variables of interest. All the control variables are measured at the time of the first survey, and thus, control variables related to financial behavior have not been influenced by the potential treatment. 10 From table 6, we can observe that after controlling for student background, the treatment group performs somewhat worse than the control group in the pre-survey. However, the treatment effect estimate is positive and significant, suggesting that the treatment group caught up and performed better than the control group in the second survey. The average treatment effect suggests an improvement in financial knowledge by 0.342 standard deviation in the intervention group. The size of improvement is in line with the effects reported by Kaiser and Menkhoff (2020) for financial literacy interventions in general.  In specification 2, we also consider that the teaching started and finished at different times in different classes and also control for parent's financial literacy. Classes where teaching had already started performed significantly better in the first survey and subsequently improved their scores less in the second survey (results available from the authors). Including these controls increases the main coefficient of interest (Intervention * Post) slightly.

Financial knowledge
Based on previous literature, we also expected that parental education would be an important control variable explaining student performance. However, in our results, this is not the case; thus, in the end, we did not include parental education among the control variables. We suspect that the lack of explanatory power may be because obtaining parental consent and answers in the parental survey already correlate with the family's socioeconomic background; thus, parental education does not contain additional explanatory power.
In specifications 3 and 4, we separately studied the treatment effect of one game-based intervention and that of two game-based interventions. The estimate of one intervention is positive, and the estimate of two interventions is higher and more strongly significant. We also tested whether these effects are of equal size. The F-test p-values are reported at the bottom of table 6. However, the null hypothesis of equal ITT effects cannot be rejected.
In specifications 5 through 8, we studied whether the interventions impact personal finance and macroeconomic and firm-related knowledge differently. Overall, we observe that the interventions significantly improve students' knowledge in both subfields and that the treatment effect estimates are similar in both cases.
In table 7, we analyzed the three interventions separately. In our specifications, we did not allow for complementarity or substitutability between the interventions because, otherwise, the number of interaction terms would grow excessively. From table 7, it is apparent that the initial level of financial knowledge varies by the intervention group, most notably for the group that was assigned to use Money Flow Challenge (MFC). This is surprising, considering that this is the only group where the treatment assignment was randomized.
Turning to treatment effect estimates, we notice that the treatment effect of the Yrityskylä game is positive and strongly significant. The estimates are similar to those in table 6. The treatment effect of Oma Onni is positive and significant at the 10 percent level in some specifications, but not when we include a full list of control variables in the estimation. Finally, the treatment effect of the Money Flow Challenge is not significant in any of our model specifications, although the point estimate is often of similar size to Oma Onni. Moreover, the F-test p-values at the bottom of table 7 show that the null hypothesis of equality of these three ITT estimates can be rejected only for personal finance-related questions.
The differences in the knowledge gains between the interventions are consistent with the amount of time used in the interventions (see also table 1). For instance, the students used the most time for Yrityskylä and the least time for Money Flow Challenge. Yrityskylä also has the biggest resources by far for pedagogical development and implementation of all these programs, which also might be reflected in its results. However, as Yrityskylä focused more on business life and work-life skills, it is somewhat surprising that its effects are even stronger on personal finance-related knowledge. This can perhaps be interpreted as transfer learning: for a successful program, the learning effects can spill over from one domain to another related domain (Kneppers et al. 2007). For the other two programs, we do not observe clear differences between personal finance and macroeconomic questions.

Financial behavior and attitudes
Students' level of financial knowledge correlates significantly with most aspects of self-reported financial behavior (correlations available from the authors). However, when we analyze whether the education intervention altered students' financial behavior, we observe that the treatment effects are negligible and statistically insignificant, with only a few exceptions. Table 8 presents the estimation results from difference-in-differences estimations regarding the behavioral variables. The outcome variables are binary variables; thus, we use probit estimation and report average marginal effects in table 8. In these estimations, we use the following control variables: gender, language, math grade, average grade, and whether there are over 100 books at the student's home. 11 Based on the results in table 8, we can observe that only in two of the savings questions (Savings and Finds saving profitable) is the treatment effect significant. When we separate between one and two interventions, we somewhat puzzlingly obtain a negative treatment effect for two treatments in the Saves regularly as well as Bank account questions but a positive effect in the Savings and Finds saving profitable questions. In conclusion, we find no clear effects on savings behavior.
In table 9, we report the further treatment effect estimates for financial behavior and attitudes. These variables are ordinal variables. The first four variables (Finds personal finance easy, Plans the use of money, Impulse shopping, Interest in economic issues) take values between "1" and "5, " where "1" means that the student strongly disagrees with the statement and "5" means that the student strongly agrees with the statement. The final variable is formed by aggregating all five savings-related variables in table 8. Thus, this variable takes values between 0 and 5, where high values indicate that the student has answered positively to more savings questions. The results are based on an ordered probit estimation. To save space,  Notes: inverse probability-weighted regressions. marginal effects after probit estimation. 1280 observations. Standard errors are clustered by school and are presented in parenthesis. *p < 0.10; **p < 0.05; ***p < 0.01 we report only the marginal effects for the treatment variables. The marginal effects describe the increase in the likelihood that students belong to a particular group.
According to table 9, we can observe that game-based financial education interventions positively affect students' self-reported interest in economic issues. This finding appears to be related to the fact that these students start from a lower level of interest, but they catch up by the time of the second survey. Students who participated in one treatment also report that they plan their use of money in the post-survey more often than those in the control group. With respect to the other behavioral variables, we do not observe statistically significant treatment effects.
We also separately estimated the financial behavior for the three interventions. In line with tables 8 and 9, these estimations did not reveal systematic results of the effectiveness of the individual programs. The results are available from the authors.
Overall, we conclude that even though game-based education interventions improve students' financial knowledge, we find only a very weak impact on their financial behavior and attitudes.

False discovery rates
As previously described, we have analyzed the effects of several different interventions on various knowledge and behavioral outcomes. This raises concerns of multiple inferences and that some statistically significant results may occur by chance and, thus, do not reflect real treatment effects. To consider this possibility, we calculated the False Discovery Rate (FDR) q-values following Anderson (2008). The FDR is the proportion of rejected null hypotheses that are false rejections (type I errors). Thus, the q-values can be interpreted analogously to the standard p-values.
For the statistically significant treatment effect estimates in tables 6 and 7, the corresponding q-values were less than 0.1 except for the treatment effect of a single program in columns 6 and 8 in table 6 and the treatment effects of the Oma Onni program in columns 1 and 5 in table 7. Thus, our main findings that the game-based interventions have a positive effect on financial knowledge and that two interventions appear to work better than one remain valid after considering multiple inferences.
With respect to the behavioral outcomes in table 8, the q-values of the statistically significant treatment effects were less than 0.1 for other outcome variables, but not for Finds saving profitable. We also observed unexpected negative treatment effects when we separated between one and two interventions in table 8. These negative treatment effects had q-values of 0.099, which could support our conclusion that these negative effects are likely to be false discoveries. Finally, in table 9, the q-values of the statistically significant treatment effects were less than 0.1, except for Impulse shopping and Interest in economic issues for participation in a single intervention. Notes: inverse probability-weighted regressions. marginal effects after ordered probit estimation. 1280 observations. *p < 0.10; **p < 0.05; ***p < 0.01 Notes: Kernel-based propensity score matching and difference-in-differences estimation with a common support. *p < 0.10; **p < 0.05; ***p < 0.01 Table 4 lists significant imbalances in the pretreatment covariates between the treatment and control groups, which can compromise the parallel trends assumption. We controlled for these observable differences in our baseline estimations, and we now conduct further robustness checks.

Robustness tests
First, we estimated the treatment effect by applying propensity score (PS) matching and difference-in-differences estimation. The first step was to calculate the probability of treatment using probit estimation (results available from the authors). We estimated the probability conditioning on gender, language, math grade, average grade, and whether the teaching had started. The choice of variables was based on the observed differences in table 5. All variables are statistically significant in the probit estimation, except for math and average grade, which are individually insignificant but jointly significant. We do not include other variables in the probit estimation because including additional but statistically not significant regressors may lead to high variance and problems with the common support condition (Caliendo and Kopeinig 2008). The matching reduces the differences in the covariates, and the remaining differences are statistically insignificant.
After the estimation of the PS, we estimated the average treatment effect on the applicable students using difference-in-differences estimation and kernel matching. The results are presented in table 10. The PS matching results show that the treatment effect on financial knowledge is approximately 0.3 standard deviation. The estimate is statistically significant but slightly lower than the estimates in table 5, where the estimates are in the range of 0.35 to 0.42. Further tests, which used coarsened exact matching on the same set of variables, again confirmed the main results. We also studied the effects on behavioral outcomes using the matching methods previously described. (These results are available from the authors.) PS matching showed statistically significant effects on the following outcomes: Savings, Finds personal finance easy, and Interest in economic issues. The first and third results are in line with tables 8 and 9, but the second result is new.
The validity of the parallel trends assumption also could be compromised if teaching in the control group and treatment group differed in other respects outside of the game-based interventions. We also asked teachers about the number of hours used for teaching and whether any other games were used to see whether the treatment and control groups differed. The use of other games was more common in the control group. Furthermore, control group teachers used slightly more hours to teach economics. Thus, we included these variables and their interaction with Post in the difference-in-differences estimations. However, these variables were not statistically significant and did not influence the main coefficient of interest (results available from authors). It is worth noting that the other games that the teachers mention are short games that take only a fraction of study lessons to play. Thus, it is plausible that their effect would be small.
Another difference between the students is their mother tongue, and thus, the study material they use in class. As table 5 indicates, Swedish-speaking students form a larger share of the control group than the treatment group. In the PS matching above, we matched the students based on language as well as other characteristics. Furthermore, we re-estimated table 6 without the Swedish-speaking students. There was very little change in the treatment effect estimates.
Finally, there is attrition in our sample because all students did not answer the post-survey. We accounted for this, using inverse probability weighting in our regressions. As a further analysis of the implications of this attrition, we conducted unweighted regressions and constructed bounds on the treatment effects on financial knowledge following Drexler, Fischer, and Schoar (2014). While some of the treatment effects in tables 6 and 7 are sensitive to the assumptions about the attrition process, this is not the case when the full list of control variables is included or when we analyze the effects of participation in two interventions or in Yrityskylä program. 12

Conclusions
In this article, we studied the impact of game-based interventions in teaching economics relative to the impact of traditional teaching methods in Finnish lower-secondary schools. Our findings indicate that learning outcomes are better with game-based interventions when the relevant measure is economic knowledge. This applies both to personal finance questions as well as to more macro-and firm-oriented questions. There is also some evidence that game-based interventions positively impact the interest of students in economic issues.
However, the impacts on (self-reported) behaviors are weak. Methodologically, the challenge is that student behaviors are hard to measure for under-aged students who do not make important economic decisions independently. Future studies could investigate how interventions could influence behavioral outcomes within game-based simulations.
The study took place in the context of Finnish 15-year-old students. The academic performance for this group has been found to be relatively high compared with other countries, such as the United States. The recent OECD (2020) study showed this comparison also to be true for financial literacy. The question arises as to what extent the results can be generalized to other settings. However, the external validity problem of the samples drawn from certain contexts is almost always present in empirical work. Although most published studies on economic education have been conducted in the United States, studying a European country may provide further validity to the research on economic education. Additionally, the specific elements in economic games in the Finnish context that would question the generalizability of the results are not clear. The game-based approaches described in this article have their counterparts in other countries as well, including the United States.
While similar results have been obtained in prior literature, a novelty of our findings is that instead of focusing on a situation where there is a games intervention vs. an absence of intervention, these findings relate to comparisons between two different types of teaching approaches. The findings strongly support pedagogical innovations in the field of personal finance and economics. The use of game-based learning approaches, which are a subset of active learning approaches, shows promise in simultaneously making studying economics more practical and fun. The strongest learning effects appear to be related to a game that students play in teams and that has diverse learning objectives, including interpersonal and communication skills. This game can also be related to content other than just personal finance. An interesting question for future studies of the use of games in financial education is to see whether direct approaches work best or whether learning is enhanced by only tangentially participating in games that relate to financial literacy. The direct approach is taken by many available mobile games, while there are examples of indirect approaches as well, e.g., economics applications in blockbuster games such as Minecraft or The Sims.

Funding
This work was supported by the research grants 305292 and 327241 from the Academy of Finland. The Academy of Finland did not participate in the research in any other than funding role, and it has no independent interest in the research results.
Notes 8. However, we also tested whether the treatment effects were heterogeneous with respect to students' academic performance. We did not find any statistically significant heterogeneity and, thus, do not report these results. 9. Unweighted difference-in-differences estimations yielded very similar results. 10. The full-result table with coefficient estimates for the control variables is available on request from the authors. 11. We do not include the following as control variables: whether the student talks about financial matters at home, has a bank account, or receives pocket money from parents, because these reflect student's behavior and are potential outcome variables. 12. The results are available upon request.