RETHINKING HEURISTICS – CHARACTERIZATIONS AND VIGNETTES

The concept of “heuristics” or “heuristic strategies” is central to (mathematical) problem solving and related research; however, there is no generally accepted definition of this term. Trying to clarify the concept might help avoiding misunderstandings and difficulties in dealing with studies that use different terms meaning the same concepts or that use the same terms meaning different concepts. Therefore, the research presented in this paper aims at a clarification of the term “heuristics” and suggestions for the use of it in future research. Building on previous work from last year’s ProMath conference, the consequences of using different characterizations of “heuristics” on vignettes (= short, completed scenes) of problem solving attempts are investigated. The conceptualization of “heuristics” has a significant influence on the types and numbers of perceived heuristics, which in turn affects empirical studies that identify and analyze heuristic strategies.


Introduction
Problem solving is an integral part of both mathematics - Halmos (1980) even calls it the "heart of mathematics" -and of mathematics education (cf.Schoenfeld 1992) and mathematics in school (cf. NCTM 2000) respectively.When actively engaging in and researching on (mathematical) problem solving, heuristics play a central role (cf. Schoenfeld 1985).This article theoretically deals with research on problem solving with a focus on heuristics.Different conceptualizations of the term "heuristics" will be compared and discussed referring to vignettes (= a short description of a process) of problem solving attempts.It will be shown that different conceptualizations could lead to different interpretations of empirical data and, thus, to different results.
The understanding of a concept like heuristics can be implicit or explicit, it can be based on (prototypic) examples or on (rigorous) definitions (e.g., Goldstein 2005, ch. 8).For a scientific debate, the need arises for definitions or characterizations as the basis of understanding.And even in practice-oriented situations, an example-based understanding might lead to imprecise conclusions or even misunderstandings.
In the course of this paper, vignettes are used to illustrate the impact of different understandings and characterizations of heuristics on the issue.A discussion on a purely theoretical basis would be very abstract and hard to follow.

Theoretical Background
Problem solving is a very important part of mathematics and thus also fundamental for school mathematics (cf.Schoenfeld 1992).The term "problem solving" has different meanings ranging from solving routine tasks to solving perplexing or difficult situations (cf. ibid.).This article refers to the latter interpretation which is also the position of Mayer and Wittrock: "When you are faced with a problem and you are not aware of any obvious solution method, you must engage in a form of cognitive processing called problem solving.Problem solving is cognitive processing directed at achieving a goal when no solution method is obvious to the problem solver […]." (Mayer & Wittrock 2006, p. 287) It is important to note that the attribute "problem" depends on the solver, not on the task.A difficult problem for one student can be a routine task for another student.Thus, research on problem solving should focus on the problem solving process.
A major role in those processes is played by heuristics (cf.Schoenfeld 1985, p. 44 f.), which are "rules of thumb for successful problem solving" (ibid., p. 23) or "methods and rules of discovery and invention" (Pólya 1945, p. 112) if he did so, it would be conscious act of skillful -even heuristical?-calculation.
As in the example of calculating a discount, in empirical research there will always be some uncertainty in recognizing heuristic acts and distinguishing them from non-heuristic acts.Most researchers try to overcome this uncertainty by developing coding manuals.
Acceptable or even good interrater-agreement in the application of such manuals helps to approach heuristic acts empirically.For the sake of simplicity and practicability, many studies then count the number of heuristic acts and calculate its interrelation with other aspects of problem solving -even though this might not be the "best" way of approaching heuristics.In the end there remains one essential question: is it possible to "count" and "measure" a way of thinking?

Problem Solving Research Regarding Heuristics
As stated above, in studies on problem solving heuristic acts are often coded and counted.
There are several studies that report a positive correlation between the use of heuristics and the success in solving problems or the development of problem solving abilities.For example: • Kilpatrick (1967) showed weak positive correlations of 56 8 th graders in task-based interview situations regarding the use of heuristics and the success in solving presented problems.
• Lucas (1974) compared the problem solving performances of two groups of university students (30 students overall) -one control group and one experimental group that was exposed to a heuristics training for 13 weeks.In the posttest interviews, the experimental group was significantly favored (using chi-square tests) in both the scores used to measure problem solving ability and the frequency of using certain heuristic strategies.
• Kantowski (1974) exposed eight 9 th graders to a heuristic training of 5 months.She also used tasked-based interviews in which the students were asked to "think aloud".For her analysis, she divided the group into successful and unsuccessful students (median split).The better performing students showed significantly higher frequencies in the use of goal-oriented heuristics.Kantowski's data also indicate that the tendency to use heuristics increases as the problem solving ability develops.• Rott (2012) analyzed videotaped problem solving processes of 32 5 th graders working in pairs on two problems.The results show significant moderate positive correlations between the number of heuristics used by the pupils and their success in solving the problems.
Trying to summarize the findings of these studies, there seems to be a positive correlation (not necessarily in a statistical sense) between the use of heuristics, the success in problem solving and increasing competencies in problem solving respectively.However, these studies rarely address the question how the use of heuristics actually affects problem solving competencies.
Three of these studies, Koichu et al. (2007), Kantowski (1974), andRott (2012), will be discussed more thoroughly (see below) with a focus on their methodology regarding the identification of heuristics.

Defining the Term "Heuristics"
Despite its importance for problem solving (research), there is no generally accepted definition of the term "heuristic".McClintock (1979, p. 174) calls heuristics "the ancient, illdefined discipline" and other authors approve that there is no agreed upon characterization (e.g., Romanycia & Pelletier 1985;Rott 2014).In fact, in most studies the term "heuristic" is not introduced by a proper definition but only by examples of heuristic strategies and techniques (like "working backwards" or "examining special cases").It will be shown that this lack of characterizations and definitions can be problematic.
Trying to obtain a proper definition, Rott (2014) compared different conceptualizations of the term "heuristic" according to a set of categories that were derived both deductively (from the literature) and inductively (by analyzing and comparing the conceptualizations).
The results of this research can be summarized this way: There are several categories in which the characterizations of different authors differ significantly.For example, regarding the distinction of algorithms and heuristics, some authors "include [into the concept of heuristics] some procedures that would better be classified as algorithms" (Kilpatrick 1967, p.19), whereas others define heuristics as the opposite of algorithms (cf.Kantowski 1974).
Some definitions see using heuristics as an unconscious act (e.g., Kahneman 2012), opposed to other characterizations in which using heuristics is an act one is aware of (e.g., Koichu et al. 2007).Also, there are authors that explicitly include metacognitive actions into their definition of heuristics (e.g., Koichu et al. 2007) whereas others explicitly differentiate between metacognition and heuristics (cf.Schukajlow & Leiss 2011 who distinguish between cognitive and metacognitive strategies).Examples for different characterizations are given in Table 1. 1ott ( 2014) additionally asked experts of problem solving research for their level of agreement with the characterizations discussed previously and for their reasons to agree or disagree to those characterizations via an e-mail questionnaire.There were no characterizations (from a set of 9) that had been favored or rejected by all 18 experts that took part in that study.The reasons for approval that were shared amongst the experts had mostly been a familiarity with the characterizations (the experts had identified the ones by Pólya and Schoenfeld even though the characterizations had been anonymized).Reasons for approval and rejection regarding content-related reasoning differed between the experts.
Using conceptualizations that differ in the ways described above lead to different ways of interpreting empirical data.The definitions that are used in constructing coding manuals do significantly influence what (empirically observed) actions are regarded as heuristics.To show this potential effect of different characterizations, three studies that used coding manuals to identify heuristic techniques within empirical data will be presented: Koichu, Berman, and Moore (2007); Kantowski (1974);and Rott (2012).These studies have been chosen, because they (a) provided characterizations of the term heuristics (which cannot be taken for granted), (b) provided insights into their coding manuals, and (c) promised to lead to interesting results regarding the comparison presented here."A precise definition of heuristic in information-processing terms will probably include some procedures that would be better classed as algorithms and will almost necessarily exclude some procedures that aid in problem solving.Let us forego such precision, therefore, and define a heuristic as any device, technique, rule of thumb, etc. that improves problem-solving performance.We consider heuristics to be typically provisional, without guarantee of effectiveness, but we do not attempt to contrast them with algorithms."Heuristics is a collective term for devices, methods, or (cognitive) tools, often based on experience.They are used under the assumption of being helpful when solving a problem (but do not guarantee a solution).There are general (e.g., "working backwards") as well as domainspecific (e.g., "reduce fractions first") heuristics.Heuristics being helpful regards all stages of working on a problem, the analysis of its initial state, its transformation as well as its evaluation.Heuristics foster problem solving by reducing effort (e.g., by narrowing the search space), by generating new ideas (e.g., by changing the problem's way of representation or by widening the search space), or by structuring (e.g., by ordering the search space or by providing strategies for working on or evaluating a problem).Though their nature is cognitive, the application and evaluation of heuristics is operated by metacognition."Kantowski (1974, p. 6 f.)

C14
"With some experience, the problem solver will use a more sophisticated heuristic approach, that is, he will employ non-algorithmic devices that help to reduce the search space.Heuristics may be described as rules for selecting search paths through a problem space; the theory of problem solving is concerned with systems of heuristics or methods of search which will exploit the information in the task environment.[...]" C15 Kahneman (2012, p. 98) "The technical definition of heuristic is a simple procedure that helps find adequate, though often imperfect, answers to difficult questions.The word comes from the same root as eureka." ["Pólya's heuristics are strategic procedures that are deliberately implemented by System 2. But the heuristics that I discuss in this chapter are not chosen; they are a consequence of the mental shotgun, the imprecise control we have over targeting our responses to questions." "How happy are you with your life these days?"!What is my mood right now?" "How popular will the president be six months from now?" !"How popular is the president right now?"] Elaboration strategies control linking new information with previous knowledge.Examples for elaboration strategies include remembering a similar task or searching for suitable mathematical methods.

•
The function of the organization strategies is to better structure present information.For example, a typical organization strategy while solving a modelling task is to draw a sketch.
[…] [From these cognitive strategies,] the metacognitive strategies like planning, control, and regulation are to be differentiated.Working on a mathematical task, the solving process can explicitly be planned.Within this process, the learner needs to realize how to proceed to reach his goal -the solution of the task.With the help of control strategies, the solving process is being monitored.[…] The metacognitive strategies do not -unlike the cognitive strategiesinfluence the direct information processing.They rather control the selection and application of cognitive strategies." The characterizations and the coding manuals of these three studies are then used to interpret and to analyze three constructed examples of problem solving situations.These examples are given in the form of vignettes.They have been constructed by the author of this paper who is experienced in analyzing problem solving processes (secondary school as well as university level) with a focus on heuristics.

Research Intention
The research intention of this article is to help clarifying the (ill-defined) term "heuristic" and to provide a better understanding of it.In this study, the influence that different characterizations have on identifying heuristics in (real and fictional) problem solving situations will be analyzed.This also raises issues about the comparability of studies on heuristics and their results.

Comparison of methods used to analyze heuristics of the selected studies
To be able to apply the different characterizations and their coding manuals respectively to the vignettes, the methodology of the chosen studies had to be studied carefully.First of all, it has been confirmed that the coding manuals match the underlying characterizations.An overview of the methods used to analyze heuristics of the selected studies is presented below.Koichu et al. (2007) conducted thinking-aloud interviews with individual participants that lasted 30 to 90 minutes.The interviewees were 8 th graders that participated in a problem solving intervention.The videos of these interviews were transcribed and analyzed by segmentation into content units.These units -determined as the largest unbroken parts of the transcript that bear a particular heuristic interpretation -were coded in terms of a protocol coding scheme (cf.Table 2 for an excerpt).Koichu evaluated the coding procedure Table 2 An excerpt of the protocol coding scheme (Koichu et al. 2007) Heuristic behaviors Description 1. Planning Evaluating of whether or not it is worthwhile to make a particular problem-solving step or several steps before doing it.
2. Self-evaluating Evaluating of whether or not it was worthwhile to use a particular problem solving method.
2a. Local self-evaluating Self-evaluating after a particular step of a solution.

Activating a previous experience
Activating knowledge and problem-solving methods that are known to the student from his or her past learning and problem-solving experience.

Creating a model
Representing a problem by means of a representational system different from a given representational system.For example, translating a word problem into algebraic equation.

Exploring particular cases
Approaching a given problem by consideration of its particular cases and (generic) examples.

Finding what is easy to find
Search for additional data, which can be derived easily from the given.For example, two angles of a triangle are given, and a student finds the third angle, even though he or she is uncertain how to use it to progress the solution.
10. Generalization Approaching a more general problem than a given one, when it may simplify the solution.Kantowski (1974) audiotaped the problem solving processes of 9 th graders.She used a coding scheme in the form of a checklist developed by Kilpatrick (1967) based on Pólya's (1945) famous list of problem solving techniques.For each problem her students had worked on, she calculated a "Process-Product-Score" that included (amongst others) checking for a correct solution as well as for devising a plan and for the absence of computational errors.The use of "goal oriented heuristics" was also coded but not part of the score.Indications for those heuristics were given in a checklist that followed Pólya's list of heuristics (cf.Table 3).Kantowski did not compute interrater reliability as she was the only one to code her data.
Rott (2012) videotaped the processes of 5th graders working in pairs on mathematical problems.All processes were coded by the author and trained research assistants with an agreement rate of 85% and more for identifying and characterizing heuristics using a coding manual that contained general and task-specific instructions (cf.Table 4).After calculating the intercoder-reliability, all differing codes have been recoded in agreement of the coders.Coasters: positions of the two squares which make it evident that the marked area amounts to one fourth of a square.

Backtracking
Working forwards until being stuck; then tacking back steps to try alternate ways.
Number Series: working till getting stuck (e.g., at 8 or 9) and then deleting numbers till being able to proceed with another number.

Construction and coding of the vignettes
The vignettes have been constructed by using tasks that were familiar to the author and for which he knew authentic solutions by a broad variety of students.
• Problem 1 is the geometrical problem of proving Thales' Theorem.Pólya (1945, p. 25) calls tasks like this "problems to prove".For its solution an auxiliary line has to be drawn from the center of the circle to point C and the triangles that come to existence because of this line have to be identified as isosceles.
• Problem 2 is an arithmetical problem that can be solved either by excessive manipulation of equations or by looking for patterns and working on sub-problems.
• Problem 3 is a simple arithmetical problem whose solution has to be monitored thoroughly because the numbers suggest a wrong solution and even university students often get this task wrong. 2or each task, three different solutions by prototypical students are given, to cover different styles of working on problems.The students' names3 hint at their type of solving problems: • Alice mostly uses algorithmic approaches -up to the point where it could be discussed whether she really engages in problem solving; • Emma is supposed to be an expert problem solver that uses a variety of heuristics and metacognitive strategies; • Nora is supposed to be a novice problem solver who is not familiar with heuristics and metacognitive actions and easily abandons working on a task.
Transcripts of real problem solving attempts could have been given, but that idea was discarded in favor of constructed ones.Constructed situations get their message communicated more precisely without the need to explain the circumstances of their origin.
The vignettes are presented in Tables 5.a, 6.a, and 7.a.
The pointed problem solving situations of these three fictional students have been analyzed and coded with each of the three coding manuals outlined above.Most relevant for the research presented here is whether a stated action is being recognized as heuristic or not.Therefore, the codings are mostly given as Y (yes) or N (no); they are presented in Tables 5.b, 6.b, and 7.b.

Table 5.a Problem 1 and according vignettes
Problem 1: You are given a circle with center M and three distinct points A, B, and C on its circumference.AB is a diameter of the circle.Show that the angle ∠ACB is a right angle (Thales' Theorem).
After reading the problem, Alice says: "Oh, I think I remember most of this proof."She silently draws a line from M to C and shades the angles.Then she writes down the following lines: Emma comments on her actions: "Let's have a look.Auxiliary lines are always helpful in Geometry."She draws a line from M to C and says: "Giving names to the angles might be useful."[a, b, g, d, m1, and m2] She looks at the figure for some time and mumbles: "Hmm, what am I missing?(…) Is there some information I haven't used so far?"After another thirty seconds, she adds: "Oh, I see.This is three times the radius.The triangles are isosceles.""Okay, how does it help me?The base angles are equal."She renames the angles in her figure [a1, a2, b1, b2, m1, and m2] and continues: "Now I should be able to solve this problem algebraically.The sum of the interior angles of a triangle is always 180°" She writes: Nora starts measuring the angles to convince herself that ∠ACB is indeed a right angle: "Okay, this really is 90°.But how can I show it?If I remember correctly, I have to name the angles."[a, b, g] After a minute, she goes on: "I have no idea how to show this.I learned this in school, some time ago.That's just the way it is.And the three angles always add up to 180°, I remember that, too."Problem 2: * Show that for all a, b, c > 0 the following inequation is true: Alice says: "This seems to be easy.I have to get rid of the fractions, first." She hesitates: "Okay, this doesn't seem to be that easy, I have to get rid of the brackets as well." She concludes: "Everything is greater than zero, thus the inequation is true.
Emma begins by saying: "If I try to eliminate the brackets, I'll miscalculate for sure.This looks like cyclic permutation to me, does this help me?"After scribbling some lines on her paper, she looks at the following inequation: Emma continues, "I can compare the fractions piece by piece." 1  +  > 1  +  +  She comments on it: "Okay, this seems to be correct.And it is true for (a+c) and (b+c) as well." After writing an answer, she looks at the problem again."Is 1/(a+b) really greater than 1/(a+b+c) for all a, b, and c?Of course, it is!By rearranging it, I can see that it is true for a, b, and c greater than zero."Alice says: "This seems to be easy.I just have to set up a system of equations." +  = 1.1 ∧  −  = 1 She proceeds without commenting on her actions: Emma begins by saying: "This seems to be easy.The bat costs $1 and the ball $0.1."She writes down an answer, scratches her head and says: "Stop.This way, the bat costs only $0.9 more than the ball.It has to cost $1.1."Before correcting the numbers, she hesitates again: "This is still not right.This time, bat and ball would cost $1.2 in total.""Of course," she continues, "I didn't pay attention.The ball has to cost $0.05."She then finishes her answer and states: "Now, the bat costs $1.05 and together they cost $1.1." Like Emma, Nora answers immediately: "This is easy, the ball costs $0.1."She writes down "$0.1"; she underlines her answer and smiles.
* Source: Kahneman & Frederick (2002), a task often used in research on critical thinking.Please note that according to Kahneman (2012) the answer "$0.1" would be a product of the use of heuristics by subconscious thinking procedures.

Results
The vignettes are presented in Tables 5.a, 6.a, and 7.a and the related results of the coding processes are presented in Tables 5.b, 6.b, and 7.b.
Table 8 provides a summary of the codings obtained with the different coding manuals.
Reported and compared are the numbers of actions in the vignettes and of actions that have been coded as "heuristic".Applying different characterizations (and thus different coding manuals) to the same vignettes leads to explicitly It can be seen that different actions are regarded as being "heuristic" -in particular the use of metacognitive strategies and algorithmic procedures causes differences.Working on problem 1, Emma uses equations to look for information on the angle at point C. Using Kantowski's manual this action is being recognized as algorithmic and, therefore, not heuristic.Koichu's and Rott's coding manuals identify this action as heuristic, because Emma is not sure about the outcome of her manipulations when she starts using equations.
When Nora measures the angle in the figure given with problem 1, Kantowki's manual does not recognize this action as heuristic, because the information that the angle is 90 degrees has already been given and measuring the angle does not reduce the search space.
Rott's manual also leads to this action not being coded as heuristic, because evaluating given information would be recognized as being metacognitive and, thus, not being heuristic.On the other hand, Koichu et al. would recognize this action as heuristic because it helps Nora to better understand the problem.
These differences in recognizing and coding heuristics have important implications on the analyzed studies' results.For example, Koichu et al. (2007) reported an increased number of heuristics used in the interviews after their heuristic training.Without distinguishing between heuristic strategies and metacognitive actions it remains unclear whether the success of the training is due to the improved use of problem solving strategies, the better use of control strategies, or a combination of both effects.

Discussion and Conclusions
The results presented in this article show that different characterizations of the term "heuristic" lead to different coding manuals for researching on heuristics in empirical studies.The different coding manuals then lead to different actions being recognized as heuristic and, therefore, being counted and further analyzed in the purpose of those studies.
This observation has clear implications for research projects and their results.Unless it is clearly stated whether observed actions (declared as "heuristic") are cognitive, metacognitive, or both, the validity of these studies' results is questionable.
The comparability of the results of such studies is limited, as long as it is unclear what kind of actions are being analyzed and counted or interpreted.A study could report on high numbers of "heuristic actions" referring to "planning" and "reflecting actions" only; such a result would be worthless for meta-studies on heuristics that exclude metacognitive actions.
Several studies on heuristics report unsatisfactory or even confusing results (especially regarding the learnability of heuristics).This could be due to the fact that these studies actually report on different concepts.
Therefore, studies focusing on heuristics should provide explicitly stated (and discussed) characterizations.The three studies used in this article are commendable exceptions; by presenting their definitions and methodologies, they allow a thorough discussion of their results.
The clarification of the concept of heuristics is sparsely studied.Future approaches on this topic could analyze more research projects that study the use of heuristics in problem solving processes.Another approach would be to study the history of heuristics starting with the ancient Greeks (cf.Zimmermann 1991).
] cognitive and metacognitive learning strategies related to the solving process […] […] the group of the cognitive strategies.[…].• Repetition strategies are responsible for the selection and storage of new information in the solving process.[…] • +  ⇔  +  +  >  +  Nora tests the inequation for several numerical examples.She inserts large numbers for a, b, and c as well as very small numbers like 0.0001.She then tries the inequation for a = b = c, especially for a = b = c = 1 and discards the idea to insert 0 for the variables.She concludes: "This is true for every example I tried.But I don't know why." * Source: Bruder (2003): http://www.math-learning.com/files/learnmethod.pdf (29.04.2014)

•
Koichu, Berman, and Moore (2007)describe a study in which 37 8 th graders participated in a 5-month heuristic training.The students had been interviewed at the beginning, in the middle, and at the end of the study, working on three problems in each interview.The results show that the students' mean scores in a

Table 1
Characterizations chosen for the discussion in this article If algorithms are heuristics, they are the least interesting sort for our purposes."We refer to the concept of heuristics as a systematic approach to representation, analysis and transformation of scholastic mathematical problems that actual (or potential) solvers of those problems use (or can use) in planning and monitoring their solutions.Some heuristics are narrow and domain-specific [...], whereas others are universal and cut across many problemsolving domains [...].[...] Heuristics at large can be seen as a cognitive tool used to approach problems, effectiveness of which is never known in advance."

Table 5 .b
Codings of heuristic actions regarding the vignettes of problem 1

Table 6 .
a Problem 2 and according vignettes

Table 6 .b
Codings of heuristic actions regarding the vignettes of problem 2

Table 7 .
a Problem 3 and according vignettes Problem 3: * A bat and a ball cost $1.10 in total.The bat costs $1 more than the ball.How much does the ball cost?

Table 7 .
b Codings of heuristic actions regarding the vignettes of problem 3

Table 8
Number of coded heuristics compared to the number of stated actions Emma hesitates at the beginning of working on problem 2 and thinks about a way to tackle this task.This action is considered as a heuristic by bothKoichu et al.   and Kantowski; Rott,on the other hand, recognizes a behavior like this as metacognitiveimportant for working on a problem but not a heuristic action.