Multimedia in Geographic Education
J.B. Krygier, Catherine Reeves, David DiBiase, and Jason Cupp
Greatly revised version in Journal of Geography in Higher Education. 21:1, 1997. pp. 17-39.
... to introduction to "Design, Implementation, and Evaluation..."
Copies of the multimedia resources discussed in this paper are free and can be downloaded via the WWW.
MULTIMEDIA IN GEOGRAPHIC EDUCATION: Design, Implementation, and Evaluation of Multimedia Resources for Geography and Earth Science Education
ABSTRACT: This paper describes an educational application of multimedia for geography and earth science education based on the assumption that multimedia is more than mere technology. This paper argues for an approach to educational multimedia design focused on a coherent set of multimedia design guidelines informed by a broad array of evaluation functions. Further, it is argued that such design and evaluation guidelines must be shaped by broader educational and content (geography and earth science) goals. It is suggested that this approach to the design, implementation, and evaluation of educational multimedia resources may guide other similar projects.
INTRODUCTION
Multimedia is receiving tremendous attention in both academic and popular media. Proponents of multimedia claim that it has the ability to change the way we understand, think, learn, and work, and have heralded the end of printed books and static graphics. Multimedia, then, is seen as much more than mere technology. Both research and educational applications of multimedia are garnering increasing attention in cartography and geography. Little is known about the prospects of multimedia as a research or educational method in geography and the earth sciences, and there is a paucity of literature on the topic. This paper focuses on multimedia as an educational method and confronts issues including the planning, design, production, and evaluation of multimedia in the classroom. I argue that it is fundamentally important to avoid approaching multimedia as mere technique independent from broader educational and content (geography and earth science) goals and issues. This paper details an approach to the design, implementation, and evaluation of educational resources wedded to an articulated sense of educational and content goals.
I begin this paper by briefly reviewing a general definition of multimedia and how our educational multimedia project arose within the context of "technology classrooms" at the Pennsylvania State University. In the first section of the paper I describe our overall multimedia design strategy. I describe an approach to educational multimedia design based on coherent and consistent graphic design principles and the matching of educational and content goals to particular multimedia forms and functions. A coherent design strategy should guide the production and implementation of educational multimedia resources from the onset of the project. In the second section of the paper I describe the multiple functions of evaluation used to inform our multimedia design strategy. I describe an approach to educational multimedia evaluation based on a broad range of evaluation functions and methods, also implemented from the onset of the project. Evaluation must play an informative role throughout the design, production, and implementation of multimedia resources, rather than only being implemented at the end of the process. Finally, design and evaluation strategies should be bound together by an iterative design approach where the goals and expertise of content experts and educators shape and are shaped by the goals and expertise of multimedia designers.
1. SETTING THE CONTEXT
Multimedia is typically defined as an array of representational forms (text, image, map, diagram, sound, video, etc.) and hypermedia is multimedia with substantive links between the various representational forms (Andrews and Tilton 1993). For convenience, I will collapse the two terms into one (multimedia) in this paper. Multimedia does not necessarily require computers. For example, geographic educators often combine the use of slides, overheads, chalkboards, movies, videos, and sound recordings in their lectures and academic presentations. Further, atlases have a long tradition of integrating text, images, maps, diagrams, and graphs. Thus the ideas behind multimedia is not completely new to geographers and cartographers and we can draw on past experience to assist in the design and production of multimedia. On the other hand, approaching multimedia as nothing substantially different from what we have done in the past is problematic and may limit our understanding of its possibilities.
The Pennsylvania State University has been developing "technology classrooms" for the past five years (Morrow and Boettcher 1995). These classrooms are equipped with an array of computers, software, network connections, and projection equipment. The College of Earth and Mineral Sciences (EMS) has made funds available for the development of educational multimedia resources for the course entitled "Gaia: An Introduction to Earth Science" taught in the College. The Gaia course is a large-enrollment course taught in several sections each semester by different faculty.
The Deasy GeoGraphics Laboratory, affiliated with the Department of Geography and the College of EMS, offered the expertise required to design, produce, implement, and evaluate multimedia teaching resources for the Gaia course (DiBiase and Krygier 1994). Two graduate students, one undergraduate student, the director of the Deasy GeoGraphics Laboratory, and an educational technology specialist have spent the last two years working on multimedia resources for the Gaia course. The resources have been produced on the Macintosh using Macromedia Director authoring software.
Our design, production, and evaluation strategy has been to synthesize an awareness of geographical and educational goals with multimedia design goals. Frequent meetings with content experts and students as well as ongoing evaluation methods have informed our approach to educational multimedia design from the onset of the project. The result of our work is not only a series of multimedia resources for teaching a single course, but a detailed planning, design, production, and evaluation strategy currently being used to develop materials for other courses. The process which produced our design strategy, the gist of this paper, is summarized in figure 1. Figure 1 reveals two primary interactions: those between content / educational goals and multimedia design goals, and, within the context of the designer's goals, interaction between multimedia design guidelines and evaluation guidelines. I stress that the process of such interactions is integrated, as the diagram suggests. The following description of the process, however, is structured in terms of our multimedia design guidelines (in section two) and our multimedia evaluation guidelines (in section three). The content/educational goals and the multimedia desginer's goals are subsumed in these two sections.
2. DESIGNING EDUCATIONAL MULTIMEDIA RESOURCES
Our strategy for the design and production of educational multimedia resources is divided into two subsections. The first concerns the design of the general interface and structure for all of our resources, which we call "lectureware." The second subsection concerns a typology of multimedia forms and functions used to guide the design of individual resources - defined as particular multimedia units which explain a single concept or idea. Both our lectureware and the resource typology were informed by the evaluatory methods described in the third section of this paper.
2.1. Multimedia Design and Lectureware
The design of the general interface and overall structure of our materials has focused on the creation of what we call "lectureware." As this name implies, our materials are designed to be used during lecture by instructors. The instructor is, thus, still important: lectureware resources do not teach by themselves. Lectureware is distinguished from courseware, which students use by themselves outside of lecture. Three design strategies which shaped our lectureware are reviewed below. These strategies include resources as a single concept or idea, an easy-to-use interface, and general graphic design guidelines.
Resources as a Single Concept or Idea: Our first lectureware design strategy is that each multimedia resource should consist of one basic concept or idea. Different instructors teach the same material in different ways. Any instructor should be able to piece together a series of resources in the order he or she is comfortable with. In addition, an instructor should be able to use the same concept (resource) in several different lectures as appropriate. However, while the resources were designed to stand alone, there was a simultaneous attempt to make sure that resources can be coherently related to each other when appropriate. Some resources have been designed to aid the instructor in relating particular concepts or ideas learned to a more general goal. We have used graphic icons to make such relationships explicit. For example, an icon which summarizes the concept learned in one resource can be used at the beginning of a related resource which depends on understanding the material summarized in the icon.
Easy-to-use Interface: Our second lectureware design strategy is the provision of an easy-to-use interface. The impetus for an easy-to-use interface design is that instructors are busy people with little time to learn a complex new computer interface. Further, the instructor must be able to concentrate on the material being taught during a lecture, and cannot be distracted by a convoluted, confusing, or inconsistent user interface. To this end, we have focused on three design goals for our user interface.
First, we have constructed a simple menu-driven interface allowing easy access to the selected list of resources and other basic commands (such as editing a lecture menu, blanking the screen, or quitting). Second, we provide a "lecture building" resource which allows the instructor to view available resources, search for particular topic, and add resources to a "lecture list." This lecture list is then installed as a universal menu item used to access the resources during the lecture. Finally, we have endeavored to provide a consistent set of navigational buttons to guide movement within individual resources. Our general goal is to make sure that the same buttons do the same things and are located in the same place on the screen from resource to resource.
General Graphic Design Guidelines: Our third lectureware design strategy consists of a series of overall design guidelines used to shape our resources. Strict consistency in design is used to minimize confusion and to maximize the manner in which the different resources can be used together. Our goal is to make sure that a series of resources looks like they were designed to be used in that order.
These three design strategies - resources as a single concept or idea, easy-to-use interface, and general graphic design guidelines - linked to educational and content goals, have been used to shape the concept of lectureware. Our approach to lectureware is complemented by and closely interrelated to our multimedia resource typology which is used to shape the design of particular multimedia resources.
2.2. Multimedia Design and the Resource Typology
The lectureware design guidelines are complemented by what we have termed a "resource typology." The typology provides a means of linking content goals to appropriate representational forms, and assists in making design decisions about particular resources. Our resource typology consists of two dimensions (figure 2: Resource Typology). The first dimension consists of a range of representational forms, including imagery, maps, diagrams, graphs, and tables, which encompass the range of representational forms we have to select from in producing our resources. The second dimension of the resource typology encompasses a range of resource functions. The resource functions include static, animated, sequential, hierarchical, and conditional resources.
2.2.1. Resource Forms
The resource forms are useful for matching particular educational goals to appropriate representational forms. For example, an instructor may be using a table of data values in class only because they have not found the time to convert the data into a graph. Careful consideration of educational goals in tandem with a consideration of available resource forms has allowed us to avoid replicating inadequate or inappropriate materials into our resources.
Beyond an important role for matching educational and content goals to appropriate resource forms, the resource form continuua helps guide consistent graphic design guidelines. For example, graphs have consistent design guidelines for colors, line widths, typography, and placement. Maps are usually designed to have labels which can be turned on and off (to avoid visual clutter). Such guidelines make the production of individual resources easier, and insure consistency from resource to resource. Thus the resource forms serve as a means of matching educational goals to appropriate representational form while facilitating particular graphic design and production decisions.
2.2.2. Resource Functions
Combined with the resource forms, resource functions facilitate matching educational goals to a logical level of multimedia functionality (or interactivity), while providing guidance for consistent navigation and graphic design. Resource functions include static, animated, sequential, hierarchical, and conditional resources.
Static resources include images, maps, diagrams, graphs, and tables that do not require interactive performance. While static resources may be appropriate given particular educational goals, we have tended to avoid them, focusing instead on providing resources which help explain concepts and ideas in ways not provided by slides or overheads.
Animated resources express change or motion when activated by the user. Interactivity is limited to starting and stopping the resource. Animated maps and diagrams are useful in representing phenomena over time. In many cases, however, we have converted material that could be presented as an animation into a sequential resource, as discussed below.
Sequential resources allow instructors to construct a linear sequence of elements one component at a time through a series of button clicks. The difference between an animation and a sequential resource is one of control over the materials being presented. An animation can be started, watched, then stopped. A sequential resource requires interactivity by the user in order to construct, step by step, an understanding of some final idea or concept. This difference has important effects on the material being presented and on the design process. In an attempt to take advantage of the interactive capabilities of multimedia, and to maximize the instructor's ability to effectively present a sequence of information that results in the understanding of a concept or idea, we have often imposed sequential functionality on material that could be presented as an animation.
Hierarchically nested resources are embedded with hidden information that can be revealed by selecting linked information in the initial display. Hierarchical resources move away from a linear construction of an idea or concept to a nonlinear means of exploring the depth of information embedded in a particular concept. Hierarchically nested resources are particularly important when the educational goal is to interrogate a concept which has depth and detail, rather than a concept constructed from a sequence of elements.
Conditional resources produce a graphic or numeric solution, according to the rules of an underlying algorithm, in response to the user's manipulation of an initial display. Conditional resources respond to the particular desires of the user and are not limited by either a sequential or hierarchical framework. Conditional resources provide a different kind of information than sequential or hierarchical resources, focusing on "what if" types of interactions rather than the construction of concepts (linearly with sequential and non-linearly with hierarchical resources).
The range of resource forms and resource functions can serve as a means by which general and particular educational goals can be matched to appropriate resource forms and functions while simultaneously providing consistency in the graphic design of the resources. The resource typology, then, is at the core of the planning, design, and production process for educational multimedia resources. In practice many resources combine a varied set of forms and functions. Nevertheless, the first (and most fundamental) step in the process of creating successful educational multimedia resources is the critical and logical connection of educational and content goals to appropriate resource forms and functions.
Given coherent educational and design goals, we have created a series of resources which have been implemented in the classroom. Design guidelines have been informed by evaluation methods since the onset of the project. Evaluation should be used not only as a means of assessing the impact of existing resources, but in shaping and informing the design process. The different functions of evaluation and their impact on the design of educational multimedia resources is detailed in the next section.
3. EVALUATING EDUCATIONAL MULTIMEDIA RESOURCES
Evaluation is useful for informing the design of educational multimedia resources but not for prescribing it. Given the new (and in many ways unexplored) technologies available for instruction, coherent, carefully designed, and innovative examples of educational technology need to be developed (Ebel 1982). A broader sense of evaluation must be adopted which can assist in shaping and informing the design of innovative educational multimedia resources. The products of this process may then be evaluated using more traditional impact evaluation methods. To this end, we have adopted a four part approach to evaluation as described by Reeves (1992) consisting of evaluation functions and methods (figure 3: Evaluation Functions and Methods).
Broadly defined, evaluation serves four (often interrelated) functions: goal refinement, documentation, formative evaluation, and impact evaluation. Each of these evaluation functions can be facilitated with a range of evaluation methods, including interviews, focus groups, questionnaires, observations, ratings assessment, expert review, and achievement tests (Reeves 1992).
3.1. Evaluation I: Goal Refinement Function
Reeves (1992) defines goal refinement as a "clear cut vision of what the [educational] goals ... should be" (p. 520). While these goals may change or evolve in practice, it is important to begin the process of conceptualizing, designing, and producing educational multimedia resources with specific objectives in place. We have attempted to assess goals as seen by course instructors, students, and administrators (who provide funds for developing such resources). These goals, in turn, informed the initial design of our lectureware. By paying attention to these differing (yet usually resolvable) goals from the onset we have been able to shape and inform both general and particular design goals. As an initial evaluatory step, goal formation is fundamental in shaping an overall design plan and for providing a set of explicit project goals.
3.2. Evaluation II: Documentation Function
Reeves (1992) defines documentation as a simply keeping a record of "what is actually done" throughout the process of creating educational resources (p. 521). From the beginning of the project we have compiled extensive documentation detailing what we thought we were doing, problems, ideas for changes, and reformulated goals. Information drawn from this documentation can be used to make future projects more efficient. A second important role for documentation is as a record of the implementation of the resources in the classroom. The project manager attended nearly every lecture taught with the resources, and the resulting documentation helped us to fix bugs and reshape our resources, thus serving a formative evaluation role (to be discussed below). A third important documentary source are the yearly status and future planning reports compiled by the project managers. These reports not only document what we accomplished, but forced the project managers to confront looming and unresolved issues. Careful and methodical documentation plays a fundamental evaluatory role in the planning, design, and production of educational multimedia resources. Such documentation is one of several means by which our resources were shaped and informed by evaluatory feedback.
3.3. Evaluation III: Formative Evaluation Function
Flagg (1990) defines formative evaluation as "the systematic collection of information for the purpose of informing decisions to design and improve the product" (p. 1-2). The idea driving formative evaluation is the realization that there is no "theoretical basis for a priori determination of the nature of instructional interventions, especially those involving new forms of technology" (Reeves 1992 p. 523). Rigid evaluatory experimentation is not able to produce viable innovations with new instructional technologies, nor can it expect to play an evaluatory role until viable educational resources are produced.
Documentation of the implementation of our resources in the classroom has proven particularly useful in re-forming many of our resources and design guidelines. Implementation documentation produced several different kinds of re-formative ideas. Particular problems with resources often surface during their use in the classroom, regardless of how carefully a resource has been tested. More substantial problems are also evident, such as when the instructor gets confused by the design of a resource. Besides problems with particular resources or the lectureware functionality itself, classroom implementation often suggested the need to create new resources when the instructor had to turn to the chalkboard or an overhead to explain a concept not contained in our set of resources.
Other important methods of formative evaluation involved seeking out instructors and other individuals with content and design expertise to evaluate resources. The result of the overlapping evaluatory processes of goal refinement, documentation, and formative evaluation is a substantive and informed critique of our original design goals. This critique was used to rethink, reshape, and reform our original ideas into the design ideas as presented in the first part of this paper. Once the kinks are worked out, the evaluation of the impact of the multimedia resources on students in the classroom can be considered.
3.4. Evaluation IV: Impact Evaluation Function
Impact evaluation considers a range of methods which seek to assess the impact of given educational resources on student learning. This type of evaluation is appropriate when coherent, carefully designed, and innovative examples of educational technology have been produced, shaped by goal refinement, documentation, and formative evaluation functions. Impact evaluation methods include the traditional notion of evaluation as measured assessment. Yet impact evaluation is more than this. As Reeves (1992) notes, there are a range of impact evaluation methods besides those which measure quantitative learning outcomes. Different impact evaluation methods pose different kinds of questions and provide different answers. Subsequently, more can be learned if a range of impact evaluation methods are brought to bear on educational multimedia resources.
Effective and useful impact evaluation methods are often difficult to design. If we cannot convincingly demonstrate the quality of traditionally taught lectures, how are we to assess lectures which adopt educational multimedia methods? Further, while evidence for the effectiveness of particular educational multimedia resources exists (Podell et. al 1993) such studies say little or nothing about the quality and effectiveness of any particular application. In an attempt to assess the impact of our particular application of educational resources we have employed two impact evaluation methods: focus groups and questionnaires. Both are qualitative forms of impact evaluation, but have provided a general sense of the impact of our multimedia resources on the students. Information from both of these methods of impact evaluation have led to modifications of particular resources as well as a very general evaluation of some of the fundamental goals of our multimedia resources. Generally positive feedback from students has provided encouragement and has suggested that our resources have been refined enough to begin formulating more quantitative impact evaluations.
Focus Group Impact Evaluation: Focus groups are seen as relevant as our lectureware is used with entire classes rather than individual students (Monmonier and Gluck 1994). The first semester the resources were implemented provided an opportunity to elicit feedback from a group of students in the course. The feedback from meetings with the students provided us with a overall sense of the impact of our resources on the students, what they liked and didn't like, and how they would like to see the resources modified. Students in the focus groups were very positive about the resources and the course, but were also able to articulate problems which impeded their goals in the course. The focus groups, then, served as a rough check on our original goals, provided us with ideas for refining the resources, and provided a forum in which students could refine and define their educational goals in the midst of fundamental changes in the classroom.
Questionnaire Impact Evaluation: The second impact evaluation method we employed was the use of questionnaires. Two sections of the Gaia course were taught by two different instructors the semester that our resources were first used. One section was taught without the lectureware and the other was taught with the lectureware. The first questionnaire was completed by students in the section of the course that was taught without the lectureware. These students saw the lectureware used in one review lecture and were queried about the advantages and disadvantages of the multimedia materials. A second questionnaire was distributed to students in the section of the course taught with the lectureware. Again, students were queried about the use of the lectureware and its positive and negative attributes. The responses to the latter questionnaire were similar, although more favorable, to the responses to the former questionnaire. In addition, comments culled from our focus groups closely correlated from comments elicited from the questionnaires.
Overall we are encouraged by the initial responses to our resources. The majority of students who saw the multimedia resources for one day were in favor of it being used for the entire class or at least in conjunction with other teaching methods. A number of students who were exposed to the use of our resources for the entire semester mentioned that they could not imagine learning the material without the resources. A high percentage of the respondents mentioned that the courseware was interesting and that it helped them to visualize and understand difficult concepts. They found the step by step build up of ideas within sequential resources and the explicit relations between different resources useful in understanding concepts and their interrelations. Students particularly liked resources with 3-D graphics, movement, and conditional interactivity.
Negative comments were split between technical difficulties and concerns about the atmosphere for learning. The most frequently mentioned problem with the use of lectureware was that technical difficulties (bugs, crashed computers, fumbling with the computer) disrupted class and wasted students' time. Lighting problems were the second largest concern among students who were in the multimedia section of the course. The more serious critiques cannot be as easily fixed as technical bugs and lighting. Some students had difficulty gleaning the key points from complex resources. Others were bothered by the difficulty of depicting the gist of complex resources in their notes. And finally, numerous students in the section of the class that saw the multimedia resources for only one day expressed concern that the instructor would be distanced from the class by focusing attention on the computer and not the students. This problem, however, was expressed by only two of the students in the section of the course which used the resources during the entire semester.
In sum, the use of focus groups and questionnaires for impact evaluation has helped shape modifications of particular resources as well as provide us with a rough evaluation of the fundamental goals of our multimedia resources. Qualitative methods of impact evaluation, along with goal refinement, documentation, and formative evaluation have helped inform original educational and design goals. Further, the responses from students have provided us with some key issues which more quantitative methods of impact evaluation may address. For example, students strongly prefer multimedia resources with some kind of movement. Is this preference based on a desire to be entertained, or is it based on the understanding that movement (particularly on graphs, diagrams, and maps) is something not offered by traditional educational resources and enhances the understanding of certain concepts and ideas? Students also claim enhanced understanding, particularly from some of the more complex resources, in comparison to static depictions of the same materials in their course reader. Impact evaluations which attempt to assess such issues are currently being developed.
CONCLUSIONS
This paper presents an integrated approach to the design, implementation, and evaluation of multimedia educational resources. The first section of the paper describes an approach to educational multimedia design based on consistent and coherent graphic design principles and the matching or educational and content goals to particular multimedia forms and functions. Such a design strategy should guide the production and implementation of educational multimedia resources from the onset of the project. The second section of the paper describes an approach to educational multimedia evaluation based on a broad range of evaluation functions and methods implemented from the onset of the project. Evaluation, broadly conceptualized as goal refinement, documentation, formative evaluation, and impact evaluation, should play an informative role throughout the design, production, and implementation of multimedia resources rather than only at the end of the process. Finally, design and evaluation strategies should be bound together by an iterative design approach where the goals and expertise of content experts and educators shape and are shaped by the goals and expertise of multimedia designers. It is hoped that this approach to the design, implementation, and evaluation of educational multimedia resources may guide other similar projects, posing and addressing important questions about the prospects and impact of multimedia on earth science and geographic education.
REFERENCES
Andrews, S. and D. Tilton. 1993. "How Multimedia and Hypermedia are Changing the Look of Maps." Proceedings of AUTOCARTO 11, Minneapolis MN. pp. 348-366.
DiBiase, D. and J. Krygier. 1994. "Multimedia at Penn State." Cartographic Perspectives 19. pp. 40-42.
Ebel, R. 1982. "The Future of Educational Research." Educational Researcher 11:8. pp. 18-19.
Flagg, B. 1990. Formative Evaluation for Educational Technologies. Hillsdale NJ: Lawrence Erlbaum.
Monmonier, M. and M. Gluck. 1994. "Focus Groups for Design Improvement in Dynamic Cartography." Cartography and Geographical Information Systems 21:1. pp. 37-47.
Morrow, C. and J. Boettcher. 1995. "Technology Classrooms: Design and Implementation, a Penn State Perspective." Syllabus 8:6. pp. 18-20.
Podell, D., S. Kaminsky, and V. Cusimano. 1993. "The Effects of a Microcomputer Laboratory Approach to Physical Science Instruction on Student Motivation." Computers in the Schools 9:2/3. pp. 65-73.
Reeves, T. 1992. "Evaluating Schools Infused with Technology." Education and Urban Society 24:4. pp. 519-534.
...back to "Design, Implementation, and Evaluation..."