AECT
Getting Started
in
Instructional Technology Research
Steven M. Ross
     University of Memphis
Gary R. Morrison
      Old Dominion University

AECT Home Membership Information Publications Events       Jobs    

Table of Contents
1: Getting Started as a Researcher: Designing and Conducting Research Studies in Instructional Technology

1. Selecting a Topic

2. Identifying the Research Problem
3. Conducting a Literature Search
4. Stating the Research Questions
5. Determining the Research Design
6. Determining Methods
7. Identifying Data Analysis Procedures
2: Writing Proposals for Acceptance at AECT
3: Practicing What We Preach: Making Effective Presentations at AECT
4: How to Get Research Articles Published in Professional Journals

References
Authors

How do you get started in planning a research study?

The main problem for many beginning researchers is putting the cart before the horse, which in research parlance translates into selecting methods, a design, or worse, a statistical analysis procedure (as in "Won't I need to use analysis of variance to get my dissertation approved?") before clearly deciding what questions they want to investigate. The usual result of such "backward planning" is a collection of methods that lack a unifying purpose of direction. To help you avoid this error, the following pages will present a seven-part model that will guide you through a logical sequence of planning steps.

1. Selecting a Topic

What general topic area interests you? Is it adaptive instruction, learner control, the benefits of animation, how computers are used in the classroom, the design of distance education, or some other focus? Think of a research study as something that you will live with for awhile, perhaps a year and probably more (not too much more, we hope, if you are doing a thesis or dissertation!). A "shot-gun" marriage with a topic area that doesn't interest you, but your advisor imposes or colleague suggests is almost certain to be an unpleasant experience. Research can be fun and involving when you have a natural curiosity about the subject; it can be drudgery when you don't.

2. Identifying the Research Problem

The next step is to determine what problem or concern to investigate within your topic area. Sometimes the problem will be immediately identifiable based on your present knowledge and interests. In other instances, it may be only a general idea that needs to be more carefully considered and sharpened. For example, a few years ago, one of us became interested in children's solving of math word problems. The particular idea was making word problems more interesting to students to increase motivation and learning. The basis for a research problem was thus established, but some library time was needed to sharpen the premises and specific focus. Through a review of the literature, the problem evolved into one concerning whether meaningful learning could be enhanced by using a microcomputer to personalize problems for each student. In selecting your topic, a key concern is the importance of the problem to the field.

Conducting research requires too much time and effort to examine trivial questions that do not expand existing knowledge. Experienced researchers are usually attuned to important topics, based on their knowledge of the literature and current research activities. Novices, however, need to use care when establishing support for their idea from recent research and issues-oriented publications (see Step 3). For experts and novices alike, it is always a good practice to use other researchers as a sounding board for a research focus, before getting too far into the study design (Steps 4 and 5). Regardless of your focus, a thorough literature search is essential here and/or at various other places in the planning process. We turn to this activity next.

3. Conducting a Literature Search

Although there are several approaches to doing a literature review, a few key ideas and techniques can help you improve your efficiency. A starting point for a search is electronic databases such as Psychological Abstracts, or ERIC that includes Current Index to Journals in Education (CIJE) and Research in Education (RIE). You might also have access to a database such as FirstSearch that combines several databases related to our field. To gain the most efficiency in using these indexes, you must start with two or three key terms. You can start with some key terms and see if the search is productive. If not, try another set of terms. Some databases will suggest alternative key words. For example, while searching a database for abstracts recently, we searched the term "instructional technology" did not find any results. The results, however, suggested we try "educational technology" which produced a series of abstracts. As a last resort, you can look through a thesaurus for specific database to find other key words. Articles listed in these indexes are classified according to established lists of terms. Thus, a few minutes spent locating key terms can make your search time more efficient. A second approach is to find a citation in one of the indexes for a key article you have read and note the terms used to classify the article. These terms are your starting point for the literature review and may need to be revised as you make progress.

The Internet and electronic databases have made it much easier to locate a host of articles related to our research topic. The key is to find the correct search terms. You can start by using terms your own key words or those from a thesaurus or "known" article. As you find relevant articles, you can scan the abstract for other terms to help you either broaden or narrow your search for relevant articles.

Applications such as EndNote provide an excellent tool for both searching and compiling bibliographies for your research. You can use these applications to search different indexes, or you can collect the data and import it into the application. Once you have complied a set of references, you can use the application to help write your paper and create the reference list.

The next step is to retrieve the articles your search has identified. The easiest articles to retrieve are those with Internet listings. Simply visit the site with your browser and save the document (we would caution you about using a large number of these non-refereed papers in a scholarly paper). Make sure you include the URL for citing the document (you can find current information on citing Internet resources at www.apa.org/journals/webref.html). If you have access to a database that includes full text articles, you can download it, have it emailed to you, or print the article or paper. A trip to the library, however, requires some planning. First, prepare a note card (or a printed page) for each reference. Second, sort the cards chronologically and then alphabetically by journal. Third, gather your supplies, which should include extra note cards, a pen and pencil, paper clips or Post-It® flags or page markers, and money for the copy machines. Fourth, read through you index cards and classify the articles on their relevance to your study (e.g., "critical, "probable, or "long shot").

When you arrive at the library, ask a librarian where information is kept concerning the location of journals. For example, current journals are probably located in one place, bound journals in another location, and microfilmed journals in another room. Organize your research according to the importance of the articles ("critical" ones first). You might want to make a photocopy of important articles and only make notes on others.

Your initial search will probably identify one or more key articles. One way to expand the search is to read the references cited in those articles. This approach, however, only provides you with older articles. Another option is to search through the Social Sciences Citation Index to see if your key articles have been cited by others. This method provides more recent articles on your topic. To broaden your search, a third strategy is to examine indexes of journals in related fields and of related international journals. A final strategy to locating some of the newest papers is search through convention programs such as AECT and AERA for recently presented papers that have not yet been published.

Problem Articles. There may be times when you cannot locate a particular journal or ERIC paper. There are seven strategies for obtaining these elusive papers. First, check with your librarian to see if another library in the area has the journal. Many libraries have such a list and exchange agreements with other libraries. Second, order a copy through inter-library loan. This process may take a few days and possibly cost a few dollars, but it is quite reasonable. Third, write to the author and request a copy. Current addresses are available through association directories such as AECT, AERA, and APA. Most authors are pleased to know that someone is reading their research and may even include additional articles. You can increase the chance of receiving the article if you include a self-addressed-stamped-envelop that makes it simple for the author to place a copy of the paper in your envelop and put in the mail. Fourth, try to order the article through companies such as Carl UnCover or ISI Document Solution. Fifth, ask your librarian to help you locate the journal publisher and then call and request to purchase a single back issue. Sixth, if you are searching for a paper in ERIC, check Psychological Abstracts and CIJE for a three-year period after the paper was presented to see if it was published in a journal. Seventh, ask a librarian to help you determine if the article is available electronically.

4. Stating the Research Questions

We have now reached what is probably the most critical step in the planning process. Once defined, research questions provide the basis for planning all other parts of the study--design, materials, and data analysis. Unfortunately, it is sometimes difficult to convince beginning researchers of this sequence. As stated earlier, a common tendency is to focus much too early on which treatments to compare or what design (e.g., experimental or correlational) to use. These decisions are obviously important, but they cannot be reasonably addressed until you know exactly what you want to find out. Research questions convey those goals.

At this point you have identified a problem focus (Step 2) and are familiar with the literature (Step 3), Based on this background, what specific questions do you want your study to answer? As you think of questions, write them down in a list. Don't worry about their exact phrasing, whether others will like them, or having too many. Before you actually begin the study, there will be many opportunities (perhaps a lot more than you want!) to present your ideas to others and obtain reactions. Chances are that you will also revisit the library to read more about a specific theory or prior study. Thus, this initial list will probably not be the final one, but will serve as a useful starting point for planning how the study will be performed. As an example, Table 1 lists the questions used in the personalization study referred to earlier. In the following pages, we will return to this table to illustrate how research questions directly guide subsequent planning steps.

Note, however, that if you are conducting a qualitative study, you will want to induce (derive) hypotheses from your observations, not deduce the answers to predefined hypotheses . Thus, qualitative research uses an open, flexible, and inductive orientation.

Table 1: An Example of Research Questions and Procedural Planning for an Actual Study

Questions

Design

Instrumentation

Statistical Analysis

1. Will personalizing contexts improve problem solving relative to using concrete or abstract contexts?

Experimental: Three Groups

a. Achievement tests

One-way Analysis of Variance

2. Will personalizing contexts improve attitudes toward the lesson?

Experimental: Three Groups

b. Attitude survey

One-way Analysis of Variance

3. Are attitudes and problem-solving performance related?

Correlational

a and b

Pearson Correlation

4. What are the logistics of personalizing problems?

Descriptive

c. Experimenter journal

Narrative report (Qualitative)

5. What are teacher's reactions about the practicality and value of the personalized strategy?

Descriptive

Interview

Narrative report (Qualitative)

5. Determining the Research Design

Your research design will be primarily determined by two factors: (a) what the research questions require (of course!), and (b) what is feasible given the resources or conditions at hand. Space restrictions limit how much can be said here about different design approaches, but, as a brief introduction, we'll note three general categories. Experimental-type designs are used to test hypotheses regarding causation, for example, that a particular instructional strategy leads to better student performance. Making causal inferences requires a high degree of experimental control, such that all conditions for the "treatment" group and the control group are identical except for the particular strategy being tested. A true-experimental design, in which subjects are randomly assigned to treatments, is best able to achieve this control, but is oftentimes impractical. A quasi-experimental design, which uses intact groups (e.g., two existing third grade classes), is less rigorous but generally easier to implement. Two additional design categories are the correlational and descriptive approaches. Correlational studies examine how variables relate to one another (e.g., amount of computer experience and attitudes toward word-processing) rather than whether one causes the other. Descriptive studies depict conditions as they exist in a particular setting (e.g., the number of teachers at different grade levels who use computer-based instruction). With descriptive studies, you may use qualitative data sources (field notes from observations, research journals, interviews), quantitative sources (frequency counts, percentages, descriptive statistics), or both.

Equipped with a basic knowledge of design approaches, the next task is to determine what is needed for each research question. Your study may therefore include several approaches, for example, an experimental part and a descriptive part. To illustrate, the second column of Table 1 lists the designs used to address the research questions in the personalization study. Note that where treatment groups were compared on some dependent (outcome) measure, an experimental design was specified. Question 3, concerning a relationship between variables, naturally required correlation. Questions 4 and 5 were descriptive in nature, concerning the researcher's experiences in implementing the personalization strategy and the teachers' reactions to its instructional use. Keep in mind that as a beginning researcher you do not have to be a research design expert; colleagues (or your advisor and committee) can provide assistance in this area.

6. Determining Methods

Methods of the study include (a) participants (subjects), (b) materials and instrumentation, and (c) procedures. Start with participants. What kind and how many participants does your research design require? For the present example (Table 1), it was decided that to manipulate the three treatments (Questions 1 and 2), we would need about 60 elementary school students. Two or three teachers would also be needed for the strategy evaluation (Question 5).

Remember, the challenging part of data analysis is selecting appropriate methods and using them in the right ways. Qualitative analysis requires careful and systematic coding and reliability checking; be sure to reference a qualitative research text if you are unfamiliar with these methods. The same recommendation applies to quantitative analysis; Table 2 is provided here to give you a head start.

Next, consider materials and instrumentation. Sometimes the specific resources needed will be obvious. When uncertainties exist, construct a listing on a question-by-question basis. This procedure is illustrated in Table 1 (column 3) for the identification of assessment instruments for the personalization study. Basic to all five-research questions was obviously the need for a computer-based mathematics lesson that would convey the personalization and control strategies. A specific lesson idea and target grade level evolved through discussion with teachers and examination of curriculum materials. Because the lesson would be unique to our study, we were faced with designing our own achievement test (a common situation in educational research) and attitude survey to inquire about the particular materials used. As you plan your own research, a good rule to follow is to try to obtain existing validated instruments where you can; when existing measures provide a poor fit with your research questions, you will be better off developing your own using the guidelines described in most educational measurement texts.

Table 2. Common statistical Analysis Procedures Used in Educational Technology Research

Analysis

Types of Data

Features

Example

Test of Causal effects?

t-test
(independent samples)

Independent variable = nominal; Dependent = one interval-ratio measure

Testing the differences between 2 treatment groups

Does the problem-based treatment group surpass the traditional instruction treatment group?

Yes

t-test

(dependent samples)

Independent variable = nominal (repeated measure); Dependent = one interval-ratio measure

Testing the difference between 2 treatment means for a given group.

Will participants change their attitudes toward drugs, from pretest to posttest, following a videotape on drug effects?

Yes

Analysis of variance

(ANOVA)

Independent variable = nominal; Dependent = one interval-ratio measure

Testing the differences between 2 or more treatment means. If ANOVA is significant, follow-up comparisons of means are performed.

Will there be differences in learning between three groups that paraphrase, summarize, or neither?

Yes

Multivariate analysis of variance (MANOVA)

Independent variable = nominal; Dependent = two or more interval-ratio measures

Testing the difference between 2 or more treatment group means on 2 or more learning measures. Controls Type I error rate across the measure. If MANOVA is significant, an ANOVA on each individual measure is performed.

Will there be differences between 3 feedback strategies on problem-solving and knowledge learning?

Yes

Analysis of covariance (ANCOVA) or multivariate analysis of covariance (MANCOVA)

Independent variable = nominal; Dependent = one or more interval-ratio measures; Covariate = one or more interval-ratio measures

Replicates ANOVA or MANOVA but employs an additional variable to control for treatment group differences in aptitude and/or to reduce error variance in the dependent variable(s).

Will there be differences in concept learning between learner-control, program-control, and advisement strategies, with differences in prior knowledge controlled?

Yes

Pearson r

Two interval-ratio measures

Tests relationship between two variables.

Is anxiety related to test performance?

No

Multiple linear regression

Independent variable = two or more interval-ratio measures; Dependent = one interval-ratio measure

Tests relationship between set of predictors (independent) variables and outcome variable. Shows the relative contribution of each predictor in accounting for variability in the outcome variable.

How well do experience, age, gender, and grade-point average predict time spent on completing a task?

No

The last step in planning the methodology is to develop a procedural plan. What logical sequence of steps will provide the information needed to answer the research questions? What materials will be used or information given to subjects at each step? You may find it helpful to begin by constructing a general outline. Once you become more confident about the basic procedures, the important task will be to describe them in narrative form (as required in a prospectus or proposal). Our best advice: Write the narrative so that another person could conduct the study for you just by reading it (this does not imply, however, that you'll ever find such a person). The more precise you are, the better basis you'll have for conducting the study and your committee will have for evaluating it.

7. Identifying Data Analysis Procedures

Some beginning researchers can't wait to identify the statistics they will use, doing this step (unfortunately) as the first rather than last step in the planning process. Others would be most happy if the statistical component somehow magically disappeared. For most types of research in instructional technology, some statistical analysis will be needed. The good news is that if you have gone through the preceding six steps, specifying an analysis plan is very straightforward and easy. If statistics aren't your strength, no need to worry. Show someone who knows statistics (usually there's such a person on your committee or ask a colleague) a clear set of research questions and design descriptions, and he/she will be able to readily identify what analyses are needed. Your task is to learn from a practitioner's standpoint why those analyses are appropriate, what types of information they provide, and how to interpret those results. Interested readers can examine in Table 1 (last column) a listing of the analyses selected for the personalization study. Note that each is matched to a specific research question (getting the picture?); also note that mixture of quantitative and qualitative analysis procedures was used.

If you are conducting a qualitative study, a common misconception is that data analysis is easier or less complicated than is the case for quantitative research. In fact, many would argue that the opposite is true! Qualitative analysis requires careful organization and synthesis of notes and transcripts to derive patterns and ideas to form the basis for conclusions. Readers of qualitative studies need to be convinced that the data analysis was done thoroughly, systematically, and reliably. Numerous textbooks and other reference materials, along with computer software, are available to help the qualitative researcher achieve these standards.

In summary, we again caution you about one of the most common mistakes made in planning research. It is thinking about treatments first (as in, "I think it would be good to cross a paraphrasing strategy and a mathemagenic strategy with using color and monochrome monitors. That would even get us a 2 x 2 multifactor design!") and meaningful research questions second. The result is often a collage of uninteresting, disconnected findings that fail to advance either theory or practice. Using a systematic planning approach, such as the present seven-step model, will help you to identify and investigate problems that interest you and are currently of interest in the field. More importantly, a systematic approach is likely to result in a higher quality study and a much more positive experience for you as a new researcher. And, if you begin now, you may even complete your study in time for our fourth article in the series, on how to write research reports for publication in professional journals such as ETR&D.


Updated April 3, 2007
Copyright © 2001, The Association for Educational Communications and Technology

AECT
1800 North Stonelake Drive, Suite 2
Bloomington, IN 47404

877.677.AECT (toll-free)
812.335.7675