Finding the right tools

At first, clinical psychology student Robert Lippy thought he had found the perfect measurement tool for his dissertation on the impact that training and exposure to other cultures have on the cultural competency of the military's primary-care physicians. The instrument he had identified seemed ideal because-unlike many of the other instruments he had reviewed--it was designed specif-ically with primary-care physicians in mind. His advisers didn't agree, noting that information about the instrument's reliability and validity was lacking.

"They said, 'It looks like a good instrument, but we don't really know anything about it scientifically,'" says Lippy, a fourth-year student at the Uniformed Services University of the Health Sciences, a lieutenant in the U.S. Navy Reserve and an intern at the National Naval Medical Center in Bethesda. "I had to go back to the drawing board." Lippy once again reviewed the psychological literature on cultural competence and found an empirically based instrument that satisfied his advisers.

As Lippy learned, it's not enough to find a measurement tool that appears to be right for your research at first glance. Students need to know how to choose an appropriate instrument, assess its quality and use it ethically, say psychologists and other measurement experts.

CHOOSING AN INSTRUMENT

When it comes to choosing and using instruments, students should take a systematic approach, says Jeffery P. Braden, PhD, chair of APA's Committee on Psycho-logical Tests and Assessment and director of the school psychology program at North Carolina State University. He and others have several suggestions:

• Know what you need. A good first step is to check the literature to see what instruments researchers in your area have used, suggests Braden. But keep in mind that while there are hundreds of instruments out there, there may be only one that suits your specific needs. A student who's interested in measuring math anxiety may find an instrument with that phrase in the title and assume it's appropriate for their purposes, says Barbara S. Plake, PhD, former director of the Buros Center for Testing and an emerita professor at the University of Nebraska. "But it may be measuring math anxiety in a different population or via a different mechanism than you're interested in," she warns.

And you might find an instrument that asks the very questions you need answered, but still have to reject it on practical grounds, adds Braden. The instrument might take too long to administer, for example. Or it might cost more than you can spend.

• Befriend your librarian. Once you've established what you're looking for, head to the library. To find commercially available tests, Plake recommends starting your search at the Buros Institute's Test Reviews Online at www.unl.edu/buros. The database contains information on almost 4,000 commercially available psychological, educational and business tests dating back to the mid-1970s. Search for keywords in the "scores" category, suggests Plake, noting that this approach will uncover tests that measure things that may not be in the test's title. Students can either purchase test reviews for $15 each or consult the particular volume of Buros's "Mental Measurements Yearbook" that contains the desired review. Buros's "Tests in Print" also allows researchers to search tests according to what they measure, says Plake. The book contains information about every commercially available English-language test currently in print.

When it comes to finding noncommercial tests, says APA Publisher Gary VandenBos, PhD, one major resource is the eight-volume "Directory of Unpublished Experimental Mental Measures" published by APA. The directory describes experimental measures from studies published in 36 top journals.

APA's PsycINFO database is another resource. In addition to searching by instrument name, you can also search by area and then check the "tests and measures" information in abstracts, says PsycINFO Senior Director Linda Beebe.

Yet another resource is the Health and Psychosocial Instruments (HAPI) database created by Behavioral Measurement Database Services and available at most university libraries through the bibliographic database OVID. (See www.ovid.com/site/products/ovidguide/hapidb.htm.) HAPI describes measures appearing in journals dating back to 1985.

"We literally go through every journal and identify the instruments," explains Behavioral Measurement Database Services Director Evelyn Perloff, PhD, an emerita professor of nursing and psychology at the University of Pittsburgh.

In addition to its database, the company has hundreds of instruments on file that it can mail to researchers; the handling fee is $20 per instrument. The company also provides free consultation to researchers.

• Do some quality control. "Many people believe that if an instrument's been published commercially, it must be a fine product," says Plake, who was co-editor of "Tests in Print" and the "Mental Measurements Yearbook" until her recent retirement. "But even with major publishers, you have to make sure that the test has adequate psychometric properties."

That means checking that the instrument is both reliable and valid, meaning it not only produces the same result each time you administer it but also measures what it's supposed to measure. Publishers will generally provide a "specimen set" consisting of a copy of the test, a technical manual and other information, says Plake.

Also consult reviews of the instrument, recommends Braden, citing the "Mental Measurements Yearbook" and PRO-ED Inc.'s "Test Critiques" as useful resources. Or type in the name of the instrument on PsycINFO and choose "review" as the type of document requested. Use the Social Sciences Citation Index to search for articles citing the particular instrument so that you can see who has used the tool and for what purpose.

• Get permission. Once you've found the perfect instrument, you need to get consent before you can start using it. Although some instruments are in the public domain, most aren't. In these cases, you'll have to buy an instrument from a publisher if it's a commercially available product or get permission from an author if it's not. "Most researchers are eager to have students use their instruments, because it means someone's paying attention," says Braden. Contacting an author also gives you a chance to discuss the instrument's strengths and weaknesses, he adds.

• Pay up. If you'll be buying a commercial product, keep in mind that costs can range from a few cents for each copy of a questionnaire to hundreds of dollars for individually administered intelligence tests. You may have to buy a manual or send data to the publisher for scoring, which can boost costs.

But students can often wrangle a discount, says Braden. "What a lot of students don't know is that there's usually a substantial discount if the instrument is going to be used for research and that research is unfunded," he says. "Most of the major test publishers will take off 40 to 50 percent."

BUILDING YOUR OWN

Can't find what you need? In more than a few circumstances, says Braden, students will decide there really isn't a good measure already out there.

Don't despair, he says. Instead, think about constructing your own instrument. SurveyMonkey.com and similar Web sites allow researchers to create online surveys, he notes. Or work with your adviser to create a more traditional instrument.

One of Braden's former students developed an instrument to identify students having trouble learning as his master's thesis, then used it for his doctoral research. The instrument-the Academic Competence Evaluation Scales--is now available commercially from Harcourt Assessment Inc. "He and his adviser are making money on it," laughs Braden.

For more information, see APA's "Frequently Asked Questions: Finding Information About Psychological Tests."

Rebecca A. Clay is a writer in Washington, D.C.