It is time to learn about the effects of your environmental education (EE) program! What do the patterns, differences, and relationships in the data suggest about how well the program is achieving its objectives?
What type of analysis do I need?
There are many different types of analyses that vary in complexity. To help you determine which type of analysis to choose, consider the following:
- Whether you collected quantitative or qualitative data,
- The resources (expertise, time, funding) you have available for analysis,
- The evaluation questions you want to answer (i.e., what analysis will sufficiently answer them), and
- How rigorous and credible the analysis needs to be to satisfy stakeholders.
Adapted from Fitzpatrick, Sanders, & Worthen, 2004
The paragraphs below discuss types of analyses according to whether you collected quantitative or qualitative data, and point you to software that can be used to analyze these data. Based on the information and resources in this section you should gain insight into what type of analysis to conduct for your evaluation, and how to conduct at least some of these analyses yourself. In addition, you should have an improved understanding of what questions to ask of an evaluation expert to help you conduct your own or clarify her/his analysis.
Get expert help with analysis! BEGINNER
How do I analyze quantitative data?
Before you can analyze quantitative data, you will need to code data and enter data into a spreadsheet or a statistical analysis program (see Step 5).
Descriptive Statistics
You will want to begin with descriptive analyses of your data. These analyses are called descriptive because they allow you to summarize large amounts of information. Descriptive statistics include frequencies (counts), percents, ranks, measures of central tendency (e.g., mean, median, and mode), and measures of variability (e.g., range and standard deviation). In many cases, descriptive statistics will be sufficient to answer most stakeholders’ questions (Fitzpatrick et al, 2004). For example, you can calculate the mean response to a question or the percent of respondents who answered a question in a particular way. If you asked respondents to indicate how likely it was they will recycle batteries as a result of your program (on a scale from 1=extremely unlikely to 7=extremely likely), you may be able to report that the mean response was 6.5 or that 85% indicated that they will (by adding those that selected 5 or above).
Statistical significance is not the same as substantive significance!
When a difference is statistically significant, it means that the difference is probably not due to chance. This does not tell you if the difference is meaningful or trivial! Effect sizes provide you with a quantitative way to assess to what extent a significant difference may also be substantively important. To learn more about effect sizes, review the section on power analysis, statistical significance, and effect size.
Inferential Statistics
After conducting descriptive analyses, you may want to conduct more complex inferential analyses. These analyses include testing for significant differences. For example, you can test whether participants in your program scored higher on a multiple choice exam than individuals in a control group. You can even conduct these analyses in such a way that you can “control” for influences other than your program such as how participants performed before your program or based on their gender. For example, were female participants more likely to correctly answer the questions than male participants?
The resources below can help you learn more about how to analyze quantitative data:
- Analyzing Quantitative Data (.pdf)
Taylor-Powell, E. (1996). University of Wisconsin Extension Program, Development and Evaluation Unit
Beginner
This guide presents an introduction to descriptive statistics and explains how to use them in evaluation. - Inferential Statistics
Trochim, W. K. (2006).
Beginner
This page describes how inferential statistics can be used in outcome evaluations. It includes links to descriptions of many types of procedures used in inferential statistics; including t-tests, ANOVA, Analysis of Covariance (ANCOVA), regression analysis, cluster analysis, and regression. - Guide to Program Evaluation for Aquatic Educators (.pdf)
Recreational Boating and Fishing Foundation. (2006).
Beginner Intermediate
The section in Chapter 5, entitled “Analyze the Evaluation Data,” provides an introduction to analyzing quantitative EE evaluation data. It presents the steps involved in analysis, summarizes the purpose of statistical analysis, and explains how to reach conclusions from the analysis. A table gives an overview of the different types of statistical analyses and their uses in evaluations of EE and aquatic education programs. - Hyper Stat Online Statistics Textbook
Beginner Intermediate
This online statistics textbook, available for free, offers a detailed discussion of how to use both descriptive and inferential statistics.
Be sure to analyze data at the appropriate level! INTERMEDIATE ADVANCED
What software can I use to analyze quantitative data?
Software programs are invaluable in carrying out most quantitative statistical procedures. In fact, many of the procedures would be impossible for most individuals to carry out without such software.
Fortunately there are many software programs to choose from including inexpensive or free ones:
- Using Excel for Analyzing Survey Questionnaires (.pdf)
Beginner
This guide provides step-by-step instructions on using Microsoft Excel to:- Create a database of responses
- Code data
- Display frequencies and percentages
- Calculate an average (mean), range, standard deviation, etc.
- Create cross-tabulations and pivot tables
- Statpages.org
Beginner Intermediate Advanced
This online clearinghouse offers links to hundreds of free resources for statistical analysis, including downloadable software packages, online statistics books, tutorials, and hundreds of pages that perform statistical calculations. This is a good place to go once you have a basic understanding of quantitative analysis and an idea of what you need for your own analysis. - Open Stat
Intermediate Advanced
Open Stat is a free software package for performing statistical analyses. It is relatively simple to use and you can paste data into it from Microsoft Excel. There are some unique aspects to how it works, but it supports a variety of simple to complex analysis procedures, including frequencies, t-test, ANOVA, multiple regression, factor analysis, etc. - SPSS
Intermediate Advanced
SPSS is one of the most popular software packages for statistical analysis and data management, though it is not free. It includes tools to help you organize, prepare, and understand your data, perform statistical analyses including basic descriptive statistics, regression, and many advanced procedures. It also helps you prepare results for presentation and reporting. - AM by AIR
Advanced
This software package was designed for "analyzing data from complex samples, especially large-scale assessments." It has many uses, from simple statistics such as frequencies, to more complex procedures such as regression and complex weighting schemes. Users can import data from a variety of sources (e.g., Excel, SPSS, etc.) and output can be displayed in a text file or a web browser.
How do I analyze qualitative data?
Though sample sizes in qualitative data tend to be smaller, the data sets themselves are typically large, complicated, and often messy to organize and analyze. In qualitative analysis, there are also fewer rules and standard procedures to guide the process than in quantitative analysis. However, good qualitative analysis, like good quantitative analysis, is highly systematic and disciplined.
Most of the qualitative analysis you are likely to conduct for an evaluation is what is called content analysis. Content analysis involves systematically analyzing the content of data, breaking it into meaningful pieces, and organizing those pieces in a way that allows their characteristics and meaning to be better understood.
Miles and Huberman (1994) propose breaking qualitative analysis into the following steps:
Data reduction
This step involves selecting, focusing, condensing, and transforming data. The process should be guided by thinking about which data best answer the evaluation questions.
Data display
This involves creating an organized, compressed way of arranging data (such as through a diagram, chart, matrix, or text). The display should help facilitate identifying themes, patterns, and connections that help answer your evaluation questions. This step usually involves coding, where you mark passages of text (or parts of images or sections of a video, etc.) that have the same message or are connected in some way, and you write an accompanying explanation of what the selected passages have in common.
Conclusion drawing and verification
During this last step, revisit the data many times to verify, test, or confirm the themes and patterns you have identified.
While the three steps provide a good overall guideline, you may need to cycle through the steps repeatedly. Qualitative analysis is a cyclical and iterative process, with many rounds of investigating evidence, modifying hypotheses, and revisiting the data from a new light. You will need to reexamine data repeatedly as new questions and connections emerge and as you gain a more thorough understanding of the information you collected. Throughout the process of examining and reexamining data, concentrate on the following:
- Patterns, recurring themes, similarities, and differences
- Ways in which these patterns (or lack thereof) help answer evaluation questions
- Any deviations from these patterns and possible explanations for these
- Interesting or particularly insightful stories
- Specific language people use to describe phenomena
- To what extent patterns are supported by past studies or other evaluations (and if not, what might explain the differences)
- To what extent patterns suggest that additional data may need to be collected
The following resources will help guide you through the process of qualitative data analysis:
- Analyzing Qualitative Data (.pdf)
Taylor-Powell, E. and Renner, M. (2003). University of Wisconsin Extension Program, Development and Evaluation Unit
Beginner
This guide walks you through the steps of content analysis. The steps include focusing the analysis, categorizing data, identifying patterns and connections, and interpretation. The guide offers many examples, useful tips, and pitfalls to avoid. - User-Friendly Handbook for Mixed Method Evaluations
Frechtling, J. and L. Sharp, (1997). National Science Foundation.
Beginner Intermediate
Chapter 4, “Analyzing Qualitative Data,” provides a detailed description of qualitative analysis based on the Miles and Huberman (1994) three-step process (1. data reduction, 2. data display, and 3. conclusion drawing and verification). The chapter also provides advice on how to judge the quality of qualitative analysis, and includes a list of tips. - Online QDA website (Quantitative Data Analysis)
Beginner Intermediate Advanced
This website provides a wealth of information to support qualitative data analysis, including introductory background and extensive information on different software packages.
Many minds working together make light work.
What software can I use to analyze qualitative data?
Using a computer program can enrich and improve the process of qualitative analysis by speeding up some analysis tasks (e.g. marking, coding, and moving data segments) and facilitating complex questioning of the data (Barry, 1998). Software cannot, however, help you pick out the meaningful codes, themes, or categories, or eliminate the challenging intellectual work of qualitative analysis (Frechtling and Sharp, 1997).
There are many software programs available for purchase and there are some that can be downloaded for free. Not surprisingly, free programs tend not to be as powerful or perform as many functions. They also are not updated as frequently. Nevertheless, such programs may be worth looking into if their capabilities meet your needs.
Do not use software just because it exists!
Two popular free programs are:
- AnSWR
AnSWR is described as “a software system for coordinating and conducting large-scale, team-based analysis projects that integrate qualitative and quantitative techniques.” The website provides more details, an overview of what the program can do, manuals, installation instructions, and more. - CDC EZ-Text
According to the website, CDC EZ-Text “is a software program developed to help researchers create, manage, and analyze semi-structured qualitative databases.” The website provides further description of the program, a frequently asked questions section (including information on how this program differs from AnSWR), and installation instructions.
The following resources will help you sift through many of the qualitative analysis software programs available to find one that will best meet your needs:
- CAQDAS – A Primer
Thomas Konig. Loughborough University
Beginner Intermediate
This article focuses on what it calls Computer-Assisted Qualitative Data Analysis Software (CAQDAS), describing in general what it can and cannot do, when to use it and what methodologies it works well with. A link will take you to a review of several popular programs. - CAQDAS Networking Project
Beginner Intermediate Advanced
This site contains a variety of resources on computer assisted data analysis for qualitative research. The link to “Qualitative Resources” may be especially useful if you are looking into software. It includes access to a list of online articles and a qualitative software discussion list amongst other options. - Choosing Qualitative Data Analysis Software: Atlas/ti and Nudist Compared
Christine A. Barry. (1998) Sociological Research Online
Beginner Intermediate
This article discusses the advantages and disadvantages of computer assisted qualitative data analysis software, reviews the primary differences between various software programs, and compares the two most popular programs, Nudist and Atlas/ti, in detail. - A Software Sourcebook: Computer Programs for Qualitative Data Analysis.
Weitzman, E.A., and Miles, M.B. (1995). Thousand Oaks, CA: Sage.
Intermediate Advanced
This book provides a comprehensive review of your options for qualitative analysis software. The authors present strengths and weaknesses of the programs, which are grouped into six types depending on their primary function. The book helps you determine what software is most appropriate for your analysis based on the amount, types, and sources of data you have to analyze and the types of analyses you want to conduct.
Analysis by hand – crude, but effective!
References
Barry, C. (1998). "Choosing Qualitative Data Analysis Software: Atlas/ti and Nudist Compared." Retrieved September 2006 from: http://www.socresonline.org.uk/3/3/4.html.
Fitzpatrick, J.L., J.R. Sanders, and B.R. Worthen. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines. Boston, MA: Pearson Education, Inc.
Frechtling, J. and L. Sharp. (1997). User-Friendly Handbook for Mixed Method Evaluations. Downloaded September 17, 2006 from http://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm.
Seidel, J. (1998). Qualitative Data Analysis. The Ethnograph v5 Manual, Appendix E. Downloaded September 18, 2006 from: http://www.qualisresearch.com/.
Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage.