Computer Science 494: Research Methods
Study Guide
Unit 2: Sampling, Measurement, Reliability and Validity
This unit contains the following sections:
- 2.1 Preface
- 2.2 Learning Outcomes
- 2.3 Sampling
- 2.4 Measurements
- 2.5 Reliability and Validity
- 2.6 References
2.1 Preface
Sampling involves techniques for finding an appropriate representative sample from the population or dataset of interest. Measurement determines how values are assigned to outcomes of the experiment. Reliability refers to the consistency of measurements or the ability of a measuring tool to provide consistent measurements. Validity has two aspects: internal validity, which refers to the extent to which a causal relationship between the dependent and independent variables is demonstrated, and external validity, which refers to the extent to which an experimental result can be generalized to a larger setting. This unit discusses these four concepts in greater detail.
We anticipate that you will need about 7 hours (420 minutes) to complete this unit.
2.2 Learning Objectives
After completing Unit 2, you should be able to
- define the sampling needs for a given research context.
- compare measurement types, and discriminate between them.
- create tests of reliability and validity.
2.3 Sampling
We estimate that this segment of Unit 2 should take about 3 hours (180 minutes) to complete.
Reading Assignment 2.3.R1
This reading introduces the concept of sampling. Make certain that you follow each of the links, including the link to “External Validity” in the menu at the left-hand side of the screen.
Trochim, W.M.K. (2006a). Sampling. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/sampling.php
Reading Assignment 2.3.R2
The Wikipedia page below contains a comprehensive outline of topics related to sampling.
Sampling (statistics). (2014, March 2). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Sampling_(statistics)&oldid=597750580
Viewing Assignment 2.3.V1
Watch the following video on probability sampling:
Bencivenga, R. (2011). Probability Sampling. Retrieved from http://vimeo.com/30606397
2.4 Measurements
We estimate that this segment of Unit 2 should take about 2 hours (120 minutes) to complete.
Reading Assignment 2.4.R1
This reading introduces the concept of measurement. Make certain that you follow each of the links, including the link to “Construct Validity” in the menu at the left-hand side of the screen.
Trochim, W.M.K. (2006b). Measurement. Retrieved January 12, 2012, from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/measure.php
Reading Assignment 2.4.R2
The reading below provides an introduction to how software quality is measured:
Software quality. (2014, March 5). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Software_quality&oldid=598237325
Reading Assignment 2.4.R3
Reread the Research Methods Knowledge Base discussion of levels of measurement:
Trochim, W.M.K. (2006c). Levels of measurement. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/measlevl.php
Reading Assignment 2.4.R4
The readings below discuss tests that can be used to measure behaviour (intelligence, language aptitude and learning style).
Weschler Adult Intelligence Scale. (2014, 28 February). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Wechsler_Adult_Intelligence_Scale&oldid=597462445
Modern_Language_Aptitude_Test. (2013, March 15). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Modern_Language_Aptitude_Test&oldid=544473415
Reading Assignment 2.4.R5
This reading introduces the concept of scaling. Make certain that you follow each of the links.
Trochim, W.M.K. (2006d). Scaling. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/scaling.php
2.5 Reliability and Validity
We estimate that this segment of Unit 2 should take about 2 hours (120 minutes) to complete.
Reading Assignment 2.5.R1
The tutorials below introduce the concepts of reliability and validity. Note that the links within the tutorials are likely to be inactive, as the tutorials are available only through the Internet Archive.
Colosi, L. (1997). Reliability and validity: What’s the difference? Retrieved August 11, 2010, from http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi2.htm
Note: This URL was inactive on January 12, 2012, but the website was available through the Internet Archive: http://web.archive.org/web/20100811193936/
Rymarchyk, G. [2004]. Validity. Retrieved August 11, 2010, from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/tutorial/Rymarchk/rymar2.htm
Note: This URL was inactive on January 12, 2012, but the website was available through the Internet Archive: http://web.archive.org/web/20100505014032/
Reading Assignment 2.5.R2
Read or reread each of the discussions from the Research Methods Knowledge Base listed below. Make certain that you follow each of the links provided within the articles.
Trochim, W.M.K. (2006e). Introduction to validity. Retrieved from http://www.socialresearchmethods.net/kb/introval.php
Trochim, W.M.K. (2006f). External validity. Retrieved from http://www.socialresearchmethods.net/kb/external.php
Trochim, W.M.K. (2006g). Construct validity. Retrieved from http://www.socialresearchmethods.net/kb/constval.php
Trochim, W.M.K. (2006h). Reliability & validity. Retrieved from http://www.socialresearchmethods.net/kb/relandval.php
Trochim, W.M.K. (2006i). Qualitative validity. Retrieved from http://www.socialresearchmethods.net/kb/qualval.php
Trochim, W.M.K. (2006j). Internal validity. Retrieved from http://www.socialresearchmethods.net/kb/intval.php
Trochim, W.M.K. (2006k). Conclusion validity. Retrieved from http://www.socialresearchmethods.net/kb/concval.php
Assignment 1
Review Assignment 1 on the course home page, and do whatever you can at this point.
2.6 References
Bencivenga, R. (2011). Probability Sampling. Retrieved from http://vimeo.com/30606397
Colosi, L. (1997). Reliability and validity: What’s the difference. Retrieved August 11, 2010, from http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi2.htm.
Note: This URL was inactive as of March 6, 2014, but the website was available through the Internet Archive: http://web.archive.org/web/20100811193936/
Modern_Language_Aptitude_Test. (2013. March 15). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Modern_Language_Aptitude_Test&oldid=544473415
Rymarchyk, G. [2004]. Validity. Retrieved August 11, 2010, from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/measure.php
Note: This URL was inactive as of March 6, 2014, but the website was available through the Internet Archive: http://web.archive.org/web/20100505014032/
Sampling (statistics). (2014, March 2). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Sampling_(statistics)&oldid=597750580
Software quality. (2014, March 5). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Software_quality&oldid=598237325
Trochim, W.M.K. (2006a). Sampling. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/sampling.php
Trochim, W.M.K. (2006b). Measurement. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/measure.php
Trochim, W.M.K. (2006c). Levels of measurement. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/measlevl.php
Trochim, W M.K. (2006d). Scaling. Retrieved from the website of the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/scaling.php
Trochim, W.M.K. (2006e). Introduction to validity. Retrieved from http://www.socialresearchmethods.net/kb/introval.php
Trochim, W.M.K. (2006f). External validity. Retrieved from http://www.socialresearchmethods.net/kb/external.php
Trochim, W.M.K. (2006g). Construct validity. Retrieved from http://www.socialresearchmethods.net/kb/constval.php
Trochim, W.M.K. (2006h). Reliability & validity. Retrieved from http://www.socialresearchmethods.net/kb/relandval.php
Trochim, W.M.K. (2006i). Qualitative validity. Retrieved from http://www.socialresearchmethods.net/kb/qualval.php
Trochim, W.M.K. (2006j). Internal validity. Retrieved from http://www.socialresearchmethods.net/kb/intval.php
Trochim, W.M.K. (2006k). Conclusion validity. Retrieved from http://www.socialresearchmethods.net/kb/concval.php
Weschler Adult Intelligence Scale. (2014, 28 February). Wikipedia. Retrieved March 6, 2014, from http://en.wikipedia.org/w/index.php?title=Wechsler_Adult_Intelligence_Scale&oldid=597462445