Florida Alternate Assessment
Item Characteristics Study:
Analysis of Item Response Data
Summary of Results

2010–2011

Prepared by Measured Progress for the
Florida Department of Education

TABLE OF CONTENTS

Contents
Chapter 1: OBJECTIVE
Chapter 2: RESEARCH QUESTIONS
Chapter 3: METHOD
3.1 Design
3.1.1 Design Specifics
3.2 Materials
3.3 Process
3.4 Analyses
3.5 Results
3.5.1 Observed p-values
3.5.2 ANOVA
3.5.3 Individual Item Sets
Chapter 4: DISCUSSION

Chapter 1: OBJECTIVE

The Florida Alternate Assessment (FAA) is based on a tiered level of complexity. Next Generation Sunshine State Standards Access Points, approved by the Florida State Board of Education, creates the frameworks upon which alternate assessment items are constructed. A single item consists of three questions, one at the Participatory level of complexity (least challenging), one at the Supported level, and one at the Independent level (most challenging). The FAA was developed to allow students an opportunity to progress through all three levels of complexity per item. Teachers present questions within an item to students until an incorrect answer is given at the Participatory, Supported, or Independent level. 

This method of test construction theoretically permits an increasing level of complexity for the questions within an item. In order to confirm that the questions within each developed item are in the desired order of hierarchical difficulty, it is necessary to examine the scores obtained via an administration method that provides the opportunity for a student to respond to all questions within an item (irrespective of achieving a correct score at any one level of complexity). 

The purpose of this study is to examine the hypothesis that the difficulty of test questions within items increases with each level of complexity (i.e., Independent questions are more difficult than Supported questions, which are in turn more difficult than Participatory questions). 

Chapter 2: RESEARCH QUESTIONS

1. Does the difficulty of test questions within items increase as expected, if difficulty is measured by calculating item average scores?
2. Is the increase in difficulty observable for all items in the test?
3. Is the increase in difficulty observable at all grade levels tested? 

Chapter 3: METHOD

3.1 Design
We administered the entire test to a sample of students without using any of the scaffolding or stopping guidelines (i.e., all students were administered all test questions regardless of whether they correctly answered the lower-complexity questions). 

3.1.1 Design Specifics
-  Mathematics and Reading
-  Grades 3, 4, 5, 7, 8, and 9
-  Separate test kits were produced based on grade and content area. Each kit had 20 Form A items from the spring 2010 FAA. Each item was composed of three questions – one each at the Participatory, Supported, and Independent levels of complexity.
-  Test kits were administered to 500 students per grade and content area. The resulting usable sample sizes ranged from 444 for several items in grade 8 mathematics and reading to 483 for several items in grade 9 reading.
-  Items were scored on a 1-0 (right-wrong) metric, without scaffolding (i.e., the possible scores on each three-question item were 0, 1, 2, or 3 instead of 0, 1, 2, 3, 6, or 9).

3.2 Materials
-  Mathematics kits (grade specific). Included: test booklet (mathematics only), response booklet (mathematics only), and mathematics cards (where applicable).
-  Reading kits (grade specific). Included: test booklet (reading only), response booklet (reading only), passage booklet, reading cards (where applicable), and reading strips (where applicable).
-  Scannable student answer sheet. Space for teachers to write/bubble in student information. Space for teachers to record student scores.
-  Administration guide.  Measured Progress document that included information such as administration practices, rules for scoring student responses, test security information, return of materials to Measured Progress (including 1-day UPS pick up date).

3.3 Process
-  Student recruitment. To maximize school district and teacher representation, each district’s proportion by grade of alternate assessment participants from the spring 2010 assessment was calculated by the Florida Department of Education (FLDOE). This number was then stratified within district and grade on the following variables: gender, race/ethnicity, and level assessed in spring 2010. Due to security issues, students in grade 4 were administered test kits based on the grade 3 spring 2010 FAA, students in grade 5 were administered test kits based on the grade 4 spring 2010 FAA, and so on. The FLDOE was responsible for the student recruitment process.
-  Pre-study Webinar: sessions were held one week before the assessment window opened. 
-  Administration window: October 25–November 5, 2010.
-  Post-study Webinar: session was held one week after the assessment window closed. 
-  Printing, distribution, and login. Measured Progress was responsible for the printing, shipping, and return of all materials for this study. Materials were printed, shipped, and logged in a secure manner, similar to the processes utilized by Piedra Data Services. One-day UPS pickup and return service labels were provided for the return of all materials to Measured Progress. All student answer documents were scanned at Measured Progress and this information was provided to the Data and Reporting Services (DRS) department for data analysis.

3.4 Analyses
Item statistics were examined for the modified assessment to investigate whether item means decrease as students progress from the Participatory (P) level to the Supported (S) level of an item, and from the supported to the Independent (I) level. First, the item means were simply observed to see if the means seem to confirm the expectations. Next, the item means were converted to delta values and an analysis of variance (ANOVA) was conducted. The items were treated as a random effect (theoretically assumed to be randomly selected from a larger hypothetical pool), and the different levels of complexity within an item were treated as a three-level factor with repeated measures (because the same students took all three levels within an item). When complexity was significant, post-hoc Tukey tests were conducted to determine which levels were significantly different from one another. 

To determine whether there were any items for which the effects of complexity were not statistically significant, an analysis was conducted on each pair of questions within each item. For each pair of questions, the difference score was calculated for the students who took both questions, and the mean of these difference scores was calculated. Next, a bootstrap resampling procedure was applied to this sample of difference scores. For each bootstrap sample, the mean of the difference scores was calculated. The standard deviation of these bootstrap means was then taken as an estimate of the standard error. Then the observed mean difference was divided by the standard error to obtain a Z statistic to test the null hypothesis of Z = 0 against the alternative hypothesis of Z > 0 (the order of the difference was always taken in such a way that the complexity assumption would indicate a positive value). In this way, one-tailed hypothesis tests were conducted to compare the three levels of complexity within each item in each grade and in each content area.

3.5 Results

3.5.1 Observed p-values
Looking at the observed P, S, and I p-values for each of the items (20 items in each of six grades for two content areas), in 209 out of 240 cases, the p-values were ordered as expected with the p-value for P being greater than that for S, which in turn was greater than that for I. The results are shown in Tables 1 and 2 (see tables and figures at end of document), for mathematics and reading, respectively.

3.5.2 ANOVA
Given the p-value results, it was not surprising that the repeated-measures ANOVA results indicated that the complexity factor was statistically significant (p < 0.0001) for every grade level for both content areas. The ANOVA hypothesis testing results are presented in Tables 3 and 5, for mathematics and reading, respectively. Furthermore, the R-squared values (variance accounted for) across the 12 combinations of grade and content area ranged from 47% to 80% (mean = 0.68, sd = 0.09), which are considered to be large effect sizes. The ANOVA effect size results are presented in Tables 4 and 6, for mathematics and reading, respectively.

The post-hoc Tukey tests indicated that the average complexity delta values for each grade and content area were significantly different from each other in the expected manner in all 12 cases at level 0.05 (one-tailed test). 

3.5.3 Individual Item Sets
These analyses showed that the desired pattern among the P, S, and I questions was statistically significant (P significantly greater than S, and S significantly greater than I) at level 0.05 for 75.8% of the items (182 out of 240). For both content areas, the nonsignificant results tended to occur for P vs. S for grade 3, but for S vs. I for the other grades. 

Furthermore, in 238 out of 240 cases, P was significantly greater than I. In two items (reading grade 3, item 5; and mathematics grade 5, item 17), the p-value of the P question was less than the p-value for the I question, but the difference was not statistically significant. In all the other items, the P question had a greater p-value than that of the I question, and in every case the difference was statistically significant. 

The paired-difference hypothesis testing results for all the pairs of questions in the items are displayed in Figures 1 and 2, for mathematics and reading, respectively.

Chapter 4: DISCUSSION

The results clearly indicate that the P, S, and I questions are functioning as desired for the Florida Alternate Assessment for both content areas and for all grade levels. Even at the level of the individual items, where the statistical power is much less because of the reduced sample size, nearly 80% of the items displayed statistical significance in complete support of the Complexity Assumption. Furthermore, in 99% of the items the P vs. I difference was statistically significant in accordance with the Complexity Assumption.

[Table 1. The p-Values for the Mathematics Items] 

[Table 2. The p-Values for the Reading Items]

[Table 3. ANOVA Hypothesis Testing Results for Delta Values for Mathematics Items]

[Table 4. ANOVA Effect Size Results for Delta Values for Mathematics Items]

[Table 5. ANOVA Hypothesis Testing Results for Delta Values for Reading Items]

[Table 6. ANOVA Effect Size Results for Delta Values for Reading Items]

[Figure 1. Paired Difference Hypothesis Test Results for All Pairs of Questions in the Mathematics Items. The horizontal line shows the Z statistic criterion of 1.645, above which a Z is statistically significant at level 0.05 for the one-tailed tests.]

[Figure 2. Paired Difference Hypothesis Test Results for All Pairs of Questions in the Reading Items. The horizontal line shows the Z statistic criterion of 1.645, above which a Z is statistically significant at level 0.05 for the one-tailed tests.]