Spending not related to proficiency

Last night I posted a response to "Save Our Schools" blogpost claiming that spending does result in better student achievement. SOS was merely repeating some dubious findings from another organization, the Federal Education Budget Project.

Yesterday I showed that, contrary to their claims, spending was not correlated with higher levels of student achievement. In fact, the results were random. Of course, I looked at the average score, not proficiency levels which is what the Federal Education Budget Project used.

The Federal Education Budget Project compares state's "current spending" (which excludes capital projects and debt repayment" with the percentage of students scoring proficient (the second highest possible category on the NAEP).

So how strong is the relationship:

ANALYSIS 1: Just testing the strength of the relationship the Federal Education Budget Project claims exists, exactly how they report the data.

This chart above is from the Federal Education Budget Project. Note, they've zoomed in to make the graph appear more dramatic. They've also failed to include a regression line to show the trend. This leads me to believe they may be trying to mislead the public. Below is my graph - same data, same X and Y axis scale but I've included a regression line.

Finally, a second graph - same data, but I've included the full range of possible X and Y axis values and a regression line. (Remember the possible percentage of students who can score proficient is not 10 percent through 45 percent but 0 percent through 100 percent) Note: be wary of any graph that does not include 0 as the starting point for the X and Y axis.

Not only does a more honest graph show that the seemingly dramatic results are kinda flat, but the math shows nothing may be going on at all.


Current Spending and the Percentage of Students Scoring Proficient:  R Square 0.046311 | Adjusted R Square 0.026848 | P-value 0.129378 | t Stat 1.542535 | Coefficients 0.000477

When looking at proficiency rather than scale score the R2 jumps from 0.034 to 0.046. This is still very weak. The P-value and t-stat still suggest there is not enough certainty to declare that a relationship exists.

ANALYSIS 2: for this analysis I look at current spending and the percentage of students scoring proficient or better (why stop at just proficient when students can also score advanced.

Current Spending and the Percentage of Students Scoring Proficient or Better:  R Square 0.065282 |       Adjusted R Square 0.046207 | P-value 0.070358  | t Stat 1.849933 | Coefficients 0.000639

The relationship grows stronger but the P-value and t-stat still fall short of us being confident enough that the relationship did not occur at random.

ANALYSIS 3: For this analysis I control for poverty, ELL students and students with learning disabilities and compare current spending with the percentage of students scoring proficient or better.

Current Spending and the Percentage of FRL-NonELL-Non-SD Students Scoring Proficient or Better:   R Square 0.004456 | Adjusted R Square -0.01586 | P-value 0.641637 | t Stat 0.468314 |Coefficients 0.000118

Any hope you may have had for finding a relationship between spending and student proficiency on the 8th grade NAEP reading exam should be tossed out the window. Once you control for the differences in student poverty, the number of English language learners and students with disabilities the relationship disappears into random nothingness.

Random, like that...