Let's look at tests at the state level, national level, and tests that are part of international comparisons. Reading First fails at all three levels, producing no improvement.
"No improvement" really means failure: According to the Center for Education Policy, students in Reading First get an extra 100 minutes per week of reading instruction, or an extra semester every two years.
If Reading First were only mildly effective, there would be obvious improvement. In other words, we are comparing extra reading instruction to not teaching reading at all. (See Gerald Coles' book Reading the Naked Truth for a discussion of the importance of considering the question "compared with what"?) Of course, not all children are involved in Reading First, but this massive extra instruction should make a noticeable difference if it is effective.
A 2007 report from the Center on Education Policy resulted in claims of victory from the Bush administration. The report examined test score data on tests of reading from individual states, and focused on the percentage of children to achieve "proficiency."
I did a comparison of gains made before NCLB was implemented (2001-02) and after (2002-06) and found that the rate of improvement for 4th graders increased by less than one third of a percent, moving from 1.2 percent more children classified as proficient each year before NCLB to 1.51 percent after NCLB for the 11 states where data were available.