Beginning in 2009, Brescia chose to incorporate standardized testing into the academic assessment strategy, and the instrument chosen was the PROFICIENCY PROFILE FROM ETS. The Proficiency Profile evaluates skills in reading, critical thinking, and math, as well as content areas of natural science, humanities, and social sciences. The advantages of the test are numerous and include:
- ease of implementation;
- rapid benchmarking with over 200 colleges;
- validation and vetting of the testing materials by ETS.
Students are pre-tested as freshmen, and members of the same cohort are tested again as seniors. Individual scores for each student are available to the institution and range from 400 to 500 for both tests. The mean of the 2011 freshman cohort of 137 students was 431.03, as seen here in Table 1:
TABLE 1: STANDARD VERSION TEST
Number of Students Tested: 137
Number of Students Included in These Statistics: 137
In the fall of 2014, students with senior status were post-tested. This cohort of 45 students (who had been tested as FTFT freshmen) had a mean score of 447.69, as seen here in Table 2:
TABLE 2: POST-TEST DATA— ABBREVIATED FORM OF TEST (40 minutes)Number of Students
Number of Students Included in These Statistics: 48
This population overlapped with the original entering cohort by 82%. (The remaining 18% entered in the previous year; their aggregate cohort data is not presented here.) Since individual pre- and post-test scores were available for the 45 students, the increase in scores was analyzed using a one-tailed t-test. The one-tailed t-test reveals the difference between the two groups of scores to be significant. The University has concluded that overall learning occurred between the freshman and senior years, though a cohort group of 45 is admittedly a small sample size; 50 or more is recommended.
Performance was also analyzed with regard to gender. The mean score of the females in the cohort of 45 when they were freshmen was 424, and in their senior year was 434. The average GPA of graduating females was 3.22. The mean score of males in the cohort of males when they were freshmen was 433.5, and when they were seniors rose to 445. Graduating males had a mean GPA of 2.98.
In an effort to try to get a broader perspective regarding learning in the skill and content areas, the aggregate data seen here in Table 3 can be helpful:
The content and skill areas are evaluated with sub-scores ranging from 100 to 130. With regard to skills, students transitioned in critical thinking from a mean of 108.17 to 112.19; in reading from 112.77 to 118.48; in writing from 111.69 to 116.96, and in math from 110.49 to 112.74. In an effort to contextualize the data, the scores were BENCHMARKED in comparison to national sub-score achievement posted on the ETS website for both freshmen and seniors. ETS posts the distribution of institutional mean scores for more than 200 colleges that use the Proficiency Profile for pre- and post-testing. This data essentially helps institutions benchmark their own student performance against that of all the other Proficiency Profile users.
Brescia does a dual analysis: comparison internally between freshmen and seniors, and externally against national averages. Analysis of Brescia University data against national benchmarks indicates that Brescia admits a low-scoring population as freshmen (the “lower selectivity” criteria noted in the Noel-Levitz report cited above). However, Brescia is able to lift the initially low-scoring student into a higher ranking in almost all areas:
- Critical Thinking — from the 18th to the 31st percentile
- Reading — from the 8th to the 27th percentile
- Writing — from the 14th to the 77th percentile
However, in math skills, Brescia students moved from the 21st to the 14thpercentile between freshman and senior year.
In the content areas, the mean transitioned thus:
- Humanities — from the 14thto the 31st percentile
- Social Sciences — from the 9th to the 12th percentile
- Natural Sciences — from the 10th to the 23rd percentile
Table 4 further ranks student performance levels as proficient, marginal, and not proficient, based on criteria established by ETS. As can be seen here in Table 4, there were substantial increases in the number of students moving into the “proficient” level by senior year for Reading, Writing, and Math:
Table 5 from the ETS website provides the proficiency level for the average of students at 200 colleges:
This data reveals that, in comparison with other colleges using the Proficiency Profile from ETS, Brescia University data is:
- on par with other colleges in the area of Reading
- above the norm with other colleges in the areas of Writing
- below the norm with other colleges in the area of Math
In future, it might be helpful to select a smaller group of peer institutions by which to benchmark; ETS does not provide the Selectivity Ratings of other colleges that use their products.