10. What are the critical aspects of the Assessment Plan?

ASSESSMENT OF CANDIDATE PERFORMANCE

Principles for Performance-Based Assessment Systems in Professional Education Programs

Assessing what professional educator candidates know and can do is critical to implementing the performance-based standards of the National Council for Accreditation of Teacher Education (NCATE) and its affiliated national professional specialty organizations. Given the complexities of teaching and other educational professions, the range of knowledge, skills, and dispositions to be assessed, the multiple purposes for which assessment results are used, and the stakes associated with the outcomes, assessment in professional education programs and units needs to include multiple measures implemented on a systematic and ongoing basis as part of a comprehensive system. This document outlines principles set forth by the NCATE Specialty Area Studies Board for performance-based assessment systems at the program level.

Although assessment systems will vary across programs and units, they generally should a) address the knowledge, skills, and dispositions to be acquired by professional educator candidates as set forth in program goals; b) be consistent with the standards of relevant national and state accrediting/approval bodies; c) have multiple means for measuring candidate performance and impact; and d) provide on-going, systematic information useful for decision-making. It is particularly critical that assessment systems provide credible results that are collected and used in a fair, valid manner consistent with their intended purpose(s).

Assessment systems should have the following characteristics:

1. The system is driven by a conceptual framework and program values which espouse assessment as a vehicle for both individual and program self-evaluation and improvement. Assessment is planned and implemented by key stakeholders in a manner consistent with the method of inquiry in the discipline and is considered a means to an end rather than an end in itself.

2. The system includes components which work together in a synergistic manner to address the knowledge, skills, and dispositions of candidates across program goals, objectives and curriculum consistent with the performance-based standards of the respective national professional specialty. Assessment is a goal-oriented process linked to program purposes/goals and national standards.

3. Multiple measures are planned and administered on a systematic, ongoing basis throughout the program beginning with the admissions process. The system includes quantitative and qualitative measures useful for formative and summative assessment. One or more measures designed to yield evidence of positive candidate impact on students is included in the system.

4. The system includes one or more measures which have been created, reviewed, and/or scored by specialty professionals external to the program. Such professionals include those with relevant specialized expertise whose primary responsibility is not to the program/unit, such as field-based master teachers, clinical teachers, intern supervisors, and/or supervisors/employers of program candidates/graduates.

5. The system is clearly delineated. Measures and associated criteria or rubrics (including minimal proficiency levels), as well as policies and practices for obtaining and using results, are described in program documents in a manner which candidates and other stakeholders can understand. Candidates are made aware of program standards and assessment requirements to which they will be held and are provided with models and/or examples of performance and the instruction and support needed to attain such levels.

6. The assessment methods and corresponding criteria included in the system are sufficiently comprehensive and rigorous to make important decisions about the proficiencies of candidates and to safeguard those they may potentially serve. Critical decision-making points are delineated in the system. Decisions that are made reflect the application of relevant criteria and use of results in a manner which discriminates acceptable versus unacceptable performance.

7. The system includes policies and procedures for the gathering, use, storage, and reporting of individual results. Such policies address the rights of individuals (e.g., those afforded candidates by the Family Educational Rights and Privacy Act; confidentiality/anonymity of survey responses). Individual candidate results are reported in a clear manner which acknowledges the source(s) and limitations of the data, individual strengths, and areas of needed or potential improvement.

8. The system includes a structure and procedures for sampling, analyzing, summarizing, and reporting aggregated results. Data are gathered on an ongoing basis and are summarized in a manner which reflects pass rates, the range of performances, and/or the "typical" or "average" performance (e.g., mean, median, or modal performance) as appropriate to the types of measures. Summaries of results are provided to key program stakeholders in a clear manner which acknowledges the source(s) and limitations of the data, data collection and reporting time frame, program strengths, and areas of needed or potential improvement.

9. The program and its assessment system foster the use of results for individual candidate and program improvement. Assessment results are regularly reviewed in relation to program goals and objectives as well as to relevant state and national standards and stimulate changes designed to optimize success.

10. The system has a mechanism and procedures for evaluating and improving itself and its component assessment methods. Evidence of the reliability and validity of the system and its component measures is gathered and used to make decisions about their ongoing use and/or revision. Evidence should address the ability of the system to comprehensively assess performance in a credible manner which is valid, fair, and unbiased.

Characteristics of Sound Evidence

Sound evidence usually exhibits several qualitative characteristics:

• Results from planned, purposeful, and continuing evaluation of candidate proficiencies, drawing on diverse sources;

Monitoring of candidate performance is embedded in the ECIT preparation program and conducted on a continuing basis. This monitoring is planned in response to faculty decisions about the points in the program best suited to gathering candidate performance information, consistent with the institution’s own context and mission. Typically such information is gathered at candidate entry, in coursework, in connection with field experiences, prior to the start of practica, and at completion of the program.

All information about candidates’ proficiencies, from all sources, is drawn on by the unit for continuous evaluation of candidate progress and program success. Excerpts, summaries, and samples from this array of information are provided for use by NCATE in its program quality reviews. Institutions will usually begin to plan their assessment system around activities that are the direct responsibility of the ECIT preparation unit. Examples of assessments that might be used or created within the program include end-of-course evaluations but also tasks used for instructional purposes such as projects, journals, observations by faculty, comments by supervisors, samples of candidate work, and other information that would commonly be available for faculty use in determining the adequacy of the candidate’s accomplishments in a course.

The monitoring information from the ECIT preparation program can be complemented by evaluations originating from external sources that supply information on candidate proficiencies. Examples from outside the unit are candidate performance evaluations during induction years and follow-up studies; performance on state licensure exams that assess candidates’ knowledge and skills; and academic subject knowledge end-of-course examinations, essays, or other demonstrations of achievement.

• Represents the scope of the standards for ECIT preparation;

Candidate performance evidence is congruent with the knowledge and skills in the AECT standards. Institutions determine the best way to demonstrate that all aspects of the standards are covered, but avoid treating each individual statement in the standards and supporting explanations in an individual, serial, and fractionated way. Instead, faculty think through how all their existing assessment information can be marshaled, and what additional information is needed, to demonstrate candidate proficiency across the standards. The usefulness and value of information derived from tests are the key determinants in decisions to use or exclude them from an institution’s performance measurement system.

• Measures the different "attributes" of standards in appropriate and multiple ways;

One conclusion about the current state-of-the art in assessment is that no single test or measurement of teacher candidates is sufficient by itself to represent these different attributes and the full scope of the standards. Multiple measures provide wide opportunities for candidates to demonstrate their accomplishments in relation to the standards. It is anticipated that institutions will draw on the extensive range of available assessment forms, including objective tests (which may be useful to gauge proficiencies in standards calling for candidate knowledge) and also portfolios, observations, reflections, teaching demonstrations, analytic work, candidate work samples, and other forms of evaluative information demonstrating proficiency in technology use. Consider as well external evidence of graduate success (surveys, licensure tests, employer induction year assessments), artifacts produced by the candidates (products, plans, assessments, case studies), reflective essays, attestations of accomplishments by supervisors, awards and recognitions, professional service, and scholarly activities.

• Results from rigorous and systematic efforts by the institution to set performance levels and judge accomplishments of its candidates;

Faculty establish written and shared explanations of what is valued in a candidate’s response to an assessment—the qualities by which levels of performance can be differentiated—that serve as anchors for judgments about the degree of candidate success. The terms "rubrics" and "criteria" are frequently used in assessment to designate these explanations for levels of performance. These may be stated in generic terms or may be specific to particular assessment tasks. They may define acceptable levels of performance for the institution and one or more levels below (such as borderline, or unacceptable) and above (such as exemplary), or they may be in the form of criteria defining the institution’s expectations for success. The rubrics or criteria are "public," that is shared with candidates and across the faculty. Faculty teach, advise, and prepare candidates for success in meeting critical external performance expectations, as expressed, for example, in state licensure test pass scores.

The institution judges individual candidate proficiencies, and also summarizes and analyzes the proportions of candidates who reach levels expressed in the rubrics or criteria. These results are used both for advisement of individual candidates, and also for strengthening of the courses and experiences offered by the institution to prepare elementary teacher candidates. The summary of results from the faculty judgments in applying the rubrics or criteria are used for the NCATE submission. Examples of candidate work are attached to the institutional submission where that is a useful way to assist reviewers’ understanding of the levels of proficiency reached by candidates.

• Provides information that is credible—accurate, consistent, fair and avoiding bias;

The institution gathers information on the accuracy (or validity) and consistency (or reliability) of its assessments. Accuracy is an expectation that the assessment information measures what is important for the decision to be made and that it represents the performances, competencies, and dispositions that are intended (that is, included in the AECT standards). Consistency is an expectation that successive samples of performances from the same candidate are reasonably related. Assessment systems must also be fair, avoiding bias and providing equitable treatment. These are matters that require professional judgment and are often determined through peer review, evaluations by external experts, or formal validation studies.

• Makes use of appropriate sampling and summarizing procedures.

In preparing the program submission, the institution samples and summarizes information about candidate proficiencies. Sampling refers both to representing the domain of the standards and representing the full range of the program’s candidates. The candidate sample might be taken from the cohort of candidates completing the program in a specific academic year and previous completers so that information about performance of candidates from their entire preparation experience and into employment can be available for demonstration of candidate proficiency. Of course, anonymity of individual candidates and the students of those candidates must be protected.

Candidate proficiency results are summarized through averages, spread of scores, and distributions of rubric scores. Summary results are requested because NCATE’s interest is in making decisions about program quality, rather than decisions about individual candidates. These summaries are made meaningful through illustrations such as samples of examination questions, examples of written responses, and analytic materials intended to inform reviewers of the proficiencies that candidates achieve in relation to the standards.

Of course, institutions that have sound evidence systems use the data to advise individual candidates and to strengthen teaching, courses, experiences, and programs.

These qualities of assessment evidence are not, themselves, the requirement for submission. The submission is developed to describe the results of the assessment evidence.

Assessment Systems: An Explanation of the NCATE Transition Plan

NCATE understands that the development of an assessment system an take several years. It has made provisions for the development of assessment systems in its Transition Plan. The Transition Plan identifies levels at which the first two standards must be addressed at the time of the unit's on-site visit. During visits, NCATE expects the unit to make available performance assessment data that are currently collected. These data include assessments conducted internally by the unit and external data such as state licensing tests, program review reports, and graduate and employer surveys. In addition to performance data that are already collected, the unit must, at a minimum, present a plan of the unit's assessment system during the first year that the standards are effective, fall 2001 and spring 2002. Increasing implementation of the plan is required in each subsequent year. Consequently, all units1 should follow the schedule outlined in the Transition Plan for the development and implementation of their assessment systems. Units may move at a faster pace at their discretion.

This paper is written to detail NCATE expectations for institutions that are developing assessment systems and have visits scheduled for fall 2001 and beyond. At a minimum, all institutions must have a plan for their assessment system during the 2001-2002 academic year. Plans for assessment systems should include the six items detailed in this paper and should be developed collaboratively by the units' educational communities. In addition, units should review the appropriate components of an assessment system detailed in the rubric and supporting explanation for Standard 2 of the NCATE Unit Standards.

According to the Transition Plan, units with visits in fall 2001 and spring 2002 should have a plan for an assessment system that:

1. Identifies transition points at the unit and/or program level.
   
   Transition points are key points in a program when the unit assesses candidate knowledge, skills, and dispositions, and determines if candidates are ready to proceed to the next stage. The NCATE standards require transition points upon entry, prior to entering clinical practice, prior to exiting clinical practice, and upon program completion. While the four transition points stated above must be in place for all programs, institutions may have, at their discretion, additional transition points in their programs.
   
   The unit should anticipate a course of action if assessments indicate that candidates are not yet ready to proceed to the next stages of their programs.
   
   Possible actions might include remediation, r~taking assessments, denial of advancement, and academic probation.
   
   
2. Identifies the major assessments to be used at the stated transition points.
   
   Major assessments can include tests, portfolios, essays, student teaching/internship evaluations, student work samples with reflections, etc. The assessments should be linked to the learning outcomes in the conceptual framework and should reflect institutional, state, and national standards. The assessments must address candidate content knowledge, pedagogical knowledge, professional knowledge, pedagogical content knowledge, dispositions, and candidates' positive impact on K-12 student learning.
   
   At this stage in the development of the assessment system, the unit must decide what assessments will be used and when. NCATE does not dictate the types of assessments used, however, evaluations of student teaching/internship and state licensing assessments, where applicable, are expected to be two of the components of the assessment system. NCATE supports the use of multiple assessments in evaluating candidates. The types of assessments may change over time based on considerations of fairness, accuracy, and consistency.
   
   While a review of all assessments in all courses may be in order to address unit and program coherence and the modeling of best practices by faculty, only major assessments should be identified as part of the assessment system. The instruments for these assessments may not yet be developed, but the unit must indicate a timeline for their development. See Item 3 below.
   
   The units should specify which elements of the standards each assessment is designed or being designed to evaluate. The same assessments must be administered to all candidates in a given cohort in a given program/unit. This means that all candidates in a program that reach the designated transition points should be required to complete the same assessments. When including course-based assessment(s) as part of the assessment system, the unit should ensure that candidates in different sections of the same course are administered the same assessment(s). Inevitably, assessments will differ across programs. The unit must coordinate the effort and ensure that each program is operating in the context of an overall assessment system.
   
   Assessments must be fair, accurate, and consistent. To ensure this, the unit may need to provide time and resources for the review of curricula to ensure that candidates have the opportunity to learn the materials assessed. In addition, the unit must provide time and resources for piloting assessments, developing benchmarks, ratings assessments, and analyzing the extent to which the assessments were successful in measuring targeted candidate knowledge, skills, and dispositions.
   
   
3. Identifies a timeline for the development and implementation of the major assessments.
   
   The timeline should include major steps in the development and implementation of assessments. These steps may include preparation, the development of drafts, piloting of drafts, benchmarking, ensuring that the assessments meet their intended goals, and refining the assessment instruments. Steps may also include revising graduate surveys, re-thinking faculty evaluations, and developing better focused employer surveys. The timeline should indicate what is to be done, by when, and what committee/person will be responsible for completing the tasks. It can be written in increments such as weeks, months, quarters, semesters, etc.
   
   
4. Identifies the design for the collection, analysis, summarization, and use of data.
   
   Once the system is in place, data from the system, particularly student assessment data, must be compiled at regular intervals. It is anticipated that these intervals will correspond with the transition points discussed above. Decisions should be made about student progress at each interval. The plan for the unit's assessment system should identify how the data will be generated and when the data will be collected. Will portfolios be submitted and evaluated by a committee? What is the content for the portfolios? Will candidates have to sit for an exam that is then graded by two faculty members? Will they be required to complete a student teaching/internship assignment evaluated by peer review?
   
   Further, once the assessments are evaluated, the data must be summarized and analyzed. The purpose of summarization and analysis is to enable the unit to look across the cohort to examine strengths and weaknesses, to identify trends in comprehension of knowledge, skills, and dispositions, and to pinpoint where additional support and academic work needs to be done. The summarization and analysis of graduate and employer surveys can further help the unit identify programmatic strengths and weaknesses. Similarly, the summarization and analysis of faculty evaluations can inform the unit of professional development needs. Summarization and analysis shifts focus from individuals to programs. Programmatic analysis, in turn, should lead to program change and improvement. For more information on the summarization of data, see Sampling and Summarizing Candidate Performance Information, a paper by Emerson Elliot available on the NCATE website.
   
   The Transition Plan indicates that units should have a design for systematic collection, analysis, summarization, and use of data. The unit should describe in its plan for an assessment system what that system is expected to look like when it is operational. When will data be collected? How will it be collected? Who will do the evaluating? Who will summarize and analyze it? When will this take place? How will it be done? When will it be shared, with whom will it be shared, and what mechanisms will be in place to ensure that the data is used to improve the programs?
   
   
5. Identifies aspects of the system that address unit operations.
   
   While the direct assessment of candidates is important, other mechanisms can and should be used to gauge the quality of the unit and its programs. The NCATE 2000 standards require that the unit collect data from applicants, candidates, recent graduates, faculty and other members of the professional community. The collection, analysis, and use of these data must be built in to the unit assessment system. The evaluations and surveys used to gather information are instruments that should be revised, when necessary, to reflect the unit's mission and philosophy. These instruments should also reflect the extent to which the unit is meeting the learning expectations stated in the conceptual framework.
   
   Other aspects of unit operations that could also be evaluated as part of the unit assessment system include the effectiveness of advisement, record keeping, the admissions system, student teaching placement, governance structures, etc.
   
   In addition, provisions should be made to study the extent to which the requirements at the various transition points are adequate predictors of candidate success. The types of questions that might be posed by these studies include: are candidates who barely met admissions requirements scoring as well on the assessments as those who exceeded the requirements? Do candidates with high scores on the assessments have higher scores on employer surveys than those who had weaker scores?
   
   
6. Identifies how information technology will be used in the maintenance of the assessment system.
   
   Tracking student progress and unit operations will likely require the use of computers and computer programs. The unit must have the capacity to retrieve and manipulate data. The unit should describe the role of information technology in the assessment system. The type and complexity of data management systems will depend, in part, on the size of the unit. Small units may be able to store assessment system data is Excel spreadsheets, while larger units may require more sophisticated software.
   
   Units may be at different points in the development of assessment systems; at a minimum, all of the items listed above should be described in one document that includes all programs.

During fall 2001 and spring 2002, BOE members will base their assessment of Standard 2 on the extent to which institutions have addressed the Transition Plan, which is explained in the items above. If an institution exceeds these minimum requirements in the development of its system, then that institution should include the additional information in the document. This document could be the unit's response to Standard 2 in the Institutional Report, or it could be a separate document that the unit makes available to the team via the web or at the time of the visit.

Visits in Fall 2002 through Spring 2005

Institutions with visits in subsequent years should be implementing their plans as delineated in the Transition Plan. For example, institutions with visits in fall 2002 and spring 2003 should have a plan for their assessment systems and they should have developed some performance assessments. Rubrics/criteria for scoring the assessments should be under development and steps or activities to ensure fairness, accuracy and consistency should have been planned. Some data collection should be underway.

Similarly, institutions with visits in fall 2003 and spring 2004 should have an assessment system plan that is inclusive of the six items above. In addition, the assessments should be fully developed. The rubrics/criteria for scoring should also be developed. The unit should be implementing the assessment system by using the assessments and rubrics/criteria with candidates and other members of the educational community. Data from the assessments should be collected and analysis should have begun. These steps should be in the timelines articulated in the assessment system plan.

Finally, institutions with visits in fall 2004 and spring 2005 should have fully developed systems that are being implemented, evaluated, and refined. The system should be clearly articulated and steps or activities to ensure accuracy, fairness, and consistency should be ongoing. In addition, institutions should have evidence that data from internal assessments, as well as external assessments (state tests scores, graduate and employer surveys, etc.), are being used to make changes and improve programs.