USVI Public Records

Document text

DOCUMENT RESUME ED 360 474 CE 064 023 AUTHOR McCaslin, N. L.; Headley, W. Scot TITLE A National Study of Approved State Systems of Performance Measures and StandPrds for Vocational Education. INSTITUTION Ohio State Univ., Columbus. Graduate School. PUB DATE Jun 93 NOTE 59p. PUB TYPE Reports Research/Technical (143) EDRS PRICE MF01/PC03 Plus Postage. DESCRIPTORS Accountability; Educational Legislation; Evaluation Criteria; Evaluation Methods; Federal Legislation; National Surveys; *Performance Tests; *Program Evaluation; Questionnaires; Secondary Education; *State Programs; *State Standards; *Student Evaluation; Tables (Data); Validated Programs; *Vocational Education IDENTIFIERS Carl D Perkins Voc and Appl Techn Educ Act 1990; Job Training Partnership Act 1982; *Performance Monitoring ABSTRACT A national study examined the system of performance measures and standards that had been approved in each of the 50, states, the District of Columbia, Puerto Rico, Guam, and the Virgin Islands in response to the 1990 Perkins Act. Descriptive survey and content analysis research methods were used to develop summary sheets categorizing the types of performance measures approved by each state/territory and the methods adopted to assess performance. The numbers of performance measures approved for secondary and postsecondary vocational education averaged 10 and 8, respectively. Most states (30) had developea a single set of measures to assess both basic and advanced academic skills at the secondary level. Other areas in which performance measures had been approved included the following: competency attainment, work skill attainment, program completion, high school graduation, placement, percentage served, and gender mix. In nearly every state, systems of core standards and measures of performance for secondary and postsecondary education had been developed and implemented in line with the requirements of the 1990 Perkins Act; however, the standards, performance measures, and types of assessment techniques approved by each state varied greatly from state to state. (Contains 13 tables, 26 references.) (MN) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************************************************** N. T.HE OHIO SfAIE I UNIVERSITY A National Study of Approved State Systems of Performance Measures And Standards for Vocational Education N. L. McCaslin W. Scot Headley S DEPARTMENT OF EDUCATION e f Resea-, AN:f .fnpfc..pmen, bot d T:ONAL RE SOLIRCFS ,N,-ORIOATION CFNTFR f 14f, sas Deer ,oproduced as r e.vea N., r. Orgalwabon I ""atTe ef. M'S CPW rj, n ,er esse.', ,r,p5P^I R. "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY At-) TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Comprehensive Vocational Education Program The Ohio State University 208 Agricultural Administration Building 2120 Fyffe Road Columbus, Ohio 43210 June 1993 Mr COPY 11111J;iiLe'Lq FOREWORD The Carl D. Perkins Vocational and Applied Technology Education Act Amendments of 1990 included a requirement that states develop a system of performance measures and standards for use in program evaluation and improvement. This type of requirement had previously been included in the Job Training Partnership Act. However, this was the first time it had been included in vocational education legislation. Implementing a system of performance measures and standards is a large undertaking. Since this was the first time it had been included in vocational education, information is needed regarding the implementation process. This information could be used for improving vocational education programs and guiding future policy initiatives. As of December 1, 1992, no information was available concerning the actual systems of performance measures and standards that had been approved by each state. The project staff is indebted to the State Directors of Vocational Education and their staff, who provided the information on the performance standards and measures systems. This study was sponsored by the Graduate School at The Ohio State University through a seed grant for the Comprehensive Vocational Education Program. Special appreciation is extended to Dr. Roy A. Koenigsknecht, Dean of the Graduate School for providing this funding. Additionally, the researchers are grateful to the following members of the Coordinating Council for the Comprehensive Vocational Education Program: Drs. Aaron J. Miller, Sharon V. Redick, and Michael Scott. The Comprehensive Vocational Education Program is an interdepartmental effort of three colleges. The colleges and the Deans include: Agriculture, Dr. Bobby D. Moser; Education, Dr. Nancy L. Zimpher; and Human Ecology, Dr. Jerelyn Schultz. Ms. Connie Myers provided secretarial services to the project. R. Kirby Barrick Professor and Chair Coordinating Council for Comprehensive Vocational Education ii CONTENTS FOREWORD ii LIST OF TABLES iv INTRODUCTION 1 VOCATIONAL EDUCATION'S INVOLVEMENT WITH ACCOUNTABILITY AND EVALUATION 2 THE JTPA INFLUENCE ON ACCOUNTABILITY AND EVALUATION EFFORTS IN VOCATIONAL EDUCATION 5 USING PERFORMANCE MEASURES AND STANDARDS 7 PROBLEM STATEMENT 9 PURPOSE AND OBJECTIVES 9 METHODOLOGY 10 FINDINGS 11 Approved Measures for Secondary Vocational Education. . . 11 Approved Measures for Postsecondary Vocational Education. 18 Types of Assessment Techniques for Academic and Other Performance 24 CONCLUSIONS 25 RECOMMENDATIONS 29 REFERENCES 31 APPENDIX A 33 APPENDIX B 37 APPENDIX C 43 iii c., LIST OF TABLES TABLE 1. PAGE Measures of Academic Performance for Secondary Vocational Education 12 2. Other Measures of Performance for Secondary Vocational Education 16 3. Measures of Academic Performance for Postsecondary VocP.tional Education 20 4. Other Measures of Performance for Postsecondary Vocational Education 22 5. Assessment Techniques for Basic and Advanced Academic Performance 26 6. Assessment Techniques for Competency and Workskill Attainment 27 7. Techniques for Assessing Reading Performance 45 8. Techniques for Assessing Language Performance 46 9. Techniques for Assessing Math Performance 47 10. Techniques for Assessing Science Performance 48 11. Techniques for Assessing Other Academic Performance. . . 48 12. Techniques for Assessing Competency Attainment 49 13. Techniques for Assessing Workskill Attainment 50 iv INTRODUCTION In the last decade there has been an increasing level of dissatisfaction with the quality of public education in the United States. The initial sound of alarm regarding the inadequacies of our education system began with the release of A Nation at Risk (National Commission on Excellence in Education, 1983). Since that time a number of other National reports have signaled similar concerns. Among these studies were: The Forgotten Half: Non College Youth in America (The William T. Grant Foundation Commission on Work, Family and Citizenship, 1988); America's Choice: Hi h Skills or ow Wa es! (Commission on Skills of the American Workforce, 1990); America 2000: An Education Strategy (U. S. Department of Education, 1991); What Work Requires of Schools: A SCANS Report for America 2000 (Secretery's Commission on Achieving Necessary Skills, 1991); Education Counts (Special Study Panel on Education Indicators, 1991); and Learning a Living: A Blueprint for High Performance (Secretary's Commission on Achieving Necessary Skills, 1992). Each of these studies called for major changes in the U.S. educational system by stressing the gap between the demands of the future and the present level of preparadness of America's youth to meet these requirements. Accompanying the need for improved educational programs was a call for better evidence regarding the accountability of public education. Hill and Bonan (1991) defined accountability in the following manner: Accountability is a relationship between two persons in which four conditions apply: first, one person expects the other to perform a service or accomplish a goal; second, the person performing the activity accepts the legitimacy of the other's expectation; third, the person performing the activity derives some benefits from the relationship; and fourth, the person for whom the activity is performed has some capacity to affect the other's benefits. (p. 35) The acceptance of the concept of accountability in education has been widespread. In fact, as White (1990) explained: Accountability has become a guiding principle in the way states have approached new ways to improve education. Thus, systems to measure accountability have become a major 'business' in The United States (Odden, 1990). Schools are expected to act like businesses, and account for their successes and failures; and though it has never been easy to apply quantitative measures to complex educational processes and outcomes, more and more school systems have been doing just that. (p. 1) 1 Accountability systems in education have traditionally relied on reviews of inputs and processes of the educational systems. However, the focus on accountability systems in education has been changing to one of assessing outcomes (McCaslin, 1990). The National Governors' Association has led a movement to reorient accountability to include outcomes (Henry, McTaggert, and McMillen, 1992). "Education 2000", the current statement of the national goals for U.S. education, also reflected an emphasis on accountability. Accountability efforts can offer important information for improving the educational system. Apling (1989) stated: To improve the education of America's children, educational programs at every level must set standards of excellence and hold students, teachers, and schools accountable for meeting those standards. (p. 4) The movement toward more emphasis on accountability and evaluation has had an impact on vocational education. At the time of the hearings to reauthorize the Carl D. Perkins Vocational Education Act, the Office Technology Assessment (1989) reported that, "there is now widespread consensus for including the vocational education system in the national debate over school reform and academic excellence" (p. 2). Later, Jennings (1991) remarked: The newly reauthorized Carl D. Perkins Vocational and Applied Technology Act is a call from Congress to vocational educators. The message: It's time to get off the sidelines and get into the game. With dozens of reports indicating that a decade of state and local reform efforts have failed to improve U.S. students' performance, vocational education is being asked to play a leadership role. (p. 18) Vocational education has considered accountability and evaluation essential activities for many years. The next section highlights these efforts. VOCATIONAL EDUCATION'S INVOLVEMENT WITH ACCOUNTABILITY AND EVALUATION The 1963 Vocational Education Act was one of the first examples of federal legislation requiring states to conduct evaluations of their programs. In response to this mandate, most vocational education programs conducted follow-up studies to determine the degree to which vocational education graduates found placements in jobs that were related to their training to meet this requirement. Additionally, states often used program reviews conducted by local educa',:ion personnel and external evaluators to chart their future directions. 2 The 1968 Vocational Education Lmendments continued to emphasize state evaluation activities. It expanded the concept by giving evaluation responsibilities to state advisory councils. The Education Amendments of 1976 (P.L. 94-482) further expanded the responsibility for evaluation and had at least 28 references to different forms of evaluation (Wentling, 1980). These evaluation requirements were designed to increase the responsiveness of vocational education to changing labor markets and requirild states to evaluate programs every five years and to determine the extent to which programs completers and leavers: (1) found employment in occupations related to their training, and (2) were considered by their employers to be well-trained and prepared for employment. Seven different groups were specifically charged with conducting vocational education evaluations in the 1976 Amendments: (1) the state boards of vocational education, (2) the state advisory councils for vocational education, (3) the National Advisory Council for Vocational Education, (4) the U.S. Office of Education's Bureau of Occupational and Adult Education, (5) the National Institute of Education, (6) The National Center for Education Statistics, and (7) the U.S. Office of Education's Office of Evaluation and Dissemination. The Carl D. Perkins Vocational Education Act (P.L. 98-524), enacted in 1984, charged the states to: develop measures for the effectiveness of vocational education programs including measurements such as-- (i) the occupations to be trained for reflect a realistic assessment of the labor market of the state, (ii) the levels of skills to be achieved in particular occupations, which will reflect the hiring needs of employers; and (iii) the basic employment competencies to be used in performance outcomes, which will reflect the hiring needs of employers; (p. 2447) The Carl D. Perkins Vocational and Applied Technology Education Act of 1990 continued to include evaluation specifications. It required states to develop core standards and measures of performance for secondary and postsecondary vocational education programs. These performance measures were to include: (1) measures of learning and competency gains, including student progress in the achievement of basic and more advanced academic skills; 3 Li (2) 1 or more measures of performance, which shall include only-- (3) (A) competenc/ attainment; (B) job or work skill attainment or enhancement including student progress in achieving occupational skills necessary to obtain employment in the field for which the student has been prepared, including occupational skills in the industry the student is preparing to enter; (C) retention in school or completion of secondary school or its equivalent; and (D) placement into additional training or education, military service, or employment; incentives or adjustments that are-- (A) designed to encourage service to targeted groups or special populations; and (b) for each student, consistent with the student's individualized education program developed under section 614(a)(5) of the Education of the Education of the Handicapped Act, where appropriate; and (4) procedures for using existing resources and methods developed in other programs receiving Federal assistance. (p.770-771) The Carl D. Perkins Vocational and Applied Technology Act of 1990 was authorized during a time when "performance incentives and quality indicators were very much in vogue" (White 1990, p. 2). Congress had authorized the largest yearly outlay for vocational education ever (1.6 billion dollars) and wanted increased le;countability. Therefore, Congress mandated that states "...set up extensive systems for evaluating programs and the effects of Perkins dollars" (Wilcox 1991, p. 16). Although the concern for quality and accountability was a major reason for establishing statewide systems of performance measures and standards, several studies and groups also had recommended the use of performance measures and standards. The Office of Technology Assessment (1989) reported: "The education and business communities now increasingly support the view that more exacting measurement of the quality of high school vocational programs may be needed" (p. 1). The recommendation of The National Assessment of Vocational Education concerning accountability for vocational education was accepted by Congress when the reauthorization of Perkins was debated (Apling, 1989). Another important influence, as cited by White (1990), for the adoption of performance measures and standards was the National Association of State Directors of Vocational Education. This association had supported the adoption of an accountability system consisting of a system of performance measures and standards for local vocational 4 programs. National reports concerning accountability had not only directed attention to the quality of educational programs but "...also raised many questions about how effectiveness is to be defined and attained" (Ewell 1988, p. 53). Apling (1989) stated the reason for using performance standards: Performance standards have been mandated for various social programs because of concern that these programs are not working as well as they could and that adopting principles of the marketplace will improve these programs. (p. 3) Although the Carl D Perkins Vocational and Applied Technology Education Act Amendments of 1990 presented a specific requirement for a system of performance measures and standards, this was not its first appearance on the national policy scene. The Job Training Partnership Act (JTPA) had previously required a system of performance measures and standards. THE JTPA INFLUENCE ON ACCOUNTABILITY AND EVALUATION EFFORTS IN VOCATIONAL EDUCATION Performance standards were first adopted as an instrument of national human resource policy in the employment and training area with the passage of the Job Training Partnership Act of 1982 (P.L. 97-300). The measures that were developed for these programs included: placement and retention in unsubsidized employment, earnings, and reductions in public assistance, Performance standards had made a major impact on how these programs were conducted. Butler (1988) had made the following assessment of the use of performance measures in JTPA: The most significant outcome-oriented practice has been the development of formal national, state, and locally- administered systems of outcomes measures, and standards for aggregated program achievement, This so called `performance standard' system-driven by a nationally- derived set of outcome numbers against which the performance of local administrative entities, called Service Delivery Areas, or SDAs, (and in turn, states) are measured-has become the basis for judgement about the effectiveness of local programs, and when they are aggregated, about the system as a whole. This rigorously-applied set of measures and standards has the effect of turning the system on its head to comply. The measures have recently been expanded, and there have been annual adjustments in the standards, but in general there has been great consistency over the several years of JTPA., providing a good basis of experience with outcome measures for a national program. (p. 2) 5 For the federal government, priorities in implementing JTPA 1982 were to hold local providers responsible for the outcomes that were attained, to encourage efficient service, to create incentives for effective management of local programs, and to foster acceptance of the program by business and industry (Dickinson & West 1988). Though the record for JTPA was positive--especially compared to the Comprehensive Employment and Training Act (CETA), the federal training program that preceded JTPA--some concerns were raised. Among the concerns reported were: (a) inappropriate targeting of participants and services (Apling, 1989), (b) questionable performance measures (Frazier, 1991), (c) problems in defining services, outcomes, etc. (Frazier), and (d) inconsistent and incomplete data (Office of Technology Assessment, 1989). The influence of performance measures for program accountability can become so great, that it results in the unintended consequences such as reduced services to targeted groups. (Dickinson & West 1988) Some problems were reported when agencies received incentives and adjustments for serving special populations. Apling (1989) reported that in JTPA programs where adjustments were in use, data was difficult to obtain and verify concerning the special conditions that warranted adjustments. When asked to justify funding for programs on the basis of performance standards, JTPA representatives at times found it difficult to produce hard data to support their claims. Further, there were questions about the programs becoming so outcomes driven, that the mission of the program became secondary to producing adequate "numbers." Butler (1988) suggested that: There is an underlying policy question abort the application of performance standards under JTPA. That is, do performance standards achieve the 'investment in human capital' agenda of the Act, or does the approach artificially force states and localities to opt for quick numbers which will yield only short-term results? Unfortunately there are no reliable data which will answer that most basic question...there is little systematic information on what services were actually provided and virtually none at all on who else might have been served had the programs been designed or supported differently-or even on who applied for services but was rejected on grounds that they might not succeed. Instead, we must rely on very partial official data on a few aggregate measures, and on the opinion and self- reporting by professionals involved in the program. (p. 7) Regardless of the problems associated with performance measures and standards, there are advantages to their use. Butler (1988) stated: JTPA has demonstrated that it is possible to establish a national program substantially driven by clearly- specified, measurable outcomes. Moreover, it is possible to tie those outcomes to funding, not only at the federal and state levels, but even down to the level of actual service delivery...As policies which underlie vocational education are developed, JTPA's experience should encourage planners that a focus on outcomes can be achieved. (p. 21). The experience in evaluating local JTPA programs was judged to be of sufficient value by Congress that a similar system of performance standards and measures were included in The Carl D. Perkins Vocational and Applied Technology Education Act of 1990. As Apling (1989) reported to Congress: The use of performance-based management principles in the public sector with an emphasis on program outcomes or the `bottom line' reflects in part a belief that outcome testing will improve the accountability, management, cost ef fectiveness, and ultimate nerformance of public programs. (p. 4) Since the passage of the Carl D. Perkins Vocational and Applied Technology Education Act of 1990, states have been revising and developing their systems of performance standards and measures. The next section presents information related to using performance measures and standards. USING PERFORIeNCE MEASURES AND STANDARDS The impact of the 1990 Amendments will be felt in the amount of attention paid to the outcomes of programs. However, measures and standards must be carefully constructed so as not to hinder the process they seek to evaluate. States must make choices such as whether to emphasize the development of academic measures centered on benchmarking (meeting an outside accepted standard) or on value added. As Ewell (1988) stated: From a policy perspective, however, the issue can be real and concrete: Are institutions and programs to be judged primarily in terms of the degree to which they `develop talent' or in terms of the degree to which their ultimate products meet accepted standards? (p. 64) Some have questioned the merits of these systems. Apling (1989), for example, reported that: Unless carefully drawn and implemented , perform. ince standards may not improve performance and might adversely influence performance; the diversity and complexity of vocational education in this country may make performance 7 1. 2 standards even more difficult to develop and implement than they have been for other programs (p. 3) To accept measures of outcomes alone as an evaluation scheme would not prove useful. McCaslin (1990) indicated: ...if information on student achievement in vocational and academic skills is all that is available, how can teachers and administrators know what worked and did not work? Upon what basis should decisions to add and drop programs be made? What should be changed in order for vocational students to achieve at higher levels? (p. 7) White (1990) echoed this concern by citing Asche: Outcome indicators do not provide information on how or why such outcomes are produced or what to do if results are judged unsatisfactory.(p. 31) Others have expressed confidence in the use of a system of measures and standards. One example is the Oklahoma Department of Vocational and Technical Education (1992) who published the following: We believe that the requirements set out in Carl Perkins support the delivery of high quality vocational programs. (p. 2.3) When used within the context of a broader and more comprehensive system of evaluation, the use of performance measures can be elpful. If needs and processes are evaluated in addition to outcomes (McCaslin, 1990), a much more valuable picture of the status of the program can be determined. Systems of measures and standards can provide information for comparing outcomes across programs and states. When dealing with occupational training, there is an exilectation in the workplace, that measures and standards can be used as benchmarks for performance. Measures and standards for vocational programs also can serve to provide the framework for assessing student progress against an accepted standard, and for identifying programs where outcomes are low, suggesting inadequacies in the system. The use of systems of performance measures, though valued in the JTPA model, will undoubtedly bring a host of problems for vocational education. These problems include the questionable validity of information gathered on program participants (Office of Technology Assessment 1989), especially if the information is self reported; the difficulty in assessing the effects of a program in the development of academic skills (Office of Technology Assessment); the challenges in meeting a mixed set of standards (Apling, 1989); insufficient investment to insure high quality for 8 all the measures (White, 1990); and difficulty in achieving program improvqment when evaluation focuses on outcomes (McCaslin, 1992). It could well be that vocational educators, feeling similar pressure to perform as did JTPA professionals, could develop practices of screehing potential students, collecting inadequate data, and even altering the types of courses offered in an attempt to make the data look good. Will it happen? Only thorough investigation over time will reveal the answer to that question. PROBLEM STATEMENT The move to implement a system of performance measures and standards in vocational education is a large undertaking. The states had until September 25, 1992 to implement the systems of standards and measures (Federal Register, August 14, 1992). Many of the states found it necessary to develop new evaluation procedures as a result of the mandates of the Carl D. Perkins Vocational and Applied Technology Education Act of 1990. Yet, relatively little information is available regarding this process. Hoachlander and Rahn (1992) gathered information in 1991 from the states in an effort to determine the expected makeup of the systems. However, as stated by Hoachlander and Rahn, "The systems actually implemented in fall 1992 may look substantially different, as states continue to develop performance measures and standards." (p. 2) By December 1, 1992, no information was available concerning the actual systems of core standards and performance measures that have been adopted by each state. This information would be useful in further developing and improving the system of performance measures and standards and in meeting the requirements of the Carl D. Perkins Vocational and Applied Technology Education Act of 1990. PURPOSE AND OBJECTIVES This study was sponsored by the Graduate School at The Ohio State University through a seed grant for the Comprehensive Vocational Education Program. The purpose of this study was to examine the system of performance measures and stundards that had been approved in each state of the United States in response to the Carl D. Perkins Vocational and Applied Technology Act of 1990. The specific objectives were: 1. To determine what types of performance measures had been approved in each state. 2. To identify how states have decided to assess their performance. 9 METHODOLOGY Descriptive-survey and content analysis research methods were used in this study. The names and addresses of the 54 state directors of vocational education were used to generate the population for this investigation. For the purposes of this study, a state was defined as including all 50 states in the United States and the District of Columbia, Puerto Rico, Guam, and The Virgin Islands. An initial letter (see Appendix A) was sent to each state director on November 24, 1992 requesting documents describing the system of performance measures-and standards that had been approved by their state board for vocational education. This strategy was used to minimize the amount of time and energy that would be required to provide the information. Approximately four weeks later, a followup letter, containing the original request, was again sent to the 19 state directors thtt had not responded. A third and final follow-up letter was sent on February 1, 1993 to the remaining 7 state directors that had not responded. On March 3, 1993 phone calls were made to the remaining 5 state directors from which no response had been received. As of April 15, 1992, 52 of the 54 states (96%) had responded with information concerning their approved systems of measures for secondary vocational education programs and 50 (93%) had responded with information concerning their approved systems of measures for postsecondary programs. Two states (Iowa and The Virgin Islands) indicated that their system of measures and standards had not been approved by their state boards. Additionally, Georgia and Arkansas did not report their system of postsecondary performance measures and standards. Once the documents had been reviewed, a content analysis was conducted. The analysis resulted in a listing, by states, of the measures that the states had adopted, using the categories listed in section 115 of the Carl D. Perkins Vocational and Applied Technology Education Act of 1990. The following categories were established: (a) basic academic skill, (b) advanced academic skill, (c) competency attainment, (d) workskill attainment, (e) retention/completion, (f) placement, (g) service to special populations, and (h) other measures. A summary sheet was developed for each state, categorizing the data on adopted measures. On March 12, 1993, the summary sheets were mailed to the state directors of vocational education (a copy of the summary sheet, instructions and the cover letter are contained in Appendix B). Personnel from each state were asked re-Aew, verify and amend the listing as necessary. When discrepancies occurred, a further review of the documents was carried out. If necessary, a followup call was made to the state director of vocational education's office for additional clarification. 10 FINDINGS This section reports the measures that states have had approved in implementing their statewide system of core standards and measures of performance. First, the measures approved for secondary vocational education will be presented. These will be followed by the measures approved for postsecondary vocational education. Approved Measures for Secondary Vocational Education The vast majority of the measures reported by the states have been approved for implementation in 1993. However, several measures were reported that were approved for implementation in 1994 or later. As indicated previously, Iowa and the Virgin Islands had not yet approved standards for their system. Therefore, information from these areas was not included il this report. The average number of performance measures approved for secondary vocational education was 10. Academic Skills The information presented in Table 1 identifies the areas for which states have approved measures of learning and competency gains in academic skills. These academic skills include two types: basic and advanced. For purposes of this report the academic skills were classified as either reading, language, mathematics, science or "other". Thirty states were using the same set of measures for both basic and advanced academic skills. The remaining twenty two states were using different sets of measures for basic and advanced academic skills. At the secondary vocational education level, two of the reporting states did not indicate that they had approved any basic or advanced academic skill measures. Basic Academic Skills. The area that states most often had approved as a measure of basic academic skills (see Table 1) was mathematics (85%). This was followed by reading (80%), language (77%) and science (35%). A total of 44% of the states indicated that they had approved an "other" measure of basic academic skills for their system of standards and measures of performance. Examples of "other" basic academic skills included measures in areas such as social studies, grade point averages, critical thinking, and problem solving. Advanced Academic Skills. The areas in advanced academic skills (see Table 1) that were reported approved by each state followed a pattern similar to that reported for basic academic skills. More than three-fourths of the states reported a mathematics measure for advanced academic skills (76%). About two- thirds (65%) of the states reported using reading and language measures. Less than half of the states reported using measures in _ Table 1 Measures of Academic Performance for Secondary Vocational Education by_State " State Basic Academic Skill Advanced Academic Skill Reading Language Math Science Other Reading Language Math Science Other Na AL AK AR AZ* CA* CO* CT DE FL GA* HI* ID* IL IN* IA KS* KY* LA* ME* MD MA MI* MN MS* MO* MT NE* x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x It x x x x x x x x x x x x Note: * = The measure is the same for both basic and advanced academic skill. NO OW Ilk NW Mt Mit MP ilk WI 11111 IMIS 11111 41111 Mr INN IS Mk IS OW 11111 11111 Oa 11111 SS on am ea an ea WS NM (Table 1 continued) Basic Academic Skill Advanced Academic Skill State Reading Language Math Science Other Reading Language Math Science Other NV x x x x x NH* x x x x x x NJ x x x x x x x x x NM x x x x x NY x x x x x x x x NC* x x x x x x x x ND* x x x x x x x x x x OH* x x x x x x x x OK x x x x x x x x x x OR x x PA x x x x x RI* x x x x x x SC* x x x x x x SD* x x x x x x TN* x x x x x x x x TX x x x UT x x x x x x x VT* x x x x x x VA* x x x x x x WA x x x x WI* x x x x x x x x x x WV x x x WY* x x x x x x x x x x DC* x x x x x x x x x x GM* x x PR* x x x x VI Total 43 40 46 19 24 35 35 41 22 23 Percent 80 77 85 35 44 65 65 76 41 43 the area of science (111%) of the states. In the area of "other" advanced academic skills, 43% of the states had approved measures such as critical thinking, problem solving, and social studies. Six of the states reporting did not include any approved measures of advanced academic skills. Other Measures of Performance Table 2 presents information on other measures of performance that have been approved by the states. Only two states had not yet approved any measures of performance for secondary vocational education programs. These areas are discussed below and include: competency attainment, work skill attainment, program completion, high school graduation, placement, percent served, and gender mix. Competency Attainment. Competency attainment was generally defined by the states as basic employability skills. Approximately one-half of the states (44%) reported using competency attainment performance measures. Work Skill Attainment. This area tended to be defined by the states as including measures of specific occupational skills attainment. About three-fourths of the states (72%) reported that their states had approved performance measures on work skill attainment. Program Completion. States generally referred to program completion as the rate at which students fulfilled the requirements of their program. Almost one-half (46%) of the states reported having this type of performance measure approved. Hiah School Graduation. HigA school graduation referred to the rate of students who had successfully completed the requirements for graduation in their school or its equivalent (e.g., General Equivalency Diploma). One-half (50%) of the states reported using this as a performance measure. Placement. This measure of program effectiveness has been used for some time in vocational education. Related placement refers to individuals who have obtained employment in an area closely related to their area of educatin and training. Any placement refers to obtaining any type of job after completing their program of studies. Approximately one-half (46%) of the states reporting using related placement as a performance measure. Whereas, sixty one percent of the states reported using any placement as a performance measure. Nine states reported using both types of placement as performance measures. A total of 92% of the states reported using some type of placement measure. Percent Served. This performance measure referred to the percent of the high school aged special population students that were enrolled in vocational education programs. Slightly more than 14 one-half (52%) of the states reported that they were using this type of performance measure. Gender Mix. Gender mix referred to the percentage of male and female students who were enrolled invocational education programs. Approximately one-third (31%) of the states indicated that this type of measure was being used in their system. 15 Th Table 2 Other Measures of Performance f r Secondary Vocational Education b State Competency Work Skill Program High School Placement Percent Gender State Attainment Attainment Completion Graduation Related Any Served Mix AL x x AK x x AR x x x AZ x x CA x x CO x x x x CT x x DE x x FL x GA x x x x HI ID IL IN IA x x x x x x x x x x x x x x x x x x KS x x x s,- x KY x x x x x x LA x x x ME x x x x x MD x x x x x MA x x x x MI x x x x x x x x MN x x MS x x x x MO x x x MT NE Note: 1-Refers to a measure of service to special population students. age op rat ism 1.1 ani Vat owl MO MP OW AO IMP 1111111 ims 6111.011111111,111111-11111111,111.4111111111-111111Mill111111111111111110 (Table 2 continued) Competency State Attainment Work Skill Attainment Program High School Completion Graduation Placement Percent Served Gender Mix Related Any NV x x x NH x x x x x NJ x x x x x NM NY x x x x x x x x NC ND x x x x x x x OH x x x x x x OK x x x x OR PA x x x x x x x RI SC x x x x x x x x x x SD TN x x x x x x x x TX x x x x x UT VT x x x x x x x x x VA WA x x x x x x x x WI x x x x WV x x x x WY DC GM x x x x x x x x PR x x X x VI Total 24 39 25 27 25 33 28 17 Percent 44 72 46 50 46 61 52 31 Approved Measures for Postsecondary Vocational Education Again, most of the measures reported by the states were approved for use in 1993. However, some of the measures were approved for use after 1993. Iowa, and the Virgin Islands reported that they had not yet approved standards for postsecondary vocational education programs. Information concerning postsecondary measures for Georgia and Arkansas was unavailable. Therefore, information from these areas was not included in this report. The average number of performance measures approved for postsecondary vocational education was 8. A total of four states did not report having any basic or applied academic skill measures approved for their postsecondary programs. Academic Skills The information presented in Table 3 identifies the ar-as for which states have approved measures of learning and competency gains in academic skills for postsecondary vocational education programs. These academic skills include two types: basic and advanced. For purposes of this report the academic skills were classified as either reading, language, mathematics, science or "other". Nineteen states were using the same set of measures for both basic and advanced academic skills. The remaining thirty-five states were using different sets of measures for basic and advanced academic skills. Basic Academic Skills. The area that states most often had approved as a measure of basic academic skills (see Table 3) was math (56%). This was followed by language (54%), and reading (48%). Science was reported being used as measures by 15%. A total of 48% of the states indicated that they had approved an "other" measure of basic academic skills for their system of standards and measures of performanc:... Examples of "other" basic academic skills for postsecondary vocational education included the following: course completion, GPA, social studies, and thinking skills. One state did not report any approved basic academic skill measures. Advanced Academic Skills. Table 3 also reports the areas in advanced academic skills that were reported approved for postsecondary vocational education in each state. In the advanced academic skills area, 48% of the states repoited they were using performance measures related to mathematics and 44% reported using L. asures related to language. Approximately one-third (30%) of the states were using measures in the reading area. Only eight states (15%) were using performance measures related to science. Approximately 40% of the states reported using performance measures related to other advanced academic skill areas. This "other" category includnd measures such as: problem solving, higher order thinking, and interpersonal relations. Three states did not report 18 any approved advanced academic skill measures for their postsecondary vocational education programs. Other Measures of Performance The information in Table 4 presents information on other measures of performance that had been approved by the states for their postsecondary vocational education programs. At the postsecondary level, six states had not yet had any measures of performance approved. The other measures of performance were similar to those presented for secondary vocational education and included: competency attainment, work skill attainment, program completion, placement, percent served, and gender mix. These performance measures will be discussed below. Competency Attainment. States tended to define competency attainment as the development of employability skills. Only one- third (33%) of the states reported that they had approved competency attainment performance measures for postsecondary vocational education programs. Work Skill Attainment. This area was generally defined by the states as including measures of the extent to which students had developed specific occupational skills. Approximately two-thirds of the states (63%) reported that they had approved this type of a performance measure. Program Completion. States tended to refer to program completion as a measure of the ratio of students who initially enrolled to those who met the requi-aments/outcomes of the program. Nearly three-fourths (70%) of the states reported having approved this type of performance measure. Placement. Two types of placement rates were reported by the states: related placement and any placement. Related placement refers to individuals who have obtained employment in an area closely related to their area of education and training. Any placement refers to obtaining any type of job after completing their program of studies. Approximately one-half (48%) of the states reporting using related placement as a performance measure. Fifty six percent of the states reported using any placement as a performance measure. Ten states reported using both types of placement as performance measures. A total of 88% of the states reported using either related or any type of placement as an approved performance measure. Percent Served. This performance measure referred to the percent of the special population students that were enrolled in postsecondary vocational education programs. Slightly more than one-half (56%) of the states reported that they were using this type of performance measure. 19 C:1 Table 3 Measures of Academic Performance for Postsecondary Vocational Education by State Basic Academic Skill Advanced Academic Skill State Reading Language Math Science Other Reading Language Math Science Other AL x x x x x x x x x x AK x x x AR AZ* x x CA x x x x x CO* x x x x x x x x CT x x x x x DE* x x x x x x FL x x x GA HI* x x x x x x ID* x x x x x x IL x x x x x x IN x x x IA KS x x x x KY x x x x LA* x x x x x x ME* x x x x x x MD x x x x x x x x MA MI x x x x x x x MN* x x MS* x x MO* x x x x x x x x MT x x x x NE* x x x x x x Note: * = The measure is the same for both basic and advanced academic skill. 1011111P111111-1111111-11101111111111111111111111111111111.1111111111011111111111111 MI OS Ili MN SO OM al ON ION Mil OM ON OM ill (Table 3 continued) Basic Academic Skill Advanced Academic Skill State Readin Lan ua e Math Science Other Reading Language Math Science Other NV NH* NJ NM NY NC ND* OH OK* OR PA* RI SC* SD TN TX UT VT VA WA WI* WV* WY DC GM PR* VI Total 26 29 Percent 48 54 x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 30 8 27 16 24 26 8 24 56 15 48 30 44 48 15 43 Table 4 Other Measures of Performance for Postsecondary Vocational Education by State State Competency Attainment Work Skill Attainment Program Completion Placement Percent Gender Served Mix Related Any AL x x x x x AK AR AZ x x x CA x x x CO x x x x x CT DE x x x x x x FL x x x GA HI x x x x ID x x x IL x x x x IN x x x IA KS x x x x KY x x x x LA x x x ME x x x x MD x x x MA x x x MI x x x x MN x x x x MS x x x MO x x x MT x x x x x NE x x x x x Note: 1-Refers to a measure of service to special population students. MIN IMP OM MO Mk OW UV OS MI MI IIIII MO IN MN NI 111111 pre as es es am es am cas as es as me es at an am es is am (Table 4 continued) State Competency Attainment Work Skill Attainment Program Completion Placement Percent Served' Gender Mix Related Any NV x x NH x x x x NJ x x x x x NM x x x x x x NY NC ND x x x x x x x x x OH OK x x x x x x x x x OR PA x x x x x x x RI SC x x x x x x x x x SD TN x x x x x x x TX x x x x x x UT VT x x x x x x x x x VA x x x x WA x x x x x x WI WV x x x x x x x x WY DC x x x x x x x GM PR x x x VI Total 18 34 38 26 30 30 17 Percent 33 63 70 48 56 56 31 Gender Mix. Gender mix referred to the percentage of male and female students who were enrolled in these postsecondary vocational education programs. Slightly less than one-third (31%) of the states indicated that their system was using this type of measure. A. es o ssessme echn es ademic a d Ot s q ce States reported using a wide variety of types of assessment instruments, procedures and processes for measuring academic and other types of performance. Four general categories were identified for academic performance, these included: state developed assessment techniques, locally developed or selected techniques, nationally recognized measures, and other assessment techniques. Five general categories of assessment techniques were identified for use with other performance. Types of measures used for other performance included: state developed, locally developed or selected, nationally recognized, occupational certification or licensure, and other assessment techniques. Locally selected or developed assessment techniqqes included all those left to the discretion of the local education agency. In some states, the local agency develops the measuring instrument and submits it for state approval. In other states, the state publishes a list of suggested assessment instruments from which the local agency may select. In still other states, the local may choose from any available nationally recognized, state developed or locally developed assessment instrument with no state approval. Nationally recognized instruments include standardized tests of academic or occupational performance. These may also include nationally recognized checklists and competency guides. Occupational certification or licensure includes assessment procedures adopted by national or state occupational boards for certifying competency in specific occupations. Examples include nursing boards, cosmetology boards, and automotive certification agencies. The other category includes instruments to measure student performance in coursework. These include successful completion of courses, grades, and the use of student portfolios. The inforaation presented in Table 5 identifies the number of states using the various assessment techniques used in measuring basic and advanced academic performance. The most frequently used techniques for assessing academic performance for secondary vocational education were state developed. More than one half of the states reporting the use of measures of reading, language and math used state developed assessment techniques, and about three- fourths of the states reporting the use of a science measure were using state developed techniques. 24 The most frequently used assessment techniques for assessing academic performance for postsecondary programs were locally selected or developed. About fifty percent of the states reporting using reading, language, and math as measures of academic performance were using locally developed or selected assessment techniques. Almost 90% of those ,:epsrting the use of a measure of science used locally developed msures for assessing performance. Table 6 summarizes the assessment techniques used in measuring competency attainment and workskill attainment. The most popular method for assessing competency attainment in both secondary and postsecondary programs reported by the states were locally selected and developed assessment techniques. Approximately two-thirds of the states reporting the use of a competency attainment measure for both secondary and postsecondary programs were using locally selected or developed methods. Most frequently used among the methods for assessing workskill attainment were locally selected or developed techniques. Among states reporting a measure of workskill attainment, about 70% were using locally selected methods. Slightly over half (55%) of the states reporting workskill attainment as a performance measure were using locally selected assessment techniques. Information concerning the specific types of assessment instruments and procedures for measuring academic and other performance of secondary and postsecondary vocational programs is contained in Appendix C (Tables 7-13). State developed assessment techniques included statewide testing instruments, competency checklists, common curriculum requirements or high school proficiency exams used across the state to assess the academic or other performance of vocational students. Some of these instruments are used by the states in assessing the performance of all secondary students in the state, while others are intended for use with vocational students only. CONCLUSIONS Based on the findings presented in the previous section, a number of conclusions have been developed. These conclusions are presented in this section. In nearly every state, systems of core standards and measures of performance for secondary and postsecondary education had been developed and implemented in line with the requirements of Section 115 of the Carl D. Perkins Vocational and Applied Technology Education Act of 1990. Only two states had not yet had their system approved and were unable to provide the researchers with their approved measures for secondary vocational education. At the postsecondary level, information was not available from 2 states. 2 5 Table 5 Assessment Techniques for Basic and Advanced Academic Performance Assessment Technique States Using Post- Secondary secondary N %1 %1 Reading State.Developed Locally Selected/Developed Nationally Recognized Other 23 12 15 15 54 28 35 35 3 12 15 57 9 35 12 46 Language State Developed 23 58 3 10 Locally Selected/Developed 13 33 14 48 Nationally Recognized 11 28 6 20 Other 12 30 15 52 Math State Developed 25 54 3 10 Locally Selected/Developed 14 30 15 50 Nationally Recognized 13 28 7 23 Other 14 30 14 47 Science State Developed 14 74 0 0 Locally Selected/Developed 10 53 7 88 Nationally Recognized 3 15 0 0 Other 7 37 4 50 Other State Developed 7 29 2 7 Locally Selected/Developed 4 17 8 30 Nationally Recognized 0 0 0 0 Other 3 13 9 33 Note: 'Percent of the total number of states reporting a performance measure in this category. 26 Table 6 Assessment Techniaues for Competency and Workskill Attainment Assessment Technique States Using Post- Secondary secondary N %1 %1 Competency Attainment State Developed 7 29 2 11 Local Selected/Developed 16 67 12 67 Nationally Recognized 5 21 5 28 Occupational Certification or Licensure 2 8 0 0 Other 7 29 7 39 Workskill Attainment State Developed 11 28 4 12 Local Selected/Developed 27 69 19 55 Nationally Recognized 2 51 1 3 Occupational Certification or Licensure 5 13 7 21 Other 8 20 8 24 Note: 'Percent of the total number of states reporting a performance measure in this category. 27 State systems tended to have more measures approved for secondary vocational education than they did for postsecondary vocational education programs. The avrerage number of performance standards that had been approved for scondary vocational education programs was 10. An average of 8 performance measures had been approved for postsecondary vocational education programs. It appeared that performance measures listed in the Carl D. Perkins Vocational and Applied Technology Education Act of 1990 were generally accepted by the states. Of the measures of performance included in the Carl D. Perkins Vocational and Applied Technology Education Act of 1990, only competency attainment had been approved by less than 50% of the states. States have taken the requirement that measures of academic skills be included in their systems seriously. Academic skill measures were approved more often for secondary vocational education programs than they were for postsecondary vocational education programs. Mathematics and reading were the most frequently approved basic and advanced academic skill measure for secondary vocational education programs, lollowed by language measures. Science measures were approved least often as secondary vocational education academic skill measures. For postsecondary vocational education, mathematics and language were the most frequently approved basic and advanced academic measures followed by reading measures. Science measures were also approved least often at the postsecondary education level. States have also responded positively to the requirement that at least one or more measure of performance be included in their system of core standards and measures of performance. At the secondary vocational education level, work skill attainment measures were reported as being approved most often. The next most frequently approved measures dealt with placement of any type, followed by program completion, and high school graduation. For postsecondary vocational education, program completion measures were approved most often. The second and third most often approved measures were work skill attainment and placement of any type, respectively. About one-half of the measures for both secondary and postsecondary vocational education explicitly addressed the extent to which they were serving special populations. In some cases, it was not obvious as to whether special population measures could be obtained from the state's records or not. Specific measures related to the gender mix of individuals served by vocatiorll education were not widely used by the states. Approximately one-third of the states had approved measures of gender mix for both secondary and postsecondary vocational education programs. 28 4t Attempts at standardizing performance standards and measures across the nation will prove challenging due to tile diverse nature of the approved measures and standards currently in use in the states. Also, there is great variability among the types of assessment techniques being used. Finally, the states have left much of the authority for assessing performance to the local agency. RECOMMENDATIONS The requirement that states develop a system of standards and measures of performance for secondary and postsecondary programs is new in vocational education legislation. This initial experience should be monitored in order to see how future policy initiatives related to these measures and standards might be improved. The following specific recommendations are offered: 1. Information should be collected regarding the rationale states used in selecting their standards and measures. 2. The strengths and weaknesses of the various measures of performance should be assessed in order to determine their relevancy for future use. Additionally, the validity and reliability of these measures should be established. 3. States should critically review their approved system of standards and measures of performance in order to identify the major facilitators and barriers they have encountered in its development and implementation. Information also should be collected regarding how states offered incentives and made adjustments to encourage service to targeted populations. 4. Ef forts should be made to determine how the state approved measures and standards compare with business and industry standards. 5. Further research on the state systems of measures and standards should be conducted. Specifically, additional analysis of the level and type of standards employed by the states would be useful. Vocational education has been concerned with evaluation for many years. At the national level, vocational education legislation has included emphasis on evaluation since the passage of the Vocational Education Act of 1963. The inclusion of requirements for a state system of performance measures and standards in the Carl D. Perkins Vocational and Applied Technology Education Act of 1990 continued this emphasis. This study examined those systems in early 1993. These performance measures and standards should continue to be monitored as they are further refined and developed. This information is needed to provide information for impl:oving vocational education programs and guide future policy initiatives. 29 REFERENCES Apling, R. N. (1989). Vocational education performance standards. Washington DC: Congressional Research Service. Butler, E. P. (1988). The search for the bottom line in vocational partnership act? Washington DC: National Assessment of Vocational Education. Carl D. Perkins Vocational Education Act. Public Law 98-524 -October 13, 1984-98th Congress: An Act to Amend the Vocational Education Act of 1963. Washington, DC: W.S. Government Printing Office. Carl D. Perkins Vocational and Applied Technology Education Act Amendments of 1990 (Public Law 100-392) Section 115. (1990). Washington, D.C.: U.S. Congress. Commission on Skills of the American Workforce. (1990). America's choice: High skills or low wages! Rochester, NY: National Center for Education and the Economy. Dickinson, K. P. & West, R. W. (1988). Evaluation of the effects of JTPA performance standards on clients, services, and costs. Washington DC: National Commission for Employment Policy. Ewell, P.T. (1988). Outcomes, assessment, and academic improvement: in search of usable knowledge. In J.C. Smart (Ed.). Higher education: Handbook of theory and research: Vol. 1. New York: Agathon Press. Federal Register. (August 14, 1992). State vocational and applied technology programs and national discretionary programs of vocational education; final rule. (34 CFR Part 400, et al.) Washington DC: U.S. Department of Education. Frazier, F. (1991). Amending the job training partnership act: Inadequate oversight among issues that need to be addressed. Washington DC: Government Accounting Office. Henry, G.T., McTaggert, M.J., & McMillian, J.H. (1992). Establishing Benchmarks For Outcome Indicators. Evaluation Review. 16(2), 131-150. Hill, P. T. & Bonan, J. J. (1991). Decentralization and accountability in public education. Santa Monica, CA: RAND. Hoachlander, E. G. & Rahn, M. L. (1992). Performance measures and standards For vocational education: 1991 survey_results. Berkeley: National Center for Research in Vocational Education. 31 Jennings, J. F. (1991). Congressional intent. Vocational Education Journal. (Feb. 1991) 18-19. McCaslin, N. L. (1990). A Framework for evaluating local vocational education programs. (Information Series No. 344). Columbus: Eric Clearinghouse on Adult, Career, and Vocational Education. (ERIC Document Reproduction Service No. ED327738) McCaslin, N. L. (1992). Outcomes Assessment. In D.D. Bragg (Ed.). Alternative approaches to outcomes arsessment. Berkeley: National Center for Research in VocatA.onal Education. National Commission on Excellence in Education. (1983). A nation at risk. Washington, DC: U.S. Department of Education. Office of Technology Assessment. (1989). Performance standards for secondary school vocational education. Washington, DC: Author. (ERIC No. ED 313 591) Oklahoma Department of Vocational and 2echnicai Education. (1992). Measuring learning and competency gains: The pretest/posttest approach. Stillwater, Ok: Author. Secretary's Commission on Achieving Necessary Skills. (1991). What work requires of schools: A SCANS report for America 2000. Washington, DC: U.S. Department of Labor. Secretary's Commission on Achieving Necessary Skills. (1992). Learnin a livin : A blue rint for hi h erformance. Washington, DC: U.S. Department of Labor. Special Study Panel on Education Indicators. (1991). Education counts. Washington, DC: National Center for Education Statistics. United States Department of Education. (1991). America 2000: An education strategy. Washington, DC: Author. Wentling, T. L. (1980). Evaluating occupational education and training programs. Boston: Allyn and Bacon. White, B. (1990). Vocational education effectiveness indicators Honolulu: University of Hawaii. (ERIC Document Reproduction Service No. ED 324 468) Wilcox, J. (1991). The Perkins act at a glance. Vocational Education Journal. (Feb. 1991) 16-17. William T. Grant Foundation Commission on Work, Family and Citizenship. (1988). The forgotten half: pathways to success for America's youth and young families. Washington, DC: Author. 32 IN MI UM MOM an ON IS NM OM OM NO 11 November 25, 1992 Dear 2 The implementation of the Carl D. Perkins Act of 1990 has required states to change how their students are being assessed and programs are being evaluated. The statewide system of performance measures and standards was to be established by September, 1992. Many states also have revised their procedures for assessing program quality. Information on these recently approved standards and measures and the criteria used to assess program quality have not been summarized on a national basis. We are gathering this information from each state. We need your help in providing the following: A list of the performance standards and measurzr for secondary and postsecondary vocational education that have been approved by your state board (Section 115). The procedures you will use to make local modifications based on economic, geographic, or demographic factors, or the characteristics of the population to be served (Section 115). Manuals, procedures, and criteria used to assess local program quality (Section 116). The name of the individual(s) responsible for secondary and/or postsecondary vocational education evaluathn in each state. A one-page form has been included to provide us with this information. This information would be especially valuable in developing high quality vocational education programs and contributing to the study of vocational education's effectiveness. It would also be important in preparing new policies and legislation for vocational education. We need to receive this information by December 1, 1992. Thank you for your assistance in providing us with this information. We will provide you with a summary of the information we collect. Sincerely, N. L. McCaslin Associate Professor William S. Headley Graduate Research Associate 35 IIIII Maliell IS OR INA Ile 1111 MO MO IS OE MS MN March 11, 1993 1- Dear 2- Thank you for sending us the information on the statewide system of measures and standards required by The Carl D. Perkins Act of 1990. In oer to ensure the accuracy of the information we will be repotting, we are requesting that you review our findings for your state. Please check the summary sheet for measures adopted for use in secondary and postsecondary programs. Feel free to make any necessary additions, deletions, or corrections. If the summary is correct, simply write "OK" on the summary sheet and return it to us. The enclosed instruction sheet will provide details to assist in the review of these materials. We would appreciate any comments concerning this summary. A stamped, addressed envelope is provided for your use in returning the summary forms. The final summary of the findings for all the states will be published and made available to you as soon as we have received the returned and corrected summary sheets. Therefore, we need to have this information by March 29, 1993. Don't hesitate to call if you have any questions. Thank you for your assistance in reviewing these materials. Sincerely, N. L. McCaslin Associate Professor William S. Headley Graduate Research Associate 39 CD Measures Adopted for Use with Secondary Programs 3" Basic Academic Skill Advanced Academic Skill Reading Language Math Science Other Reading Language Math Science Other Other Academic Measures Determined Social Course Critical Problem Personal by program Studies Work Thinking Solving Qualities Competency Work Skill Attainment Retention/Completion Placement Related Any II Special Populations Other Enrollment Retention Placement GPA Gender Mix Enrollment Collaboration Student Satisfaction Other Career Plan Guidance Program Prior Earnings Services Features Information 1111111111111111411111111111-111111111-11111111111111111111111110111111111111111-11111 Instruction Sheet 1. Read over the attached summary sheet. An "x" has been placed under the headings indicating your state has a measure in this area. Blank spaces indicate no measure in this area. 2. Please cross off any incorrect "x" and add any that should be present. 3. Some b4..ates are phasing in measures, the measure is listed on the summary sheet even if it is to be phased in at a later date. 4. The notes below will help in the interpretation of the headings. a. If your state shows an * in the academic skill area, we interpreted your system as using the same set of measures for both basic and advanced academic skill. b. Determined by program means that each vocational program area (e.g. agriculture, marketing, business) in the state will determine the specific academic measures to be used. c. Social studies includes citizenship, American history, etc. d. Under Placement, both Related and Any include military service and further training or education. e. Under Special Populations and Other, Enrollment refers to measures of numbers of students in programs or ratios of students compared to other groups of students, etc. f. Program Features refers to measures of curriculum, teacher, or other programmatic measures. 5. If the summary is correct, simply write "OK" on the summary sheet and return it to us. 6. Please return the summary forms in the enclosed stamped, addressed envelope. 7. Your comments are appreciated. If you need additional clarification or desire additional information, call Scot Headley at 614-292-6321. 41 5 1111110111-1111MMMISMIII Table 7 Techniques for Assessing Reading Performance Assessment Technique State Developed States Using Post- Secondary secondary 23 3 State high school proficiency exam 10 Other state developed assessment 13 3 Local Selected or Developed 15_ Nationally Recognized 9 Iowa Test of Basic Skills 3 Test of Adult Proficiency 3 ASSET 1 Test of Adult Basic Education 2 6 California Test of Basic Skills 1 California Achievement Test 1 Metropolitan Achievement Test 1 Stanford Achievement Test 1 1 Stanford 8 3R battery 1 ACT Work Keys 1 Gates-MacGinite Reading Test 1 VTECS item banks 1 Other 15 12. Course completion 9 9 Portfolios 3 Grade advancement 1 GPA 2 GED criteria 2 1 45 53 Table 8 Techniques for Assessing Language Performance Assessment Technique State Developed States Using Post- Secondary secondary a State high school proficiency exam 9 Other state developed assessment 14 3 Local Selected or Developed 13 Nationally Recognized 11 6 Iowa Test of Basic Skills 2 Test of Adult Proficiency 1 ASSET 1 Test of Adult Basic Education 2 5 California Test of Basic Skills 1 Metropolitan Achievement Test 1 Stanford Achievement Test 1 Stanford 8 3R battery 1 ACT Work Keys 1 VTECS item banks 1 Other 12 15 GPA 1 Course completion 8 11 Portfolios 3 Institutional requirements 1 GED criteria 1 1 Student opinion survey 1 46 Table 9 Techniques for Assessing Math Performance Assessment Techniaue States Using Post- Secondary secondary State Developed State high school proficiency exam 3 9 State developed assessment 16 3 Local Selected or developed Nationally Recognized Iowa Test of Basic Skills Test of Adult Proficiency ASSET Test of Adult Basic Education California Test of Basic Skills Metropolitan Achievement Test Stanford Achievement Test Stanford 8 3R battery ACT Work Keys VTECS item banks 3 2 1 2 5 1 1 1 1 1 1 1 Other lA Course completion 10 10 GED 1 Portfolios 3 Institutional requirements 1 Grade advancement 1 GPA Student opinion survey 1 47 !7i Table 10 11 Techniques for Assessing Science Performance States Using Post- Assessment Technique Secondary secondary State Developed 11 0 State high school proficiency exam 7 State developed assessment 7 Local selected or developed la 7 Nationally Recognized 2 1 California Test of Basic Skills 1 Stanford Achievement Test 1 IIIowa Test of Basic Skills 1 Other 7 4 Course completion 6 2 GED criteria 1 Portfolios 1 Institutional requirements 1 II II II 11 11 II II Table 11 I Techniques for Assessing Other Academic Performance Assessment Technique States Using Post- Secondary secondary State Developed 7 exam 5 a State high school proficiency State developed assessment 2 2 Local Developed or Selected 4 8 Other 3 9 Course completion 3 5 GED criteria 1 GPA 3 II II II II II 1/ 48 1 b Table 12 Techniques for Assessing Competency Attainment Assessment Technique State Developed Local Developed or Selected Nationally Recognized States Using Post- Secondary secondary 7 16. 5 2 12 5 Jobs for American Graduates Test 1 1 Workplace Readiness Assessment 2 2 Work Keys ACT 1 1 Youthwork Instrument 1 1 Occupational Licensure or Certification Other 2 0 Completion of vocational coursework 6 6 Portfolios 1 1 49 Table 13 Techniques for Assessing Workskill Attainment 11 States Using Post- 11 Assessment Technique Secondary secondary State Developed 11 6 5 22 2 4 3 1 12 1 State developed tests State developed checklist Local Developed or Selected Nationally Recognized Measures VTECs materials 1 NOCTI Exams 1 1 Occupational Licensure and Certification 5 7 Other Measures 8 8 Portfolios 2 Degree 1 Completion of coursework/program 5 3 GPA in vocational courses 1 3 Student opinion survey 1 II II II II 11 II II 50 1 IIIIIIMMINMIMIMINIMI , i