USVI Public Records

Document text

ED 107 710 DOCUMENT RESUME . 95 TM 004 537 AUTHOR Womer, Frank B.; Lehmann, IrvinJ. TITLE 1973 Assessment Workshops: Final Report. SPONS AGENCY Education Commission of the States, Denver, Colo. National Assessment of Educational Progress.; Office of Education (DREW), Washington, D.C. PUB DATE 15 Aug 73 GRANT OEG-0-9-08771-2468 NOTE 59p. EDPS PRICE MF-$0.76 HC-$3.32 PLUS POSTAGE DESCRIPTORS Educational Accountability; Educational Administration; *Educational Assessment; Educational Needs; *Participant Satisfaction; Program Evaluation; *State Departments of Education; State Programs; Surveys; *Workshops ABSTRACT Three 3-day assessment workshops were held in Boulder, Colorado from June 19-29, for personnel in the assessment field from state departments of education. Seventy-six participants from 35 states, Puerto Rico, the Virgin Islands and the District of Columbia attended. Two of the three workshops concentrated on National Assessment as one model for large-scale assessments. Presentations were made by National staff members. The third workshop was held between the other two. The F:ogram for this workshop concentrated on seven different state assessments, a short up-dating on National Assessment, and small group discussion sessions. Presentations were made by state directors of assessment, contractors representatives, and selected National Assessment staff. Evaluations of the workshops were quite positive. In all three workshops over 90 percent of the responses indicated a desire to attend a follow-up workshop in 1974. Most questions elicited very favorable reactions. The major suggestions for improvement were for more discussion time and better speakers. The things liked best were the opportunity to interact with other assessment persons and to hear about other programs. (Author/DEP) S U S DEPARTMENT OF HEALTH, EDUCATION & WELFARE NATIONAL INSTITUTE OF EOUCATION THIS DOCUMENT HAS BEEN REPRO OUCEO EXACTLY AS RECEIVED FROM THE PERSON OR ORGANISATION ORIGIN AT,P4t. IT POINTS OT vIEW OR OPINIONS STATIC/ DO NOT NECESSARILY LEPNE SENT oT TiCsAL NATIONAL INSTITUTE Ot EOUCATION POSITION OR PO( ICY FINAL REPORT .....,. 1973 ASSESSMENT WORKSHOPS Frank B. Womer Irvin J. Lehmann Sponsored by the National Assessment of Educational Progress Education Commission of the States Denver, Colorado August 15, 1973 I I CD CD 3 FINAL REPORT 1973 ASSESSMENT WORKSHOPS Prepared by Frank B. Womer, Professor of Education Bureau of State Services University of Michigan, Ann Arbor and Irvin J. Lehmann, Professor of Evaluation Services Michigan State University, East Lansing Sponsored by the National Assessment of Educational Progress James A. Hazlett, Administrative Director J. Stanley Ahmann, Staff Director Education Commission of the States Denver, Colorado Wendell H. Pierce, Executive Director This publication was prepared pursuant to Grant No. 0EG-0-9.08771-2468 with the Office of Education, U S. Department of Health, Education, and Welfare Grantees undertaking such projects under government sponsoiship are encouraged to express freely their professional judgment in the conduct of the project. Points of view or opinions stated do not, therefore, necessarily represent official Office of Education position or policy. 4 TABLE OF CONTENTS ABSTRACT. . ACKNOWLEDGMENTS . INTRODUCTION . . Page . iii . iv-v vi WORKSHOP PURPOSES . . . 1 WORKSHOP PLAN . . . 3 WORKSHOP I SCHEDULE . . 5 WORKSHOP II SCHEDULE . . . 7 WORKSHOP III SCHEDULE . . . 10 EVALUATION OF WORKSHOP EXPERIENCES WORKSHOPS I AND III. . . . 12 EVALUATION OF WORKSHOP EXPERIENCES WORKSHOP II. . . 29 WORKSHOP STAFF . 37 WORKSHOP PARTICIPANTS . . . 39 EVALUATION FOR WORKSHOPS I AND III. . 45 EVALUATION FOR WORKSHOP II . . . 47 ii J 1 6 s ASSESSMENT WORKSHOPS ABSTRACT Three 3-day assessment workshops were held in Boulder, Colorado from June 19-29, for personnel in the assessment field from state departments of education. Seventy-six participants from 35 states, Puerto Rico, the Virgin Islands and the District of Columbia attended. Two of the three workshops concentrated on National Assessment as one model for large-scale assessments. These were held at the beginning and the end of the nine days. These workshops were for personnel who had not attended a similar workshop in either 1971 or 1972. Presentations were made by National Assessment staff members. Attendance was 25 and 17. The third workshop was held between the other two. Attendees were personnel who had attended a National Assessment workshop before, as well as any of the attendees at the other two workshops who cared to stay. The program for this work- shop concentrated on seven different state assessments, a short up-dating on National Assessment, and small group discussion sessions. Presentations were made by state directors of assessment, contractors representatives, and selected National Assess- ment staff. Attendance was 53. Evaluations of the workshops were quite positive. In all three workshops over 90 percent of the responses indicated a desire to attend a follow-up workshop in 1974. Most questions elicited very favorable reactions. The major suggestions for improve- ment were for more discussion time and better speakers. The things liked best were the opportunity to interact with other assessment persons and to hear about other programs. iii ACK NOWLE DG ME NTS The success of these three workshops was dependent in large measure on the various presenters and discussants. We were fortunate in having a goodly number of excellent presenters, both from the National Assessment staff and from various to departments of education and contracting agencies. Those who presented various state assessment plans were Lorrie Shepard, California; Dale Carlson, California; Jim Fisher, Maryland; Roberta Keiter, Maryland; Jim Impara, Florida; Keith Cruse, Texas; John Adams, Minnesota; Gordon Ascher, New Jersey; and Dan Schooley, Michigan. Contractors' representatives were Dave Bayless, Research Triangle Institute; Bill Schabacker, Educational Testing Service; and Fred Finch, CTB/McGraw-Hill. Three presenters on the topic of dissemination made major contri- butions, Louis Rubin, professor, University of Illinois; Erwin Bettinghaus, professor, Michigan State University; and Gerald Miller, professor, Michigan State University. The Cooperative Accountability Project (CAP) deserves thanks for providing the honoraria for Dr. Bettinghaus and Dr. Miller, who have just authored a new CAP publication on dissemination. Within the National Assessment staff we are particular:y indebted to Todd Rogers for his very lucid explanations and for the amount of time he gave us. Linda Mughrabi was invaluable in handling the myriad details of iv 8 conference operations. And Jack Schmidt, our National Assessment colleague in directing the workshop, must receive our spegial thanks for both his planning and operational skill. It was a real pleasure to work with all of these professional colleagues. v 9 Frank B. Womer Irvin J. Lehmann INTRODUCTION This third series of assessment workshops was decided on almost at the conclusion of the 1972 workshops. It was obvious then that state directors of assessment and-their staffs were anxious to continue the brief but well-established tradition of the workshops. A six-month follow-up confirmed that decision. Initial plans called for two workshops, one to be devoted to National Assessment and one to state assessments. A mid-winter survey- of-interest demonstrated that the number of "new" participants would be too large to handle effectively in a single workshop. A decision was made to offer two identical National Assessment workshops, one before and one after the state assessment workshop for previous workshop participants. Program development was related as much as possible to the evaluations received from attendees in 1972, within the usual constraints of staffing and time. A slight shift in local locale was necessary because of crowded conference conditions at the University of Colorado. vi 10 WORKSHOP PURPOSES In the last two years the trend to state assessments has moved from a position of "interest" to a position of "implementation." At least a third of the states have gathered assessment data in one form or another and others are very close to that point. Almost all states are engaged in some phase of planning for or implementing an assessment. In this climate it is imperative that information about large-scale assessments should be shared between and among the various states and the National Assessment project. A workshop to bring together the directors of assessment and members of their staff is a logical vehicle. National Assessment, as a nationwide project, is in a position of natural leadership to handle this need. Previous workshops had concentrated on National Assessment as one model of assessment. For persons who had attended one of those workshops it was more logical to plan a program that would emphasize planning and activities within several states that are implementing assess- ment in a variety of ways. Any cursory examination of state plans shows one that, while no two are identical, there are certain common elements such as "use of released National Assessment exercises" by some, or "use of a regression model of analysis of results" by some, or "problems of reporting to lay publics" by others. Thus, one purpose of the 1973 workshops was to provide a sharing of state assessm,nt plans. A second major purpose of the workshops was to continue the activities of the previous two years, an indepth review of National Assess- ment policies and procedures, for interested state department personnel who had not attended a previous workshop. This was felt to be important for two reasons: (1) some states had not sent anyone to a previous work- shop, and (2) many states had had changes in and/or additions to their assessment staff. Clearly we are past the stage of deciding whether to do state 1 11 assessments or not. We are in a rapidly expanding period of implemen- tation. The ultimate success of any of these programs, as well as the National Assessment project, will depend upon the quality of the program and the relevance of the results. A sharing of information among the states and National Assessment certainly is one essential ingredient for ultimate success. 2 WORKSHOP PLAN The basic ingredients for program planning were: (1) previous experience in 1971 and 1972; (2) six-month evaluations of workshop participants in 1972; (3) a pre-registration of participants; and (4) knowledge of existing state assessment plans. In addition, the work of the Cooperative Accountability Project in contracting for monographs on dissemination of assessment results and developing large-scale assessments contributed to the planning. Rather large pre-registrations for the National Assessment Model workshop led to a decision to split the group into two (hereafter called Workshop I and Workshcp III). Identical programs were planned for the two groups (persons who had not participated in the similar workshops in 1971 or 1972). These workshops were condensed into 3-1/2 and 3 days respectively, by reducing the time devoted to certain topics, by eliminating specific discussions of state assessments, * and by utilizing evening sessions. While attendees did not divide themselves equally between Workshops I and III, the division did permit greater flexibility of discussions. Attendance was 25 and 17 respectively. 1973 Workshop Schedule June 19 20 21 22 23 24 25 26 27 28 29 30 Workshop IX X X X weekend Workshop II X X X Workshop III X X X X *Attendees interested in specific state assessments were invited to attend Workshop II also. 3 13 Workshop II was planned to concentrate primarily on state assessment programs, except for one-half day devoted to "%la's New" in National Assessment. One full day was scheduled for presentation and discussion of four topic areas of assessment, as each area is being handled in four different states -- Calfornia, Florida, Maryland, and Texas. Another day was planned to discuss how three different states used contracting agencies to handle a specific phase of their assess- ment program. The state-contractor combinations were Minnesota-Research Triangle Institute; New Jersey-Educational Testing Service; and Michigan-CTB/ McGraw-Hill. A third day was planned in small group discussion sessions on topics chosen by conference participants, and individual consultation with National Assessment staff. Attendance at Workshop II was 53 (including 18 persons who also attended I or III). Three outside speakers were engaged to discuss the general topic of dissemi- nation of educational information. Drs. Gerald Miller and Erwin Bettinghaus, Professors of Communication at Michigan State University, spoke at Workshops I and III respectively. Dr. Louis Rubin, Professor at Large and Communications Specialist, University of Illinois, spoke at a conference dinner during Workshop II. * * * * The location of the workshop was at the Royal Inn, adjacent to the University of Colorado campus in Bouldar, Colorado. Conference arrangements were made through the University, but meeting rooms were more convenient at the Inn. Some participants were housed in a University dormitory; others in the Royal and Holiday Inns. The University Club was used for the conference dinner. 4 14 WORKSHOP I SCHEDULE June 19-22, 1973 June 19, 1:00 - 3:00 Conference Registration 3:00 - 4:00 Introductory Remarks: Chairman Iry Lehmann Welcome: James Hazlett, Administrative Director, NAEP Why Assess?: Chairman Frank Womer NAEP Slide Show: Jack Schmidt, State Assessment 4:00 - 5:30 Developing Goals and Objectives: Hal Wilson June 20 9:00 - 10:00 a. m. Developing NAEP Objectives: Scott Newcomb 10:00 - 10:30 Break 10:30 - 12:00 Developing Assessment Questions, Items, Exer- cises: Hal Wilson 12:00 - 1:30p. m. Lunch 1:30 - 3:00 NAEP Exercise Development: Nancy Burton 3:00 - 3:30 Break 3:30 - 4:30 Principles of Sampling: Don Searls 7:00 - 9:00 NAEP Sampling: Todd Rogers June 21 9:00 - 10:00 a. m. Discussion of Sampling: Don Searls, Todd Rogers 10:00 - 10:30 Break 10:30 - 12:00 NAEP Data Collection and Scoring: Roger Talle, Sue Oldefendt 12:00 - 1:30 p. m. Lunch 1:30 - 3:00 NAEP Data Analyses: Sue Sherman 3:00 - 3:30 Break 3:30 - 4:30 NAEP Data Analysis and Report Preparation: Charlie Gadway June 22 9:00 - 10:30 a. m. Discussion of Data Analysis and Report Prepara- tion: Rex Brown 10:00 - 10:30 Break 10:30 - 12:00 Principles of Dissemination Introduction: Tom Fisher, Michigan Department of Education Presentation: "Dissemination of Assessment Information" - Gerald R. Miller, Professor, Michigan State University (this presentation sponsored jointly by NAEP/ Cooperative Accountability Project) 12:00 1:30 p.m. Lunch 1:30 3:30 NAEP Utilization/Applications/Interpretation: Ken Seaman and U/A staff Discussion of Utilization/Applications/Interpretation Services Available from Utilization/Applications: Jack Schmidt and U/A staff 3:30 - 4:00 4:00 - 4:30 Break Windup and Evaluation: Womer and Lehmann 16 6 WORKSHOP II SCHEDULE June 25-27, 1973 June 25 Directors of Assessment from four different states, California, Texas, Florida and Maryland discussed alternative approaches to assessment in purposes and materials development, data collection, analysis and dissemination. 8:00 - 9:00 a.m. 9:00 - 10:30 10:30 - 11:00 11:00 - 12:00 12:00 - 1:30 1:30 - 2:45 Conference Registration Vurposes, Objectives, Exercise Development Chairman: Frank Womer Participants: California: Dale Carlson, Lorrie Sr .paid Florida: Jim Impara Maryland: Jim Fisher, Roberta Keiter Texas: Keith Cruse Break Sampling and Data Collection Chairman: Iry Lehmann Participants: Dale Carlson Lorrie Shepard -Jim Impara Jim Fisher Roberta Keiter Keith Cruse Lunch Analysis and Report Writing Chairman: Iry Lehmann Participants: Dale Carlson Jim Impara Lorrie Shepard Jim Fisher Roberta Keiter Keith Cruse 2:45 - 3:15 Break 17 7 3:15 - 4:30 6:00 - 9:00 June 26 9:00 - 10:30 a.m. 10:30 - 11:00 11:00 - 12:00 12:00 - 1:30 p.m. Interpretation/Dissemination/Utilization/ Applications Chairman: Frank Womer Participants: Dale Carlson Lorrie Shepard Jim Impara: Jim Fisher Roberta Keiter Keith Cruse National Assessment Dinner Chairman: Ken Seaman, Director of Utilization/Applications, NAE Speaker: Louis Rubin, Communications Specialist, University of Illinois "Dissemination of Educational Perform- ance Information" ill What's New at NAEP? (Objectives, Exercise Development, Sampling, Data Collection): George Johnson, Associate Staff Director, NAEP Break What's New at NAEP? (Analysis Reporting, Dissemination): George Johnson Lunch Afternoon Sessions: A team from each of three states discussed one aspect of assessment in that state. Each team consisted of a State Director of Assessment and a representative from a contracting agency that worked with that state. 1:30 - 2:45 Development of a State Assessment Plan in Minnesota Chairman: Fank Womer Participants: John Adams, Minnesota Dave Bayless, Research Triangle Institute John Pyecha, Research Triangle .T.-r Institute 18 8 2:45 - 3:15 3:15 - 4:30 Break Development of Objectives and Exercises in New Jersey Chairman: Iry Lehmann Participants: Gordon Ascher, New Jersey William Schabacker, ETS 7:00 - 9:00 Development and Tryouts of Exercises in Michigan Chairman: Jack Schmidt, NAEP Participants: Dan Schooley, Michigan Fred Finch, CTB/McGraw-Hill June 27 9:00 - 10:00a.m. Small Group Discussions on topics selected by conference participants 10:00 - 10:30 Break 10:30 - 11:30 Repeat of Small Group Discussions 11:30 - 12:00 Windup and Evaluation: Iry Lehmann 2:30 - 4:00 (Optional) Individual consultation at NAEP offices in Denver with NAEP staff 9 19 4 ) June 27 8:00 - 9:00 a. m. 9:00 - 10:00 10:00 10:30 10:30 11:30 11:30 1:00 p.m. 1:00 2:30 2:30 3:00 3:00 4:30 7:30 - 9:00 June 28 8:30 - 9:30a.m. 9:30 - 10:00 10:00 - 12:00 12:00 - 1:30p.m. 1:15 - 2:45 2:45 - 3:00 3:00 - 4:30 4:30 - 5:30 WORKSHOP III SCHEDULE June 27-29, 1973 Conference Registration Introductory Remarks: Chairman Frank Womer Welcome: George Johnson, Associate Staff Director, NAEP .4. Why Assess ?: Chairman Frank Womer NAEP Slide Show: Jack Schmidt, Break Development of Goals and Objectives: Hal Wilson Lunch Development of NAEP Objectives: Scott Newcomb Break Development of Asiessment Questions, Items, Exercises: Hal Wilson NAEP Exercise Development: Nancy Burton Principles of Sampling: Todd Rogers Break NAEP Sampling: Todd Rogers Lunch NAEP Data Collection and Scoring: Roger Talle, Wayne Martin Coffee Principles of Data Analysis and Technical Report Preparations: Ina Mullis, Dave Wright NAEP Data Analysis and Report Preparation: Rex Brown 10 20 June 29 9:00 - 10:30 a.m. Principles of Dissemination Introduction: Dan Schooley, Michigan Department of Education Presentation: "Dissemination of Assessment Information" - Erwin P. Bettinghaus, Professor, Michigan State University (this presentation sponsored jointly by NAEP/ Cooperative Accountability Project) 10:30 - 11:00 Break 11:00 - 12:00 Discussion of Principles of Dissemination: Bettinghaus 12:00 - 1:30 p. m. Lunch 1:30 - 3:30 NAEP Utilization/Applications Interpretation Jack Schmidt and U/A staff 3:30 - 4:00 Coffee 4:00 - 4:30 Windup and Evaluation: Womer and Lehmann 11 EVALUATION OF WORKSHOP EXPERIENCES WORKSHOPS I AND III On the final day of the workshop, the participants were asked to complete an evaluation form. The purpose of this evaluation was two-fold: (1) to provide the workshop staff with some objective evidence concerning the impact of the workshop experience on the participants, and (2) to pro- vide the workshop staff with feedback that could be of value in the planning and conduct of future workshops. The major questions asked in the evaluation form for Workshops I and III were as follows. The evaluation of Workshop II is covered separately in the following section of this report. 1. Relative value of major presentation sessions. 2. General impressions of the workshop experience. 3. Length of workshops. 4. Suggested improvements for future workshops. 5. Most and least valuable workshop experience. 6. Degree of satisfaction with workshop experience. 7. Suggested improvements. Twenty-seven participants were registered for Workshop I. Of these, two only attended one or two sessions and therefore were not considered in determining the population size. Of the 25 eligible respondents, completed evaluation forms were received from 24 participants (96 percent). The results are based on the replies received from these 24 participants. Seventeen participants were registered for Workshop III. Of these, one participant attended about one-half the workshop and two attended only one day. The former was included but not the latter. Responses were received from all the participants. In a few instances, the total number of responses to a particular question will be less than the base number of participants because of failure to respond to that question by one or more persons. 12 Relative Value of Major Presentation Sessions One of the more difficult aspects in the planning of any learning experience is to offer experiences that will be most beneficial to a majority of the Itarners, Plar.ning a workshop is no different. On the basis of evaluations conducted with previous workshop participants, the directors planned and organized a series of speakers and topics within the constraints that the general topic was National Assessment and that most speakers would be from the NAEP staff, The participants were asked to rank, in order of importance, the eight major presentation sessions in relation to their present and possible future work. Using a weighting scheme where 1 = the activity ranked first, the mean ratings and rank order are presented below. Session Mean Rating WS I WS III Rank Order WS I WS III Developing objectives 2.5 3.4 1 2 Sampling 4. 5 2. 0 5 1 Scoring 5.1 5.7 8 8 Report preparation 4.9 4. 9 7 4 Exercise development 3. 4 5. 1 2 5 Data collection 4. 8 5.2 6 6 Data analysis 4. 0 5.4 3. 5 7 Dissemination 4. 0 4. 1 3.5 3 It is evident from the results that there are both similarities and differences between the participants in the two workshops in their ratings of the various content sessions. For example, participants in both workshops rated developing objectives and dissemination in the top three; and rated scoring of least value. In contrast, whereas the Workshop III participants felt that the sampling session was of most value to them, this was not true for Workshop I participants, who ranked sampling fifth. Workshop I par- ticipants ranked exercise development second, whereas it was fifth for Workshop III. 13 -.: 23 Although there may be marked differences in the rank order of a particular session (for example, report preparation) there is little or no difference in the mean rating between some of them. This is an artifact of the ranking procedure and suggests that one should look both at the mean rating and rank order before drawing any conclusions, especially where there is a marked discrepancy in rank order. There is no denying the fact that there is a marked difference between the participants in the two workshops insofar as the sampling and the exercise development are concerned. This may be due, in part, to the differences between the participants in terms of training, experience, and job responsibilities. Also it might be explained in part by differences in the speakers and format employed in the two workshops. Whereas Work- shop I used a team approach for sampling, only a single presenter was used in Workshop III. On the other hand, speakers were identical for exercise development in both workshops. It may be of interest to note that two of the participants did not answer the question, but said "all topics are of great importance in the planning and implementation of an assessment program." Informal comments could lead one to feel that this sentiment was shared by other participants, even though they answered the question. Another point of interest is the greater homogeneity of responses for Workshop I. The mean ratings went from 2. 5 to 5.1, a difference of only 2.6 ranks. For Workshop III, however, the mean ratings went from 2.0 to 5. 7, a difference of 3. 7 ranks. This may have been due to a leveling effect introduced by a larger group of raters, or the interests within Work- shop I may have been more heterogeneous (resulting in less differentiation when rankings were averaged). In Workshop I, developing objectives and exercise development were clearly of greatest intent, whereas no one topic was clearly of least interest. In Workshop III sampling and developing objectives were of greatest interest and scoring of least interest. 14 f 24 General Impressions of Workshop Experience II The reactions from the two workshops are so divergent that it does not seem appropriate to discuss them collectively. Rather, the data are presented separately, with an atte-npt to interpret the variability in results. The data from this question are presented below. Area Workshop I Good Average Poor Workshop III Good Average Poor Overall workshop organization 14 8 2 8 7 0 Workshop content 11 10 2 9 6 0 Speakers/discussion leaders 9 13 1 9 6 0 Workshop materials 17 7 0 10 5 0 Living facilities 20 2 1 9 3 2 Opportunity for group discussion 9 8 7 12 3 0 Opportunity for individual interaction 11 10 3 12 3 0 Availability of workshop co- directors 17 6 0 13 2 0 The major differences between Workshops I and III were in terms of: (1) the time frame (Workshop I was conducted in 3-1/2 days; Workshop III in 3 days); (2) some presenters; and (3) the size of the groups. The most positive reactions (positive in the sense that 70 percent or more of the participants rated the question "good") for the Workshop I participants were living facilities (83%), availability of co-directors (74%), and workshop materials (71%). In contrast, "goodness" reactions for the Workshop III participazits were availability of co-directors (87%), opportunity for discussion (80%), opportunity for individual interaction (80%). Thus, three of the eight questions suggested considerable satisfaction in each workshop. However, only one question -- availability of co-directors -- received this level of satisfaction from the participants of both workshops. Slightly more than one-half of the participants -- 58% and 53% in Workshops I and III respectively rated overall workshop organization as "good." 15 25 The Workshop I participants were more negative in their reactions in comparison to their counterparts in Workshop III. For example, the only question that received any "poor" ratings from Workshop III partici- pants was living facilities (two out of fourteen, o'r 14 percent). For Workshop I six of the questions elicited at least one "poor" response, out of 24 respondents. But for only two of these six questions was the percentage greater than 10 percent of the respondents (3 or more). For "opportunity for individual interaction" there were 12 percent "poor" ratings, and for "opportunity for group discussion" there were 29 percent "poor" ratings. This is very understandable when one considers the size of the two workshops: Workshop I at one time had 27 participants whereas Workshop III at one time had a maximum of 17 participants. Also, the fact that the participants were housed in different locations (more of the Workshop III people lived in the same place), might explain some of the findings. It is interesting to note, however, that of a total of 16 "poor" responses from Workshop I, 9 of them came from three respondents. This suggests that Workshop I may not have met the needs of three of the participants too well. The matter of group dynamics must also be considered when attempting to interpret the markedly different results to the question on opportunity for group discussion. The memory of the directors suggests that roughly comparable amounts of time were spent in group discussion in Workshop I and Workshop III. But it is possible that a more limited number of the participants in Workshop' I actually engaged in discussion and raised ques- tions, and that others, perhaps not by choice, were precluded from doing so because a few used up the available time. Or, it is possible that the sheer size of the group and the tight scheduling prevented everyone from being able to participate in the discussion. These reactions suggest that the size of future introductory workshops be seriously considered by the staff. iv 16 26 The question on quality of speakers/discussion leaders received more "average" ratings than "good" ratings in Workshop I. Certainly the speakers at the workshops differed markedly in their teaching ability and if one had to select a single reaction that was voiced most frequently, it centered about the quality of the speakers. Specific comments were made by six Workshop I participants and by three Workshop III participants. It is difficult to control this problem completely, because not all researchers are automatically good "teachers. " Certainly efforts should be made to exercise some greater control intt he future. Comments made by more than one participant in Workshop I were as follows: "More careful culling of speakers... some are very knowledgeable but poor speakers" (6) "Better coordination among speakers" (3) "Co-directors were attentive to the group's needs" (2) "Too lecture-orientednot a workshop" (2) "NAEP people tended to speak to each other rather than answer the participants' questions" (3) The other comments, made by only a single participant, were con- cerned with such things as physical facilities, the technical nature of the content, living facilities, and the like. The only comment made by more than one person in Workshop III was related to the quality of the speakers used (3). The other comments, made only once, dealt again with physical facilities, evening sessions, and the technical nature of some of the material. In summary, it would appear that the majority of the participants in both Workshops I and III had positive reactions and felt that most of the experiences offered would be or are of value to them. The staff are con- cerned about the average and negative comments related to the speakers and opportunity for discussion and will take them into considerattion in future planning. 17 2:7 Length of Workshop On the basis of remarks made at the workshops in 1971 and 1972, and because of time constraints for some of the NAEP staff who were serving as speakers and resource personnel, it was decided to reduce the length of the workshops from five to three and one-half days and to three days. This was done without omitting any of the essentials about National Assessment covered in previous workshops but by removing one day on state assessments. This did result in long days and some evening sessions. The participants were asked for their reaction to this condensation and the results were as follows: 1. Nineteen participants in Workshop I and 10 participants in Workshop III stated it was satisfactory; 2. Two participants in Workshop III said that the material should have been spread over more days; and, 3. Four participants in Workshop I and one participant in Work- shop III said that less material should be covered in the same number of days. Initially, the co-directors were concerned that the participants would be "worked" too much considering the length of the day sessions and the evening sessions. It would appear that the majority of the participants -- 76 percent and 77 percent in Workshops I and III respectively -- felt that they were not being pushed. However, some of the comments made on the evaluation sheet such as "too scheduled-missed opportunity to interact, " "avoid night meetings, " "although much material was covered in a short time, to reduce the amount of time would be undesirable and to increase the time too much would be impractical, " suggest that some consideration be given to the question of workshop length another year. If it would be considered desirous to permit more group interaction and group discussion, it would appear that the length of the workshop would need to be increased or the enrollment limited to about 15 to 20. 18 28 4 Four of the Workshop I and six of the Workshop III participants made specific comments. Three of the former and one of the latter group said that the sessions were too long and that evening sessions following a very full day are almost too much. One Workshop III participant said that "some of the macerial was repetitious... some could be condensed further... emphasis could be shifted." Most and Least Valuable Workshop 1.:xperience As would be expected with a diversity in the training, experience, and job responsibilities of the workshop participants, those experiences most valuable to some would be less valuable to others. In fact, it would appear that a common frame of reference is the nature of one's job respon- sibility. As some participants stated, "sessions on , because the concerns discussed were more directly related to my present tasks." The most common comments made as "most valuable" experiences were that the workshop provided them with an opportunity: (1) to meet with others who had similar problems; (2) to hear about National Assess- ment and thereby develop a better understanding of that model and see how it could fit (either directly, partially, or completely) into their own state assessment plans; and (3) to discuss points of mutual concern. Possibly the following comment succinctly portrays the feeling expressed by a majority of the participants: "The real strength of this first week in my opinion is the quality of informed personnel whom I have had the good fortune to meet and whom I consequently will pursue to provide profes- sional assistance as necessary." This feeling is one that has been present in all the workshops and jt is evident that further contacts are being made between the various state assessment personnel attending the workshop. On more than one occasion the directors have heard about participants calling each other to discuss problems or to offer suggestions. Specific comments that were made were as follows: "Discussion of the development of objectives and exercises. These two activities are necessary in any assessment pro- gram" (6) 19 29 "Meeting NAEP staff and the opportunity to interact with people from other states" (6) "Familiarization with NAEP" (5) "Lea-,-ting about pitfalls one falls into in an assessment pro- gram ana alternative actions" (4) "Sampling and dissemination sessions were excellent" (4) "Knowing what's going on in other SDE's" (4) "Learning about NAEP sampling techniques" (3) "Principles of dissemination by Miller" (3) Two other comments accurately portray the consensus of feeling of the majority of the participants regardless of which workshop they attended. They are "Sharing of ideas among participants. Each one now has the opportunity to improve what he has been doing in his particular state." "It has enabled me to make a much more accurate assess- ment of the potential value and feasibility of adapting parts of the NAEP model for state assessment in and the care with which such decisions must be approached. The opportunity to exchange ideas and information and the under-. standing of NAEP (which was the objective) were the most valuable aspects of the workshop." It is interesting to note that over 20 percent of the Workshop I participants commented that the most valuable experience was learning about NAEP whereas none of the Workshop III participants mentioned this. The least liked portion of the program by the participants in both workshops were the sessions on report preparation (10 persons). However, judging from the various comments made on the evaluation form as well as from those garnered from informal discussions, it would appear that this may have been due to the manner in which the material was presented and the fact that some presenters were not very dynamic. Next in order of "least" liked were the sessions on sampling. Six respondents indicated 20 sampling, all of them from Workshop I. Three persons from Workshop I felt that "repetition of information already known" was least liked, and three indicated that the "speakers" were least liked. Two persons in Workshop I wrote in "data collection." From Workshop III no response was greater than one, except "report preparation." And three participants in Workshop III said that everything was of value. Some of the individual comments made by the Workshop III participants are: "Friday PM session--too much on NAEP problems. " "Exercise Development" "Utilization/Application" "Report writing--insufficient detail. Maybe optional examples should be discussed." The participants in Workshop I had more to say on this question than their counterparts in Workshop III. The six persons who indicated sampling as least liked in Workshop I indicated that it was because of the technical nature of the material. These participants expressed a desire for a less-technical presentation, and would rather have had a survey of sampling presented than a discussion of how a particular sample is selected. One reaction was "...the sampling discussion (which was too quick, too complicated), although it was directed by the most dynamic speaker." Inasmuch as some of the sessions that were least liked were tech- nical and some were of a non-technical nature, one might conclude that the least liked feelings are related either to specific content or to the pre- senters -- their organization, content, presentation, and the like. Other individual comments made as "least liked" were as follows: "General lecture attitude" "Failure to elaborate on points" "Time precluded questions and discussion" In the main, the majority of participants had more "most" than "least" comments about their workshop experience. Three of the -,a- 21 31 participants in Workshop III and two persons in Workshop I said that all the sessions were good and valuable. The least liked reactions related primarily to the presenters and to the technical nature of some of the material. In addition, some of the "least" comments were related to "lack of the respondent's responsibility for or contact with this activity." The reactions to the questions about most and least liked portions of the program suggest that state department assessment personnel want to know more about what's going on in other assessment programs and want to have the opportunity to meet and discuss problems of mutual con- cern. In a sense, the workshop serves as a cathartic agent and helps to give confidence to those who are working in a very difficult area. Degree of Satisfaction with Workshop Experience There were three specific questions asked in order to provide objective evidence regarding the future of the workshop. One of these was "Would you recommend the workshop for other State Department personnel?" Another was "Would you recommend this workshop for local education agencies (LEA's)?" A third question was "Would you be interested in attending a follow-up workshop next summer?" The responses to these three questions were: Question Number of Participants Responding Workshop I Yes No ? Workshop III Yes No ? Recommendation for other State Department Personnel Recommendation for LEA's Attend follow-up workshop 19 13 20 4 10 1 1 1 3 13 2 12 1 7 2 1 4 1 Over 80 percent of the participants in both workshops (83 percent and 86 percent in Workshop I and III) indicated that they would recommend this type of workshop to their colleagues in the state department. The 22 reasons most often given were: "It would provide them with a better understanding of assess- ment" (t)); "These people would benefit greatly from a basic orientation to NAEP aria thereby might avoid some of the problems NAEP encountered" (5); "Need to know what's going on" (3) "Exchange of ideas" (2); "It would help facilitate communication among staff" (2). A comment echoed by many of the participants was "particularly for 'non-assessment' oriented staff who are important to an overall effort but who do not icnow the why or wherefor of assessment (and program plan- %. ning)." AnNher comment made ay some of the participants either explicitly on the questionnaire or in informal discussions was "no matter how much we caution pe p le about rushing into state assessment without adeqaute i and careful plan. rig, they don't believe us. There are those among us who want to collect data tomorrow -- they need to hear first-hand of the problems involved." For the five participants who answered in the negative, the consensus was that unless the person is actually involved in a program, he need only have a superficial knowledge and understanding which could be obtained from existing documents. One participant felt that the material is too soph.tsticated for most state education agency (SEA) personnel and felt that a less technical information session for the generalist, eurri^ulum specialists, and department managers would be most helpful. There was considerable variation between the two groups insofar as recommending the workshop to LEA's. Slightly more than one-half (54 percent) of Workshop I people but only 13 percent of Workshop III partici- pants answered in the affirmative. And there were more participants in Workshop III (four persons versus one person) who indicated uncertainty. 23 33 Why such a marked difference? The written comments do not contain an explanation. Of these who said they would recommend a NAEP workshop for LEA's, the only comment made by more than one person was "It would broaden their perspective on assessment and help to break dowr. fear" (three persons). The reasons given by the participants who said "no" to the idea of a NAEP workshop for LEA's said such things as: "The NAEP approach is not applicable to LEA's"; "Only good for large LEA's"; "Such sessions should be offered by the SEA to give it a state flavor and gear it more to their policy system." Some of those who responded "no" did comment that there is the need for a workshop dealing with basic questions and issues where utility should be the first consideration. Considering the fact that the workshop was not organized for LEA's the response may be more positive than one had a right to expect. In fact, had the responses been too positive one might make a case that the workshop organized for state department personnel left something to be desired. What is evident is that, as presently constituted and conducted, this workshop is designed specifically for those persons involved in state assessment programs or for the very large LEA's. * Possibly the best indicator of the success of the workshop can be found in looking at the number of participants who indicated that they would like to return for a follow-up workshop. Collectively, 39 participants (82 percent) said they would be interested in attending a follow-up workshop next summer; 3 participants (6 percent) said no; and 4 participants (8 percent) said perhaps. When the results are broken down by workshop, the percentage differences are minor. 24 34 iP Suggested Improvements Although there were open-ended sections following various ques- tions, where the respondents could offer suggestions for improving future workshops, there were two questions specifically designed to do this. They differed slightly in their orientation. One question called for improvements for future meetings whereas the other was entitled final comments, and the respondents were told to offer suggestions for similar and different types of workshops. Since the questions are getting at essentially the same thing -- that is, how to improve on the present model -- the responses will be considered together. Many of the sug- gestions were common to the participants, regardless of the workshop they attended: (1) select the speakers with greater care (eleven responses); (2) avoid evening sessions (two responses); (3) try to have more free time built into the program so that people can collect their thoughts and converse with others (three responses); (4) try to house people in a central location to make interaction more feasible (two responses); (5) keep the workshop size small to permit discussion and interaction (two responses); and (6) spend more time looking at issues, the "why's," the relationship of the results to the improvement of education, and alternative approaches to the NAEP model (three responses). In addition, there were suggestions common to both groups that concerned themselves with physical facilities of the meeting rooms, coffee breaks, and the like which although valuable will not be discussed at this time. These additional points, however, will be considered by the co-directors in planning future workshops. Three respondents suggested that each speaker prepare a brief out- line which is distributed either at the beginning of the workshop or mailed to the participants before the workshop. This might eliminate premature questions and discussion as well as focus the participant's thoughts, if the materials were read prior to attendance. Three, or four individuals suggested that the total group be broken down into smaller groups, each of which would, in an actual working 25 35 situation, carry through a simulated assessment program from the beginning of identifying objectives to the preparation of a final report. Another type of stratification suggested was on the basis of interest areas. Still another type of grouping that was suggested was that there be a few major topical sessions where all participants would be present and then a series of small groups where selected areas would be con- sidered in more depth. These comments suggest that some of the partici- pants would like to attend a "working-through simulated set of experiences" type workshop rather than having the material presented to them in either a lecture or discussion medium. It should be noted that an attempt some- thing along these lines was tried in a previous workshop; but the partici- pants were not too pleased, feeling that they might have missed something by being in one group rather than another. Two people asked whether it would be possible to have a 1-1/2 day information-type session showing NAEP products and discussing NAEP's future plans. Two participants commented upon the marathon-speed in which the workshops were conducted, stating that at times the audience missed some important material either because of the tempo and/or because they were tired. Six participants commended the co-directors and staff on their presentation of an excellent workshop. One participant said, "Though I may have sounded negative at times, I felt good about the workshop and I felt it was worthwhile." Other com- ments were "the workshop has been worthwhile... it was well organized"; "the program content was most appropriate"; and "the directors set a pleasant atmosphere for an open conference which encouraged discussion." In summary, it is evident that the majority of the participants in both Workshops I and III were satisfied with their experience. Conclusions and Recommendations On the basis of the workshop evaluation, as well as from informal discussions with the participants, it is evident that Workshop I and III 26 36 were quite successful. It would appear that most of the participants were pleased with the overall structure, content, materials, facilities, and opportunity for individual interaction. However, about a third of Workshop I were dissatisfied with the opportunity given them to engage in group discussion, and many felt that the speakers were- average. Many of the participants stated that as a result of attending the workshop, they developed a better understanding of National Assessment in general, and their local plans in particular. Many stated that they were able to meet their colleagues from other states and discuss simi- larities, differences, problems, solutions, and the like to a common goal -- that of undertaking a state assessment. This, of course, was a major purpose of the workshop. It was not the purpose of the workshop to promote a particular assessment model (the National Assessment model). But, it was the intent to give participants a detailed view of National Assessment so that they could use the information in developing their own assessment plans. In this way, hopefully, future state and local assessments can profit by National Assessment experiences, and avoid some of the problems experienced by NAEP. As one previous workshop participant stated, "there is no reason to re-invent the wheel when it may only require some slight modification." Based upon the various responses made on both the questionnaire and in informal discussion, we offer the following recommendations: 1. National Assessment should sponsor another workshop for new personnel in state departments undertaking or contemplating conduct of an assessment program. 2. Although three-fourths of the respondents indicated satisfaction with the amount of time for the workshop (three days) the specific suggestions for more discussion (Workshop I), the concern about evening sessions, and a potential desire for program differentiation suggest that serious consideration should be given to the time needed for this workshop. The six-month follow-up survey should include questions that seek more information on this point. f 27 37 3. Consideration should be given to reorganizing the overall workshop so that all participants need not all attend the same sessions. Some participants commented that some of the sessions were too elementary or that they were already familiar with the material or that they had no interest or responsibility in that area. Pos- sibly the general discussions of NAEP methodology should be reduced and participants be given more time to pursue their own specific interests. 4. The maximum size for a workshop like these two should be 15-20. A large meeting begins to resemble a conference, and lessens the opportunities for participant interaction. 5. If pre-registration indicates an enrollment larger than 20 the total group should be broken down according to their familiarity with the NAEP model. A person just appointed to head an assess- ment program might or might not be very knowledgeable about assessment in general and the NAEP model in particular. 6. More careful attention should be paid to the selection and prepara- tion of speakers. There were instances where knowledgeable speakers were ineffective "teachers." Outside consultants should be used when it is known that they are stimulating teachers and also have the appropriate skills. 7. An attempt should be made to cater to some of the needs of par- ticular states but not at the expense of the overall philosophy in which these workshops are conceived and conducted. 8. The 1974 meeting should place more emphasis on discussing the manner in which the data are disseminated and development of objectives. 9. The 1974 meeting should include more "discussion" sessions, with time being made available for such sessions. 10. National Assessment should sponsor a follow-up assessment meeting in 1974 for persons who attended these two workshops. The majority of the participants in both Workshops I and III indicated that they would be interested in returning next summer for a follow-up meeting. This relates to a suggestion for the continuation of the Workshop II approach (see recommendations after the next section). 28 e., ipla qt.. L-1 EVALUATION OF WORKSHOP EXPERIENCES WORKSHOP II Workshop II was designed for persons who were already familiar with National Assessment. The program concentrated primarily on state assessments. While 53 persons attended one or more sessions, only 44 were in attendance for the entire session. Of these, 40 responses to the evalua- tions were received. In a few instances the total number of responses to a particular question will be less than 40 because of failure to respond to that question by one or two persons. Reactions to Conference Structure 1. 2. 3. 4. 5. 6. 7. 8. The results for the first question are: Overall workshop organization Workshop content Speakers Workshop materials Living facilities Opportunity to interact individually with other conference participants Availability of NAEP staff Availability of workshop co-directors Good Average 36 4 31 9 17 23 26 12 31 6 29 9 30 6 35 4 Poor 0 0 0 1 0 2 1 0 ? 1 These results showed that about 90 percent of the respondents felt that the overall organization was good, as was the availability of the work- shop co-directors. About 75 percent rated the following categories "good": (1) workshop content; (2) living facilities; (3) opportunity to interact; and (4) availability of NAEP staff. Thus, six of the eight questions indicated considerable satisfaction with the workshop. 29 39 Workshop materials were felt to be good by two-thirds of the respondents. This is positive, but may suggest that some thought should be given to improvement another year. The only response that was rated less than good by a majority of the respondents was the question on speakers -- 58 percent responded "average"; 42 percent responded "good." This clearly needs some attention for next year. Some effort needs to be made to improve the quality of presenting material. Since not everyone is really accomplished at "teaching" this may be difficult to change. And since many of the presenters were chosen because of their position in a particular program, it may be difficult to change. However, greater efforts could be made to encourage speakers to prepare themselves thoroughly, to have handouts and/or satisfactory audio-visual materials, etc. Some respondents make comments after these questions. Those made by more than one respondent are as follows: 4 Best workshop ever attended 4 Good state assessment programs 3 Poor opportunity to interact 3 All participants should be housed together 3 Too much scheduled * 4, Reactions to Specific Sessions * * * The responses to the next question are given below: 1. (California, Florida, Maryland, Texas) Good Average Poor ? 1.1 Purposes, objectives, exercisers 22 12 2 2 1.2 Sampling & data collection 21 14 1 2 1.3 Analysis and report writing 19 16 1 2 1.4 Dissemination 21 16 0 2 30 40 Good Average Poor ? '4. NAEP dinner program 17 16 1 0 3. What's New at NAEP 6 15 17 0 4. Minnesota assessment 23 14 2 0 5. New Jersey assessment 26 8 0 2 6. Michigan assessment 22 11 2 0 7. Small group discussions 20 9 1 1 8. Individual consultation in Denver 2 0 0 0 From these tabulations one might conclude that the reactions to the various presentations of state assessment plans were quite consistent, with the following numbers rating them as "good" -- 22, 21, 19, 21, 23, 26, 22 (questions 1, 4. 5, and 6). However, percentages ranged from 50 to 72. In toto, the three separate presentations (Minnesota, New Jersey, and Michigan) were received a bit better than the four "combined" presen- tations. Perhaps of greater significance is the fact that the respondents reacted quite differently to different sessions. Only four respondents rated all the sessions as either good or average; everyone else cast a split ballot. This suggests that everyone felt that at least some of the state assessment sessions were good. In fact, only two respondents did not check at least one session as being good. So, different needs were being met. The NAEP dinner program received a 50-50 reaction. But four respondents either double-checked that program as being outstanding or very good or added a footnote to that effect. The only generally negative reaction was to the discussion of what's new in NAEP. One can only speculate as to whether there is a lack of interest in NAEP activities or whether the things that are new are not of great interest or whether too much time was devoted to it. The small group discussion sessions were well received. Perhaps more time should be devoted to discussion sessions in the future. * * * 31 41 Reactions to Length of Workshop The question about balance of time and material resulted in 24 respondents saying that it was satisfactory; 9 saying that material should have been spread over more days; and 4 saying that less material should be covered in the same number of days. The co-directors had been con- cerned about "working" the participants too much. But only about a third of the respondents felt that the balance was unsatisfactory. Some individual comments were made to that question. Six respon- dents suggested eliminating evening sessions; three suggested adding a day or two; two suggested that the schedule was OK; and two suggested fewer days but the same amount of material. Most and Least Liked Sessions Two of the questions dealt with those things that were of most value and those of least value. There was considerable agreement on each question. Eighteen persons specifically mentioned things like "sharing ideas with others"; "interacting with others"; or "discussing problems of state assessment," as being most liked. Thirteen persons specifically mentioned that the most liked reaction was hearing about other state assessments. These two reactions were the overwhelming ones. No other single reaction was mentioned by more than two respondents each. The points that were mentioned by at least two persons were: (1) total impact of workshop; (2) sampling; (3) getting help from NAEP; and (4) the state plus contractor presentations. The least liked program was "What's New in NAEP." Eighteen persons mentioned that one. Of the eighteen, seven pointed out that it was repetitious of things covered in WorkshopI which they attended. Three other reactions were mentioned as least liked by more than one person. They were:, (1) technical discussions; (2) too much detail; and (3) the dinner meeting. 32 42 These reactions confirm those from other years that tell us that state department assessment personnel want to know more about what is going on in other states ar_d want to be able to share and discuss their own assessment questions and concerns. Recommendation for Workshop for Others Thirty-four of the respondents (90 percent) indicated that they would recommend a workshop of this nature for other state department personnel. This certainly suggested an overwhelming feeling of satis- faction with the total impact. Seventeen of the 34 said that their reason was because non-assessment personnel in state departments (curriculum specialists were most often mentioned) need to know more about assessment. Sixteen of the 34 indicated that they would recommend the workshop because they received a great deal from it themselves. Thus, respondents really reacted to this question quite differently -- some in relation to its suitability for non-assessors, some in relation to its suitability for assessors. In one sense, then, it turned out to be a double-barreled question, which is difficult to interpret except to conclude considerable satisfaction. e,c When asked if they would recommend this workshop for LEA's, the responses were more mixed.. Twenty-one said no, 13 said yes, and five said maybe. Of those who said no the most common reasons were: (1) it's the SEA's job to deal with LEA's, or (2) it doesn't apply to LEA's, or (3) too technical. Of those who said yes the most common reasons were: (1) LEA's need to know about assessment too, or (2) would recom- mend for large cities only. 33 43 Suggestions for Improvement Two of the questions were completely open-ended. One asked for suggestions for improving the meeting; one asked for "final comments," most of which were suggestions for improvements. So it seems appro- priate to summarize those comments together. Certain points were repeated often enough (from three to eight times each) to warrant mentioning them. One common response was "to continue with presentations of specific state plans" (seven responses). This suggestion reinforces the very positive responses that were received on questions relating to specific state assessment plans. The small group discussion sessions were asked for again (five responses) with several respondents suggesting that more time be allowed for each session. The hour allocated was not felt to be sufficient, accord- ing to some comments. Quite a few respondents (eight) suggested more "focused" sessions, in which a specific topic would be addressed in some depth -- perhaps a half day devoted to it (or more). Specific topic suggestions were "some subject area such as reading"; "what's new in various states or NAEP"; "new technology"; "assessing change." Seven respondents suggested that the 50 states were different enough on some bases (size, geography, level of assessment "sophistication" or group vs. individual pupil data) to ask for breaking up the workshop. One type of suggestion was to move to mini-conferences or to regional con- ferences. Another type of suggestion was to offer more alternative ses- sions within an annual meeting. Five respondents suggested the need for better organization of presentations and/or for better speakers. It was felt that some presenters had done a much better job of preparation than others. Three persons called for more attention to the relationship of 34 44 assessment to instruction, of focusing on how assessment results can be used more effectively. Several suggested bringing more curriculum specialists froni their own staffs to the meeting. A great maw- comments were individual ones. They will not be listed here, but vill be reviewed carefully in planning for future activities. One individual suggestion will be quoted however. It is "Fine each speaker 10 for each time he says, 'you know. The fund would pay all refreshment costs." Best Time for 1974 Meeting When asked to suggest the time for a similar meeting in 1974, the res,ilts were as follows: May 5 June 4 July 9 August 4 October 1 Before June 1 Summer 1 After July 1 3 J. Would You Like to Attend in 1974? The most rewarding response, from the directors point of view, was made to the last question. "Would you be interested in attending a follow-up workshop next summer?" Thirty-seven persons said yes, two persons said perhaps, and no one said no. This response leads directly to our first recommendation, below. 35 45 Recommendations Based on the Evaluation The directors' evaluation of the workshop evaluations has led us to the following conclusions: 1. National Assessment should sponsor a summer workshop (or con- ference) on large-scale assessment for state department personnel in 1974. Aside from the quality of the program, the fact that the meeting brings together state assessment workers is itself a most important contribution to the assessment movement. 2. The 1974 meeting should place considerable emphasis upon what is being done in state assessments. 3. The 1974 meeting should include more "discussion" sessions, with more time available for such sessions. 4. Efforts should be made to improve the "quality" of presentations, either by judicious selection of presenters or more specific direc- tions, to the extent that this can be done. The directors do not feel, however, that an important topic, or a given state program should be eliminated from consideration because of an inability to guarantee a "stimulating" presentation. 5. Efforts should be made to focus on specific assessment "issues" of concern to a number of states, and not confine the efforts to things of interest to all states. 6. Efforts should be made to increase the "individualization" of the program by offering more choices rather than concentrating as much on total group sessions. This suggests a movement toward a "conference" format in contrast to a "workshop" format. 7. Consideration should be given to inviting a greater variety of state department personnel to at least some of the meetings next year, in particular, curriculum specialists. A conference format would lend itself to that possibility. 8. The six-month follow-up evaluation of workshop attendees should be focused on identifying specific program content that will be of maximum utility. Suggestions made in the recent evaluation should be used as one basis for developing the questionnaire. The directors are pleased with the evaluations of Workshop II, both because of indications of satisfaction but also because of the many specific suggestions for improvement. 36 46 WORKSHOP STAFF DIRECTORS Frank B. Womer Irvin J. T_,c.-hmann The University of Michigan Michigan State University NAEP STAFF Assessment Workshop Coordinator Jack Schmidt, State Assessment Consultant Abbey, Janet, Secretary, Utilization/Applications Brown, Rex, Report Coordinator, Utilization/Applications Burton, Nancy, Area Development Monitor, Exercise Development Crane, Robert, Report Writer, Utilization/Applications Gadway, Charles, Writer Analyst, Research and Analysis Hazlett, James, Director NAEP, and Director, Elementary and Secondary Education Services, ECS Herman, Magdalen, Program Consultant, Utilization/Applications Johnson, George, Associate Staff Director, Administration Martin, Wayne, Analyst, Research and Analysis Mughrabi, Linda, Secretary, Utilization/Applications Mullis, Ina, Analyst, Research and Analysis Newcomb, Scott, Area Development Monitor, Exercise Development Oldefendt, Sue, Analyst, Research and Analysis Phillips, Donald, Analyst, Research and Analysis Pratt, Ted, Dissemination Coordinator, Utilization/Applications Rogers, Todd, Sampling Coordinator, Research and Analysis Salazar, Teresa, Program Consultant, Utilization/Applications Seaman, Ken, Director, Utilization/Applications Searls, Donald, Director, Research and Analysis Sherman, Sue, Analyst, Research and Analysis Talle, Roger, Director, Operations Trussel, Vicky, Editor, NAEP Newsletter, Utilization/Applications Vandermyn, Gaye, NAEP Public Information Officer, Communications Department/ECS Wilson, Hal, Director, Exercise Development Wright, David, Data Coordinator, Research and Analysis * SPEAKERS Bettinghaus, Erwin, Professor of Communication, Michigan State University Miller, Gerald, Professor of Communication, Michigan State University Rubin, Louis, Professor of Communication, University of Illinois 37 47 STATE DEPARTMENT PRESENTERS Adams, John, Director, Educational Assessment, Minnesota Ascher, Gordon, Director, Educational Assessment, New Jersey Carlson, Dale, Director, Statewide Assessment Programs, California Cruse, Keith, Program Director, Needs Assessment, Texas Fisher, James, Consultant, Division of Instruction, Maryland Keiter, Roberta, Assistant Director for Test Administration, Montgomery County Public Schools, Rockville, Maryland Impara, James, Administrator, Statewide Assessment, Florida Schooley, Dan, Director of Test Development, Michigan Shepard, Lorrie, Program Evaluation & Research, California ASSESSMENT CONTRACTORS PRESENTERS Bayless, David, Senior Educational Sampling Statistician, RTI Pyecha, John, Senior Educational System Analyst, RTI Finch, Fred, Manager, Custom Instrument Development, CTB/McGraw-Hill Schabacker, William, Elementary and Secondary Programs, ETS * * * * * UNITED STATES OFFICE OF EDUCATION Majors, Dexter, Program Officer, State Agency Cooperation Division Song, Tongsoo, NCES, Office of Education * * * * * COOPERATIVE ACCOUNTABILITY PROJECT Ludka, Art, Assistant Director Olson, Arthur, Director * * 38 48 WORKSHOP PARTICIPANTS ALABAMA Russell Chandler Research Consultant Office of Planning & Evaluation 412 State Office Building Montgomery 36104 Frank T. Speed, Director Departmental Services Room 416, State Office Bldg. Montgomery 36104 Phone: 205-269-7216 ARIZONA Eugene Hertzke Superintendent, Creighton District 2702 E. Flower Street Phoenix 85016 CALIFORNIA Dale Carlson, Director Statewide Assessment Programs Department of Education 721 Capitol Mall Sacramento 95814 Lorrie Shepard, Consultant Office of Program Evaluation and Research Department of Education 721 Capitol Mall Sacramento 95814 Phone: 916-322-2200 COLORADO John F. Haberbosch, Senior Consultant, District Planning Services Department of Education State Office Building Denver 80203 John W. Helper, Consultant Department of Education Room 430, State Office Bldg. Denver 80203 Phone: 303-892-2238 39 49 Jim Hennes Planning & Evaluation Unit Department of Education State Office Building Denver 80203 Phone: 303-892-2217 CONNECTICUT Joseph J. Cashman Educational Consultant Bureau of Management & Finance State Department of Education PO Box 2215 Hartford 06115 George Kinkade State Department of Education State Capitol Building Hartford 06115 Phone: 203-566-4382 FLORIDA James C. Impara, Administrator Department of Education 117 M.J. Suite 10 Tallahassee 32304 Tom Seals, Evaluation Consultant Department of Education Suite 10 M.J. Tallahassee 32304 Gambhir M. Shrestha Statistician, Evaluation Section S.-tte Department of Education Tallahassee 32304 GEORGIA Peter Thomas, Consultant Statewide Testing State Office Building, Room 258 Atlanta 30334 HAWAII Janet I. Sumida, Evaluator State Department of Education 1270 Queen Emma Street Honolulu 96813 KENTUCKY William W. Field Pfogram Coordinator LEA Title I Projects Division Compensatory Education Department of Education Frankfort 40601 David W. Shannon, Head Office of Planning & Evaluation Department of Education Frankfort 40601 Eugene Tallent, Coordinator Planning and Evaluation Bureau of Instruction Department of Education Frankfort 40601 Phone: 502-564-3233 Donald S. Van Fleet, Director Division of Evaluation State Department of Education Frankfort 40601 Phone: 502-564-4394 MAINE Joseph Pecoraro Educational Planner Office of Planning, Evaluation, and Research State Department of Educational and Cultural Services Augusta 04330 Phone: 207 - 289 -2373 207-289-2321 40 50 MARYLAND Robert W. Armacost, Specialist Office of Advanced Planning & Development Baltimore City Public Schools 2330 St. Paul Street Baltimore 21218 James L. Fisher Department of Education PO Box 8717 Friendship International Airport Baltimore 21240 Phone: 301-796-8300 M. Roberta Keiter Assistant Director, MCPS Dept. Pupil & Program Appraisal 850 North Washington Rockville 20850 Phone: 301-279-3596 James Morgan Montgomery County Public Schools 850 North Washington Rockville 20850 Phone: 301-279-3595 MASSACHUSETTS James F. Baker Associate Commissioner Department of Education 182 Tremont Boston 02111 Phone: 617-727-8477 Steve Kaagan, Executive Planner State Education Department 178 Tremont Boston 02111 Phone: 617-727-5700 MICHIGAN Thomas H. Fisher Research Consultant Educational Assessment Program Department of Education PO Box 420' Lansing 48902 Phone: 517-373-1830 Dan Schooley Director of Test Development Department of Education PO Box 420 Lansing 48902 Phone: 517-373-1830 MINNESOTA John W. Adams, Director State Educational Assessment Department of Education 731 Capitol Square Building St. Paul 55101 Floyd Keller Director of Instruction Department of Education 550 Cedar Capitol Square St. Paul 55101 Phone: 612-296-4060 MISSOURI John F. Allan, Chairman Statewide Assessment Project Director of Planning & Evaluation Department of Education Jefferson City 65101 Phone: 314-751-3501 MONTANA J. Michael Pichette Reporting Services Coordinator Research, Planning, Development, and Evaluation Office of Superintendent of Public Instruction Helena 59601 Phone: 406-449-3693 41 51 NEVADA R. H. Mathers, Consultant Department of Education Heroes Memorial Building Carson City 89701 Phone: 702-882-7111 NEW JERSEY Gordon Ascher, Director Office of Educational Assessment Department of Education 225 W. State Street Trenton 08625 Phone: 609-292-7983 Joyce Matthews, Program Planner Office of Assessment Department of Education 225 W. State Street Trenton 08625 Phone: 609-292-6939 NEW MEXICO Alan D. Morgan, State Director Evaluation and Assessment Department of Education Santa Fe 87501 Phone: 505-827-2928 NEW YORK Alan Robertson, Director Division of Education Evaluation Office of Research, Communica- tions, & Evaluation State Department of Education Albany 12224 Phone: 518-474-3236 NORTH CAROLINA William J. Brown Director of Research State Department of Public Instruction Raleigh 27602 Robert C. Evans, Director Assessment and Evaluation Unit Division of Research State Department of Public Instruction Raleigh 27611 OHIO John Adams, Chief of Planning Division of Planning & Evaluation 65 South Front Street Columbus 43215 OREGON R. B. Clemmer Coordinator, Planning & Evaluation Department of Education 943 Lancaster Drive, NE Salem 97310 Phone: 503-378-3074 Robert H. Mattson College of Education University of Oregon Eugene 97401 Phone: 503-686-3405 Charles Nelson Evaluation Specialist Planning and Evaluation Department of Education Salem 97310 Phone: 503-378-3074 42 .. 52 PENNSYLVANIA William W. Burson Educational Research Associate Division of Educational Quality Assessment PO Box 911, DOE Harrisburg 17109 Phone: 717-787-4234 Thomas E. Kendig, Chief Division of Educational Quality Assessment PO Box 911, DOE Harrisburg 17126 Phone: 717-787-4234 RHODE ISLAND Carol A. Kominski Statewide Testing Consultant Division of Research & Planning State Department of Education Rogers Williams Building Providence 02808 SOUTH CAROLINA W. D. Ellis, Director Office of Research Department of Education Columbia 29201 Phone: 803-758-2169 Charles R. Statler Head Supervisor State Testing Program Office of Research Department of Education Columbia 29201 Phone: 803-758-2301 SOUTH DAKOTA Henry Kosters, Associate Superintendent, Planning Division Department of Public Instruction Pierre 57501 TENNESSEE Thomas G. Caulkins Director of Evaluation State Department of Education 263 Cordell Hull Building Nashville 37219 Phone: 615-741-3206 John E. Cox Assistant to the Commissioner State Department of Education 142 Cordell Hull Building Nashville 37219 Phone: 615-741-2927 John N. Hooker, Director State Testing Services 1000 White Avenue University of Tennessee Knoxville 37916 Phone: 615-974-5385 Ross Mol, Director Program Planning State Testing Services 1000 White Avenue University of Tennessee Knoxville 37916 Phone: 615-974-5385 TEXAS Keith L. Cruse Program Director Needs Assessment Texas Education Agency 201 East Eleventh Street Austin 78701 Ronald Knight Office of Planning Texas Education Agency 201 E. Eleventh Street Austin 78701 Phone: 512-475-2066 43 53 VIRGINIA James M. Bagby Supervisor of Testing State Department of Education Richmond 23216 Phone: 804-770-2615 WASHINGTON Gordon Ensign, Supervisor Program Evaluation State Superintendent of Public Instruction Old Capitol Building Olympia 98504 Phone: 206-753-3449 Robert C. Munson, Supervisor Program Evaluation State Superintendent of Public Instruction Old Capitol Building Olympia 98504 Phone: 206-753-3449 WEST VIRGINIA Philip F. Thornton, Consultant Assessment and Evaluation Department of Education Building 6B, Room 363 Charleston 25305 Phone: 304-348-2698 WISCONSIN James H. Gold Assessment Coordinator Research, Assessment, Evaluation Department of Public Instruction 126 Langdon Madison 53702 Phone: 608-266-7941 Robert H. Gomoll, Section Chief Research, Assessment, Evaluation Department of Public Instruction 126 Langdon Madison 53702 Phone: 608-266-1782 WYOMING Peggy A. Bagby Assistant Director, WYENAP College of Education University of Wyoming Laramie 82070 PUERTO RICO Martin Alvarado Martinez Department of Education Hato Rey, Puerto Rico 00919 Phone: 765-9350 VIRGIN ISLANDS Kurt Komives Evaluation Specialist ESEA, Title III Department of Education PO Box 630 St. Thomas 00801 OTHER PARTICIPANTS Joane Binkley ERIC Clearinghouse on Social Studies Education 855 Broadway Boulder, Colorado 80302 Robert Fox ERIC Clearninghouse on Social Studies Education 855 Broadway Boulder, Colorado 80302 George Heigho, Project Director Guidance Programs Laboratory Science Research Associates 259 East Erie Chicago, Illinois 60611 Phone: 312-944-7552, ext. 187 Bernard H. McKenna Professional Associate Instruction & Professional Development National Education Association 1201 Sixteenth , NW, Room 715 Washington, D.C. 20036 John O'Neill, Manager Measurement Research Center PO Box 30 Iowa City, Iowa 52240 Phone: 319-356-3234 Barbara Sizemore Associate Secretary American Association of School Administrators 1801 N. Moore Arlington, Virginia 22209 Phone: 703-528-0700 Lyman J. Smith, Director Assessment Operations Science Research Associates 259 East Erie Chicago, Illinois 60611 44 . 1973 ASSESSMENT WORKSHOP EVALUATION FOR WORKSHOPS I AND III* June 1973 Boulder, Colorado The purpose of this evaluation is to determine the impact of your experi- ence here in relation to your local assessment needs. Because of a possible follow-up, we request that you identify yourself so that we can collate the results of the two evaluations. All information will be kept confidential. Only group results will be reported. Thank you in advance for your cooperation. 1. Name of participant: 2. State: 3. Listed below are the eight major topical areas of this workshop. Please rank them in descending order of importance to you in the performance of your present and (possible) future work. (Use "1" for most important.) ( ) Development of Objectives ( ) Exercise Development ( ) Sampling,- ( ) Data Collection ( ) Scoring ( ) Data Analysis ( ) Report Preparation ( ) Dissemination 4. What are your general impressions of the value to you of the following: a. Overall workshop organization good average b. Workshop content good average __poor poor c. Speakers/discussion leaders good average poor d. Workshop materials good average poor e. Living facilities good average poor f. g. Opportunity for group discussion good average poor Opportunity to interact individually good average poor with other conference participants h. Availability of workshop co-directors good average poor Comments: 5. The workshop this year is very condensed; a great deal of material is covered in a very short time, including evenings. Please give us your evaluation of this concern. ) The amount of material and period of time was satisfactory. ( ) The amount of material was OK, but should have been spread over more days. ( ) The number of days was OK, but less material should be covered. Comments: *This version is condensed slightly by reducing the space available for writing comments. 45 55 6. What improvements would you recommend for future workshops (content, speakers, facilities, organization, length, etc.)? 7. What was of MOST value to you in this workshop? Why? 8. What was of LEAST value to you in this workshop? Why? 9. Would you recommend this workshop for other State Department personnel? Yes No Why/Why Not? 10. Would you recommend this workshop for LEA's? Yes No Why/Why Not? 11. Final Comments (please give us your suggestions for improvements or for new or different workshops): 12. Would you be interested in attending a follow-up workshop on national and state assessments next summer? Yes No 46 56 1973 ASSESSMENT WORKSHOP EVALUATION FOR WORKSHOP II June 1973 Boulder, Colorado The purpose of this evaluation is to determine the impact of your experi- ence here in relation to your local assessment needs. Because of a possible follow-up, we request that you identify yourself so that we can collate the results of the two evaluations. All information will be kept confidential. Only group results will be reported. Thank you in advance for your cooperation. 1. Name 2. State: 3. What of participant: are your general impressions of the value to you of the following: a. Overall workshop organization good average poor b. Workshop content good average poor c. Speakers good average poor d. Workshop materials good average _poor e. f. Living facilities good average poor Opportunity to interact individually good average poor with other conference participants g. Availability of NAEP staff good average poor h. Availability of workshop co-directors good average poor Comments: 4. What are your general impressions of the value to you of the following specific sessions: 4.1 Four state part discussion (California, Florida, Maryland, Texas) a. b. c. d. Purposes, objectives, exercises good average poor Sampling and data collection good average poor Analysis and report writing good average poor Dissemination good average poor 4.2 National Assessment dinner good average poor program _ 4. 3 What's New at NAEP good average poor 4.4 Minnesota assessment good average poor 4.5 New Jersey assessment good average poor 4. 6 Michigan assessment good average poor 4.7 Small group discussions good average poor 4.8 Individual consultation in Denver good average poor (omit if not applicable) *This version is condensed slightly by reducing the space writing comments. 47 57 available for 5. The workshop this year is very condensed; a great deal of material in covered in a very short time, including evenings. Please give us your evaluation of this concern. ( ) The amount of material and period of time was satisfactory. ( ) The amount of material was OK, but should have been spread over more days. ) The number of days was OK, but less material should be covered. Comments: 6. What improvements would you recommend for future workshops (content, speakers, facilities, organization, length, etc. )? 7. What was of MOST value to you in this workshop? Why? 8. What was of LEAST value to you in this workshop? Why? 9. Would you recommend this workshop for other State Department personnel? Yes No Why/Why Not? 10. Would you recommend this workshop for LEA's? Yes No Why/Why Not? 11. Final Comments (please give us your suggestions for improvements or for new or different workshops): 12. Would you he interested in attending a follow-up workshop on national and state asse=ssments next summer? Yes No 48 58 Education Commission of the States INTIPI The Education Commission of the States is a nonprofit organization formed by interstate compact in 1966. Forty-seven states and territories are now members. Its goal is to further a working relationship among state governors, legislators and educators for the improvement of education. This report is an outcome of one of many Commission undertakings at all levels of education. The Commission offices are located at 300 Lincoln Tower, 1860 Lincoln Street, Denver, Colorado 80203. 59