tv Politics and Public Policy Today CSPAN April 11, 2016 11:12am-1:13pm EDT
deputy director of the centers for disease control and prevention, dr. anne schuchat, live here on c-span3. and the house and senate both in session this week, the senate meeting today at 3:00 eastern to resume debate on a bill to extend funding for the federal aviation administration, and a confirmation vote on a judicial nomination at 5:30. the house returns from a three-week break. today a short pro forma session and tomorrow resuming live business with a bill to make the zika virus vaccine eligible for priority reviews by the fda. they'll also take up legislation that requires the homeland security department to prepare a southwest border threat analysis. live coverage of the house over on c-span and the senate on c-span2. and tonight at 8:00, a look at the implications of allowing women to serve in combat roles in the military. current and former military officials will address troop morale, recruitment and training, and how other countries have integrated women into combat. here's a preview. >> i spent four years of my life
after 9/11 in iraq and afghanistan virtually every year from 2003 to 2013 deployed. i've been in every environment you can imagine across the spectrum, and i've served alongside women in iraq and afghanistan and elsewhere in a variety of roles, and firsthand know the value and experience they bring to the table across many, many venues. they are flying combat close air support and helicopters and jets. they are on convoy security, which is probably the most dangerous job in iraq and afghanistan is driving logistically between operating bases and combat outposts. so, you know, my comments and my viewpoints, you know, are from a very different perspective. and i listen to the comments coming from kate and elliott about changing a culture, and particularly, the marine corps, and i've read their, some of the writings on the topic, you know. and i'm looking at it from, you know, we're talking about holding people to standards, and the reality is the fact that, you know, they've come out with standards -- of all the services
and all the branches, it was the marine corps alone that said, you know, the air force, the navy, the army, they just kind of punted. they don't want to deal with the topic. the marine corps said straight up, you want to hold the women to the standard. and every one of them, all the leadership from sec-def carter to secretary of navy davis, west point colonel herring, who filed the lawsuit against women in combat, they all said the same thing -- we just want women to be given the chance and compete at the same standard as the men and to be given the chance to go fight and die for their country just like young men have been doing since our nation's history. >> you can watch the entire discussion on women and combat tonight at 8:00 eastern over on c-span. madam secretary, we proudly give 72 of our delegate votes to the next president of the united states.
♪ ♪ new america recently held a discussion on the debate over whether college students should take assessment tests. one of the concerns dealt with how such a test would be validated and scored, given the different disciplines offered at a university. >> good morning, everybody. hi. thanks so much for coming. my name is kevin carey. i direct the education policy
program here at new america. we really appreciate you all coming here on a rainy spring morning in washington, d.c. the subject of our event today is assessment in higher education, a seemingly technical and maybe seemingly kind of boring topic that i actually think is becoming central to a lot of the discussions we have about american higher education. and i say that because i have found that assessment of college student learning is often the place that you end up at after trying to deal with a bunch of other naughty questions in american higher education. it is not the thing that people talk about often first when they talk about college. our national conversation around college is very much focused on price and debt and cost. but when you have a debate about
the price of college and you go to colleges and universities and say why are you so expensive, why do people keep having to borrow so much more money to attend your institutions, what you get back is a conversation about value. often, colleges will frame the value of higher education in economic terms, particularly if the question is framed in economic terms, and we've become very used to talking about using terms like return on investment and now the u.s. department of education is publishing information about how much graduates at individual colleges make. and from my perspective, that makes a lot of sense, but at the same time, i don't think anyone would assert that the value of a higher education can entirely or even substantially be encompassed only in a dollar figure. so, if you're going to measure the value of college in some way other than how much a diploma's worth in the labor market, it leaves you to wonder how to do that and how you can gather information about how much
students learn. if you're dissatisfied with the accreditation system as many people here in washington, d.c., seem to be -- there have been hearings in congress over the last couple of years criticizing or denouncing the accreditors in different ways because of a dissatisfaction with the way they handle quality control and other aspects of higher education. again, it leads you to a question of, if not accreditation, then what? if you want to think about innovation and higher education and maybe finding new ways to use the federal financial aid system to give entrepreneurs or people the ability to create new systems or new ways of approaching higher education, but at the same time, you're mindful of the potential for fraud and abuse, it raises the question, how would you find out whether or not some new, innovative higher education operator is really doing a good job? and more broadly, even beyond the kind of policy-type questions that we like to debate here in washington, just the central importance of our colleges and universities, the
millions of students moving through them, the very high-stakes nature of that process ends in the traditional lack of a really solid research base around college student learning. there is research out there, there are studies out there, but surprisingly little, given how many people go to college, given implicitly the value proposition and the promise that is made for higher education. we don't know all that much, really, i would say, about how much students learn, particularly at the individual student level or the departmental level or the institution level. so, for this and many reasons, new america invited fredrik deboer, who is with us today, to write a white paper that was released last week about the state of college assessment today, where it is, where it's going, and what it means. and what i think you find from reading the paper, which is both an excellent synthesis of where things are now and really a
provocative look at where things can be going, you'll find that there's more out there than people realize. there are actually some very smart people, and we have some of them in the room today to talk about this who have been doing great work over the last decade and even more developing new ways of assessing college student learning. people have become very interested in these assessments. the broader idea of using standardized assessments in learning is one that is actually more familiar to us from the k-12 arena, where it has been and remains very controversial, yet very much a part of the fabric of our k-12 schools. there's a lot of i think trepidation, some of it warranted, some of it not, in higher education about whether or not it's either possible or appropriate to assess people at a more consistent or standardized means. there are a lot of very complicated technical questions about how to do that, and then there are a lot of, in some ways, broader philosophical questions about the meaning of
assessment and how it ought to relate to higher education. so, that's what we're going to talk about this morning. we are grateful for you in the audience and everyone watching out there on c-span, and we are going to start with a presentation from fredrik deboer, who's going to talk to us about his research and the paper that he's written. so, fredrik, thank you. >> good morning. thank you for coming. i'd like to thank kevin and new america for bringing me here and for commissioning the paper. it's really a great opportunity. i try never to read too much when i present, but i do like to hear myself talk, so if i'm going on a little bit long, you guys can throw something at me. so, i guess the thing that i really want to talk to you all about is why i would happen to come to write a paper like this when i consider myself someone who's still within the liberal arts or the humanities and someone who opposes the sort of corporate turn in higher education. you know, why would i come to
write a paper like this? it is the case that now most of my research is quantitative in nature, social scientific and empirical, but i grew up in the humanities and consider myself fundamentally a humanist. so, i want to tell you a little it about where the research came from and how i can try to synthesize those sort of parts of myself. so, like most research, my interest in standardized testing in college came from my own local context and my own life. when i was getting my doctorate at purdue, which i completed last may, a controversy erupted there over proposed implementation of the collegiate learning assessment plus, which is one of the major tests of -- standardized tests, excuse me, of college learning today. the mitch daniels administration -- so, mitch daniels is the former republican governor of indiana and is now the president of purdue -- wanted to implement this test at wide scale in the university. they wanted a very large portion of the incoming freshmen and the outgoing seniors to take this test in order to monitor undergraduate learning. the daniels administration has
made value their sort of keystone word. for example, in indiana, it's now dotted with billboards that say "education at the highest proven value," to sort of sell purdue. this was bound to be controversial for a few reasons, the first of which is that mitch daniels' administration has been controversial at the school. this is for a variety of reasons, maybe the biggest one being that he does not have an academic background, which made many of the faculty unhappy when he was hired on. it's also the case that the way in which he was selected was controversial given that he appointed all of the trustees who, in turn, appointed him. so, when he was governor, he appointed the trustees that then appointed him as the president, which was also controversial. but assessment in particular became a linchpin of a lot of other issues that have been sort of bubbling along on the college since his appointment, and that's because the assessment in a very deep and real way asks what we value in the university,
and it is inevitable that assessment will to some degree control learning. i think that you can have minimally invasive assessment. that's part of why i wrote the white paper. it's part of what i'll talk to you about today. but there is no doubt that assessment is always going to have some impact on how learning happens on college campuses. if it didn't, why would you do it? and so, the question is then was this in the mind of the faculty a way for the daniels administration to sort of take control, to wrestle control of undergraduate learning away from the faculty senate where it had always been sort of invested? and that was sort of the proxy issue. so, the fight was about this test, but it was also about faculty control of learning and the gradual deprofessionalization of the professory that's happening n d nationwide. so, when i was doing my dissertation research, the point that came out again and again
from faculty is we want to assess, make sure our students are doing a good job, but the question is who controls it. and one of the biggest issues with standardized tessing is it is, in fact, standardized. and what many faculty members don't like is it removes the local control and local definition of what success means. and yet, i still think that standardized tests can be useful, even though i myself believe in and understand a desire for faculty control and for local context. for a little bit of background. so, i've spent sort of the past five or six years becoming versed in educational measurement, statistics, psychometrics and related topics within assessment. i would prefer you not to try to quiz me on that stuff right now, but i've done a lot of work, and i came up with a very conventional kind of liberal arts background. i got a ba in english and philosophy, for example. so, 10, 12 years ago i was reading thomas hardy, and these days, for some reason, i'm staring at a spreadsheet all the time, which wouldn't ordinarily be what i enjoy.
but i wanted to acquire this at least basic literacy and quantitative skills, because it seems clear to me that some people -- and i don't think everybody should be this way, not even a large proportion of us -- but some of us within the humanities have to be able to speak the language of numbers in the social sciences, because that is the language of power. it is abundantly clear that the policy world speaks in a certain kind of language, and that language is statistics that language is validity, reliability, and i became concerned that that was not a skill set that was in the hands of most humanists. and so, one of the things i ended up finding in my dissertation research is humanities professors were constantly complaining that they were cut out of major policy initiatives and think tanks and commissions, but then the question becomes, you know, what would you talk about if everyone else is talking about numbers? that's a bigger fight, but that was sort of why i'm here, and i still maintain to myself a sort of core belief that i'm pursuing
what the humanities are all about. anyway, so, the story of purdue had kind of an anticlimactic resolution. it became a very big kind of local controversy. the local paper actually ran a front-page story that said "daniels and faculty in battle of wills," which, of course, made that true. like, if it wasn't a battle of wills before, as soon as the paper said that it was, both sides kind of dug in, because now they had to save face. what ended up happening is that they delayed, they sort of -- they did what, you know, universities always do, which is they had committees with subcommittees and those subcommittees had subcommittees, and there was kind of a delaying action. eventually, what happened is that the students wouldn't sign up to take the test, so they had this sort of big kind of battle going on, but one of the major impacts and what actually happened was that you couldn't get an adequate sample size of the size that the daniels administration wanted. so, they're still going forward with the cla at the school and it's going to be interesting to see what happens, but in much smaller numbers than they had originally proposed. you know, 18 and 22-year-olds
aren't exactly eager to take a standardized test that they didn't have to take in the past, so that's one of the issues with this sort of thing. but i do think that the controversy's really interesting and important, and i think we're going to see these debates play out in many schools across the united states, because assessment is not going away and standardized assessment is not going away. we've had a success in presidential administrations, both the george w. bush administration, the barack obama administration, who have had educational officials who have made a strong call for more standardized assessments going into college, and this is an issue on which the political parties, although they have disagreements about who will do what and the sort of dynamics of how it's going to happen, there is great agreement between them on this issue, and people within higher education cannot just sort of close their ears to this. although this particular plan was eventually scuttled, the obama administration was going to try to tie performance on standardized assessments and college rankings to availability
of federal aid. that is a very big stick indeed. that is the kind of thing that no college, even the really elite colleges, can afford to ignore. so, if we're going to move forward as a system of higher education that can define what's going to happen to itself, rather than have assessment happen to it -- so, there's, in my field, in composition, there's a guy named edward white, and his white's law is assess or be assessed, meaning that if you don't perform assessment yourself and if you aren't willing and eager to get involved in assessment, then you will end up finding assessment being done to you. and the reason why i'm so invested in this is for that exact reason. i think that faculty can take an active role in assessment, that we can get out ahead of these problems and that we can become a major force in shaping how assessment happens. if, on the other hand, faculties simply say we are not going to do this, then it's going to happen anyway, and it's going to happen in a way that does not reflect faculty interests.
i think that that's reality. some people see that as fatalism, but. i want to make a few things clear to people within the humanities world who disagree with me. to begin with, we're already assessing in many ways. there's already all kinds of assessment that have happened on campus. the problem is that many of these assessments are ad hoc, they're idiosyncratic, they're lacking in validity and reliability evidence, they can't talk to each other from one campus to the other. i mean, so, the entire process of accreditation that kevin mentioned, that is supposed to and has always supposed to have had an assessment function. the idea of assessing colleges is not some new neoliberal enterprise. it's always been central to the process of accreditation. the fact that the accreditation process is seen as toothless is not, you know, really a good thing. if we're going to have an accreditation process, it should mean something, right? and the fact that so many schools have become used to a context where the accreditation process doesn't threaten them means we have to reform the
system. but i'll name another form of assessment. rankings like the u.s. news and world report rankings, in a real sense, that is an assessment. i think that it's a very wrong-headed kind of assessment and i think it's casually destructive to what we really value on college campuses, but in a very real way, they are saying let's assess the quality of these schools. and they put out a list of rankings that many students and parents pay a lot of attention to. and one of the things that means is that through a lack of assessment data, we simply perpetuate the elite. there's many things that elite colleges do well -- harvard, yale, stanford, university of chicago. i don't mean to demean them. they do a lot of things well. but the notion that we know that they teach undergraduates better has no evidentiary basis, right? so, the sense that harvard is the best university in the world has almost nothing to do with how well it teaches undergrads. that data does not exist. and because they skim off the top and they take only the truly elite high school students, you could probably put those students into any particular university and see them excel.
and so, when they sort of report how well their students do after graduation, you can fairly ask, well, does that come from your undergraduate education? in my hometown, i went to public high school, the local private high school always bragged about the standardized test scores, like s.a.t. scores, that its graduates got, which they would ignore the fact that you had to do well on a standardized test to get in, so it's like, you know, having like a height minimum and then bragging about how tall your student body is, right? that is sort of the system that is perpetuated right now. a lack of assessment data allows elite institutions to maintain the fiction that they teach better without having to provide any evidentiary basis to do so. and this is one of the reasons why assessment is a social justice issue, because if we want to make colleges true vehicles for equality, we have to be able to generate data that says, in fact, what is perceived as being a sort of elite university has no solid basis for saying that it is. so, i mean, you guys -- you know, every year there's a
controversy about one school or the other slipping or rising or falling a little bit in the rankings of "u.s. news az newz. repor report", but it's not like harvard's going to show up tomorrow and be at 50, right? that doesn't happen. so, we're just shuffling around these very elite institutions at the top. but it's also the case, and i think that, you know, in the simplest terms, higher education is under really profound threat right now. we have had massive amounts of defunding at the state level. we have a lot of people in the policy world who think that physical colleges in the traditional liberal arts should be replaced altogether with online-only programs, certifications like that. i think that it is profoundly naive to think that online education is going to sweep in and within a generation we'll see a reduction of colleges of 90%, something i read a lot about. for one thing, i think that just underestimates the persistence of institutions and the inertia of how hard it is to change these large institutions. but i also think that we do a
much better job of educating than online-only education. i think what limited evidence we have now suggests that that's true. but if we're going to say that, we have to generate data to say it. you know, the same people who tell me that we shouldn't be assessing hate the idea of online-only education. they hate the idea of the demise of the liberal university. well, if that's the case, then you need to be able to say to the rest of the world, we do something very well. i think that we need to assess more in college, and i think this can have a lot of benevolent institutions. and the message should read everyone involved, we take a lot of resources from society. college is very expensive. we are draining hundreds of thousands of dollars on the backs of young people who then graduate into an uncertain economic climate. colleges receive hundreds of millions of dollars from the federal government. and we have an invested college with dysfunction, somewhat unfairly, but dysfunction of being an absolutely key linchpin
of having a healthy economy. and there's no enterprise in the world where we pay hundreds of millions of dollars and no one bothers to ask how well we're doing, besides the defense industry. so, i think that we can do more assessment, but i also agree with skeptics that this kind of assessment is very hard, okay? and one of the things that i want to insist to everybody, i think it's useful in a context like this in the policy world, is these problems are not just political or theoretical in nature. one of the frustrations of being a humanist and talking about assessment is when i'm in the other world, in the policy world or the educational testing world, the psychometrics world, assumption is that resistance to these tools are always about political resistance or self-interested resistance of the faculty. that when we say these are hard to do, it's really about we just don't want anyone to check our work. but from a pure social science,
it's hard to run the assessme s assessments. i think there is a lot of data we can collect to guide decisions, but we can't underestimate that. one of the biggest things, for example, we know that colleges have profoundly different populations coming in every year, right? part of the reason why elite colleges invest so many resources in their admissions process to make it truly exclusive is because they know that it works to pull sort of the cream off the top who are going to go on and excel. so, one of the things that we see that's true of these standardized tests in college is that the best predictor of how well both your freshmen and your senior populations will do is their s.a.t. scores coming in. in other words, we know very well with a great deal of certainty, with some variation, of course, in the world of variability, but we know very well how most colleges are going to stack up in rankings simply based on how they stack up on their average s.a.t. score incoming. so, we need to use value-added models and things like that in order to correct for differences
in incoming population. purdue is a fantastic public university with a great student body. we still accept about 60% of our applicants, which, by the way, you know, that is actually quite elite in context. so, i don't think most people understand this. going to an exclusive college i extremely rare. this is one of the things that people in journalism and the policy world i think often underestimate. there is over 3,000 accredited, or about 3,000 accredited four-year institutions in the united states. maybe 125 of them -- and that's a generous estimate -- reject more students than they accept, okay? so, the vast majority of colleges are taking a majority of the students who apply. and in fact, most colleges take essentially any student that applies. they need to simply for pure economic reasons. so, if we're going to compare colleges, we need to bear in mind that they have very different populations, and we need to have a way to correct for that. there's also all kinds of issues with scaling, traditional sort of testing things. we know how to address those
things pretty well, but let's be clear that they are empirical problems. they are not just political problems. they're not just self-interested professors saying we don't want to be tested. it's a false choice is my major point i want to make here, between sort of invasive testing, testing everyone all the time, constantly testing these students, having them have a sort of teach to the test kind of attitude, dramatic in college or no assessment. this is one of my great frustrations, is conversation so often boils down to either we enact something like no child left behind for college or we continue to do almost nothing that's rep kabul. that's a false choice. i think we can have assessment that can still provide a lot of good data. we don't need to test all the students all the time. one of the frustrations with k-12 debate is that it is often
premised on the idea that we need to do a census style for testing. in other words, in k-12, almost all students are tested almost all the time, right? and this is part of what parents hate about it. but we have the power of inferential statistics. we can form stratified samples representative of a student body and extrapolate from that sample. we know how to do that. we can take a sample of students from the average college, make sure it is adequately diverse in terms of the racial makeup, gender, the majors involved, whatever you want, and we can develop a sample, have students take these tests and understand very well how our student body are doing. that's an ability that we have, and it is frustrating that that ability from both people who push more testing and those who resist it is often ignored. we don't have to test everybody all the time. we have the beauty of a stratified sampling. i want to say it's essential to involve faculty at every stage of the process, and that's true whether you like faculty or not.
luckily, there is still some power invested in faculties, even in a lot of institutions where the higher ed striation has clawed back a lot of power. at purdue, for example, you know, president daniels has a real strong mandate -- and i should say that although i'm critical of him on some things, i have agreed with some of his initiatives, by the way -- but he has a strong mandate. he has the backing of the board of trustees. some people can say he's still the most powerful politician in indiana, even though he's not the governor anymore. but he still was unable to just sort of implement what he wanted. you have to work with the faculty. in my paper i talk about this at length, and you can read my recommendations that you let a faculty-controlled disciplinary assessment. i think academically adrift is a flawed text for a variety of reasons. one of the things that bothers me about it is they didn't even really try to assess disciplinary knowledge. so, these tests that we're going to talk about today, these are
tests of general ability, so often defined as like critical thinking or academic aptitude. they don't attempt to assess what you learned in your major classes. in other words, they are not even attempting to say how well did a nursing major learn nursing, how well did a computer scientist learn to code? you have to do that because obviously, you can't have a standardized test that works across the institution testing major knowledge for people who are not in that major. but what that means is that when people say there is limited learning on college, often they're talking about not looking at what most people are considering the most important thing you learn in college, your major. that's the important way to involve faculty. say you will always control disciplinary assessment. the computer science program gets to define success for computer science. we need that to be more standardized. we need that to be more interoperable, so we need people to talk with other kinds of assessment, but you are in control of that. that's a great way to involve faculty. finally, let me close by saying this while i have maybe the attention of the testing
industry. i think that we have to have greater access to data and information and mechanisms of standardized testing instruments if we're going to implement these at scale. we need to open up the books on these tests a little bit. to their credit, major test developers do a lot of internal research and they have proven themselves willing to publish research that is critical of their own instruments. so, ets, for example, who's often a boogieman in these conversations -- and i have a list of problems with them -- but they're very progressive in having independent researchers investigate their instruments and say, you know what, this is the problem. they're very willing to do that, and the testing industry in general is willing to do that. the fact of the matter is internal research can never replace truly external review. even the most principled researchers can't audit themselves. and so, some information is made available by testing companies. it's my opinion that not nearly enough is made available and that what is made available frequently has requirements for access that are too onerous.
i'll tell you a story. when i was still in course work, i was taking a seminar in language testing, and i needed a data set. my professor said, well, you can use this data set i have from a testing company that will remain nameless. so, i did research, i did the papers, turned it in. i turned it in. she thought it was very good. i said, well, i'd like to publish this paper. she said, well, here's the problem, that data set is a proprietary data set of this testing company. so, you have to write up the paper, you have to submit the draft to the testing company. the testing company then will decide whether to review it or not for you. if they decide to review it, that could take six months. they would send it back and say this stuff has to change. i would change it and send it back for review again. if they thought the reviews were good enough, they'd send it back to me. then i could submit it to ab an academic journal, which would take three months or longer to referee the paper. the journal would send back the paper to me with their recommended revisions. if i made those revisions, because the draft had changed, i
would then need to submit it back to the tessing company. the testing company could then sign off on the revisions or give me revisions to the revisions. once the testing company was satisfied, then i could submit it back to the journal, but of course, i would have revised it again in ways that the journal hadn't advised me to, so they could come back and tell me again. it can easily take three years or more for this process to play out when you're trying to publish this kind of data. early career academics, time is of the essence. if you are a grad student, you need to publish and get stuff on your cd to get a job. if you are pretenure, you need to get stuff in there when your tenure clock has run out. so, this poses a powerful disincentive for people to publish. there's got to be a better way to do that. i think that it's possible -- now, the concerns of the testing industry, which i recognize, are number one, test security. so, the fear that giving people data will make it easier to game or cheat on the test and industry trade secrets. i do think that we can provide data and still sort of address
those things. i mean, a test like the s.a.t., people have been trying to game it for years, and still, it's remarkably durable to cheating. so, whatever "princeton review" or another test prep company tells you, they have not been able to demonstrate that they can really game those tests up. and i think that, you know, the core of assessment is saying trust us is not enough, right? i mean, what we're talking about here is universities saying, trust us, our students are learning, and policymakers and stakeholders saying, trust us is not good enough. if that's true for the faculty, it has to also be true for t testing companies. in other words, testing companies can't expect for us to accept trust us either. okay? and i think that there's all kinds of data that you can put out there. so, for multiple-choice tests, we should, you know, external researchers should be able to do traditional item analysis, things like facility index and discrimination indices and stuff that really test developers can tell you much better about than
i can. for written responses for essay tests, i think it's appropriate for test developers to provide a research corporates, so a collection of real essays that have been generated and scored internal to the organization. give us those scores, give us the essays, give us the prompts that they were used, make the machine readable so we can use corpse linguist and things like that. ultimately, what is appropriate and fair for test companies to reveal and put out there will be a matter of negotiation and they're going to tend to err on the side of giving less and we'll ask for more, but i think we need to have more access to information than we do now. and that can help the test companies, too, because one of the major bones of contention with faculty is they say we can't look behind the curtain. i'm an expert in educational testing or i'm an expert in developmental psychology or i'm an expert in statistics, a professor might say, why can't i look at your mechanism? and i think we can both sort of serve the interests of test security and trade secrets and
open up the books a little bit. anyway, i'm going to stop talking now. but i want to close by saying, again, the notion that we either have to have no expansion of assessment or we have to have a high-stakes, perpetual testing regime that dramatically changes the university is just wrong. there is plenty of opportunity, there is many opportunities for us to gather data, to make that data publicly available, to better understand how our students are doing in clennell, how well our institutions are serving students who are graduating again with hundreds of thousands of dollars of student loan debt. we can gather information and still keep all the good things about college, but there has to be external pressure on institutions because the default sort of institutional response of colleges and universities to make a change, and we're in crisis, so we need to change. thank you. >> that was terrific.
in your remarks, you mentioned both at purdue university, the cla plus being the instrument there, also the very well-known and controversial book "academically adrift," also based on the cla plus. we are very pleased to have roger benjamin, the president of the council for aid to education, the man who in many ways is responsible for the cla plus. joining us today. i have to also note that we invited alberto asereda from ets to speak today. he was on his way down from new jersey and there was a problem with traffic, so he's not going to be able to make it, but he did send me his remarks and i am going to reflect some of those, and i suppose you can be a little tougher on ets now, not that we wouldn't have been anyway. so, roger, i would love to start by hearing some of your thoughts on freddie's white paper and presentation today. >> well, i enjoyed it. and first let me say that i am
the director of cae. i'm a political economist, and i also was an academic for a long time and still think the title professor is the one i like the best. and then in that role, i was dean of the arts and sciences, and i view myself as fundamentally a product of the liberal arts. and i'm married to an extremely distinguished art historiahistod her world is illuminated french manuscripts, and she wouldn't be bothered to attend this subject because she's totally focused on the data in that world. and let me just say something on behalf of ets. one of the problems that we've got is a paucity of measurement
scientists, psychomatricians, there are only a handful each year being turned out by iowa, iowa state, illinois, minnesota, places like that and yet as kevin noted in his intro, increasingly assessments being recognized as a very important subject. we just don't have the numbers, let alone the quality of people in that field that we're going to need going forward. the importance of ets cannot be overstated. i think the -- for example, the director of research for ce, steve kline, who just retired is a young person worked under sam
messic to design the nap and the mate rick sampling approach. the nap is the gold standard for the kind of test that makes a lot of sense. and richard shavelson was heavily involved with us. he is a professor at stanford has ties to ets. ets is doing excellent work. lydia lu is distinguished researcher there working in -- with assessment in higher education. dan mccaffery who used to be a colleague of mine and steve kline's when we were at rand in the late '90s now has a chair -- he is one of the most thoughtful people, a statistician who really knows a great deal about the issues that fredrik was
talking about. i want to say a little bit -- the bottom line is fredrik's paper is a serious critique and he poses problems that are important. and what i want to do is just make a couple of comments that are simulated by his argument and then i want to just tell you a little bit about a new way we might think about framing research on higher education, leading perhaps with educational assessment but going beyond that a little bit. i think that accountability did have a lot to do with the ramp up of assessment in higher education. i'm talking about the spellings commission and so on. but i want to note that in the
case of our group, reform and not accountability was the principle motivator. steve and i, some other colleagues at rand in the late '90s began to think about introducing assessment to higher education because we felt that it was a good idea. and steve had introduced performance assessment to the bar association via the clinical part of the bar exam, which is in all the major states in the late '70s, early '80s. and we thought that in the knowledge economy it's very important for the next generation students to be able to improve their critical thinking skills, their ability to access structure and use information, because you can google for simply facts and you really need to be able to be
stronger critical thinkers. and my colleagues and i set out with that premise and we still hold it. the board, for example, led by richard atkinson, former president of the university of california, doug bennett frommerlam. michael low max, sara tucker who ran the national hispanic association, michael crow and others really believe that this was a worthwhile undertaking and we still do. we focussed on no stakes value added approach because we thought the improvement question was a good way to start for post secondary education. we were starting to do research and development at a period in which no child left behind and the test corruption in k through
12 was at issue. and so we decided to really take this value-added approach to scale. and it is a worthwhile endeavor. it turns out that there's about a .44 standard deviation of growth, at least based on the cla across 1,200, 1,400 test administration. and that's a very significant growth in social science. controlling for entering competencies through the s.a.t. or the a.c.t. and now a matching model that we created. and the amount of growth in similarly situated institutions
varies as much as 1.0. there's a large number -- there are about 20% of the colleges that really do demonstrate best practices in important ways. obviously those in the middle and then 20% at the bottom are in somewhat problematic. about four years ago we began to develop a version of the cla that was reliable and valid at the institutional -- sorry, at the student level. and the no-stakes approach did not cause problems for motivation -- motivational problems. it affected the institutional level results but it did affect student motivation, so we've created badges, mastery levels
for the cla plus results for students and we have cla plus career connect and employers are beginning to actually ask students whether or not -- indicate that they would be interested in getting these results. so we're trying to solve the motivational issue that way. i think it's true that we could have more -- much more reliability and validity data, but the truth of the matter is also we have a paper on our website the case for critical thinking test and performance assessment and it has about 100 external publications. and all peer reviewed papers that are published by cla, ets papers and there are many, many dozens of them are peer reviewed
several independent reviewers take a look at them. that's on our website. now, the idea -- it's under the label of going forward, pasters, give you something to think about this morning. pasters quadrant in higher education. it's a name of a paper that i've just drafted. if you're interested, i'm going to leave my card somewhere and i'll be glad to send you the draft. donald stokes wrote an intriguing book called pastry quadrant, the relationship between basic research and technological change. and in it, he reverses the time honored assumption that basic research drives applied research. that's the way it goes. in fact, he has numerous -- he points out numerous cases of
which practical problems drive research and narrative kind of is his major exhibit. he was passionate about doing something about tainted milk that was killing millions of babies. and along the way in his professional career, he invented the building blocks for micro biology. and stokes himself talks about its importance, democracy, use-inspired research to help frame the way we think about public policy issues. and i came up with three or four historical cases in which this approach focussed interdisciplinary research using the value system of science, peer review transparency and the ability to replicate results as guiding principles. we in the middle of the civil war, congress founded the land
grant university and the purpose of the land grant university was to make agricultural more of a science. and over time we've done a very good job at that. to me, the green revolution, et cetera, yes, there are complaints about modified genetic crops today, but i think it's a success story. health after the flexner report in 1910, there was a commitment to make health much more of a science. and that's been a steady climb. and in the recent decades with me lekular biology, i think basically most -- the tide has turned. there are tensions between the clinicians, the practitioners and the scientists, but again it's been a good story.
finally rand, which i've just noted where i was for a decade, they're at the end of the war when we got worried about the soviets, congress founded rand and the goal was to task a group of researchers to come up with better objective tools of decision making for national security decision makers to make better decisions. and in the first decade they invented game theory, cross benefit analysis, prototype for the internet. not bad. and i just then would say that my argument in this paper is that higher ed is the next obvious candidate for this kind of focussed approach. why? well, first of all, i think there is increasing agreement that human capital is our
principal resource any nations principal resource and the k through 16 is the venue for preserving and enhancing that capital. and there is, therefore, reason that policymakers at every organization more and more focussed on how to make sure that the system improves. the hiring sector is very important because it's the apex. it sets the standards for parents, students and teachers to move toward. that's why it's important. what evidence of problems are that make it warranted specifically? i really won't go into them, but they're the usual suspects. but i would say if you just look at the higher education price index and its growth over the last 40 years compared to the
cpi, it shows cross disease problem abundantly in evidence. and therefore any increase in funds that come into the sector for public or private institutions goes with inflation and i think the primary symptoms that we read about, the student loan debt, is a huge problem. cost is a big problem. and then one that i'll mention that i think is very important and that is almost unnoticed is the demise of the tenured track and the tenured faculty actually running the sector. now, we only have 25% or less of the whole sector that are tenured or tenured track and they're primarily in the top institutions. and department gofer nans is the way we set about organizing the
sectors decisions about what to teach, recommend who to recommend to teach it and who to assess. so, big problems. in addition, you've got an absence of equal opportunity. that's something i'm very interested in. you don't have a level playing field for the students in the less selective colleges that are of high ability to have a crack at the good jobs as they leave college. it's true that 30% of the students who graduate from the selective colleges score in the top 5 or 10% of our test and only 10% in the less selective colleges. when you actually do the math, you're talking about a million, 1.5 million students in the less selective colleges, pell grant students and also students from underrepresented groups that
should be given opportunities. so there's an inequality -- feeds into the inequality problem. that's a huge issue that tony is talking about that i think is one that's a major public policy issue. finally, there's a big debate about whether it's education is a public or private good. and it's frozen. it's becoming what we call a common pool problem. leaders don't have tools to address this. and that's my argument. now, the faculty are -- when you talk about scrutiny and bringing research to bear on the sector, people -- leaders, higher ed leaders, faculties are worried about scrutiny causing negative publicity, other budget issues and so on. and there's something i call --
the department-bagsed governance itself is a double edged sword. the relative atommy afforded our colleagues is one rft reasons we have the greatest research universities on the planet. you don't -- and you don't want to do anything to disturb that. at the same time, those same principles mean that the faculty in departments view their role as sank row sank in determining what subjects to teach, how to teach and how to determine success, et cetera. third party interventions from the outside are not looked at favorably. well, all right, so i think not only do we need -- we need to really consider upping our game, this youth inspired basic
research notion in which the fields of inquiry that share adherence to the value system of science led -- not just led by joined educational assessments one but joined by cognitive science, micro, macro economics, educational technology really can be very, very helpful joined with practitioners who are very, very important, which is a point that fredrik makes to move forward. and the principles for engagement i'm just going to simply assert are these principles of peer review and transparency in results and the need for rep -- to be able to replicate results. is it too costly?
well, no. we now have data-mining tools which allow the data that we generate from these cross dis i disciplinary activities to be fed to the right groups, and i think it is possible and i talk about that in the paper as well. so, those are my comments on an important discussion in front of us. >> thanks, roger. i have a couple of questions for both of you but you should feel free to ask questions of your own. i'm going to start with this question of disciplinary assessment, which i think is really interesting. and by way of anti-dote, a couple of years ago i was talking to man named carl wineman, director of white house council of science advisers, was that his title? something like that. so carl won the noble prize in
physics and then having done that decided to do something really hard which is to think about how to do a better job of teaching undergraduates physics. like a lot of people who come to washington from the outside for a limited time at some point, society, gee, i would like to get something done before i go that's real. the thing he decided he wanted to get done was he had this idea of requiring any university that took federal research money to report back how they were assessing the quality of their instruction in undergraduate physics. i said to him, as long as we're having crazy, crazy ideas -- because of course this didn't happen, as long as we're having crazy ideas, why not have them give the same exam to their undergraduate physics students on my theory that physics is maybe one of the less contested realms particularly at the
introductory level and report how well their students are doing. and he just laughed at me and said you're a fool for even bringing that up. and if i said something like that, it would torpedo this effort which of course was torpedoed any way. i want to take seriously, fredrik, what you said, which is that the resistance among institutions and faculty to assessment is not just political. it's size for someone in my position to sort of frame it as such and i do because i do think it is sometimes as such, but if you take learning seriously, cognition seriously, assessment seriously and want to do a good job, you have to recognize how complicated it is. are there, however, opportunities in some realms where we could move relatively quickly to high quality disciplinary assessment? where are the opportunities now? >> so, i guess the -- i agree with your colleague in saying that i think that sort of trying to impose from the outside like a -- this is the national
physics knowledge test is precisely what will make faculty holler the most and probably should. that is the kind of local control i want to maintain. to make sort of the case for that local control, some fields people will acknowledge off the bat are sort of having v a political opponent that influences these decisions, so maybe the first or one of the first mitch controversies at purdue was over howard zinn who is a controversial figure in history and mitch daniels indicated that he was not a serious historian. and that sort of -- and that sort of thing, if you try to sort of impose from the outside a test on top of those people, it feels like history we expect that. look at a field like computer science. a lot of people they know tend to think of computer science -- the teaching is teaching of coding. coding is one aspect of the practical layer of computer science when, in fact, computer science is filled with theory, right? computer science programs differ
enormously in terms of their approach to teaching theory in practice to the kind of languages that they think students should be learning to what order they should be learning them in. professors are very passionate disagreements about how to teach computer science and what's the way to prepare students for the workplace of the future. so that's the kind of sort of top down thing that i would like to avoid. now, i do think that there is what i would view more of a bottom up kind of approach which we're seeing more and more which is, for example, and i'm blanking on the name, but the american historical association or something like that, it's sort of the professional organization of the history professors has recently been coming out with policy documents about here is how to do an assessment that reflects the breadth of knowledge everybody should be if you're graduating with a degree in history while still maintaining that individual local control over sort of the actual curriculum. the ncte the national council
teachers of english, for example, will have a major conference in a couple weeks. one of the things they get to do is get together and vote on these policy documents, for example, they have a goals, means and outcome student for every student leaving composition should do. that empowers faculty. there's plenty of people who don't agree with those policy documents. they come from a mutual agreement from faculty in these professional os who can then define their own standards. that's the way you do it. if it seems like some fed is sort of saying here is what you have to teach in your political science class, that becomes a profound problem, i think. also we have to acknowledge -- so purdue has a huge ag program. it's not surprising that purdue's agriculture program does a lot of work in soy and corn and not pineapple. education is a big one. your educational program will be deeply influenced by what is in
the required curriculum of k through 12 public skooms in your local context. i do think there is the opportunity for professional organizations of faculty to use their decision making powers, however democratic that might be to create standards. but one test that is sort of applied to all programs does not seem feasible to me. >> just to add to that, the gates foundation is just publishing now a study in which they've engaged the professional associations of 12 to 14 of the core disciplines and they've got -- they each field in physics and chemistry and history and so on set up committees to do the kind of thing that fredrik just indicated. and i haven't seen the results of that yet, but that's going to be quite interesting. now, the problem is one of the
problems is the only metric that is the basis for the incentive system for faculty right now is research based. and frankly unless and until we get an adapted incentive system that encourages faculty to spend more time on these issues, guess what, not much is going to happen. so that's another problem. >> so is it possible, however, that one federally posed test on everyone that sounds bad. that's on one extreme on the choice where we could find some middle ground on that or nothing for anyone. i was interested in what you said the use of statistics. we know we can do. we have things like the nap the faculty -- national narrative of stagnation is a testament to the power of statistics because every two years we pick a
different group of 17 years olds, give them the same test and get exactly the same number. that's amazing. we know the testing part is working well. but when you start to break things up by discipline and have differences of philosophy as you said inside of departments, you quickly get to pretty small numbers, even in big numbers which are ultimately not as big as they seem. they're more collections of smaller discreet things. but there might be 10 or 15 or 20 departments around the country that should have enough of a shared philosophical basis where if they got together they could voluntarily agree upon some set you want to call them standards, assessments, et cetera, the pool of students you could draw conclusions. am i right in saying that basically never happens. >> it doesn't really happen. again, so i love purdue and i've had a great time there, so i don't seem to be bashing it
during my doctoral thesis. i interviewed a lot of people for it. i interviewed provost who said purdue is more like 11 private colleges rather than one public institution. everyone has their own feoff tum. it's a futile instruction. within the institution you don't have a consistency in undergraduate. you can use the accreditation process to say we believe fundamentally in the value of faculty control of curriculum. however, we think that faculty should be able to point to some kind of sort of curriculum guidelines that have been generated interior to a discipline. created by faculty on a national level. say this is a document that we believe in. here is our general goals, means and outcome statement. within that, there will still be a lot of opportunity for students to specialize in different areas and for faculty
to teach in different ways, but that we have this framework. when we assess, we are assessing based on that framework. i think it's appropriate and possible to generate that sort of thing. in order for that to happen, there has to be some kind of a policy lever to make it happen because, again, inertia is the most powerful force in higher education, right? so this hasn't happened in the past, so it will continue not to happen unless someone provides incentives for it to happen. you mentioned the nap, a lot of times when people ask me what i want in terms of a national assessment regime, the nap for college sounds good, right? we have all these problems with standardized tests and yet the nap manages to avoid almost all of them. no kid is coming home crying because they had to go through yet another round of the nap, right? teachers aren't complaining that the nap is disrupting their classroom, forcing them to teach to test. pearson doesn't -- the pearson company doesn't exert too much
control over k through 12 curriculum because of the influence of the nap and sort of the testing materials that they put together. and again, that is relatively very small numbers of students are taking it. the same students aren't taking it over and over again. and it is minimally invasive and yet we have very powerful data on what is going on in the k through 8 level thanks to that test. >> to both of you, have we backed into something through the piac test. either of you can explain what that is or i'm happy to. >> well, no, because that focuses on non18 to 22-year-olds. it is very important for -- >> international test of adult litterac literacy. >> the 15-year-old pea pisa
test, there is still an effort that's actually being renewed to develop a higher education version of that, which is another subject. but actually we're involved in that. we suggested years ago when we got started at cae something like the college nap and my friends and colleagues at dupont circle said in your dreams, roger. you've lost your mind. they were not happy to go down that road then. probably not now, but i don't think -- these problems are going to continue to grow. i'm talking about the need for -- there is a role for standardized assessments with anything that has stakes attached to it because if you can't -- if you can't see the results of a test being reliable and valid, how can you make
decisions on it? that's the problem. and yet fredrik has correctly identified the importance of staying true to academic relative aon themy in local control for assessment. >> yeah. let's talk about that. i think the purdue example is instructive. mitch daniels in some ways is an extreme test case of a politically empowered president. i suppose it's possible to envision in a sort of small not well known private college president having more power, but in a big consequential institution like purdue, you couldn't come in with more backing than he had and yet it ends up being as any kind of thing is a negotiation with the faculty. so, is there -- you kind of
framed your remarks, fredrik, towards the faculty, you said assess or be assessed. is there a more positive message that could be given to them as in if you're serious about your teaching, this will help you be better at your job? >> sure. i mean, again -- so i frame it -- one of the frustrations for me is that almost no one of the people who object to assessment likes the status quo when you sort of break things down. i say do you like the news and world report, no, i hate it. then the reason that the u.s. news and world report exists because parents and students don't feel like they have information about what's a good college. and as reductive as they are and as many parts of their instrument i disagree with, if you're an 18 or 19-year-old, you're looking at, hey, this is at 3 and this is at 7, i know which school i'm going to. i also say to them -- if we think about why students are choosing to go to college. it's really important to remember our conception of college is very much bound up in
a kind of elite school where you go away to school somewhere in the northeast and all your peers come from all over the world. in fact, for -- for a public university, the median distance from home to school is 18 miles. okay? a majority of college students go to school within 30 miles of where they grew up. geography is number one for the large majority of students. that's the number one determining factor. and in fact, one of our problems we have we have these higher ed deserts where there's no institutions. it seems impossible. there are no institutions that are appropriate for students. talking about the median student, not attending an exclusive college, graduated near the middle of their high school class and mostly going on geography. for the kind of students that are cross shopping that are looking at different institutions, that are saying this is a good school, this is a good school, which one do i go to? what is motivating their advice?
part of the reason why tuition is so incredibly high is because we are building things like at purdue $95 million gym, which construction began in the heart of an employment depression. in that $95 million gym, we have, for example, a pool with a vortex so you can run against the current. we have a 55 foot climbing wall. many colleges now have dining halls with full time sushi chefs. we have lazy river rides. we have dorms that many of which are now based on the idea that we should get as many singles as possible instead of the traditional let's squeeze four kids into the room because it's cheap. why are colleges investing in that? number one, it's their own fault. make good decision for your student body. the real reason they're doing it is because their research shows it works. right? they invest a lot of money and decided when we had the student we wanted, they didn't go to our school, they went to a
competitive institution. that research tells them it's not because of the quality of the faculty which students don't care or understand about. it's not because of the faculty to student ratio which is a made up number that means nothing. because on the campus visit the dorm looks nice, the gym looks nice and the dining hall looks nice. when you're 18, 19 years old, you don't always have maybe the most sort of long-term thinking in mind and it sure would be cool to have that 55 feet of vertical climbing surface for the next four years and 22, 23 might as well be 73 as far as your long-term planning goes. at the center of it, there's the decision of a 17-year-old. and unfortunately parents aren't doing, in my opinion, their job in dissuading students from being pulled into that sort of thing. if you want to make the positive case to professors, you can say, the system through which students are choosing a college right now is no good for a variety of reasons. if we can assemble powerful data
that demonstrates we teach people well and teach them cheaply, that can create an incentive that can hopefully change. i do think as long as students know about gyms, dining halls that will be a pull. but i can foresee a situation in which actual educational quality becomes a factor in determining a college. right now, it simply isn't at scale. it's just not a very big factor. >> so i want to take a few seconds just to read a few of the comments that were sent in by alberto from ets. i'm going to cherry pick them but we'll make them available so you can see everything he wanted to say. largely i think in agreement with your paper he talks about higher education remains the only american institution not driven by evidence of effectiveness. of course, i find ets to be a fascinating organization for many reasons, partly because even as you correctly say, in
this one way of thinking about standardized testing or higher education, we're nowhere. but in many ways they drove the development of standardized testing through the creation of ets and has used it to sort students forever. not just at the elite level but also the sorting remediation level which brings in the majority of students one way or the other. they're all about that for getting in. the gre, the mccat, the lsat, et cetera, et cetera. it's testing in between where we end up a mirror image conversation. he talked about one of the reasons we were hoping to hear from ets they also fair to say in some ways a competitor to cae and i would say from my observation and i've been watching cla for more than ten years now in response to the attention that came to the cla i think ets felt like it needed to step up and be able to say we
can do this also. so, they've developed something called the heightened tool, which is -- i'm just reading this, suite of six assessment too manies for critical thinking, written communication, digital information litsy, civic competency, that's interesting and engagement and then intercultural competency and diversity. building on what you say you d and getting even more things that colleges are interested in. so, it seems like we're developing more of these tools going forward. do you think roger will more people jump in to this market opportunity if i can call it that? are there other players out there who will want to try to be the providers of these assessments? >> well, right now it's a bit like the wild west because the people -- the space that everybody is focussed on is from
college to career and there's a recognition that there's an opportunity because, for example, of great inflation students particularly in the less selective colleges when they leave college their transcript doesn't really differentiate them very much from other students. so, employers are searching for assessments which allow them to differentiate incoming students. you have some of these tests, these cognitive skill tests like the cla or the cap or a chunk at least of proficiency profile, but you have a tremendous amount of interest now in gallops wellness assessment based on surveys but persistence, collaboration, entrepreneurship, things like that.
there's a whole range of tests by providers who want to fill that space. the challenge they have is are their measures reliable and valid? if proficiency profile and cla plus are contested, frankly these really soft kinds of measures have got even more difficult along those lines. >> that's -- starts to move as farther outside -- >> grit is the latest. >> im ask one more question and then we'll go to questions from the audience. and this is actually a little more of a technical question but something that seems important to me, one of the ways that these instruments including the cla have been validated is by comparing s.a.t. scores. the fact that there's a strong positive correlation between these results is seen as an indicator of validity.
but we don't say that they're measuring the same thing. the same is true for the gre. this is an exam that's designed for graduate admissions. what is the difference between what you're trying to measure and what those more traditional exams measure and do those differences show up in the data? if you give the same person the two tests, why would they score differently? >> well, they do score differently. the cla adds about 19% of variance on top of the s.a.t. generally. the cla is a performance assessment and therefore by definition it's actually measuring the extent to which students are able to perform and demonstrate their writing ability and their critical thinking skills. >> everything in the s.a.t. and more? >> no, no, no. they're different tests. they're different tests.
and there's an important role for the s.a.t. the s.a.t. predicts freshman grades. it predicts -- it will -- it is a predictor of senior gpa, but i don't think anybody -- the ets would not assert that they're actually measuring the skills that students need to be successful in college and actually in the workplace. these performance assessments -- by the way, when we started they were very novel, but because of the race to the top and all the testing -- all of the money put into other development, they're becoming pretty common place now and not just -- >> talking about the exams --
>> yeah. one of the attractions is they mimic real world problems and they actually provide clues for faculty or teachers or the students to figure out what it is they need to do to improve. >> a few things i want to say. the first is, i have issues with ets and the s.a.t. i find it unhelpful when people try to attack them based on their validity or reliabilitrel. i can't tell you how many times i have people confidently assert that the s.a.t. doesn't predict college success. the s.a.t. is remarkably strong predictor of your grades and four-year and six-year graduation rates. it's important to say that one of the original justifications for the test was to keep students who were not adequately prepared for college out of college, which you can say that's just sort of function of sort of keeping the elite elite
and that's true, but it also might have the benefit of keeping studenting from taking on student loan debt when they won't graduate. one of the biggest problems we have is this chunk of people who can report some college on their resume but not a degree who have tens of thousands of dollars of student loan debt which is a big larger conversation. the ets does have problems but we know their tests are very predictive. in terms of how the s.a.t. and the cla plus are different. the mechanisms are very different in the sense that cla plus has a performance test which is a written response which tries to show a lot of different things in concert together. ets is among other things addicted to multiple choice tests because multiple choice tests are much easier to determine sort of reliability data and it's easier to get a higher reliability number, but a lot of college professors, including in the hard sciences, don't like multiple choice tests because they think frankly it's
too much of a reduction of what we're actually trying to do. ets will counterthat. if we have a written test, like the performance test, those will correlate strongly with multiple choice so let's use multiple choice. in the paper i provide a qualifier of the cla plus. i think that we should not be investing so much in multiple choice testing of college students because one of the things that we should know college students can do is to take a look at lots of different kinds of data which is what the performance task provides. come to a decision on issue of controversy and articulate that decision in writing. the fact that those scores will toend be highly correlated with the s.a.t. scores or multiple choice scores is not a question of interest to me. i want authenticity in my instrument and many professors agree. as far as the sort of relative placement let me just quickly say so at the institutional level, cla plus is an r squared. the way in which the two test
scores vary together. the amount of the variance in one test that explains the other is about .75, around there. in other words, you can expect s.a.t. scores to explain about three quarters of the variance in scores of the cla plus. for the proficiency profile, ets is higher, .85. these are very high numbers. this gets at something that is much bigger at college education but refers to education at large, it's a dirty secret that people don't talk about because it's impolite. look at education at large, relative performance throughout life tends to be remarkably static. meaning that of course there's variation and individuals can rise and fall, but if you look at students who are in the low groups in first grade and third grade and fifth grade, they're going to continue to tend to be in the low groups in eighth and tenth and 12th grade. third grade reading level is actually a strong predictor of whether or not you'll complete college in six years. now, that is something that is a
very large conversation in which a lot of people don't spend a lot of time talking about because it really does seem to curse people to bad educational outcomes. there's a lot going on there, but we should expect any valid tests of academic preparedness to have at least fairly strong amount of staticness between different levels because that is what we see throughout education, throughout life. it doesn't mean learning isn't happening, right? just because the cla test results correlate so highly with the s.a.t., doesn't mean students aren't learning. it simply means their relative placement is remaining the same. so you can look at a standard plot of how schools are doing. as dr. benjamin said, almost every institution is showing strong gains. we can look at the freshman administration the senior administration and there's a solid gap there. some aren't large enough, some are very large. because harvard's kids are coming at so much of a higher
level than middle state universities are, middle state university kids learn so did harvard kids, it remains static. >> three things, wait for the microphone. two, please identify your name and affiliation and three, please phrase your question in the form of a question. ben. >> thankings. i'm ben with the rockefeller institute at suni. all of this is a pretty heavy lift, given the sort of culture of college and universities you talked about the cla, cla plus institution level testing. you talked about individual assessment again with the cla plus and talk about the nap-like approach, which presumably could give national results, regional results, institution-type results. given how hard all of this is, where would you start? what's most important? >> well, i would start with
number one like recognizing that, yes, the lifting is heavy, but the stakes are very high. when i say we're in crisis, i mean that we are graduating thousands of students into a brutal job market that remains even with all the recovery in the job market remains the case that we're graduating students to the youngest college educated workers continue to face very high unemployment numbers. and the amount of student loan debt they're carrying is unconsciousable and assessment appears to be one of the levels we can finally start to press colleges for having allowed tuition to go that high. i think that the system should feel incredible shame over the fact that, you know, within ten years $100,000 year tuition will probably be on the horizon and that is a mortal failure for an institution that has been invested with incredible sort of responsible. where would i start? we're already starting. tests like the cla plus are
being implemented in a lot of places. the problem is that they're being implemented piecemeal and being implemented without a lot of -- one of the things that tends to happen right now, i agree a no stakes approach is better than ahigh stakes approach. institutions will have this test come out and if their school looks good, they'll put it on the website and if they don't, they'll put the report in a drawer somewhere. we should create incentives to say we're putting together this college score card any way. it's great but again it has these gaps in terms of if we don't know how to compare students from different institutions, harvard kids come in with certain structural economic advantages, people will read that data incorrectly. we say if you want to be able to sort of correct for those problems, you can look at actual test result that talks a little bit about testing. we have these mechanisms. i understand that people don't want to have the federal
government say this is the one test because this is a very big business and that will end up rewarding one institution if you do that. but these tests we can compare their scores because they're norm referenced. in other words, because they're compared to relative placement to other students, we can develop a system where students can look up on the college score card, okay, in addition to this economic data from ten years after graduation or whatever, here is the sort of average change in their cla plus score or the average change in the proficiency profile. bigger picture, you know, i understand why people were horrified about the obama administration idea of cutting off federal aid, but if there's one thing i know about colleges is that they're not going to move unless someone is able to provide a strong incentive to say you have to change. and again, we are in mortal crisis right now because we cannot demonstrate our value to the young people who that we
then impoverish for the first 10, 15, 20 years of their lives. >> yes, in the back. >> hi. you've certainly not disappointed in all the provocative things that have been said today. so had a hard time figuring out what i want to ask about. so my problem with nap, i think, is that it allowed -- i thought it was interesting that you started your presentation talking about learning about numbers and policy discussions because i think antic dote is how policy is made in washington. so nap allowed for an argument that all k through 12 was failing. and as a parent of a first and a fourth grader, i can tell you the testing regime is a disaster. so how do we avoid those problems if you want a national assessment? and two, with students -- so i didn't introduce myself. debby attenberg. we have a chemistry major, he
wants to be a chemist. why is he going to take a test on sort of this critical thinking. we have a hard time getting students now to take all of the different kinds of surveys. we have to offer $5 gift cards and things like that. why are students going to take this test? so those are my two questions. how are you going to avoid the problems you've seen in k through 2 and why would students take this test? >> that's a very good question -- very good questions. i also think that the standardized testing regime in k through 12 is a disaster. i want to put out a better system together, it's less invasive to avoid that problem. one of the things that are romantic -- we have a romantic national idea about colleges, one of the things that obscures is that colleges and universities are immensely powerful political lobby. there are line of scrimmages and universities in every state which means there are college and administrators who are tight
with ere senator. they are in a large majority of congressional districts. this obama plan was snuffed like that by college presidents when they pushed back. the relative power between institutions, between public elementary school or at a university the difference is just vast. especially with private institutions. and particularly because the most elite institutions that have the most weight are also all private by and large. and again my fears that we're going to do too little and not too much. i also think that it's relevant to say that we have lived through no child left behind and we are seeing a bipartisan recognition that there were many sort of disastrous outcomes from that and i would like to think that we would be informed by those negative outcomes to not try to do a similar thing in college.
finally, you know, college avoids some of the problems that k through 12 school does because it remains the fact that only a third of american adults over 25 has a college degree, right? going to college remains rare and the hardest to educate students are already getting filtered out which isn't necessarily a good thing depending on what you think college should be for. it does remove some of the issues of accounting for that problem when you're doing this kind of testing. i will say, you know, we've never been very high on comparative educational testing. one of the lies of the current moment is that we have sort of fallen from some peek where we used to be good in educational testing as long as there's been international comparisons, the u.s. has performed poorly. motivation is the question. i said in the talk that et is good about publishing research that makes its own instruments look bad. that's what the data says. i admire that about them. they did a fantastic paper a few years ago, they split into two
groups, control and experimental. one group of students was told that the test score would follow them. it would go on their transcripts and employers and graduate schools would see it. the control group was not. the test group outperformed. in other words, students perform much differently if they thought the test mattered. whatever your problems with the s.a.t., we know that students invest maximum effort in the s.a.t. because it has stakes for their life. right now this kind of testing does not have high stakes for them. we can motivate them to take the test. but motivating them to invest their top effort is the major empirical challenge is motivation. how can that change over time? one of the things that is happening right now, purdue is proposing to student that is a can put that on their transcript or resume. right now employers don't know what that means. if i say to somebody in college, this student got 1,200 on the
s.a.t.s, the test is so widespread that people can interpret that. right now, cla plus scores aren't interpretable. you would hope over time as more and more of these tests happen, i scored in x percentile of the test i took, that would become to be more meaningful to employers. and one of the other things that this whole regime might do, there's a signaling and screening argument for college that colleges valuable not because of how it educates you because it simply acts as a screen to remove certain people from the population and that employers want to know you've been vetted by something like the harvard admissions department. this would be a good opportunity to find out if that's true, right? if it really is just a screening process, then the test results wouldn't matter to employers. but that is the key issue. i've always identified that as the biggest problem. >> i just have another couple of quick comments. if the common core was a good idea, is a good idea, it's a
move toward deeper learning, it's the same effort in k through 12 that i referenced in terms of higher ed, but the rollout has been a disaster. now, compare that to what happened when in the reagan administration we adopted medicare and medicaid and in order to, in fact, decide co-payer kinds of norms and premises that would make sense when you ended up rolling it out, hew then did an rfp and spent 100 million on a 16-year random experiment. it was actually run by rand, which really gave policymakers on all sides of the equation very solid reference about at least what to do in terms of where the price -- the pain
points were in copayments and what made sense. imagine if we had done something like that for this common core that we actually went out and did a test, beta test, pilot test. the point i'm trying to make, testing when it's actually important and costs significant amounts of money really can pay off. and then finally, just to comment on the chemistry major, most jobs right now if you look at the network and the department of labor, there are hundreds upon hundreds of jobs occupations that as requisites require these kinds of critical thinking skills. most students are not going to get a job based on their undergraduate major. they'll get looked at first because of their employability more generally. >> questions? yes, sir.
>> thank you. it has been provocative. i'm terry rhodes with the association of american colleges and universities. and i know the focus here is on standardized tests, but the recommendations, frederick, that are in your paper, is your conclusion that nobody is working on those? that there are no alternatives? that the heavy lifting isn't already going on and that there are results from it that are positive? >> i wouldn't say there's no -- i think there are institutes interested in the questions and they're working on solving them and the problem is they're working on them at the institutional level. it is not clear to me, one of the issues with sort of the local control that we have in the university system they want to preserve is there is a very weird mix of federalism inside the american university system where if you were to design a system from scratch you would
not have the kind of mixture of things that colleges have local control over and those which they don't, right? one of the things that happen is we built a university system as a finishing school for elite, white male wasps and then sort of tried to expand that and groom it into being a system for everybody and so there's all these weird legacy situations in terms of who has control over what in this system. i believe that many institutions are, woing hard. i believe that many professional organizations are working hard. i do not believe that there is the necessary kind of concert of movement that there is a necessary coordination and necessary information gathering to ensure that that is resulting in outcomes that will work for everybody. i don't at all mean to imply that other people are doing these things. i do think that it's important for the kind of people that are in my world, and the people who are inclined to be skeptical
toward exterior assessment of any kind to recognize the moment of danger that we are in. does that answer your question? sorry. i'm not supposed to do a floel-up, i guess. they're utilized by many institutions reflecting the local, but they also have validity and reliability. do they have the scaleability in terms of the nationwide experience, experiment? the reality is that there say 12-state, multi-state collaborative learning assessment that's going on right now that will, indeed come out with the validity and reliability at scale. we have a lot of less than national reliability and validity data around it, but it addresses all of what you're talking about in terms of the local control by faculty and other educational professionals. it has standards without the standardization of a single
test. it is based upon the work that students are doing in the curriculum and co curriculum assigned by faculty that is if they're going to have motivation they're going to have it because it results in a grade that allows them to move forward and we're externally validating it on a national scale. so it's way beyond that you're suggesting, but it's not quite there yet, but it will be and we'll have results in august and september and there are 12 states and 100 institutions participating in it and hundreds validating it and judging it and scoring it. i think that's fantastic. i think we need to expand that 100-university scale to 2,000-university scale that we need long term -- validity is never a destination, right? validity is a vector, and we need to continue to validate these things over time. i'm eager in collecting as many
different kinds of information as i can in part because that's the only way to not fall victim to any one instrument. i applaud what is going on there and i can't wait to see the results and i also want to keep the pressure on to continue to gather more in different kinds of data so eventually all of these things can cross-validate each other and one of the things i'm interested in is not allowing a single test to become the criterion of success. so i appreciate that faculty-controlled assessment and hopefully in ten years, 20 years we will have that kind of data that will demonstrate the validity and hope to validate each other and other instruments that we can continue to work on. i think that's very important work. >> we are out of time so please join me in thanking our panelists. [ applause ] and expected to start shortly, live coverage from the white house here on c-span3 of the daily briefing with josh
ernest. today he'll be joined by dr. anthony fauci and dr. ann shucket. she joined us earlier and here is our briefing with her while we wait for the briefing to begin. >> we are joined on the washington journal by dr. ann shucket and is admiral in the cdc response efforts on health emergencies from anthrax to sars and today she's in washington, d.c. for a white house meeting on the zika virus. what can you tell us about this meeting at the white house. it's a very scary, new infection that seems to be link ed ed to pregnant women and we want to know people know what's going on and we've seen increases in
puerto rico and we just met with over 30 states to help them get ready for the mosquito season. >> who will be at this meeting today? is the president going to be there? >> no, this is update for the press with tony fauci and myself. >> and the president's budget this year for the cdc, $7 billion, just under $7 billion. how much of that this year will go to that zika fight? >> there were programs within the regular budget, but what happened is the administration has adapted some of the resources that were designated for ebola or for the strategic national stockpile and other uses to deal with the mrrjs right away while congress considers a supplemental request. >> the head of the cdc has expressed some concern about the research for using ebola fund for the zika fight. why the concern to use it for the public health crisis.
>> there is disease in guinea and liberia and about 1,000 contacts are being followed now for disease and we know that we're not going to be lucky enough to just have one infectious disease problem at a time. there is a big yellow fever outbreak in angola right now. so we're very concerned that we need to address multiple threats at the same time. that said, the zika problem is at our doorstep. it's in port roik owe and pregnant women are terrified there of the harm that may occur to their babies and so we're doing everything we can to protect them. >> as we said, our program today is focusing on the cdc. they've been helpful enough to have three guests with us from the cdc. we're talking about various issues with the agency and with its history and it's celebrating its 70th year and it's 70th birthday this year. what were the zikas and what were the e bowl as back in 1946 when the cdc started? >> the cdc started really out of an agency that was dealing with malaria, and malaria was one of
the big concerns then, but other infectious diseases like typhus and smallpox were an issue and typhus is not something that you hear about every day. we're still dealing with ma lair wra and we're still dealing with zika and cdc. >> one of the articles i read called you a virus hunter at the cdc. why do you have that name? >> i've been working on disease and vaccine-preventable diseases. a lot of work on influenza. a lot of work on unusual viruses like the severe acute respiratory syndrome virus that affected china ask toronto and other countries back in 2003, but really i'm about prevention and dr. ann schuchat.
if you're in the mountain or pacific time zones it's 202-748-8001. with zika, when it comes to zika, how closer we to a zika vaccine? >> people always want a vaccine when something is new and scary, but vaccines take a while to develop. they used to take about 10 to 15 years to develop, but with some of the newer technologies and science we have an opportunity to fast track that and the nih has candidate reserves for the zika and we don't have a vanning saeb that could be in general use for at best case, two years after that. >> the house tomorrow taking up a bill that would make zika virus vaccines eligible for fda priority reviews. what does that mean and explain why that's important. >> you know, it takes a while to do the studies to evaluate a potential vaccine, but there's
also a regulatory process and i would say that fda has been extremely committed to rapid review of emergency products. last year we were testing an ebola candidate vaccine in west africa and the fda was excellent in looking at all of the proposals rapidly, but a fast track review will speed up the time where the vaccines are being looked at. to get a vaccine for zika virus, but what we need to do now is everything we need to do to protect pregnant women in the meantime. that means reducing mosquito biteds with long pants and that means insect control and in puerto rico they're beginning that right now with mosquito control inside the house and outside the house and it really means good mosquito surveillance to know whether they're resistant to the different pesticides that are out there
and what is the better way to spend our money right now? is it on the vaccine research or the mosquito bite prevention and the monitoring, fo efforts? >> we need a vaccine, but the mosquito surveillance and the mosquito control is urgent. it's actually an emergency and the big emergency place is in puerto rico. at the meantime, we also need to learn as much as we can about this virus so we can get better information to pregnant women about what to expect and how they can protect themselves. >> the latest numbers on zika in the united states. travel-associated zika viruses. there are 346 cases in the united states, cdc with an updated map and these numbers from april 6th from last week. explain what a travel-associated zika virus is. >> right. about 40 million people travel to south america, central
america, caribbean every year from the united states and what we see is that travelers can goat b get bitten by a mosquito and get infected with the illness. in the continental u.s. the only people who developed zika virus infection have been people who have traveled or a small number of people who have had sexual contact with someone who traveled and got zika. so we've not yet had mosquito-transmitted zika virus in the states, but we have had in puerto rico, the u.s. virgin islands and american samoa. the continental u.s. in alaska haven't yet had local transmission. we know the mosquito is present in 80 states and the aej us aegyp aegypti. and we are taking your questions
with dr. anne schuchat. we'll start with cal this morning. >> my question is can a whole group of people -- >> can a whole group of people get post-traumatic syndrome brought on by the federal government because they're taking half our pensions away and we're too old to get a part-time job and a lot of us are in such poor health that we could aren't get a part-time job even if we wanted to? >> you know, post traumatic stress syndrome is a difficult condition and there are so many challenges in life that can trigger it. i'm so sorry for your situation, and i really hope the best for you. >> and we'll go to john in rockville, maryland. you're on with dr. anne
schuchat. the hospitals and nursing homes in this country there seem to be a large percentage of input with respect to care and the second question is the cdc considered to be a part of the executive branch of the government, exactly what is the cdc as an entity? >> thanks so much. you know, the cdc works closely with the health care system, one of our -- >> all right. good afternoon, everybody. i hope you had a good weekend. as advertised, we are going to do a follow-up to the zika briefing that we did have a briefing here months ago, so i am joined once again by dr. anne schuchat for the centers of disease control and dr. anthony fauci. we have learned about this virus
and the risk it poses to the american public and they'll able to provide us additional information about the funding requests that the administration put forward to the united states congress a couple of months ago. as we discussed in the briefing last week, we have not seen the kind of response from congress that we would expect and frankly, we have not seen the kind of response from congress that we have seen in the interest of the safety and well-being of the american people and so both dr. schuchat and dr. fauci should be able to provide you some insight into how those resources will be used. each of them will make a brief opening statement and for questions on this topic, we'll do those at the top so they can answer them and then we'll let them go and we can discuss the wide array of other topics that are likely to come up today. dr. schuchat, why don't i turn it over to you first? >> thanks so much. it's a pleasure to be here
today. since we last discussed the zika virus we continue to be learning pretty much every day and most of what we're learning is not reassuring. we have learned that the virus is linked to a broader set of complications in pregnancy, not just the microcephaly, but also prematurity, eye problems and some other conditions. we have learned that the mosquito vector, the aegis aegypti mosquito is linked to a broader range of states in the u.s. instead of 12 states where the moss kquito aedus aegypti is present. and not just the first trimester, but potentially throughout the pregnancy. this information is, of course, of concern and cdc has been
working 24/7 to protect pregnant women and with the state departments that are the front line of defense to learn as much as we can about the mosquito that can spread the virus and about the virus tuesday and to work with other countries to learn what we may be seeing later in the continental u.s. >> we are quite concerned about puerto rico where the virus is spreading throughout the island. we think there could be hundreds of thousands of cases of zika virus in puerto rico and perhaps hundreds of affected babies. we know that the pregnant women in puerto rico are very keen to protect themeses and to have community protection and we're working closely with the authorities in puerto rico to support that response with mosquito control beginning and with the distribution of what we call zika prevention kits for pregnant women. we have learned that it's spread
through sexual transmission and we have updated guidance, particularly for pregnant women. >> the other thing that we've learn side that there is a resounding interest in prevending this disease and controlling it as well as we can on april 1st, the cdc convened a zika action plan summit in atlanta. leaders from more than 30 states and territories joined in atlanta to do zika action planning to get ready for zika, and to increase mosquito surveillance and to increase human surveillance and birth defect surveillance so people have the best information to protect themselves and their families. so that's what's been going on since we talked a couple of weeks ago and i'll turn things over to dr. fauci. >> thank you very much, anne.
in a similar vein since we spoke last that the cdc had their summit in atlanta, we had a meeting on the monday and tuesday of that week in rockville and north bethesda. we are learning more and more about this and i'll give you a brief summary about that, but the more and more we learn the more and more you learn about the scope of what the virus is doing. bottom line is we still have a lot to learn. the first is a very important study at the very fine molecular level looking at the virus and seeing if it was different from dengue because the question we keep asking, it's transmitted by the same mosquito. what is the difference between zika and dengue? it looks very much the same molecularly except there is a short run of aminoacids, namely the building blocks of protein that are the part of the virus that binds to cells. so it may be that that's the clue of why it acts different,
particularly being neurotropic or having a propensity to affect neurological tissues. the good news is we developed two animal models, two mouse models since we spoke last and it is underscored as dr. schuchat said, when you infect a mouse there is a propensity to infect neurological tissue. the monkey now you can get a monkey pregnant and look at the difference between a virus and a pregnant monkey and a monkey who is not pregnant and what we've seen and this is just preliminary data and it's quite scintillating is that the virus stays around the blood significantly longer in the pregnant monkey than it does in the non-pregnant monkey and the reason that's important, you might remember, we had a case here of a washington resident who was infected during pregnancy, affected the fetus and that person had weeks verimia and that was unusual
because it only lasts a couple of days. in vitro studies and putting them in neurostem cells showing that it has a very strong propensity to destroy tissue which could explain why besides interfering with the development of a fetus, it might directly attack brain tissue even when the fetus is later on in the period of gestation. we also are continuing with the vaccine studies that i mentioned to you. i told you that we would have our first vaccine candidate in phase one in september. that looks like it's on time. we're producing it in our pilot plant outside of bethesda and we will then be processing it to be able to get it through the fda to put into a human and then finally, we have a screening program for drugs that i mentioned to you. we hadn't screened for drugs and we now screened 52 drugs and have some degree of activity. caution, that doesn't mean they'll turn out to be good drugs, but they do have some activity. so in summary, a lot of things
have gone on and things that are pointing to serious issues that we need to address, but we learned an awful lot since we spoke and we need to learn a lot more because this is a very unusual virus that we can't pretend to know everything about it that we need to know. thank you. >> questions on this? jeff, do you want to start? >> thank you. doctor, you expected there to be hundreds of thousands of cases in puerto rico. do you have a range of how many you expect in the united states? >> most of our predictions come from what we saw with dengue virus and chikungunya virus and those viruses are fred into a the same mosquito. in puerto rico they range between 25% and 80% of the population getting infected with one or the other viruses over the course of one or multiple seasons. in the continental u.s., we have seen travel-associated cases of chikungunya or dengue.