Skip to main content

tv   Key Capitol Hill Hearings  CSPAN  April 9, 2016 12:00am-4:27am EDT

12:00 am
>> you cannot understand the constraints on us today if you don't understand the king case. so to contrast what happened in
12:01 am
the past and shed light on where we are today there is much more significant accountability and oversight constraints with respect to the fbi's surveillance activities than there were in the past. just this last week the director of the fbi was telling a story about how on his glass-top desk, underneath the glass, he has the one-page argueization hoover submitted to bobby kennedy for observation for surveillance if dr. king. it has no limitation on dureration, the scope, what could be done with what was collected -- duration -- it was a broad authorization. it director keeps that on his
12:02 am
desk right now and every morning he is responsible for certifying applications under the foreign intelligence surveillance act which was enacted after the king survilance. so the director literally puts the stack of applications on top of the one-pager every morning. the applications are like this. each one of them is quite extensive and thick with numerous recitations required by the law and i can talk about those in a second. but the point is that the era of today, the protections in place today, are substantially
12:03 am
different than that one point time. you have layer upon layer of requirements that are an effort to have countability and oversight with respect to every application, the director or the deputy director, or some other high ranking official inside the government has to sign certification that meets certain statutory requirements and explains the purpose and necessity of the surveillance. that is what it is in essence. and in addition, every application has to be signed pie the attorney general, deputy attorney general or the deputy general for national security. they have to sign their name and are accountable. something that didn't exist at the time is the foreign
12:04 am
intelligence agency court which is a real court of judges who review these applications and the government can't go forward with these surveillances except in limited circumstances if the fisa judge hasn't signed off. permanent oversight committees didn't exist during the dr. king era. that is a hugely important oversight mechanisms that exists today but didn't in the past. in addition, if i may take two more minutes, the statute itself has a number of standards set forth in it that again constrain the power of the government and focus the government's attention to make sure we are doing these types of highly intrusive
12:05 am
surveillance. that the government is doing them in a way that is consistent with the constitution, the statute, and i would submit with american values. so for example, there are specific standards about what it means to be a legitimate target under fisa court. what is the standard? probably clause. there are limitations on the duration of the surveillance and the facilities so the phone or where the search is conducted. those have to be described and the judges had to make determinations under the law with respect to what they authorize. importantly, there is limitations on what the government can do with the
12:06 am
information once collected. one of the most problematic things is what the fbi did with the information after it collected it. the amount of information that was disclosed to other people and the sensitivity to other people that was exposed was quite astounding. today there are several important parts of the statue that limit that type of use of material. in particularly, the information can't be used for an unlawful purpose. only a lawful purpose. there are court-ordered procedures that apply for m minimization and that means they have to limit the collection consistent with the government's interest in obtaining, producing and disseminating foreign intelligence information.
12:07 am
so what i guess i am trying to say is the structure -- the government learned a lot from these cases. congress learned a lot. congress passed law. the government has been implementing those laws for a long time. we as the fbi have learned a lot from it and recognize to this day, right now, and every single day with the director having this on this desk, that we need to stay focused on this particular case but the larger issue it raises with respect to it being cautious about our own exercise of power and the beliefs we have when we are thinking about exercising the power that you have given us under law questioning our thinking and why we are doing something and pushing ourselves to make sure we are not deceiving ourselves and we are not unwear of our own biases and
12:08 am
focused on what we should be focused on when we engage in investigative activity and use investigative tools. this is something i want to tell you this is something we think about very much today. it is very much in our mind. with that, i will turn it over and look forward to your questions. >> thank you so much. >> i think it is important to recognize and underscore something you said, mr. baker. and that is it is pretty interesting. in dprcongress, you have folks talking about the dr. king case like ellison and rand paul. but it is important to say in the executive branch, the one person, and i am repeating myself prom this morning but i want to make sure everyone hears
12:09 am
me is dr. comey. he trains all fbi agents on the case. the director case a speech about hard truths and at the start of it he told this story you think is an introductory speech story and talked about his grandfather being a police cheese in yonkers and his grandfather became unpopular after he apparently cut a fire hose that bootleg les were using to transport beer from the place they brewed to the beer to the speakeasies in the neighborhood. the story is inocilous until you realize wait it minute, there was a time when beer was illegal in this country.
12:10 am
and then you think not too long ago it was illegal to be gay. there was a time it was illegal to be japanese and living on the west coast. japanese american living in your home in the western participant -- part of the united states. and there was a time when it was effectively illegal to be a civil rights leader in modern america. so how can the fbi say it should always be able to access the information on on phone to root out illegal conduct when throughout history the law has been misused in ways that hurt vulnerable people. >> there is lots there. >> we are most definitely not saying we should always have access to everybody's phone. that is absolutely not what we interested in.
12:11 am
we want access to data that we think is necessary in order to conduct our investigations that you want us to conduct and you, the america people, have charged us to conduct. when we have appropriate legal authority to do so. when we have a search warrant issued by an independent judge that says there is probably cause to believe there is evidence of a crime on that particular device what we are saying is we will try to pursue that and get the evidence off the particular device because that is what we are expected to do. we are investigators so we investigate. what we have been trying to say with respect to what you are referencing is increasingly for a variety of reasons, including encryption, we are unable to obtain evidence in a case in a situation where data is maybe stored on a phone or in a
12:12 am
situation where we have trying to do wire tap and the data is in motion and because it is encrypted we are unable to access that. what we are trying to say and director comby said it and where did and i am trying to say how we think about it. we think of our sebselves as th service of the american people. we belong to you. you own us. we follow our direction. it is our responsibility to tell you how it is going and make sure you understand what difficulties we are encountering because you want us to protect you. we have trying to tell people when it comes to electronic surveillance it has been very effective and over the years it is much less effective today. what do you want us to do about that? the laws today, generally
12:13 am
speaking, do not enable us to get access to this type of data. is that okay with you? is that all right in situations where you have the woman in louisiana who has been murdered and her mother thing thinks there may be evidence on her phone and we cannot access. is that okay when you is a situation involving a child and a child predator? is it okay when we know we have leaders in isil in syria communicating with people in the united states and people in the using encrypted communication. we can see they are communicating but can't see what they are saying. as a society, are we willing to live with the inevitable public safety costs that are associated
12:14 am
with encryption which of course enhances all our privacy include mine. we are charged with dealing with cybersecurity threats so encryption is great in that regard. we love encryption. it helps us in so many ways as a society. but it has cost. we need to think about it as as society how we will deal with those cost. maybe we say we will except it. maybe that is it. the value associated with encryption is so tremendous, great and beneficial we will deal with the cost. i spent most of my career de department of justice and beingae fbi now one thing that is impressive is the proximate of giving advice and i am closer to agents going into the houses
12:15 am
and putting their lives at risk and i am therefore closer to the bad guys and collectively we are closing to the victims. we have the face the victims. there are no demons on any side. the companies are trying to do things they think are legitimate and i don't question their moeft motives. long answer to an important question. >> i want to open it up to the audience but first i have question. can y one nugget to consider. i think one of the most powerful things in the book is how the fbi uses information. a lot of people know there was a wire tap in your book "the fbi versus martin luther king" you talk about how the fbi didn't just takeant the information or
12:16 am
use it for blackmail but they brought up the press. i invite you -- you are not supposed to ask questions if you are a lawyer but i invite you to speculate what would dr. king's life been like in the absence of that surveillance? >> two points. >> dr. king was able to survive as a landmark civil rights figure because the journalism standards were set by bradley not by nick denton. in today's america dr. king's private would have destroyed him before 1963. and the degree of sexual voyeurism that hoover,
12:17 am
phillips -- head quarter supervisor on the king case -- the degree of voyeurism was not limited to dr. king by any means. the bureau tried to do that in many instances. but american journalist back then nobody touched it. nowadays you can see what has become of this culture. second point which albert touched on, too, scores of people, probably several hundred people in washington, knew what was going on. knew that all of this sexual information was being passed around to catholic bishops, heads of foundations, religiousx journalistic, political figures all over the government. not a single whistleblower
12:18 am
stepped forward. i mentioned the organizational culture of the hoover fbi, the organizational culture of the hoover fbi meant there was never a whistleblower. whether we cite someone like jeffrey sterling, an african-american who is in jail that we should know, or we think about a great american patriot that is living under vladimer putin it teaches us whistleblowers are necessary to american freedom. without whistleblowers you get a hoover fbi. >> ladies and gentlemen, we have time for one or two questions. i know. trust me. we could stay here all day but mr. baker has to be general
12:19 am
council of the fbi and garrow has many things to do. >> my question has to do with what the lesson the fbi is drivi driving currently from the martin luther king and how does that apply to new technology? not just under surveillance authorities but the fbi's racial and ethnic mapping program which we know from public disclosures of the manual, the guide is permitted or believes it has the authority to collect publically demographic information about race in the community. the aclu has filled public records request about the program. how that authority is being impleme implemented and not leading to wide scale racial profiling and
12:20 am
intelligence gathering in investigations. >> thank you. >> so sitting here i cannot give the details of that program because i don't have them in my head. let me say this. the protection of civil liberties is baked into anything we do there. there are limits to ensure civil liberties are protected. we have people in the operati operational management teams that are excellent. they are all over these kind of issues. we are in a challenging environment, right? so the world is constantly changing.
12:21 am
the technology and tools, analytical tools we have available to us haare constantl evolving. we have to do -- make use of these tools but do them in a way that fits into constitution. we with constantly worried about things that i am guessing you are worried about. we want to make sure what we are doing is defensible because we know to the public and to congress and because of whistleblowers we are going to be held accountable. at some point all of this stuff comes out. we want to make sure what we are doing is defensible. sometimes it is hard to figure out in the moment what that
12:22 am
means. we have robust discussions and debate within the fbi about what is the appropriate thing to do, what are the requirements, and there a lot of pressures on us in terms of the fisa discussion. you get critiques also it is too slow, there are too many requirements and that slows us down especially when dealing with al-qaeda. so you have all of these pressures and the idea is put in place the appropriate mechanisms, structures and people who can make the right decisions. >> for the second and final question i want to recognize that a lot of folks are watching at home from live stream and we have a para scope feed. so i am going to give you mr. baker a question from parascope.
12:23 am
i will paraphrase a little bit. the question is where minority populations are often disenfranchised how does that affect the totality of the american people who are quote unquote authorizing the fbi to do what it does? to collect this data or not. and the questioner asks isn't it true many of the american people who want this data are not the ones at risk of being surveyed? [applause] >> trying to make sure i understand the question. tough one for the fbi to answer. we are, as i said before, we are the servants of the american people and the american political institutions that tell us what to do in particular congress and the courts. that is a question about the nature of u.s. democracy that i am not sure that i can or should
12:24 am
really answer. >> i would like to just use a phrase i mentioned to jim yesterday. a phrase that pushes back against today's theme. muslims are the new blacks. think about the political culture today with communities under suspicion. >> we will talk about that immediately after lunch actually so stick around. >> we face old questions in new context. >> ladies and gentlemen, i think it is important to recognize the unique conversation we have going on. i want to reiterate what professor butler said. we invited a lot of the nation's top intelligence officials to
12:25 am
come and many said yes and that is important and we really appreciate that. >> ladies and gentlemen, thank you very much for coming. we have so much for you in the afternoon including a discussion of precisely overlap between the african-american population and the muslim population from professor zee. we will reconvine on time and that panel is not to be missed. we have a cafeteria, we have food trucks, we have lots of businesses in the neighborhood. one more round of applause and see you at 1:05.
12:26 am
[inaudible discussions]
12:27 am
12:28 am
[inaudible discussions] >> welcome back to government surveillance session at georgetown.
12:29 am
we have a debate about predictive prisoning and predictive sentencing. a police officers' decision to stop a citizen on the stret for years was predicated as having reasonable suspicion. and up until recent history that was a human decision and of course driven by bias. what happens when a computer makes that decision? what happens when a computer not only makes the decision about woo to stop, arrest, what happens when the computer makes the decisionoon what that individual's sentence should be. those are the questions at the heart of the panel and we have an extroidinary panel. we have adjunct profess of law
12:30 am
here. mr. ferguson, sonya star, and we have christy lane scott from the united states department of justi justice. if you join me on welcoming our panel on predicted arresting and sentencing. we have four panelist, one moderator and 45 minutes. sonya, you can start with talking about predictive sentencing. >> i am not a surveillance expert so i am learning a lot. i am here talking about data-dri data-driven tools used to pro ticket the future crime risk by
12:31 am
the sentence. these tools pervade every step of the criminal justice system from bail, sentencing, corrections, to parole decisions. my work has a focus on critiquing their use in sentencing and parole but many issues are similar across the board. a risk assessment is one that bases a prediction about the client's future risk of crimes is based on an algorithm. the most poplar ones are corporate products and their formulas are proprietary so we know something about the variables but defendants don't get to know exactly how their risk core is calculated.
12:32 am
the algorithms vary. all contain variables related to criminal history. the rest vary but there is a few different categories. there is social economic variables like past and present house employment and housing, available to pay pills. next is related to family background. do you have family members in prison? do you have friends with criminal records? next is demographic particularly gender, age and marriage status. and psycho socio assessment barriers are done and they ask if you have oppositional attitudes toward the criminal justice system and authority.
12:33 am
race is not one of the variables you see but this isn't always true. many parole boards up in till the 1970s used predictions based explac explicitly on race. most of the variables are listed are race correlated because they are indicators of poverty. and so we can expect these instruments to have a racially disparate impact. in my view, many of the variables go into the scores are simply inappropriate and possibly illegal. most are merely parkers of poverty. i am not saying those things are correlated or predictive of recidivism rates. crime is obviously correlated with poverty in our society although we should remember that the extent of that correlation
12:34 am
may be made worse by the fact recidivism rates don't measure crime but getting caught. you are seeing people who have these characteristics are more likely to have been arrested and that may reflect differences but it may reflect policing differences. but even if we assume all of those characteristics actually predict crime lates that doesn't mean we should not use them when deciding how much to punish a person. when we tell a judge sentence this person based on a risk score and treat people in poverty as risk factors we are explicitly telling judges we want to literally punish people for their poverty. that is their score is determined not by what they did
12:35 am
in the case but by who they are and who their families are and how much money they have. i set forth a constitutional argument against the use of some of these variables. it is a law review article and i don't have time to go through the details. i will say it is not boundary pushing. there is quite strong existing supreme court precedent that comes from the state trying to evoke a probationer who lost his job and the state tried saying there is studies showing people that lost their job and poor people are more dangerous and the court said you cannot lump all poor people together and say just because this person shares a characteristic with people who have committed an elevated level of crime in the past that this person is more dangerous and that is nothing more than punishing someone for their poverty. that is exactly what today's
12:36 am
rick assessments do. i think they run up against that doctrine i think as well as our moral intuition. it is surprising this risk assessment instruments have been embraced by progressive advocates of criminal justice reform. it is a prong in the pending criminal justice reform pieces of legislation. why is that? i think in part people hope it will help reduce incarceration by allowing judges to identify low-risk people who don't need to be incarcerated. my view is that, you know, that reducing incarceration is an important objective but that adopting it through measures that prevent the very people that have been the most dispor portionately incarcerated is not the way to go about it.
12:37 am
i have much more to say but i am getting a big stop sign so i will stop. [applause] >> andrew is now going to briefly talk about the predicted policing. >> in my opening i would like to make one major point which is when the comes to predictive policing the color of surveillance is not white. it is not black. it is not brown. it is dark, it is hidden, secret and hard to figure out. predictive policing, the idea of that say crime data can allow predictive hot spots, hot places, hot people, through algorithm number crunching is not necessarily discriminitory. but it it has impact on communities of color.
12:38 am
the police data comes from the police and all of the explicit and implicit biases in policing show up in the data. the real question is could this be better? i want to briefly sketch out the reality and scope of predicting policing and give you terms and we will discuss it in the question and answer. to give you the sense of scale it is being used in new york, los angeles, seattle, santa cruz, kansas city and a >> hoshost of smaller cities. you have ideas you can predibt where burglars and car thefts might be. and you have your violence crimes; gang shootings, bars and clubs. and person-based predictions. we can identify the people involved in criminal activity and target them through public health model or surveillance.
12:39 am
for predictive policing, type of crime, location of crime, and time of crime are fed into a computer algorithm to predict crime. it is maybe a city block and police handed maps with these red boxes and they go patrol them. the logic is not some algorithm magic. it is just some crime is c contagious. some car theft rings take place in various parking lots because there is some environmental vulnerability. there is a contagious factor and the algorithms work on that. there are other sophisticated
12:40 am
programs that add in the time of day and weather to forecast the risk of weather at a particular time in a particular neighborhood. seco secondly there is subject-based policing mainly predicting people. chicago has a list of 400 people they believe are more likely to be the victim of violent crime or perpetrators. they get a knock on the door by a detective, social working, and maybe a community football coach, knock knock knock. you go left, you go to jail. the idea of targeting these people is at the heart of chicago's policing. i want to address how constitutional rights might be affected by this policing. the 4th amendment protects us
12:41 am
from unreasonable stops and searching. if you are in th box and the police have been told to be on the outlook for the suspicious behavior is your 4th amendment being violated? if you are one of it top people suspected in violent crime should your 4th amendment hold? it is shedding light on predicting policing. thank you for being here and my time is over. [applause] >> now we will talk about what sheri is seeing for predicting policing and sentencing at the community level. >> i work for the center for
12:42 am
media justice as an organizer and i am an organizer with our bay area. i am from california. i will talk about three things when it comes to predictive policing. a little around t the data, the money and lack of accountability. and what is happening on the ground. the first one is, and it what kind of brought up, but it is where the data is neutral which we know is not the case. we know we have historical studies proven over and over again that various communities might actually do more crime yet other communities are policed for that crime. if that is the case, and t theitathe data going into the studies, it will only produce inaccurate
12:43 am
information that relies on racist human assumptions. that is number one. the other one is there is an influx of money, it was discussed earlier in one of the panels, around muna coming in. a lot of that has gone into these predictive policing softwares that are unproven. and there are several deparliamedepartmen departments that don't work. and because they are not coming from grants they are not coming from the inflated budget. we have communities fighting for
12:44 am
community policing and structures in their cities but millions are being funneled for technologies that don't have the same level of accountability. then the last one is around what is happening on the ground. we have algorithms used to predict and fight crime but what it means is it a continuation of the militarization of our police department. they spoke about the use of drones and militarization but these pro pre-dictive policing technologies and practices were created for the military. they were created to find hot spots in iraq and afghanistan and are now being used police departments across the country because it can take public dollars into large corporations
12:45 am
without the level of transparency that is required generally for police departments. the other thing i would say is there is a lot of different groups starting to work on this. you have the baker center based out of oakland that is working to produce technology that is around holding police accountability. we had the technology earlier that brandon spoke around police accou accountability and transparency. they are talking about using technology mechanisms to hold police accountable but know about the ways in which predictive sentencing is being used against us. the last thing i saw in the conversations i had with activist and organizers across the country is really we have to
12:46 am
have a sense of it. we know there is money in it. we know that because of the work of organizers across the country, police are getting a massive spotlight on them. so the answer is how can we use technology to be able to solve that? and how can we take the human nature out of it and instead of using technology to really reproduce historical harm and using the technology to really reproduce structural racism. [applause] >> we will conclude with christie who will tell us about how the government uses technology. >> i want to specifically thank georgetown for inviting me to this very important event.
12:47 am
i think it is most critical at this juncture. policing after 9/11 is a way to avoid preventable crime tactics. this term is new but the types of detective work to solve crimes is not. there are enhanced technical capabilities, but there is a need to make sure civil liberty predictions are adequately protected. i want to talk about some things the prior guest have made about the funding. the term policing or data analytics raises fear that the police may engage in legal practices and overstep their bounds and use that information in an intelligence that bridges
12:48 am
the 4th amendment and 5th amendment and privacy laws. they are able to arrest individuals before a crime is committed but making predictions is one half of prediction-led policing and the other part is ac acting on the prediction. it is important to underscore the relationship between federal law enforcement and state and local law enforcement. it is important for the federal government to provide strong policy considerations that protect communities of color and to insure they are not negatively impacted or targeted on the bases of race. there is a need to understand that what information is collected is clear. in december 2014, then attorney general eric holder updated the youth of race policy and this policy builds on the 2003
12:49 am
guidance and confirms that law enforcement agencies conduct behavior in an unbias manner. moreover, and vitally important, the practices are not effective from the law enforcement perspecti perspective. and former attorney general eric holder noted it is not good racket -- practice. in the same, privacy officials need to be engage said in policy developments about what information can and should be shared with other agencies. these responsibilities are within various offices within the department and we take a multi layered approach
12:50 am
protecting civil liberties and rights. as mentioned in 2014, the white house released a report titled big data, seed seasons opportunities, preserving value and this was the result of president obama's big 90-day data report. this report called on the department of justice to examine the u.s. of predictive policing and the technologies. as we know, the data is only as valuable and as good as it is collected. if you are relying on older data and not raw data that is actionable you will not get the results law enforcement needs for their mission. one of the things that was important was this report within this task and that was the department established guidelines for the use of state
12:51 am
and local law enforcement because a lot of this is targeted at their level. the federal government can't specifically tell state and local law enforcement how to conduct business we can establish conduct policies that can be used as a milestone or a model for future policing activities. thank you. [applause]. >> thank you, everyone. i would like to start off by asking a very blunt question. and that is are predictive policing and sentencing racial profiling by another name? >> just the short answer to that i would say yes. there are two parts. the first one is the fact it relies on old data that is not neutral. that is the crux of it.
12:52 am
the other one is very similar to what was said around we have these algorithms that are propritory so we don't know the factors and how the maps are being created. so because of that there is a lot not there. between the two of those it is. it is being able to take communities that are already overpoliced and finding new reasons and neutral reasons to police them again and police them more. >> i think one thing we have not explicitly talked about is the need for transparency. when president obama took office one of the things we mandated was open government, open data. and i think the government can do, if a real concrete way, is to require within the federal government certainly, but basically encourage the state and local law enforcements to
12:53 am
have some of privacy policy and impact assessment that let's the public know how the systems are developed and what is the risk of using that data so there is fair impact with respect to different communities being targeted or not being targeted. >> and sonia and andrew, would that persuade some of your concerns? >> i think with transparency, the key issue we need to focus on, is that we can begin demanding those. there is not transparency right now. there are certain proprietary technologies that don't share the information. and there are some that are recognizing they need to and they will. we think perhaps competitor advantage might win out. the police that control the data are fought giving it up. -- not.
12:54 am
it is easy to label things as racial profiling where it is more complex. you cannot remove race from policing or the reality but without the transparency we are left with those labels. i think it is important for the federal government to lead the way to demand that transparency so we can see what is happening and pull it apart and have data scientist look at it and look at the consequences. we don't have that now but need it. >> i think -- can have their parole denied on the bases of alleg alleg algorit algorithms they don't have access to. i don't think transparency is enough. there is a substance problem on which the bases the decision is being made. and telling people what the
12:55 am
bases is is part of being able to allow them to make the arguments about what is wrong with it. i think it is racial profiling but not literally in the sense race is not included in the instrument. it is certainly profiling and profiling on the bases of a bunch of factors that are inappropriate to use. i wish people would use the phrase profiling more often whether you attend the racial or not because instead when we talk about these predictive technologies what tends to happen, especially when it is in the discussion of progressive policy reform is we use euphemistic language and refer to the sentencing predictions as evidence-based sentencing.
12:56 am
that is a euphemism because the one thing they are not based on -- did that work? [applause] >> evidence-based sentencing -- the one thing thes predictions are not based on at all is the evidence in the individual's case. the evidence comes from a program. and i think they take a process that is inherently subjective. and they make it scientific. but, you know, the language shouldn't blind us to the fact that we are in fact like calculating these scores based
12:57 am
inputs and if you had a judge saying i was going to give you probation but your parents went to prison and you only make minimum wage so i am sending you to prison would have people in the courtroom gasping. and when we do this assessment with a risk score we should have this gasp and i think it will help for us to refer to it as profiling rather than these sort of technical terms. [applause] >> can big data alone every justify a stop? >> i say no. i think what happened is our 4th amendment suspicion for stop is
12:58 am
factors that warrant criminal activity is afoot is a pretty loose and weak standard. it is a totality of circumstances which takes into account anything you want. you can imagine being that officer on the street and you have been told to go to this block and be on the lookout for burglar. there is a person holding a bag and normally that is not a crime but the algorithm told you to be on the lookout for burglars and burglars have bags. can you stop him? that is the question. and i think the answer to that is no. i hope the answer to that is no. but the courts will have to wrestle with that. how do you conceptualize the algorithm tips? it is based on someitate -- some data but it lacks a nexus of what you saw and the officer herself her himself doesn't
12:59 am
understand the bases of it and yet it has been a justification for stops but we have not seen the case yet but it will be a difficult question for the court to decide if this is enough and my hope is the judge will review that and say no but they would have to have read my article. ...
1:00 am
one of the things to remember is that there's a lot of information we already know. we already know where poverty is in cities. we are ready know where there are bad schools. we are ready know which communities need helpful. we don't need big data and new algorithms to figure that out. the question is why and to whose prophet are we using these tools we already have answers to these questions. >> i don't necessarily, i can't speak to the benefit part of it. i do know that one of the goals was to really challenge the federal government to come up with concrete benefits for the
1:01 am
use of big data. as you know there are some benefits to whether or not it's a health context or social work in terms of understanding someone's medical history, someone's education and those sorts of things. i do think a lot of times the discussion is cast in a negative and i think society defends it. >> in the correction system, sometimes very similar algorithms or the same algorithms are used for something called needs assessment in which they are come up basically it's a more benign purpose in which essentially the profile of a person is used to match them to services that they might need. now, the problem with that is the insights that the data tell us in this case, i think that
1:02 am
really have been long known and you don't really need to stamp a number or a score on a particular individual to know. right, we know, if it's true for instance that unemployment is a financial predictor of recidivism. we can take that insight, which there are decades of research supporting at least some strength of relationship and say yes, provide job training to people in prison or job placement services on reentry. those are things that you don't actually need to do, to assign scores to people to do. likewise, substance abuse treatment, probably the people who have addiction problems need it. this is not rocket science. but there's obviously, i'm not saying there is no potentially beneficial use of gathering data that people, that can help with the reentry process.
1:03 am
i'm not hostile to the use of data. i'm an empirical researcher on the justice system. i think we need empirically tested things. i'm just not entirely sure that scoring people is really the the way to go. >> i think that's the public health approach to violence is not something that should be discounted. i think it's been shown to reduce violence. there certainly youth violence. i don't think it has to be connected with police. i think the people who are most likely to be shot or to shoot someone can be, intervention can happen on the community level without police. i think that is the difference. it doesn't have to go through the police although thoughts perhaps where the funding is and why it happens. you could just have the social worker in the football coach knock on the door and tried to reach the same place of identifying the young men,
1:04 am
primarily men of color who have the most risk of getting shot. in chicago, there are a lot of young men of color being shot. too many. something needs to be done. if the identification process has to be the same as the remedy involving law enforcement is the question. >> we will open it up to the audience. >> my question is one of the ethical concerns of allowing the state to essentially experiment on individuals, including these are people who are getting locked up. one of the ethical concerns of doing social experiments? >> so i would actually say it's really great. one of the things that came up as you were talking was the use of science and data.
1:05 am
when we think about the history and especially as it relates to black people and people of color, the use of science and data has been used for hundreds of years to justify the continuation of white supremacy and for me, this being able to take the people out of it and being able to make it more neutral by the use of buddha and data and big data analog rhythms , it continues with the use of science and ethical things that can come from that. >> does anyone else want to speak on that? >> there are certainly ethical questions but police have been collecting data on individuals forever. you can trace that back to the 18th century and see that. there are ethical questions. there are folks that take
1:06 am
seriously some of the concerns because you can hear the response of this idea about the future minority report is not minority report. it's something more sophisticated but less sophisticated, i guess. it causes concern in communities and probably for good reason. >> so i think the question, when you phrase it as what are the implications of experimenting on people who have no choice but to participate, that sounds bad. really depends on what you're talking about. i think there is actually a role for experimentation in the justice system when it comes to police trying a new method. if there's nothing objectionable about the method but they're just randomizing it across
1:07 am
precincts is a pretty good way of testing effectiveness. likewise in corrections, if you're going to assign people to drug treatment, you're not really sure which drug treatment program works better, randomization is a good way to assess those things. but that is a service to help people and presumably it has informed consent. so there's experiments and experiments. i don't the facts you are collecting data on something inherently means you're doing something bad. [inaudible]
1:08 am
they have the along rhythm that predict things that have been correlated is that bias? >> that's a good question. i think one of the things is sort of this notion that race should be a factor in there have been policies on many years on the use of race and it's intended to not use race as a basis for any particular vision on individual. we have the office of justice program and other entities that give out but they are very interested in training and
1:09 am
working in the same way they do to deploy these technologies and working in the aftermath of ferguson with police departments to specifically understand and identify but some of these officers, whether or not, they may have preconceived notions of different groups of people. i think training is one way to address those issue but any system that is based primarily on race would be unlawful under the constitution. >> you want to respond? >> okay. >> thank you again. so when you're talking about big data methodologies, i think increasingly what you are seeing is not that method but big data type things like random forest
1:10 am
or neural networks. the difference there is that those are fundamentally designed to not be interpretable. their black box. as a practitioner i know input and output but i don't know how the waiting was determined. what is it mean to make something like that available to the public when even the professionals have a hard time interpreting what exactly is happening and how it's working? >> that does make it even harder. the encrypting methods that are being used in the process to tend to be a little bit more old technology. they're not really big data. their studies of a few thousand offenders using very traditional methods.
1:11 am
those get translated, but that shows the effect that each variable had. you can transported into a risk score, then the probation officer can easily calculate each calculate each risk after. so that said, there is a bunch of work on how to get better prediction using machine learning techniques. i think at least you could, it may be harder to receive transparency on that but maybe you could make available what the factors are.
1:12 am
i don't know. i have to think more about it. for me because i'm concerned with the substance of whether the variables are being included are appropriate, to me it doesn't matter if you feed inappropriate values into a regression -based system or whether you feed them into a machine learning base system. either way it's like garbage in, garbage garbage out. those unconstitutional variables in unconstitutional risks are out. >> we have time for one more question and then we have to wrap up. wrap up? okay. thank you all. [applause]. >> that was a really interesting panel. you know when we think about the struggle for racial justice and how advancements have come,
1:13 am
sometimes they come through the law like by winning court cases and lobbying and sometimes they come from taking it to the streets. when you want to know how to take it to the streets in d.c., u.s. jane. he's going to give us a tutorial. you say that chief lanier is thinking about using some sort of predicting instrument. let's say that we don't like that. how could we make our chief abandon that? how could we make our mayor tell her to abandon that. we want the answer to that question. u.s. this over here. ps eugene, he is a black live smatter activist. he is an author. he he has had a number of successes.
1:14 am
now he is going to tell us how to do it. [applause]. >> all tried to talk loud. is my microphone on? can you hear that? good. fantastic. first of four for most, thank you. i'm extremely grateful to be here. that's a great opening example because chief lanier is looking to work on the issues here. but set that aside, were part of the black lives matter ecosystem folks in this movement would agree to this degree that liza called business as usual? we live in capitalist society and we need a certain amount of stability to work. that's why it's business as usual. if we really want to force
1:15 am
change, we have the push disruption. it has its own uses and we have used it in ways in washington d.c. in ways that have been effective. it's essentially, here's an issue and we have all these institutions, how do we just tweak them just a little bit. how do we cut out that scar tissue and take it out so much of what has happened is a lived experience.
1:16 am
these things aren't actually separate at all. they are coexisting with each other and they aren't seen separate because of the way we interact. we don't really need a scalpel in some ways. we .2 issues that are not just about how to we use the law, but is the law itself, the constitution itself, even adequate? you have civil right guarantees but not social and economic rights. is that even adequate in the 21st century? please of the questions we are asking. i think the answers are entrenched in big societal issues. we really have to act as an instrument that can disrupt business as usual to get big conversations going. they don't actually take place the way they are. disruption is one of the keys.
1:17 am
it can happen in a lot of different ways. you have what a lot of people are thinking about, yesterday bill clinton got into it with the black lives matter folk. obviously this is something he never thought he would have to account for post- presidency now he's being challenged in the direct way. people are revisiting it and talking about it. that's one. i think in some ways of the more acceptable one but it's an easy one to talk about, disrupting politicians. there is also another layer of disruption that exists and that's what we saw in ferguson. that's what kicked this off with people rising up in baltimore. that led to a lot of realities. the difference between the two is one is a spontaneous engine spontaneous action.
1:18 am
the other is difficult to control because it does not appear to be plan. when we disrupted the mayor we didn't just say were going to go disrupt the mayor because real by the mayor we work climate and scream, what we said as we have no ability to get ourselves into the conversation until we are able to get into the conversation. the black lives matter was the many thousands of people that we could find. we wanted to iterate on the backend with arm information and then assess the background of political policy.
1:19 am
another crime bill has passed. it's all about the public health approach to criminal justice previous question. we tried to take a blunt approach and find ways to be aggressive. we are asking people to report to us what they see and what happened in the videos that they have. we will map it so we can push back. we'll try to complete a community of people. we start to create a community and a feedback loop where people are as dependent on the government to hold them accountable.
1:20 am
we are learning what it means to have lease and have goals. this is a movement. a static movement that is acting that's our approach. some level of disruption to start conversation. we won have campaigns and political practices that allow us to come from behind and engage in our conversation once it's already started to give it direction and hopefully be able to organize people to succeed. thank you so much. thank you for coming and the opportunity. i just want to open it up for folks to just go with whatever you've got. [applause]. >> thank you so much for breaking it down. i have a couple questions.
1:21 am
you're speaking to people who are studying these questions from an academic perspective. what structural recommendation do have beyond having conferences like this to examine and better connect the policy arguments to the consequences of people on the ground. >> i think that's a good question. i think one way is that we have to, how much can an institution do this. i think what's interesting, if it's within the institution, it still without the institution. by their own initiative they're reaching out. how how is it that faculty members and institutions can push forward organizations and get the resources they need to help make the connections and create the spaces to bring people in. i think it's the only way it can really happen is for people who
1:22 am
really feel the movement and are engaged in the movement to move the institution forward. i think there are ways but on the same token sometimes the institution is going to be resistant and i think that's okay. i think the importance of understanding that's okay and sometimes it can all come from this is also important for us. my question, what you say about the movement for black lives and how you're dealing with the totality of issues that are very deep and very complex. my question for you is, were sitting here talking about surveillance of communities of color. many of us are thinking, how can we actually make a difference in this area? my question to you is is working on an issue like surveillance helpful. i can see it potentially be
1:23 am
coming one of the other and i don't know what the percentage perception is. >> i would say it's helpful. i think from the point of view in general, big thing, all the time people stop me on the streets and they say man i love what you're doing but just don't get killed. >> i think we need to understand something even when it's challenging it's easier to deal with. i think pushing back on these practices which are clearly inappropriate. the government occupied several of these. these people are completely nonviolent.
1:24 am
we actually have to surreal them even more because we have to prove that their violent. we have this whole range of political surveillance. i think the work to try to push back on that and understand that templates buy lights on that is incredibly key to our work and help others understand what they're going up against. i think there is an important decision about their own ability as well [applause]. >> one more. so were working on surveillance, i just have a question where are we standing or failing to explain the nuances of this in a way that connects.
1:25 am
[inaudible] to the actual impact that it has on activists and how can we connect those two stations in a way that allows the activists to do their reports more effectively and more safely? >> i don't know if you are feeling. i think most people who are activists actually know about that. they try to learn about what's going on in the patriot act and what other things are going on. it's from the point of view of what we do. i think there are connections that can be made in more information that needs to be put out there about what types of political surveillance are going on. i think we scratch the surface. some of that information getting out there will help but i think a lot of it is succeeding in getting out there. for this example, every black lives matter activists is using
1:26 am
the knowledge of encryption. there's groups that are going around giving people trading on things like that about all these different things. i think there is a high level of consciousness. keep doing the work you're doing [applause]. wow, how about around of applause. applause. thank you all. [applause]. >> so when activism movement for black lives are using encryption, are they being paranoid? that's a question that we are going to take up now. we've heard a lot about the history and now let's talk about current events. let's talk about whether and what the fbi, the department of homeland security at your local police department is doing to activists like eugene? what are they doing? to start us in this conversation, we've been talking a lot about data.
1:27 am
later in the afternoon we will have real data from people who have done all of your information requests about black lives matter. they're going to review some of your findings. but now a history lesson that starts with september 11 in our professor will be a real live professor. he is associate professor at texas a&m. she teaches at the area of national security and tort. before she joined the academy she worked other places in the department of homeland security. [applause].
1:28 am
there we go, alright. thank you so much for inviting me. it's great to be back in georgetown. i was an adjunct here before i went to texas a&m. it's a great to see professor butler and many other colleagues. i'm in terms in the turnout and diversity of the turnout. what i want to talk about today, much of it is in an article so i can't go into the details, but what i want to talk about today is the way in which the relationship between the war on drugs affected the war on terror and the war on terror is now boomerang back into what i think is still a war on drugs and drugs is a war on the blacks and muslim.
1:29 am
i realize that's a provocative statement, but i think if you look at the way that these so-called domestic wars are being enforced, the numbers in the data is clear if you look at the names of them are to. in britain, after they had experienced terrorist attacks, the americans adopted this quote in said let's see if we can copy the brits. they have essentially said why don't you use the communities to defeat terrorism which brings up a ideas of community policing and the war on crime and war on drugs. so, just to to give you a brief history and we had a previous speaker who talked about this, the the african-american community, in my opinion, there's a lot of truth to it.
1:30 am
it is the base line of understanding race in america and discrimination in america. we need to understand the history in order to see how discrimination against other communities are showing up. they're not necessarily identical but certainly connected. in the 60s and 70s, and this is part of the surveillance against rodney king and malcolm ask, and other political groups. what we are seeing here is very connected. it's the same law enforcement agencies. it's the same laws. they've been expanded after 911. they are effectively the same infrastructure. they're just targeting different communities and that does not mean, and we've seen today that those cities are not run.
1:31 am
we've also seen to expand the left-wing political group, particularly with regard to surveillance which is the hook that leads to the prosecution and incarceration. the derivative effect is the chilling effect in terms of political dissent through engagement and activism. i put it? there because if you ask any police officer in a on studies and they show that police officers at the local bug goal level have said if you want to know where the biggest threat to homegrown terrorism, it's actually more from right wing political groups that are white supremacist groups and militants and domestic militants, people who are, some of them who are
1:32 am
completely protected by what they do in the first amendment. ironically, that is not where the law enforcement resources are being targeted. in 2009 there was a report that essentially argued that homegrown white wing extremism is a problem and something that law enforcement needs to focus on. if you go and look online, the response and the backlash was quite notable and dhs had to withdraw that report. so what i look at specifically is countering violent extremism. i don't know how many of you know what that is but it's a big term in d.c. when i wrote that paper, this wasn't the case. it means that the industrial complex has officially taken off and there are now financial interests in the academic sector, in the private sector in the government sector. it's something you should be familiar with. it's very connected to what has been happening and what will
1:33 am
experience to african-americans and other minority communities. so you have a policing infrastructure, and i was very glad to see the fbi general counsel here because i took note of the fact that the laws have changed to some extent. one could argue for the better for the civil rights perspective and some could argue for the worse. just because the laws have changed to create more clinical and legal rights. they may have overstepped the laws in the 60s and 70s, but be careful, sometimes the response is just to legalize what they were doing. it's no longer extra judicial or extra- legal, but it is the same action as, so after 911,
1:34 am
information sharing. there is information sharing between all of the federal agencies. it's all there and it's all very secretive. you have dhs, dod, the police department, police department, and all different types of security department. at this point in time, we're pretty confident they are focusing on muslim american communities but again, it starts with one particular group and then it expands. you also have an entire infrastructure and these are state entities for the police department essentially produce reports. they are not necessarily related to criminal activity but they are just the form of spying. they do ask businesses to do the
1:35 am
same thing in suspicious business reports. it's all tied to national security. many businesses are afraid to push back because they don't want trouble with the government. there's the politics of counterterrorism. even members of the public are told, if you push back and defend civil liberties and there's the terrorist attack, what is on your hands. everyone becomes very risk adverse. you have police working with all these programs. the problem is it's very opaque and there's very little oversight :
1:36 am
>> >> for the drug cartels. this is not as new as the thought it was and these are some of the things that the federal agents are doing. many are things that have been happening or will continue to have been the use of informants or undercover agents are staying operations to have
1:37 am
ethnic mapping that is institutionalize the fbi will tell you this is not racial profiling we're trying to understand the community to know what we police but if you look at the other end of the black box to see the prosecution the identities are consisted of a group it is hard to take that statement as the good faith statement and we have seen through litigation of mass surveillance of muslim communities. many student groups had informants and undercover agents to infiltrate them with the intelligence data base and at that point aside from the privacy issues that is a way to start targeting people that leads to of
1:38 am
parma of people's rights once you're on their radar it is hard to get out a you don't find out intelligence too late to. this is not similar to post an 11 counterterrorism but to show you how fed 11 has effectively pushed the war on crime paradigm on stair raids to give it new breath and money and energy so all of these individuals have to get paid end the way they get paid is 90 plot we find those liken california was anglo who converted fbi
1:39 am
informant he converted at the mosque in essentially went around to create a terrorist plot with the people of the mosque the mosque reported him to asea fbi and said this guy is crazy there is something wrong with him. the fbi disregarded it and kept working with him have you found anybody? maybe you have a relationship with the woman if she can tell you something. very illicit activities he turned on them ultimately and sued the fbi and told the public and how he bush is paid over six digits a very dirty game maybe that is the exception of the commandeers do we say that? of course, then there is "the patriot act" this is
1:40 am
like three or four weeks of counterterrorism but that changed and expanded the authority particularly of the fbi significantly prime rarely by lowering the standards which they could get these types of warrants. instead of relevance to be indicative of or a purpose rather than significant purpose than that does mean a lot for the lawyers because of that higher burden. this is the business record this week in peak you get a warrant but you don't have to tell the person you are searching their home menu comeback and then you tell them but the roving wiretaps is essentially is not connected to a particular phone or a device but allegedly it is with the
1:41 am
persons you can see how that can become very broad. the national security letters that are subpoenas that existed before and 11 but the patriot act expanded them so they don't have to be as relevant. the way they are usually issued to banks and businesses and third parties if they persuade them it is relevant is to include the name of a muslim sounding person on the subpoena that they say we will cooperate. so there is a lot of racial in this process and then you have the of fisa court i appreciated the general counsel's comments about the court however it is far from perfect because it is a
1:42 am
secret court the complete exception of the american legal system. you have judges to get an application from the fbi and there are internal checks within the department of justice but there is no adversary no one on their side to say i want to challenge it to it is a secret court you have the judiciary sometimes they say it could be to differential they don't want the blood on your hands am not sure i am convinced the fisa court is a good check if that is what you look to solving the problems. you also have the threat
1:43 am
assessments selling the fbi to follow you physically coming interview people and ally who they are and without anything relevant it can be on a nonaccrual basis they're very problematic the way they can focus you may know from the news there are tracking devices and of course, we know about the militarization of the police. so these are ways it has expanded their power. what about guidance on the use of race? >> in 2003 under bush there was the exemption for national security. if it was related the fbi have law-enforcement cady
1:44 am
use race not individuals with this particular suspect but in general. eric holder change that and said we will apply the guidelines to national security and intelligence and expanded that to include religion and gender identity sexual orientation but the hypothetical is as isis tries to recruit muslims that gives us the basis to keep an eye out oftentimes that is the argument we know there between 15 and 25 dead from the middle eastern country originally that doesn't give those communities of lot of assurances to be based on their identity.
1:45 am
all of that is to say we will push back. is the adversarial prosecutorial criminal law enforcement approach not rehabilitative it is punitive so there is a movement now getting the language of community policing. but i think it is a wolf in sheep's clothing it is a euphemism without a distinction. so from the doj policing security have a program just expanded you already have the infrastructure. this is assuming that it works there is a debate if
1:46 am
it works but this is the more traditional model. one thing that is different from the war on drugs if working with the police to protect the communities it could be a gay or a drug dealer or people in the community so they're working with the police with a common interest. the incentive for the communities is to beg them not to infringe on civil liberties. that is why they participate they say stop profiling yet the airport stop selective supporting us in our religious leaders.
1:47 am
that was called communities engagement at the time but there is all whole different purpose and conflict of interest. this is the distinction without a difference all of this terminology goes back to counterterrorism so for that reason i seem to be suspect with the narrative of let your guard down and trust us and tell us about your communities you'll be better off from a civil liberties perspective. but there are, the problem is they talk partnership with the act adversarial. it depends a what they can improve and how they behave the go to the hour reach meetings and gather information we have no way to know if it goes into the
1:48 am
intelligence data base but talk is cheap. prove that are there any laws that prohibit that? there at the table i have witnessed situations in boston someone who was in the of reach and in prosecuting a muslim individual that was participating in the efforts then they grabbed him for immigration issue is a huge conflict of interest but they say what is the problem with that with a straight face. then tactics and false statements to the little meet with people as they meet with them in a nonthreatening environment then they give voluntary interviews to individuals
1:49 am
because it don't terrorism to happen in the united states and isn't necessarily related to terrorism. it turns out he went to visit his mother but was scared to tell them. if he cooperates if he is willing to be an informant they will drop the charges it is high risk now we see a troubling trend. and with this application for naturalization. so it is the al capone method. and to come into the community it is very problematic.
1:50 am
set-tos skip those civil-rights implications. i will talk about the assumptions. the vast majority of those getting gauge of terrorism recently if they have come or part of the safe operation there is a small minority those that were born and raised your. in there is more numbers of home grown terrorism suspects not related the
1:51 am
heather assumes there freely operating if you infiltrate you will find them. the user to are victims of state operations that get them to go to the middle east. the parents have no idea that families don't know so it's not even an effective and then the community is collectively responsible for the punishment. now were the fbi is proposing the muslim
1:52 am
communities to put them quit intervention there is absolutely no safe harbor in the gauges in the terrorist act. to the prosecutor civilly? and doesn't end up putting you in jail. seven to stand up for your rights if they find out the fbi betrayed your trust. they will not. this is the apparatus and it is very entrenched this is something you should get to
1:53 am
know more about. [applause] stick just want to say as we transition to the next panel to reiterate when you hear muslim a very large number of african-americans there is a significant overlap to treat out there right statistic about one-fourth are african-american. i am tremendously excited and as i mentioned a lot of philosophers have done a lot
1:54 am
of thinking but yet so few have realized the nature is always risky but i will for this purpose. of that society in the way the police surveil the area is the area they're working. one of the unfortunate part of the trail because of modern surveillance. we're joined by professor brown the assistance of the most powerful work to update the overture to asks a
1:55 am
question to basket interesting group sharing this is my co-host professor butler. >> the fbi so we went to go a little deep to that theory to the next. and to check out surveillance.
1:56 am
as day visual technology to ask a simple question how to the blind as he raised? the way that we ender's the increase and what it means and he will tell us what he has discovered. >> it is good to be here and part of the panel. to be across america this is part of a problem. for those who were allies
1:57 am
but now the enemy. with those racially in distinct groups they were inherently suspicious and it is a matter of national security. and they said with exacting precision to tell the difference between the enemy. and then also to with the bodily descriptions. with that relationship with the race still at the political nature how they're seen as visually obvious because those circumstances
1:58 am
change the way that you see it. those that are collecting information better visually obvious. but from the example of world war ii especially with race and human behavior is very little to do with visual perception. but to be self-evident to see race that may come from something other than with visually obvious. to exploit this idea i conducted a series of interviews that were totally blinded ask some of their understanding of race. it represents the first time anyone understood the blind community. is largely assumed it is significant to blind people
1:59 am
as the genesis that is thought to be the utopia and with fat and ability to see. however some spine people are part of the same social environment they may be able to speak to their understanding of race. with those that our less accessible to them that sharpens our understanding and more over they can provide insight for social conditions that make it possible. so the target population that have been totally blind
2:00 am
since birth to also used as comparison cases i interviewed over those there is snowball sampling that was prescribed by a third-party. the first up is the to common-sense approaches for some people have an understanding of race and they think those of have a diminished understanding some of the interest of time it is clear people they get is obvious and isn't important to blind people. to put simply race is understood visually prettied
2:01 am
is associating race with skin color. many of you may be reaching the reflects the general awareness but the finding should goes much deeper with the visual understanding the way they think about it in their response rate. the brilliant respondents it is a way to divide human beings. another said race is skin color. . .
2:02 am
2:03 am
2:04 am
2:05 am
2:06 am
2:07 am
2:08 am
2:09 am
2:10 am
2:11 am
2:12 am
2:13 am
2:14 am
2:15 am
2:16 am
2:17 am
2:18 am
2:19 am
2:20 am
2:21 am
2:22 am
2:23 am
2:24 am
2:25 am
2:26 am
2:27 am
2:28 am
2:29 am
2:30 am
2:31 am
2:32 am
2:33 am
2:34 am
2:35 am
2:36 am
2:37 am
2:38 am
2:39 am
2:40 am
2:41 am
2:42 am
2:43 am
2:44 am
2:45 am
2:46 am
2:47 am
2:48 am
2:49 am
2:50 am
2:51 am
2:52 am
2:53 am
2:54 am
2:55 am
2:56 am
2:57 am
2:58 am
2:59 am
3:00 am
3:01 am
3:02 am
3:03 am
3:04 am
3:05 am
3:06 am
3:07 am
3:08 am
3:09 am
3:10 am
3:11 am
3:12 am
3:13 am
3:14 am
3:15 am
3:16 am
3:17 am
3:18 am
3:19 am
3:20 am
3:21 am
3:22 am
3:23 am
3:24 am
3:25 am
3:26 am
3:27 am
3:28 am
3:29 am
3:30 am
3:31 am
3:32 am
3:33 am
3:34 am
3:35 am
3:36 am
3:37 am
3:38 am
3:39 am
3:40 am
3:41 am
3:42 am
3:43 am
3:44 am
3:45 am
3:46 am
3:47 am
3:48 am
3:49 am
3:50 am
3:51 am
3:52 am
3:53 am
3:54 am
3:55 am
3:56 am
3:57 am
3:58 am
3:59 am
4:00 am
4:01 am
4:02 am
4:03 am
4:04 am
4:05 am
4:06 am
4:07 am
4:08 am
4:09 am
4:10 am
4:11 am
4:12 am
4:13 am
4:14 am
4:15 am
4:16 am
4:17 am
4:18 am
4:19 am
4:20 am
4:21 am
4:22 am
4:23 am
4:24 am
4:25 am
4:26 am

262 Views

info Stream Only

Uploaded by TV Archive on