A searchable audio archive from the 2013-2016 legislative sessions of the North Carolina General Assembly.

searching for


Reliance on Information Posted The information presented on or through the website is made available solely for general information purposes. We do not warrant the accuracy, completeness or usefulness of this information. Any reliance you place on such information is strictly at your own risk. We disclaim all liability and responsibility arising from any reliance placed on such materials by you or any other visitor to the Website, or by anyone who may be informed of any of its contents. Please see our Terms of Use for more information.

Joint | March 17, 2015 | Chamber | Health and Human Services

Full MP3 Audio File

Thank you, members of the committee, members of the public. I'll begin this morning by recognizing our pages. First, we have in the Senate Deshonda Brown. Are you here? Okay, in the back, sorry. Senator Robinson, and Sierra Cox from Gaston with the speaker coming in. Our Sergeant-at Arms today from the House, we have Young Bay, Bill Morris, Jim Moran; and from the Senate, Marcus Kitz and Dale Huff. If we have any other comments from members or the chair, the Chairman or others today? Seeing none of those, we're going to begin today with a presentation from the Pew-MacArthur Foundation. We have Gary VanLandingham and Josh Waters presenting here today. They have some investment information and other things that we're looking for in operations, and we look forward to their presentation. The floor is yours. You generally have to push and hold on the mike when the green light's on. SPEAKER CHANGES Thank you, Mr. Chairman, thank you, members of the Committee. I'm Gary VanLandingham, I'm director of the Pew-MacArthur Results First Initiative, and with me here is Josh Waters and Tom Conroy, who heads up our fiscal and economic policy portfolio at Pew. We're very happy to be here to talk about the Results First Initiative. And Results First is a partnership between the Pew Charitable Trust and the John D. and Catherine T. MacArthur Foundation, which are two organizations that are passionately involved in helping government make better choices by using data. And the good-news situation I think is out there is that there's a lot of information currently available on what works. Tens of thousands of evaluations have been done across the country looking at a large number of programs. And collectively, as a country, we know a lot now about what works and what doesn't. The challenge has been how do we bring that information into the process so it can help inform the tough choices that you have to make. Because as you know, policy making happens in real time, and when you ask your staff, "tell me what works in criminal justice," or "tell me what works in child welfare, because we really want to make good decisions," you don't want to hear back from them, "Great question, Senator, give me about a year-and-a-half and I'll read through 30,000 journal articles across 15 different periodicals and look through all the literature and tell you, sort of collectively, what we know about what works," because nobody has time to do that. The good news information is that there is that information out there, and what we've been doing with Results First is making that information accessible and, we hope, useful to legislators to help inform the tough choices that are before you. And what we'd like to do this morning is to talk for about 20 minutes about what Results First is, how it operates, and then how some of the other states have been using this over the last few years to help inform the tough choices that you're facing. In the policy challenge these are things which you'll recognize very well. Everybody wants to make strategic choices, recognizing that there's limited resources the taxpayers have given you, and the issue is how do we invest that money to achieve the best results for policy makers. And unfortunately the way that the budget process works in almost all states, it doesn't give that information. What it tells you is here's how much money you spent on programs last year. And then what that does is it focuses the debate on what do we do to balance the allocation this year to the needs that we see out here. And unfortunately, the budget process doesn't give you information that's really critically important if you're trying to make strategic choices, because the budget documents don't tell really important things, like what programs are out there? What are we really funding? What is in the base? That information typically isn't there. I was legislative staff in Florida for many years before I joined Pew. I can guaranty you that Florida does not have a comprehensive list of everything that it funds. The legislature does not have that, the governor's office does not have that. Most agencies don't have a comprehensive list of the things that they administer, because these programs have been established for years. Oftentimes, choices on what programs are being operated are decentralized, and at agency it's being run by the field offices, or it's being--we have a contract with a provider. What that means is that we don't really have a good idea of what programs are operating out there, and we don't know how well those programs are doing. The budget process typically doesn't have information on how much we're spending on each program and what those programs are accomplishing for us. Now, most states have some performance measurement systems, but typically those systems aggregate information

But such a high level but it doesn't really help you tell what are we exactly accomplishing in each individual program and can we compare programs to say which ones of these really are working best for us and as we're doing results vs. bringing all finding ways to bring an information into the process to give you really can't beat the private sector investment analysis perspective of what our investment choices what happens we put money into those different investment choices in our their best buys out there are things which based on North Carolina to meet population characteristics of programs that are out there that the people that you're trying to serve and we know about the program's what really represent the best ways to invest money to get the outcomes of the one with G4 missing symptoms. All.(SPEAKER CHANGES) Our solution is to bring the seventh in the process to make it user friendly this is being called the money ball approach to government all money ball would be few river back a few years ago was a movie auditors also been a book that was based on the experience of the Oakland A's all baseball team which unfortunately has won 25% of the money to fuel the winning team and a big market seems like the Yankees help but they're the same goal of how to weave using the resources we have, we build 18 to allow this to achieve our goals, FOB warning that the maximum number of games and, ideally winning the championship with 1000 by using data that was healthier in more creative ways and other teams they were able to field a very competitive team with 25% of resources that other teams have and it was worked out very well so this is really the money ball approach to government how can we use data that's out there to help fueled the best team the best programs the weekend when she the outcomes of the one which you four cities what to do a sister walk you through record of 30,000 foot level what this looks like our approach really is fairly basic it starts off with one side and if I was in the base what programs are out here and what we know about the effectiveness of these programs and that it will lead a former minotaur read this is a type of information that we wish we are able to retrieve all and basically this is simply a list of programs and the budget that is out there for each of those programs and this is useful information but it doesn't really tell us how well these programs are offered ample we do next is compare this information to what the research shows about the effect was is information of all these inventions and there are a number of organizations out there that are systematically going through all the research we have on programs in identifying how effective they are in use our national research clearinghouses and there's a large number of exploring houses what they all do about the same thing a big goal for the research in the classified programs five COL Woodworth the evidence is and what they show about the facts in this news programs and what we've done with results first as the first of a barn owl system would do with your staff is to really compare what programs are being operated to(SPEAKER CHANGES) what that the national research shows about the fact is these programs and Liu C is that these research scoring all those have some would differ nomenclature to make this pretty easy to understand what we've done is with color benefit all the best are rated clerk password programs about a defined by the 30 houses are green that shows that there's a lot of evidence of these programs are likely to be successful if implemented appropriately other programs have some research behind them but could there be a little bit more risky investments these are yellow programs is a promising programs begun some research behind and it shows that this is probably a good use of investment choice but not quite as his regrets investment opportunities as programs it is a lot of evidence that they work out for slider some programs out there which states are operating at the research shows don't work of that sells a good idea of what when you reduce the test these things they don't really achieve the outcomes which we thought that the fugitive and to reinforce with some programs alter the state's monopoly next legal research shows make things worse of that actually creates negative consequences for people want to go through them and I believe that is not what in East Asia be doing so what we're doing first working with the staff used to identify what programs they would operate and was no about the effectivenes s of programs nobody is saying what would have really lost do first is to give you some of the core branded consumer reports assessment of how risky use the investment in North Carolina in these programs would typically finding is that usually about 1/4 money that the state is operating and independent policy area is being spent on programs that there's a lot of evidence that these are the likely to be a factor of about 10% is often spent on programs that are promising an hour for slider someone is being spent on programs that the research shows are defective debate that may have a couple stories behind them but when tested they simply aren't achieving........

Research shows that kids to go for this program have better educational outcomes are more likely to stay in school them or what to do graduate and kids a budget from Playskool we know, better outcomes in light of day or more money you become taxpayers as a tax consumers so on average kid to go through this program, 19500 dollar benefit for the state compared with kids too. (SPEAKER CHANGES)Once the other alternative program healthcare cost and ago down for a couple reasons that the key to go through this are less likely to commit crime the most likely going to get into fights in the show opened ers' in half to beat, Medicaid and expenditures for them and also these kids are more likely to become employed a drug culture rents and therefore IP disc also on average about $400 in benefits come about white to the eldest of the benefits all promote long-term incumbents about $37,000 in benefits for the investment of $3300 or about 0:49 return on investment four for watching state so not a bad way out the best in mighty good outcomes and were able to predict 10 this looks like it works were able to resolve first is to really look of qualls portfolios it's good to know about the program works it's more prone to know how it works compared to other programs and that this is again watching stayed in output looking across programs. For the child welfare and an internal justice is a subset of programs of this assessment kick and it can look at ways to use that there really typically isn't a big relationship between what program costs and whisper projected to which she four a state some programs are pretty cheap and it produced good outcomes some progress a more expensive to produce OK outcomes but on a dollar for dollar basis going through some of the discipline investment approach of analyzing programs for it looks really help states identify what looks like the best dollar for dollar investment and we see that some programs can produce very strong outcome soul All Star (SPEAKER CHANGES) programs also $100 foot through them and watching state about $2000 into an outcome for 123 dollar for term investment for rebel wrist and put into this all it did in comparison parents as teachers and other programs serving children all basically by 1 to 1 record, not a bad outcome of boat is simply not predicted as much for corona basketball the program is putting money into a different program at four for the programs out there which is as I mentioned, don't work, scared straight to use a relatively cheap program called cost about $66.00 and watching status on digital thunder prison for bank of unfortunately does research shows the highly effective program if you're trying to increase crime all be because rather than its caring escape and stay on the straight and narrow sending a child to prison for they basically demystify as prison for the many still here just don't you would like to keep regard to what I did a blend of care they hear things like what you really wanna make money at the thought of doing tax instead, soaking again came to go through this outcome to actively harmed by the program and that's not what states want to be doing so what is really allows states to do is to look across different investment choices in identifying one return on investment would you get by putting money into these programs and really being able to compare about based on what would be able to get from these programs and visual powerful way of using this information into the process and ideally finding ways of targeting might force programs that work better about spending more money to solve social programs to spending the money to scan the system Alps worse things as simply worked better than most it's been doing in the past when a kid Alistair was over to Josh Waterston to talk about publicity but has been used by some other states to to help inform the budget foursomes from pigs.(SPEAKER CHANGES) Again and wrestlers on the senior Susie with universal results for system and mike area from address experience working for state legislatures in the past and as a buzz Ellis and budgets are so on second base with very some are some points. , Zoos share with you some experiences that some oversee partners, magnesium results first approach, you can see on the mound well we are working in 20 jurisdictions right now we're part of 20 for six in 16 states and a pilot project working with four counties and California of Boston season dark, because it really is the mother ship our project and one sewers would vault of econometric model that we work with states implement it was a working in the states, El Nino taken the standard DCO non profit approach and come in and take all your day job to get back to DC analyzing a black boxing comeback in six months with sweeping recommendations of Bowie par with the state be provided beyond the model figuring them say along with other tools.........

Is really to pass the building projects, so the mottled becomes the North Carolina results first model from all the school's first city for North Carolina, may stay in the state 35 both pointed warning of legislative auditor of big thing about its absence, with recommendations of Bowie do is Wes Dworkin states elbowed passing we provide training four DO for the schools, we provide, hoping Allies in entering the results and community knows to both legislators and seawater groups com 47 by the software ongoing Federal assistance and boxes of money we really come back and help your share those results and we'll build a learning community among our states so really are states can learn from each other and using the approach and in the state so far our products and other only about four years now are Severson moved around $80,000,000 in funding from programs in im and policies that it's about sevens are shown out for two projects and policies that person to work so sore share with you some examples of New Mexico is one of our star states are one of the first eighths to one results first on (SPEAKER CHANGES) Mrs. Bobbitt implemented the model and all the old policy areas, it produced a couple of very innovative reports one of them will be called the cost of doing nothing report and what they did was they walked out the criminal justice system and New Mexico using the results first approached things using that's been determined that if nothing was done and recidivism rates remain the same for one cohorts one year of five people release from prison, cost than $360,000,000 over the next 15 years and for relatively small state like a population was like an excellent for Corsi but some of them in a monsoon reducing bats recidivism rate by 10% to say the state millions of dollars and one N supplements who is using U.S. use the outer sole source approached apart around $50,000,000 in funding of the projects and programs that all works projects that two com all this amounts of 41,000,000 has been started four of its pace early childhood programs at least on CU of pregame,(SPEAKER CHANGES) but since we're 43 Com's for literacy programs and for some home visiting programs I was another see working and, also is wonderful for some issue of early on and go really sorry we're sure about with the Iowa deals with Iran to massive I was treatment program so I won't have an idea that the domestic violence freeman program really wasn't working the way showing all been using the results first approached they found stacks of the data showed that for every dollar they are investing in the domestic violence treatment program the sea was losing $3.00 and sell rather than just cut this program because services needed to be provided two, fenders domestic violence offenders of the work of university is developing new program with a new domestic domestic violence in the program summa to develop what works and so, results are positive base Oreille by piling up projects in one judicial district and announcing the projects, see one calm and star Gary likes to share is backed by a project of bat program to program their bomb is actually required in 20 states seven the program that doesn't work 20 states are required by statute to implement , and the other example a share is in Mississippi , as it is a law degree work and no results first from the workfare styles and their ones that appear to Mather, valuation job and though this past session 2014 implement three bills truly support the work of results first personal be implemented was a bill that defines a new space policy making of policy programs research space policy making just to get everybody on the same page is using the tabular in so that the legislative branch one of communicating with the executive branch about these of programs and policies everyone knows what everyone else is talking about it is invariably the bill that required on several state agencies to program in the forties when Gary said,(SPEAKER CHANGES) none of the season working with another season we've ever worked with at a comprehensive list of what they're buying and it's really the first key step 89 and reliever understanding what you're buying with state money is available list of what you're spending on com so is a city pass legislation that requires of four key state agencies suits are developing a list of prominent story of what they're actually spending money on in the final bill of Babel minutes, really gets a job supporting a work by requiring TV reported in shared between the executive branch and the legislative branch stores than just a couple minutes you're just on UW from the suggested model implementation as well as the roles of both are per states and the results first still reeling from all adults com by Washington State drizzling to be the adult criminal and juvenile justice model and see what this .................

Ensues this pity with employment forms the core of the model and if you think about it makes sense because one of the social policies that states implement also lead impacts the criminal justice system mental health policies if those people are assisted a missile with nickel all five issues all will eventually wind up producing one of the criminal justice system similar to build justice policies from some really it's been close the core of the approach bomb that is in a cell in Unicef to run the analyses on the criminal justice system by the water the DO needs to be populated and the bulk and we'll just a small in order to realize the benefits from other (SPEAKER CHANGES) policy areas bob after Doug okamoto's to some point we ought to suggest the states with actual justice child welfare league are other bigger other core components of a model that really affect the benefits of from the other components and finally also a missile also since you spent a full education from some states are working with oracle mining malls will call series of the same time I'm also working with the state of Colorado see this tender doing adult criminal justice juvenile justice and child welfare simultaneously all existed to really get the core keys a model bills as quickly as possible say could work on the other areas as well so the role of our partner states would bar with the state we ask banks, the gates are bob or actually requires us to get letters of commitments from both from one side of your set from thing authors as well as the governor and is the cause the services that we provide state are free of charge we provide a model we provides technical assistance to provide the other fools no to the foresee partners, and so for Hr Investments in helping the states are working with the state's we aspects BBS have one is a commitment from the leaders of all that state doesn't mean that the results have to be used in this means that they're committed to looking at the results in taking this into consideration and the policymaking process, additionally we ask that a policy work group B appointing N. Tom in some states and six legislators in some states legislators and zombies any branch officials other states of executive branch officials depending on ball where the mall and work as housing estates and that really is to direct the work of the staff workers, (SPEAKER CHANGES) Selig said the priorities of what programs look at what all serious a look at and really depends at the time line for moving forward from third as the distaff or group be established with a person who really takes responsibility for providing work forward as a staff from product manager, Tim this week, soulful and working with other states Tom in order to both communicates between B OM the pols were probing the staff or group and a mix of work things on an assembly session,(SPEAKER CHANGES) and finally as the state's collaborate with five's to strengthen the model and Siebel the warning committee of states, we work with our see partners of final whites Beyer policies are and we say those backs the mall developers to work with filling out those particular areas of the model, one of the reason we heard about from foresee partners is the political arena and so we're now working with the losses in suits to build out those areas of the model and sell these are the services that we provide, we provide software retraining news staff repro to provide ongoing technical assistance and using this pope interpret the results and really comply with sheer lessons learned between states com and also extended up to the model of good model is is is populated and is some cause populate the states and the boss is the data and Tom we were the states to build out and really make a specific states centric model and so without I will preclude my presentation and about two questions representatives to thank you Mr. Chairman and I've been an active in politics one and a ticket really paranoid from cell on the entire thread to and no one can skins about his word of mouth and a ball approached and I'm I'm happy it's very promising but I know we have programs that work here mean we are the hallmark of an securing france's Campbell is ours spring training is ours and our programs were built around its daring would we keep the programs that week after working poor are you looking at examples so we use to react education Riyadh model for our O and four are from barely held use that there be heavier, squeaker singer and ..................

It uses that model. It's a model like system of care is a similar model where you look for the students strengths. You identify people you require them to go home every weekend. So you don't just treat them or punish them you build a community where they can exist. So I guess my, sounds to me like what you are providing is a package where you have program that have names like functional family whatever and so is that the same program that we have or is it a different program and would you bring in your programs and we buy your programs and replace ours with what you have. Or do you look at the programs we have and you say your model here is very close to what we have we'd like to tinker with it a little bit so that's the kind of question I had. [SPEAKER CHANGES] Mr. Chairman, great question representative. Couple of answers to it one is we're not here to sell anything. What we're trying to do is to provide you with the best information that is nationally available about the effectiveness of the programs. ?? is in the model because that program has been researched. What we use with the research that is in the model is really the best information that's available. And every state also has program which have been developed have been studied here. If they're not in the model then you can add those programs to the model when North Carolina specific evaluations. So if UNC or Duke or other universities have done evaluations of them and your confident that those evaluations are well done you can add them to the model. Our goal here isn't to force you to do anything. It's to give you tools that become North Carolina's tools. They're your data, they're your tools and the choices of what will be done with them will be yours. What we're seeing is that all states have information that's never been evaluated. I mean it's never been nationally reported. Everybody does evaluations unfortunately because a lot of evaluations are never published that data is never being able to be shared. So as we're working with states we're seeing hey North Carolina has got some great programs we can add to the model, New Mexico's got some great programs we can add to the model and then to build that that national sort of conversation and ability to share information from each other and help all states make the best choices. [SPEAKER CHANGES] One of the things I've talked to staff about before that is a frustration I think is that you can have a lot of research on a model and you can bring the model in but you don't fund all of the components and then you pretend that you have a evidence based model and you wonder and I think lots of times you look ahead and we don't evaluate whether we're getting the results we want. So I guess maybe that's a comment how do you deal with that I mean sometimes the it makes the program we need to save money so we think we can cut out this part of the program and not lose anything. So is it, is that part of your program to promote a fidelity to the model? [SPEAKER CHANGES] That's a very powerful issue. State run into trouble in a couple of areas. One is that investing money in programs the research shows are not effective. The other is exactly what your talked about investing in a good program but not doing it with fidelity. And therefore doing a good program badly doesn't get you where you want to be. We did a report last year which really lays out a comprehensive set of things states should be paying attention to in doing programs. And you know it's not just picking good programs it's making sure that they're being implemented well tracking outcomes to make sure your getting what you pay for using the evaluation of resources to look at programs that look promising but really need to be proven here in North Carolina. And really coming up with a comprehensive holistic way of looking at this. But yes that is a big issue programs needs to be operated with fidelity. There's recent research now which shows some of these areas what are the key things you have to do to get the outcomes you want to achieve. How much can you tweak the program before it's no longer that program. And unfortunately your totally right states when money is tight tend to cut monitoring and things like that. What that means is that after while you lose confidence that agencies will be able to deliver what your giving them money to then to deliver. [SPEAKER CHANGES] I have one more question but I [SPEAKER CHANGES] For follow up.

Do you recommend that if we take a program that is proven and have the data that the studies have been done, that we continue to do our own evaluation of it? To collect data and analyze that? Or is it more important to analyze whether or not we're being honest, true to the model, or both? [SPEAKER CHANGES] Mr. Chairman, Representative, that's a choice for the state to make. I think the question is what's known about the program, does the state have confidence it's being done with fidelity? If you have questions about it than you can direct the legislative evaluation division to go out and do an evaluation because it really is building the discipline into the system that, "This is what we're buying, we have confidence that the agencies will deliver it well, we have an outcome monitoring system that tells us are we getting where we should be going, and if not, what mid-course corrections can we take?" [SPEAKER CHANGES] Senator Tucker. [SPEAKER CHANGES] Thank you, Mr. Chair. Gentlemen, again, thank you for being here. I love the operative word you use, free. We get charged a great deal at this state for using consultants. This question may be for staff, has the department ever submitted to you, Ms. Jacobs, the list of programs that we're funding now, on a spreadsheet, and what the expense is on each one of them so that we could take that information and convey it to the PEW Center and they could evaluate what programs we had prior to them coming back with some recommendations? [SPEAKER CHANGES] Senator Tucker, members of the committee, we do not currently have a comprehensive list of every program in Health and Human Services, the department may have one. We have requested, for this session, a list of programs focused on the ages zero to five that we're talking about this session. They do have a database that they have online that's called Open Windows where they have information on various programs. I am not sure if that is up to date, I'm not sure if there's someone here from the department can talk about their Open Windows database and what it consists of. I will tell you that for a lot of programs it does not have outcome information, that's they key component that we're missing. It has some budget information but it does not necessarily have the outcome information that the committee needs to make decisions. But I would say that Open Windows may be the most reliable source that I'm aware of that the department maintains, but as far as a comprehensive list of programs funded by every source that's within the Department of Health and Human Services, I'm not aware of one other than that database. [SPEAKER CHANGES] Okay, one follow-up, Mr. Chair. Mr. [??], you're sitting back there, you had done an evaluation at HHS, were you able to find an inventory of all the programs that we fund? Meaning I'm just asking do we know where our money's going and whether we're effective or we just continue to keep looking at a number that we spent the last budget cycle and we just add on to it and it must be a good program and go forward. Were you able to find out any real outcome, evidence based evaluation from what your group's work did? [SPEAKER CHANGES] Mr. Chairman, Senator Tucker, we do not have a comprehensive inventory. Now, we've gone in selectively at HHS, a few years ago we looked at grant programs and the non-profits that administered that received those grant funds and we found the overwhelming majority of those non-profits did not have outcome measures in HHS at the time. The previous administration was not requiring those outcomes. Now they have been working hard on contracts with their grantees, but that's a selective inquiry, it's not the comprehensive inventory that you're talking about. As you know, PED is sent in on a project by project basis and we haven't done anything like that, no sir. [SPEAKER CHANGES] One more question. [SPEAKER CHANGES] Final follow-up. [SPEAKER CHANGES] Mr. Landingham, is this scenario that's just been spoken to about the list of inventory, and I certainly don't know all the programs in HHS, and we go through the budget process, is this fairly common when you come into a state? I think I heard you imply that, is that fairly common that a lot of states don't know where they're spending their money when it comes to a lot of programs, but primarily what we're talking about today in HHS? [SPEAKER CHANGES] Mr. Chairman, Senator, that is exactly the

the situation we find in every state that we work in. Typically it takes several months to work with staff and their working with the agencies to come up with that list. That really, I think, points to that the challenge that all states face in making strategic budget choices is that if you don't know what's currently being funded it's very difficult to say, "Well, we should cut this," because there's no information to really target cuts so legislators do what they can do, which is cut things across the board. Which, unfortunately then, hinders the best programs while preserving those that have failed. [SPEAKER CHANGES] Okay, thank you, Mr. Chair. [SPEAKER CHANGES] Representative Avila. [SPEAKER CHANGES] Thank you, Mr. Chairman. I have a question, when you've gone in to these various states, what collaborate group generates the most success? Is it a department similar to HHS says we need issues taken care of? Is it when it's a legislative invitation? Is it when you've got the governor behind it? What is the most productive scenario of support that once we get this information we can turn it into actionable, productive outcomes? [SPEAKER CHANGES] Mr. Chairman. So it really does depend on the state where the work is housed. In our most successful states, the work is either housed in the legislative budget or policy analysis shop or the governor's office budget or policy analysis shop. Really, the policy group that directs the work is very important and for that to be successful you need strong leadership on the group itself and that usually comes from legislators or from executive branch officials. As for the staff work group, the most successful states, the work has been housed in the legislative budget shop or the governor's budget shop. So, really the staffing work group is a key component in that as well. [SPEAKER CHANGES] Senator McKissick. [SPEAKER CHANGES] Thank you. I have a couple of questions and I guess this is for Mr. VanLandingham, in terms of the way you conduct your analysis, you initially start out by saying you look to what's been successfully done in other states, but I"m trying to understand how that would function within the context of knowing that different states have different programs. How do you get the apples to apples comparison, but more importantly how do you account for important variables such as management and staffing? Because you can have very similar programs but if they're ineffectively managed or organizationally they're not structured right, or the resource is not provided, they may appear to be quite dissimilar or function in different ways. So in terms of getting that initial comparative analysis, how do you go about doing it to get a true apples to apples comparison and then customize all of that knowledge to look at what we're doing here in North Carolina? [SPEAKER CHANGES] Mr. Chairman, Senator, the way that the process works is we'll work with your staff first to do the inventory, what programs are there, and then the question is what's known about programs like that? So if it's functional family therapy but with a North Carolina bent, well how much of a diversion does that look like it is and what's the research show about the impact of those, that those tweaks to a program? Then the next level of analysis would be working with your staff to make a call on how much confidence does the state have on the management of this program? And that's important for a couple of things, one is that by using the model, it tells you where a program should be able to be in terms of the outcomes that it delivers. Washington state ran in to this problem exactly like this, the legislature had been investing in functional family therapy and wanted to know, "Are we getting what we're paying for? Your model says that it's a 22% reduction in recidivism, tell us if we're getting that." So they task their evaluators to go out and look at functional therapy to see if they're getting that 22% reduction in recidivism. If, after doing that study, they came back and they said, "Well, the good news, members, is in about half the state we're getting 22%, just like the best research predicted. The bad news is in the other half of the state you're not getting anywhere near that, and in some cases the providers are making things worse because they're not doing this program. They have not hired good staff, they're not managing them, they're doing their own thing and what they're doing doesn't work."

and are you happy with the way that that's working or do you want to put some more provisions in place and put some pressure on management to do the program? I think the strength of this approach is it allows you to tell, "Here's what our program should be able to accomplish based on what's known about programs like this," and then you can customize that by how much confidence you have in management and use outcome data that may be collected by the state and evaluations that may be collected by the state to come up with your own assessments of where we may be. For some of these programs you may say, "Well, based on this program, if done well, we should be able to solve half of the problems this client group has, unfortunately our best data shows were only half of the way there, so what do we need to do to improve our game so that we're able to serve these people more effectively and get the outcomes that we're able to predict? [SPEAKER CHANGES] Follow-up, Mr. Chair. Let me ask this, assuming you sit with staff, you come up with that list of programs, based upon what you know about today about North Carolina, what kind of time frame would you be looking at for being able to conduct your analysis, to come back and provide meaningful, insightful data based upon what is effective, what's ineffective, and what can be done better? What kind of timeframe are we looking at. I guess it depends upon the umber of programs that your inventory covers and what the scope is, but do you have any thoughts about that at this time. [SPEAKER CHANGES] Mr. Chairman, Senator, what we're finding is that in the states that we're working with, they can get the initial components of the model done in an interim period. Now, obviously they can't do everything the model does. The model covers eight different policy areas now, so states need to pick and choose what you want to get done the first year. Typically the program inventory, assuming that you get strong executive branch buy in, which is why the letter from the governor is so important, that typically takes a couple of months to go through that process and then getting data on North Carolina so were able to make North Carolina specific predictions, that, again, will take a couple of months. So, typically it's taken around six to seven months for states to get the initial cut of the model operational in the first couple of policy areas and then expanding into new policy areas over time. But our goal really is that this becomes a resource that's useful to legislature that becomes part of the process and that takes time. Typically, states have some of the data that you want to have, you use proxies for the rest of it, and next interim you clean that up a little bit. Washington State has been doing this for 15 years and they're continually tweaking this model to provide more information, deeper information, and hopefully more useful information to the process each year. [SPEAKER CHANGES] Last follow-up. As I recall, PEW Charitable Trusts worked with us years ago when we were looking at the justice reinvestment act and providing some assistance to our sentencing commission and to help us construct what looked like would be a good way of moving forward with that initiative, and of course data has been forthcoming, it's coming back in to us but the way you work with us on that project, of course it would be different from this one but in what ways would it be similar, in what ways would it be dissimilar. Because I'm familiar with what you did on that occasion. [SPEAKER CHANGES] Mr. Chairman, Senator, what the Public Safety Performance Project does, which is a judicial reinvestment initiative that you mentioned, is that they work with stakeholders in a state to reach a consensus on what's driving the criminal justice populous, population growth, what are options for dealing with that, and then helping build a consensus within the state to pass a law that would implement those choices. Then, after that passes, they move on to work with other states. But it's more of a policy focus. Results First really is much more of a focus on individual programs, how is the state spending money, is there better investment choices the state could be making to predict it to get better outcomes? So it's not based on that policy focus but also then we typically work with states for many years once we start because this work expands. Because it's more programmatic focus, the politics are different. There are some winners and losers occasionally. If you find that a program that the state's been investing in isn't working then you have to deal with that but it tends to be more, probably, staff-oriented at the beginning because your staff will be working to collect this data and to bring it into the process and then you choose what to do with it during the budget deliberations. [SPEAKER CHANGES] Thank you, sir. We look forward

Revere systems surrounds bank in a stereo and I think you are Mr. M landing one years there, and several questions why is condescending to start out with the first thing is I've bent Allied with and the HHS there is some evidence based programs in an era fact one of the things they advocate at the Federal government to one of the things a gel get funding in London City of Atlanta is based not all are in cell nine vehicle issue back to locate your own community and Batman in terms of money you use that for the first question(SPEAKER CHANGES) I can see you he is are in this novel how we can determine went into a retooling clue me at all at opprobrium sinking in Kittitas and get all this round program evaluation within fences and maternal and child health and say we've gotten spurring which are looking at what is already the air in terms of playing as evidence bates outcomes already had been sheer what baby and Cindy M. Ickes chapel a cost based on what those outcomes are at which a determined that because when I look into a model year and you begin stop about adult visa gravity is crying something else and sent me know because their crops in your read across departments when I look into mom was cross several departments in early childhood education is in a climate of education something else is in the kitchen sink their cell Halley to determine what any of that are used to bail them out all of four North Carolina(SPEAKER CHANGES) and then the other thing issues say a 6 to 7 months you can see now comes over six and seven months old at which you collect the data looking at what we have here at one to determine what programs within what which which programs that are collected data from sell you can develop a model bat well B evidence the eighth of Mr. Min senator B in terms of deciding what to look four beat him and didn't see all this is where you have to make some priorities, are you primarily in arrested early education for early intervention program for human arrested him programs which were reduced incidence of toledo's and neglect all and focus that the working on no specific areas of 10 disappear model is based on programs rather than debate and sees so it because water agencies are very broad so we we wouldn't be able to do everything set department doubts about deal for the first thing would be what you care about what you want your staff to be able to come back to you with with information on all within department focusing on that all and then once viewed the Tory has been donned him a false areas that venture of correcting the work begun on what is known about those programs all user evidence behind the more are the homegrown programs do they have local valuation city will come to know all and reporting about that, and infirm so the dead raises on number of Falwell points, but you can consider in terms of your article for those programs are being done what are they parted towards the pick the critical populations that if you only to beat our constant state so it really gives you a way out of four focusing your attention but did it starts with a priority what you care about an apartment that the weekend Potomac and uses pro, out in our approach is very we search oriented spaced on what we know about programs in the fall series promised here and in an ideal respect the work of pew charitable trusts are you know the year Friday at it from, the trust for a long time in terms of health and human services(SPEAKER CHANGES) but my concern still using about the developer mama based on what we tell you we won a look at an issue only be an Al do we use tactical knowledgeable to the next program of minutes they argue training staff here we go to work with their own university assistance and can select as the polls and is that contain NAFTA 6 to 7 last year for E of course you can't sell how do we continue to work with many in terms of making sure where making sure that bears the Galatea to the mile in specially FDA components to web and continue to evaluate programs and real, of miss chairman senator bob we've worked with..............

states for multiple years in doing this and we've been working with some of these states for four years, so we have not walked away from any state. We view this as a long-term partnership because you're totally right, there's more analysis that can be done, there's more programs, there's more data that can be done but our partnership with the state would be long standing to really build this as deeply as it can be done in the state, to answer the questions that you want your staff to answer. Again, we view this as capacity building. It's not us coming in and doing this work. It's working with your staff, whether it's in PED or whether it's in Appropriations staff or it's a partnership with the executive branch, we work with them to train them in what this approach is, provide the tools, and then provide technical assistance to help them do all of these deep dives. But it is something that, as you have more questions, your staff will continue to do more work. [SPEAKER CHANGES] One last follow-up, Mr. Chair. Long-term, you would stay here, is long-term free, too? [SPEAKER CHANGES] Mr. Chairman, Senator, yes it has been so far. We've been working with states, I'll look at my boss, Mr. Conroy, with this but this has been fully funded by the PEW and MacArthur foundations for the last four years. We're confident we'll be fully funded for the next three years after that. Just like you can't commit future legislatures, I can't commit the board to doing things forever but this has been the basis of this work to date. [SPEAKER CHANGES] I think that's an excellent answer and I would only add that if we are able to help the state of North Carolina generate savings by allocating your limited and constrained resources towards programs that work, that generate better outcomes for your citizens, and cut programs that don't work and make your budget decision making life easier, and ultimately save millions of dollars for the residents of North Carolina, there's a value to that that we would be providing to you and perhaps several years down the road, nothing in the short term, but several years down the road we would like to entertain a conversation with you about the value that we are providing and whether or not compensation would be appropriate or whether free is the model going forward. [SPEAKER CHANGES] Senator Pate. [SPEAKER CHANGES] Thank you, Mr. Chair. Gentlemen, it seems to me that someone has to see what the target really is before we let the arrow go. Is there some sort of study that your folks are capable of making with partnership with our government to see how we get to, first of all where we want to go and secondly how we get there? [SPEAKER CHANGES] Mr. Chairman, Senator, as part of what we call the due diligence process, it really is you assessing us at whether this partnership makes sense for the state and us assessing whether we think the state is well positioned to move forward. We're confident that North Carolina has the legislative staff with a skill set necessary to do this. We haven't done a data assessment to get an idea of where the data sources will be needed to populate this model and we'll have some conversations in subsequent weeks to talk to people about that. So it really is a choice that you would make, are you confident with where you want to go with this? We can share case studies of other states that give you an idea of what this has looked like, we'd be happy to give you names of both members and staff in other states that you get an idea of what this has meant to them. But we do really want everybody to be going in to this eyes open because the good work that Washington State and the research clearing houses have done meant that this information on what works is available to us. It doesn't take an army and a state to do this work but typically it's been a staff work group over an interim who leads this. It needs to be people who have the technical skills to do this, who are part of the process, that are honest brokers for you, who can do this work long-term. So you have to make the decision, who do you want to have do this, what capacity would they have over the interim to do this, how do you want to structure this? We'll make a decision whether we think that the state has the data

Speaker changes:to move forward north Carolina is more respected for legislative staffing we have so we are ?? to that level but we want his to be that something both sides get into because you want this to be open eye because you want this to be successful little time partnership Speaker changes:i have Representative ?? Speaker changes:Mr.Chairman i would like to address Mr ?? if i could I'm very frustrated about all this I'm gonna say my history with at and my be you could share some of my rectify i was assistant cabinet secretary ?? in north Carolina state government the governors came in again at ?? one party rule on the conservative platform got the brightest platform in the conservative community hospital governor corporates ?? to do executive studies i was asked in my department in implementing those points so i work with ?? with the nine cabinet secretaries at that time and so i worked lot with process HSS now to implementation of these states i have ever been frustrated in my life these were people these implemented bought in long time state employees very little got implemented so i left the early part of administration i know the early part of doing that much time ?? eight years they have elected another conservative governor and he did ?? studies all that happen is ?? i want more money i want are consolidate and are organize more anything and not knowing t ll if it nor administered i did not work into the process very ?? so bout ten years later i ran into county commission of the biggest population county in the state then i became chairman two years later we did ?? you name it so you did the study votes when they come in i git the ?? and we re determined that we gonna do it with five ?? and we did it we went with 21 department 12 department we did with 105 of the county employees it can be done I'm telling you north Carolina if you leave it up to the governor and executive branch it will be flop we did not meant let them do it we need let them do it we need to tell them that's gonna be funded we have an excellent program general assembly sections with excellent employees or they do the great job they could be the why you call the facilitator,they are great but they cant do these ?? can you help I'm really frustrated Speaker changes:look Mr.chairman Representative ?? that is key problem through the DNA of the government how do we need lines to do the performance that need to happen that is why e both let into ?? because you are totally right the agencies can prevent ?? until something else goes on and we want this to be something else for anyone some of the best works that have been an done and some of the state done because ?? the department of correction realize it didn't make sense to do filed program and this gives the ?? which ideally do the same as the legislative works and that's why the policies based on let's step back the ways on let's look at what re we doing is it likely to work and then directs thing ?? actually I'm focusing on different set of questions thinks that ?? which is very much into the process Speaker changes: Representative former butter-field Speaker changes: thank you Mr.Chair

You talk about states, states have been involved because 5:00 PM comedy, the Waco is up Mr. Members in the Bork early working width 16 states and woodworking was almost 8424 years ago and noticed it to talk about lessons learned about nine eight commemorate with all the states can you give us a specific examples of some of the lessons learned about the freeze to death while all Mr. Mandala representative 56 impalas is warranted a week or use to do some really important to get that the high level commitment from states to do this, we started off by working with some agencies and the challenge was they could do the analysis point that the staff from a digital system made a student of their own bodies stafford loans that the legislature so the report that going places about 41 help but this volume. It did to fix a educational process for agreed to change their thinking disorder asking for questions on what works with identify policy lovers about the legislature's can use to insert move the system court says all for example Mississippi.(SPEAKER CHANGES) Passing a wallet says visible we mean by government so the way this has come forward with no we wanted to win new program no questions are one of Azusa the best likely to work show was the evidence because no one fun things that are seven inspired so you can really use building this into the process of feet before the cube losses were born and we've identified calls to lovers and other states have used across wouldn't unquestioned DD down from outmanned Andre am all for what would happen to 1D6 to seven month. In North Carolina this was done out Mr. (SPEAKER CHANGES)and all we have more plans in other states have used out and then we'll we'll give those to staff the customer call System Works here in North Carolina all together is to be that the ownership of staff level so that is one key questions is pretty one have do this work and then we'll share information with him to tell bill duplication point said Wells made to Mr. Kerr question for any of the presenters are listening to your model it seems like if you applied that model two of which are doing you might be overlooked state government and the size of our government and its population and everything your North Carolina and review that vs. overstates it to work with in other words can you give us a range of potential savings is no throttles HHS Dept is the smoothest operation you've ever saying what would be the savings that we'd say anything says Brian zucchini you ever saying what would be the savings would say enough guts nonmember kitchen range, asking for is there some, ballpark you could give us today or later,(SPEAKER CHANGES) Mr. Murdoch said Europe five hesitate to give a vehicle without the fickle into the left of albeit is very difficult to compare states because states organize and sells differently so just basic things like a leap employees a state as compared with other state defers because some states don't help functions of a given to local governments in some states do so did those comparisons between states are worse are fraught with difficulties so I think it utilities are of your call logic of this worried much more meaningful than mine in terms of accomplishing that would be with a partner whether you think that it's doing the things that should be doing in doing dumb well home center put items to homage to the Willingham since we're picking on you and we'll all slide in the fragment politics is an error here all the time and bob, Landis and other states put everything could change 10 home of the even numbered year which November krstic also did the general assembly is reelected and nor faces elections and the governor of every four years and the entire thrust build a program could change as a result of that they'll be you, they bet in your long-term relationships with the states of miss chairman senator one of things said that the thing support is why keep expectations for reasonable this isn't an approach that comes in and says our goals and take the politics of the budget process may call it a star is the key is how many decisions have to be made because Baltic Sea is how all the divers e people the state have to come together and reach an agreement, spam state's resources NI doing something there...................

To live with no 1 May be totally happy with all so we got one say the state's a poll to kill the process two dozen for the politics eight and 88 new and very powerful way and Wilbur seeing even safer working with many of which have party change to stop the both the second branch was in the brakes with a working with these events will this work continues because you're at the end of the (SPEAKER CHANGES) I think people will argue in a shed about the size of government and give priority to government but people generally agreed that we should be doing things that work and this gives you a way of saying regardless of how much money you want input and HHS for other agencies one what is a research shows works you're all doubt would get the outcomes that the state's elected leadership wants to achieve and you so most essential for the political someone will be based on 55 building this beta ID into the system giving you much more comprehensive information as part of prophecy, (SPEAKER CHANGES) before we see that does survive changes in party missed racial because this is basically makes a lot of sense would also was on the clinical of your own and do the only indirect bob but direction events can prove scores of Ike and Tina Christian Slater I think in any other questions in terms thank you very much for presentation we all look for the continuing the conversation from heavy with the agenda this committee will send a jar of a.............