1 00:05:31,727 --> 00:05:37,050 [COUGH] Get started, okay, 2 00:05:37,050 --> 00:05:44,231 good morning, let's get started. 3 00:05:44,231 --> 00:05:48,788 So today Katie Clift speaking on some 4 00:05:48,788 --> 00:05:53,940 complexity in healthcare data systems. 5 00:05:53,940 --> 00:05:56,937 Kinda continuing the theme we talked earlier about, 6 00:05:56,937 --> 00:06:00,799 the last couple of weeks, of us looking at how systems thinking helps 7 00:06:00,799 --> 00:06:03,408 with complex problems in different domains. 8 00:06:03,408 --> 00:06:07,922 We've had systems engineering two weeks ago and 9 00:06:07,922 --> 00:06:11,893 then system-centered design last week. 10 00:06:11,893 --> 00:06:16,650 And now we're gonna see some of the same themes of how a systems 11 00:06:16,650 --> 00:06:19,320 approach helps in a different domain. 12 00:06:19,320 --> 00:06:24,234 >> Okay, so without further ado, I am Katie. 13 00:06:24,234 --> 00:06:27,295 Little bit of an introduction, by day I work at a large, 14 00:06:27,295 --> 00:06:29,850 highly federated healthcare organization. 15 00:06:31,010 --> 00:06:36,712 I go to grad school in the evenings, on the weekends I'm a soccer mom. 16 00:06:36,712 --> 00:06:38,853 But I'm really excited to be here today, 17 00:06:38,853 --> 00:06:42,782 because it is rare that I actually get to present this to system scientists. 18 00:06:42,782 --> 00:06:46,650 So bring your insights, bring your questions, help me make this better. 19 00:06:46,650 --> 00:06:52,200 I'm kind of close to this work, so I wanna hear from you. 20 00:06:52,200 --> 00:06:57,183 So a little bit of a prelude for the presentation that you're about to see. 21 00:06:57,183 --> 00:07:03,502 I mentioned in the bio that I've taken the scenic journey to my graduate degree, and 22 00:07:03,502 --> 00:07:08,937 I'm targeting completion next term, but I started in the Fall of 16. 23 00:07:08,937 --> 00:07:14,366 And one of the classes that I took was holistic strategies for problem solving, 24 00:07:14,366 --> 00:07:19,727 so that kind of set the foundation of systems thinking, system application. 25 00:07:19,727 --> 00:07:25,134 So fast forward to Spring of 2018, alongside another class I did a small 26 00:07:25,134 --> 00:07:30,640 reading and conference, focused on system approaches to managing change. 27 00:07:30,640 --> 00:07:34,872 And so this is my Dorothy in The Wizard of Oz, and Paul was there, and 28 00:07:34,872 --> 00:07:40,244 Shane was there, and others were there, so thank you for being there on the journey. 29 00:07:40,244 --> 00:07:45,071 And it was a really kind of learning through dialogue approach. 30 00:07:45,071 --> 00:07:49,137 And a couple of the methods that we focused on really resonated with me and 31 00:07:49,137 --> 00:07:51,613 you'll see kind of played out in this work. 32 00:07:51,613 --> 00:07:56,343 They're bold in that list, the Viable Systems Model by Stafford Beer, 33 00:07:56,343 --> 00:08:00,960 Soft Systems Methodology, Checkland, and Critical Systems. 34 00:08:00,960 --> 00:08:05,236 Heuristics by all which, okay? 35 00:08:05,236 --> 00:08:09,170 So fast forward to depart from it. 36 00:08:09,170 --> 00:08:14,608 All of a team that I built this presentation for a deliverable and 37 00:08:14,608 --> 00:08:19,241 follow me which was artificial life Joe's class, and 38 00:08:19,241 --> 00:08:23,588 he asked me to come today, and do a reprise of this. 39 00:08:23,588 --> 00:08:28,428 So what the presentation is really focused on in my work life, 40 00:08:28,428 --> 00:08:33,921 in about this time last year there is initiation of a project to develop 41 00:08:33,921 --> 00:08:38,971 a shared platform for data and analytics in this organization. 42 00:08:38,971 --> 00:08:44,447 So that's really the focus, the focal problem of to which 43 00:08:44,447 --> 00:08:49,711 we are applying these systems methods moving forward. 44 00:08:49,711 --> 00:08:51,986 And the last thing that I think I'll say, 45 00:08:51,986 --> 00:08:56,864 before we move on is I'm currently taking Marty's class, which is called philosophy. 46 00:08:56,864 --> 00:09:01,640 So you'll see some of these ideas kind of 47 00:09:01,640 --> 00:09:06,283 floated to the topic of this, as well. 48 00:09:06,283 --> 00:09:10,012 Any questions before we move on? 49 00:09:10,012 --> 00:09:13,504 All right, let's go. 50 00:09:13,504 --> 00:09:17,063 Large major projects in a highly federal healthcare organization. 51 00:09:17,063 --> 00:09:18,574 What does that mean? 52 00:09:18,574 --> 00:09:21,351 Okay, so on the left, you've got the status quo. 53 00:09:21,351 --> 00:09:24,164 On the right, you've got the vision of the future for this. 54 00:09:24,164 --> 00:09:30,823 Share data and analytics platform, and the status quo goes something like this. 55 00:09:30,823 --> 00:09:34,603 We've got source data in various systems of record. 56 00:09:34,603 --> 00:09:36,122 We've got claim systems. 57 00:09:36,122 --> 00:09:38,706 We've got medical record systems. 58 00:09:38,706 --> 00:09:41,055 We've got finance and membership data systems. 59 00:09:41,055 --> 00:09:45,985 Transactional data goes into these transactional systems, but 60 00:09:45,985 --> 00:09:49,670 they're not great for analytics or reporting. 61 00:09:49,670 --> 00:09:54,851 So what happens is that you've got many disparate teams in a highly 62 00:09:54,851 --> 00:10:00,406 distributed analytics environment, building kind of custom views of 63 00:10:00,406 --> 00:10:06,788 the data to serve their stakeholders who are sometimes the same people at the top, 64 00:10:06,788 --> 00:10:12,182 but not always the same people on top and sometimes different needs. 65 00:10:12,182 --> 00:10:16,929 And so what we've realized is that there's probably a 10x 66 00:10:16,929 --> 00:10:19,822 multiplication of the source data. 67 00:10:19,822 --> 00:10:21,712 There's a lot of duplication. 68 00:10:21,712 --> 00:10:24,043 Yep, there's a lot of silos. 69 00:10:24,043 --> 00:10:26,361 It's really hard to find anything. 70 00:10:26,361 --> 00:10:29,647 And at the top, you've got executives that are like wait, 71 00:10:29,647 --> 00:10:33,213 why does this report say something different than that report? 72 00:10:33,213 --> 00:10:35,580 I thought I was kind of looking at something that was fundamentally 73 00:10:35,580 --> 00:10:36,137 the same idea. 74 00:10:36,137 --> 00:10:44,387 Yeah, yeah, so t here in lies the problem of the status quo and 75 00:10:44,387 --> 00:10:49,473 nobody set out to build this, right? 76 00:10:49,473 --> 00:10:54,553 It was a lot of agents with the motivations that they had project based 77 00:10:54,553 --> 00:11:01,400 motivations, short timelines, stakeholders that wanted something yesterday, right? 78 00:11:02,510 --> 00:11:06,309 So this is what we're steering towards and 79 00:11:06,309 --> 00:11:11,385 this is a super simplified version of the architecture. 80 00:11:11,385 --> 00:11:14,230 But at the base, you've got foundation services. 81 00:11:14,230 --> 00:11:15,252 You've got a platform. 82 00:11:15,252 --> 00:11:17,352 We're headed towards Azure. 83 00:11:17,352 --> 00:11:21,811 We're set that whole topic for a slide about what that's good idea but 84 00:11:21,811 --> 00:11:26,271 foundation services are gonna be things like your machines that run, 85 00:11:26,271 --> 00:11:29,462 your tools that you need to do stuff on these data. 86 00:11:29,462 --> 00:11:35,392 Foundation data is hey, let's take all these source data that everybody uses. 87 00:11:35,392 --> 00:11:37,102 Put that in a raw layer. 88 00:11:37,102 --> 00:11:39,393 That really nobody can do analytics on it. 89 00:11:39,393 --> 00:11:44,374 It's just a copy of that data refine them in ways 90 00:11:44,374 --> 00:11:48,872 that are needed for analytics generally. 91 00:11:48,872 --> 00:11:53,642 So an example of this is there are actually, at least seven clinical record 92 00:11:53,642 --> 00:11:58,960 systems in the organization that I work with, because it's so highly federated. 93 00:12:00,250 --> 00:12:05,637 Stitching them together to say, okay, despite the fact that they're 94 00:12:05,637 --> 00:12:10,937 kind of serving different parts of the organization, fundamentally 95 00:12:10,937 --> 00:12:16,532 when you act when you want someone's A1C which is a measure of diabetes. 96 00:12:16,532 --> 00:12:20,151 You should be able to get that across the entire population. 97 00:12:20,151 --> 00:12:23,064 Not we're gonna do this system and that system and this system. 98 00:12:23,064 --> 00:12:27,842 So there's some things that from the refined layer, 99 00:12:27,842 --> 00:12:32,511 be able to say can we just get that across the board and 100 00:12:32,511 --> 00:12:35,924 then enriched data takes many forms. 101 00:12:35,924 --> 00:12:39,133 DX, DG is like a risk score, what? 102 00:12:39,133 --> 00:12:43,676 How risky is this person given their diagnoses for 103 00:12:43,676 --> 00:12:46,494 becoming ill in the next year? 104 00:12:46,494 --> 00:12:49,971 For example, geography is another enrichments. 105 00:12:49,971 --> 00:12:53,131 Where do they live relative to the census data. 106 00:12:53,131 --> 00:12:54,372 There are many examples. 107 00:12:54,372 --> 00:12:57,630 Those are just a couple off the top. 108 00:12:57,630 --> 00:13:02,280 So imagine, if you will that this is what the foundation 109 00:13:02,280 --> 00:13:07,050 of the future looks like and how do the needs of the stakeholders get met. 110 00:13:07,050 --> 00:13:12,411 They get met through what are being called tenants. 111 00:13:12,411 --> 00:13:16,618 They're focused on the needs of decision makers, but 112 00:13:16,618 --> 00:13:20,559 still using the same raw, refined enriched data. 113 00:13:20,559 --> 00:13:25,342 So the question becomes great, that's a great picture and 114 00:13:25,342 --> 00:13:30,221 what might you do to reduce the likelihood that you're gonna 115 00:13:30,221 --> 00:13:34,830 end up again with something that kind of looks like this? 116 00:13:36,400 --> 00:13:40,062 And one of the leads of this project was like you know what? 117 00:13:40,062 --> 00:13:44,543 I think we're gonna need to be really somewhat intentional out of 118 00:13:44,543 --> 00:13:48,380 the gate to apply some standards and practices, okay? 119 00:13:48,380 --> 00:13:49,272 Some governance. 120 00:13:49,272 --> 00:13:52,930 Dare I say governance, it's kind of a dirty word in analytics. 121 00:13:52,930 --> 00:13:57,362 But just from like governance and even some of our analysts are like yeah, 122 00:13:57,362 --> 00:13:58,301 you know what? 123 00:13:58,301 --> 00:14:04,572 This is really hard to use, because it is it's hard to find stuff. 124 00:14:04,572 --> 00:14:08,327 >> Okay, do other different hours for 125 00:14:08,327 --> 00:14:12,593 clinical finance, membership or art? 126 00:14:12,593 --> 00:14:15,053 Is it the same group of analysts? 127 00:14:15,053 --> 00:14:18,239 >> So at this level, it's not so 128 00:14:18,239 --> 00:14:23,087 much analysts working on specific systems. 129 00:14:23,087 --> 00:14:28,580 You've got people that serve numbers around claims. 130 00:14:28,580 --> 00:14:31,211 Putting data into these systems and 131 00:14:31,211 --> 00:14:36,130 like I need one particular piece of data for one particular thing. 132 00:14:36,130 --> 00:14:39,071 They're not really doing analytics. 133 00:14:39,071 --> 00:14:42,869 Where analytics gets interesting is when you wanna put these things 134 00:14:42,869 --> 00:14:44,190 together, right? 135 00:14:44,190 --> 00:14:49,722 So let's say you've got a grandma who's sick and her claims information, 136 00:14:49,722 --> 00:14:54,832 and her granddaughter's information that takes care of right like. 137 00:14:54,832 --> 00:14:59,405 So that's really where the analytics start to get interesting and duplicative, and 138 00:14:59,405 --> 00:14:59,922 siloed. 139 00:14:59,922 --> 00:15:03,528 And because you've got a lot of agents or 140 00:15:03,528 --> 00:15:08,619 analysts in here trying to answer similar questions from 141 00:15:08,619 --> 00:15:14,040 different angles that they duplicate the source data a lot. 142 00:15:15,410 --> 00:15:15,923 It's a good question. 143 00:15:15,923 --> 00:15:20,076 Other questions clarification >> Seems like one of the challenge is that 144 00:15:20,076 --> 00:15:24,685 there are people who can navigate this system and just Job security for them. 145 00:15:24,685 --> 00:15:26,683 >> The incumbents way. 146 00:15:26,683 --> 00:15:28,443 No, yes, sir. 147 00:15:28,443 --> 00:15:30,023 Yes, thank you. 148 00:15:30,023 --> 00:15:31,777 Yes, there is a little bit of that. 149 00:15:31,777 --> 00:15:33,355 Actually more than a little bit. 150 00:15:33,355 --> 00:15:38,656 >> [LAUGH] >> Other thoughts, observation, questions? 151 00:15:38,656 --> 00:15:41,556 No, let's keep going. 152 00:15:41,556 --> 00:15:46,714 Okay, so the standard objective of the standards and 153 00:15:46,714 --> 00:15:50,962 practices work is to identify prioritize. 154 00:15:50,962 --> 00:15:56,863 Establish or adopt standards and practices to send system that is emerging. 155 00:15:56,863 --> 00:15:58,755 This is a systems formation event. 156 00:15:58,755 --> 00:16:01,370 Writ large, right? 157 00:16:01,370 --> 00:16:03,800 It's exciting, it's also a little bit terrifying. 158 00:16:03,800 --> 00:16:08,260 So the questions that I came to were, okay, 159 00:16:08,260 --> 00:16:12,820 what can I bring to bear from cybernetics and complex adaptive 160 00:16:12,820 --> 00:16:17,490 systems to inform the approach to design and implement standards and practices? 161 00:16:18,540 --> 00:16:22,340 It's a really somewhat abstract problem, so I'm gonna need a little bit of 162 00:16:22,340 --> 00:16:26,170 an abstract toolbox to look into and guide the thinking on this. 163 00:16:26,170 --> 00:16:28,710 Otherwise, it's gonna be a hot mess and it still might be a hot mess. 164 00:16:28,710 --> 00:16:30,865 So, we'll save that one for a little bit later. 165 00:16:32,630 --> 00:16:35,015 So what tools, what methods might apply? 166 00:16:35,015 --> 00:16:39,279 And then what complexity related risks might arise as this project matures? 167 00:16:39,279 --> 00:16:45,249 I expect there to be some really good dialogue from you guys on that, 168 00:16:45,249 --> 00:16:47,046 cuz there are many. 169 00:16:47,046 --> 00:16:52,550 Okay, so before we go too much further, I felt the need to define complex 170 00:16:52,550 --> 00:16:57,010 adaptive systems, and I chose Holland's definition of complex adaptive systems. 171 00:16:57,010 --> 00:17:00,120 And I just looked at this from his text. 172 00:17:00,120 --> 00:17:03,690 So let's see, systems change and reorganize their component parts to adapt 173 00:17:03,690 --> 00:17:06,440 themselves to the problems posed by their surroundings. 174 00:17:06,440 --> 00:17:07,452 Yeah, that applies. 175 00:17:07,452 --> 00:17:10,820 Moving target, yep, that too. 176 00:17:10,820 --> 00:17:11,340 What else? 177 00:17:12,540 --> 00:17:16,620 Mechanisms that mediate these systems are much more alike than surface observations 178 00:17:16,620 --> 00:17:21,334 would suggest, and that's what we're banking on. 179 00:17:21,334 --> 00:17:26,208 So, raise your hand if you agree with this 180 00:17:26,208 --> 00:17:31,236 definition of complex adaptive systems. 181 00:17:33,742 --> 00:17:39,691 Okay, raise your hand if you think that the project description that 182 00:17:39,691 --> 00:17:45,550 I showed is likely to manifest itself as a complex adaptive system. 183 00:17:46,640 --> 00:17:49,228 Okay, so, that's what I'm banking on as well, thank you all for 184 00:17:49,228 --> 00:17:50,073 raising your hands. 185 00:17:50,073 --> 00:17:53,732 >> [LAUGH] >> And I'll pay you [LAUGH] all right. 186 00:17:58,906 --> 00:18:01,275 Okay, so what do we expect from a complex adaptive system? 187 00:18:01,275 --> 00:18:04,565 Well, we expect some nonlinearity, yep. 188 00:18:04,565 --> 00:18:07,695 We expect emergence, we expect unintended consequences. 189 00:18:07,695 --> 00:18:10,725 If you push the system, it's probably not gonna go in the direction that you want it 190 00:18:10,725 --> 00:18:16,450 to, so you're gonna need another lever. 191 00:18:16,450 --> 00:18:19,610 And the agents are going to change the environment and 192 00:18:19,610 --> 00:18:23,420 be changed by the environment, so let's let them do that. 193 00:18:23,420 --> 00:18:25,162 Let's help them do that, 194 00:18:25,162 --> 00:18:29,527 in ways that actually trend us towards standardization, okay? 195 00:18:32,757 --> 00:18:34,728 Other questions, observations, thoughts? 196 00:18:34,728 --> 00:18:38,751 Moving right along. 197 00:18:38,751 --> 00:18:42,560 Okay, so we're opening the system's toolbox. 198 00:18:42,560 --> 00:18:45,680 And the first question that I had to wrestle with when I had to go and 199 00:18:45,680 --> 00:18:49,990 talk about this in the organization was, so, why do we need standards? 200 00:18:49,990 --> 00:18:51,700 What is the purpose of standards? 201 00:18:52,970 --> 00:19:00,400 And I had to go with the adaptive cycle from what, Holland and Gunderson. 202 00:19:00,400 --> 00:19:03,290 Raise your hand if you're familiar with the adaptive cycle. 203 00:19:03,290 --> 00:19:08,240 Okay, our system scientists are, this comes out of ecosystems 204 00:19:08,240 --> 00:19:12,280 science, okay? 205 00:19:14,010 --> 00:19:19,480 And what it says is that, many systems have a growth phase, 206 00:19:19,480 --> 00:19:25,980 rapid growth and change, this is when things are changing a lot. 207 00:19:25,980 --> 00:19:31,488 And then they get to a state of maturity where conservation is really more focal, 208 00:19:31,488 --> 00:19:33,274 there's efficiencies. 209 00:19:33,274 --> 00:19:36,454 But what starts to happen over here is that, things get so 210 00:19:36,454 --> 00:19:40,894 rigid that they can't change anymore, and then they go through this release, 211 00:19:40,894 --> 00:19:47,270 renewal, repeat, Process. 212 00:19:47,270 --> 00:19:51,040 So this is the panarchy and the adaptive cycle. 213 00:19:51,040 --> 00:19:55,803 I don't ever go to my corporate colleagues and say, 214 00:19:55,803 --> 00:20:00,686 let me tell you about the adaptive cycle [LAUGH] never. 215 00:20:00,686 --> 00:20:01,368 What I do [LAUGH] is, you know what? 216 00:20:01,368 --> 00:20:03,830 Let's look at the current state. 217 00:20:03,830 --> 00:20:05,750 Let's look at the status quo. 218 00:20:05,750 --> 00:20:09,780 And I think it's probably pretty safe to say 219 00:20:09,780 --> 00:20:12,950 that we've got some extremes going on in our status quo. 220 00:20:12,950 --> 00:20:17,910 We've got chaos on one hand, cuz we've got a little bit of a data swamp, 221 00:20:17,910 --> 00:20:20,590 a lot of duplication, it's hard to find stuff. 222 00:20:20,590 --> 00:20:23,495 When you do, you're kind of like, do I trust it, hopefully? 223 00:20:23,495 --> 00:20:27,570 But we also have rigidity. 224 00:20:27,570 --> 00:20:32,390 One anecdote is, it costs a quarter of a million dollars to add a field. 225 00:20:34,390 --> 00:20:36,350 Just gonna leave that hang for a moment. 226 00:20:36,350 --> 00:20:38,401 That doesn't scale, and 227 00:20:38,401 --> 00:20:43,166 that's the rigidity that we face with part of this problem. 228 00:20:43,166 --> 00:20:48,270 Okay, so we've got an adaptive cycle playing out. 229 00:20:48,270 --> 00:20:50,810 We've got this new project, okay, we need to solve this problem. 230 00:20:50,810 --> 00:20:51,353 Let's have a new project. 231 00:20:51,353 --> 00:20:58,118 Now I'm gonna draw a picture, because what has become evident to me is that, 232 00:20:58,118 --> 00:21:05,006 if this is the project [INAUDIBLE] if this is the project, this is the adaptive. 233 00:21:05,006 --> 00:21:09,180 Okay, there's the platform. 234 00:21:09,180 --> 00:21:14,240 So we're changing the platform, and that has changed on one level. 235 00:21:14,240 --> 00:21:17,972 We're changing how the data producers act. 236 00:21:17,972 --> 00:21:19,162 My gosh, this is gonna be so hard. 237 00:21:19,162 --> 00:21:20,113 [LAUGH] it's a good idea. 238 00:21:20,113 --> 00:21:23,434 We're changing how the analysts act. 239 00:21:23,434 --> 00:21:26,120 We're changing how their end users act. 240 00:21:26,120 --> 00:21:30,090 And most importantly, we're changing the polity. 241 00:21:30,090 --> 00:21:31,690 We're changing how it's governed. 242 00:21:31,690 --> 00:21:36,058 We're changing who's allowed to do what, Wayne, to your point. 243 00:21:36,058 --> 00:21:43,290 There's incumbents in all of these, and so it's turtles all the way down. 244 00:21:43,290 --> 00:21:47,522 And what I get to is, my gosh, this is the sand pile of self-organized criticality. 245 00:21:49,905 --> 00:21:55,351 So questions, observations, crazy talk. 246 00:21:58,066 --> 00:21:58,660 >> Is there an antidote? 247 00:21:58,660 --> 00:22:01,099 >> Well, let's keep talking. 248 00:22:01,099 --> 00:22:05,651 [LAUGH] >> Are you in the omega phase, 249 00:22:05,651 --> 00:22:09,166 the system is about to crash, is that what? 250 00:22:09,166 --> 00:22:12,966 I mean, has it gotten too complex, too chaotic, 251 00:22:12,966 --> 00:22:16,236 and too rigid that it's not sustainable? 252 00:22:16,236 --> 00:22:23,345 And that you're trying to get it into this alpha phase of new ideas? 253 00:22:23,345 --> 00:22:26,756 >> Well, that's a really great and interesting, I mean, 254 00:22:26,756 --> 00:22:30,890 the system in its current form has to actually keep running, right? 255 00:22:30,890 --> 00:22:34,890 So, elements of- >> Yeah, but it could be very clumsy. 256 00:22:34,890 --> 00:22:36,900 >> It's clumsy, it's become inefficient. 257 00:22:36,900 --> 00:22:38,860 It's becoming expensive to maintain. 258 00:22:38,860 --> 00:22:41,870 That's why there is this big project underway. 259 00:22:41,870 --> 00:22:46,810 But for the interim, it does have to actually continue to do its thing. 260 00:22:46,810 --> 00:22:52,452 But we do need this growth phase before it gets so 261 00:22:52,452 --> 00:22:57,823 bad, that status quo has actually crashed. 262 00:22:57,823 --> 00:23:03,840 Now, the challenge is that you need some of these people who, 263 00:23:03,840 --> 00:23:08,017 so far, are focused on the current state. 264 00:23:08,017 --> 00:23:11,708 You actually need them to help start informing the new state, and so 265 00:23:11,708 --> 00:23:14,710 that's a balance in the dance as well. 266 00:23:14,710 --> 00:23:17,850 >> Well, a key aspect is that, you're gonna have to make a significant 267 00:23:17,850 --> 00:23:21,430 investment, cuz you have to maintain the bus while re-engineering the bus. 268 00:23:21,430 --> 00:23:24,370 You can't do that without a serious budget. 269 00:23:24,370 --> 00:23:27,555 So there has to be commitment at the top of your organization for 270 00:23:27,555 --> 00:23:29,248 a very large amount of research. 271 00:23:29,248 --> 00:23:33,540 >> And they would say that they are doing that, and 272 00:23:33,540 --> 00:23:35,779 the question becomes- >> Is it enough? 273 00:23:35,779 --> 00:23:36,839 >> Yes, indeed, and 274 00:23:36,839 --> 00:23:41,361 is everybody okay with the fact that it's gonna get worse before it gets better? 275 00:23:41,361 --> 00:23:42,338 >> Exactly. 276 00:23:42,338 --> 00:23:44,890 >> And no is the answer to that, by the way. 277 00:23:44,890 --> 00:23:49,191 [LAUGH] >> Because I guess my perspective is 278 00:23:49,191 --> 00:23:54,089 a little bit different, cuz my organization is making an enormous amount 279 00:23:54,089 --> 00:23:59,483 of investment in a new, Building and all the kinda operating system. 280 00:23:59,483 --> 00:24:00,400 And I think, 281 00:24:00,400 --> 00:24:06,408 it's interesting because I feel like it's not only about the amount of investment, 282 00:24:06,408 --> 00:24:11,700 but kinda the philosophy and the mental model that you bring to bear. 283 00:24:11,700 --> 00:24:16,837 Which is, I think, cuz if you have a certain kind of mentality to 284 00:24:16,837 --> 00:24:23,229 how you change things, you could be bringing some old stuff into the mix, too. 285 00:24:23,229 --> 00:24:28,260 Or best practices, one person's conception of best 286 00:24:28,260 --> 00:24:35,097 practices may not be the true best practices that speaks to the culture. 287 00:24:35,097 --> 00:24:36,044 >> Yeah. >> That's there already. 288 00:24:36,044 --> 00:24:36,762 >> Yeah. >> So I think, 289 00:24:36,762 --> 00:24:40,409 it's definitely about the amount, you need some investment. 290 00:24:40,409 --> 00:24:41,210 >> Yeah. 291 00:24:41,210 --> 00:24:42,986 >> But you also need a different mental model. 292 00:24:42,986 --> 00:24:48,093 And at the same time, I think the process of how you make 293 00:24:48,093 --> 00:24:52,767 the change could be also a very important factor. 294 00:24:52,767 --> 00:24:56,127 >> I agree wholeheartedly, mindsets matter a ton. 295 00:24:56,127 --> 00:24:58,898 >> Yeah. 296 00:24:58,898 --> 00:25:02,766 >> We don't wanna pay Cal PATHS is what we talk about in the project, but 297 00:25:02,766 --> 00:25:04,576 that's easier said than done. 298 00:25:04,576 --> 00:25:06,047 >> Yeah. 299 00:25:06,047 --> 00:25:09,103 >> I was just curious, speaking a little bit to that, 300 00:25:09,103 --> 00:25:12,595 has this been from the top down to create an army to do this? 301 00:25:12,595 --> 00:25:15,148 How much buy-in is there across the organization from the top to 302 00:25:15,148 --> 00:25:16,055 the bottom to do this? 303 00:25:16,055 --> 00:25:17,279 >> It varies. 304 00:25:17,279 --> 00:25:21,390 It varies and it is a highly federated organization. 305 00:25:21,390 --> 00:25:25,195 So- >> Some of those federations are very much 306 00:25:25,195 --> 00:25:28,860 into it and others are like, yeah, it's probably just fine the way it is. 307 00:25:28,860 --> 00:25:30,636 >> I've seen this tribe before. 308 00:25:30,636 --> 00:25:31,399 >> Right. >> Right. 309 00:25:31,399 --> 00:25:35,828 [LAUGH] >> And 310 00:25:35,828 --> 00:25:39,068 one of the things I've always been interested in is this panarchy model. 311 00:25:39,068 --> 00:25:44,245 Where you've got this connectedness access along the bottom, right? 312 00:25:44,245 --> 00:25:46,393 How many components, how connected they are? 313 00:25:46,393 --> 00:25:49,796 I always wonder whether, so as you're coming around this curve, 314 00:25:49,796 --> 00:25:53,712 it's like connectedness increases for a little bit until it gets too much. 315 00:25:53,712 --> 00:25:55,671 >> Sure. >> And sort of precipitates a collapse and 316 00:25:55,671 --> 00:25:56,526 then it comes back. 317 00:25:56,526 --> 00:25:57,784 >> Yeah. 318 00:25:57,784 --> 00:26:02,167 >> Are you saying that or are they saying that reducing connectedness is part of 319 00:26:02,167 --> 00:26:04,190 the way to solve the problem? 320 00:26:04,190 --> 00:26:06,697 Or that it's a consequence of the collapse, right? 321 00:26:06,697 --> 00:26:07,744 I guess it could be either or. 322 00:26:07,744 --> 00:26:11,660 >> I think it's either or, I couldn't go binary on that answer. 323 00:26:11,660 --> 00:26:12,628 >> But in your case, 324 00:26:12,628 --> 00:26:16,569 it's going to mean reducing the number of elements that are out there and 325 00:26:16,569 --> 00:26:21,102 players and then the relationships between those players to try to streamline it. 326 00:26:21,102 --> 00:26:24,810 >> Well and reducing the duplication of the data elements. 327 00:26:24,810 --> 00:26:28,249 >> Yeah. >> Is one aspect of it, but increasing, 328 00:26:28,249 --> 00:26:34,621 in some respects, the connectivity of the agents working within the system. 329 00:26:34,621 --> 00:26:36,249 And we'll see more about that and 330 00:26:36,249 --> 00:26:38,940 that may be actually a little fraught with peril, so. 331 00:26:38,940 --> 00:26:42,268 >> Yeah, I mean, so yeah, you're gonna be adding some connections, but 332 00:26:42,268 --> 00:26:44,356 you should hopefully be removing a lot more. 333 00:26:44,356 --> 00:26:48,104 >> Yeah. >> Making the whole thing look just more 334 00:26:48,104 --> 00:26:49,514 streamlined. 335 00:26:49,514 --> 00:26:50,551 >> Yes. 336 00:26:50,551 --> 00:26:52,368 >> Eliminating meaningless connection. 337 00:26:52,368 --> 00:26:52,983 >> Agreed. 338 00:26:52,983 --> 00:26:54,446 >> Yeah. 339 00:26:54,446 --> 00:26:56,980 >> So I see that you have a problem. 340 00:26:56,980 --> 00:27:00,075 And I'm wondering kinda in the spirit of evolution, 341 00:27:00,075 --> 00:27:03,946 if there's been any punctuating events as you're writing this? 342 00:27:03,946 --> 00:27:07,384 And whether those are leverage points where they come in. 343 00:27:07,384 --> 00:27:12,257 >> Yeah, no, that is a great, I love punctuated equilibrium, by the way. 344 00:27:12,257 --> 00:27:17,615 I think it's not just the internals of the system have become inefficient, 345 00:27:17,615 --> 00:27:22,902 there are external pressures that are driving this as well, regulatory. 346 00:27:22,902 --> 00:27:27,910 You need to be really efficient, effective maybe, with your response 347 00:27:27,910 --> 00:27:32,340 to regulatory requirements they are coming fast and furious. 348 00:27:32,340 --> 00:27:36,148 So that external pressure, I think, 349 00:27:36,148 --> 00:27:40,199 is starting to nudge the system as well. 350 00:27:40,199 --> 00:27:41,910 >> External changes fast. 351 00:27:41,910 --> 00:27:45,792 >> Yeah, did that get to what you were thinking? 352 00:27:45,792 --> 00:27:48,887 >> Actually, I'm not invested in your answer very much. 353 00:27:48,887 --> 00:27:53,280 [LAUGH] But I'm just wondering, are there any moments that you're hanging 354 00:27:53,280 --> 00:27:57,741 this process on to or whether you've just recognized, kind of ethically, 355 00:27:57,741 --> 00:28:00,620 that there's a bad problem and it needs fixing. 356 00:28:00,620 --> 00:28:05,490 >> I think it's more than ethics, but it's an internal problem. 357 00:28:05,490 --> 00:28:09,455 And there are some externalities that are inspiring the change. 358 00:28:09,455 --> 00:28:13,123 >> Wondering if the realization of that figure on the bottom right, 359 00:28:13,123 --> 00:28:14,399 was one of the points. 360 00:28:14,399 --> 00:28:16,489 How much does it cost to have a field? 361 00:28:16,489 --> 00:28:21,144 All right, maybe we can just- >> I think there might be a little bit of 362 00:28:21,144 --> 00:28:22,004 that, yeah. 363 00:28:22,004 --> 00:28:24,716 >> And that's what you see at the analysts level and you're like, what? 364 00:28:24,716 --> 00:28:29,645 And then add that up to the executive level, you're like, hang on, no. 365 00:28:29,645 --> 00:28:34,642 >> What they're telling you is, if I add this field, I have to go find all 366 00:28:34,642 --> 00:28:39,162 the other places that that has touched and that's [INAUDIBLE]. 367 00:28:39,162 --> 00:28:41,491 >> Not to be there. 368 00:28:41,491 --> 00:28:44,070 And what they're really saying is we don't even know what they are. 369 00:28:44,070 --> 00:28:45,890 >> Yeah. >> So we'd have to go find out. 370 00:28:45,890 --> 00:28:46,954 >> Yep. 371 00:28:46,954 --> 00:28:49,013 >> Probably need several people to do that. 372 00:28:49,013 --> 00:28:49,986 >> More than several. 373 00:28:49,986 --> 00:28:56,735 [LAUGH] Okay, so, let's see here. 374 00:28:56,735 --> 00:29:01,227 Okay, the next question that I would confront was really about, and 375 00:29:01,227 --> 00:29:03,947 this comes from a dear colleague of mine. 376 00:29:03,947 --> 00:29:06,173 And I was like, okay, we're gonna do standards and practices. 377 00:29:06,173 --> 00:29:08,438 And he just kind of sat there with his arms crossed and 378 00:29:08,438 --> 00:29:11,337 gave me the stink eye like, what do you mean by standards, right? 379 00:29:11,337 --> 00:29:14,532 This is an analyst who was like, you can keep your governance far, 380 00:29:14,532 --> 00:29:15,470 far away from me. 381 00:29:15,470 --> 00:29:18,096 I just need to get my job done, thank you. 382 00:29:18,096 --> 00:29:23,011 And so it became clear to me that there did need to be some kind of reckoning 383 00:29:23,011 --> 00:29:26,410 with the fact that, what do we mean by standards? 384 00:29:26,410 --> 00:29:28,570 Is everything going to be a standard? 385 00:29:28,570 --> 00:29:30,054 What does that continuum look like? 386 00:29:30,054 --> 00:29:33,550 And the truth is no, not everything is gonna be a standard only 387 00:29:33,550 --> 00:29:37,670 the things that really have to be standards enforced will be standards. 388 00:29:40,250 --> 00:29:44,396 How people access data, that will be standardized. 389 00:29:44,396 --> 00:29:48,458 It will be, in fact, even informed by policies that are already in place. 390 00:29:48,458 --> 00:29:50,014 It will be monitored. 391 00:29:50,014 --> 00:29:54,570 There is the security aspect around all of that. 392 00:29:55,820 --> 00:29:58,534 Best practices are a little more like guidelines than guard rails. 393 00:29:58,534 --> 00:30:03,514 This is a way to get to the objective of standardization 394 00:30:03,514 --> 00:30:06,337 without saying, thou shall. 395 00:30:06,337 --> 00:30:10,784 An example of that that has actually been working, it seems really banal. 396 00:30:10,784 --> 00:30:15,289 But what is the cover page for the self-service report that you write? 397 00:30:17,060 --> 00:30:20,003 Well, we did it. 398 00:30:20,003 --> 00:30:21,520 What assumptions were made? 399 00:30:21,520 --> 00:30:23,054 When was it relevant? 400 00:30:23,054 --> 00:30:24,395 I'm kind of winging it. 401 00:30:24,395 --> 00:30:25,989 Is it sensitive? 402 00:30:25,989 --> 00:30:26,992 Sensitive regarding what? 403 00:30:26,992 --> 00:30:30,324 Well sensitive according to policy document. 404 00:30:30,324 --> 00:30:32,510 So that's not gonna be policed. 405 00:30:32,510 --> 00:30:34,790 No one's gonna say, you didn't follow the standard. 406 00:30:34,790 --> 00:30:39,949 But if it's a really good example, that analysts, I've been an analyst. 407 00:30:39,949 --> 00:30:41,298 I don't want to think about that stuff. 408 00:30:41,298 --> 00:30:45,350 I just want to have a template that I fill in that makes my job easier. 409 00:30:45,350 --> 00:30:48,800 And I go and ask my manager if there's a couple of hard questions. 410 00:30:48,800 --> 00:30:51,994 Is this restrictive? 411 00:30:51,994 --> 00:30:52,870 That kind of thing. 412 00:30:52,870 --> 00:30:57,494 So it became clear to me that we really had to be 413 00:30:57,494 --> 00:31:03,107 thoughtful of who we were putting these standards on. 414 00:31:03,107 --> 00:31:07,655 And at least recognize the value judgments that they 415 00:31:07,655 --> 00:31:10,555 would bring to the table on that. 416 00:31:10,555 --> 00:31:11,910 And how things would be interpreted. 417 00:31:13,860 --> 00:31:18,489 Okay, moving on cuz this is really just another lens on that. 418 00:31:18,489 --> 00:31:22,438 And we also recognized that there are numerous stakeholders across this 419 00:31:22,438 --> 00:31:23,310 value chain. 420 00:31:23,310 --> 00:31:30,456 So the heart of this slide really is how we are organizing standards and practices. 421 00:31:30,456 --> 00:31:32,470 It is a hierarchical framework. 422 00:31:32,470 --> 00:31:34,615 This is the top of the hierarchy. 423 00:31:34,615 --> 00:31:38,234 So people, actors, we call them personas, 424 00:31:38,234 --> 00:31:41,965 interact with this system to get things done. 425 00:31:41,965 --> 00:31:46,645 To operate platforms, to collect and integrate this system of record data, 426 00:31:46,645 --> 00:31:50,190 to prepare the data for business use So 427 00:31:50,190 --> 00:31:55,580 this is a way to kind of begin to say, okay, who are the people that need to 428 00:31:55,580 --> 00:32:00,560 do something with this system, so they need to be able to interact with it? 429 00:32:00,560 --> 00:32:01,649 Pretty straightforward stuff, but. 430 00:32:01,649 --> 00:32:06,570 >> Who are the people who are suffering most of these people? 431 00:32:06,570 --> 00:32:07,941 >> Great question. 432 00:32:07,941 --> 00:32:13,440 [COUGH] I think that everybody, the decision 433 00:32:13,440 --> 00:32:17,930 makers probably aren't getting what they want as fast as they want it. 434 00:32:17,930 --> 00:32:21,315 The analytics leaders talk to the decision makers and are like, 435 00:32:21,315 --> 00:32:24,385 yeah, I know, but you have no idea how many people I need to 436 00:32:24,385 --> 00:32:27,093 talk to to get the question answered that you want. 437 00:32:27,093 --> 00:32:29,378 And by the way, it's gonna cost you a quarter million dollars and 438 00:32:29,378 --> 00:32:31,340 I'm pretty sure you're not gonna give that to me, right. 439 00:32:31,340 --> 00:32:32,832 So that's right. 440 00:32:32,832 --> 00:32:37,220 And then your data engineers and your data consultants and 441 00:32:37,220 --> 00:32:41,310 your data scientists are like, seriously, I can actually build a really great 442 00:32:41,310 --> 00:32:45,840 decision tool if this wasn't such a fragmented mess, right. 443 00:32:45,840 --> 00:32:50,690 Or if I do build you something, then I'm gonna have a little bit of anxiety about 444 00:32:50,690 --> 00:32:55,320 the quality of the data that went into that decision that you're making, and 445 00:32:55,320 --> 00:32:58,214 let's hope it's not life or death, right. 446 00:32:58,214 --> 00:33:04,110 Like that, so there's motivation across the board, I think. 447 00:33:04,110 --> 00:33:05,640 There's also resistance across the board, 448 00:33:05,640 --> 00:33:08,520 to your point, because there's some incumbency here. 449 00:33:08,520 --> 00:33:13,760 And so the other thing to kind of get back to that cohesion, adaptation, balance. 450 00:33:13,760 --> 00:33:18,830 If anything, these are the folks in particular 451 00:33:18,830 --> 00:33:24,920 that do adaptability well, they can change in a heartbeat. 452 00:33:26,638 --> 00:33:31,800 Less so [LAUGH] because there are a lot of policies and procedures and change 453 00:33:31,800 --> 00:33:39,280 requests and things that I think initially were intended towards systems of record. 454 00:33:39,280 --> 00:33:47,180 You shouldn't really change a claim system after it's implemented very often. 455 00:33:47,180 --> 00:33:50,690 But they've applied some of those same rules and thinking to analytics 456 00:33:50,690 --> 00:33:55,380 systems that actually do need a little bit more flexibility. 457 00:33:56,690 --> 00:34:01,580 >> A big concern often is that by putting in standards you're gonna suppress 458 00:34:01,580 --> 00:34:02,920 innovation and creativity. 459 00:34:02,920 --> 00:34:04,020 >> Absolutely. 460 00:34:04,020 --> 00:34:05,170 >> And so that's a difficult- >> Yeah 461 00:34:05,170 --> 00:34:05,790 >> Balance, 462 00:34:05,790 --> 00:34:07,238 the bigger your organization- >> Yep. 463 00:34:07,238 --> 00:34:11,730 >> The different ways you try to solve that puzzle. 464 00:34:11,730 --> 00:34:12,820 >> Yeah, we'll get to that. 465 00:34:12,820 --> 00:34:19,157 That's actually one of the complexity related risks and it is. 466 00:34:19,157 --> 00:34:19,800 Other thoughts? 467 00:34:19,800 --> 00:34:20,604 >> Can I say something about the previous slide? 468 00:34:20,604 --> 00:34:25,300 >> Yeah. 469 00:34:25,300 --> 00:34:29,372 >> Or suggest to have something. 470 00:34:29,372 --> 00:34:32,837 So degrees of freedom fits really well for 471 00:34:32,837 --> 00:34:36,995 this, where thinking of having just standards, 472 00:34:36,995 --> 00:34:41,945 having lots and lots of specific rules about specific cases 473 00:34:41,945 --> 00:34:46,410 is like having all of this data about how to do things. 474 00:34:46,410 --> 00:34:52,300 But coming up with best practices is finding structure in that data, 475 00:34:52,300 --> 00:34:55,610 find the commonality that rules, and reducing the degrees of freedom. 476 00:34:55,610 --> 00:34:56,653 >> Right. >> So saying, 477 00:34:56,653 --> 00:35:01,904 instead of having all these specific rules about how to do things, 478 00:35:01,904 --> 00:35:06,780 instead say, we have a principle of appreciating elegance or 479 00:35:06,780 --> 00:35:11,910 shortness of an answer, of privacy, love, a few things. 480 00:35:11,910 --> 00:35:15,340 So you have a few of these guidelines that are high level rules 481 00:35:15,340 --> 00:35:17,780 that actually create all the specific rules. 482 00:35:17,780 --> 00:35:18,680 >> Yeah. 483 00:35:18,680 --> 00:35:21,700 >> So if you can- >> Distill, yeah. 484 00:35:21,700 --> 00:35:22,961 >> Get people to look at the high level ones, 485 00:35:22,961 --> 00:35:24,269 you don't have to have all the specifics. 486 00:35:24,269 --> 00:35:25,250 >> Right. 487 00:35:25,250 --> 00:35:28,295 >> So I mean it depends on the degrees of freedom constraints, you're looking for 488 00:35:28,295 --> 00:35:29,760 things that have fewer constraints. 489 00:35:29,760 --> 00:35:31,080 >> Yeah. 490 00:35:31,080 --> 00:35:34,870 >> And explains what you're supposed to do and then- 491 00:35:34,870 --> 00:35:35,664 >> Yeah, and 492 00:35:35,664 --> 00:35:40,528 there's a delicate balance there I also have found between the, 493 00:35:42,991 --> 00:35:46,769 How much specificity do you have in that answer, 494 00:35:46,769 --> 00:35:51,940 general principles versus sometimes we need a little more stuff. 495 00:35:51,940 --> 00:35:54,637 [LAUGH] >> Some are gonna be exceptions, 496 00:35:54,637 --> 00:35:58,486 they're not gonna cover everything, but. 497 00:35:58,486 --> 00:35:59,820 >> Yeah, Dr. Shey. >> Yeah, just in that, 498 00:36:01,790 --> 00:36:05,850 in general this looks like maybe you're prepared to have sort of a light touch 499 00:36:05,850 --> 00:36:10,990 where you have smart people working for you that are creative problem solvers. 500 00:36:10,990 --> 00:36:14,100 And so if you're telling them what your high level objectives are, 501 00:36:14,100 --> 00:36:16,350 this is the result that we want. 502 00:36:16,350 --> 00:36:19,390 We're not exactly sure how we want you to go about doing it, but 503 00:36:19,390 --> 00:36:21,950 this is what we would hope that you would give to us. 504 00:36:21,950 --> 00:36:26,910 And then, in standards terms, there are some hard limits on 505 00:36:26,910 --> 00:36:29,830 what you're allowed to do or what they should be allowed to do. 506 00:36:29,830 --> 00:36:34,350 And so you say, okay, well if you can keep 507 00:36:34,350 --> 00:36:38,670 the standards limited to the things that you absolutely require. 508 00:36:38,670 --> 00:36:40,868 And then just say, okay, within those limited constraints, 509 00:36:40,868 --> 00:36:43,620 do whatever you want to solve the problem and then tell us how you solved it. 510 00:36:43,620 --> 00:36:45,405 >> Right, yes. 511 00:36:45,405 --> 00:36:49,318 >> And that's letting your people be smart people and 512 00:36:49,318 --> 00:36:54,640 not mechanical components in an alienating system. 513 00:36:54,640 --> 00:36:56,990 >> Absolutely, absolutely. 514 00:36:56,990 --> 00:37:02,115 And we'll see more of that, it does not sell up. 515 00:37:02,115 --> 00:37:03,350 Other thoughts? 516 00:37:03,350 --> 00:37:08,400 Okay, I didn't actually even pick this particular, but it is now. 517 00:37:08,400 --> 00:37:12,910 So another thing with a critical systems heuristics model that 518 00:37:12,910 --> 00:37:17,090 really was I felt important to start with was boundaries. 519 00:37:17,090 --> 00:37:20,370 What are the boundaries of this system? 520 00:37:20,370 --> 00:37:26,280 Well, we've got to go kind of from infrastructure, so 521 00:37:26,280 --> 00:37:30,350 how do we operate the platforms, all the way to the insights that we're delivering. 522 00:37:30,350 --> 00:37:35,280 Those are the boundaries of where system standards and practices apply. 523 00:37:35,280 --> 00:37:40,550 The other boundary critique that is relevant is, this is for this big project, 524 00:37:40,550 --> 00:37:45,500 it is not for the entire analytics landscape within this organization. 525 00:37:45,500 --> 00:37:48,323 Although, if we drew a Venn diagram, 526 00:37:48,323 --> 00:37:52,920 it's a pretty big part of it, it's not the entirety of it. 527 00:37:52,920 --> 00:37:57,540 The other relevant thing from this, and this was actually based on these 528 00:37:57,540 --> 00:38:01,176 categories of, well kind of high level organization and 529 00:38:01,176 --> 00:38:06,271 then the categories and subcategories within there but I don't actually see. 530 00:38:06,271 --> 00:38:08,452 But that's really where the heart of the standards are. 531 00:38:08,452 --> 00:38:14,066 All in is about 100 documents, just my god, they're not all standards. 532 00:38:14,066 --> 00:38:18,433 >> [LAUGH] >> Try to find the line on that one. 533 00:38:18,433 --> 00:38:20,444 What was I gonna say about this? 534 00:38:20,444 --> 00:38:26,993 So pretty rigid hierarchy to give some structure to this system. 535 00:38:26,993 --> 00:38:31,104 How does it operate? 536 00:38:31,104 --> 00:38:34,770 And so we put this forth, we said these are the bounds of the system. 537 00:38:34,770 --> 00:38:39,510 We've said, here's a template for which we want you to document around. 538 00:38:39,510 --> 00:38:44,300 What's the purpose of said standard or practice? 539 00:38:44,300 --> 00:38:47,500 What do you do for this? 540 00:38:47,500 --> 00:38:52,412 And they're supposed to be concise document, 541 00:38:52,412 --> 00:38:56,231 but it is light touch in a lot of ways. 542 00:38:56,231 --> 00:38:59,680 This is a more functional view of this system. 543 00:39:01,100 --> 00:39:03,830 So what we have here on the left is an image that 544 00:39:03,830 --> 00:39:05,930 you're not gonna be able to see at all, but go to Wikipedia. 545 00:39:08,120 --> 00:39:10,604 The Viable Systems Model from Stafford Beer. 546 00:39:10,604 --> 00:39:13,620 So a couple of key elements from this model. 547 00:39:15,640 --> 00:39:18,630 System one is value creation. 548 00:39:18,630 --> 00:39:20,130 What value does thsi system create? 549 00:39:20,130 --> 00:39:24,210 System two is coordination, system three is control, 550 00:39:24,210 --> 00:39:27,330 system four is adaptation or evolution, how does it evolve. 551 00:39:27,330 --> 00:39:33,020 And then system five which is not on here from my perspective is identity. 552 00:39:33,020 --> 00:39:37,830 So that's that the Viable System Model, kind of a very, very cursory level. 553 00:39:37,830 --> 00:39:41,818 How does that apply to our approach to standards and practices?. 554 00:39:41,818 --> 00:39:44,780 So we turn the corner from the structural view, structural and 555 00:39:44,780 --> 00:39:49,330 hierarchical view of standards, and we've said, you know what, There are actually 556 00:39:49,330 --> 00:39:55,830 four key things that analysts and operators within this system care about. 557 00:39:55,830 --> 00:39:59,940 The stakeholders care about the metrics that they're delivering, the information. 558 00:39:59,940 --> 00:40:01,370 Is it timely? 559 00:40:01,370 --> 00:40:02,880 Is it correct? 560 00:40:02,880 --> 00:40:04,130 Is it trustworthy? 561 00:40:04,130 --> 00:40:09,066 Does it conflict with something else that somebody else produced? 562 00:40:09,066 --> 00:40:11,000 So that's the value delivery. 563 00:40:11,000 --> 00:40:16,480 Data modeling and patterns is loosely, this is probably the most tenuous one, 564 00:40:16,480 --> 00:40:18,260 loosely based on the coordination function. 565 00:40:19,490 --> 00:40:23,630 These are simple things that can really eat a ton of analysts' time if you have to 566 00:40:23,630 --> 00:40:25,120 do them over and over again. 567 00:40:25,120 --> 00:40:27,379 How do we format dates? 568 00:40:27,379 --> 00:40:31,700 Do we have leading zeros on a lookup field or on something that's kind of the key? 569 00:40:31,700 --> 00:40:32,803 You either do or you don't? 570 00:40:32,803 --> 00:40:35,660 Please don't make me have to sort through that problem. 571 00:40:35,660 --> 00:40:40,010 I will waste far too many cycles and far too much money on that. 572 00:40:40,010 --> 00:40:42,680 Data access and use, this is the control system. 573 00:40:42,680 --> 00:40:45,579 Can I get access to the data that I need to do my job? 574 00:40:47,570 --> 00:40:49,920 Period, full stop, right? 575 00:40:49,920 --> 00:40:54,100 And then capabilities and tools is adaptability. 576 00:40:56,890 --> 00:41:02,690 My environment will put pressures on me to use simulation tools. 577 00:41:02,690 --> 00:41:06,770 This is actually in my inbox yesterday about simulation tools, so 578 00:41:06,770 --> 00:41:09,100 that's becoming a thing. 579 00:41:10,100 --> 00:41:14,900 So we need flexibility to be able to apply capabilities and tools. 580 00:41:14,900 --> 00:41:19,218 While at the same time, we don't need no 581 00:41:19,218 --> 00:41:24,060 vendor left behind, because that happens. 582 00:41:24,060 --> 00:41:27,350 Thoughts on this, too loose of an application? 583 00:41:27,350 --> 00:41:30,550 >> So what personas are sitting in these advisory groups? 584 00:41:30,550 --> 00:41:34,930 >> Wonderful question. 585 00:41:34,930 --> 00:41:39,550 I would say a lot of the IT, it's a good mix of IT and 586 00:41:39,550 --> 00:41:43,970 analytics in capabilities and tools. 587 00:41:43,970 --> 00:41:49,550 It's a good mix of IT and analytics and data modelling and patterns that, 588 00:41:49,550 --> 00:41:54,594 it's actually a good mix of IT and analytics across each of these. 589 00:41:54,594 --> 00:41:59,123 I would say the metrics delivery is the most analysts dominated one. 590 00:41:59,123 --> 00:41:59,828 Great question. 591 00:41:59,828 --> 00:42:03,373 >> And the use one would have your end users as a level. 592 00:42:03,373 --> 00:42:06,450 >> Well, and the end users are still pretty far from this system. 593 00:42:06,450 --> 00:42:09,813 This is really analyst focused, and it's, 594 00:42:09,813 --> 00:42:14,688 if the other view was a hierarchical view of how the system works. 595 00:42:14,688 --> 00:42:19,725 This is a non hierarchical network of why the system works. 596 00:42:19,725 --> 00:42:25,290 And I am banking on self organization around advisory groups 597 00:42:25,290 --> 00:42:30,650 that the analysts and IT community can galvanize around. 598 00:42:30,650 --> 00:42:34,014 >> That's worrisome, I mean you don't really want your end users to be far away 599 00:42:34,014 --> 00:42:36,874 from where they're residing- >> You're right, no, you're right. 600 00:42:36,874 --> 00:42:41,700 >> [LAUGH] >> It depends on, so this system it sounds 601 00:42:41,700 --> 00:42:46,936 like you're talking about, isn't necesarily the one that is, 602 00:42:46,936 --> 00:42:51,123 well [CROSSTALK] it's not the one all your doctors and 603 00:42:51,123 --> 00:42:54,194 practitioners, this is a- >> Your executives 604 00:42:54,194 --> 00:42:56,250 are not involved in this. 605 00:42:56,250 --> 00:43:01,730 >> In a larger hospital, this is, already at, the third, or fourth level up, right? 606 00:43:01,730 --> 00:43:08,068 This is a command and control type system, so you're not. 607 00:43:08,068 --> 00:43:13,010 Metrics delivery for this system is a successful command and control for 608 00:43:13,010 --> 00:43:14,270 the hospital. 609 00:43:14,270 --> 00:43:14,965 >> Right. 610 00:43:14,965 --> 00:43:18,365 >> But at the same time, when you're entering in a new record, 611 00:43:18,365 --> 00:43:22,900 that's the end user entering in data that needs to make its way into this system, 612 00:43:22,900 --> 00:43:27,700 into that raw data part and pull up a record needs to come out of a system. 613 00:43:27,700 --> 00:43:32,318 Is either this one or is very much consistent with this system. 614 00:43:32,318 --> 00:43:37,265 >> So that's a really interesting kind of case study cuz when I think of that 615 00:43:37,265 --> 00:43:41,670 person, she is a nurse practitioner in an exam room. 616 00:43:41,670 --> 00:43:44,250 And she is interacting with a system of record. 617 00:43:44,250 --> 00:43:48,784 And her, it's a very transactional thing and 618 00:43:48,784 --> 00:43:52,279 that data then flows in first here. 619 00:43:52,279 --> 00:43:58,210 And what we hope is that she entered it in a somewhat standardized way. 620 00:43:58,210 --> 00:44:05,720 If she didn't, we may or may not be able to influence that. 621 00:44:05,720 --> 00:44:09,559 We may just have to adapt to it, at this level. 622 00:44:09,559 --> 00:44:14,561 But we only have to do it once-ish and we have to do it well. 623 00:44:14,561 --> 00:44:18,948 >> Yeah, so that's the question is how much will those people really be impacted 624 00:44:18,948 --> 00:44:21,090 by these changes to the system? 625 00:44:21,090 --> 00:44:23,468 So you can keep it to nothing to be- >> Limited. 626 00:44:23,468 --> 00:44:25,250 >> But otherwise- >> They're limited. 627 00:44:25,250 --> 00:44:27,486 And so who are your end users in this? 628 00:44:27,486 --> 00:44:33,300 Well, they are the executives and what they care I think most about is this. 629 00:44:33,300 --> 00:44:35,466 This is your value delivery, right? 630 00:44:35,466 --> 00:44:40,252 And they also care about this because if they have problem there then they 631 00:44:40,252 --> 00:44:41,360 have a problem. 632 00:44:41,360 --> 00:44:43,250 These, not so much. 633 00:44:43,250 --> 00:44:49,832 Right, just get it done and quickly, that kind of thinking. 634 00:44:49,832 --> 00:44:54,639 So, What has happened 635 00:44:54,639 --> 00:44:59,853 previously though is it was all, that's done over in IT, and 636 00:44:59,853 --> 00:45:05,644 then you throw it over the wall, and that was rather ineffective I think. 637 00:45:05,644 --> 00:45:09,069 And I think that's probably part of the reason why it cost quarter million dollars 638 00:45:09,069 --> 00:45:09,730 to add a field. 639 00:45:09,730 --> 00:45:14,476 So let's see if there's anything else I wanna say about this, and 640 00:45:14,476 --> 00:45:17,671 let me know if you have other questions. 641 00:45:17,671 --> 00:45:21,487 Okay, there's one more thing that I want to say about this. 642 00:45:21,487 --> 00:45:26,040 So structure function in history, take Marty's class, system philosophy. 643 00:45:26,040 --> 00:45:29,900 We talked about structure, that's the hierarchical view. 644 00:45:29,900 --> 00:45:31,549 This is the functional view. 645 00:45:31,549 --> 00:45:34,170 And there's also a historic element of this. 646 00:45:34,170 --> 00:45:39,720 So rewind I would say three, 647 00:45:39,720 --> 00:45:42,790 at least three, probably no more than five years ago. 648 00:45:42,790 --> 00:45:45,360 The guy that I used to work for 649 00:45:45,360 --> 00:45:48,610 that Jackson currently works for that was on Michael. 650 00:45:48,610 --> 00:45:50,450 Michael was his dissertation advisor. 651 00:45:52,800 --> 00:45:54,550 Came up with, you know what? 652 00:45:54,550 --> 00:45:57,780 The analytics community needs to galvanize around some of these key things. 653 00:45:57,780 --> 00:45:59,790 Guess what the key things were? 654 00:45:59,790 --> 00:46:05,740 Well, they access and use, capabilities and tools and metadata. 655 00:46:05,740 --> 00:46:10,640 And then also learning and training and that's really that's being actually 656 00:46:10,640 --> 00:46:13,950 addressed in a different part of the system for this project. 657 00:46:13,950 --> 00:46:18,510 So there is some history to this functional view and need for 658 00:46:18,510 --> 00:46:20,434 these advisory groups. 659 00:46:20,434 --> 00:46:24,482 I think the question then could become, well, are you sure those? 660 00:46:24,482 --> 00:46:25,430 You've tried that before, right? 661 00:46:25,430 --> 00:46:27,720 [LAUGH] Are you sure that that's the right approach? 662 00:46:27,720 --> 00:46:32,800 And I think the only thing I would say to that is I think it was the right approach. 663 00:46:32,800 --> 00:46:35,530 Perhaps just a little too early for the problem. 664 00:46:35,530 --> 00:46:41,608 The problem wasn't ready for the solution is my current worldview of that. 665 00:46:41,608 --> 00:46:44,653 >> But there's vestiges of that coming forward? 666 00:46:44,653 --> 00:46:47,848 >> Yeah, yeah, I know, yes, indeed. 667 00:46:47,848 --> 00:46:50,125 >> I was gonna say you have the advantage now that people have already sort of 668 00:46:50,125 --> 00:46:51,135 been primed for this process. 669 00:46:51,135 --> 00:46:54,068 >> You got it, yep. 670 00:46:54,068 --> 00:46:58,480 >> You've gotten rid of some of that fear of change resistance. 671 00:46:58,480 --> 00:47:00,651 >> Yeah. >> Where it's like well we did this before 672 00:47:00,651 --> 00:47:03,940 and nothing really happened so we can do it again and we'll see. 673 00:47:03,940 --> 00:47:07,750 And then you can actually change it- >> I'm not entirely sold. 674 00:47:09,030 --> 00:47:10,580 I'm gonna keep going. 675 00:47:10,580 --> 00:47:15,960 I did not think I was gonna take the whole hour, but this is fun, okay. 676 00:47:15,960 --> 00:47:18,950 All right, what complexity, this was research question number two. 677 00:47:18,950 --> 00:47:22,430 What complexity related risks might arise as this project matures? 678 00:47:22,430 --> 00:47:23,590 Half dependence. 679 00:47:23,590 --> 00:47:27,800 This is why I just like this Lorenz Curve, look it up. 680 00:47:27,800 --> 00:47:28,704 It's really cool. 681 00:47:28,704 --> 00:47:32,355 >> [LAUGH] >> And I am not gonna do it justice, 682 00:47:32,355 --> 00:47:33,510 I'm not even gonna try. 683 00:47:33,510 --> 00:47:41,090 But what you sort of have are these attractor basins, if you will. 684 00:47:41,090 --> 00:47:44,130 And you've also got kind of some path dependants around here and 685 00:47:44,130 --> 00:47:48,598 some parameters and My working assumption is that, 686 00:47:48,598 --> 00:47:54,211 the advisory groups are functioning as attractors in some way, right? 687 00:47:54,211 --> 00:48:00,231 They're bringing together really smart people in ways so much more effective than 688 00:48:00,231 --> 00:48:05,760 I ever could, if I was trying to stand at the top of some mountains, right? 689 00:48:05,760 --> 00:48:07,576 I'm not doing that, I'm actually kind of allowing, 690 00:48:11,022 --> 00:48:14,444 Analysts to bring each other together. 691 00:48:14,444 --> 00:48:20,990 But path dependence is definitely something that I can see from a mile away. 692 00:48:20,990 --> 00:48:25,730 For example, one of the advisory groups that I'm pretty close with is celebrate. 693 00:48:25,730 --> 00:48:28,420 And we have this two really smart 694 00:48:28,420 --> 00:48:31,870 people that are doing dash boarding in some what different way. 695 00:48:31,870 --> 00:48:36,626 And you are like, my gosh, I can actually see them diverging, and 696 00:48:36,626 --> 00:48:39,339 not could I get us to standardization. 697 00:48:39,339 --> 00:48:43,945 And so, path dependence is one of the things that I have my eye on. 698 00:48:43,945 --> 00:48:48,460 The other complexity related risk that I do actually think a lot about, 699 00:48:48,460 --> 00:48:51,560 kind of gets back to the turtles all the way down. 700 00:48:51,560 --> 00:48:55,562 It's complexity catastrophe or combinatorial complexity. 701 00:48:55,562 --> 00:49:00,753 [INAUDIBLE] there is a lot of handoffs along 702 00:49:00,753 --> 00:49:05,350 this value chain, a lot of places for 703 00:49:05,350 --> 00:49:09,662 things to get difficult or stuck. 704 00:49:09,662 --> 00:49:16,130 And I think that's one of the things that I worry about is, 705 00:49:16,130 --> 00:49:22,470 just freezing the system, because of that complexity. 706 00:49:22,470 --> 00:49:25,109 >> That's also already sort of the problem you have now, right? 707 00:49:25,109 --> 00:49:28,560 Is that you have too many moving parts. 708 00:49:28,560 --> 00:49:30,156 >> Yep. >> Get too many connections, and 709 00:49:30,156 --> 00:49:32,381 it's not clear which are needed and when not. 710 00:49:32,381 --> 00:49:34,889 >> Yep, You got it. 711 00:49:34,889 --> 00:49:40,200 >> So, on some level, yeah, you don't want to recreate your problem, 712 00:49:40,200 --> 00:49:43,319 but you wouldn't be worse off. 713 00:49:43,319 --> 00:49:45,472 >> I agree. It is just definitely something that I 714 00:49:45,472 --> 00:49:48,790 have my eye on and- >> Would you say that 715 00:49:48,790 --> 00:49:52,570 communications becoming more automated with this new system? 716 00:49:52,570 --> 00:49:55,210 >> I think that would be yes. 717 00:49:55,210 --> 00:50:00,270 That would be great, I think they also need to become more transparent. 718 00:50:01,590 --> 00:50:05,360 And I would actually, if prioritizing, I would say transparency first, 719 00:50:05,360 --> 00:50:11,110 then automation, because let me give you an example of not transparent. 720 00:50:11,110 --> 00:50:15,300 Let's say you need to get information about the business rules or 721 00:50:15,300 --> 00:50:18,780 an algorithm that was applied to a specific report. 722 00:50:18,780 --> 00:50:22,976 Well, that could take you six months and six emails, or 723 00:50:22,976 --> 00:50:28,085 you could go to a website that's available to anyone and look it up. 724 00:50:28,085 --> 00:50:29,448 And that will take you six minutes. 725 00:50:29,448 --> 00:50:33,685 And so transparency is my lead foot in this entire project. 726 00:50:33,685 --> 00:50:37,351 And it's actually always a tough sell with a comments or like that. 727 00:50:37,351 --> 00:50:38,374 That's not how we do it. 728 00:50:38,374 --> 00:50:39,577 So, 729 00:50:47,716 --> 00:50:48,535 [INAUDIBLE] okay? 730 00:50:48,535 --> 00:50:55,608 Is anybody else familiar with McKelvey? 731 00:50:55,608 --> 00:50:59,640 It's UCLA, kind of like a social organization, 732 00:50:59,640 --> 00:51:03,672 kind of a guide business, really more business. 733 00:51:03,672 --> 00:51:10,590 But this to me, was what are the sources of organizational tension? 734 00:51:12,630 --> 00:51:17,220 Diverse agendas, different frames of reference, emergent strategies. 735 00:51:17,220 --> 00:51:21,840 I love this one, garbage can decision process [LAUGH] a little bit, 736 00:51:21,840 --> 00:51:26,900 and do some autonomous processes in structured and unstructured processes. 737 00:51:26,900 --> 00:51:32,450 So, when I kind of boiled these up, it seemed to me that 738 00:51:32,450 --> 00:51:37,604 the sources of organizational tension are stemming from multiple perspectives. 739 00:51:37,604 --> 00:51:43,040 And self-organizations have the solution and attention, 740 00:51:43,040 --> 00:51:50,240 I think, in some respects on this, okay? 741 00:51:50,240 --> 00:51:51,790 So, what are we gonna do about it? 742 00:51:51,790 --> 00:51:54,534 Well, we've used some systems tools to guide our thinking, 743 00:51:54,534 --> 00:51:57,301 we're not completely winging it, so that's the good news. 744 00:51:57,301 --> 00:52:00,203 Again, this is more of McKelvey. 745 00:52:00,203 --> 00:52:05,303 But what do you do when you are in an environment that 746 00:52:05,303 --> 00:52:10,890 has co evolutionary dynamics, which we are, I think. 747 00:52:12,360 --> 00:52:17,220 So, risk mitigation is really about the agents, and 748 00:52:17,220 --> 00:52:21,480 about the policies within the environment side as opposed to standards of practices. 749 00:52:21,480 --> 00:52:24,570 So, for the agent heterogeneity, right? 750 00:52:24,570 --> 00:52:27,180 So diversity matters. 751 00:52:27,180 --> 00:52:31,260 Capability, right, you need them to be smart and want to continue to learn. 752 00:52:32,290 --> 00:52:33,450 This is a great one. 753 00:52:33,450 --> 00:52:36,814 And actually, the evidence that small world networks weak-tie connections. 754 00:52:38,240 --> 00:52:41,540 Advisory groups are weak-tie connections. 755 00:52:41,540 --> 00:52:46,589 But I mean, I'm forcing that, you guys should don't talk to me, don't talk to 756 00:52:46,589 --> 00:52:51,891 each other, come up with a recommendation, and I will sell it to the stakeholders. 757 00:52:51,891 --> 00:52:56,631 So weak tie connections is really, I think, important, 758 00:52:56,631 --> 00:53:00,999 because they help disrupt path dependence, right? 759 00:53:00,999 --> 00:53:02,810 When things get rigid. 760 00:53:02,810 --> 00:53:05,930 Your weak-tie connections can help to de-rigidify. 761 00:53:07,900 --> 00:53:11,130 Agents have to know why they're doing stuff, right? 762 00:53:11,130 --> 00:53:14,010 Guess what? Regulatory requirements are changing fast, 763 00:53:14,010 --> 00:53:15,630 we have to change that too. 764 00:53:15,630 --> 00:53:20,850 If your agents don't have any visibility into that, if you're an analyst and don't 765 00:53:20,850 --> 00:53:26,011 have any visibility into that, that's rate limiting to the health of your system. 766 00:53:26,011 --> 00:53:29,090 Yeah, these are kind of the same. 767 00:53:29,090 --> 00:53:33,480 Stay close to your industry, which means stay close to your end users, 768 00:53:33,480 --> 00:53:38,150 because they're the ones who are really close to those contextual drivers. 769 00:53:38,150 --> 00:53:41,799 And then this one I like a lot, lower threshold gates for environmental signals. 770 00:53:41,799 --> 00:53:45,655 Have you ever encountered the not invented here mindset? 771 00:53:45,655 --> 00:53:51,056 That's kind of the anecdotes of the not invented here mindset. 772 00:53:51,056 --> 00:53:53,850 Okay, about the policies. 773 00:53:53,850 --> 00:53:55,560 Wayne, this is your point. 774 00:53:55,560 --> 00:53:57,817 Watch for order creation that inhibits innovation. 775 00:53:57,817 --> 00:54:03,710 Great, you've got all these standards, everybody knows exactly what to do, right? 776 00:54:03,710 --> 00:54:07,230 That's not going to be good for the long term health of your system. 777 00:54:08,570 --> 00:54:14,041 Habitual bureaucratic responses. 778 00:54:14,041 --> 00:54:16,157 I'm sorry, we can't do that for you, 779 00:54:16,157 --> 00:54:18,930 because we need you to fill out this 16 page form. 780 00:54:20,650 --> 00:54:23,620 No, not that, let's not do it that. 781 00:54:23,620 --> 00:54:28,741 >> The only way to come back with a six page description of your request. 782 00:54:28,741 --> 00:54:34,305 >> [LAUGH] yeah, wait in my inbox for six weeks, okay? 783 00:54:34,305 --> 00:54:38,401 And then open organizational boundaries to facilitate flow of new ideas from 784 00:54:38,401 --> 00:54:40,270 the competitive environment. 785 00:54:40,270 --> 00:54:42,919 These are kind of related. 786 00:54:42,919 --> 00:54:46,918 We've talked about some all ready, not invented here, 787 00:54:46,918 --> 00:54:50,850 standards actually could be ignored or rejected. 788 00:54:50,850 --> 00:54:53,077 That's why I say we need both transparency, and 789 00:54:53,077 --> 00:54:54,498 we need broad participation. 790 00:54:54,498 --> 00:54:58,050 Perfect as the enemy of the good. 791 00:54:58,050 --> 00:55:01,435 These advisory groups, they're like baby birds right now. 792 00:55:01,435 --> 00:55:04,862 [LAUGH] I don't want to hold them too tightly, but 793 00:55:04,862 --> 00:55:07,965 I need to protect them at least a little bit. 794 00:55:07,965 --> 00:55:10,895 And I think to give some cohesion to them, 795 00:55:10,895 --> 00:55:14,860 we do need kind of the top down approval mechanism. 796 00:55:14,860 --> 00:55:15,600 Why are you doing this? 797 00:55:15,600 --> 00:55:21,843 Well, because your boss's boss's boss is on the approval committee. 798 00:55:21,843 --> 00:55:28,658 This is kind of a direct, directly from the system's philosophy, right? 799 00:55:28,658 --> 00:55:31,750 It's an imperfect system. 800 00:55:31,750 --> 00:55:33,730 I'm okay with that. 801 00:55:33,730 --> 00:55:37,030 I think the system's tools have been useful to, 802 00:55:38,130 --> 00:55:43,030 The system formation event, in a way that was better than completely winging it. 803 00:55:43,030 --> 00:55:45,674 >> The key here though, is to address what is important. 804 00:55:45,674 --> 00:55:46,850 >> Yeah. 805 00:55:46,850 --> 00:55:49,625 >> Going after the dotting the i's and 806 00:55:49,625 --> 00:55:55,651 crossing the t's might be infeasible in keep you from resolving [INAUDIBLE]. 807 00:55:55,651 --> 00:55:58,955 What you wrote after that doesn't resonate across the lines as much as 808 00:55:58,955 --> 00:56:00,980 some of the others that haven't. 809 00:56:00,980 --> 00:56:02,918 >> Yeah, >> That's it. 810 00:56:02,918 --> 00:56:06,970 I think to get people we need to do the right thing. 811 00:56:06,970 --> 00:56:08,721 >> Yes. >> But not perfectly means. 812 00:56:08,721 --> 00:56:14,074 >> Well and that gets to what you were saying Joe, and 813 00:56:14,074 --> 00:56:19,811 that's I get to your high level guiding principles as 814 00:56:19,811 --> 00:56:25,180 opposed to the specifics and you need both, but. 815 00:56:25,180 --> 00:56:28,610 >> Yeah, it seems to me that the underlying risk there is and 816 00:56:28,610 --> 00:56:33,541 just a tendency of people to compare the strategy to some hypothetical perfect way 817 00:56:33,541 --> 00:56:39,140 of solving problems instead of comparing the new strategy to the current strategy. 818 00:56:39,140 --> 00:56:42,064 Which is the the actual relevant comparison. 819 00:56:42,064 --> 00:56:43,603 >> That's a great observation. 820 00:56:43,603 --> 00:56:47,421 >> So it's yeah, okay, that this is the best way to do it. 821 00:56:47,421 --> 00:56:48,304 >> He cares a lot. 822 00:56:48,304 --> 00:56:49,560 [CROSSTALK] >> [LAUGH] 823 00:56:49,560 --> 00:56:50,382 >> But is it better than what 824 00:56:50,382 --> 00:56:51,013 we're doing now? 825 00:56:51,013 --> 00:56:52,253 Yes, so. 826 00:56:52,253 --> 00:56:53,075 >> Right. 827 00:56:53,075 --> 00:56:54,275 >> We should do it. 828 00:56:54,275 --> 00:56:55,381 >> Yeah. 829 00:56:59,286 --> 00:57:01,274 >> Along the way. 830 00:57:01,274 --> 00:57:03,589 >> [LAUGH] >> You guys have been great. 831 00:57:03,589 --> 00:57:11,123 [APPLAUSE] >> Very nice. 832 00:57:11,123 --> 00:57:12,278 >> I do have one more thing. 833 00:57:12,278 --> 00:57:14,150 >> Good luck. 834 00:57:17,725 --> 00:57:19,239 >> Let's see here, let's go here. 835 00:57:19,239 --> 00:57:23,184 This was in my inbox this morning and 836 00:57:23,184 --> 00:57:29,499 [LAUGH] it seems apropos because these are very generalist 837 00:57:29,499 --> 00:57:36,400 tools that I've applied to a pretty abstract problem. 838 00:57:36,400 --> 00:57:41,996 And we do need general centers, so your in the right place. 839 00:57:41,996 --> 00:57:44,818 >> [LAUGH] >> Cool. 840 00:57:46,484 --> 00:57:49,839 That's all I got. 841 00:57:49,839 --> 00:57:53,543 How many people do you think it is across the board involved from this whole 842 00:57:53,543 --> 00:57:56,365 process, he said it's not big users, there's like- 843 00:57:56,365 --> 00:57:59,410 >> So, we just do the best to deliver or 844 00:57:59,410 --> 00:58:01,657 to receive benefit from? 845 00:58:01,657 --> 00:58:05,838 >> Anyway we would be involved, will be the agents of your system here, 846 00:58:05,838 --> 00:58:07,650 that you are talking about. 847 00:58:07,650 --> 00:58:11,387 >> This is probably like 500 to 1000 and 848 00:58:11,387 --> 00:58:15,835 then the end users are probably like 10,000. 849 00:58:15,835 --> 00:58:20,583 >> Is it specific to the state or is it? 850 00:58:20,583 --> 00:58:23,459 [INAUDIBLE] >> Yes, yes. 851 00:58:23,459 --> 00:58:24,241 >> Okay. Got it. 852 00:58:24,241 --> 00:58:25,306 >> Yeah. 853 00:58:25,306 --> 00:58:35,306 [CROSSTALK] 854 00:59:04,863 --> 00:59:06,004 >> People are excited to be involved. 855 00:59:06,004 --> 00:59:07,467 [CROSSTALK] >> They want to be involved. 856 00:59:07,467 --> 00:59:08,011 >> Yeah. 857 00:59:08,011 --> 00:59:09,807 >> They don't feel that this is an. 858 00:59:09,807 --> 00:59:10,601 >> Yeah, no. 859 00:59:10,601 --> 00:59:12,138 >> Extra. 860 00:59:12,138 --> 00:59:18,026 >> I would say more people than not, like they want to be involved. 861 00:59:18,026 --> 00:59:28,026 [CROSSTALK]