1 00:00:00,000 --> 00:00:04,170 No. Well, I guess like whoa, 2 00:00:04,170 --> 00:00:05,250 it says it's like it was in 3 00:00:05,250 --> 00:00:09,390 progress. Anyway. It's recording. 4 00:00:09,390 --> 00:00:13,980 Okay. Great. Alright. Thanks, Wayne. 5 00:00:13,980 --> 00:00:16,904 I'd like to thank everyone for joining me. 6 00:00:16,904 --> 00:00:21,015 I really appreciate you showing up today. 7 00:00:21,015 --> 00:00:24,809 I would also like 8 00:00:24,809 --> 00:00:26,910 to thank the members of my committee for 9 00:00:26,910 --> 00:00:29,700 providing support to me at 10 00:00:29,700 --> 00:00:33,524 a time when nobody has any spare bandwidth. 11 00:00:33,524 --> 00:00:35,040 I'd also like to thank 12 00:00:35,040 --> 00:00:36,749 the PO organization and 13 00:00:36,749 --> 00:00:38,309 especially the Bainbridge chapter 14 00:00:38,309 --> 00:00:39,209 for awarding me with 15 00:00:39,209 --> 00:00:40,409 a dissertation grant that 16 00:00:40,409 --> 00:00:42,409 allowed me to complete this work. 17 00:00:42,409 --> 00:00:45,235 And I think we have some people 18 00:00:45,235 --> 00:00:46,779 here who are new to systems science. 19 00:00:46,779 --> 00:00:47,859 So I'm going to try to keep 20 00:00:47,859 --> 00:00:50,604 this accessible to newcomers. 21 00:00:50,604 --> 00:00:52,690 And just to confirm you're seeing 22 00:00:52,690 --> 00:00:54,949 my slides, right? 23 00:00:55,500 --> 00:01:00,100 Yes. Right. Okay. So here's 24 00:01:00,100 --> 00:01:02,304 an overview of what I'll be covering. 25 00:01:02,304 --> 00:01:04,465 First, I'm going to give some background 26 00:01:04,465 --> 00:01:06,219 about causal loop diagrams, 27 00:01:06,219 --> 00:01:08,259 what they are, how they're used. 28 00:01:08,259 --> 00:01:10,884 I'll go over my research questions briefly, 29 00:01:10,884 --> 00:01:11,889 and then we'll dig into 30 00:01:11,889 --> 00:01:13,299 the methods and results from 31 00:01:13,299 --> 00:01:15,579 my free papers and 32 00:01:15,579 --> 00:01:17,529 then bring it together with some things, 33 00:01:17,529 --> 00:01:21,070 synthesis and discuss the main takeaways. 34 00:01:21,070 --> 00:01:22,719 And then we'll have some Q and a 35 00:01:22,719 --> 00:01:24,644 like Wayne mentioned. 36 00:01:24,644 --> 00:01:27,230 Now. To start, 37 00:01:27,230 --> 00:01:28,520 causal loop diagrams are 38 00:01:28,520 --> 00:01:30,320 a method for systems science that 39 00:01:30,320 --> 00:01:31,790 describe in a visual 40 00:01:31,790 --> 00:01:33,560 way how different things in 41 00:01:33,560 --> 00:01:34,940 this system are connected to one 42 00:01:34,940 --> 00:01:37,445 another through causal relationships. 43 00:01:37,445 --> 00:01:39,619 And because my research takes 44 00:01:39,619 --> 00:01:42,109 a deep dive into causal loop diagramming, 45 00:01:42,109 --> 00:01:43,640 I wanted to spend a few minutes 46 00:01:43,640 --> 00:01:45,710 going over the mechanics of how they work. 47 00:01:45,710 --> 00:01:48,815 And so this is one classic example. 48 00:01:48,815 --> 00:01:51,140 So let's say you have a population of 49 00:01:51,140 --> 00:01:53,180 chickens, all things being equal. 50 00:01:53,180 --> 00:01:54,634 If you have more chickens, 51 00:01:54,634 --> 00:01:56,165 you have more eggs. 52 00:01:56,165 --> 00:01:57,529 An era with the plus sign 53 00:01:57,529 --> 00:01:58,789 means that the more chickens you have, 54 00:01:58,789 --> 00:02:00,320 more eggs you have as a result, 55 00:02:00,320 --> 00:02:01,939 while if you have more eggs, 56 00:02:01,939 --> 00:02:02,810 that means you end up 57 00:02:02,810 --> 00:02:04,414 with even more chickens. 58 00:02:04,414 --> 00:02:06,349 And this forms a feedback loop. 59 00:02:06,349 --> 00:02:07,430 And this is one is what we 60 00:02:07,430 --> 00:02:09,110 call a reinforcing loop 61 00:02:09,110 --> 00:02:10,549 because it describes 62 00:02:10,549 --> 00:02:12,724 an exponential growth or decline. 63 00:02:12,724 --> 00:02:15,260 So by itself, this diagram tells a story of 64 00:02:15,260 --> 00:02:16,280 a chicken population that 65 00:02:16,280 --> 00:02:18,769 grows exponentially without limits. 66 00:02:18,769 --> 00:02:20,330 Okay? So what if instead 67 00:02:20,330 --> 00:02:22,550 of living in a limitless void, 68 00:02:22,550 --> 00:02:24,619 your chickens live by a busy road. 69 00:02:24,619 --> 00:02:26,660 While we all know what happens when you 70 00:02:26,660 --> 00:02:29,255 have more chickens and more roads, 71 00:02:29,255 --> 00:02:32,285 you get more chickens making bad decisions. 72 00:02:32,285 --> 00:02:34,580 Which meets them are road crossings. 73 00:02:34,580 --> 00:02:35,929 And if you have more road crossings, 74 00:02:35,929 --> 00:02:38,689 what do you get? Fewer chickens? 75 00:02:38,689 --> 00:02:40,070 And so the red arrow with 76 00:02:40,070 --> 00:02:41,510 the negative sign by it means that 77 00:02:41,510 --> 00:02:42,770 the causal relationship works 78 00:02:42,770 --> 00:02:44,299 in the opposite direction. 79 00:02:44,299 --> 00:02:46,699 Road crossings, fewer chickens. 80 00:02:46,699 --> 00:02:48,080 And this forms what we call a 81 00:02:48,080 --> 00:02:49,894 balancing loop because by itself, 82 00:02:49,894 --> 00:02:52,129 it describes a system that over time 83 00:02:52,129 --> 00:02:54,710 comes into balance or homeostasis. 84 00:02:54,710 --> 00:02:56,420 And zooming out, we can see 85 00:02:56,420 --> 00:02:58,069 that the diagram is a whole, 86 00:02:58,069 --> 00:02:59,900 tells a story about what we call the 87 00:02:59,900 --> 00:03:02,345 causal structure of this system. 88 00:03:02,345 --> 00:03:04,009 If you have chickens and a road 89 00:03:04,009 --> 00:03:05,284 and confined space, 90 00:03:05,284 --> 00:03:06,230 these feedback loops 91 00:03:06,230 --> 00:03:08,194 describe what can happen. 92 00:03:08,194 --> 00:03:10,429 And so causal loop diagram notation 93 00:03:10,429 --> 00:03:12,514 with the arrows that go in one direction 94 00:03:12,514 --> 00:03:13,880 that have a positive or negative 95 00:03:13,880 --> 00:03:15,410 valence allow quite a bit 96 00:03:15,410 --> 00:03:16,999 of information to be summarized 97 00:03:16,999 --> 00:03:19,009 in a compact form. 98 00:03:19,009 --> 00:03:21,110 And causal loop diagrams contain 99 00:03:21,110 --> 00:03:23,509 several types of causal structures. 100 00:03:23,509 --> 00:03:25,400 Variables are anything that 101 00:03:25,400 --> 00:03:27,094 exists in the system. 102 00:03:27,094 --> 00:03:30,169 Anything I guess that exists in the system. 103 00:03:30,169 --> 00:03:32,629 Causal links or individual relationships 104 00:03:32,629 --> 00:03:34,025 between variables. 105 00:03:34,025 --> 00:03:36,619 Feedback loops are reinforcing or balancing 106 00:03:36,619 --> 00:03:39,784 structures that explain non-linear behavior. 107 00:03:39,784 --> 00:03:42,950 And archetypes are certain configurations of 108 00:03:42,950 --> 00:03:44,479 feedback loops and links 109 00:03:44,479 --> 00:03:46,310 that reflect common situations. 110 00:03:46,310 --> 00:03:47,645 And so Tragedy of 111 00:03:47,645 --> 00:03:50,225 Commons is one that you might have heard of. 112 00:03:50,225 --> 00:03:51,919 And there's a hierarchy here 113 00:03:51,919 --> 00:03:53,329 where the structures containing 114 00:03:53,329 --> 00:03:55,250 the most causal information are 115 00:03:55,250 --> 00:03:58,410 less commonly found and vice-versa. 116 00:03:58,420 --> 00:04:01,220 So causal loop diagramming started in 117 00:04:01,220 --> 00:04:03,679 the System Dynamics field is a way to 118 00:04:03,679 --> 00:04:05,600 sketch out the main structure of 119 00:04:05,600 --> 00:04:08,284 a system when building a computational model. 120 00:04:08,284 --> 00:04:10,760 And these diagrams have also been used as 121 00:04:10,760 --> 00:04:12,830 standalone tools to help people 122 00:04:12,830 --> 00:04:14,119 get a better understanding 123 00:04:14,119 --> 00:04:15,649 of complex systems. 124 00:04:15,649 --> 00:04:17,239 And they allowed you to get 125 00:04:17,239 --> 00:04:19,010 a bird's eye view of 126 00:04:19,010 --> 00:04:20,089 the key relationships and 127 00:04:20,089 --> 00:04:21,499 dynamics in the system, 128 00:04:21,499 --> 00:04:22,669 what the drivers are, 129 00:04:22,669 --> 00:04:24,454 where the points of leverage may be. 130 00:04:24,454 --> 00:04:26,090 And in my research, I focus on 131 00:04:26,090 --> 00:04:28,294 two different uses of these diagrams. 132 00:04:28,294 --> 00:04:29,524 The first is when 133 00:04:29,524 --> 00:04:31,610 public sector organizations use them to 134 00:04:31,610 --> 00:04:34,025 describe their social programs 135 00:04:34,025 --> 00:04:34,849 are supposed to make 136 00:04:34,849 --> 00:04:36,695 a difference in people's lives. 137 00:04:36,695 --> 00:04:38,959 In the field of program evaluation calls 138 00:04:38,959 --> 00:04:42,275 this sort of explanation program theory. 139 00:04:42,275 --> 00:04:44,435 And this is an example of 140 00:04:44,435 --> 00:04:45,679 a causal loop diagram used 141 00:04:45,679 --> 00:04:47,524 to describe program theory. 142 00:04:47,524 --> 00:04:49,520 You can see interventions acting on 143 00:04:49,520 --> 00:04:51,845 variables that work together in the system. 144 00:04:51,845 --> 00:04:53,659 And more and more, people are using 145 00:04:53,659 --> 00:04:55,699 causal loop diagrams for program theory as 146 00:04:55,699 --> 00:04:57,529 a way to incorporate complexity 147 00:04:57,529 --> 00:04:59,885 into their understanding of social programs. 148 00:04:59,885 --> 00:05:02,089 And they can use them to better understand 149 00:05:02,089 --> 00:05:03,710 how their intervention is 150 00:05:03,710 --> 00:05:05,854 supposed to act on parts of the system. 151 00:05:05,854 --> 00:05:07,700 It helps bring together 152 00:05:07,700 --> 00:05:09,319 an understanding of how contacts and 153 00:05:09,319 --> 00:05:11,060 the intervention worked together 154 00:05:11,060 --> 00:05:13,294 to produce whatever outcomes you see. 155 00:05:13,294 --> 00:05:15,530 And so prior to this, nobody had 156 00:05:15,530 --> 00:05:17,450 brought these examples together and 157 00:05:17,450 --> 00:05:19,639 studied how they were using 158 00:05:19,639 --> 00:05:20,899 causal loop diagramming and 159 00:05:20,899 --> 00:05:22,489 how they might be able to do it better. 160 00:05:22,489 --> 00:05:24,470 So in the first paper, my dissertation, 161 00:05:24,470 --> 00:05:25,940 I do a systematic review of 162 00:05:25,940 --> 00:05:28,624 these studies to do just that. 163 00:05:28,624 --> 00:05:31,054 I also examine how 164 00:05:31,054 --> 00:05:32,659 causal loop diagrams can be used to 165 00:05:32,659 --> 00:05:34,759 describe individuals mental models. 166 00:05:34,759 --> 00:05:37,369 And a mental model is a person set of 167 00:05:37,369 --> 00:05:39,019 assumptions and beliefs about 168 00:05:39,019 --> 00:05:40,940 how the world works. 169 00:05:40,940 --> 00:05:43,099 There are two key questions when developing 170 00:05:43,099 --> 00:05:44,329 causal loop diagrams or 171 00:05:44,329 --> 00:05:45,740 any other systems model, 172 00:05:45,740 --> 00:05:47,899 whose perspective or mental model 173 00:05:47,899 --> 00:05:49,325 are you going to represent? 174 00:05:49,325 --> 00:05:50,809 And how we will get 175 00:05:50,809 --> 00:05:53,000 that perspective from someone's head and into 176 00:05:53,000 --> 00:05:56,090 the model in the system science field has 177 00:05:56,090 --> 00:05:57,799 had a lot of scholarship around how 178 00:05:57,799 --> 00:05:59,599 to engage groups in model building, 179 00:05:59,599 --> 00:06:00,835 but those methods aren't 180 00:06:00,835 --> 00:06:02,720 o is feasible to use. 181 00:06:02,720 --> 00:06:04,534 And they're, oh, Israel, there are only 182 00:06:04,534 --> 00:06:06,769 really designed for working with groups. 183 00:06:06,769 --> 00:06:08,299 And so there's a need for 184 00:06:08,299 --> 00:06:09,620 methods that can use 185 00:06:09,620 --> 00:06:11,209 one-on-one interviewing to 186 00:06:11,209 --> 00:06:13,744 capture individuals mental models. 187 00:06:13,744 --> 00:06:15,830 When the interviews are a great way to get 188 00:06:15,830 --> 00:06:18,080 information from people about 189 00:06:18,080 --> 00:06:19,819 their experiences or their perspective 190 00:06:19,819 --> 00:06:21,560 of a topic because they're 191 00:06:21,560 --> 00:06:22,789 a pretty comfortable format 192 00:06:22,789 --> 00:06:23,690 for people and they 193 00:06:23,690 --> 00:06:25,429 allow people to explain 194 00:06:25,429 --> 00:06:27,740 what they think in their own words. 195 00:06:27,740 --> 00:06:29,359 And so it's more accessible 196 00:06:29,359 --> 00:06:31,279 than asking them to engage 197 00:06:31,279 --> 00:06:32,270 directly and modeling when 198 00:06:32,270 --> 00:06:32,899 they're not used to 199 00:06:32,899 --> 00:06:35,555 that sort of sort of thing. 200 00:06:35,555 --> 00:06:37,099 So people working in 201 00:06:37,099 --> 00:06:38,240 system dynamics or abused 202 00:06:38,240 --> 00:06:39,530 interviews in 203 00:06:39,530 --> 00:06:40,939 model building for quite some time. 204 00:06:40,939 --> 00:06:42,260 But this has typically 205 00:06:42,260 --> 00:06:43,790 been done in formerly and 206 00:06:43,790 --> 00:06:45,440 without a systematic way to 207 00:06:45,440 --> 00:06:47,765 go from interview data to model. 208 00:06:47,765 --> 00:06:49,970 And what I mean by systematic Here 209 00:06:49,970 --> 00:06:52,234 is a method that is transparent, 210 00:06:52,234 --> 00:06:53,449 reproducible, 211 00:06:53,449 --> 00:06:55,280 tracks modeler influence and 212 00:06:55,280 --> 00:06:57,635 is as accurate as possible. 213 00:06:57,635 --> 00:07:00,410 And in 2012, Kim 214 00:07:00,410 --> 00:07:02,509 and Anderson outlined a method for 215 00:07:02,509 --> 00:07:04,220 identifying causal links in 216 00:07:04,220 --> 00:07:05,869 qualitative data and using them 217 00:07:05,869 --> 00:07:08,059 to assemble causal loop diagrams 218 00:07:08,059 --> 00:07:09,859 in stock and flow diagrams, 219 00:07:09,859 --> 00:07:11,375 which are closely related. 220 00:07:11,375 --> 00:07:13,460 An example is shown on the right here. 221 00:07:13,460 --> 00:07:15,500 And when studying this method 222 00:07:15,500 --> 00:07:17,375 and trying it out on some interview data, 223 00:07:17,375 --> 00:07:19,369 I had, I came across 224 00:07:19,369 --> 00:07:21,050 some limitations that I hadn't seen 225 00:07:21,050 --> 00:07:23,104 mentioned in the literature before. 226 00:07:23,104 --> 00:07:24,995 The main one was that the method 227 00:07:24,995 --> 00:07:27,200 involved coding only for 228 00:07:27,200 --> 00:07:30,229 individual causal links rather than 229 00:07:30,229 --> 00:07:31,910 larger causal structures like 230 00:07:31,910 --> 00:07:33,874 feedback loops are archetypes. 231 00:07:33,874 --> 00:07:35,359 And when I was using it, 232 00:07:35,359 --> 00:07:36,589 I found that I miss 233 00:07:36,589 --> 00:07:38,120 some important information 234 00:07:38,120 --> 00:07:39,500 that was being conveyed 235 00:07:39,500 --> 00:07:41,869 because people don't necessarily spell out 236 00:07:41,869 --> 00:07:43,910 every individual causal link in 237 00:07:43,910 --> 00:07:45,410 their mental model when 238 00:07:45,410 --> 00:07:47,090 they're being interviewed. 239 00:07:47,090 --> 00:07:49,955 So to illustrate this point a bit more. 240 00:07:49,955 --> 00:07:52,309 If someone says they're 241 00:07:52,309 --> 00:07:54,739 learning to play the piano, for example, 242 00:07:54,739 --> 00:07:56,600 or an organization says they're training 243 00:07:56,600 --> 00:07:57,799 their employees to have 244 00:07:57,799 --> 00:07:59,420 a certain proficiency. 245 00:07:59,420 --> 00:08:01,159 The underlying causal structure is 246 00:08:01,159 --> 00:08:03,245 a goal directed feedback loop, 247 00:08:03,245 --> 00:08:05,960 balancing feedback loop, which is shown here. 248 00:08:05,960 --> 00:08:08,405 And so if there's a gap 249 00:08:08,405 --> 00:08:11,450 between desired and actual skill level, 250 00:08:11,450 --> 00:08:13,339 the person engages in some training 251 00:08:13,339 --> 00:08:14,884 to improve their skill. 252 00:08:14,884 --> 00:08:16,760 And over time, the current skill level 253 00:08:16,760 --> 00:08:19,070 should trend toward the desired level. 254 00:08:19,070 --> 00:08:20,810 And this is a very common 255 00:08:20,810 --> 00:08:22,160 causal structure and fits 256 00:08:22,160 --> 00:08:23,959 any situation where there's 257 00:08:23,959 --> 00:08:26,794 a gap between the desired and actual state, 258 00:08:26,794 --> 00:08:28,189 which prompts us to 259 00:08:28,189 --> 00:08:29,810 make an improvement in something. 260 00:08:29,810 --> 00:08:32,420 And this is a structure we all understand, 261 00:08:32,420 --> 00:08:33,440 even if we haven't heard of 262 00:08:33,440 --> 00:08:34,925 a goal-directed feedback loop. 263 00:08:34,925 --> 00:08:36,529 So when someone says 264 00:08:36,529 --> 00:08:37,759 they're led me to play the piano, 265 00:08:37,759 --> 00:08:39,694 we understand what that means. 266 00:08:39,694 --> 00:08:40,849 There's, it's gap that they're 267 00:08:40,849 --> 00:08:42,320 trying to address through 268 00:08:42,320 --> 00:08:43,670 some kind of training such as 269 00:08:43,670 --> 00:08:45,575 practicing or taking lessons. 270 00:08:45,575 --> 00:08:46,790 And they expect to try 271 00:08:46,790 --> 00:08:48,439 to get better over time. 272 00:08:48,439 --> 00:08:50,030 And so that phrase learning to 273 00:08:50,030 --> 00:08:51,320 play the piano is 274 00:08:51,320 --> 00:08:52,580 shorthand for this entire 275 00:08:52,580 --> 00:08:55,054 causal structure that isn't, 276 00:08:55,054 --> 00:08:57,319 isn't explicitly mentioned but 277 00:08:57,319 --> 00:08:59,494 is referred to in the interview. 278 00:08:59,494 --> 00:09:01,130 And so this gets at 279 00:09:01,130 --> 00:09:02,600 a key critique I have of 280 00:09:02,600 --> 00:09:04,190 Kim Anderson's method is if 281 00:09:04,190 --> 00:09:05,809 you're only cop coding for 282 00:09:05,809 --> 00:09:07,970 individual causal relationships that are 283 00:09:07,970 --> 00:09:09,469 explicitly mentioned 284 00:09:09,469 --> 00:09:10,805 in your qualitative data, 285 00:09:10,805 --> 00:09:12,169 you're gonna mess larger 286 00:09:12,169 --> 00:09:13,759 causal structures because so 287 00:09:13,759 --> 00:09:15,440 much of our communication 288 00:09:15,440 --> 00:09:17,299 is implied or implicit. 289 00:09:17,299 --> 00:09:20,120 So to really capture someone's mental model, 290 00:09:20,120 --> 00:09:20,810 you need to be able to 291 00:09:20,810 --> 00:09:22,730 recognize and track these 292 00:09:22,730 --> 00:09:23,750 larger imply 293 00:09:23,750 --> 00:09:26,119 causal structures like feedback loops. 294 00:09:26,119 --> 00:09:28,219 So in my second paper, I try to address 295 00:09:28,219 --> 00:09:30,679 this gap by modifying their method. 296 00:09:30,679 --> 00:09:31,970 And then in the third paper, I 297 00:09:31,970 --> 00:09:34,010 present interview strategies designed 298 00:09:34,010 --> 00:09:35,750 for eliciting data suitable 299 00:09:35,750 --> 00:09:38,180 for analysis with this method. 300 00:09:38,180 --> 00:09:40,085 Okay, so that was a lot of 301 00:09:40,085 --> 00:09:41,300 background information 302 00:09:41,300 --> 00:09:42,470 to bring it altogether. 303 00:09:42,470 --> 00:09:43,759 And I'm a research focused 304 00:09:43,759 --> 00:09:45,935 on two main questions. 305 00:09:45,935 --> 00:09:47,539 The first was How have 306 00:09:47,539 --> 00:09:49,760 causal loop diagram has been used to describe 307 00:09:49,760 --> 00:09:52,790 and analyze complexity aware program theory 308 00:09:52,790 --> 00:09:53,929 or program theory that tries 309 00:09:53,929 --> 00:09:55,834 to incorporate complexity. 310 00:09:55,834 --> 00:09:57,470 And how can 311 00:09:57,470 --> 00:09:59,360 interviews be designed, conducted, 312 00:09:59,360 --> 00:10:01,579 and analyzed to identify and 313 00:10:01,579 --> 00:10:04,685 diagram participant's mental models. 314 00:10:04,685 --> 00:10:07,970 So to get into the methods and 315 00:10:07,970 --> 00:10:10,730 results of each paper, 316 00:10:10,730 --> 00:10:12,620 we'll start with the first one. 317 00:10:12,620 --> 00:10:14,539 For this paper, I 318 00:10:14,539 --> 00:10:16,489 conducted a systematic review to 319 00:10:16,489 --> 00:10:17,795 locate studies that use 320 00:10:17,795 --> 00:10:20,779 causal loop diagramming for program theory. 321 00:10:20,779 --> 00:10:23,585 So first, I searched 322 00:10:23,585 --> 00:10:26,090 several databases with relevant search terms 323 00:10:26,090 --> 00:10:28,535 to loci over 300 articles. 324 00:10:28,535 --> 00:10:30,469 I then reviewed the abstracts 325 00:10:30,469 --> 00:10:31,549 that those articles to 326 00:10:31,549 --> 00:10:34,250 find ones that fit a few main criteria. 327 00:10:34,250 --> 00:10:36,319 So to be included the articles had to be 328 00:10:36,319 --> 00:10:38,195 evaluation studies and how to use 329 00:10:38,195 --> 00:10:40,670 causal loop diagramming for program theory. 330 00:10:40,670 --> 00:10:42,860 And what that means is that the diagram 331 00:10:42,860 --> 00:10:46,039 had to include the program's intervention. 332 00:10:46,039 --> 00:10:51,080 Not just mapping dynamics of a system, 333 00:10:51,080 --> 00:10:53,449 like background information on a system. 334 00:10:53,449 --> 00:10:55,610 A lot of articles were excluded because they 335 00:10:55,610 --> 00:10:58,309 use diagramming just for needs assessment, 336 00:10:58,309 --> 00:10:59,449 which by itself doesn't 337 00:10:59,449 --> 00:11:00,859 qualify as program theory, 338 00:11:00,859 --> 00:11:02,149 has to have that intervention 339 00:11:02,149 --> 00:11:04,610 component than it did at 340 00:11:04,610 --> 00:11:08,240 closer review of 60 full-text articles 341 00:11:08,240 --> 00:11:10,145 to see if they fit the criteria. 342 00:11:10,145 --> 00:11:12,709 And in the end, 22 articles relating 343 00:11:12,709 --> 00:11:15,799 to 13 studies were included in the analysis. 344 00:11:15,799 --> 00:11:17,149 So after I had my sample, 345 00:11:17,149 --> 00:11:18,470 I analyze the articles to 346 00:11:18,470 --> 00:11:19,970 identify how the diagrams 347 00:11:19,970 --> 00:11:21,050 are developed and used, 348 00:11:21,050 --> 00:11:22,550 what they looked like and how well they 349 00:11:22,550 --> 00:11:23,720 conform to standards 350 00:11:23,720 --> 00:11:25,414 of causal loop diagramming, 351 00:11:25,414 --> 00:11:28,190 as well as the strengths and limitations of 352 00:11:28,190 --> 00:11:29,389 using this approach that 353 00:11:29,389 --> 00:11:32,179 the authors mentioned in their papers. 354 00:11:32,179 --> 00:11:33,649 I'll flip through 355 00:11:33,649 --> 00:11:35,119 a few examples from these studies. 356 00:11:35,119 --> 00:11:37,024 You can get a general feel for them. 357 00:11:37,024 --> 00:11:39,050 This one was part of a series 358 00:11:39,050 --> 00:11:41,344 of diagrams on this topic. 359 00:11:41,344 --> 00:11:42,769 It describes how teaching 360 00:11:42,769 --> 00:11:43,910 fellows communication 361 00:11:43,910 --> 00:11:45,530 skills improved through interactions 362 00:11:45,530 --> 00:11:47,389 with students and teachers. 363 00:11:47,389 --> 00:11:49,849 This one is created to understand why 364 00:11:49,849 --> 00:11:51,499 a leadership development program for 365 00:11:51,499 --> 00:11:54,275 Health System Managers and Ghana failed. 366 00:11:54,275 --> 00:11:57,230 In this pretty messy one was created 367 00:11:57,230 --> 00:11:59,690 as part of a realist evaluation to 368 00:11:59,690 --> 00:12:01,310 evaluate a 369 00:12:01,310 --> 00:12:03,319 performance-based financing intervention 370 00:12:03,319 --> 00:12:05,929 in Uganda and healthcare sector. 371 00:12:05,929 --> 00:12:07,910 And this one looks at how 372 00:12:07,910 --> 00:12:10,205 a pen-based digital learning intervention 373 00:12:10,205 --> 00:12:12,320 affect student learning in the US. 374 00:12:12,320 --> 00:12:14,059 And so there are three main ways 375 00:12:14,059 --> 00:12:15,499 the diagrams were developed. 376 00:12:15,499 --> 00:12:18,350 Some were using participatory methods 377 00:12:18,350 --> 00:12:19,789 that group model building. 378 00:12:19,789 --> 00:12:23,825 Others were developed using 379 00:12:23,825 --> 00:12:25,490 secondary analysis of 380 00:12:25,490 --> 00:12:27,739 existing evaluation data. 381 00:12:27,739 --> 00:12:29,089 And some of the researchers who took 382 00:12:29,089 --> 00:12:32,239 this approach used Kim Anderson's method, 383 00:12:32,239 --> 00:12:33,890 which I mentioned earlier to 384 00:12:33,890 --> 00:12:36,140 analyze that prior data. 385 00:12:36,140 --> 00:12:37,580 And then some articles 386 00:12:37,580 --> 00:12:38,720 either didn't describe how they 387 00:12:38,720 --> 00:12:40,429 developed their models are used 388 00:12:40,429 --> 00:12:43,125 a modeller lead sort of standard approach. 389 00:12:43,125 --> 00:12:45,550 So some key strengths of 390 00:12:45,550 --> 00:12:47,740 causal loop diagram in this study offers 391 00:12:47,740 --> 00:12:50,409 identified include an better understanding 392 00:12:50,409 --> 00:12:52,554 the system components and behavior, 393 00:12:52,554 --> 00:12:54,190 clarifying the intervention and 394 00:12:54,190 --> 00:12:57,295 implementation and communicating findings. 395 00:12:57,295 --> 00:12:58,810 As some limitations are 396 00:12:58,810 --> 00:13:00,279 that it's resource and time 397 00:13:00,279 --> 00:13:01,870 intensive and for the people 398 00:13:01,870 --> 00:13:03,984 using prior evaluation data, 399 00:13:03,984 --> 00:13:06,880 many of them felt constrained by the focus or 400 00:13:06,880 --> 00:13:08,529 scope of that data because it wasn't 401 00:13:08,529 --> 00:13:10,825 collected with modeling in mind. 402 00:13:10,825 --> 00:13:13,060 And one thing I noticed was that 403 00:13:13,060 --> 00:13:15,069 these studies largely did not cite each 404 00:13:15,069 --> 00:13:16,254 other and didn't have 405 00:13:16,254 --> 00:13:19,150 certain foundational literature in common. 406 00:13:19,150 --> 00:13:21,220 Which means that the researchers 407 00:13:21,220 --> 00:13:23,424 probably decided to use 408 00:13:23,424 --> 00:13:25,904 this causal loop diagramming approach 409 00:13:25,904 --> 00:13:27,919 on their own based on the needs of 410 00:13:27,919 --> 00:13:29,779 their study rather than responding 411 00:13:29,779 --> 00:13:33,479 to an established methods literature. 412 00:13:33,520 --> 00:13:35,870 Another key finding was that 413 00:13:35,870 --> 00:13:37,820 many of the articles included in this study 414 00:13:37,820 --> 00:13:39,530 did not closely adhere to 415 00:13:39,530 --> 00:13:41,690 the standards of causal loop diagramming, 416 00:13:41,690 --> 00:13:43,729 both in visual layout and 417 00:13:43,729 --> 00:13:45,844 how they develop their models. 418 00:13:45,844 --> 00:13:47,600 So for example, 419 00:13:47,600 --> 00:13:50,690 some researchers developed their diagrams by 420 00:13:50,690 --> 00:13:53,539 taking a comprehensive inventory 421 00:13:53,539 --> 00:13:56,104 of all possible variables. 422 00:13:56,104 --> 00:13:59,180 And then inventoried also 423 00:13:59,180 --> 00:14:00,889 what all the possible relationships 424 00:14:00,889 --> 00:14:02,285 between those variables. 425 00:14:02,285 --> 00:14:03,890 And then saw where 426 00:14:03,890 --> 00:14:06,649 the feedback loops may be from there. 427 00:14:06,649 --> 00:14:09,559 Whereas in system dynamics, 428 00:14:09,559 --> 00:14:11,209 it's more of an iterative process 429 00:14:11,209 --> 00:14:12,079 where you want to match 430 00:14:12,079 --> 00:14:15,485 the model structure to a certain behavior. 431 00:14:15,485 --> 00:14:16,129 And you might do 432 00:14:16,129 --> 00:14:18,635 an exercise, the inventories, 433 00:14:18,635 --> 00:14:21,020 variables, relationships, but it's sort of 434 00:14:21,020 --> 00:14:23,835 part of this larger iterative process. 435 00:14:23,835 --> 00:14:26,440 So the main takeaway from this paper was that 436 00:14:26,440 --> 00:14:27,550 causal loop diagramming for 437 00:14:27,550 --> 00:14:29,904 permanent theory has advantages. 438 00:14:29,904 --> 00:14:31,509 And there's a need for 439 00:14:31,509 --> 00:14:33,774 development of best practices. 440 00:14:33,774 --> 00:14:35,589 So in the second paper, 441 00:14:35,589 --> 00:14:37,090 I modified Kim and 442 00:14:37,090 --> 00:14:38,350 Anderson's method 443 00:14:38,350 --> 00:14:40,120 for generating causal loop diagrams 444 00:14:40,120 --> 00:14:41,890 will qualitative data by borrowing 445 00:14:41,890 --> 00:14:45,145 from qualitative methods in system dynamics. 446 00:14:45,145 --> 00:14:47,200 And I'm a bird to 447 00:14:47,200 --> 00:14:48,340 the revised method is 448 00:14:48,340 --> 00:14:49,494 causal structure mapping, 449 00:14:49,494 --> 00:14:51,444 just to have a shorthand for it. 450 00:14:51,444 --> 00:14:53,995 So let's compare these two methods. 451 00:14:53,995 --> 00:14:55,690 So inhuman Anderson's method, 452 00:14:55,690 --> 00:14:57,069 they start with qualitative data, 453 00:14:57,069 --> 00:14:58,180 which is a transcript of 454 00:14:58,180 --> 00:15:00,115 a meeting or an interview. 455 00:15:00,115 --> 00:15:01,870 The first step is to do 456 00:15:01,870 --> 00:15:03,940 a standard qualitative analysis 457 00:15:03,940 --> 00:15:06,570 of the text to discover themes in the data. 458 00:15:06,570 --> 00:15:08,479 Then they go back and identify 459 00:15:08,479 --> 00:15:09,770 causal relationships in 460 00:15:09,770 --> 00:15:11,029 the text which are 461 00:15:11,029 --> 00:15:14,195 individual causal links between variables. 462 00:15:14,195 --> 00:15:16,519 Then they combine those links 463 00:15:16,519 --> 00:15:18,724 into Word and arrow diagrams, 464 00:15:18,724 --> 00:15:22,010 which are small linear model segments. 465 00:15:22,010 --> 00:15:23,495 They do this manually. 466 00:15:23,495 --> 00:15:25,460 And then they combine 467 00:15:25,460 --> 00:15:26,690 these wording arrow diagrams 468 00:15:26,690 --> 00:15:28,069 into a combined model, 469 00:15:28,069 --> 00:15:29,720 which is that this is 470 00:15:29,720 --> 00:15:31,834 also done manually by the modeler. 471 00:15:31,834 --> 00:15:33,349 And then they go back and create 472 00:15:33,349 --> 00:15:34,459 a table summarizing 473 00:15:34,459 --> 00:15:35,809 connections between the model 474 00:15:35,809 --> 00:15:37,519 and the source data. 475 00:15:37,519 --> 00:15:39,995 And the rabbis method that I proposed. 476 00:15:39,995 --> 00:15:41,900 I start by focusing the inquiry, 477 00:15:41,900 --> 00:15:43,639 by familiarizing myself with 478 00:15:43,639 --> 00:15:45,259 the source data in reviewing 479 00:15:45,259 --> 00:15:47,840 the the transcripts and 480 00:15:47,840 --> 00:15:50,149 refining my research questions. 481 00:15:50,149 --> 00:15:52,250 And then I used atlas TI, 482 00:15:52,250 --> 00:15:52,970 which is a type 483 00:15:52,970 --> 00:15:55,325 of qualitative analysis software 484 00:15:55,325 --> 00:15:57,349 to code for different types 485 00:15:57,349 --> 00:15:58,880 of causal structures. 486 00:15:58,880 --> 00:16:00,830 What this means is I use 487 00:16:00,830 --> 00:16:03,500 the software to select portions of text 488 00:16:03,500 --> 00:16:05,539 and then assigned a certain code 489 00:16:05,539 --> 00:16:08,074 corresponding to either causal link, 490 00:16:08,074 --> 00:16:10,130 feedback loop or other structures. 491 00:16:10,130 --> 00:16:11,600 And when I assign that code, 492 00:16:11,600 --> 00:16:12,920 I attached a comment in 493 00:16:12,920 --> 00:16:14,390 which I translated what the purchase 494 00:16:14,390 --> 00:16:15,710 faint was saying into 495 00:16:15,710 --> 00:16:17,479 an abbreviated form of 496 00:16:17,479 --> 00:16:20,100 causal loop diagram notation. 497 00:16:20,410 --> 00:16:23,270 And then after I was done coding, 498 00:16:23,270 --> 00:16:24,529 I used the software to run 499 00:16:24,529 --> 00:16:26,330 a query to identify all of 500 00:16:26,330 --> 00:16:27,410 the quotations that I had 501 00:16:27,410 --> 00:16:29,584 coded as having causal information. 502 00:16:29,584 --> 00:16:31,940 So the query had the direct quote from 503 00:16:31,940 --> 00:16:35,299 the participant alongside the codes and 504 00:16:35,299 --> 00:16:38,449 then the notes I had taken in for all of 505 00:16:38,449 --> 00:16:39,890 the quotations that I coded as 506 00:16:39,890 --> 00:16:42,005 having a feedback loop or archetype. 507 00:16:42,005 --> 00:16:44,780 I reviewed the quotation and my notes to 508 00:16:44,780 --> 00:16:46,339 create freehand sketches 509 00:16:46,339 --> 00:16:48,274 of the causal structures. 510 00:16:48,274 --> 00:16:50,194 And the advantage of using 511 00:16:50,194 --> 00:16:51,440 free hand sketching during 512 00:16:51,440 --> 00:16:52,520 this phase is that it 513 00:16:52,520 --> 00:16:53,930 allows the modeller to review 514 00:16:53,930 --> 00:16:56,270 the source text closely and do 515 00:16:56,270 --> 00:16:58,069 some trial and error when creating 516 00:16:58,069 --> 00:16:59,179 a sketch that captures 517 00:16:59,179 --> 00:17:01,235 what the participant saying. 518 00:17:01,235 --> 00:17:04,220 I created a causal mapping table, 519 00:17:04,220 --> 00:17:05,060 which is a spreadsheet. 520 00:17:05,060 --> 00:17:07,009 The lists all of the causal relationships I 521 00:17:07,009 --> 00:17:08,195 identified through 522 00:17:08,195 --> 00:17:09,829 the coding and through the sketching. 523 00:17:09,829 --> 00:17:12,530 So the causal links were taken directly from 524 00:17:12,530 --> 00:17:14,120 the Query Report and 525 00:17:14,120 --> 00:17:15,139 the sketches were broken 526 00:17:15,139 --> 00:17:16,895 down into causal links. 527 00:17:16,895 --> 00:17:18,500 So then I included 528 00:17:18,500 --> 00:17:20,360 the quotation number as well, 529 00:17:20,360 --> 00:17:22,099 which was automatically generated 530 00:17:22,099 --> 00:17:24,350 through the software 531 00:17:24,350 --> 00:17:26,809 and tracked whether the relationship was 532 00:17:26,809 --> 00:17:28,069 mentioned explicitly 533 00:17:28,069 --> 00:17:29,509 by the interview participant, 534 00:17:29,509 --> 00:17:31,310 was implied, or was 535 00:17:31,310 --> 00:17:33,274 it an assumption that I made? 536 00:17:33,274 --> 00:17:35,600 Then I uploaded the spreadsheet to Kalhu, 537 00:17:35,600 --> 00:17:38,285 which is a web-based visualization platform. 538 00:17:38,285 --> 00:17:40,189 I then reorganize the position of 539 00:17:40,189 --> 00:17:42,170 the variables and made some minor edits to 540 00:17:42,170 --> 00:17:44,300 combine synonyms or leak segments 541 00:17:44,300 --> 00:17:46,790 that were logically tied. 542 00:17:46,790 --> 00:17:49,594 So to compare these two methods, 543 00:17:49,594 --> 00:17:52,730 a key difference is where in the process does 544 00:17:52,730 --> 00:17:54,050 the modeler engage 545 00:17:54,050 --> 00:17:55,550 directly with the source data, 546 00:17:55,550 --> 00:17:55,940 which is 547 00:17:55,940 --> 00:17:57,845 an interview transcript in this case. 548 00:17:57,845 --> 00:18:00,710 So in Kamen Anderson's method on the top, 549 00:18:00,710 --> 00:18:03,920 this only happens early on in their process. 550 00:18:03,920 --> 00:18:07,790 And in the newer method at the bottom, this, 551 00:18:07,790 --> 00:18:10,130 the use of software allowed me to engage 552 00:18:10,130 --> 00:18:11,299 with the quotations during 553 00:18:11,299 --> 00:18:13,085 more of the process. 554 00:18:13,085 --> 00:18:15,889 Okay, so now let's look at where the modeler 555 00:18:15,889 --> 00:18:17,270 constructs loops and 556 00:18:17,270 --> 00:18:19,039 larger structures of the model, 557 00:18:19,039 --> 00:18:20,960 which is the point at which the modeler 558 00:18:20,960 --> 00:18:23,330 has the greatest influence in this process. 559 00:18:23,330 --> 00:18:25,130 So in Kim Anderson's method, 560 00:18:25,130 --> 00:18:27,499 this happens when assembling the Word 561 00:18:27,499 --> 00:18:30,109 and arrow diagrams into a composite model. 562 00:18:30,109 --> 00:18:31,759 Which means that the model 563 00:18:31,759 --> 00:18:32,929 there isn't engaging with 564 00:18:32,929 --> 00:18:34,819 that source text when 565 00:18:34,819 --> 00:18:36,514 making those judgment calls. 566 00:18:36,514 --> 00:18:39,050 They just have that the word a narrow table. 567 00:18:39,050 --> 00:18:41,000 So in the revised method, 568 00:18:41,000 --> 00:18:42,379 these decisions happen when 569 00:18:42,379 --> 00:18:44,539 reviewing the quotations in the query. 570 00:18:44,539 --> 00:18:45,800 And so this helps align 571 00:18:45,800 --> 00:18:47,119 the modelers decisions 572 00:18:47,119 --> 00:18:48,260 about model structure with 573 00:18:48,260 --> 00:18:49,730 the participant's mental model 574 00:18:49,730 --> 00:18:50,180 because they're 575 00:18:50,180 --> 00:18:53,000 engaging directly with that source material. 576 00:18:53,000 --> 00:18:55,490 Another key difference is that because 577 00:18:55,490 --> 00:18:56,509 the newer approach uses 578 00:18:56,509 --> 00:18:58,414 software to help with coding, 579 00:18:58,414 --> 00:18:59,900 an automatically generated 580 00:18:59,900 --> 00:19:01,099 quotation numbers can be 581 00:19:01,099 --> 00:19:04,460 used to track model segments to source data, 582 00:19:04,460 --> 00:19:06,979 which is much more tedious. 583 00:19:06,979 --> 00:19:08,704 And the previous method. 584 00:19:08,704 --> 00:19:11,329 Also the creation of a composite model, 585 00:19:11,329 --> 00:19:13,490 a streamline with the use 586 00:19:13,490 --> 00:19:13,579 of 587 00:19:13,579 --> 00:19:16,610 visualization software than the newer method. 588 00:19:16,610 --> 00:19:18,440 And so I piloted this 589 00:19:18,440 --> 00:19:20,149 mapping approach by conducting 590 00:19:20,149 --> 00:19:21,529 a secondary analysis of 591 00:19:21,529 --> 00:19:22,700 interview data from 592 00:19:22,700 --> 00:19:24,919 an existing study that I'm involved in. 593 00:19:24,919 --> 00:19:26,510 And that's through my work at 594 00:19:26,510 --> 00:19:27,560 the Oregon rural practice 595 00:19:27,560 --> 00:19:28,639 based research network or 596 00:19:28,639 --> 00:19:30,499 or brand OH, US, EU. 597 00:19:30,499 --> 00:19:32,779 And it's an implementation science study 598 00:19:32,779 --> 00:19:34,924 which six staff members from our brand. 599 00:19:34,924 --> 00:19:36,680 Work with primary care clinics 600 00:19:36,680 --> 00:19:38,179 across the state to improve 601 00:19:38,179 --> 00:19:39,650 how the clinic screen for 602 00:19:39,650 --> 00:19:41,705 entry unhealthy alcohol use, 603 00:19:41,705 --> 00:19:43,489 and this type of assistance provided to 604 00:19:43,489 --> 00:19:45,664 clinics is called practice facilitation. 605 00:19:45,664 --> 00:19:47,090 And there's a whole literature on 606 00:19:47,090 --> 00:19:48,679 how practice facilitators can 607 00:19:48,679 --> 00:19:52,054 be most effective in our work with clinics. 608 00:19:52,054 --> 00:19:54,530 And so the research question for 609 00:19:54,530 --> 00:19:55,730 the study has to do with 610 00:19:55,730 --> 00:19:57,470 how practice facilitators Taylor, 611 00:19:57,470 --> 00:19:59,509 the support they provide based 612 00:19:59,509 --> 00:20:00,800 on the needs of the clinic 613 00:20:00,800 --> 00:20:02,689 and their own expertise. 614 00:20:02,689 --> 00:20:06,814 And as part of this research, 615 00:20:06,814 --> 00:20:08,555 raising the method I just 616 00:20:08,555 --> 00:20:11,059 outlined to help address 617 00:20:11,059 --> 00:20:13,190 the question what our practice facilitators, 618 00:20:13,190 --> 00:20:15,439 mental models of practice change. 619 00:20:15,439 --> 00:20:16,909 So the interviews will be 620 00:20:16,909 --> 00:20:18,109 conducted over six months over 621 00:20:18,109 --> 00:20:19,219 the course of the study and we'll 622 00:20:19,219 --> 00:20:20,689 create maps of each interview. 623 00:20:20,689 --> 00:20:21,889 And so at the end of the study, 624 00:20:21,889 --> 00:20:22,910 we can examine how 625 00:20:22,910 --> 00:20:23,959 their mental models changed 626 00:20:23,959 --> 00:20:26,764 over time and compare between individuals. 627 00:20:26,764 --> 00:20:28,219 And so the interviews I used for 628 00:20:28,219 --> 00:20:30,739 this paper where the baseline interviews. 629 00:20:30,739 --> 00:20:33,679 And so I applied this method. 630 00:20:33,679 --> 00:20:35,914 Yet to those six baseline interviews. 631 00:20:35,914 --> 00:20:39,154 The figure might be hard to see, but the, 632 00:20:39,154 --> 00:20:41,120 it shows an example of 633 00:20:41,120 --> 00:20:43,715 a quotation from an interview at the top. 634 00:20:43,715 --> 00:20:45,350 The code that I applied 635 00:20:45,350 --> 00:20:47,014 to it during analysis. 636 00:20:47,014 --> 00:20:50,255 Hi, which indicated I saw a feedback loop. 637 00:20:50,255 --> 00:20:51,950 The notes that I inserted 638 00:20:51,950 --> 00:20:53,060 into the comment field, 639 00:20:53,060 --> 00:20:55,430 what I was doing 640 00:20:55,430 --> 00:20:58,039 the coding to summarize the loop that I saw. 641 00:20:58,039 --> 00:20:59,480 And then the final diagram 642 00:20:59,480 --> 00:21:00,830 which was in the model. 643 00:21:00,830 --> 00:21:03,199 And this feedback loop, this explains how 644 00:21:03,199 --> 00:21:05,780 the more clinicians and staff see 645 00:21:05,780 --> 00:21:06,979 the positive impacts of 646 00:21:06,979 --> 00:21:08,255 doing a brief intervention 647 00:21:08,255 --> 00:21:09,364 for someone who screen 648 00:21:09,364 --> 00:21:11,089 positive for unhealthy alcohol use, 649 00:21:11,089 --> 00:21:13,070 the more buy in they have for 650 00:21:13,070 --> 00:21:14,359 improving how they do 651 00:21:14,359 --> 00:21:15,619 screening and brief intervention, 652 00:21:15,619 --> 00:21:17,839 which is a reinforcing loop. 653 00:21:17,839 --> 00:21:20,495 Here's an example of the same concept. 654 00:21:20,495 --> 00:21:23,000 Clinician and staff buy-in in 655 00:21:23,000 --> 00:21:24,199 diagrams representing 656 00:21:24,199 --> 00:21:27,019 four different participants mental models. 657 00:21:27,019 --> 00:21:29,629 And so this shows how some of 658 00:21:29,629 --> 00:21:31,160 the variables and causal links 659 00:21:31,160 --> 00:21:33,050 were similar between participants. 660 00:21:33,050 --> 00:21:34,850 But there were also quite a few differences 661 00:21:34,850 --> 00:21:36,919 in how densely tied 662 00:21:36,919 --> 00:21:38,839 the variable was to others and whether it 663 00:21:38,839 --> 00:21:40,520 was connected to a feedback loop. 664 00:21:40,520 --> 00:21:41,929 For example. 665 00:21:41,929 --> 00:21:43,940 As I'll show you a few examples 666 00:21:43,940 --> 00:21:45,410 of complete diagrams. 667 00:21:45,410 --> 00:21:47,209 And these diagrams are definitely large, 668 00:21:47,209 --> 00:21:49,565 but they reflect quite a bit of the content 669 00:21:49,565 --> 00:21:52,520 from one hour semi-structured interviews. 670 00:21:52,520 --> 00:21:53,990 And I'll talk a bit later about 671 00:21:53,990 --> 00:21:55,789 how this use of causal loop diagramming 672 00:21:55,789 --> 00:21:57,890 is different from the more traditional uses 673 00:21:57,890 --> 00:21:59,615 of it in system dynamics. 674 00:21:59,615 --> 00:22:01,100 You'll see this one has 675 00:22:01,100 --> 00:22:02,975 quite a few feedback loops 676 00:22:02,975 --> 00:22:04,515 that don't actually doesn't have any, 677 00:22:04,515 --> 00:22:05,570 which reflects that 678 00:22:05,570 --> 00:22:06,769 the participant responded to 679 00:22:06,769 --> 00:22:07,790 questions by listing 680 00:22:07,790 --> 00:22:09,319 things they thought were important. 681 00:22:09,319 --> 00:22:10,070 They didn't go into 682 00:22:10,070 --> 00:22:11,420 a whole lot of detail about how 683 00:22:11,420 --> 00:22:12,740 the variables related to each 684 00:22:12,740 --> 00:22:14,884 other or changed over time. 685 00:22:14,884 --> 00:22:16,519 And the interviewer, 686 00:22:16,519 --> 00:22:17,810 which was me in this case, 687 00:22:17,810 --> 00:22:19,670 could have probably done a better job asking 688 00:22:19,670 --> 00:22:20,780 follow-up questions to 689 00:22:20,780 --> 00:22:22,220 elicit that information. 690 00:22:22,220 --> 00:22:23,509 So this is experience is 691 00:22:23,509 --> 00:22:24,890 one of the things that kind 692 00:22:24,890 --> 00:22:25,969 of put me in the direction of 693 00:22:25,969 --> 00:22:28,579 my third paper Ravel described later. 694 00:22:28,579 --> 00:22:30,664 Here's another example. 695 00:22:30,664 --> 00:22:32,540 When I was using this mapping approach, 696 00:22:32,540 --> 00:22:34,429 I found that it is easier to use and a lot of 697 00:22:34,429 --> 00:22:36,649 ways and less tedious and the prayer method, 698 00:22:36,649 --> 00:22:39,169 but it was definitely still time-consuming. 699 00:22:39,169 --> 00:22:41,629 So the two main takeaways from 700 00:22:41,629 --> 00:22:43,849 this paper that prior methods for 701 00:22:43,849 --> 00:22:45,409 generating causal loop diagrams from 702 00:22:45,409 --> 00:22:47,390 qualitative data did not do 703 00:22:47,390 --> 00:22:48,739 a good enough job capturing 704 00:22:48,739 --> 00:22:51,560 feedback loops in larger causal structures. 705 00:22:51,560 --> 00:22:53,989 And that causal structure mapping 706 00:22:53,989 --> 00:22:55,309 leverages software and 707 00:22:55,309 --> 00:22:57,379 a closer analysis of source texts to 708 00:22:57,379 --> 00:22:58,489 produce diagrams of 709 00:22:58,489 --> 00:23:00,304 individuals mental models. 710 00:23:00,304 --> 00:23:02,450 And main limitation of the study was that 711 00:23:02,450 --> 00:23:03,559 this method was only used 712 00:23:03,559 --> 00:23:04,954 on one small data set. 713 00:23:04,954 --> 00:23:09,019 So future research could apply 714 00:23:09,019 --> 00:23:11,105 this method in various settings 715 00:23:11,105 --> 00:23:13,850 and in test out further. 716 00:23:13,850 --> 00:23:15,349 So during this research, 717 00:23:15,349 --> 00:23:16,370 I also wondered whether 718 00:23:16,370 --> 00:23:17,540 the final diagrams could 719 00:23:17,540 --> 00:23:18,559 have been improved if 720 00:23:18,559 --> 00:23:19,369 the questions I had 721 00:23:19,369 --> 00:23:20,570 asked during the interviews were more 722 00:23:20,570 --> 00:23:23,825 tailored to identifying causal structures. 723 00:23:23,825 --> 00:23:26,225 Which brings me to the third paper. 724 00:23:26,225 --> 00:23:28,189 For this paper, I wanted 725 00:23:28,189 --> 00:23:29,869 to take a critical look at how 726 00:23:29,869 --> 00:23:31,759 interviews can be designed to elicit 727 00:23:31,759 --> 00:23:32,570 information about 728 00:23:32,570 --> 00:23:34,174 participant's mental models. 729 00:23:34,174 --> 00:23:36,200 And so the system scientists have 730 00:23:36,200 --> 00:23:37,459 used interviews for 731 00:23:37,459 --> 00:23:39,259 quite some time to gather information, 732 00:23:39,259 --> 00:23:41,659 but there hasn't been a lot, 733 00:23:41,659 --> 00:23:42,499 much written about how 734 00:23:42,499 --> 00:23:43,729 to design those interviews. 735 00:23:43,729 --> 00:23:45,470 So for this paper, I integrated 736 00:23:45,470 --> 00:23:46,970 some strategies from 737 00:23:46,970 --> 00:23:49,070 three different areas of research. 738 00:23:49,070 --> 00:23:51,695 The first is qualitative interviewing. 739 00:23:51,695 --> 00:23:53,540 There are different flavors 740 00:23:53,540 --> 00:23:54,739 of qualitative interviewing, 741 00:23:54,739 --> 00:23:56,090 but many of them are based 742 00:23:56,090 --> 00:23:58,580 on constructivist philosophical assumptions, 743 00:23:58,580 --> 00:24:00,275 which means they ask 744 00:24:00,275 --> 00:24:01,879 open-ended questions and try to 745 00:24:01,879 --> 00:24:03,259 capture the experience of 746 00:24:03,259 --> 00:24:06,289 the interview participant in their own words. 747 00:24:06,289 --> 00:24:08,330 This approach works great 748 00:24:08,330 --> 00:24:10,100 for standard qualitative research 749 00:24:10,100 --> 00:24:11,405 because it lets the participant 750 00:24:11,405 --> 00:24:14,389 focus on what's important to them. 751 00:24:14,389 --> 00:24:16,400 For model building. 752 00:24:16,400 --> 00:24:18,919 This open-ended neutral approach 753 00:24:18,919 --> 00:24:21,109 works well for initial, 754 00:24:21,109 --> 00:24:22,489 an initial interview with 755 00:24:22,489 --> 00:24:23,690 someone when you trying to 756 00:24:23,690 --> 00:24:25,160 capture the main gist of what 757 00:24:25,160 --> 00:24:26,885 they're saying or their perspective. 758 00:24:26,885 --> 00:24:28,609 But the more I looked into it, 759 00:24:28,609 --> 00:24:31,129 the more I realized that the sort of 760 00:24:31,129 --> 00:24:33,409 constructivist perspective 761 00:24:33,409 --> 00:24:35,090 visited these assumptions 762 00:24:35,090 --> 00:24:37,039 aren't really enough for 763 00:24:37,039 --> 00:24:38,555 an entire modelling project. 764 00:24:38,555 --> 00:24:40,445 Because when you're trying to create 765 00:24:40,445 --> 00:24:41,690 a model to reflect someone 766 00:24:41,690 --> 00:24:43,039 else's perspective. 767 00:24:43,039 --> 00:24:44,765 There's a back and forth 768 00:24:44,765 --> 00:24:46,744 where you're trying to bring 769 00:24:46,744 --> 00:24:49,339 your understanding what they think and 770 00:24:49,339 --> 00:24:50,539 the line into line 771 00:24:50,539 --> 00:24:52,084 with what they actually think. 772 00:24:52,084 --> 00:24:54,290 So you're doing this back and forth 773 00:24:54,290 --> 00:24:55,670 and you 774 00:24:55,670 --> 00:24:56,179 refining 775 00:24:56,179 --> 00:24:57,770 your understanding of their mental model. 776 00:24:57,770 --> 00:24:58,640 There needs to be a bit 777 00:24:58,640 --> 00:24:59,809 when you need to be able to be a bit 778 00:24:59,809 --> 00:25:02,734 more explicit in the questions you ask. 779 00:25:02,734 --> 00:25:05,059 Then is, is sort of 780 00:25:05,059 --> 00:25:06,019 typical and standard 781 00:25:06,019 --> 00:25:07,910 qualitative interviewing. 782 00:25:07,910 --> 00:25:08,899 So that's where 783 00:25:08,899 --> 00:25:10,669 a realist interviewing comes in. 784 00:25:10,669 --> 00:25:12,890 Realists believe that a real-world exists, 785 00:25:12,890 --> 00:25:14,689 but that individual experience plays 786 00:25:14,689 --> 00:25:15,829 a big role in shaping 787 00:25:15,829 --> 00:25:17,434 our understanding of that reality. 788 00:25:17,434 --> 00:25:19,234 So in realist interviewing, 789 00:25:19,234 --> 00:25:20,869 instead of posing a question 790 00:25:20,869 --> 00:25:21,739 and letting the participant 791 00:25:21,739 --> 00:25:22,610 go where they want with it, 792 00:25:22,610 --> 00:25:23,929 the realist interviewer takes 793 00:25:23,929 --> 00:25:25,190 a much more active role 794 00:25:25,190 --> 00:25:27,109 in steering the conversation. 795 00:25:27,109 --> 00:25:29,315 There interview questions are 796 00:25:29,315 --> 00:25:30,889 designed around validating or 797 00:25:30,889 --> 00:25:33,800 falsifying certain hypothesis that 798 00:25:33,800 --> 00:25:36,455 the interviewer comes to the interview with. 799 00:25:36,455 --> 00:25:38,629 My one researcher, men's Ohno, 800 00:25:38,629 --> 00:25:40,430 proposed a three-phase process 801 00:25:40,430 --> 00:25:41,479 for realist interviewing, 802 00:25:41,479 --> 00:25:44,794 which I borrowed for. For this research. 803 00:25:44,794 --> 00:25:46,879 I also wanted to make sure 804 00:25:46,879 --> 00:25:48,170 that the strategies 805 00:25:48,170 --> 00:25:49,669 we're focused on getting information 806 00:25:49,669 --> 00:25:50,689 about causal structures. 807 00:25:50,689 --> 00:25:51,739 So I borrowed from 808 00:25:51,739 --> 00:25:54,139 system dynamics modelling for that. 809 00:25:54,139 --> 00:25:56,239 So this figure shows how 810 00:25:56,239 --> 00:25:57,620 the process that I came up with is 811 00:25:57,620 --> 00:26:00,230 a back and forth with the investigator 812 00:26:00,230 --> 00:26:01,730 or the interviewer on one side 813 00:26:01,730 --> 00:26:03,185 and the participant on the other. 814 00:26:03,185 --> 00:26:04,909 And over time their understanding 815 00:26:04,909 --> 00:26:06,559 comes into closer alignment. 816 00:26:06,559 --> 00:26:09,289 So in the first step, the interviewer defines 817 00:26:09,289 --> 00:26:10,790 the research questions and set 818 00:26:10,790 --> 00:26:11,869 some initial boundaries 819 00:26:11,869 --> 00:26:13,265 for the investigation. 820 00:26:13,265 --> 00:26:15,109 Which could mean that they define 821 00:26:15,109 --> 00:26:16,429 a certain problem or system 822 00:26:16,429 --> 00:26:18,184 behavior they want to understand. 823 00:26:18,184 --> 00:26:19,850 Decide who they want to interview, 824 00:26:19,850 --> 00:26:22,400 or decide that certain things 825 00:26:22,400 --> 00:26:24,845 are outside the scope of the research. 826 00:26:24,845 --> 00:26:27,094 The interviewer then decides, 827 00:26:27,094 --> 00:26:28,760 designs an interview guide and 828 00:26:28,760 --> 00:26:29,780 conducts initial 829 00:26:29,780 --> 00:26:31,399 interviews with participants. 830 00:26:31,399 --> 00:26:32,929 In these interviews are 831 00:26:32,929 --> 00:26:34,909 exploratory and open ended, 832 00:26:34,909 --> 00:26:37,280 but the interviewer follows some guidance for 833 00:26:37,280 --> 00:26:38,450 what to listen for and 834 00:26:38,450 --> 00:26:39,769 how to ask follow-up questions, 835 00:26:39,769 --> 00:26:42,260 which I'll detail in a bit. 836 00:26:42,260 --> 00:26:44,479 After the initial interviews, 837 00:26:44,479 --> 00:26:46,040 the recordings are transcribed 838 00:26:46,040 --> 00:26:47,119 and analyzed according to 839 00:26:47,119 --> 00:26:49,190 the causal structure mapping method 840 00:26:49,190 --> 00:26:51,649 I illustrated and paper too. 841 00:26:51,649 --> 00:26:53,570 So causal loop diagrams are produced 842 00:26:53,570 --> 00:26:56,030 for each interview using that approach. 843 00:26:56,030 --> 00:26:58,520 And after the diagrams are created, 844 00:26:58,520 --> 00:26:59,904 the interviewer analyze it. 845 00:26:59,904 --> 00:27:01,369 Some do identify gaps and 846 00:27:01,369 --> 00:27:03,379 places that need clarification. 847 00:27:03,379 --> 00:27:05,689 Then they produce an interview guide 848 00:27:05,689 --> 00:27:07,759 based on that analysis. 849 00:27:07,759 --> 00:27:09,830 And then follow-up interviews are 850 00:27:09,830 --> 00:27:11,384 conducted that are more 851 00:27:11,384 --> 00:27:13,010 pointed and based on more 852 00:27:13,010 --> 00:27:14,659 of a realist approach designed to 853 00:27:14,659 --> 00:27:16,430 flesh out the modelers understanding 854 00:27:16,430 --> 00:27:19,055 of that participants perspective. 855 00:27:19,055 --> 00:27:21,559 The data from the follow-up interviews is 856 00:27:21,559 --> 00:27:24,934 analyzed using the same mapping method. 857 00:27:24,934 --> 00:27:27,380 And then in sort of an abbreviated form, 858 00:27:27,380 --> 00:27:29,240 and then I integrate 859 00:27:29,240 --> 00:27:31,490 it with the previous diagrams. 860 00:27:31,490 --> 00:27:33,830 And then the model can either be considered 861 00:27:33,830 --> 00:27:35,539 finished or there can be 862 00:27:35,539 --> 00:27:37,010 an optional third session 863 00:27:37,010 --> 00:27:38,360 where the models presented 864 00:27:38,360 --> 00:27:39,439 to the participants for 865 00:27:39,439 --> 00:27:41,750 feedback and an additional editing. 866 00:27:41,750 --> 00:27:44,209 So if doing that participatory review, 867 00:27:44,209 --> 00:27:46,145 final revisions will be made and then 868 00:27:46,145 --> 00:27:47,390 documentation produced 869 00:27:47,390 --> 00:27:49,710 after that final review. 870 00:27:49,810 --> 00:27:52,939 So in my research, I also describe 871 00:27:52,939 --> 00:27:55,730 some interview strategies customized 872 00:27:55,730 --> 00:27:57,530 to the needs of causal loop diagramming. 873 00:27:57,530 --> 00:27:58,865 And I'll briefly go over 874 00:27:58,865 --> 00:28:00,859 how to approach the interview, 875 00:28:00,859 --> 00:28:02,629 what to ask, what to listen for, 876 00:28:02,629 --> 00:28:05,360 and how to probe or ask followup questions. 877 00:28:05,360 --> 00:28:06,859 So in broad strokes, 878 00:28:06,859 --> 00:28:08,524 the initial interview takes 879 00:28:08,524 --> 00:28:10,610 wherever constructivist orientation and 880 00:28:10,610 --> 00:28:12,950 a follow-up interview is a bit more realist. 881 00:28:12,950 --> 00:28:15,170 And interviews are problems 882 00:28:15,170 --> 00:28:16,550 centric in the sense that they're 883 00:28:16,550 --> 00:28:18,319 focused on what is relevant to 884 00:28:18,319 --> 00:28:21,139 understanding a certain problem or a system. 885 00:28:21,139 --> 00:28:24,319 And this is, this comes from system dynamics. 886 00:28:24,319 --> 00:28:27,664 So introductory questions can build rapport, 887 00:28:27,664 --> 00:28:30,050 get people talking, make people comfortable. 888 00:28:30,050 --> 00:28:31,609 You can have questions about 889 00:28:31,609 --> 00:28:33,079 problem definition that are 890 00:28:33,079 --> 00:28:35,285 either open ended. 891 00:28:35,285 --> 00:28:37,910 If you want the participants to take more of 892 00:28:37,910 --> 00:28:38,779 an active role in 893 00:28:38,779 --> 00:28:40,159 defining what the problem is. 894 00:28:40,159 --> 00:28:42,350 Word it could be more narrow depending on 895 00:28:42,350 --> 00:28:43,490 how you define 896 00:28:43,490 --> 00:28:45,290 those boundaries ahead of time. 897 00:28:45,290 --> 00:28:47,089 I, key variables and 898 00:28:47,089 --> 00:28:49,699 relationships are clearly important. 899 00:28:49,699 --> 00:28:51,230 If the mental model has to do 900 00:28:51,230 --> 00:28:52,835 with a certain intervention. 901 00:28:52,835 --> 00:28:55,280 Like for an evaluation and program theory, 902 00:28:55,280 --> 00:28:57,605 the interviewer can ask about 903 00:28:57,605 --> 00:29:01,055 the rationale or logic behind the program. 904 00:29:01,055 --> 00:29:03,200 Questions can also address 905 00:29:03,200 --> 00:29:05,239 desired future outcomes and 906 00:29:05,239 --> 00:29:07,025 we're back to potential causes. 907 00:29:07,025 --> 00:29:09,049 And so far these are relatively 908 00:29:09,049 --> 00:29:10,430 standard for semi-structure 909 00:29:10,430 --> 00:29:12,665 qualitative interviews. 910 00:29:12,665 --> 00:29:14,870 Now the criteria for what 911 00:29:14,870 --> 00:29:16,700 interviewers should listen for and 912 00:29:16,700 --> 00:29:18,889 how to ask questions or more 913 00:29:18,889 --> 00:29:20,089 specialized to the needs 914 00:29:20,089 --> 00:29:21,710 of causal diagramming. 915 00:29:21,710 --> 00:29:23,719 So the main thing to listen 916 00:29:23,719 --> 00:29:25,610 for is a coherent narrative 917 00:29:25,610 --> 00:29:27,260 or explanation 918 00:29:27,260 --> 00:29:29,450 of the participant's mental model. 919 00:29:29,450 --> 00:29:30,515 So the interviewer should 920 00:29:30,515 --> 00:29:31,940 actively listen and make sure they 921 00:29:31,940 --> 00:29:34,909 fully understand what the participant saying. 922 00:29:34,909 --> 00:29:36,470 If there's missing pieces and ask 923 00:29:36,470 --> 00:29:38,989 follow-up questions to fill those gaps. 924 00:29:38,989 --> 00:29:42,185 If someone says something isn't relevant, 925 00:29:42,185 --> 00:29:44,119 that can also be important information 926 00:29:44,119 --> 00:29:45,679 about boundaries and boundaries 927 00:29:45,679 --> 00:29:48,020 include which factors participants 928 00:29:48,020 --> 00:29:49,100 think are the most important 929 00:29:49,100 --> 00:29:50,209 and which are recruit, 930 00:29:50,209 --> 00:29:51,544 which players are involved, 931 00:29:51,544 --> 00:29:53,750 that kind of thing are not involved. 932 00:29:53,750 --> 00:29:55,850 And sometimes people make 933 00:29:55,850 --> 00:29:58,310 explicit references to causal structures. 934 00:29:58,310 --> 00:30:01,294 So for reinforcing loops, 935 00:30:01,294 --> 00:30:02,794 someone could talk about 936 00:30:02,794 --> 00:30:04,969 a vicious cycle or a cascade, 937 00:30:04,969 --> 00:30:06,529 something amplifying, 938 00:30:06,529 --> 00:30:08,689 or something getting out of control. 939 00:30:08,689 --> 00:30:11,539 For. One of the main clues about 940 00:30:11,539 --> 00:30:13,610 a balancing loop is someone talking 941 00:30:13,610 --> 00:30:14,719 about working toward a goal 942 00:30:14,719 --> 00:30:16,114 like I described earlier. 943 00:30:16,114 --> 00:30:17,629 Bouncing feedback is also in 944 00:30:17,629 --> 00:30:19,369 play when someone talks 945 00:30:19,369 --> 00:30:21,680 about something reaching homeostasis 946 00:30:21,680 --> 00:30:24,259 are recovering or stabilizing. 947 00:30:24,259 --> 00:30:26,240 It's also good to listen for 948 00:30:26,240 --> 00:30:27,650 system archetypes which are 949 00:30:27,650 --> 00:30:29,599 certain causal configurations that 950 00:30:29,599 --> 00:30:31,760 can apply to various contexts. 951 00:30:31,760 --> 00:30:33,290 So if someone talks about 952 00:30:33,290 --> 00:30:35,165 short-term solutions undermining, 953 00:30:35,165 --> 00:30:37,250 undermining or fundamental solutions 954 00:30:37,250 --> 00:30:39,290 that shifting the burden on the left, 955 00:30:39,290 --> 00:30:41,029 success to the successful 956 00:30:41,029 --> 00:30:42,079 is in the upper right, 957 00:30:42,079 --> 00:30:45,080 which describes resources being shifted from 958 00:30:45,080 --> 00:30:46,310 an exploited group to 959 00:30:46,310 --> 00:30:48,739 a more ruling group or class. 960 00:30:48,739 --> 00:30:50,450 If someone talks about something 961 00:30:50,450 --> 00:30:51,530 being an arms race, 962 00:30:51,530 --> 00:30:53,179 that's the escalation archetypes of 963 00:30:53,179 --> 00:30:54,769 these terms can clue 964 00:30:54,769 --> 00:30:56,734 you in to these structures. 965 00:30:56,734 --> 00:30:58,910 That's also important to point out that 966 00:30:58,910 --> 00:31:01,490 references to causal structures are, 967 00:31:01,490 --> 00:31:03,529 are often also implied. 968 00:31:03,529 --> 00:31:05,869 So the interviewer needs to 969 00:31:05,869 --> 00:31:08,360 check in and make sure that their, 970 00:31:08,360 --> 00:31:10,579 their understanding and clarifying 971 00:31:10,579 --> 00:31:12,740 when need to, when need be. 972 00:31:12,740 --> 00:31:15,769 So the last design criteria 973 00:31:15,769 --> 00:31:18,050 I propose has to do with probes, 974 00:31:18,050 --> 00:31:19,804 which are follow-up questions. 975 00:31:19,804 --> 00:31:21,350 An interviewer asks to get 976 00:31:21,350 --> 00:31:22,220 more clarity on 977 00:31:22,220 --> 00:31:23,869 something the participant says. 978 00:31:23,869 --> 00:31:25,624 So in a modelling interview, 979 00:31:25,624 --> 00:31:27,065 the interviewer should listen 980 00:31:27,065 --> 00:31:28,909 closely and be prepared to steer 981 00:31:28,909 --> 00:31:31,069 the conversation with probes 982 00:31:31,069 --> 00:31:32,450 to make sure they have 983 00:31:32,450 --> 00:31:33,499 the information they need. 984 00:31:33,499 --> 00:31:34,910 So this can be done through 985 00:31:34,910 --> 00:31:35,989 paraphrasing an idea 986 00:31:35,989 --> 00:31:37,849 back or asking them to 987 00:31:37,849 --> 00:31:40,114 fill certain gaps in understanding. 988 00:31:40,114 --> 00:31:41,885 Asking why and how 989 00:31:41,885 --> 00:31:43,519 questions are important because they 990 00:31:43,519 --> 00:31:45,439 prompt the participant to connect 991 00:31:45,439 --> 00:31:47,779 ideas until that causal story. 992 00:31:47,779 --> 00:31:49,610 What if questions can help you 993 00:31:49,610 --> 00:31:50,690 more information about 994 00:31:50,690 --> 00:31:52,999 underlying structure as well. 995 00:31:52,999 --> 00:31:55,460 So to summarize the advantages 996 00:31:55,460 --> 00:31:57,695 and limitations of this approach. 997 00:31:57,695 --> 00:31:59,419 An advantage is that it 998 00:31:59,419 --> 00:32:01,369 provides a method for understanding 999 00:32:01,369 --> 00:32:02,720 an individual's mental model 1000 00:32:02,720 --> 00:32:03,499 in a way that can be 1001 00:32:03,499 --> 00:32:05,509 compared with other individuals 1002 00:32:05,509 --> 00:32:07,670 or with other points of time. 1003 00:32:07,670 --> 00:32:10,489 The method is participatory in the sense 1004 00:32:10,489 --> 00:32:11,840 that it engages people 1005 00:32:11,840 --> 00:32:13,039 in a modelling process. 1006 00:32:13,039 --> 00:32:14,179 But it's also flexible 1007 00:32:14,179 --> 00:32:16,009 because it only requires one-on-one 1008 00:32:16,009 --> 00:32:17,689 interviews rather than getting 1009 00:32:17,689 --> 00:32:18,859 a whole group at 1010 00:32:18,859 --> 00:32:20,945 the same place in the same time. 1011 00:32:20,945 --> 00:32:23,719 And also leverages the advantages of 1012 00:32:23,719 --> 00:32:27,620 causal structure mapping and the precision 1013 00:32:27,620 --> 00:32:30,019 and tracking embedded in 1014 00:32:30,019 --> 00:32:31,924 this approach makes it 1015 00:32:31,924 --> 00:32:34,699 suitable for use in research. 1016 00:32:34,699 --> 00:32:37,909 So however, the method is time consuming and 1017 00:32:37,909 --> 00:32:39,439 requires both qualitative 1018 00:32:39,439 --> 00:32:42,184 and modeling expertise. 1019 00:32:42,184 --> 00:32:45,020 And there are a few different kinds 1020 00:32:45,020 --> 00:32:46,744 of applications of this method, 1021 00:32:46,744 --> 00:32:48,274 a potential applications 1022 00:32:48,274 --> 00:32:49,700 in system dynamics modelling. 1023 00:32:49,700 --> 00:32:51,889 It could be used to provide more rigor and 1024 00:32:51,889 --> 00:32:53,299 transparency when using 1025 00:32:53,299 --> 00:32:55,535 interviews to build models, 1026 00:32:55,535 --> 00:32:58,219 which could increase model credibility. 1027 00:32:58,219 --> 00:32:59,869 It can also be used as 1028 00:32:59,869 --> 00:33:01,445 a precursor or to 1029 00:33:01,445 --> 00:33:03,650 compliment group model building. 1030 00:33:03,650 --> 00:33:05,390 In program evaluation, 1031 00:33:05,390 --> 00:33:06,859 this approach could be used to help 1032 00:33:06,859 --> 00:33:08,179 map program theory or to 1033 00:33:08,179 --> 00:33:10,940 better understand stakeholder perspectives. 1034 00:33:10,940 --> 00:33:12,589 And then in qualitative research, 1035 00:33:12,589 --> 00:33:13,489 I think there's a lot of 1036 00:33:13,489 --> 00:33:16,925 potential applications to explore 1037 00:33:16,925 --> 00:33:18,889 for any research where understanding and 1038 00:33:18,889 --> 00:33:21,920 comparing mental models is important. 1039 00:33:21,920 --> 00:33:24,365 This might be used is also, I think, 1040 00:33:24,365 --> 00:33:27,949 room for expanding on this research to see 1041 00:33:27,949 --> 00:33:30,980 how shed perspectives of 1042 00:33:30,980 --> 00:33:32,750 different groups be combined 1043 00:33:32,750 --> 00:33:34,864 or compared, that kind of thing. 1044 00:33:34,864 --> 00:33:36,349 And there has been interest 1045 00:33:36,349 --> 00:33:38,629 in various fields within 1046 00:33:38,629 --> 00:33:40,640 the qualitative realm for 1047 00:33:40,640 --> 00:33:42,619 new ways to analyze 1048 00:33:42,619 --> 00:33:44,389 or navigate qualitative data. 1049 00:33:44,389 --> 00:33:45,649 And I think this research that's 1050 00:33:45,649 --> 00:33:47,974 within that broader effort. 1051 00:33:47,974 --> 00:33:51,320 So a couple takeaways from paper three. 1052 00:33:51,320 --> 00:33:53,360 Interviews conducted to inform 1053 00:33:53,360 --> 00:33:54,650 modeling should be designed 1054 00:33:54,650 --> 00:33:55,924 with modelling in mind. 1055 00:33:55,924 --> 00:33:58,070 And this paper proposes 1056 00:33:58,070 --> 00:34:00,109 an interview based method to mapping 1057 00:34:00,109 --> 00:34:01,339 mental models and design 1058 00:34:01,339 --> 00:34:03,379 criteria for interviews. 1059 00:34:03,379 --> 00:34:07,279 So I'd like to take a few minutes to bring 1060 00:34:07,279 --> 00:34:09,320 these three papers together 1061 00:34:09,320 --> 00:34:12,064 and talk about some main takeaways. 1062 00:34:12,064 --> 00:34:15,170 So to recap, in reverse order, 1063 00:34:15,170 --> 00:34:17,255 kind of going left to right here. 1064 00:34:17,255 --> 00:34:18,964 Paper three looks at 1065 00:34:18,964 --> 00:34:21,740 how to elicit mental models from 1066 00:34:21,740 --> 00:34:24,124 interview participants and gather 1067 00:34:24,124 --> 00:34:25,580 qualitative data suitable 1068 00:34:25,580 --> 00:34:27,410 for causal loop diagramming. 1069 00:34:27,410 --> 00:34:29,539 And then paper number two looks 1070 00:34:29,539 --> 00:34:31,519 at how to improve the process of cleaning 1071 00:34:31,519 --> 00:34:32,840 causal loop diagrams from 1072 00:34:32,840 --> 00:34:34,340 that qualitative data 1073 00:34:34,340 --> 00:34:36,500 using causal structure mapping. 1074 00:34:36,500 --> 00:34:38,329 And then paper number one was 1075 00:34:38,329 --> 00:34:40,669 a systematic review of how to, how to, 1076 00:34:40,669 --> 00:34:41,960 how causal loop diagrams are 1077 00:34:41,960 --> 00:34:44,540 used for mapping program theory, 1078 00:34:44,540 --> 00:34:45,559 which is one possible 1079 00:34:45,559 --> 00:34:48,394 application of these methods. 1080 00:34:48,394 --> 00:34:50,899 So there are several contributions to 1081 00:34:50,899 --> 00:34:52,100 knowledge of this research 1082 00:34:52,100 --> 00:34:53,674 that I want to highlight. 1083 00:34:53,674 --> 00:34:57,440 The methods I propose and papers 23 describe 1084 00:34:57,440 --> 00:34:58,775 the importance of capturing 1085 00:34:58,775 --> 00:35:00,109 implied information and 1086 00:35:00,109 --> 00:35:02,255 larger causal structures. 1087 00:35:02,255 --> 00:35:04,580 They also try to better account for 1088 00:35:04,580 --> 00:35:06,799 the role of the modeler as co-creator, 1089 00:35:06,799 --> 00:35:08,119 which who brings 1090 00:35:08,119 --> 00:35:10,580 a certain perspective and experience. 1091 00:35:10,580 --> 00:35:12,680 And the third one requires 1092 00:35:12,680 --> 00:35:14,000 a bit more explanation. 1093 00:35:14,000 --> 00:35:16,009 So conventionally in system dynamics, 1094 00:35:16,009 --> 00:35:17,749 causal loop diagrams are used 1095 00:35:17,749 --> 00:35:19,760 either as a stepping stone to 1096 00:35:19,760 --> 00:35:22,280 computational modelling or as 1097 00:35:22,280 --> 00:35:23,450 an educational tool to 1098 00:35:23,450 --> 00:35:25,715 describe feedback relationships. 1099 00:35:25,715 --> 00:35:27,380 And the goal with both of these is 1100 00:35:27,380 --> 00:35:28,909 to make the diagrams very 1101 00:35:28,909 --> 00:35:30,290 simple in order to 1102 00:35:30,290 --> 00:35:32,539 communicate a few key dynamics. 1103 00:35:32,539 --> 00:35:35,689 And this makes sense if you're using it 1104 00:35:35,689 --> 00:35:37,249 for education or if 1105 00:35:37,249 --> 00:35:39,634 you're going in the direction of simulation. 1106 00:35:39,634 --> 00:35:40,460 Because when you're 1107 00:35:40,460 --> 00:35:41,990 making a simulation model, 1108 00:35:41,990 --> 00:35:43,759 and here I mean computational 1109 00:35:43,759 --> 00:35:45,290 system dynamics model. 1110 00:35:45,290 --> 00:35:47,495 You're trying to create a model that 1111 00:35:47,495 --> 00:35:49,070 reproduces the behavior you 1112 00:35:49,070 --> 00:35:51,440 see in a real-world system. 1113 00:35:51,440 --> 00:35:53,330 As causal loop diagrams. 1114 00:35:53,330 --> 00:35:55,220 The structure of computational model is 1115 00:35:55,220 --> 00:35:56,929 your hypothesis about how 1116 00:35:56,929 --> 00:35:58,489 this real-world system works. 1117 00:35:58,489 --> 00:36:00,200 And so you want your hypothesis to be as 1118 00:36:00,200 --> 00:36:01,760 tiny as possible so you 1119 00:36:01,760 --> 00:36:03,679 can understand what shaping 1120 00:36:03,679 --> 00:36:05,269 the behavior of the system because you're 1121 00:36:05,269 --> 00:36:07,924 going from model two 1122 00:36:07,924 --> 00:36:10,205 to trying to reproduce that behavior. 1123 00:36:10,205 --> 00:36:11,180 And you want to understand 1124 00:36:11,180 --> 00:36:12,725 how your model does that. 1125 00:36:12,725 --> 00:36:15,080 So some researchers in 1126 00:36:15,080 --> 00:36:16,655 system dynamics or critique 1127 00:36:16,655 --> 00:36:18,199 causal loop diagrams as being 1128 00:36:18,199 --> 00:36:20,120 inferior to simulation modelling 1129 00:36:20,120 --> 00:36:21,979 or even misleading because they 1130 00:36:21,979 --> 00:36:23,960 don't quantify the relationships between 1131 00:36:23,960 --> 00:36:26,029 variables aside 1132 00:36:26,029 --> 00:36:28,010 for just that positive and negative valence. 1133 00:36:28,010 --> 00:36:29,734 And so because there's no math 1134 00:36:29,734 --> 00:36:31,669 behind the relationships 1135 00:36:31,669 --> 00:36:32,825 and causal loop diagrams, 1136 00:36:32,825 --> 00:36:34,159 there's no information about what 1137 00:36:34,159 --> 00:36:35,719 we call loop dominance, 1138 00:36:35,719 --> 00:36:38,540 which is how influential one feedback loop 1139 00:36:38,540 --> 00:36:41,359 as compared to others in the model. 1140 00:36:41,359 --> 00:36:43,609 And I have a couple of responses to that. 1141 00:36:43,609 --> 00:36:45,320 But the main one is that 1142 00:36:45,320 --> 00:36:46,609 these critiques of Judge 1143 00:36:46,609 --> 00:36:48,650 causal diagramming by 1144 00:36:48,650 --> 00:36:50,000 the standards of simulation 1145 00:36:50,000 --> 00:36:51,169 modeling when they don't 1146 00:36:51,169 --> 00:36:54,545 necessarily need to have the same goals. 1147 00:36:54,545 --> 00:36:56,240 So causal loop diagramming 1148 00:36:56,240 --> 00:36:57,410 is never going to be better 1149 00:36:57,410 --> 00:36:58,670 than simulation modelling 1150 00:36:58,670 --> 00:37:00,380 at reproducing behavior. 1151 00:37:00,380 --> 00:37:02,299 But what it's really 1152 00:37:02,299 --> 00:37:04,250 good at is representing information 1153 00:37:04,250 --> 00:37:05,810 such as someone's mental model 1154 00:37:05,810 --> 00:37:07,279 in a way that is 1155 00:37:07,279 --> 00:37:11,285 visual and that is relative, 1156 00:37:11,285 --> 00:37:12,875 comparatively simple. 1157 00:37:12,875 --> 00:37:15,469 So they do this in 1158 00:37:15,469 --> 00:37:17,989 a pretty compact and precise way 1159 00:37:17,989 --> 00:37:19,519 using a notation and 1160 00:37:19,519 --> 00:37:21,935 the causal diagramming notation 1161 00:37:21,935 --> 00:37:25,429 makes information combinable with 1162 00:37:25,429 --> 00:37:26,779 other kinds of information. 1163 00:37:26,779 --> 00:37:28,730 So from this perspective, 1164 00:37:28,730 --> 00:37:30,319 causal loop diagrams illustrate 1165 00:37:30,319 --> 00:37:31,669 relationships that people see in 1166 00:37:31,669 --> 00:37:33,589 the world as 1167 00:37:33,589 --> 00:37:36,364 sort of a roadmap of possible outcomes. 1168 00:37:36,364 --> 00:37:38,120 And this is a shift 1169 00:37:38,120 --> 00:37:40,190 towards seeing causal loop diagrams as 1170 00:37:40,190 --> 00:37:42,199 representations of knowledge or 1171 00:37:42,199 --> 00:37:43,880 information or sort of 1172 00:37:43,880 --> 00:37:45,244 a translation 1173 00:37:45,244 --> 00:37:48,740 of a certain type of information. 1174 00:37:48,740 --> 00:37:50,764 In this case, qualitative information. 1175 00:37:50,764 --> 00:37:52,849 Rather than the, rather than just as 1176 00:37:52,849 --> 00:37:55,130 blueprints are the bones 1177 00:37:55,130 --> 00:37:57,210 of a simulation model. 1178 00:37:57,310 --> 00:37:59,869 So another contribution I wanted to 1179 00:37:59,869 --> 00:38:01,880 mention is that in this research, 1180 00:38:01,880 --> 00:38:03,919 I did a lot of comparing and integrating 1181 00:38:03,919 --> 00:38:05,990 approaches from different fields. 1182 00:38:05,990 --> 00:38:07,579 And I noticed a distinction 1183 00:38:07,579 --> 00:38:09,499 between the systems approaches and 1184 00:38:09,499 --> 00:38:11,269 the more standard approaches in 1185 00:38:11,269 --> 00:38:12,410 program evaluation and 1186 00:38:12,410 --> 00:38:14,569 also an implementation science. 1187 00:38:14,569 --> 00:38:17,659 So the more common approaches, 1188 00:38:17,659 --> 00:38:18,604 the centered, 1189 00:38:18,604 --> 00:38:20,719 the intervention and the analysis. 1190 00:38:20,719 --> 00:38:22,249 And by intervention I mean anything 1191 00:38:22,249 --> 00:38:23,930 that's done to address or 1192 00:38:23,930 --> 00:38:25,099 solve a problem such 1193 00:38:25,099 --> 00:38:26,839 as part of a social program 1194 00:38:26,839 --> 00:38:28,129 with being evaluated 1195 00:38:28,129 --> 00:38:29,840 or a clinical intervention. 1196 00:38:29,840 --> 00:38:32,330 So any characteristics of 1197 00:38:32,330 --> 00:38:33,500 the people or the sighting 1198 00:38:33,500 --> 00:38:35,915 involved in the intervention or 1199 00:38:35,915 --> 00:38:38,794 kind of considered local contexts. 1200 00:38:38,794 --> 00:38:39,529 But there's still sort 1201 00:38:39,529 --> 00:38:41,375 of outside the intervention. 1202 00:38:41,375 --> 00:38:43,129 And then there's a sort of broader, 1203 00:38:43,129 --> 00:38:45,770 more social or regulatory context. 1204 00:38:45,770 --> 00:38:47,870 For example, when these aspects 1205 00:38:47,870 --> 00:38:49,894 of context are externalized and are 1206 00:38:49,894 --> 00:38:51,739 only really look that through the lens of 1207 00:38:51,739 --> 00:38:55,160 how the influence that intervention. 1208 00:38:55,160 --> 00:38:57,980 So now this way 1209 00:38:57,980 --> 00:39:00,139 of looking at things isn't wrong per se, 1210 00:39:00,139 --> 00:39:01,640 but centering the intervention in 1211 00:39:01,640 --> 00:39:03,500 this way might get in 1212 00:39:03,500 --> 00:39:07,654 the way of seeing how existing systems work. 1213 00:39:07,654 --> 00:39:09,845 So in systems modeling on the other hand, 1214 00:39:09,845 --> 00:39:11,779 what's central to the analysis is 1215 00:39:11,779 --> 00:39:13,789 a complex understanding of 1216 00:39:13,789 --> 00:39:16,310 how different parts of a system interact 1217 00:39:16,310 --> 00:39:18,575 to produce a system, a certain behavior. 1218 00:39:18,575 --> 00:39:20,270 So variables that we might 1219 00:39:20,270 --> 00:39:22,189 consider contexts such as 1220 00:39:22,189 --> 00:39:24,589 the characteristics of the people 1221 00:39:24,589 --> 00:39:26,780 or the setting involved in the intervention, 1222 00:39:26,780 --> 00:39:28,489 are included in the model based on 1223 00:39:28,489 --> 00:39:29,329 how they interact with 1224 00:39:29,329 --> 00:39:30,484 other parts of the system. 1225 00:39:30,484 --> 00:39:32,720 So after you develop a model of 1226 00:39:32,720 --> 00:39:35,090 how you understand the system to work, 1227 00:39:35,090 --> 00:39:37,220 you can identify possible leverage points 1228 00:39:37,220 --> 00:39:39,319 which are pizza places in the system. 1229 00:39:39,319 --> 00:39:41,689 An intervention might be effective. 1230 00:39:41,689 --> 00:39:44,030 Interventions are seen as acting on 1231 00:39:44,030 --> 00:39:46,549 that preexisting system and 1232 00:39:46,549 --> 00:39:47,750 centering the problem like 1233 00:39:47,750 --> 00:39:49,670 this rather than the intervention, 1234 00:39:49,670 --> 00:39:52,339 might open the door to interventions that are 1235 00:39:52,339 --> 00:39:56,299 more closely tailored to local needs. 1236 00:39:56,299 --> 00:39:58,639 So thinking about the potential 1237 00:39:58,639 --> 00:39:59,719 impact of this research, 1238 00:39:59,719 --> 00:40:00,409 there are a couple of things 1239 00:40:00,409 --> 00:40:01,895 I want to highlight. 1240 00:40:01,895 --> 00:40:03,890 Together. I think these papers 1241 00:40:03,890 --> 00:40:05,614 contribute to a broader effort 1242 00:40:05,614 --> 00:40:06,829 toward more rigor and 1243 00:40:06,829 --> 00:40:08,030 transparency for causal 1244 00:40:08,030 --> 00:40:09,814 loop diagram development. 1245 00:40:09,814 --> 00:40:11,720 And other thing is that capturing 1246 00:40:11,720 --> 00:40:12,859 the mental model of 1247 00:40:12,859 --> 00:40:14,150 an interview participant and 1248 00:40:14,150 --> 00:40:16,729 translating it into a causal loop diagram. 1249 00:40:16,729 --> 00:40:18,470 Essentially putting qualitative 1250 00:40:18,470 --> 00:40:19,580 data into a format 1251 00:40:19,580 --> 00:40:20,840 that can be integrated 1252 00:40:20,840 --> 00:40:22,505 with other data sources. 1253 00:40:22,505 --> 00:40:25,100 So for example, these diagrams could be 1254 00:40:25,100 --> 00:40:26,420 used to integrate findings 1255 00:40:26,420 --> 00:40:28,040 from scientific literature, 1256 00:40:28,040 --> 00:40:30,635 local or stakeholder knowledge and practice, 1257 00:40:30,635 --> 00:40:32,629 practitioner knowledge held by 1258 00:40:32,629 --> 00:40:34,160 people doing an intervention. 1259 00:40:34,160 --> 00:40:35,779 So this opens the door to 1260 00:40:35,779 --> 00:40:38,885 new kinds of decision support tools. 1261 00:40:38,885 --> 00:40:41,060 The biggest limitation of 1262 00:40:41,060 --> 00:40:43,400 this research is that the interview based 1263 00:40:43,400 --> 00:40:45,200 method modelling approach that I outlined in 1264 00:40:45,200 --> 00:40:48,005 the third paper has not been fully piloted. 1265 00:40:48,005 --> 00:40:50,779 I've used many of the strategies mentioned in 1266 00:40:50,779 --> 00:40:51,949 the interviews that I 1267 00:40:51,949 --> 00:40:53,090 conducted for paper too, 1268 00:40:53,090 --> 00:40:54,724 but it wasn't a formal pilot. 1269 00:40:54,724 --> 00:40:57,680 And also the methods and 1270 00:40:57,680 --> 00:40:58,850 paper two also should 1271 00:40:58,850 --> 00:41:01,475 be explored more as well. 1272 00:41:01,475 --> 00:41:03,410 Also, I drew from 1273 00:41:03,410 --> 00:41:05,299 several different fields from this research, 1274 00:41:05,299 --> 00:41:06,590 which means there were multiple 1275 00:41:06,590 --> 00:41:08,210 literatures to draw from. 1276 00:41:08,210 --> 00:41:10,849 And many different types of 1277 00:41:10,849 --> 00:41:12,515 similar sort of 1278 00:41:12,515 --> 00:41:15,590 mapping systems or interview strategies. 1279 00:41:15,590 --> 00:41:17,570 Traditions of every interviewing and 1280 00:41:17,570 --> 00:41:19,954 so exploring every possible angle. 1281 00:41:19,954 --> 00:41:21,529 Literature review wasn't feasible 1282 00:41:21,529 --> 00:41:22,595 in the time I had. 1283 00:41:22,595 --> 00:41:23,719 But there's, there's probably 1284 00:41:23,719 --> 00:41:24,949 a lot more opportunity from 1285 00:41:24,949 --> 00:41:26,480 learning from different perspectives 1286 00:41:26,480 --> 00:41:28,400 and traditions. 1287 00:41:28,400 --> 00:41:30,589 So I see this research is 1288 00:41:30,589 --> 00:41:32,150 forming the basis for 1289 00:41:32,150 --> 00:41:33,995 a lot of future research. 1290 00:41:33,995 --> 00:41:35,689 For example, Best Practices 1291 00:41:35,689 --> 00:41:37,219 for causal loop diagrams for 1292 00:41:37,219 --> 00:41:38,390 program theory could be 1293 00:41:38,390 --> 00:41:39,650 developed based on 1294 00:41:39,650 --> 00:41:41,434 their review and paper won. 1295 00:41:41,434 --> 00:41:43,700 The methods from paper was 23 should be 1296 00:41:43,700 --> 00:41:45,560 tested and refined further. 1297 00:41:45,560 --> 00:41:46,940 And like I mentioned, in 1298 00:41:46,940 --> 00:41:48,020 multiple settings 1299 00:41:48,020 --> 00:41:50,060 and guidance should be developed 1300 00:41:50,060 --> 00:41:52,339 regarding when to use these methods 1301 00:41:52,339 --> 00:41:55,729 versus other existing approaches. 1302 00:41:55,729 --> 00:41:58,254 And finally, the use of these methods, 1303 00:41:58,254 --> 00:41:59,579 sizing different kinds of 1304 00:41:59,579 --> 00:42:02,775 information should be explored. 1305 00:42:02,775 --> 00:42:04,619 So I'd like to leave you with 1306 00:42:04,619 --> 00:42:06,839 a closing thought from Donella Meadows, 1307 00:42:06,839 --> 00:42:08,429 who is a system scientist who's 1308 00:42:08,429 --> 00:42:10,755 inspired me and a lot of my work. 1309 00:42:10,755 --> 00:42:13,079 So people who are raised in 1310 00:42:13,079 --> 00:42:15,509 the industrial world and get hoot and who 1311 00:42:15,509 --> 00:42:16,710 get enthused by systems 1312 00:42:16,710 --> 00:42:17,700 thinking are likely to 1313 00:42:17,700 --> 00:42:20,265 make a terrible mistake. They like. 1314 00:42:20,265 --> 00:42:22,305 They're likely to assume that here. 1315 00:42:22,305 --> 00:42:23,609 And Systems Analysis 1316 00:42:23,609 --> 00:42:24,869 in the interconnection and 1317 00:42:24,869 --> 00:42:27,210 complication and the power of the computer 1318 00:42:27,210 --> 00:42:28,499 here at last is the key 1319 00:42:28,499 --> 00:42:30,100 to prediction and control. 1320 00:42:30,100 --> 00:42:33,019 This mistake is likely because the mindset of 1321 00:42:33,019 --> 00:42:34,700 the industrial world assumes that there 1322 00:42:34,700 --> 00:42:36,875 is a key to prediction and control. 1323 00:42:36,875 --> 00:42:38,569 But social systems are 1324 00:42:38,569 --> 00:42:40,969 the external manifestations of 1325 00:42:40,969 --> 00:42:42,350 cultural thinking patterns 1326 00:42:42,350 --> 00:42:44,240 enough profound human needs, 1327 00:42:44,240 --> 00:42:46,460 emotions, strengths and weaknesses. 1328 00:42:46,460 --> 00:42:48,019 We can't control systems 1329 00:42:48,019 --> 00:42:48,874 are figuring them out, 1330 00:42:48,874 --> 00:42:50,360 but we can dance with them. 1331 00:42:50,360 --> 00:42:52,490 Living successfully in a world of systems 1332 00:42:52,490 --> 00:42:53,719 requires more of us 1333 00:42:53,719 --> 00:42:55,264 than our ability to calculate, 1334 00:42:55,264 --> 00:42:57,019 requires our full humanity, 1335 00:42:57,019 --> 00:42:59,190 our rationality, our ability to sort out 1336 00:42:59,190 --> 00:43:01,370 truth from falsehood or intuition, 1337 00:43:01,370 --> 00:43:04,769 our compassion, our vision, and our morality. 1338 00:43:05,020 --> 00:43:07,190 And I'd like to say thank you again 1339 00:43:07,190 --> 00:43:08,630 for everyone for 1340 00:43:08,630 --> 00:43:10,309 joining me today and I look 1341 00:43:10,309 --> 00:43:13,079 forward to having some discussion. 1342 00:43:14,400 --> 00:43:16,719 Well here my clap, because I, 1343 00:43:16,719 --> 00:43:18,490 I unmuted my back. 1344 00:43:18,490 --> 00:43:21,070 All clapping. That was terrific. 1345 00:43:21,070 --> 00:43:23,079 And I love that quote at the end. 1346 00:43:23,079 --> 00:43:24,459 That was really a nice way to 1347 00:43:24,459 --> 00:43:26,799 cap off your presentation. 1348 00:43:26,799 --> 00:43:30,714 Yeah. And so I'm going to open it up now. 1349 00:43:30,714 --> 00:43:32,964 First to committee members. 1350 00:43:32,964 --> 00:43:35,169 And I guess I 1351 00:43:35,169 --> 00:43:36,609 would like to see is someone who I would 1352 00:43:36,609 --> 00:43:37,749 like to lead off and then 1353 00:43:37,749 --> 00:43:39,280 we'll take a little while to 1354 00:43:39,280 --> 00:43:40,749 give kind of a little bit 1355 00:43:40,749 --> 00:43:42,789 of a priority to committee members. 1356 00:43:42,789 --> 00:43:43,480 And then we'll open it up 1357 00:43:43,480 --> 00:43:44,379 to the audience because 1358 00:43:44,379 --> 00:43:47,150 we'd love to have your interaction as well. 1359 00:43:57,520 --> 00:44:00,739 I can jump in errand. 1360 00:44:00,739 --> 00:44:03,889 And it's more of a speculative question. 1361 00:44:03,889 --> 00:44:04,625 You know, we had 1362 00:44:04,625 --> 00:44:06,230 the opportunity to ask you a lot of 1363 00:44:06,230 --> 00:44:07,310 clarifications on 1364 00:44:07,310 --> 00:44:09,499 the exact dissertation content. 1365 00:44:09,499 --> 00:44:10,999 But one of the things I've 1366 00:44:10,999 --> 00:44:13,099 always appreciated about your work 1367 00:44:13,099 --> 00:44:14,450 coming from an implementation 1368 00:44:14,450 --> 00:44:15,530 science perspective 1369 00:44:15,530 --> 00:44:16,609 is that focus on 1370 00:44:16,609 --> 00:44:18,650 centring head of the problem or 1371 00:44:18,650 --> 00:44:21,694 the system versus centering the intervention. 1372 00:44:21,694 --> 00:44:23,389 And in your presentation 1373 00:44:23,389 --> 00:44:25,160 today when you talked about 1374 00:44:25,160 --> 00:44:27,410 those to you and the opportunity 1375 00:44:27,410 --> 00:44:29,690 that Taylor the intervention, 1376 00:44:29,690 --> 00:44:31,279 it made me actually think we 1377 00:44:31,279 --> 00:44:33,139 haven't talked about the role 1378 00:44:33,139 --> 00:44:35,359 implementation strategies might play 1379 00:44:35,359 --> 00:44:37,220 in bridging those two. 1380 00:44:37,220 --> 00:44:41,659 Um, and so this is totally beyond the scope, 1381 00:44:41,659 --> 00:44:42,725 but do you want to 1382 00:44:42,725 --> 00:44:44,540 talk about that or how you might think 1383 00:44:44,540 --> 00:44:46,459 about integrating that concept into 1384 00:44:46,459 --> 00:44:47,689 some other future work you 1385 00:44:47,689 --> 00:44:49,400 might move forward in 1386 00:44:49,400 --> 00:44:51,500 terms of the activities 1387 00:44:51,500 --> 00:44:52,684 at team might do to help 1388 00:44:52,684 --> 00:44:54,319 change or address a problem 1389 00:44:54,319 --> 00:44:56,720 and an align that to the intervention? 1390 00:44:56,720 --> 00:44:58,760 Sure. That's a good question. 1391 00:44:58,760 --> 00:45:00,439 So I mean, 1392 00:45:00,439 --> 00:45:01,999 I think there's a few things about that. 1393 00:45:01,999 --> 00:45:03,799 I think the first thing that 1394 00:45:03,799 --> 00:45:06,110 comes to mind is something that I 1395 00:45:06,110 --> 00:45:09,199 think is I don't 1396 00:45:09,199 --> 00:45:10,730 know about common knowledge or it's a, 1397 00:45:10,730 --> 00:45:13,160 it's a lesson in system science, 1398 00:45:13,160 --> 00:45:14,990 which is to know your own place 1399 00:45:14,990 --> 00:45:16,399 in the system as well. 1400 00:45:16,399 --> 00:45:19,549 And to understand when 1401 00:45:19,549 --> 00:45:21,920 you're trying to understand a system, 1402 00:45:21,920 --> 00:45:27,680 to try to understand what your influences, 1403 00:45:27,680 --> 00:45:30,995 how your perspective shapes 1404 00:45:30,995 --> 00:45:32,299 your understanding of it. 1405 00:45:32,299 --> 00:45:34,789 And then also how 1406 00:45:34,789 --> 00:45:37,580 you're positioned to understand how 1407 00:45:37,580 --> 00:45:38,719 your position to then be able to 1408 00:45:38,719 --> 00:45:41,690 understand how your where 1409 00:45:41,690 --> 00:45:43,520 your potential points of leverage are or 1410 00:45:43,520 --> 00:45:45,860 where you can have influence. 1411 00:45:45,860 --> 00:45:47,270 And so, I don't know, 1412 00:45:47,270 --> 00:45:49,490 I see implementation strategies as sort of 1413 00:45:49,490 --> 00:45:52,054 part of that conversation as part of, 1414 00:45:52,054 --> 00:45:53,914 you know, first of all, 1415 00:45:53,914 --> 00:45:55,430 when you're really trying to 1416 00:45:55,430 --> 00:45:57,679 understand what that system is, 1417 00:45:57,679 --> 00:45:59,990 is to try to, you know, 1418 00:45:59,990 --> 00:46:03,079 if this is a implementation science study 1419 00:46:03,079 --> 00:46:05,119 where you have practice facilitators working 1420 00:46:05,119 --> 00:46:07,129 with clinics to try to put 1421 00:46:07,129 --> 00:46:10,580 those people into your understanding of okay, 1422 00:46:10,580 --> 00:46:11,809 not just how to clinics work, 1423 00:46:11,809 --> 00:46:13,430 but how does that clinic 1424 00:46:13,430 --> 00:46:15,934 practice facilitator relationship work 1425 00:46:15,934 --> 00:46:17,300 or how does the relationship, 1426 00:46:17,300 --> 00:46:18,845 if you're looking at a nonprofit 1427 00:46:18,845 --> 00:46:20,869 working in a certain community, 1428 00:46:20,869 --> 00:46:22,565 how does that, how, 1429 00:46:22,565 --> 00:46:23,959 where is that non-profit 1430 00:46:23,959 --> 00:46:25,670 situated in that community, 1431 00:46:25,670 --> 00:46:26,960 and how do they, how can you 1432 00:46:26,960 --> 00:46:28,309 build on those relationships? 1433 00:46:28,309 --> 00:46:32,479 And so I would see that intervention, 1434 00:46:32,479 --> 00:46:34,189 implementation strategies is kind of 1435 00:46:34,189 --> 00:46:36,590 coming out of that understanding of those, 1436 00:46:36,590 --> 00:46:39,710 that embeddedness and then also trying 1437 00:46:39,710 --> 00:46:43,399 to kind of reassess and see. 1438 00:46:43,399 --> 00:46:44,809 When you when you have 1439 00:46:44,809 --> 00:46:46,594 an intervention finding court, 1440 00:46:46,594 --> 00:46:48,365 you incorporate that into your model, 1441 00:46:48,365 --> 00:46:49,130 you know, because that is 1442 00:46:49,130 --> 00:46:50,405 part of your system now. 1443 00:46:50,405 --> 00:46:51,380 And then trying to 1444 00:46:51,380 --> 00:46:52,955 constantly reassess as you, 1445 00:46:52,955 --> 00:46:54,259 as you go along and hopefully 1446 00:46:54,259 --> 00:46:56,705 having that kind of feedback loop of 1447 00:46:56,705 --> 00:47:02,029 constant improvement at Air. 1448 00:47:02,029 --> 00:47:03,380 And I was going to suggest if you want 1449 00:47:03,380 --> 00:47:04,864 to unshare for a second, 1450 00:47:04,864 --> 00:47:06,020 David will have the gallery 1451 00:47:06,020 --> 00:47:07,189 view and we can kinda all 1452 00:47:07,189 --> 00:47:08,449 see each other and sometimes 1453 00:47:08,449 --> 00:47:09,709 that's a good way to yeah, 1454 00:47:09,709 --> 00:47:10,759 I wasn't sure if I would have 1455 00:47:10,759 --> 00:47:12,230 to switch back to my sides, 1456 00:47:12,230 --> 00:47:13,459 but I can always do that. 1457 00:47:13,459 --> 00:47:15,200 Of course, this way people 1458 00:47:15,200 --> 00:47:16,159 can sort of see each other 1459 00:47:16,159 --> 00:47:17,269 who was in the room door. 1460 00:47:17,269 --> 00:47:18,860 Yeah, that's okay. 1461 00:47:18,860 --> 00:47:22,290 Other questions from the community? 1462 00:47:22,510 --> 00:47:27,470 Back over. Comment. Any technical question? 1463 00:47:27,470 --> 00:47:29,975 So I think my my comment 1464 00:47:29,975 --> 00:47:31,220 is what I really like about 1465 00:47:31,220 --> 00:47:32,660 your work is that you are 1466 00:47:32,660 --> 00:47:34,819 taking your using system dynamics. 1467 00:47:34,819 --> 00:47:36,800 I think in all honesty, you should say, well, 1468 00:47:36,800 --> 00:47:38,899 very much have the mindset 1469 00:47:38,899 --> 00:47:41,150 of a systems dynamics model or you're looking 1470 00:47:41,150 --> 00:47:43,159 for archetypes and you're looking for 1471 00:47:43,159 --> 00:47:44,539 the language of system 1472 00:47:44,539 --> 00:47:45,979 dynamics and those interviews, 1473 00:47:45,979 --> 00:47:47,300 you're not out there getting 1474 00:47:47,300 --> 00:47:49,954 mental models like psychologists do this. 1475 00:47:49,954 --> 00:47:51,680 You're a systems dynamics model, 1476 00:47:51,680 --> 00:47:54,965 but what you do is you are really thinking of 1477 00:47:54,965 --> 00:47:56,869 people contributing knowledge that's 1478 00:47:56,869 --> 00:47:59,225 relevant about the systems. It's not this. 1479 00:47:59,225 --> 00:48:01,400 The model will tell you all the answers, 1480 00:48:01,400 --> 00:48:03,604 but the people will tell you all the answers. 1481 00:48:03,604 --> 00:48:05,149 And I, and I think what 1482 00:48:05,149 --> 00:48:06,485 was missing a little bit 1483 00:48:06,485 --> 00:48:08,720 is clearly the integration 1484 00:48:08,720 --> 00:48:10,114 happens to the model, right? 1485 00:48:10,114 --> 00:48:11,299 Because you have all these 1486 00:48:11,299 --> 00:48:13,640 different mental models. 1487 00:48:13,640 --> 00:48:16,070 And to do program evaluation, 1488 00:48:16,070 --> 00:48:17,689 you have to smoosh them together in 1489 00:48:17,689 --> 00:48:18,980 some way or form through 1490 00:48:18,980 --> 00:48:19,999 the systems dynamics. 1491 00:48:19,999 --> 00:48:20,839 Well, and I really love 1492 00:48:20,839 --> 00:48:21,829 that because I think that, 1493 00:48:21,829 --> 00:48:24,035 that respects the knowledge of, 1494 00:48:24,035 --> 00:48:26,180 of people in social systems, 1495 00:48:26,180 --> 00:48:28,925 what system dynamics sometimes does. 1496 00:48:28,925 --> 00:48:31,174 A great job. 1497 00:48:31,174 --> 00:48:32,600 I think that's super awesome. 1498 00:48:32,600 --> 00:48:36,110 My Indian reveal technical question is, 1499 00:48:36,110 --> 00:48:37,520 so atlas TI has 1500 00:48:37,520 --> 00:48:40,519 the capabilities to define relationship. 1501 00:48:40,519 --> 00:48:41,930 You can say this concept has 1502 00:48:41,930 --> 00:48:43,385 a relationship with this concept. 1503 00:48:43,385 --> 00:48:44,810 Yet you seem to have gone to 1504 00:48:44,810 --> 00:48:47,870 a table format and using your, 1505 00:48:47,870 --> 00:48:50,779 your kind of hand-drawn things. 1506 00:48:50,779 --> 00:48:52,519 Why that why did 1507 00:48:52,519 --> 00:48:53,810 you choose to do it this way? 1508 00:48:53,810 --> 00:48:55,939 Yeah. So from what I understand, 1509 00:48:55,939 --> 00:48:57,079 I a I played 1510 00:48:57,079 --> 00:48:58,820 around with that feature and atlas TI, 1511 00:48:58,820 --> 00:49:00,320 but from what I understand, 1512 00:49:00,320 --> 00:49:03,410 and I've seen some research use 1513 00:49:03,410 --> 00:49:06,680 similar features I think, and NVivo. 1514 00:49:06,680 --> 00:49:10,745 But what I understand those concepts link, 1515 00:49:10,745 --> 00:49:12,470 you can only link codes. 1516 00:49:12,470 --> 00:49:13,729 From what I understand is 1517 00:49:13,729 --> 00:49:15,214 that you can make codes, 1518 00:49:15,214 --> 00:49:17,300 which then obviously codes are 1519 00:49:17,300 --> 00:49:19,489 tied to certain sections of text. 1520 00:49:19,489 --> 00:49:22,489 But it was what 1521 00:49:22,489 --> 00:49:24,889 I was playing around with that it 1522 00:49:24,889 --> 00:49:30,410 seemed to what I was interested in, 1523 00:49:30,410 --> 00:49:33,379 I think was, um, 1524 00:49:33,379 --> 00:49:35,899 so linking codes like 1525 00:49:35,899 --> 00:49:38,150 that I felt was interesting, 1526 00:49:38,150 --> 00:49:39,589 but it's a different type of inquiry 1527 00:49:39,589 --> 00:49:40,609 from when I was doing, 1528 00:49:40,609 --> 00:49:43,189 I felt that the what 1529 00:49:43,189 --> 00:49:45,530 was missing in some of this prior research 1530 00:49:45,530 --> 00:49:48,110 was kind of giving credence to 1531 00:49:48,110 --> 00:49:49,280 the role of the modeler 1532 00:49:49,280 --> 00:49:51,169 in their interpretive abilities. 1533 00:49:51,169 --> 00:49:53,480 And this is something that I feel like the 1534 00:49:53,480 --> 00:49:54,769 reach the writing 1535 00:49:54,769 --> 00:49:56,209 on qualitative research does 1536 00:49:56,209 --> 00:49:58,910 well and is sort of giving 1537 00:49:58,910 --> 00:50:03,020 respect or kind of recognizing that the, 1538 00:50:03,020 --> 00:50:05,929 that the analyst brings with them. 1539 00:50:05,929 --> 00:50:07,429 Some things that are important 1540 00:50:07,429 --> 00:50:08,540 for interpretation. 1541 00:50:08,540 --> 00:50:13,174 And so for the way I had, that's the, 1542 00:50:13,174 --> 00:50:16,940 this system setup is that by looking at 1543 00:50:16,940 --> 00:50:21,004 a quotation and having the, 1544 00:50:21,004 --> 00:50:22,970 the source, you know, so the, 1545 00:50:22,970 --> 00:50:24,139 the quotation number is 1546 00:50:24,139 --> 00:50:25,489 connected to a certain quotation. 1547 00:50:25,489 --> 00:50:26,570 So it's maybe a paragraph of 1548 00:50:26,570 --> 00:50:27,769 text if someone telling, 1549 00:50:27,769 --> 00:50:29,674 explaining something, you know. 1550 00:50:29,674 --> 00:50:32,750 And so it's like it's associated 1551 00:50:32,750 --> 00:50:34,295 with like a whole thought 1552 00:50:34,295 --> 00:50:36,050 or a collection of thoughts. 1553 00:50:36,050 --> 00:50:37,430 And so within that there can be 1554 00:50:37,430 --> 00:50:38,945 different things going on. 1555 00:50:38,945 --> 00:50:41,929 You know, there can be different codes, 1556 00:50:41,929 --> 00:50:44,599 there can be different levels of things. 1557 00:50:44,599 --> 00:50:45,799 And so what I wanted 1558 00:50:45,799 --> 00:50:47,060 to do was to be able to create 1559 00:50:47,060 --> 00:50:50,420 a system where the modeller can track, okay, 1560 00:50:50,420 --> 00:50:53,884 here's a meaningful section of text and then 1561 00:50:53,884 --> 00:50:55,430 hear them and apply 1562 00:50:55,430 --> 00:50:57,080 their understanding and to 1563 00:50:57,080 --> 00:50:58,279 be able to see, okay. 1564 00:50:58,279 --> 00:50:59,119 Especially when you're looking 1565 00:50:59,119 --> 00:51:01,730 at that implied information, 1566 00:51:01,730 --> 00:51:02,960 if someone makes a like 1567 00:51:02,960 --> 00:51:04,460 an offhand reference, for example, 1568 00:51:04,460 --> 00:51:06,080 to something being a vicious cycle, 1569 00:51:06,080 --> 00:51:07,219 but then doesn't say 1570 00:51:07,219 --> 00:51:08,570 anything more about that. 1571 00:51:08,570 --> 00:51:10,939 You know, it wouldn't necessarily come up if 1572 00:51:10,939 --> 00:51:13,985 all your coding for is just, 1573 00:51:13,985 --> 00:51:15,845 you know, individual concepts 1574 00:51:15,845 --> 00:51:17,150 or something like that. 1575 00:51:17,150 --> 00:51:18,980 So I wanted to be able to see, okay, 1576 00:51:18,980 --> 00:51:20,270 to have the analyst 1577 00:51:20,270 --> 00:51:21,410 to be able to look at that, 1578 00:51:21,410 --> 00:51:22,849 look at the context and 1579 00:51:22,849 --> 00:51:24,559 then be able to pull out, Oh, 1580 00:51:24,559 --> 00:51:26,689 this is what they mean because I wanted to be 1581 00:51:26,689 --> 00:51:29,225 able to, you know, 1582 00:51:29,225 --> 00:51:33,140 rely on the, the intelligence of the analyst 1583 00:51:33,140 --> 00:51:34,940 to be able to interpret 1584 00:51:34,940 --> 00:51:37,220 what the person's saying. 1585 00:51:37,220 --> 00:51:38,869 And so I think there's a lot 1586 00:51:38,869 --> 00:51:41,149 of room for improvement. 1587 00:51:41,149 --> 00:51:42,050 And what I would love to 1588 00:51:42,050 --> 00:51:43,340 see is to be able to leverage 1589 00:51:43,340 --> 00:51:46,190 the power of that software, 1590 00:51:46,190 --> 00:51:49,054 to be able to, you know, 1591 00:51:49,054 --> 00:51:50,870 do not have to do 1592 00:51:50,870 --> 00:51:52,309 the manual spreadsheet like that 1593 00:51:52,309 --> 00:51:54,154 was kind of, you know, 1594 00:51:54,154 --> 00:51:56,360 it's a lot more clunky than it needs to be if 1595 00:51:56,360 --> 00:51:57,679 the software would allow you to 1596 00:51:57,679 --> 00:51:59,419 take a section of text and say, 1597 00:51:59,419 --> 00:52:02,690 okay, heres the causal structure 1598 00:52:02,690 --> 00:52:05,270 that is tied to this. 1599 00:52:05,270 --> 00:52:06,770 I think that would be awesome. 1600 00:52:06,770 --> 00:52:08,900 But from what I understand, 1601 00:52:08,900 --> 00:52:10,610 the way codes, the 1602 00:52:10,610 --> 00:52:12,169 code connections are built like that. 1603 00:52:12,169 --> 00:52:13,310 It's like this higher level 1604 00:52:13,310 --> 00:52:19,500 those relationships. Thank you. 1605 00:52:22,630 --> 00:52:25,640 Have a couple of questions. 1606 00:52:25,640 --> 00:52:27,349 So first era and I just, I 1607 00:52:27,349 --> 00:52:28,430 just want to say thank you very 1608 00:52:28,430 --> 00:52:29,509 much for the presentation and 1609 00:52:29,509 --> 00:52:30,995 for the excellent work. 1610 00:52:30,995 --> 00:52:32,120 I also want to tell you 1611 00:52:32,120 --> 00:52:32,989 that this is probably one of 1612 00:52:32,989 --> 00:52:35,119 the best dissertation research 1613 00:52:35,119 --> 00:52:36,229 presentations that i've seen. 1614 00:52:36,229 --> 00:52:38,689 So congratulations on doing 1615 00:52:38,689 --> 00:52:39,949 an excellent job all 1616 00:52:39,949 --> 00:52:42,004 around. I'm super impressed. 1617 00:52:42,004 --> 00:52:43,669 I have a couple of 1618 00:52:43,669 --> 00:52:45,560 different kinds of questions 1619 00:52:45,560 --> 00:52:49,070 related to the program evaluation piece, 1620 00:52:49,070 --> 00:52:50,659 the program's theory piece of things being 1621 00:52:50,659 --> 00:52:53,479 the program evaluator of the group. 1622 00:52:53,479 --> 00:52:55,609 And one of the questions I want to 1623 00:52:55,609 --> 00:52:58,280 ask you about is this idea of 1624 00:52:58,280 --> 00:53:02,870 where this approach to developing 1625 00:53:02,870 --> 00:53:05,030 diagrams and to developing program theory 1626 00:53:05,030 --> 00:53:07,870 fits within the idea of, 1627 00:53:07,870 --> 00:53:10,204 of how generally program theory is developed. 1628 00:53:10,204 --> 00:53:11,659 Because I think that what 1629 00:53:11,659 --> 00:53:12,950 you're describing here is 1630 00:53:12,950 --> 00:53:15,604 an incredibly important approach to 1631 00:53:15,604 --> 00:53:16,940 making explicit a lot 1632 00:53:16,940 --> 00:53:18,320 of assumptions and mental models, 1633 00:53:18,320 --> 00:53:19,339 both about problems and 1634 00:53:19,339 --> 00:53:22,385 interventions that as you know, 1635 00:53:22,385 --> 00:53:23,810 we've talked about are typically 1636 00:53:23,810 --> 00:53:25,310 not made explicit by people 1637 00:53:25,310 --> 00:53:26,660 who are engaged in a lot of 1638 00:53:26,660 --> 00:53:28,589 public and social service work. 1639 00:53:28,589 --> 00:53:30,260 That people don't have the time 1640 00:53:30,260 --> 00:53:31,730 or don't take the time to make 1641 00:53:31,730 --> 00:53:33,169 explicit a lot of these assumptions 1642 00:53:33,169 --> 00:53:33,889 about what they're doing 1643 00:53:33,889 --> 00:53:35,045 and how they're doing it. 1644 00:53:35,045 --> 00:53:37,370 But in the developmental program theory from, 1645 00:53:37,370 --> 00:53:38,870 from a program evaluation model 1646 00:53:38,870 --> 00:53:43,265 is that asking stakeholders in 1647 00:53:43,265 --> 00:53:45,200 programs to make 1648 00:53:45,200 --> 00:53:48,380 explicit their assumptions and 1649 00:53:48,380 --> 00:53:51,560 their models about the program 1650 00:53:51,560 --> 00:53:52,160 and what they're doing and 1651 00:53:52,160 --> 00:53:52,670 how they're doing it 1652 00:53:52,670 --> 00:53:54,109 as incredibly important piece 1653 00:53:54,109 --> 00:53:55,010 of program theory and 1654 00:53:55,010 --> 00:53:56,270 developing program theory. 1655 00:53:56,270 --> 00:53:58,445 But so also is 1656 00:53:58,445 --> 00:54:00,349 that juxtaposition with what people 1657 00:54:00,349 --> 00:54:01,279 think is happening or 1658 00:54:01,279 --> 00:54:02,209 where they assumed to be 1659 00:54:02,209 --> 00:54:03,259 happening and what their 1660 00:54:03,259 --> 00:54:04,594 experience tells them as happening. 1661 00:54:04,594 --> 00:54:07,385 With existing research that 1662 00:54:07,385 --> 00:54:09,169 has been done about larger issues in terms of 1663 00:54:09,169 --> 00:54:10,489 how we understand the problem being 1664 00:54:10,489 --> 00:54:11,839 addressed and how other kinds of 1665 00:54:11,839 --> 00:54:13,429 interventions that may be similar 1666 00:54:13,429 --> 00:54:15,110 have addresses these problems. 1667 00:54:15,110 --> 00:54:16,250 And so I'm wondering within 1668 00:54:16,250 --> 00:54:17,750 your model here about 1669 00:54:17,750 --> 00:54:19,160 how this helps to make 1670 00:54:19,160 --> 00:54:20,285 the stakeholder positions 1671 00:54:20,285 --> 00:54:21,800 so much more explicit. 1672 00:54:21,800 --> 00:54:23,119 How is that then 1673 00:54:23,119 --> 00:54:25,369 juxtaposed with existing research that, 1674 00:54:25,369 --> 00:54:28,490 that Ben is juxtaposed 1675 00:54:28,490 --> 00:54:30,079 together to develop a program theory that 1676 00:54:30,079 --> 00:54:31,595 is that has been formed by 1677 00:54:31,595 --> 00:54:33,440 both the larger issues as 1678 00:54:33,440 --> 00:54:34,820 well as those localized issues as 1679 00:54:34,820 --> 00:54:37,369 you've spoken so eloquently about? 1680 00:54:37,369 --> 00:54:39,109 Yeah. No, that's a great question. 1681 00:54:39,109 --> 00:54:41,360 Let me share again this slide that I 1682 00:54:41,360 --> 00:54:42,979 didn't share early and that I 1683 00:54:42,979 --> 00:54:45,630 think speaks to this. 1684 00:54:50,850 --> 00:54:53,980 So this is from 1685 00:54:53,980 --> 00:54:58,270 a proposal for a K award that I have. 1686 00:54:58,270 --> 00:55:00,624 I'm still waiting to hear background, 1687 00:55:00,624 --> 00:55:02,080 so I don't know if I'm going to do 1688 00:55:02,080 --> 00:55:03,730 this research, but it's, 1689 00:55:03,730 --> 00:55:06,250 the idea here is that I'm using 1690 00:55:06,250 --> 00:55:09,249 this coding analysis to describe 1691 00:55:09,249 --> 00:55:12,685 stakeholder mental models and then 1692 00:55:12,685 --> 00:55:16,644 integrating those diagrams with 1693 00:55:16,644 --> 00:55:18,805 different forms of data. 1694 00:55:18,805 --> 00:55:21,369 So I think a strength of this, 1695 00:55:21,369 --> 00:55:23,799 of the causal loop diagramming form 1696 00:55:23,799 --> 00:55:25,375 in particular is that because 1697 00:55:25,375 --> 00:55:28,809 it's ultimately kinda decomposable 1698 00:55:28,809 --> 00:55:30,939 to spreadsheet form, right? 1699 00:55:30,939 --> 00:55:33,460 And so it's, it's, it's, 1700 00:55:33,460 --> 00:55:36,849 you can, because it 1701 00:55:36,849 --> 00:55:38,619 provides a sort of common language. 1702 00:55:38,619 --> 00:55:40,600 And so in this example, 1703 00:55:40,600 --> 00:55:43,689 we're integrating were I proposed to 1704 00:55:43,689 --> 00:55:45,340 map stakeholder perspectives 1705 00:55:45,340 --> 00:55:47,320 from patients and clinicians. 1706 00:55:47,320 --> 00:55:50,169 This is an implementation science study. 1707 00:55:50,169 --> 00:55:52,239 And then integrating it with 1708 00:55:52,239 --> 00:55:54,624 meeting transcripts that they have. 1709 00:55:54,624 --> 00:55:56,200 Clinic contact logs, which is 1710 00:55:56,200 --> 00:55:58,240 another type of qualitative information. 1711 00:55:58,240 --> 00:56:00,294 Scientific literature as well, 1712 00:56:00,294 --> 00:56:02,229 participatory review, 1713 00:56:02,229 --> 00:56:04,194 you also engaging people in it, 1714 00:56:04,194 --> 00:56:05,980 and then outcomes data as well. 1715 00:56:05,980 --> 00:56:07,930 So using even the quantitative data 1716 00:56:07,930 --> 00:56:09,594 to help form, 1717 00:56:09,594 --> 00:56:11,109 to help shape okay, 1718 00:56:11,109 --> 00:56:14,080 where, which parts of a system 1719 00:56:14,080 --> 00:56:17,329 is maybe more relevant. 1720 00:56:17,329 --> 00:56:19,234 So I see a lot of, 1721 00:56:19,234 --> 00:56:20,839 this is where I think there's a lot of 1722 00:56:20,839 --> 00:56:25,610 promising research in this, 1723 00:56:25,610 --> 00:56:28,385 for this research proposal and talking about 1724 00:56:28,385 --> 00:56:31,384 customizing evidence-based interventions 1725 00:56:31,384 --> 00:56:32,750 using these diagrams. 1726 00:56:32,750 --> 00:56:34,699 But I think the, the main idea 1727 00:56:34,699 --> 00:56:36,184 being that, you know, 1728 00:56:36,184 --> 00:56:38,554 I would love for there to be more 1729 00:56:38,554 --> 00:56:40,159 and for me that I'd 1730 00:56:40,159 --> 00:56:41,539 love to be involved in this research, 1731 00:56:41,539 --> 00:56:43,549 but for there to be more research on how 1732 00:56:43,549 --> 00:56:48,530 to translate different types of information. 1733 00:56:48,530 --> 00:56:50,644 So information from literature, 1734 00:56:50,644 --> 00:56:52,790 for example, into, into diagrams. 1735 00:56:52,790 --> 00:56:54,575 And so there's, there's some research 1736 00:56:54,575 --> 00:56:58,880 out there already on. 1737 00:56:58,880 --> 00:57:03,740 I'm using either automated AI type methods 1738 00:57:03,740 --> 00:57:06,695 to kind of understand 1739 00:57:06,695 --> 00:57:07,880 causal information or 1740 00:57:07,880 --> 00:57:09,754 identify causal information. 1741 00:57:09,754 --> 00:57:11,959 I think there's, you know, 1742 00:57:11,959 --> 00:57:13,339 there's maybe some pros and cons to 1743 00:57:13,339 --> 00:57:15,125 that line of inquiry, but there's, 1744 00:57:15,125 --> 00:57:17,419 there's some research out there about how to 1745 00:57:17,419 --> 00:57:20,330 use how to use 1746 00:57:20,330 --> 00:57:22,774 causal loop diagrams in 1747 00:57:22,774 --> 00:57:26,570 systematic reviews of scientific literature. 1748 00:57:26,570 --> 00:57:28,159 And so I think there's, because 1749 00:57:28,159 --> 00:57:30,154 these diagrams are, 1750 00:57:30,154 --> 00:57:32,209 it can kinda put information 1751 00:57:32,209 --> 00:57:34,009 on the same plane. 1752 00:57:34,009 --> 00:57:36,289 E. I think there's a lot of room for 1753 00:57:36,289 --> 00:57:38,210 development of those methods as far as 1754 00:57:38,210 --> 00:57:40,040 like what that might actually look like 1755 00:57:40,040 --> 00:57:43,519 in a evaluation, I think, 1756 00:57:43,519 --> 00:57:47,015 you know, I think we would, it would, 1757 00:57:47,015 --> 00:57:48,919 we'd have to kind of 1758 00:57:48,919 --> 00:57:53,180 balance the use of methods like this 1759 00:57:53,180 --> 00:57:56,479 for sort of being systematic and 1760 00:57:56,479 --> 00:57:58,729 being maybe a bit more precise 1761 00:57:58,729 --> 00:58:01,655 with other methods to meet other needs. 1762 00:58:01,655 --> 00:58:03,530 Because maybe it would be 1763 00:58:03,530 --> 00:58:05,390 blended with some group modelling to 1764 00:58:05,390 --> 00:58:07,309 kind of engage people and make them feel 1765 00:58:07,309 --> 00:58:09,649 more ownership of it. 1766 00:58:09,649 --> 00:58:12,604 Or maybe there's, you know, 1767 00:58:12,604 --> 00:58:14,989 you can incorporate components 1768 00:58:14,989 --> 00:58:16,459 of lit review that then are 1769 00:58:16,459 --> 00:58:17,900 summarized in certain areas 1770 00:58:17,900 --> 00:58:18,679 and you could be able to 1771 00:58:18,679 --> 00:58:23,555 trace where different information comes from. 1772 00:58:23,555 --> 00:58:25,580 So yeah, I don't think it needs 1773 00:58:25,580 --> 00:58:27,020 to take obviously take place 1774 00:58:27,020 --> 00:58:30,319 within the larger question of OK, 1775 00:58:30,319 --> 00:58:31,969 what's right for this argument? 1776 00:58:31,969 --> 00:58:32,449 You know, what's right 1777 00:58:32,449 --> 00:58:33,679 for that organization, right? 1778 00:58:33,679 --> 00:58:35,120 Or what's right for? 1779 00:58:35,120 --> 00:58:37,130 What are your, your evaluation questions, 1780 00:58:37,130 --> 00:58:38,480 what are your research questions? 1781 00:58:38,480 --> 00:58:40,969 And so I'm hoping this research kinda fits 1782 00:58:40,969 --> 00:58:44,225 within adding to the toolbox. 1783 00:58:44,225 --> 00:58:45,664 But it doesn't necessary 1784 00:58:45,664 --> 00:58:47,270 definitely doesn't replace 1785 00:58:47,270 --> 00:58:49,970 the traditions and the systems 1786 00:58:49,970 --> 00:58:51,695 that are already there. 1787 00:58:51,695 --> 00:58:53,509 Thank you. That's super helpful. 1788 00:58:53,509 --> 00:58:54,290 And actually, I think what you 1789 00:58:54,290 --> 00:58:55,550 did was just lead into 1790 00:58:55,550 --> 00:58:56,855 my next question which is 1791 00:58:56,855 --> 00:58:57,770 and I don't think I 1792 00:58:57,770 --> 00:58:58,790 could like Melinda question. 1793 00:58:58,790 --> 00:59:00,170 It's it's a little bit you know, 1794 00:59:00,170 --> 00:59:01,730 it's a little bit outside of the scope 1795 00:59:01,730 --> 00:59:04,610 of what you've done here, which is fantastic. 1796 00:59:04,610 --> 00:59:07,070 But whenever I kind of thought you've given 1797 00:59:07,070 --> 00:59:10,249 to implementing that's not 1798 00:59:10,249 --> 00:59:11,299 just in a research context, 1799 00:59:11,299 --> 00:59:13,159 but actually in practice in terms of working 1800 00:59:13,159 --> 00:59:16,610 with public and non-profit organizations 1801 00:59:16,610 --> 00:59:18,049 to implement more of this approach. 1802 00:59:18,049 --> 00:59:19,099 You know, at the beginning you talk 1803 00:59:19,099 --> 00:59:20,929 about this idea 1804 00:59:20,929 --> 00:59:22,310 of introducing more complexity 1805 00:59:22,310 --> 00:59:23,929 into programs, into program theory. 1806 00:59:23,929 --> 00:59:25,579 And for someone like me 1807 00:59:25,579 --> 00:59:27,800 who works a lot with non-profits and 1808 00:59:27,800 --> 00:59:30,349 does a lot of work around trying 1809 00:59:30,349 --> 00:59:31,880 to introduce more evaluative thinking 1810 00:59:31,880 --> 00:59:33,409 into organizations. 1811 00:59:33,409 --> 00:59:36,680 That's hard. And introducing 1812 00:59:36,680 --> 00:59:37,909 even more complexity into 1813 00:59:37,909 --> 00:59:39,289 the Universe in a lot 1814 00:59:39,289 --> 00:59:41,135 of these non-profits and public organizations 1815 00:59:41,135 --> 00:59:42,200 exist in a universe 1816 00:59:42,200 --> 00:59:46,849 where the easiest answer is that the best? 1817 00:59:46,849 --> 00:59:48,919 Like people like simple answers, 1818 00:59:48,919 --> 00:59:50,149 especially when it comes 1819 00:59:50,149 --> 00:59:51,965 to entrenched social problems 1820 00:59:51,965 --> 00:59:53,660 and actually investing in addressing 1821 00:59:53,660 --> 00:59:55,414 those kinds of issues is incredibly hard. 1822 00:59:55,414 --> 00:59:56,570 Everybody want simple answers and 1823 00:59:56,570 --> 00:59:57,769 so I'm just curious 1824 00:59:57,769 --> 00:59:58,969 what kind of thinking you've 1825 00:59:58,969 --> 01:00:00,110 done in terms of like, 1826 01:00:00,110 --> 01:00:02,885 how would you bring this approach to funders? 1827 01:00:02,885 --> 01:00:04,039 Like how would you bring us approach 1828 01:00:04,039 --> 01:00:06,075 to organizations to 1829 01:00:06,075 --> 01:00:07,880 add even more complexity into something 1830 01:00:07,880 --> 01:00:10,100 that they already find kind of complex? 1831 01:00:10,100 --> 01:00:12,020 Yeah. No, it's a good question. 1832 01:00:12,020 --> 01:00:12,979 Um, I think there's 1833 01:00:12,979 --> 01:00:14,929 a few different answers here. 1834 01:00:14,929 --> 01:00:21,410 One is that there's an existing practice. 1835 01:00:21,410 --> 01:00:23,524 I feel the practice in literature around 1836 01:00:23,524 --> 01:00:24,709 group model building that I 1837 01:00:24,709 --> 01:00:26,585 think does a good job of. 1838 01:00:26,585 --> 01:00:29,479 It. You know, coming up 1839 01:00:29,479 --> 01:00:31,939 with ways to engage groups, 1840 01:00:31,939 --> 01:00:34,115 especially coalitions of groups, 1841 01:00:34,115 --> 01:00:38,000 in these modeling exercises. 1842 01:00:38,000 --> 01:00:39,829 And where a goal is 1843 01:00:39,829 --> 01:00:42,230 not just coming up with a good model, 1844 01:00:42,230 --> 01:00:46,009 but it's having this sort 1845 01:00:46,009 --> 01:00:47,629 of shared group process and 1846 01:00:47,629 --> 01:00:49,774 a learning process where in 1847 01:00:49,774 --> 01:00:52,250 engaging in this modelling and having 1848 01:00:52,250 --> 01:00:55,880 a a kind of artifact model 1849 01:00:55,880 --> 01:00:59,179 there to reference is a way to 1850 01:00:59,179 --> 01:01:00,679 build shared understanding within 1851 01:01:00,679 --> 01:01:02,779 a group and, and that kind of stuff. 1852 01:01:02,779 --> 01:01:05,104 And so I think there's, 1853 01:01:05,104 --> 01:01:07,100 you know, and but yeah, 1854 01:01:07,100 --> 01:01:09,950 there's definitely a high investment 1855 01:01:09,950 --> 01:01:12,559 of time and energy and all of that. 1856 01:01:12,559 --> 01:01:14,555 And I think that's 1857 01:01:14,555 --> 01:01:17,254 where I think some of this, 1858 01:01:17,254 --> 01:01:18,649 where some of the motivation for 1859 01:01:18,649 --> 01:01:20,615 my research comes in is 1860 01:01:20,615 --> 01:01:24,650 wanting to bring in people into that process. 1861 01:01:24,650 --> 01:01:26,810 But also recognize that, 1862 01:01:26,810 --> 01:01:28,144 you know, people don't, 1863 01:01:28,144 --> 01:01:29,540 can't necessarily show up for 1864 01:01:29,540 --> 01:01:31,730 a series of 12 modelling sessions, 1865 01:01:31,730 --> 01:01:33,409 synchronous, you know, 1866 01:01:33,409 --> 01:01:35,675 in person, all of that. 1867 01:01:35,675 --> 01:01:38,269 And so it's, you know, 1868 01:01:38,269 --> 01:01:41,989 I think there's different methods out there, 1869 01:01:41,989 --> 01:01:46,504 but I think there's I think there's also, 1870 01:01:46,504 --> 01:01:49,474 I think groom for development of, 1871 01:01:49,474 --> 01:01:52,039 um, you know, I see some 1872 01:01:52,039 --> 01:01:54,380 of this as maybe under the hood too. 1873 01:01:54,380 --> 01:01:58,850 Like where people can, you know, 1874 01:01:58,850 --> 01:02:00,469 not everybody in 1875 01:02:00,469 --> 01:02:02,329 an organization necessarily has to 1876 01:02:02,329 --> 01:02:05,900 be involved in the same way, 1877 01:02:05,900 --> 01:02:06,919 which I think is something 1878 01:02:06,919 --> 01:02:09,289 that traditionally a group model building. 1879 01:02:09,289 --> 01:02:09,950 It's like, okay, let's get 1880 01:02:09,950 --> 01:02:10,985 everyone in the room. 1881 01:02:10,985 --> 01:02:12,349 We have a white board handhold, 1882 01:02:12,349 --> 01:02:14,344 kind of do this together. 1883 01:02:14,344 --> 01:02:17,539 Whereas I think with, 1884 01:02:17,539 --> 01:02:19,235 with this you can 1885 01:02:19,235 --> 01:02:21,830 do some short interviews with people. 1886 01:02:21,830 --> 01:02:22,820 You can think of 1887 01:02:22,820 --> 01:02:24,290 other ways of engaging people. 1888 01:02:24,290 --> 01:02:26,315 Maybe some people are just involved. 1889 01:02:26,315 --> 01:02:28,850 The final review stage or maybe sunlight, 1890 01:02:28,850 --> 01:02:30,139 you know, so I think there's different 1891 01:02:30,139 --> 01:02:32,015 ways you can approach it. 1892 01:02:32,015 --> 01:02:34,160 As far as the broader question 1893 01:02:34,160 --> 01:02:36,080 of People don't want complexity. 1894 01:02:36,080 --> 01:02:38,570 I mean, I think there's, 1895 01:02:38,570 --> 01:02:41,209 you know, that's that's true. 1896 01:02:41,209 --> 01:02:44,585 But I feel HC there's also, you know, 1897 01:02:44,585 --> 01:02:45,800 people don't want complexity 1898 01:02:45,800 --> 01:02:46,760 until they realized that they 1899 01:02:46,760 --> 01:02:49,100 needed it from the start, you know, 1900 01:02:49,100 --> 01:02:51,979 and that hopefully there can 1901 01:02:51,979 --> 01:02:55,190 be a shift sort of more 1902 01:02:55,190 --> 01:02:58,249 broadly towards understanding the value 1903 01:02:58,249 --> 01:03:01,519 of really kind of understanding the system. 1904 01:03:01,519 --> 01:03:03,140 Doing a needs assessment of 1905 01:03:03,140 --> 01:03:05,344 where what you do, 1906 01:03:05,344 --> 01:03:06,814 what your problem is, 1907 01:03:06,814 --> 01:03:10,264 where your intervention is going to come in 1908 01:03:10,264 --> 01:03:12,620 and then trying to be 1909 01:03:12,620 --> 01:03:14,659 mindful of that relationship. 1910 01:03:14,659 --> 01:03:16,324 And that doesn't have to be 1911 01:03:16,324 --> 01:03:18,319 the sprawling complex model. 1912 01:03:18,319 --> 01:03:19,745 It can be pretty simple and 1913 01:03:19,745 --> 01:03:22,369 that sort of basic exercise of understanding. 1914 01:03:22,369 --> 01:03:24,320 Okay, what problem we're trying to solve. 1915 01:03:24,320 --> 01:03:25,864 What are the key dynamics there? 1916 01:03:25,864 --> 01:03:27,680 What buttons are we trying to push by 1917 01:03:27,680 --> 01:03:29,780 doing this? I think there's value. 1918 01:03:29,780 --> 01:03:32,600 A little bit of that can go a long way in 1919 01:03:32,600 --> 01:03:34,190 helping people understand 1920 01:03:34,190 --> 01:03:36,215 what goes wrong, leader. 1921 01:03:36,215 --> 01:03:37,699 And then one final thought on 1922 01:03:37,699 --> 01:03:39,229 that is I think there's a lot of 1923 01:03:39,229 --> 01:03:43,955 interest were some at least within funders, 1924 01:03:43,955 --> 01:03:45,260 you know, a lot of the when I 1925 01:03:45,260 --> 01:03:47,284 was doing this research, I found. 1926 01:03:47,284 --> 01:03:49,639 A lot of guides to this sort of 1927 01:03:49,639 --> 01:03:51,859 diagramming or guides to system dynamics. 1928 01:03:51,859 --> 01:03:53,030 Or there's a lot of pleasure like 1929 01:03:53,030 --> 01:03:56,450 USA ID or different folks have been, I think, 1930 01:03:56,450 --> 01:03:59,195 recognize the value in some of this 1931 01:03:59,195 --> 01:04:02,690 and try to put some of this out there as, 1932 01:04:02,690 --> 01:04:04,745 you know, here's something that could help. 1933 01:04:04,745 --> 01:04:06,229 I think there's a long way to 1934 01:04:06,229 --> 01:04:07,909 go and helping kind of integrate 1935 01:04:07,909 --> 01:04:09,049 the best practices from 1936 01:04:09,049 --> 01:04:11,330 system dynamics or the fields and kind of 1937 01:04:11,330 --> 01:04:15,035 really design strategies that are good. 1938 01:04:15,035 --> 01:04:16,309 You know, instead of just borrowed 1939 01:04:16,309 --> 01:04:18,980 from modeling to really have 1940 01:04:18,980 --> 01:04:20,810 things that are customized to 1941 01:04:20,810 --> 01:04:23,330 the needs of evaluators and evaluation. 1942 01:04:23,330 --> 01:04:25,489 So I see this as a starting point that, 1943 01:04:25,489 --> 01:04:27,019 you know, there's a lot of ways, 1944 01:04:27,019 --> 01:04:28,549 yeah, a long way to go to 1945 01:04:28,549 --> 01:04:31,279 make things actually usable. 1946 01:04:31,279 --> 01:04:36,320 And yeah, so thanks. That's a good answer. 1947 01:04:36,320 --> 01:04:39,259 I might just add that while 1948 01:04:39,259 --> 01:04:41,810 people want to be sure 1949 01:04:41,810 --> 01:04:43,865 that they are not overlooking things that 1950 01:04:43,865 --> 01:04:45,350 complexity perspective 1951 01:04:45,350 --> 01:04:47,360 would help, help them with. 1952 01:04:47,360 --> 01:04:50,119 They also would love to know 1953 01:04:50,119 --> 01:04:52,699 that you can use these methods to 1954 01:04:52,699 --> 01:04:55,370 make the whole process manageable to address 1955 01:04:55,370 --> 01:04:59,195 complexity without having to use tricky, 1956 01:04:59,195 --> 01:05:00,829 hard to understand things at 1957 01:05:00,829 --> 01:05:02,449 the basic language is pretty 1958 01:05:02,449 --> 01:05:03,890 straightforward and by using 1959 01:05:03,890 --> 01:05:05,824 it in a thoughtful way, 1960 01:05:05,824 --> 01:05:07,910 you are addressing the complexity, 1961 01:05:07,910 --> 01:05:09,770 but in a way that we can think 1962 01:05:09,770 --> 01:05:12,020 about in process and digest. 1963 01:05:12,020 --> 01:05:13,220 So and I think you 1964 01:05:13,220 --> 01:05:15,719 said that your presentation as well. 1965 01:05:16,420 --> 01:05:18,575 I'd like to open it up for 1966 01:05:18,575 --> 01:05:19,250 questions from 1967 01:05:19,250 --> 01:05:20,690 everyone and committee members. 1968 01:05:20,690 --> 01:05:21,949 Feel free to continue asking 1969 01:05:21,949 --> 01:05:23,959 questions as well. 1970 01:05:23,959 --> 01:05:26,300 And I'll just go ahead 1971 01:05:26,300 --> 01:05:28,800 and take the mic if you have a question. 1972 01:05:29,770 --> 01:05:32,119 Okay. Thank you. Minus took quarter. 1973 01:05:32,119 --> 01:05:33,889 Thank you. Brought her presentation. 1974 01:05:33,889 --> 01:05:36,890 There are really interesting 1975 01:05:36,890 --> 01:05:39,515 and a totally new area 1976 01:05:39,515 --> 01:05:41,959 of study ever seen before. 1977 01:05:41,959 --> 01:05:44,839 But my first question was, 1978 01:05:44,839 --> 01:05:47,209 have you applied this 1979 01:05:47,209 --> 01:05:53,670 to a needs discovery for marketing? 1980 01:05:54,640 --> 01:05:58,144 For marketing? I personally have. 1981 01:05:58,144 --> 01:06:01,205 I feel like there's, you know, 1982 01:06:01,205 --> 01:06:03,859 the field of system science 1983 01:06:03,859 --> 01:06:06,724 and causal loop diagramming and all of that. 1984 01:06:06,724 --> 01:06:09,350 In generally, people use things 1985 01:06:09,350 --> 01:06:12,380 for all different areas. 1986 01:06:12,380 --> 01:06:13,804 And so I'm sure someone has done it. 1987 01:06:13,804 --> 01:06:14,959 I know there's a lot of 1988 01:06:14,959 --> 01:06:16,399 people who apply these 1989 01:06:16,399 --> 01:06:18,319 sort of modeling with economics 1990 01:06:18,319 --> 01:06:19,490 and that kind of stuff. 1991 01:06:19,490 --> 01:06:21,349 But I personally haven't 1992 01:06:21,349 --> 01:06:24,090 gone in that direction. 1993 01:06:25,390 --> 01:06:28,099 Okay. Thank you. What might 1994 01:06:28,099 --> 01:06:30,359 be profitable now? 1995 01:06:30,880 --> 01:06:36,139 Can be high. 1996 01:06:36,139 --> 01:06:38,885 Erin, nice to meet you. 1997 01:06:38,885 --> 01:06:40,940 My name is Kathleen conti. 1998 01:06:40,940 --> 01:06:43,175 I do as soft systems research, 1999 01:06:43,175 --> 01:06:44,900 systems methodologies 2000 01:06:44,900 --> 01:06:46,730 and participatory methods 2001 01:06:46,730 --> 01:06:48,784 for public health. Go great. 2002 01:06:48,784 --> 01:06:51,170 And I loved your presentation. 2003 01:06:51,170 --> 01:06:53,690 I'm, it's a fabulous work 2004 01:06:53,690 --> 01:06:55,025 and I hope that 2005 01:06:55,025 --> 01:06:56,899 there's an opportunity to use it. 2006 01:06:56,899 --> 01:06:59,449 Be thinking about that as I move forward with 2007 01:06:59,449 --> 01:07:00,830 my research and planning 2008 01:07:00,830 --> 01:07:03,050 and hopefully we can connect on it. 2009 01:07:03,050 --> 01:07:06,920 My question for you stems from 2010 01:07:06,920 --> 01:07:10,834 my experience doing group model building and 2011 01:07:10,834 --> 01:07:12,215 working with mental models 2012 01:07:12,215 --> 01:07:14,465 and thinking about how one of 2013 01:07:14,465 --> 01:07:17,030 the really important elements of 2014 01:07:17,030 --> 01:07:20,480 that work is surfacing assumptions 2015 01:07:20,480 --> 01:07:24,050 and recognizing and resolving where there's 2016 01:07:24,050 --> 01:07:26,599 disagreement or conflict and 2017 01:07:26,599 --> 01:07:29,240 how people see a problem or an issue. 2018 01:07:29,240 --> 01:07:30,980 And I think this method 2019 01:07:30,980 --> 01:07:33,170 is really interesting in 2020 01:07:33,170 --> 01:07:34,850 that there are a lot of opportunities here 2021 01:07:34,850 --> 01:07:36,589 to really go in depth and away, 2022 01:07:36,589 --> 01:07:37,850 probably couldn't and 2023 01:07:37,850 --> 01:07:40,625 that group model building situation. 2024 01:07:40,625 --> 01:07:42,050 But I also see that 2025 01:07:42,050 --> 01:07:43,669 there's some opportunity there for 2026 01:07:43,669 --> 01:07:45,020 maybe identifying some of 2027 01:07:45,020 --> 01:07:46,849 those friction points. 2028 01:07:46,849 --> 01:07:47,929 And so I wondered if you 2029 01:07:47,929 --> 01:07:49,970 had been doing and he's thinking or I've 2030 01:07:49,970 --> 01:07:52,460 done any work around how to 2031 01:07:52,460 --> 01:07:56,015 pull these disparate models together, 2032 01:07:56,015 --> 01:07:57,560 maybe to identify some of 2033 01:07:57,560 --> 01:07:59,240 those points and how 2034 01:07:59,240 --> 01:08:01,085 you might use that information. 2035 01:08:01,085 --> 01:08:02,960 Yeah. I think it's a great question, 2036 01:08:02,960 --> 01:08:04,099 so yeah, I mean, 2037 01:08:04,099 --> 01:08:06,109 there's definitely trade-offs 2038 01:08:06,109 --> 01:08:08,119 between using this approach, 2039 01:08:08,119 --> 01:08:10,610 for example, and using group model building. 2040 01:08:10,610 --> 01:08:12,379 There's different things that 2041 01:08:12,379 --> 01:08:14,225 the two strategies are good at. 2042 01:08:14,225 --> 01:08:16,849 I mean, I think there's a lot of room 2043 01:08:16,849 --> 01:08:20,900 for combining them to using this, 2044 01:08:20,900 --> 01:08:22,760 this sort of incorporating 2045 01:08:22,760 --> 01:08:24,215 from individual interviews 2046 01:08:24,215 --> 01:08:26,090 either as a precursor 2047 01:08:26,090 --> 01:08:27,844 to group model building, 2048 01:08:27,844 --> 01:08:28,820 where as a way 2049 01:08:28,820 --> 01:08:30,560 to incorporate perspectives from 2050 01:08:30,560 --> 01:08:32,689 people for people who 2051 01:08:32,689 --> 01:08:34,174 can't be in the room 2052 01:08:34,174 --> 01:08:36,510 with group model building. 2053 01:08:36,760 --> 01:08:39,109 And, but yeah, I 2054 01:08:39,109 --> 01:08:40,685 think with group model building the string, 2055 01:08:40,685 --> 01:08:41,810 one of this main strengths is 2056 01:08:41,810 --> 01:08:43,549 that dynamic kind of, 2057 01:08:43,549 --> 01:08:46,370 you know, let's the, the, the, 2058 01:08:46,370 --> 01:08:48,049 the cycles of iteration or 2059 01:08:48,049 --> 01:08:49,849 back and forth is much tighter 2060 01:08:49,849 --> 01:08:51,109 when you have someone 2061 01:08:51,109 --> 01:08:52,340 in front of you and can have 2062 01:08:52,340 --> 01:08:53,690 a whole conversation right 2063 01:08:53,690 --> 01:08:56,359 there with within that group. 2064 01:08:56,359 --> 01:08:58,894 And so what I tried to do is 2065 01:08:58,894 --> 01:09:01,969 capture some of the strengths of 2066 01:09:01,969 --> 01:09:04,459 that and try to codify some of it into 2067 01:09:04,459 --> 01:09:05,570 some of that guidance 2068 01:09:05,570 --> 01:09:07,429 for conducting the interview. 2069 01:09:07,429 --> 01:09:09,739 So try to you I have 2070 01:09:09,739 --> 01:09:12,080 the interviewer B, you know, 2071 01:09:12,080 --> 01:09:12,859 because you don't have 2072 01:09:12,859 --> 01:09:14,359 that group providing that 2073 01:09:14,359 --> 01:09:16,100 those checks and balances 2074 01:09:16,100 --> 01:09:17,795 are trying to get at that, 2075 01:09:17,795 --> 01:09:21,050 that the main idea, the inner mark, 2076 01:09:21,050 --> 01:09:23,509 there's more onus on the interviewer to 2077 01:09:23,509 --> 01:09:27,185 provide that that function. 2078 01:09:27,185 --> 01:09:29,210 And so I think it's 2079 01:09:29,210 --> 01:09:30,980 that's where I think it's it's 2080 01:09:30,980 --> 01:09:34,940 definitely a it's a practice 2081 01:09:34,940 --> 01:09:36,050 that I think there's a lot of room 2082 01:09:36,050 --> 01:09:37,250 for development on, 2083 01:09:37,250 --> 01:09:40,969 you know, how do you actually get at 2084 01:09:40,969 --> 01:09:42,514 the bottom of the 2085 01:09:42,514 --> 01:09:44,989 person's mental model in an interview, 2086 01:09:44,989 --> 01:09:47,044 you have to, I think ice, 2087 01:09:47,044 --> 01:09:48,679 this research kind of starts 2088 01:09:48,679 --> 01:09:50,540 that conversation of Here's 2089 01:09:50,540 --> 01:09:51,619 some things to think about, 2090 01:09:51,619 --> 01:09:52,729 but I think there's room for a 2091 01:09:52,729 --> 01:09:54,725 lot more development and 2092 01:09:54,725 --> 01:09:56,420 in developing kind of a craft 2093 01:09:56,420 --> 01:09:58,190 around, around that. 2094 01:09:58,190 --> 01:10:00,680 And so, you know, 2095 01:10:00,680 --> 01:10:02,630 this, yeah, I think there's, 2096 01:10:02,630 --> 01:10:04,580 there's strengths and weaknesses. 2097 01:10:04,580 --> 01:10:07,310 I think it's one thing that I really tried. 2098 01:10:07,310 --> 01:10:10,459 I need that I think this fills that 2099 01:10:10,459 --> 01:10:11,600 I was felt like 2100 01:10:11,600 --> 01:10:12,710 the limitation with 2101 01:10:12,710 --> 01:10:14,539 group model building in some ways is, 2102 01:10:14,539 --> 01:10:17,074 I feel like for some people, 2103 01:10:17,074 --> 01:10:19,414 a one-on-one interview might be a bit 2104 01:10:19,414 --> 01:10:22,549 more comfortable if a situation 2105 01:10:22,549 --> 01:10:24,859 or someone might feel more 2106 01:10:24,859 --> 01:10:28,189 open to sharing what their perspective is. 2107 01:10:28,189 --> 01:10:30,109 Cuz I feel like with 2108 01:10:30,109 --> 01:10:31,789 the group modelling because. 2109 01:10:31,789 --> 01:10:33,050 You know, strength is 2110 01:10:33,050 --> 01:10:34,160 getting people to kind of 2111 01:10:34,160 --> 01:10:36,770 coalesce around a similar view. 2112 01:10:36,770 --> 01:10:37,730 But then the trade-off 2113 01:10:37,730 --> 01:10:38,689 with that is that you don't get 2114 01:10:38,689 --> 01:10:40,610 that diversity and perspectives, right? 2115 01:10:40,610 --> 01:10:42,349 And so this could maybe 2116 01:10:42,349 --> 01:10:44,149 be something where you do in 2117 01:10:44,149 --> 01:10:45,739 an individual interviews first with 2118 01:10:45,739 --> 01:10:47,539 people and you kind of get 2119 01:10:47,539 --> 01:10:49,520 segments of models maybe or maybe 2120 01:10:49,520 --> 01:10:50,929 you kind of do 2121 01:10:50,929 --> 01:10:52,639 the individual models and then see, 2122 01:10:52,639 --> 01:10:54,440 okay, what's kind of 2123 01:10:54,440 --> 01:10:56,539 diversity of perspectives that you have. 2124 01:10:56,539 --> 01:10:59,690 And then bring people together and kind of 2125 01:10:59,690 --> 01:11:01,279 have them engage in that we're at 2126 01:11:01,279 --> 01:11:02,959 where we're doing a study right now, 2127 01:11:02,959 --> 01:11:04,250 where we're doing some of that. 2128 01:11:04,250 --> 01:11:05,989 We did individual interviews with folks, 2129 01:11:05,989 --> 01:11:09,619 were doing some some mapping and 2130 01:11:09,619 --> 01:11:11,569 then bringing them together to try 2131 01:11:11,569 --> 01:11:13,955 to to reflect on that. 2132 01:11:13,955 --> 01:11:16,279 So yeah, I'd love to talk to 2133 01:11:16,279 --> 01:11:19,830 talk more. Thank you. 2134 01:11:22,660 --> 01:11:26,270 I Kinsey, My name is Nicola and that 2135 01:11:26,270 --> 01:11:27,320 internal evaluator for 2136 01:11:27,320 --> 01:11:29,509 the Native American Youth and Family Center. 2137 01:11:29,509 --> 01:11:31,205 I'm really excited to actually 2138 01:11:31,205 --> 01:11:32,899 use your work in my work. 2139 01:11:32,899 --> 01:11:34,759 So I look forward to reading 2140 01:11:34,759 --> 01:11:35,539 your dissertation with 2141 01:11:35,539 --> 01:11:37,250 its upon cardiac scholar. 2142 01:11:37,250 --> 01:11:38,929 I was just wondering, it's 2143 01:11:38,929 --> 01:11:40,249 okay if you don't have more details, 2144 01:11:40,249 --> 01:11:41,630 but if you just tell us a little bit 2145 01:11:41,630 --> 01:11:43,250 more about how you're able to so 2146 01:11:43,250 --> 01:11:44,809 wonderfully braid 2147 01:11:44,809 --> 01:11:46,549 constructivist and realist 2148 01:11:46,549 --> 01:11:48,754 interview methodologies 2149 01:11:48,754 --> 01:11:51,515 like background and such. 2150 01:11:51,515 --> 01:11:53,899 Yeah. I mean, I think I 2151 01:11:53,899 --> 01:11:57,289 just I think it came down 2152 01:11:57,289 --> 01:12:01,579 to was when I was trying to figure 2153 01:12:01,579 --> 01:12:06,499 out where what's sort 2154 01:12:06,499 --> 01:12:08,299 of like what would make when asking myself, 2155 01:12:08,299 --> 01:12:11,555 okay, what would make a good interview 2156 01:12:11,555 --> 01:12:13,489 designed for modelling, right? 2157 01:12:13,489 --> 01:12:16,880 I the the qualitative training and me, 2158 01:12:16,880 --> 01:12:19,819 he kind of originally went to OK, 2159 01:12:19,819 --> 01:12:20,599 we need to have 2160 01:12:20,599 --> 01:12:22,940 this more constructivist approach 2161 01:12:22,940 --> 01:12:24,230 that kind of gives 2162 01:12:24,230 --> 01:12:29,269 more respect or was 2163 01:12:29,269 --> 01:12:33,545 steering capacity to the participant. 2164 01:12:33,545 --> 01:12:34,699 And I think there's a lot 2165 01:12:34,699 --> 01:12:35,900 of validity in that, 2166 01:12:35,900 --> 01:12:38,149 but then I realized that that 2167 01:12:38,149 --> 01:12:40,864 can only really go so far. 2168 01:12:40,864 --> 01:12:42,259 And that when you 2169 01:12:42,259 --> 01:12:46,279 were modeling what I like about and there's, 2170 01:12:46,279 --> 01:12:49,129 there's preexisting research out there within 2171 01:12:49,129 --> 01:12:52,294 the system dynamics realm around, 2172 01:12:52,294 --> 01:12:53,420 you know, sort of what are 2173 01:12:53,420 --> 01:12:56,119 the philosophical assumptions there? 2174 01:12:56,119 --> 01:12:59,510 And I think a realist comes up, 2175 01:12:59,510 --> 01:13:01,790 realism comes up there because, 2176 01:13:01,790 --> 01:13:03,710 you know, it's, you don't want to, 2177 01:13:03,710 --> 01:13:04,940 when you're making a model, you don't 2178 01:13:04,940 --> 01:13:06,319 want to pretend that you as 2179 01:13:06,319 --> 01:13:10,100 a modeler don't have perspective, 2180 01:13:10,100 --> 01:13:12,184 are you, that you're not part of it? 2181 01:13:12,184 --> 01:13:14,059 I feel like there's a lot 2182 01:13:14,059 --> 01:13:16,624 of room for improvement within 2183 01:13:16,624 --> 01:13:18,995 the system science field for 2184 01:13:18,995 --> 01:13:21,440 better acknowledging what the role 2185 01:13:21,440 --> 01:13:22,580 of the modeler is. 2186 01:13:22,580 --> 01:13:24,139 And that there, 2187 01:13:24,139 --> 01:13:26,479 there's a tiny little assumptions 2188 01:13:26,479 --> 01:13:27,980 that you're making when you're building 2189 01:13:27,980 --> 01:13:30,740 a model and a ton of little decisions. 2190 01:13:30,740 --> 01:13:33,410 And I think a lot 2191 01:13:33,410 --> 01:13:34,670 of times that gets glossed 2192 01:13:34,670 --> 01:13:35,749 over in the process. 2193 01:13:35,749 --> 01:13:37,039 I think, especially 2194 01:13:37,039 --> 01:13:38,659 with simulation modelling, 2195 01:13:38,659 --> 01:13:39,830 because it looks so 2196 01:13:39,830 --> 01:13:41,989 quantitative and it has math behind it. 2197 01:13:41,989 --> 01:13:43,774 I think it's easy to say, 2198 01:13:43,774 --> 01:13:46,130 oh, that is the system. 2199 01:13:46,130 --> 01:13:48,050 But I think that really glosses over 2200 01:13:48,050 --> 01:13:50,465 how that's a creation. 2201 01:13:50,465 --> 01:13:52,444 That it's something that's 2202 01:13:52,444 --> 01:13:54,095 crafted by a modeler. 2203 01:13:54,095 --> 01:13:55,670 And so I think. 2204 01:13:55,670 --> 01:13:58,010 Where this research comes in, 2205 01:13:58,010 --> 01:13:59,569 is that what I wanted to do is 2206 01:13:59,569 --> 01:14:03,409 really be intentional and try to track, 2207 01:14:03,409 --> 01:14:06,260 Okay, what comes from the purchase paint, 2208 01:14:06,260 --> 01:14:07,670 what comes from a modeler? 2209 01:14:07,670 --> 01:14:09,995 How can we understand where 2210 01:14:09,995 --> 01:14:12,560 those assumptions the modeler comes in? 2211 01:14:12,560 --> 01:14:14,179 And so I felt like there was 2212 01:14:14,179 --> 01:14:16,205 a need for, you know, 2213 01:14:16,205 --> 01:14:19,085 having that constructivist orientation 2214 01:14:19,085 --> 01:14:20,390 to start kinda get 2215 01:14:20,390 --> 01:14:24,454 the purview and then to say, okay, 2216 01:14:24,454 --> 01:14:26,360 the modeler does have 2217 01:14:26,360 --> 01:14:29,390 that role and that you need to be able to 2218 01:14:29,390 --> 01:14:31,519 acknowledge that they're coming to it with 2219 01:14:31,519 --> 01:14:34,339 their own hypotheses and 2220 01:14:34,339 --> 01:14:35,210 you need to have 2221 01:14:35,210 --> 01:14:37,530 a different perspective there. 2222 01:14:41,220 --> 01:14:44,379 I have one more question. 2223 01:14:44,379 --> 01:14:46,509 The more you talk, 2224 01:14:46,509 --> 01:14:47,619 the clearer it becomes to 2225 01:14:47,619 --> 01:14:48,805 me that we are all kind of 2226 01:14:48,805 --> 01:14:52,030 hostages of our own artifacts of our model. 2227 01:14:52,030 --> 01:14:54,160 And, and I think to a large, 2228 01:14:54,160 --> 01:14:55,719 it's as a lot of the things you're doing, 2229 01:14:55,719 --> 01:14:56,830 I'm doing with a technique 2230 01:14:56,830 --> 01:14:58,210 called fuzzy cognitive maps. 2231 01:14:58,210 --> 01:15:00,910 And because fuzzy cognitive maps happened to 2232 01:15:00,910 --> 01:15:02,200 be something that you 2233 01:15:02,200 --> 01:15:03,835 combine very, very easily. 2234 01:15:03,835 --> 01:15:05,350 While of course we say, well, 2235 01:15:05,350 --> 01:15:07,929 let's do individual interviews or even less. 2236 01:15:07,929 --> 01:15:10,240 And then combine all this into a model. 2237 01:15:10,240 --> 01:15:11,275 And because they look a lot 2238 01:15:11,275 --> 01:15:12,309 like neural networks there, 2239 01:15:12,309 --> 01:15:14,170 let's try that with artificial intelligence. 2240 01:15:14,170 --> 01:15:15,610 So I think a lot of where 2241 01:15:15,610 --> 01:15:17,260 we're going with this simply 2242 01:15:17,260 --> 01:15:20,550 based on the modelling technique. 2243 01:15:20,550 --> 01:15:22,880 And I'm seeing the same in your work and, 2244 01:15:22,880 --> 01:15:25,820 and I've seen many feel now duplicating 2245 01:15:25,820 --> 01:15:27,229 a lot of the same thing 2246 01:15:27,229 --> 01:15:28,955 with the same intentions, 2247 01:15:28,955 --> 01:15:31,145 but they're stuck on their modeling method. 2248 01:15:31,145 --> 01:15:33,020 How do we get over this? 2249 01:15:33,020 --> 01:15:35,599 What, what would you suggest we do? 2250 01:15:35,599 --> 01:15:38,225 Well after Kobe, Michigan together and, 2251 01:15:38,225 --> 01:15:41,824 you know, I mean, 2252 01:15:41,824 --> 01:15:43,279 I think it's a really good question 2253 01:15:43,279 --> 01:15:44,540 and I feel like that was something 2254 01:15:44,540 --> 01:15:47,000 that one 2255 01:15:47,000 --> 01:15:48,800 of the hardest things about this research 2256 01:15:48,800 --> 01:15:51,334 was feeling parallelize that I couldn't 2257 01:15:51,334 --> 01:15:54,454 explore all of those possible roads. 2258 01:15:54,454 --> 01:15:56,090 I would, yeah, I really wanted to 2259 01:15:56,090 --> 01:15:57,964 do a deep dive into 2260 01:15:57,964 --> 01:16:00,319 fuzzy cognitive mapping and these other 2261 01:16:00,319 --> 01:16:02,569 sorts of mapping methods. 2262 01:16:02,569 --> 01:16:07,234 And so it was, you're right, 2263 01:16:07,234 --> 01:16:09,170 like there is a lot out there and I think 2264 01:16:09,170 --> 01:16:12,274 there needs to be 2265 01:16:12,274 --> 01:16:16,954 much more dialogue in 2266 01:16:16,954 --> 01:16:18,140 real life and in the 2267 01:16:18,140 --> 01:16:19,400 literature around like how 2268 01:16:19,400 --> 01:16:22,354 do we learn from each other? 2269 01:16:22,354 --> 01:16:26,150 And how do we yeah. 2270 01:16:26,150 --> 01:16:30,440 Combine or or blend methods when needed. 2271 01:16:30,440 --> 01:16:33,170 So yeah, I totally agree. 2272 01:16:33,170 --> 01:16:35,569 Someday we'll have the Vulcan mind 2273 01:16:35,569 --> 01:16:37,589 meld to accomplish that. 2274 01:16:38,080 --> 01:16:41,825 I just wanted to ask you for the record. 2275 01:16:41,825 --> 01:16:43,459 Because I think that might be 2276 01:16:43,459 --> 01:16:44,960 students watching this in 2277 01:16:44,960 --> 01:16:47,509 future years, maybe in months. 2278 01:16:47,509 --> 01:16:50,000 Even. What advice would you 2279 01:16:50,000 --> 01:16:52,819 have are two classes of people. 2280 01:16:52,819 --> 01:16:56,539 People in a room of which could be full-on. 2281 01:16:56,539 --> 01:16:57,620 Experienced practitioners 2282 01:16:57,620 --> 01:16:58,760 will want to use this method. 2283 01:16:58,760 --> 01:17:00,110 What sort of advice would you 2284 01:17:00,110 --> 01:17:01,879 have for those people? 2285 01:17:01,879 --> 01:17:03,080 And in particular, what advice 2286 01:17:03,080 --> 01:17:04,805 would you have for students 2287 01:17:04,805 --> 01:17:06,800 about how you found 2288 01:17:06,800 --> 01:17:10,049 your way to a very successful dissertation. 2289 01:17:12,630 --> 01:17:15,099 I mean, I think for this method, 2290 01:17:15,099 --> 01:17:16,119 for practitioners, 2291 01:17:16,119 --> 01:17:18,580 I may think This isn't an original thought, 2292 01:17:18,580 --> 01:17:19,869 but I think it really just comes down 2293 01:17:19,869 --> 01:17:21,699 to what are your research questions? 2294 01:17:21,699 --> 01:17:23,560 What are you trying to get? 2295 01:17:23,560 --> 01:17:26,335 What do you really, 2296 01:17:26,335 --> 01:17:29,740 I think having a dialogue and 2297 01:17:29,740 --> 01:17:33,985 a critical understanding of what yeah, 2298 01:17:33,985 --> 01:17:35,364 what are your questions? 2299 01:17:35,364 --> 01:17:36,579 What are the strategies 2300 01:17:36,579 --> 01:17:39,459 needed to answer those questions? 2301 01:17:39,459 --> 01:17:41,800 What is, what is sort of 2302 01:17:41,800 --> 01:17:43,989 the least strategy needed 2303 01:17:43,989 --> 01:17:45,685 to answer those questions to? 2304 01:17:45,685 --> 01:17:49,840 I think this research fills a certain niche, 2305 01:17:49,840 --> 01:17:51,219 but it's, you know, 2306 01:17:51,219 --> 01:17:54,049 it's not, it, It's a specific niche. 2307 01:17:54,049 --> 01:17:55,970 You know, I think there's a lot 2308 01:17:55,970 --> 01:17:57,890 of situations where standard 2309 01:17:57,890 --> 01:18:00,259 qualitative interviews or be just great or 2310 01:18:00,259 --> 01:18:01,909 modelling that's more modeler 2311 01:18:01,909 --> 01:18:03,304 lead, but I think there's, 2312 01:18:03,304 --> 01:18:06,079 there needs to be more kind 2313 01:18:06,079 --> 01:18:08,389 of operationalization or guidelines 2314 01:18:08,389 --> 01:18:11,779 developed for how to 2315 01:18:11,779 --> 01:18:14,179 use these different tools in the toolbox. 2316 01:18:14,179 --> 01:18:16,099 So I think this is a starting point, 2317 01:18:16,099 --> 01:18:18,740 but I think and I think that's, 2318 01:18:18,740 --> 01:18:22,489 it's the onus of developing that is 2319 01:18:22,489 --> 01:18:24,260 on people like us 2320 01:18:24,260 --> 01:18:27,260 who developed methods are proposed methods. 2321 01:18:27,260 --> 01:18:29,539 I think we need to. 2322 01:18:29,539 --> 01:18:32,059 I really tried to think about the, 2323 01:18:32,059 --> 01:18:33,394 the end user and, 2324 01:18:33,394 --> 01:18:36,209 and try to develop some of that. 2325 01:18:36,280 --> 01:18:39,649 I mean, as far as advice 2326 01:18:39,649 --> 01:18:43,795 for dissertation enemy at 2327 01:18:43,795 --> 01:18:45,680 first try not to do it during a pandemic. 2328 01:18:45,680 --> 01:18:49,459 I think that and if you 2329 01:18:49,459 --> 01:18:54,785 do yeah. Yeah. I don't know. 2330 01:18:54,785 --> 01:18:58,295 I think it was yeah. 2331 01:18:58,295 --> 01:19:02,600 I think I'm trying to set boundaries 2332 01:19:02,600 --> 01:19:04,369 for yourself and trying to 2333 01:19:04,369 --> 01:19:05,539 realize that you can't go 2334 01:19:05,539 --> 01:19:06,980 down on all of those roads. 2335 01:19:06,980 --> 01:19:09,305 It was important for me. 2336 01:19:09,305 --> 01:19:12,104 And try understanding that 2337 01:19:12,104 --> 01:19:13,749 doing research is 2338 01:19:13,749 --> 01:19:15,249 a learning process and you don't 2339 01:19:15,249 --> 01:19:18,520 need to necessarily stick 2340 01:19:18,520 --> 01:19:20,500 to where your original starting point wasn't 2341 01:19:20,500 --> 01:19:22,690 because you change and 2342 01:19:22,690 --> 01:19:24,160 you are understanding changes and 2343 01:19:24,160 --> 01:19:27,009 so your your research 2344 01:19:27,009 --> 01:19:28,060 should change with that. 2345 01:19:28,060 --> 01:19:29,140 And as long as you're kind of 2346 01:19:29,140 --> 01:19:32,410 being engaging or committee and engaging, 2347 01:19:32,410 --> 01:19:34,659 kind of thinking critically about that, 2348 01:19:34,659 --> 01:19:36,175 that you know, to, 2349 01:19:36,175 --> 01:19:38,020 to try to use this as a period of 2350 01:19:38,020 --> 01:19:40,704 evolution in your own understanding. 2351 01:19:40,704 --> 01:19:42,910 Which I, that's what I 2352 01:19:42,910 --> 01:19:43,570 thought was one of 2353 01:19:43,570 --> 01:19:45,640 the biggest benefits for me. 2354 01:19:45,640 --> 01:19:49,819 Great. Thanks. Any other questions? 2355 01:19:54,930 --> 01:19:58,149 Go ahead. I don't have a question, 2356 01:19:58,149 --> 01:19:59,710 but I think maybe adding 2357 01:19:59,710 --> 01:20:02,364 some compliments on would be valuable. 2358 01:20:02,364 --> 01:20:03,879 And I think, and in 2359 01:20:03,879 --> 01:20:05,470 relation to your question Wayne 2360 01:20:05,470 --> 01:20:07,329 about advice for students 2361 01:20:07,329 --> 01:20:08,829 what maybe era and isn't 2362 01:20:08,829 --> 01:20:10,480 articulating as she had 2363 01:20:10,480 --> 01:20:13,869 a very different plan about nine months ago. 2364 01:20:13,869 --> 01:20:15,669 And so I think 2365 01:20:15,669 --> 01:20:17,230 sometimes as we enter this work, 2366 01:20:17,230 --> 01:20:20,949 we might have a plan 2367 01:20:20,949 --> 01:20:22,899 and not be open to the opportunities. 2368 01:20:22,899 --> 01:20:25,240 And Aaron just did an amazing job of really 2369 01:20:25,240 --> 01:20:27,880 responding to the dynamics of 2370 01:20:27,880 --> 01:20:29,724 what was happening during 2371 01:20:29,724 --> 01:20:32,289 coded and then finding a way to 2372 01:20:32,289 --> 01:20:34,390 move forward that still allowed her to 2373 01:20:34,390 --> 01:20:37,484 advance her methods and advance your skills. 2374 01:20:37,484 --> 01:20:39,574 And not getting stuck. 2375 01:20:39,574 --> 01:20:41,330 I think the other thing I 2376 01:20:41,330 --> 01:20:43,729 might comment on and this relates to Billy, 2377 01:20:43,729 --> 01:20:45,769 your question about how do we maybe 2378 01:20:45,769 --> 01:20:47,060 Bridge and how do we bring 2379 01:20:47,060 --> 01:20:48,665 this to communities is, 2380 01:20:48,665 --> 01:20:50,105 is I think we also need 2381 01:20:50,105 --> 01:20:51,289 to think about training. 2382 01:20:51,289 --> 01:20:53,660 So Erin brings like 2383 01:20:53,660 --> 01:20:55,489 the beauty that sometimes you 2384 01:20:55,489 --> 01:20:57,559 also find in biostatisticians, 2385 01:20:57,559 --> 01:20:59,030 which is the ability to do 2386 01:20:59,030 --> 01:21:01,790 complex work and rigorous work. 2387 01:21:01,790 --> 01:21:03,799 But when you're communicating with partners, 2388 01:21:03,799 --> 01:21:06,890 simplify it in a way or focus attention 2389 01:21:06,890 --> 01:21:08,330 in a way that the end-user 2390 01:21:08,330 --> 01:21:10,445 can actually absorb, 2391 01:21:10,445 --> 01:21:12,559 unlike act on the information. 2392 01:21:12,559 --> 01:21:14,390 So I guess I would just say, 2393 01:21:14,390 --> 01:21:15,679 I think that's another 2394 01:21:15,679 --> 01:21:17,059 really important aspect 2395 01:21:17,059 --> 01:21:18,290 of this work is errands, 2396 01:21:18,290 --> 01:21:20,509 ability to distill complexity 2397 01:21:20,509 --> 01:21:21,979 so that we can all 2398 01:21:21,979 --> 01:21:26,269 enter in and understand the value of it. 2399 01:21:26,269 --> 01:21:28,219 So it's been a pleasure 2400 01:21:28,219 --> 01:21:29,630 and I would just commend you for 2401 01:21:29,630 --> 01:21:32,929 kinda those softer skills that may be 2402 01:21:32,929 --> 01:21:34,595 also are part of what 2403 01:21:34,595 --> 01:21:37,415 has allowed you to be so successful. 2404 01:21:37,415 --> 01:21:41,960 Thinking. You didn't really say 2405 01:21:41,960 --> 01:21:43,400 specifically to much about 2406 01:21:43,400 --> 01:21:44,840 your publishing plan to 2407 01:21:44,840 --> 01:21:46,550 get you at the very highest level. 2408 01:21:46,550 --> 01:21:47,839 Just talked about where this 2409 01:21:47,839 --> 01:21:49,280 might get published and 2410 01:21:49,280 --> 01:21:50,720 kind of the timeframe that you sort of 2411 01:21:50,720 --> 01:21:52,269 have mindsets putting on, 2412 01:21:52,269 --> 01:21:53,950 they'll spotted all just just talk 2413 01:21:53,950 --> 01:21:54,549 about a little bit 2414 01:21:54,549 --> 01:21:56,049 from a publishing perspective. 2415 01:21:56,049 --> 01:21:57,429 Sure, yeah, I mean, 2416 01:21:57,429 --> 01:21:58,780 the first paper with 2417 01:21:58,780 --> 01:22:00,190 the systematic review I hope 2418 01:22:00,190 --> 01:22:02,080 to publish in a journal 2419 01:22:02,080 --> 01:22:04,705 for an evaluation audience. 2420 01:22:04,705 --> 01:22:06,939 The second, third papers, 2421 01:22:06,939 --> 01:22:08,290 while the second paper, 2422 01:22:08,290 --> 01:22:13,015 the the causal structure mapping method, 2423 01:22:13,015 --> 01:22:17,530 where I'm the the interviews 2424 01:22:17,530 --> 01:22:21,130 with the the practice facilitators. 2425 01:22:21,130 --> 01:22:23,964 There's this initial paper 2426 01:22:23,964 --> 01:22:27,219 which I think I think I'm planning to copy. 2427 01:22:27,219 --> 01:22:29,290 I think both those papers are 23 2428 01:22:29,290 --> 01:22:31,570 are more for a methods audience are sort 2429 01:22:31,570 --> 01:22:34,035 of more of a systems methods audience 2430 01:22:34,035 --> 01:22:35,660 kind of within the tradition 2431 01:22:35,660 --> 01:22:37,639 of that Kim Anderson paper and 2432 01:22:37,639 --> 01:22:38,690 the other papers that 2433 01:22:38,690 --> 01:22:39,770 have followed from that. 2434 01:22:39,770 --> 01:22:41,270 So I feel like that kind of fits within 2435 01:22:41,270 --> 01:22:43,430 that tradition. 2436 01:22:43,430 --> 01:22:45,109 But then there's, Oh, 2437 01:22:45,109 --> 01:22:47,644 I'm also planning to, at the end of that, 2438 01:22:47,644 --> 01:22:49,160 the orphan study with 2439 01:22:49,160 --> 01:22:50,989 the practice facilitators will 2440 01:22:50,989 --> 01:22:54,499 do a longitudinal analysis 2441 01:22:54,499 --> 01:22:56,600 of the different map, 2442 01:22:56,600 --> 01:22:57,935 the maps from every, 2443 01:22:57,935 --> 01:22:59,450 the interviews from every six months. 2444 01:22:59,450 --> 01:23:03,590 And so that will have I figure we'll have a, 2445 01:23:03,590 --> 01:23:06,109 well at least one publication that's more for 2446 01:23:06,109 --> 01:23:08,689 a practice facilitation audience and, 2447 01:23:08,689 --> 01:23:10,730 you know, really more digs into, okay, 2448 01:23:10,730 --> 01:23:11,840 what were the findings 2449 01:23:11,840 --> 01:23:13,850 of or the content of some of 2450 01:23:13,850 --> 01:23:15,470 those maps and what can 2451 01:23:15,470 --> 01:23:17,599 that field learn from some of that? 2452 01:23:17,599 --> 01:23:18,860 So I think it's, 2453 01:23:18,860 --> 01:23:21,799 I definitely, when developing this work, 2454 01:23:21,799 --> 01:23:24,410 I kept kind of switching between 2455 01:23:24,410 --> 01:23:25,609 these multiple hats and 2456 01:23:25,609 --> 01:23:27,469 the different potential audiences. 2457 01:23:27,469 --> 01:23:29,119 I think it's something and system science 2458 01:23:29,119 --> 01:23:31,729 we always kinda have to ask is okay, 2459 01:23:31,729 --> 01:23:33,350 is this first systems audiences is for 2460 01:23:33,350 --> 01:23:35,900 the sort of subject matter audience. 2461 01:23:35,900 --> 01:23:37,909 And yeah, so I, 2462 01:23:37,909 --> 01:23:40,940 I'm looking forward to kind of trying to 2463 01:23:40,940 --> 01:23:45,300 sort that out a little bit more TO thank you. 2464 01:23:46,750 --> 01:23:49,669 Well, if there aren't additional questions, 2465 01:23:49,669 --> 01:23:51,979 I'm gonna ask the committee to move 2466 01:23:51,979 --> 01:23:55,340 into a breakout room. 2467 01:23:55,340 --> 01:23:57,509 I haven't set up. I'm just going.