1
00:00:00,000 --> 00:00:04,170
No. Well, I
guess like whoa,

2
00:00:04,170 --> 00:00:05,250
it says it's like it was in

3
00:00:05,250 --> 00:00:09,390
progress. Anyway.
It's recording.

4
00:00:09,390 --> 00:00:13,980
Okay. Great. Alright.
Thanks, Wayne.

5
00:00:13,980 --> 00:00:16,904
I'd like to thank
everyone for joining me.

6
00:00:16,904 --> 00:00:21,015
I really appreciate
you showing up today.

7
00:00:21,015 --> 00:00:24,809
I would also like

8
00:00:24,809 --> 00:00:26,910
to thank the members
of my committee for

9
00:00:26,910 --> 00:00:29,700
providing support to me at

10
00:00:29,700 --> 00:00:33,524
a time when nobody has
any spare bandwidth.

11
00:00:33,524 --> 00:00:35,040
I'd also like to thank

12
00:00:35,040 --> 00:00:36,749
the PO organization and

13
00:00:36,749 --> 00:00:38,309
especially the
Bainbridge chapter

14
00:00:38,309 --> 00:00:39,209
for awarding me with

15
00:00:39,209 --> 00:00:40,409
a dissertation grant that

16
00:00:40,409 --> 00:00:42,409
allowed me to
complete this work.

17
00:00:42,409 --> 00:00:45,235
And I think we
have some people

18
00:00:45,235 --> 00:00:46,779
here who are new to
systems science.

19
00:00:46,779 --> 00:00:47,859
So I'm going to try to keep

20
00:00:47,859 --> 00:00:50,604
this accessible
to newcomers.

21
00:00:50,604 --> 00:00:52,690
And just to confirm
you're seeing

22
00:00:52,690 --> 00:00:54,949
my slides, right?

23
00:00:55,500 --> 00:01:00,100
Yes. Right. Okay. So here's

24
00:01:00,100 --> 00:01:02,304
an overview of what
I'll be covering.

25
00:01:02,304 --> 00:01:04,465
First, I'm going to
give some background

26
00:01:04,465 --> 00:01:06,219
about causal loop diagrams,

27
00:01:06,219 --> 00:01:08,259
what they are,
how they're used.

28
00:01:08,259 --> 00:01:10,884
I'll go over my research
questions briefly,

29
00:01:10,884 --> 00:01:11,889
and then we'll dig into

30
00:01:11,889 --> 00:01:13,299
the methods and
results from

31
00:01:13,299 --> 00:01:15,579
my free papers and

32
00:01:15,579 --> 00:01:17,529
then bring it together
with some things,

33
00:01:17,529 --> 00:01:21,070
synthesis and discuss
the main takeaways.

34
00:01:21,070 --> 00:01:22,719
And then we'll
have some Q and a

35
00:01:22,719 --> 00:01:24,644
like Wayne mentioned.

36
00:01:24,644 --> 00:01:27,230
Now. To start,

37
00:01:27,230 --> 00:01:28,520
causal loop diagrams are

38
00:01:28,520 --> 00:01:30,320
a method for systems
science that

39
00:01:30,320 --> 00:01:31,790
describe in a visual

40
00:01:31,790 --> 00:01:33,560
way how different things in

41
00:01:33,560 --> 00:01:34,940
this system are
connected to one

42
00:01:34,940 --> 00:01:37,445
another through
causal relationships.

43
00:01:37,445 --> 00:01:39,619
And because my
research takes

44
00:01:39,619 --> 00:01:42,109
a deep dive into causal
loop diagramming,

45
00:01:42,109 --> 00:01:43,640
I wanted to spend
a few minutes

46
00:01:43,640 --> 00:01:45,710
going over the mechanics
of how they work.

47
00:01:45,710 --> 00:01:48,815
And so this is one
classic example.

48
00:01:48,815 --> 00:01:51,140
So let's say you
have a population of

49
00:01:51,140 --> 00:01:53,180
chickens, all
things being equal.

50
00:01:53,180 --> 00:01:54,634
If you have more chickens,

51
00:01:54,634 --> 00:01:56,165
you have more eggs.

52
00:01:56,165 --> 00:01:57,529
An era with the plus sign

53
00:01:57,529 --> 00:01:58,789
means that the more
chickens you have,

54
00:01:58,789 --> 00:02:00,320
more eggs you
have as a result,

55
00:02:00,320 --> 00:02:01,939
while if you
have more eggs,

56
00:02:01,939 --> 00:02:02,810
that means you end up

57
00:02:02,810 --> 00:02:04,414
with even more chickens.

58
00:02:04,414 --> 00:02:06,349
And this forms a
feedback loop.

59
00:02:06,349 --> 00:02:07,430
And this is one is what we

60
00:02:07,430 --> 00:02:09,110
call a reinforcing loop

61
00:02:09,110 --> 00:02:10,549
because it describes

62
00:02:10,549 --> 00:02:12,724
an exponential
growth or decline.

63
00:02:12,724 --> 00:02:15,260
So by itself, this
diagram tells a story of

64
00:02:15,260 --> 00:02:16,280
a chicken population that

65
00:02:16,280 --> 00:02:18,769
grows exponentially
without limits.

66
00:02:18,769 --> 00:02:20,330
Okay? So what if instead

67
00:02:20,330 --> 00:02:22,550
of living in a
limitless void,

68
00:02:22,550 --> 00:02:24,619
your chickens live
by a busy road.

69
00:02:24,619 --> 00:02:26,660
While we all know
what happens when you

70
00:02:26,660 --> 00:02:29,255
have more chickens
and more roads,

71
00:02:29,255 --> 00:02:32,285
you get more chickens
making bad decisions.

72
00:02:32,285 --> 00:02:34,580
Which meets them
are road crossings.

73
00:02:34,580 --> 00:02:35,929
And if you have more
road crossings,

74
00:02:35,929 --> 00:02:38,689
what do you get?
Fewer chickens?

75
00:02:38,689 --> 00:02:40,070
And so the red arrow with

76
00:02:40,070 --> 00:02:41,510
the negative sign
by it means that

77
00:02:41,510 --> 00:02:42,770
the causal
relationship works

78
00:02:42,770 --> 00:02:44,299
in the opposite direction.

79
00:02:44,299 --> 00:02:46,699
Road crossings,
fewer chickens.

80
00:02:46,699 --> 00:02:48,080
And this forms
what we call a

81
00:02:48,080 --> 00:02:49,894
balancing loop
because by itself,

82
00:02:49,894 --> 00:02:52,129
it describes a system
that over time

83
00:02:52,129 --> 00:02:54,710
comes into balance
or homeostasis.

84
00:02:54,710 --> 00:02:56,420
And zooming out, we can see

85
00:02:56,420 --> 00:02:58,069
that the diagram
is a whole,

86
00:02:58,069 --> 00:02:59,900
tells a story about
what we call the

87
00:02:59,900 --> 00:03:02,345
causal structure
of this system.

88
00:03:02,345 --> 00:03:04,009
If you have
chickens and a road

89
00:03:04,009 --> 00:03:05,284
and confined space,

90
00:03:05,284 --> 00:03:06,230
these feedback loops

91
00:03:06,230 --> 00:03:08,194
describe what can happen.

92
00:03:08,194 --> 00:03:10,429
And so causal loop
diagram notation

93
00:03:10,429 --> 00:03:12,514
with the arrows that
go in one direction

94
00:03:12,514 --> 00:03:13,880
that have a positive
or negative

95
00:03:13,880 --> 00:03:15,410
valence allow quite a bit

96
00:03:15,410 --> 00:03:16,999
of information
to be summarized

97
00:03:16,999 --> 00:03:19,009
in a compact form.

98
00:03:19,009 --> 00:03:21,110
And causal loop
diagrams contain

99
00:03:21,110 --> 00:03:23,509
several types of
causal structures.

100
00:03:23,509 --> 00:03:25,400
Variables are anything that

101
00:03:25,400 --> 00:03:27,094
exists in the system.

102
00:03:27,094 --> 00:03:30,169
Anything I guess that
exists in the system.

103
00:03:30,169 --> 00:03:32,629
Causal links or
individual relationships

104
00:03:32,629 --> 00:03:34,025
between variables.

105
00:03:34,025 --> 00:03:36,619
Feedback loops are
reinforcing or balancing

106
00:03:36,619 --> 00:03:39,784
structures that explain
non-linear behavior.

107
00:03:39,784 --> 00:03:42,950
And archetypes are
certain configurations of

108
00:03:42,950 --> 00:03:44,479
feedback loops and links

109
00:03:44,479 --> 00:03:46,310
that reflect
common situations.

110
00:03:46,310 --> 00:03:47,645
And so Tragedy of

111
00:03:47,645 --> 00:03:50,225
Commons is one that you
might have heard of.

112
00:03:50,225 --> 00:03:51,919
And there's a
hierarchy here

113
00:03:51,919 --> 00:03:53,329
where the structures
containing

114
00:03:53,329 --> 00:03:55,250
the most causal
information are

115
00:03:55,250 --> 00:03:58,410
less commonly found
and vice-versa.

116
00:03:58,420 --> 00:04:01,220
So causal loop
diagramming started in

117
00:04:01,220 --> 00:04:03,679
the System Dynamics
field is a way to

118
00:04:03,679 --> 00:04:05,600
sketch out the
main structure of

119
00:04:05,600 --> 00:04:08,284
a system when building
a computational model.

120
00:04:08,284 --> 00:04:10,760
And these diagrams
have also been used as

121
00:04:10,760 --> 00:04:12,830
standalone tools
to help people

122
00:04:12,830 --> 00:04:14,119
get a better understanding

123
00:04:14,119 --> 00:04:15,649
of complex systems.

124
00:04:15,649 --> 00:04:17,239
And they allowed you to get

125
00:04:17,239 --> 00:04:19,010
a bird's eye view of

126
00:04:19,010 --> 00:04:20,089
the key relationships and

127
00:04:20,089 --> 00:04:21,499
dynamics in the system,

128
00:04:21,499 --> 00:04:22,669
what the drivers are,

129
00:04:22,669 --> 00:04:24,454
where the points of
leverage may be.

130
00:04:24,454 --> 00:04:26,090
And in my research,
I focus on

131
00:04:26,090 --> 00:04:28,294
two different uses
of these diagrams.

132
00:04:28,294 --> 00:04:29,524
The first is when

133
00:04:29,524 --> 00:04:31,610
public sector
organizations use them to

134
00:04:31,610 --> 00:04:34,025
describe their
social programs

135
00:04:34,025 --> 00:04:34,849
are supposed to make

136
00:04:34,849 --> 00:04:36,695
a difference in
people's lives.

137
00:04:36,695 --> 00:04:38,959
In the field of program
evaluation calls

138
00:04:38,959 --> 00:04:42,275
this sort of explanation
program theory.

139
00:04:42,275 --> 00:04:44,435
And this is an example of

140
00:04:44,435 --> 00:04:45,679
a causal loop diagram used

141
00:04:45,679 --> 00:04:47,524
to describe program theory.

142
00:04:47,524 --> 00:04:49,520
You can see
interventions acting on

143
00:04:49,520 --> 00:04:51,845
variables that work
together in the system.

144
00:04:51,845 --> 00:04:53,659
And more and more,
people are using

145
00:04:53,659 --> 00:04:55,699
causal loop diagrams
for program theory as

146
00:04:55,699 --> 00:04:57,529
a way to incorporate
complexity

147
00:04:57,529 --> 00:04:59,885
into their understanding
of social programs.

148
00:04:59,885 --> 00:05:02,089
And they can use them
to better understand

149
00:05:02,089 --> 00:05:03,710
how their intervention is

150
00:05:03,710 --> 00:05:05,854
supposed to act on
parts of the system.

151
00:05:05,854 --> 00:05:07,700
It helps bring together

152
00:05:07,700 --> 00:05:09,319
an understanding of
how contacts and

153
00:05:09,319 --> 00:05:11,060
the intervention
worked together

154
00:05:11,060 --> 00:05:13,294
to produce whatever
outcomes you see.

155
00:05:13,294 --> 00:05:15,530
And so prior to
this, nobody had

156
00:05:15,530 --> 00:05:17,450
brought these
examples together and

157
00:05:17,450 --> 00:05:19,639
studied how they were using

158
00:05:19,639 --> 00:05:20,899
causal loop diagramming and

159
00:05:20,899 --> 00:05:22,489
how they might be
able to do it better.

160
00:05:22,489 --> 00:05:24,470
So in the first paper,
my dissertation,

161
00:05:24,470 --> 00:05:25,940
I do a systematic review of

162
00:05:25,940 --> 00:05:28,624
these studies to
do just that.

163
00:05:28,624 --> 00:05:31,054
I also examine how

164
00:05:31,054 --> 00:05:32,659
causal loop diagrams
can be used to

165
00:05:32,659 --> 00:05:34,759
describe individuals
mental models.

166
00:05:34,759 --> 00:05:37,369
And a mental model
is a person set of

167
00:05:37,369 --> 00:05:39,019
assumptions and
beliefs about

168
00:05:39,019 --> 00:05:40,940
how the world works.

169
00:05:40,940 --> 00:05:43,099
There are two key
questions when developing

170
00:05:43,099 --> 00:05:44,329
causal loop diagrams or

171
00:05:44,329 --> 00:05:45,740
any other systems model,

172
00:05:45,740 --> 00:05:47,899
whose perspective
or mental model

173
00:05:47,899 --> 00:05:49,325
are you going to represent?

174
00:05:49,325 --> 00:05:50,809
And how we will get

175
00:05:50,809 --> 00:05:53,000
that perspective from
someone's head and into

176
00:05:53,000 --> 00:05:56,090
the model in the system
science field has

177
00:05:56,090 --> 00:05:57,799
had a lot of
scholarship around how

178
00:05:57,799 --> 00:05:59,599
to engage groups
in model building,

179
00:05:59,599 --> 00:06:00,835
but those methods aren't

180
00:06:00,835 --> 00:06:02,720
o is feasible to use.

181
00:06:02,720 --> 00:06:04,534
And they're, oh,
Israel, there are only

182
00:06:04,534 --> 00:06:06,769
really designed for
working with groups.

183
00:06:06,769 --> 00:06:08,299
And so there's a need for

184
00:06:08,299 --> 00:06:09,620
methods that can use

185
00:06:09,620 --> 00:06:11,209
one-on-one interviewing to

186
00:06:11,209 --> 00:06:13,744
capture individuals
mental models.

187
00:06:13,744 --> 00:06:15,830
When the interviews
are a great way to get

188
00:06:15,830 --> 00:06:18,080
information from
people about

189
00:06:18,080 --> 00:06:19,819
their experiences or
their perspective

190
00:06:19,819 --> 00:06:21,560
of a topic because they're

191
00:06:21,560 --> 00:06:22,789
a pretty comfortable format

192
00:06:22,789 --> 00:06:23,690
for people and they

193
00:06:23,690 --> 00:06:25,429
allow people to explain

194
00:06:25,429 --> 00:06:27,740
what they think in
their own words.

195
00:06:27,740 --> 00:06:29,359
And so it's more accessible

196
00:06:29,359 --> 00:06:31,279
than asking them to engage

197
00:06:31,279 --> 00:06:32,270
directly and modeling when

198
00:06:32,270 --> 00:06:32,899
they're not used to

199
00:06:32,899 --> 00:06:35,555
that sort of sort of thing.

200
00:06:35,555 --> 00:06:37,099
So people working in

201
00:06:37,099 --> 00:06:38,240
system dynamics or abused

202
00:06:38,240 --> 00:06:39,530
interviews in

203
00:06:39,530 --> 00:06:40,939
model building for
quite some time.

204
00:06:40,939 --> 00:06:42,260
But this has typically

205
00:06:42,260 --> 00:06:43,790
been done in formerly and

206
00:06:43,790 --> 00:06:45,440
without a systematic way to

207
00:06:45,440 --> 00:06:47,765
go from interview
data to model.

208
00:06:47,765 --> 00:06:49,970
And what I mean by
systematic Here

209
00:06:49,970 --> 00:06:52,234
is a method that
is transparent,

210
00:06:52,234 --> 00:06:53,449
reproducible,

211
00:06:53,449 --> 00:06:55,280
tracks modeler
influence and

212
00:06:55,280 --> 00:06:57,635
is as accurate as possible.

213
00:06:57,635 --> 00:07:00,410
And in 2012, Kim

214
00:07:00,410 --> 00:07:02,509
and Anderson outlined
a method for

215
00:07:02,509 --> 00:07:04,220
identifying causal links in

216
00:07:04,220 --> 00:07:05,869
qualitative data
and using them

217
00:07:05,869 --> 00:07:08,059
to assemble causal
loop diagrams

218
00:07:08,059 --> 00:07:09,859
in stock and flow diagrams,

219
00:07:09,859 --> 00:07:11,375
which are closely related.

220
00:07:11,375 --> 00:07:13,460
An example is shown
on the right here.

221
00:07:13,460 --> 00:07:15,500
And when studying
this method

222
00:07:15,500 --> 00:07:17,375
and trying it out on
some interview data,

223
00:07:17,375 --> 00:07:19,369
I had, I came across

224
00:07:19,369 --> 00:07:21,050
some limitations
that I hadn't seen

225
00:07:21,050 --> 00:07:23,104
mentioned in the
literature before.

226
00:07:23,104 --> 00:07:24,995
The main one was
that the method

227
00:07:24,995 --> 00:07:27,200
involved coding only for

228
00:07:27,200 --> 00:07:30,229
individual causal
links rather than

229
00:07:30,229 --> 00:07:31,910
larger causal
structures like

230
00:07:31,910 --> 00:07:33,874
feedback loops
are archetypes.

231
00:07:33,874 --> 00:07:35,359
And when I was using it,

232
00:07:35,359 --> 00:07:36,589
I found that I miss

233
00:07:36,589 --> 00:07:38,120
some important information

234
00:07:38,120 --> 00:07:39,500
that was being conveyed

235
00:07:39,500 --> 00:07:41,869
because people don't
necessarily spell out

236
00:07:41,869 --> 00:07:43,910
every individual
causal link in

237
00:07:43,910 --> 00:07:45,410
their mental model when

238
00:07:45,410 --> 00:07:47,090
they're being interviewed.

239
00:07:47,090 --> 00:07:49,955
So to illustrate this
point a bit more.

240
00:07:49,955 --> 00:07:52,309
If someone says they're

241
00:07:52,309 --> 00:07:54,739
learning to play the
piano, for example,

242
00:07:54,739 --> 00:07:56,600
or an organization
says they're training

243
00:07:56,600 --> 00:07:57,799
their employees to have

244
00:07:57,799 --> 00:07:59,420
a certain proficiency.

245
00:07:59,420 --> 00:08:01,159
The underlying
causal structure is

246
00:08:01,159 --> 00:08:03,245
a goal directed
feedback loop,

247
00:08:03,245 --> 00:08:05,960
balancing feedback loop,
which is shown here.

248
00:08:05,960 --> 00:08:08,405
And so if there's a gap

249
00:08:08,405 --> 00:08:11,450
between desired and
actual skill level,

250
00:08:11,450 --> 00:08:13,339
the person engages
in some training

251
00:08:13,339 --> 00:08:14,884
to improve their skill.

252
00:08:14,884 --> 00:08:16,760
And over time, the
current skill level

253
00:08:16,760 --> 00:08:19,070
should trend toward
the desired level.

254
00:08:19,070 --> 00:08:20,810
And this is a very common

255
00:08:20,810 --> 00:08:22,160
causal structure and fits

256
00:08:22,160 --> 00:08:23,959
any situation where there's

257
00:08:23,959 --> 00:08:26,794
a gap between the desired
and actual state,

258
00:08:26,794 --> 00:08:28,189
which prompts us to

259
00:08:28,189 --> 00:08:29,810
make an improvement
in something.

260
00:08:29,810 --> 00:08:32,420
And this is a structure
we all understand,

261
00:08:32,420 --> 00:08:33,440
even if we haven't heard of

262
00:08:33,440 --> 00:08:34,925
a goal-directed
feedback loop.

263
00:08:34,925 --> 00:08:36,529
So when someone says

264
00:08:36,529 --> 00:08:37,759
they're led me to
play the piano,

265
00:08:37,759 --> 00:08:39,694
we understand
what that means.

266
00:08:39,694 --> 00:08:40,849
There's, it's
gap that they're

267
00:08:40,849 --> 00:08:42,320
trying to address through

268
00:08:42,320 --> 00:08:43,670
some kind of
training such as

269
00:08:43,670 --> 00:08:45,575
practicing or
taking lessons.

270
00:08:45,575 --> 00:08:46,790
And they expect to try

271
00:08:46,790 --> 00:08:48,439
to get better over time.

272
00:08:48,439 --> 00:08:50,030
And so that phrase
learning to

273
00:08:50,030 --> 00:08:51,320
play the piano is

274
00:08:51,320 --> 00:08:52,580
shorthand for this entire

275
00:08:52,580 --> 00:08:55,054
causal structure
that isn't,

276
00:08:55,054 --> 00:08:57,319
isn't explicitly
mentioned but

277
00:08:57,319 --> 00:08:59,494
is referred to in
the interview.

278
00:08:59,494 --> 00:09:01,130
And so this gets at

279
00:09:01,130 --> 00:09:02,600
a key critique I have of

280
00:09:02,600 --> 00:09:04,190
Kim Anderson's method is if

281
00:09:04,190 --> 00:09:05,809
you're only cop coding for

282
00:09:05,809 --> 00:09:07,970
individual causal
relationships that are

283
00:09:07,970 --> 00:09:09,469
explicitly mentioned

284
00:09:09,469 --> 00:09:10,805
in your qualitative data,

285
00:09:10,805 --> 00:09:12,169
you're gonna mess larger

286
00:09:12,169 --> 00:09:13,759
causal structures
because so

287
00:09:13,759 --> 00:09:15,440
much of our communication

288
00:09:15,440 --> 00:09:17,299
is implied or implicit.

289
00:09:17,299 --> 00:09:20,120
So to really capture
someone's mental model,

290
00:09:20,120 --> 00:09:20,810
you need to be able to

291
00:09:20,810 --> 00:09:22,730
recognize and track these

292
00:09:22,730 --> 00:09:23,750
larger imply

293
00:09:23,750 --> 00:09:26,119
causal structures
like feedback loops.

294
00:09:26,119 --> 00:09:28,219
So in my second paper,
I try to address

295
00:09:28,219 --> 00:09:30,679
this gap by modifying
their method.

296
00:09:30,679 --> 00:09:31,970
And then in the
third paper, I

297
00:09:31,970 --> 00:09:34,010
present interview
strategies designed

298
00:09:34,010 --> 00:09:35,750
for eliciting data suitable

299
00:09:35,750 --> 00:09:38,180
for analysis with
this method.

300
00:09:38,180 --> 00:09:40,085
Okay, so that was a lot of

301
00:09:40,085 --> 00:09:41,300
background information

302
00:09:41,300 --> 00:09:42,470
to bring it altogether.

303
00:09:42,470 --> 00:09:43,759
And I'm a research focused

304
00:09:43,759 --> 00:09:45,935
on two main questions.

305
00:09:45,935 --> 00:09:47,539
The first was How have

306
00:09:47,539 --> 00:09:49,760
causal loop diagram has
been used to describe

307
00:09:49,760 --> 00:09:52,790
and analyze complexity
aware program theory

308
00:09:52,790 --> 00:09:53,929
or program theory
that tries

309
00:09:53,929 --> 00:09:55,834
to incorporate complexity.

310
00:09:55,834 --> 00:09:57,470
And how can

311
00:09:57,470 --> 00:09:59,360
interviews be
designed, conducted,

312
00:09:59,360 --> 00:10:01,579
and analyzed to
identify and

313
00:10:01,579 --> 00:10:04,685
diagram participant's
mental models.

314
00:10:04,685 --> 00:10:07,970
So to get into
the methods and

315
00:10:07,970 --> 00:10:10,730
results of each paper,

316
00:10:10,730 --> 00:10:12,620
we'll start with
the first one.

317
00:10:12,620 --> 00:10:14,539
For this paper, I

318
00:10:14,539 --> 00:10:16,489
conducted a
systematic review to

319
00:10:16,489 --> 00:10:17,795
locate studies that use

320
00:10:17,795 --> 00:10:20,779
causal loop diagramming
for program theory.

321
00:10:20,779 --> 00:10:23,585
So first, I searched

322
00:10:23,585 --> 00:10:26,090
several databases with
relevant search terms

323
00:10:26,090 --> 00:10:28,535
to loci over 300 articles.

324
00:10:28,535 --> 00:10:30,469
I then reviewed
the abstracts

325
00:10:30,469 --> 00:10:31,549
that those articles to

326
00:10:31,549 --> 00:10:34,250
find ones that fit a
few main criteria.

327
00:10:34,250 --> 00:10:36,319
So to be included the
articles had to be

328
00:10:36,319 --> 00:10:38,195
evaluation studies
and how to use

329
00:10:38,195 --> 00:10:40,670
causal loop diagramming
for program theory.

330
00:10:40,670 --> 00:10:42,860
And what that means
is that the diagram

331
00:10:42,860 --> 00:10:46,039
had to include the
program's intervention.

332
00:10:46,039 --> 00:10:51,080
Not just mapping
dynamics of a system,

333
00:10:51,080 --> 00:10:53,449
like background
information on a system.

334
00:10:53,449 --> 00:10:55,610
A lot of articles were
excluded because they

335
00:10:55,610 --> 00:10:58,309
use diagramming just
for needs assessment,

336
00:10:58,309 --> 00:10:59,449
which by itself doesn't

337
00:10:59,449 --> 00:11:00,859
qualify as program theory,

338
00:11:00,859 --> 00:11:02,149
has to have that
intervention

339
00:11:02,149 --> 00:11:04,610
component than it did at

340
00:11:04,610 --> 00:11:08,240
closer review of 60
full-text articles

341
00:11:08,240 --> 00:11:10,145
to see if they
fit the criteria.

342
00:11:10,145 --> 00:11:12,709
And in the end, 22
articles relating

343
00:11:12,709 --> 00:11:15,799
to 13 studies were
included in the analysis.

344
00:11:15,799 --> 00:11:17,149
So after I had my sample,

345
00:11:17,149 --> 00:11:18,470
I analyze the articles to

346
00:11:18,470 --> 00:11:19,970
identify how the diagrams

347
00:11:19,970 --> 00:11:21,050
are developed and used,

348
00:11:21,050 --> 00:11:22,550
what they looked like
and how well they

349
00:11:22,550 --> 00:11:23,720
conform to standards

350
00:11:23,720 --> 00:11:25,414
of causal loop diagramming,

351
00:11:25,414 --> 00:11:28,190
as well as the strengths
and limitations of

352
00:11:28,190 --> 00:11:29,389
using this approach that

353
00:11:29,389 --> 00:11:32,179
the authors mentioned
in their papers.

354
00:11:32,179 --> 00:11:33,649
I'll flip through

355
00:11:33,649 --> 00:11:35,119
a few examples from
these studies.

356
00:11:35,119 --> 00:11:37,024
You can get a general
feel for them.

357
00:11:37,024 --> 00:11:39,050
This one was
part of a series

358
00:11:39,050 --> 00:11:41,344
of diagrams on this topic.

359
00:11:41,344 --> 00:11:42,769
It describes how teaching

360
00:11:42,769 --> 00:11:43,910
fellows communication

361
00:11:43,910 --> 00:11:45,530
skills improved
through interactions

362
00:11:45,530 --> 00:11:47,389
with students and teachers.

363
00:11:47,389 --> 00:11:49,849
This one is created
to understand why

364
00:11:49,849 --> 00:11:51,499
a leadership
development program for

365
00:11:51,499 --> 00:11:54,275
Health System Managers
and Ghana failed.

366
00:11:54,275 --> 00:11:57,230
In this pretty messy
one was created

367
00:11:57,230 --> 00:11:59,690
as part of a realist
evaluation to

368
00:11:59,690 --> 00:12:01,310
evaluate a

369
00:12:01,310 --> 00:12:03,319
performance-based
financing intervention

370
00:12:03,319 --> 00:12:05,929
in Uganda and
healthcare sector.

371
00:12:05,929 --> 00:12:07,910
And this one looks at how

372
00:12:07,910 --> 00:12:10,205
a pen-based digital
learning intervention

373
00:12:10,205 --> 00:12:12,320
affect student
learning in the US.

374
00:12:12,320 --> 00:12:14,059
And so there are
three main ways

375
00:12:14,059 --> 00:12:15,499
the diagrams
were developed.

376
00:12:15,499 --> 00:12:18,350
Some were using
participatory methods

377
00:12:18,350 --> 00:12:19,789
that group model building.

378
00:12:19,789 --> 00:12:23,825
Others were developed using

379
00:12:23,825 --> 00:12:25,490
secondary analysis of

380
00:12:25,490 --> 00:12:27,739
existing evaluation data.

381
00:12:27,739 --> 00:12:29,089
And some of the
researchers who took

382
00:12:29,089 --> 00:12:32,239
this approach used Kim
Anderson's method,

383
00:12:32,239 --> 00:12:33,890
which I mentioned
earlier to

384
00:12:33,890 --> 00:12:36,140
analyze that prior data.

385
00:12:36,140 --> 00:12:37,580
And then some articles

386
00:12:37,580 --> 00:12:38,720
either didn't
describe how they

387
00:12:38,720 --> 00:12:40,429
developed their
models are used

388
00:12:40,429 --> 00:12:43,125
a modeller lead sort
of standard approach.

389
00:12:43,125 --> 00:12:45,550
So some key strengths of

390
00:12:45,550 --> 00:12:47,740
causal loop diagram
in this study offers

391
00:12:47,740 --> 00:12:50,409
identified include an
better understanding

392
00:12:50,409 --> 00:12:52,554
the system components
and behavior,

393
00:12:52,554 --> 00:12:54,190
clarifying the
intervention and

394
00:12:54,190 --> 00:12:57,295
implementation and
communicating findings.

395
00:12:57,295 --> 00:12:58,810
As some limitations are

396
00:12:58,810 --> 00:13:00,279
that it's resource and time

397
00:13:00,279 --> 00:13:01,870
intensive and
for the people

398
00:13:01,870 --> 00:13:03,984
using prior
evaluation data,

399
00:13:03,984 --> 00:13:06,880
many of them felt
constrained by the focus or

400
00:13:06,880 --> 00:13:08,529
scope of that data
because it wasn't

401
00:13:08,529 --> 00:13:10,825
collected with
modeling in mind.

402
00:13:10,825 --> 00:13:13,060
And one thing I
noticed was that

403
00:13:13,060 --> 00:13:15,069
these studies largely
did not cite each

404
00:13:15,069 --> 00:13:16,254
other and didn't have

405
00:13:16,254 --> 00:13:19,150
certain foundational
literature in common.

406
00:13:19,150 --> 00:13:21,220
Which means that
the researchers

407
00:13:21,220 --> 00:13:23,424
probably decided to use

408
00:13:23,424 --> 00:13:25,904
this causal loop
diagramming approach

409
00:13:25,904 --> 00:13:27,919
on their own based
on the needs of

410
00:13:27,919 --> 00:13:29,779
their study rather
than responding

411
00:13:29,779 --> 00:13:33,479
to an established
methods literature.

412
00:13:33,520 --> 00:13:35,870
Another key
finding was that

413
00:13:35,870 --> 00:13:37,820
many of the articles
included in this study

414
00:13:37,820 --> 00:13:39,530
did not closely adhere to

415
00:13:39,530 --> 00:13:41,690
the standards of causal
loop diagramming,

416
00:13:41,690 --> 00:13:43,729
both in visual layout and

417
00:13:43,729 --> 00:13:45,844
how they develop
their models.

418
00:13:45,844 --> 00:13:47,600
So for example,

419
00:13:47,600 --> 00:13:50,690
some researchers developed
their diagrams by

420
00:13:50,690 --> 00:13:53,539
taking a comprehensive
inventory

421
00:13:53,539 --> 00:13:56,104
of all possible variables.

422
00:13:56,104 --> 00:13:59,180
And then inventoried also

423
00:13:59,180 --> 00:14:00,889
what all the possible
relationships

424
00:14:00,889 --> 00:14:02,285
between those variables.

425
00:14:02,285 --> 00:14:03,890
And then saw where

426
00:14:03,890 --> 00:14:06,649
the feedback loops
may be from there.

427
00:14:06,649 --> 00:14:09,559
Whereas in system dynamics,

428
00:14:09,559 --> 00:14:11,209
it's more of an
iterative process

429
00:14:11,209 --> 00:14:12,079
where you want to match

430
00:14:12,079 --> 00:14:15,485
the model structure to
a certain behavior.

431
00:14:15,485 --> 00:14:16,129
And you might do

432
00:14:16,129 --> 00:14:18,635
an exercise, the
inventories,

433
00:14:18,635 --> 00:14:21,020
variables, relationships,
but it's sort of

434
00:14:21,020 --> 00:14:23,835
part of this larger
iterative process.

435
00:14:23,835 --> 00:14:26,440
So the main takeaway
from this paper was that

436
00:14:26,440 --> 00:14:27,550
causal loop diagramming for

437
00:14:27,550 --> 00:14:29,904
permanent theory
has advantages.

438
00:14:29,904 --> 00:14:31,509
And there's a need for

439
00:14:31,509 --> 00:14:33,774
development of
best practices.

440
00:14:33,774 --> 00:14:35,589
So in the second paper,

441
00:14:35,589 --> 00:14:37,090
I modified Kim and

442
00:14:37,090 --> 00:14:38,350
Anderson's method

443
00:14:38,350 --> 00:14:40,120
for generating
causal loop diagrams

444
00:14:40,120 --> 00:14:41,890
will qualitative
data by borrowing

445
00:14:41,890 --> 00:14:45,145
from qualitative methods
in system dynamics.

446
00:14:45,145 --> 00:14:47,200
And I'm a bird to

447
00:14:47,200 --> 00:14:48,340
the revised method is

448
00:14:48,340 --> 00:14:49,494
causal structure mapping,

449
00:14:49,494 --> 00:14:51,444
just to have a
shorthand for it.

450
00:14:51,444 --> 00:14:53,995
So let's compare
these two methods.

451
00:14:53,995 --> 00:14:55,690
So inhuman
Anderson's method,

452
00:14:55,690 --> 00:14:57,069
they start with
qualitative data,

453
00:14:57,069 --> 00:14:58,180
which is a transcript of

454
00:14:58,180 --> 00:15:00,115
a meeting or an interview.

455
00:15:00,115 --> 00:15:01,870
The first step is to do

456
00:15:01,870 --> 00:15:03,940
a standard
qualitative analysis

457
00:15:03,940 --> 00:15:06,570
of the text to discover
themes in the data.

458
00:15:06,570 --> 00:15:08,479
Then they go back
and identify

459
00:15:08,479 --> 00:15:09,770
causal relationships in

460
00:15:09,770 --> 00:15:11,029
the text which are

461
00:15:11,029 --> 00:15:14,195
individual causal links
between variables.

462
00:15:14,195 --> 00:15:16,519
Then they combine
those links

463
00:15:16,519 --> 00:15:18,724
into Word and
arrow diagrams,

464
00:15:18,724 --> 00:15:22,010
which are small linear
model segments.

465
00:15:22,010 --> 00:15:23,495
They do this manually.

466
00:15:23,495 --> 00:15:25,460
And then they combine

467
00:15:25,460 --> 00:15:26,690
these wording
arrow diagrams

468
00:15:26,690 --> 00:15:28,069
into a combined model,

469
00:15:28,069 --> 00:15:29,720
which is that this is

470
00:15:29,720 --> 00:15:31,834
also done manually
by the modeler.

471
00:15:31,834 --> 00:15:33,349
And then they go
back and create

472
00:15:33,349 --> 00:15:34,459
a table summarizing

473
00:15:34,459 --> 00:15:35,809
connections
between the model

474
00:15:35,809 --> 00:15:37,519
and the source data.

475
00:15:37,519 --> 00:15:39,995
And the rabbis method
that I proposed.

476
00:15:39,995 --> 00:15:41,900
I start by focusing
the inquiry,

477
00:15:41,900 --> 00:15:43,639
by familiarizing
myself with

478
00:15:43,639 --> 00:15:45,259
the source data
in reviewing

479
00:15:45,259 --> 00:15:47,840
the the transcripts and

480
00:15:47,840 --> 00:15:50,149
refining my
research questions.

481
00:15:50,149 --> 00:15:52,250
And then I used atlas TI,

482
00:15:52,250 --> 00:15:52,970
which is a type

483
00:15:52,970 --> 00:15:55,325
of qualitative
analysis software

484
00:15:55,325 --> 00:15:57,349
to code for different types

485
00:15:57,349 --> 00:15:58,880
of causal structures.

486
00:15:58,880 --> 00:16:00,830
What this means is I use

487
00:16:00,830 --> 00:16:03,500
the software to select
portions of text

488
00:16:03,500 --> 00:16:05,539
and then assigned
a certain code

489
00:16:05,539 --> 00:16:08,074
corresponding to
either causal link,

490
00:16:08,074 --> 00:16:10,130
feedback loop or
other structures.

491
00:16:10,130 --> 00:16:11,600
And when I assign
that code,

492
00:16:11,600 --> 00:16:12,920
I attached a comment in

493
00:16:12,920 --> 00:16:14,390
which I translated
what the purchase

494
00:16:14,390 --> 00:16:15,710
faint was saying into

495
00:16:15,710 --> 00:16:17,479
an abbreviated form of

496
00:16:17,479 --> 00:16:20,100
causal loop
diagram notation.

497
00:16:20,410 --> 00:16:23,270
And then after I
was done coding,

498
00:16:23,270 --> 00:16:24,529
I used the software to run

499
00:16:24,529 --> 00:16:26,330
a query to identify all of

500
00:16:26,330 --> 00:16:27,410
the quotations that I had

501
00:16:27,410 --> 00:16:29,584
coded as having
causal information.

502
00:16:29,584 --> 00:16:31,940
So the query had the
direct quote from

503
00:16:31,940 --> 00:16:35,299
the participant
alongside the codes and

504
00:16:35,299 --> 00:16:38,449
then the notes I had
taken in for all of

505
00:16:38,449 --> 00:16:39,890
the quotations
that I coded as

506
00:16:39,890 --> 00:16:42,005
having a feedback
loop or archetype.

507
00:16:42,005 --> 00:16:44,780
I reviewed the quotation
and my notes to

508
00:16:44,780 --> 00:16:46,339
create freehand sketches

509
00:16:46,339 --> 00:16:48,274
of the causal structures.

510
00:16:48,274 --> 00:16:50,194
And the advantage of using

511
00:16:50,194 --> 00:16:51,440
free hand sketching during

512
00:16:51,440 --> 00:16:52,520
this phase is that it

513
00:16:52,520 --> 00:16:53,930
allows the
modeller to review

514
00:16:53,930 --> 00:16:56,270
the source text
closely and do

515
00:16:56,270 --> 00:16:58,069
some trial and
error when creating

516
00:16:58,069 --> 00:16:59,179
a sketch that captures

517
00:16:59,179 --> 00:17:01,235
what the
participant saying.

518
00:17:01,235 --> 00:17:04,220
I created a causal
mapping table,

519
00:17:04,220 --> 00:17:05,060
which is a spreadsheet.

520
00:17:05,060 --> 00:17:07,009
The lists all of the
causal relationships I

521
00:17:07,009 --> 00:17:08,195
identified through

522
00:17:08,195 --> 00:17:09,829
the coding and through
the sketching.

523
00:17:09,829 --> 00:17:12,530
So the causal links were
taken directly from

524
00:17:12,530 --> 00:17:14,120
the Query Report and

525
00:17:14,120 --> 00:17:15,139
the sketches were broken

526
00:17:15,139 --> 00:17:16,895
down into causal links.

527
00:17:16,895 --> 00:17:18,500
So then I included

528
00:17:18,500 --> 00:17:20,360
the quotation
number as well,

529
00:17:20,360 --> 00:17:22,099
which was automatically
generated

530
00:17:22,099 --> 00:17:24,350
through the software

531
00:17:24,350 --> 00:17:26,809
and tracked whether
the relationship was

532
00:17:26,809 --> 00:17:28,069
mentioned explicitly

533
00:17:28,069 --> 00:17:29,509
by the interview
participant,

534
00:17:29,509 --> 00:17:31,310
was implied, or was

535
00:17:31,310 --> 00:17:33,274
it an assumption
that I made?

536
00:17:33,274 --> 00:17:35,600
Then I uploaded the
spreadsheet to Kalhu,

537
00:17:35,600 --> 00:17:38,285
which is a web-based
visualization platform.

538
00:17:38,285 --> 00:17:40,189
I then reorganize
the position of

539
00:17:40,189 --> 00:17:42,170
the variables and made
some minor edits to

540
00:17:42,170 --> 00:17:44,300
combine synonyms
or leak segments

541
00:17:44,300 --> 00:17:46,790
that were logically tied.

542
00:17:46,790 --> 00:17:49,594
So to compare
these two methods,

543
00:17:49,594 --> 00:17:52,730
a key difference is where
in the process does

544
00:17:52,730 --> 00:17:54,050
the modeler engage

545
00:17:54,050 --> 00:17:55,550
directly with
the source data,

546
00:17:55,550 --> 00:17:55,940
which is

547
00:17:55,940 --> 00:17:57,845
an interview transcript
in this case.

548
00:17:57,845 --> 00:18:00,710
So in Kamen Anderson's
method on the top,

549
00:18:00,710 --> 00:18:03,920
this only happens early
on in their process.

550
00:18:03,920 --> 00:18:07,790
And in the newer method
at the bottom, this,

551
00:18:07,790 --> 00:18:10,130
the use of software
allowed me to engage

552
00:18:10,130 --> 00:18:11,299
with the quotations during

553
00:18:11,299 --> 00:18:13,085
more of the process.

554
00:18:13,085 --> 00:18:15,889
Okay, so now let's look
at where the modeler

555
00:18:15,889 --> 00:18:17,270
constructs loops and

556
00:18:17,270 --> 00:18:19,039
larger structures
of the model,

557
00:18:19,039 --> 00:18:20,960
which is the point
at which the modeler

558
00:18:20,960 --> 00:18:23,330
has the greatest influence
in this process.

559
00:18:23,330 --> 00:18:25,130
So in Kim
Anderson's method,

560
00:18:25,130 --> 00:18:27,499
this happens when
assembling the Word

561
00:18:27,499 --> 00:18:30,109
and arrow diagrams into
a composite model.

562
00:18:30,109 --> 00:18:31,759
Which means that the model

563
00:18:31,759 --> 00:18:32,929
there isn't engaging with

564
00:18:32,929 --> 00:18:34,819
that source text when

565
00:18:34,819 --> 00:18:36,514
making those
judgment calls.

566
00:18:36,514 --> 00:18:39,050
They just have that the
word a narrow table.

567
00:18:39,050 --> 00:18:41,000
So in the revised method,

568
00:18:41,000 --> 00:18:42,379
these decisions happen when

569
00:18:42,379 --> 00:18:44,539
reviewing the quotations
in the query.

570
00:18:44,539 --> 00:18:45,800
And so this helps align

571
00:18:45,800 --> 00:18:47,119
the modelers decisions

572
00:18:47,119 --> 00:18:48,260
about model structure with

573
00:18:48,260 --> 00:18:49,730
the participant's
mental model

574
00:18:49,730 --> 00:18:50,180
because they're

575
00:18:50,180 --> 00:18:53,000
engaging directly with
that source material.

576
00:18:53,000 --> 00:18:55,490
Another key difference
is that because

577
00:18:55,490 --> 00:18:56,509
the newer approach uses

578
00:18:56,509 --> 00:18:58,414
software to help
with coding,

579
00:18:58,414 --> 00:18:59,900
an automatically generated

580
00:18:59,900 --> 00:19:01,099
quotation numbers can be

581
00:19:01,099 --> 00:19:04,460
used to track model
segments to source data,

582
00:19:04,460 --> 00:19:06,979
which is much more tedious.

583
00:19:06,979 --> 00:19:08,704
And the previous method.

584
00:19:08,704 --> 00:19:11,329
Also the creation of
a composite model,

585
00:19:11,329 --> 00:19:13,490
a streamline with the use

586
00:19:13,490 --> 00:19:13,579
of

587
00:19:13,579 --> 00:19:16,610
visualization software
than the newer method.

588
00:19:16,610 --> 00:19:18,440
And so I piloted this

589
00:19:18,440 --> 00:19:20,149
mapping approach
by conducting

590
00:19:20,149 --> 00:19:21,529
a secondary analysis of

591
00:19:21,529 --> 00:19:22,700
interview data from

592
00:19:22,700 --> 00:19:24,919
an existing study
that I'm involved in.

593
00:19:24,919 --> 00:19:26,510
And that's through
my work at

594
00:19:26,510 --> 00:19:27,560
the Oregon rural practice

595
00:19:27,560 --> 00:19:28,639
based research network or

596
00:19:28,639 --> 00:19:30,499
or brand OH, US, EU.

597
00:19:30,499 --> 00:19:32,779
And it's an implementation
science study

598
00:19:32,779 --> 00:19:34,924
which six staff members
from our brand.

599
00:19:34,924 --> 00:19:36,680
Work with primary
care clinics

600
00:19:36,680 --> 00:19:38,179
across the state to improve

601
00:19:38,179 --> 00:19:39,650
how the clinic screen for

602
00:19:39,650 --> 00:19:41,705
entry unhealthy
alcohol use,

603
00:19:41,705 --> 00:19:43,489
and this type of
assistance provided to

604
00:19:43,489 --> 00:19:45,664
clinics is called
practice facilitation.

605
00:19:45,664 --> 00:19:47,090
And there's a whole
literature on

606
00:19:47,090 --> 00:19:48,679
how practice
facilitators can

607
00:19:48,679 --> 00:19:52,054
be most effective in
our work with clinics.

608
00:19:52,054 --> 00:19:54,530
And so the research
question for

609
00:19:54,530 --> 00:19:55,730
the study has to do with

610
00:19:55,730 --> 00:19:57,470
how practice
facilitators Taylor,

611
00:19:57,470 --> 00:19:59,509
the support they
provide based

612
00:19:59,509 --> 00:20:00,800
on the needs of the clinic

613
00:20:00,800 --> 00:20:02,689
and their own expertise.

614
00:20:02,689 --> 00:20:06,814
And as part of
this research,

615
00:20:06,814 --> 00:20:08,555
raising the method I just

616
00:20:08,555 --> 00:20:11,059
outlined to help address

617
00:20:11,059 --> 00:20:13,190
the question what our
practice facilitators,

618
00:20:13,190 --> 00:20:15,439
mental models of
practice change.

619
00:20:15,439 --> 00:20:16,909
So the interviews will be

620
00:20:16,909 --> 00:20:18,109
conducted over
six months over

621
00:20:18,109 --> 00:20:19,219
the course of the
study and we'll

622
00:20:19,219 --> 00:20:20,689
create maps of
each interview.

623
00:20:20,689 --> 00:20:21,889
And so at the end
of the study,

624
00:20:21,889 --> 00:20:22,910
we can examine how

625
00:20:22,910 --> 00:20:23,959
their mental models changed

626
00:20:23,959 --> 00:20:26,764
over time and compare
between individuals.

627
00:20:26,764 --> 00:20:28,219
And so the interviews
I used for

628
00:20:28,219 --> 00:20:30,739
this paper where the
baseline interviews.

629
00:20:30,739 --> 00:20:33,679
And so I applied
this method.

630
00:20:33,679 --> 00:20:35,914
Yet to those six
baseline interviews.

631
00:20:35,914 --> 00:20:39,154
The figure might be
hard to see, but the,

632
00:20:39,154 --> 00:20:41,120
it shows an example of

633
00:20:41,120 --> 00:20:43,715
a quotation from an
interview at the top.

634
00:20:43,715 --> 00:20:45,350
The code that I applied

635
00:20:45,350 --> 00:20:47,014
to it during analysis.

636
00:20:47,014 --> 00:20:50,255
Hi, which indicated I
saw a feedback loop.

637
00:20:50,255 --> 00:20:51,950
The notes that I inserted

638
00:20:51,950 --> 00:20:53,060
into the comment field,

639
00:20:53,060 --> 00:20:55,430
what I was doing

640
00:20:55,430 --> 00:20:58,039
the coding to summarize
the loop that I saw.

641
00:20:58,039 --> 00:20:59,480
And then the final diagram

642
00:20:59,480 --> 00:21:00,830
which was in the model.

643
00:21:00,830 --> 00:21:03,199
And this feedback loop,
this explains how

644
00:21:03,199 --> 00:21:05,780
the more clinicians
and staff see

645
00:21:05,780 --> 00:21:06,979
the positive impacts of

646
00:21:06,979 --> 00:21:08,255
doing a brief intervention

647
00:21:08,255 --> 00:21:09,364
for someone who screen

648
00:21:09,364 --> 00:21:11,089
positive for unhealthy
alcohol use,

649
00:21:11,089 --> 00:21:13,070
the more buy in
they have for

650
00:21:13,070 --> 00:21:14,359
improving how they do

651
00:21:14,359 --> 00:21:15,619
screening and brief
intervention,

652
00:21:15,619 --> 00:21:17,839
which is a
reinforcing loop.

653
00:21:17,839 --> 00:21:20,495
Here's an example of
the same concept.

654
00:21:20,495 --> 00:21:23,000
Clinician and
staff buy-in in

655
00:21:23,000 --> 00:21:24,199
diagrams representing

656
00:21:24,199 --> 00:21:27,019
four different participants
mental models.

657
00:21:27,019 --> 00:21:29,629
And so this shows
how some of

658
00:21:29,629 --> 00:21:31,160
the variables
and causal links

659
00:21:31,160 --> 00:21:33,050
were similar between
participants.

660
00:21:33,050 --> 00:21:34,850
But there were also
quite a few differences

661
00:21:34,850 --> 00:21:36,919
in how densely tied

662
00:21:36,919 --> 00:21:38,839
the variable was to
others and whether it

663
00:21:38,839 --> 00:21:40,520
was connected to
a feedback loop.

664
00:21:40,520 --> 00:21:41,929
For example.

665
00:21:41,929 --> 00:21:43,940
As I'll show you
a few examples

666
00:21:43,940 --> 00:21:45,410
of complete diagrams.

667
00:21:45,410 --> 00:21:47,209
And these diagrams
are definitely large,

668
00:21:47,209 --> 00:21:49,565
but they reflect quite
a bit of the content

669
00:21:49,565 --> 00:21:52,520
from one hour
semi-structured interviews.

670
00:21:52,520 --> 00:21:53,990
And I'll talk a
bit later about

671
00:21:53,990 --> 00:21:55,789
how this use of causal
loop diagramming

672
00:21:55,789 --> 00:21:57,890
is different from the
more traditional uses

673
00:21:57,890 --> 00:21:59,615
of it in system dynamics.

674
00:21:59,615 --> 00:22:01,100
You'll see this one has

675
00:22:01,100 --> 00:22:02,975
quite a few feedback loops

676
00:22:02,975 --> 00:22:04,515
that don't actually
doesn't have any,

677
00:22:04,515 --> 00:22:05,570
which reflects that

678
00:22:05,570 --> 00:22:06,769
the participant
responded to

679
00:22:06,769 --> 00:22:07,790
questions by listing

680
00:22:07,790 --> 00:22:09,319
things they thought
were important.

681
00:22:09,319 --> 00:22:10,070
They didn't go into

682
00:22:10,070 --> 00:22:11,420
a whole lot of
detail about how

683
00:22:11,420 --> 00:22:12,740
the variables
related to each

684
00:22:12,740 --> 00:22:14,884
other or changed over time.

685
00:22:14,884 --> 00:22:16,519
And the interviewer,

686
00:22:16,519 --> 00:22:17,810
which was me in this case,

687
00:22:17,810 --> 00:22:19,670
could have probably done
a better job asking

688
00:22:19,670 --> 00:22:20,780
follow-up questions to

689
00:22:20,780 --> 00:22:22,220
elicit that information.

690
00:22:22,220 --> 00:22:23,509
So this is experience is

691
00:22:23,509 --> 00:22:24,890
one of the things that kind

692
00:22:24,890 --> 00:22:25,969
of put me in the
direction of

693
00:22:25,969 --> 00:22:28,579
my third paper Ravel
described later.

694
00:22:28,579 --> 00:22:30,664
Here's another example.

695
00:22:30,664 --> 00:22:32,540
When I was using this
mapping approach,

696
00:22:32,540 --> 00:22:34,429
I found that it is easier
to use and a lot of

697
00:22:34,429 --> 00:22:36,649
ways and less tedious
and the prayer method,

698
00:22:36,649 --> 00:22:39,169
but it was definitely
still time-consuming.

699
00:22:39,169 --> 00:22:41,629
So the two main
takeaways from

700
00:22:41,629 --> 00:22:43,849
this paper that
prior methods for

701
00:22:43,849 --> 00:22:45,409
generating causal
loop diagrams from

702
00:22:45,409 --> 00:22:47,390
qualitative data did not do

703
00:22:47,390 --> 00:22:48,739
a good enough job capturing

704
00:22:48,739 --> 00:22:51,560
feedback loops in larger
causal structures.

705
00:22:51,560 --> 00:22:53,989
And that causal
structure mapping

706
00:22:53,989 --> 00:22:55,309
leverages software and

707
00:22:55,309 --> 00:22:57,379
a closer analysis
of source texts to

708
00:22:57,379 --> 00:22:58,489
produce diagrams of

709
00:22:58,489 --> 00:23:00,304
individuals mental models.

710
00:23:00,304 --> 00:23:02,450
And main limitation
of the study was that

711
00:23:02,450 --> 00:23:03,559
this method was only used

712
00:23:03,559 --> 00:23:04,954
on one small data set.

713
00:23:04,954 --> 00:23:09,019
So future research
could apply

714
00:23:09,019 --> 00:23:11,105
this method in
various settings

715
00:23:11,105 --> 00:23:13,850
and in test out further.

716
00:23:13,850 --> 00:23:15,349
So during this research,

717
00:23:15,349 --> 00:23:16,370
I also wondered whether

718
00:23:16,370 --> 00:23:17,540
the final diagrams could

719
00:23:17,540 --> 00:23:18,559
have been improved if

720
00:23:18,559 --> 00:23:19,369
the questions I had

721
00:23:19,369 --> 00:23:20,570
asked during the
interviews were more

722
00:23:20,570 --> 00:23:23,825
tailored to identifying
causal structures.

723
00:23:23,825 --> 00:23:26,225
Which brings me to
the third paper.

724
00:23:26,225 --> 00:23:28,189
For this paper, I wanted

725
00:23:28,189 --> 00:23:29,869
to take a critical
look at how

726
00:23:29,869 --> 00:23:31,759
interviews can be
designed to elicit

727
00:23:31,759 --> 00:23:32,570
information about

728
00:23:32,570 --> 00:23:34,174
participant's
mental models.

729
00:23:34,174 --> 00:23:36,200
And so the system
scientists have

730
00:23:36,200 --> 00:23:37,459
used interviews for

731
00:23:37,459 --> 00:23:39,259
quite some time to
gather information,

732
00:23:39,259 --> 00:23:41,659
but there hasn't
been a lot,

733
00:23:41,659 --> 00:23:42,499
much written about how

734
00:23:42,499 --> 00:23:43,729
to design those interviews.

735
00:23:43,729 --> 00:23:45,470
So for this paper,
I integrated

736
00:23:45,470 --> 00:23:46,970
some strategies from

737
00:23:46,970 --> 00:23:49,070
three different
areas of research.

738
00:23:49,070 --> 00:23:51,695
The first is qualitative
interviewing.

739
00:23:51,695 --> 00:23:53,540
There are different flavors

740
00:23:53,540 --> 00:23:54,739
of qualitative
interviewing,

741
00:23:54,739 --> 00:23:56,090
but many of them are based

742
00:23:56,090 --> 00:23:58,580
on constructivist
philosophical assumptions,

743
00:23:58,580 --> 00:24:00,275
which means they ask

744
00:24:00,275 --> 00:24:01,879
open-ended questions
and try to

745
00:24:01,879 --> 00:24:03,259
capture the experience of

746
00:24:03,259 --> 00:24:06,289
the interview participant
in their own words.

747
00:24:06,289 --> 00:24:08,330
This approach works great

748
00:24:08,330 --> 00:24:10,100
for standard
qualitative research

749
00:24:10,100 --> 00:24:11,405
because it lets
the participant

750
00:24:11,405 --> 00:24:14,389
focus on what's
important to them.

751
00:24:14,389 --> 00:24:16,400
For model building.

752
00:24:16,400 --> 00:24:18,919
This open-ended
neutral approach

753
00:24:18,919 --> 00:24:21,109
works well for initial,

754
00:24:21,109 --> 00:24:22,489
an initial interview with

755
00:24:22,489 --> 00:24:23,690
someone when you trying to

756
00:24:23,690 --> 00:24:25,160
capture the main
gist of what

757
00:24:25,160 --> 00:24:26,885
they're saying or
their perspective.

758
00:24:26,885 --> 00:24:28,609
But the more I
looked into it,

759
00:24:28,609 --> 00:24:31,129
the more I realized
that the sort of

760
00:24:31,129 --> 00:24:33,409
constructivist perspective

761
00:24:33,409 --> 00:24:35,090
visited these assumptions

762
00:24:35,090 --> 00:24:37,039
aren't really enough for

763
00:24:37,039 --> 00:24:38,555
an entire
modelling project.

764
00:24:38,555 --> 00:24:40,445
Because when you're
trying to create

765
00:24:40,445 --> 00:24:41,690
a model to reflect someone

766
00:24:41,690 --> 00:24:43,039
else's perspective.

767
00:24:43,039 --> 00:24:44,765
There's a back and forth

768
00:24:44,765 --> 00:24:46,744
where you're
trying to bring

769
00:24:46,744 --> 00:24:49,339
your understanding
what they think and

770
00:24:49,339 --> 00:24:50,539
the line into line

771
00:24:50,539 --> 00:24:52,084
with what they
actually think.

772
00:24:52,084 --> 00:24:54,290
So you're doing
this back and forth

773
00:24:54,290 --> 00:24:55,670
and you

774
00:24:55,670 --> 00:24:56,179
refining

775
00:24:56,179 --> 00:24:57,770
your understanding of
their mental model.

776
00:24:57,770 --> 00:24:58,640
There needs to be a bit

777
00:24:58,640 --> 00:24:59,809
when you need to be
able to be a bit

778
00:24:59,809 --> 00:25:02,734
more explicit in the
questions you ask.

779
00:25:02,734 --> 00:25:05,059
Then is, is sort of

780
00:25:05,059 --> 00:25:06,019
typical and standard

781
00:25:06,019 --> 00:25:07,910
qualitative interviewing.

782
00:25:07,910 --> 00:25:08,899
So that's where

783
00:25:08,899 --> 00:25:10,669
a realist interviewing
comes in.

784
00:25:10,669 --> 00:25:12,890
Realists believe that
a real-world exists,

785
00:25:12,890 --> 00:25:14,689
but that individual
experience plays

786
00:25:14,689 --> 00:25:15,829
a big role in shaping

787
00:25:15,829 --> 00:25:17,434
our understanding
of that reality.

788
00:25:17,434 --> 00:25:19,234
So in realist interviewing,

789
00:25:19,234 --> 00:25:20,869
instead of posing
a question

790
00:25:20,869 --> 00:25:21,739
and letting the participant

791
00:25:21,739 --> 00:25:22,610
go where they want with it,

792
00:25:22,610 --> 00:25:23,929
the realist
interviewer takes

793
00:25:23,929 --> 00:25:25,190
a much more active role

794
00:25:25,190 --> 00:25:27,109
in steering the
conversation.

795
00:25:27,109 --> 00:25:29,315
There interview
questions are

796
00:25:29,315 --> 00:25:30,889
designed around
validating or

797
00:25:30,889 --> 00:25:33,800
falsifying certain
hypothesis that

798
00:25:33,800 --> 00:25:36,455
the interviewer comes
to the interview with.

799
00:25:36,455 --> 00:25:38,629
My one researcher,
men's Ohno,

800
00:25:38,629 --> 00:25:40,430
proposed a
three-phase process

801
00:25:40,430 --> 00:25:41,479
for realist interviewing,

802
00:25:41,479 --> 00:25:44,794
which I borrowed for.
For this research.

803
00:25:44,794 --> 00:25:46,879
I also wanted to make sure

804
00:25:46,879 --> 00:25:48,170
that the strategies

805
00:25:48,170 --> 00:25:49,669
we're focused on
getting information

806
00:25:49,669 --> 00:25:50,689
about causal structures.

807
00:25:50,689 --> 00:25:51,739
So I borrowed from

808
00:25:51,739 --> 00:25:54,139
system dynamics
modelling for that.

809
00:25:54,139 --> 00:25:56,239
So this figure shows how

810
00:25:56,239 --> 00:25:57,620
the process that
I came up with is

811
00:25:57,620 --> 00:26:00,230
a back and forth with
the investigator

812
00:26:00,230 --> 00:26:01,730
or the interviewer
on one side

813
00:26:01,730 --> 00:26:03,185
and the participant
on the other.

814
00:26:03,185 --> 00:26:04,909
And over time their
understanding

815
00:26:04,909 --> 00:26:06,559
comes into closer
alignment.

816
00:26:06,559 --> 00:26:09,289
So in the first step,
the interviewer defines

817
00:26:09,289 --> 00:26:10,790
the research
questions and set

818
00:26:10,790 --> 00:26:11,869
some initial boundaries

819
00:26:11,869 --> 00:26:13,265
for the investigation.

820
00:26:13,265 --> 00:26:15,109
Which could mean
that they define

821
00:26:15,109 --> 00:26:16,429
a certain problem or system

822
00:26:16,429 --> 00:26:18,184
behavior they want
to understand.

823
00:26:18,184 --> 00:26:19,850
Decide who they
want to interview,

824
00:26:19,850 --> 00:26:22,400
or decide that
certain things

825
00:26:22,400 --> 00:26:24,845
are outside the scope
of the research.

826
00:26:24,845 --> 00:26:27,094
The interviewer
then decides,

827
00:26:27,094 --> 00:26:28,760
designs an
interview guide and

828
00:26:28,760 --> 00:26:29,780
conducts initial

829
00:26:29,780 --> 00:26:31,399
interviews with
participants.

830
00:26:31,399 --> 00:26:32,929
In these interviews are

831
00:26:32,929 --> 00:26:34,909
exploratory and open ended,

832
00:26:34,909 --> 00:26:37,280
but the interviewer
follows some guidance for

833
00:26:37,280 --> 00:26:38,450
what to listen for and

834
00:26:38,450 --> 00:26:39,769
how to ask follow-up
questions,

835
00:26:39,769 --> 00:26:42,260
which I'll detail in a bit.

836
00:26:42,260 --> 00:26:44,479
After the initial
interviews,

837
00:26:44,479 --> 00:26:46,040
the recordings
are transcribed

838
00:26:46,040 --> 00:26:47,119
and analyzed according to

839
00:26:47,119 --> 00:26:49,190
the causal structure
mapping method

840
00:26:49,190 --> 00:26:51,649
I illustrated
and paper too.

841
00:26:51,649 --> 00:26:53,570
So causal loop
diagrams are produced

842
00:26:53,570 --> 00:26:56,030
for each interview
using that approach.

843
00:26:56,030 --> 00:26:58,520
And after the
diagrams are created,

844
00:26:58,520 --> 00:26:59,904
the interviewer analyze it.

845
00:26:59,904 --> 00:27:01,369
Some do identify gaps and

846
00:27:01,369 --> 00:27:03,379
places that need
clarification.

847
00:27:03,379 --> 00:27:05,689
Then they produce
an interview guide

848
00:27:05,689 --> 00:27:07,759
based on that analysis.

849
00:27:07,759 --> 00:27:09,830
And then follow-up
interviews are

850
00:27:09,830 --> 00:27:11,384
conducted that are more

851
00:27:11,384 --> 00:27:13,010
pointed and based on more

852
00:27:13,010 --> 00:27:14,659
of a realist
approach designed to

853
00:27:14,659 --> 00:27:16,430
flesh out the modelers
understanding

854
00:27:16,430 --> 00:27:19,055
of that participants
perspective.

855
00:27:19,055 --> 00:27:21,559
The data from the
follow-up interviews is

856
00:27:21,559 --> 00:27:24,934
analyzed using the
same mapping method.

857
00:27:24,934 --> 00:27:27,380
And then in sort of
an abbreviated form,

858
00:27:27,380 --> 00:27:29,240
and then I integrate

859
00:27:29,240 --> 00:27:31,490
it with the
previous diagrams.

860
00:27:31,490 --> 00:27:33,830
And then the model can
either be considered

861
00:27:33,830 --> 00:27:35,539
finished or there can be

862
00:27:35,539 --> 00:27:37,010
an optional third session

863
00:27:37,010 --> 00:27:38,360
where the models presented

864
00:27:38,360 --> 00:27:39,439
to the participants for

865
00:27:39,439 --> 00:27:41,750
feedback and an
additional editing.

866
00:27:41,750 --> 00:27:44,209
So if doing that
participatory review,

867
00:27:44,209 --> 00:27:46,145
final revisions will
be made and then

868
00:27:46,145 --> 00:27:47,390
documentation produced

869
00:27:47,390 --> 00:27:49,710
after that final review.

870
00:27:49,810 --> 00:27:52,939
So in my research,
I also describe

871
00:27:52,939 --> 00:27:55,730
some interview
strategies customized

872
00:27:55,730 --> 00:27:57,530
to the needs of causal
loop diagramming.

873
00:27:57,530 --> 00:27:58,865
And I'll briefly go over

874
00:27:58,865 --> 00:28:00,859
how to approach
the interview,

875
00:28:00,859 --> 00:28:02,629
what to ask, what
to listen for,

876
00:28:02,629 --> 00:28:05,360
and how to probe or ask
followup questions.

877
00:28:05,360 --> 00:28:06,859
So in broad strokes,

878
00:28:06,859 --> 00:28:08,524
the initial interview takes

879
00:28:08,524 --> 00:28:10,610
wherever constructivist
orientation and

880
00:28:10,610 --> 00:28:12,950
a follow-up interview
is a bit more realist.

881
00:28:12,950 --> 00:28:15,170
And interviews are problems

882
00:28:15,170 --> 00:28:16,550
centric in the
sense that they're

883
00:28:16,550 --> 00:28:18,319
focused on what
is relevant to

884
00:28:18,319 --> 00:28:21,139
understanding a certain
problem or a system.

885
00:28:21,139 --> 00:28:24,319
And this is, this comes
from system dynamics.

886
00:28:24,319 --> 00:28:27,664
So introductory questions
can build rapport,

887
00:28:27,664 --> 00:28:30,050
get people talking, make
people comfortable.

888
00:28:30,050 --> 00:28:31,609
You can have
questions about

889
00:28:31,609 --> 00:28:33,079
problem definition that are

890
00:28:33,079 --> 00:28:35,285
either open ended.

891
00:28:35,285 --> 00:28:37,910
If you want the participants
to take more of

892
00:28:37,910 --> 00:28:38,779
an active role in

893
00:28:38,779 --> 00:28:40,159
defining what
the problem is.

894
00:28:40,159 --> 00:28:42,350
Word it could be more
narrow depending on

895
00:28:42,350 --> 00:28:43,490
how you define

896
00:28:43,490 --> 00:28:45,290
those boundaries
ahead of time.

897
00:28:45,290 --> 00:28:47,089
I, key variables and

898
00:28:47,089 --> 00:28:49,699
relationships are
clearly important.

899
00:28:49,699 --> 00:28:51,230
If the mental
model has to do

900
00:28:51,230 --> 00:28:52,835
with a certain
intervention.

901
00:28:52,835 --> 00:28:55,280
Like for an evaluation
and program theory,

902
00:28:55,280 --> 00:28:57,605
the interviewer
can ask about

903
00:28:57,605 --> 00:29:01,055
the rationale or logic
behind the program.

904
00:29:01,055 --> 00:29:03,200
Questions can also address

905
00:29:03,200 --> 00:29:05,239
desired future outcomes and

906
00:29:05,239 --> 00:29:07,025
we're back to
potential causes.

907
00:29:07,025 --> 00:29:09,049
And so far these
are relatively

908
00:29:09,049 --> 00:29:10,430
standard for semi-structure

909
00:29:10,430 --> 00:29:12,665
qualitative interviews.

910
00:29:12,665 --> 00:29:14,870
Now the criteria for what

911
00:29:14,870 --> 00:29:16,700
interviewers should
listen for and

912
00:29:16,700 --> 00:29:18,889
how to ask
questions or more

913
00:29:18,889 --> 00:29:20,089
specialized to the needs

914
00:29:20,089 --> 00:29:21,710
of causal diagramming.

915
00:29:21,710 --> 00:29:23,719
So the main thing to listen

916
00:29:23,719 --> 00:29:25,610
for is a coherent narrative

917
00:29:25,610 --> 00:29:27,260
or explanation

918
00:29:27,260 --> 00:29:29,450
of the participant's
mental model.

919
00:29:29,450 --> 00:29:30,515
So the interviewer should

920
00:29:30,515 --> 00:29:31,940
actively listen
and make sure they

921
00:29:31,940 --> 00:29:34,909
fully understand what
the participant saying.

922
00:29:34,909 --> 00:29:36,470
If there's missing
pieces and ask

923
00:29:36,470 --> 00:29:38,989
follow-up questions
to fill those gaps.

924
00:29:38,989 --> 00:29:42,185
If someone says something
isn't relevant,

925
00:29:42,185 --> 00:29:44,119
that can also be
important information

926
00:29:44,119 --> 00:29:45,679
about boundaries
and boundaries

927
00:29:45,679 --> 00:29:48,020
include which
factors participants

928
00:29:48,020 --> 00:29:49,100
think are the
most important

929
00:29:49,100 --> 00:29:50,209
and which are recruit,

930
00:29:50,209 --> 00:29:51,544
which players are involved,

931
00:29:51,544 --> 00:29:53,750
that kind of thing
are not involved.

932
00:29:53,750 --> 00:29:55,850
And sometimes people make

933
00:29:55,850 --> 00:29:58,310
explicit references
to causal structures.

934
00:29:58,310 --> 00:30:01,294
So for reinforcing loops,

935
00:30:01,294 --> 00:30:02,794
someone could talk about

936
00:30:02,794 --> 00:30:04,969
a vicious cycle
or a cascade,

937
00:30:04,969 --> 00:30:06,529
something amplifying,

938
00:30:06,529 --> 00:30:08,689
or something getting
out of control.

939
00:30:08,689 --> 00:30:11,539
For. One of the
main clues about

940
00:30:11,539 --> 00:30:13,610
a balancing loop
is someone talking

941
00:30:13,610 --> 00:30:14,719
about working toward a goal

942
00:30:14,719 --> 00:30:16,114
like I described earlier.

943
00:30:16,114 --> 00:30:17,629
Bouncing feedback
is also in

944
00:30:17,629 --> 00:30:19,369
play when someone talks

945
00:30:19,369 --> 00:30:21,680
about something
reaching homeostasis

946
00:30:21,680 --> 00:30:24,259
are recovering
or stabilizing.

947
00:30:24,259 --> 00:30:26,240
It's also good
to listen for

948
00:30:26,240 --> 00:30:27,650
system archetypes which are

949
00:30:27,650 --> 00:30:29,599
certain causal
configurations that

950
00:30:29,599 --> 00:30:31,760
can apply to
various contexts.

951
00:30:31,760 --> 00:30:33,290
So if someone talks about

952
00:30:33,290 --> 00:30:35,165
short-term solutions
undermining,

953
00:30:35,165 --> 00:30:37,250
undermining or
fundamental solutions

954
00:30:37,250 --> 00:30:39,290
that shifting the
burden on the left,

955
00:30:39,290 --> 00:30:41,029
success to the successful

956
00:30:41,029 --> 00:30:42,079
is in the upper right,

957
00:30:42,079 --> 00:30:45,080
which describes resources
being shifted from

958
00:30:45,080 --> 00:30:46,310
an exploited group to

959
00:30:46,310 --> 00:30:48,739
a more ruling
group or class.

960
00:30:48,739 --> 00:30:50,450
If someone talks
about something

961
00:30:50,450 --> 00:30:51,530
being an arms race,

962
00:30:51,530 --> 00:30:53,179
that's the escalation
archetypes of

963
00:30:53,179 --> 00:30:54,769
these terms can clue

964
00:30:54,769 --> 00:30:56,734
you in to these structures.

965
00:30:56,734 --> 00:30:58,910
That's also important
to point out that

966
00:30:58,910 --> 00:31:01,490
references to causal
structures are,

967
00:31:01,490 --> 00:31:03,529
are often also implied.

968
00:31:03,529 --> 00:31:05,869
So the interviewer needs to

969
00:31:05,869 --> 00:31:08,360
check in and make
sure that their,

970
00:31:08,360 --> 00:31:10,579
their understanding
and clarifying

971
00:31:10,579 --> 00:31:12,740
when need to, when need be.

972
00:31:12,740 --> 00:31:15,769
So the last design criteria

973
00:31:15,769 --> 00:31:18,050
I propose has to
do with probes,

974
00:31:18,050 --> 00:31:19,804
which are follow-up
questions.

975
00:31:19,804 --> 00:31:21,350
An interviewer asks to get

976
00:31:21,350 --> 00:31:22,220
more clarity on

977
00:31:22,220 --> 00:31:23,869
something the
participant says.

978
00:31:23,869 --> 00:31:25,624
So in a modelling
interview,

979
00:31:25,624 --> 00:31:27,065
the interviewer
should listen

980
00:31:27,065 --> 00:31:28,909
closely and be
prepared to steer

981
00:31:28,909 --> 00:31:31,069
the conversation
with probes

982
00:31:31,069 --> 00:31:32,450
to make sure they have

983
00:31:32,450 --> 00:31:33,499
the information they need.

984
00:31:33,499 --> 00:31:34,910
So this can be done through

985
00:31:34,910 --> 00:31:35,989
paraphrasing an idea

986
00:31:35,989 --> 00:31:37,849
back or asking them to

987
00:31:37,849 --> 00:31:40,114
fill certain gaps
in understanding.

988
00:31:40,114 --> 00:31:41,885
Asking why and how

989
00:31:41,885 --> 00:31:43,519
questions are
important because they

990
00:31:43,519 --> 00:31:45,439
prompt the participant
to connect

991
00:31:45,439 --> 00:31:47,779
ideas until that
causal story.

992
00:31:47,779 --> 00:31:49,610
What if questions
can help you

993
00:31:49,610 --> 00:31:50,690
more information about

994
00:31:50,690 --> 00:31:52,999
underlying
structure as well.

995
00:31:52,999 --> 00:31:55,460
So to summarize
the advantages

996
00:31:55,460 --> 00:31:57,695
and limitations
of this approach.

997
00:31:57,695 --> 00:31:59,419
An advantage is that it

998
00:31:59,419 --> 00:32:01,369
provides a method
for understanding

999
00:32:01,369 --> 00:32:02,720
an individual's
mental model

1000
00:32:02,720 --> 00:32:03,499
in a way that can be

1001
00:32:03,499 --> 00:32:05,509
compared with
other individuals

1002
00:32:05,509 --> 00:32:07,670
or with other
points of time.

1003
00:32:07,670 --> 00:32:10,489
The method is
participatory in the sense

1004
00:32:10,489 --> 00:32:11,840
that it engages people

1005
00:32:11,840 --> 00:32:13,039
in a modelling process.

1006
00:32:13,039 --> 00:32:14,179
But it's also flexible

1007
00:32:14,179 --> 00:32:16,009
because it only
requires one-on-one

1008
00:32:16,009 --> 00:32:17,689
interviews rather
than getting

1009
00:32:17,689 --> 00:32:18,859
a whole group at

1010
00:32:18,859 --> 00:32:20,945
the same place in
the same time.

1011
00:32:20,945 --> 00:32:23,719
And also leverages
the advantages of

1012
00:32:23,719 --> 00:32:27,620
causal structure mapping
and the precision

1013
00:32:27,620 --> 00:32:30,019
and tracking embedded in

1014
00:32:30,019 --> 00:32:31,924
this approach makes it

1015
00:32:31,924 --> 00:32:34,699
suitable for use
in research.

1016
00:32:34,699 --> 00:32:37,909
So however, the method
is time consuming and

1017
00:32:37,909 --> 00:32:39,439
requires both qualitative

1018
00:32:39,439 --> 00:32:42,184
and modeling expertise.

1019
00:32:42,184 --> 00:32:45,020
And there are a few
different kinds

1020
00:32:45,020 --> 00:32:46,744
of applications
of this method,

1021
00:32:46,744 --> 00:32:48,274
a potential applications

1022
00:32:48,274 --> 00:32:49,700
in system dynamics
modelling.

1023
00:32:49,700 --> 00:32:51,889
It could be used to
provide more rigor and

1024
00:32:51,889 --> 00:32:53,299
transparency when using

1025
00:32:53,299 --> 00:32:55,535
interviews to build models,

1026
00:32:55,535 --> 00:32:58,219
which could increase
model credibility.

1027
00:32:58,219 --> 00:32:59,869
It can also be used as

1028
00:32:59,869 --> 00:33:01,445
a precursor or to

1029
00:33:01,445 --> 00:33:03,650
compliment group
model building.

1030
00:33:03,650 --> 00:33:05,390
In program evaluation,

1031
00:33:05,390 --> 00:33:06,859
this approach could
be used to help

1032
00:33:06,859 --> 00:33:08,179
map program theory or to

1033
00:33:08,179 --> 00:33:10,940
better understand
stakeholder perspectives.

1034
00:33:10,940 --> 00:33:12,589
And then in
qualitative research,

1035
00:33:12,589 --> 00:33:13,489
I think there's a lot of

1036
00:33:13,489 --> 00:33:16,925
potential applications
to explore

1037
00:33:16,925 --> 00:33:18,889
for any research where
understanding and

1038
00:33:18,889 --> 00:33:21,920
comparing mental
models is important.

1039
00:33:21,920 --> 00:33:24,365
This might be used
is also, I think,

1040
00:33:24,365 --> 00:33:27,949
room for expanding on
this research to see

1041
00:33:27,949 --> 00:33:30,980
how shed perspectives of

1042
00:33:30,980 --> 00:33:32,750
different groups
be combined

1043
00:33:32,750 --> 00:33:34,864
or compared, that
kind of thing.

1044
00:33:34,864 --> 00:33:36,349
And there has been interest

1045
00:33:36,349 --> 00:33:38,629
in various fields within

1046
00:33:38,629 --> 00:33:40,640
the qualitative realm for

1047
00:33:40,640 --> 00:33:42,619
new ways to analyze

1048
00:33:42,619 --> 00:33:44,389
or navigate
qualitative data.

1049
00:33:44,389 --> 00:33:45,649
And I think this
research that's

1050
00:33:45,649 --> 00:33:47,974
within that broader effort.

1051
00:33:47,974 --> 00:33:51,320
So a couple takeaways
from paper three.

1052
00:33:51,320 --> 00:33:53,360
Interviews
conducted to inform

1053
00:33:53,360 --> 00:33:54,650
modeling should be designed

1054
00:33:54,650 --> 00:33:55,924
with modelling in mind.

1055
00:33:55,924 --> 00:33:58,070
And this paper proposes

1056
00:33:58,070 --> 00:34:00,109
an interview based
method to mapping

1057
00:34:00,109 --> 00:34:01,339
mental models and design

1058
00:34:01,339 --> 00:34:03,379
criteria for interviews.

1059
00:34:03,379 --> 00:34:07,279
So I'd like to take a
few minutes to bring

1060
00:34:07,279 --> 00:34:09,320
these three papers together

1061
00:34:09,320 --> 00:34:12,064
and talk about some
main takeaways.

1062
00:34:12,064 --> 00:34:15,170
So to recap, in
reverse order,

1063
00:34:15,170 --> 00:34:17,255
kind of going left
to right here.

1064
00:34:17,255 --> 00:34:18,964
Paper three looks at

1065
00:34:18,964 --> 00:34:21,740
how to elicit
mental models from

1066
00:34:21,740 --> 00:34:24,124
interview participants
and gather

1067
00:34:24,124 --> 00:34:25,580
qualitative data suitable

1068
00:34:25,580 --> 00:34:27,410
for causal loop
diagramming.

1069
00:34:27,410 --> 00:34:29,539
And then paper
number two looks

1070
00:34:29,539 --> 00:34:31,519
at how to improve the
process of cleaning

1071
00:34:31,519 --> 00:34:32,840
causal loop diagrams from

1072
00:34:32,840 --> 00:34:34,340
that qualitative data

1073
00:34:34,340 --> 00:34:36,500
using causal
structure mapping.

1074
00:34:36,500 --> 00:34:38,329
And then paper
number one was

1075
00:34:38,329 --> 00:34:40,669
a systematic review
of how to, how to,

1076
00:34:40,669 --> 00:34:41,960
how causal loop
diagrams are

1077
00:34:41,960 --> 00:34:44,540
used for mapping
program theory,

1078
00:34:44,540 --> 00:34:45,559
which is one possible

1079
00:34:45,559 --> 00:34:48,394
application of
these methods.

1080
00:34:48,394 --> 00:34:50,899
So there are several
contributions to

1081
00:34:50,899 --> 00:34:52,100
knowledge of this research

1082
00:34:52,100 --> 00:34:53,674
that I want to highlight.

1083
00:34:53,674 --> 00:34:57,440
The methods I propose
and papers 23 describe

1084
00:34:57,440 --> 00:34:58,775
the importance of capturing

1085
00:34:58,775 --> 00:35:00,109
implied information and

1086
00:35:00,109 --> 00:35:02,255
larger causal structures.

1087
00:35:02,255 --> 00:35:04,580
They also try to
better account for

1088
00:35:04,580 --> 00:35:06,799
the role of the
modeler as co-creator,

1089
00:35:06,799 --> 00:35:08,119
which who brings

1090
00:35:08,119 --> 00:35:10,580
a certain perspective
and experience.

1091
00:35:10,580 --> 00:35:12,680
And the third one requires

1092
00:35:12,680 --> 00:35:14,000
a bit more explanation.

1093
00:35:14,000 --> 00:35:16,009
So conventionally
in system dynamics,

1094
00:35:16,009 --> 00:35:17,749
causal loop
diagrams are used

1095
00:35:17,749 --> 00:35:19,760
either as a
stepping stone to

1096
00:35:19,760 --> 00:35:22,280
computational
modelling or as

1097
00:35:22,280 --> 00:35:23,450
an educational tool to

1098
00:35:23,450 --> 00:35:25,715
describe feedback
relationships.

1099
00:35:25,715 --> 00:35:27,380
And the goal with
both of these is

1100
00:35:27,380 --> 00:35:28,909
to make the diagrams very

1101
00:35:28,909 --> 00:35:30,290
simple in order to

1102
00:35:30,290 --> 00:35:32,539
communicate a few
key dynamics.

1103
00:35:32,539 --> 00:35:35,689
And this makes sense
if you're using it

1104
00:35:35,689 --> 00:35:37,249
for education or if

1105
00:35:37,249 --> 00:35:39,634
you're going in the
direction of simulation.

1106
00:35:39,634 --> 00:35:40,460
Because when you're

1107
00:35:40,460 --> 00:35:41,990
making a simulation model,

1108
00:35:41,990 --> 00:35:43,759
and here I mean
computational

1109
00:35:43,759 --> 00:35:45,290
system dynamics model.

1110
00:35:45,290 --> 00:35:47,495
You're trying to
create a model that

1111
00:35:47,495 --> 00:35:49,070
reproduces the behavior you

1112
00:35:49,070 --> 00:35:51,440
see in a real-world system.

1113
00:35:51,440 --> 00:35:53,330
As causal loop diagrams.

1114
00:35:53,330 --> 00:35:55,220
The structure of
computational model is

1115
00:35:55,220 --> 00:35:56,929
your hypothesis about how

1116
00:35:56,929 --> 00:35:58,489
this real-world
system works.

1117
00:35:58,489 --> 00:36:00,200
And so you want your
hypothesis to be as

1118
00:36:00,200 --> 00:36:01,760
tiny as possible so you

1119
00:36:01,760 --> 00:36:03,679
can understand what shaping

1120
00:36:03,679 --> 00:36:05,269
the behavior of the
system because you're

1121
00:36:05,269 --> 00:36:07,924
going from model two

1122
00:36:07,924 --> 00:36:10,205
to trying to reproduce
that behavior.

1123
00:36:10,205 --> 00:36:11,180
And you want to understand

1124
00:36:11,180 --> 00:36:12,725
how your model does that.

1125
00:36:12,725 --> 00:36:15,080
So some researchers in

1126
00:36:15,080 --> 00:36:16,655
system dynamics or critique

1127
00:36:16,655 --> 00:36:18,199
causal loop
diagrams as being

1128
00:36:18,199 --> 00:36:20,120
inferior to
simulation modelling

1129
00:36:20,120 --> 00:36:21,979
or even misleading
because they

1130
00:36:21,979 --> 00:36:23,960
don't quantify the
relationships between

1131
00:36:23,960 --> 00:36:26,029
variables aside

1132
00:36:26,029 --> 00:36:28,010
for just that positive
and negative valence.

1133
00:36:28,010 --> 00:36:29,734
And so because
there's no math

1134
00:36:29,734 --> 00:36:31,669
behind the relationships

1135
00:36:31,669 --> 00:36:32,825
and causal loop diagrams,

1136
00:36:32,825 --> 00:36:34,159
there's no information
about what

1137
00:36:34,159 --> 00:36:35,719
we call loop dominance,

1138
00:36:35,719 --> 00:36:38,540
which is how influential
one feedback loop

1139
00:36:38,540 --> 00:36:41,359
as compared to
others in the model.

1140
00:36:41,359 --> 00:36:43,609
And I have a couple
of responses to that.

1141
00:36:43,609 --> 00:36:45,320
But the main one is that

1142
00:36:45,320 --> 00:36:46,609
these critiques of Judge

1143
00:36:46,609 --> 00:36:48,650
causal diagramming by

1144
00:36:48,650 --> 00:36:50,000
the standards of simulation

1145
00:36:50,000 --> 00:36:51,169
modeling when they don't

1146
00:36:51,169 --> 00:36:54,545
necessarily need to
have the same goals.

1147
00:36:54,545 --> 00:36:56,240
So causal loop diagramming

1148
00:36:56,240 --> 00:36:57,410
is never going to be better

1149
00:36:57,410 --> 00:36:58,670
than simulation modelling

1150
00:36:58,670 --> 00:37:00,380
at reproducing behavior.

1151
00:37:00,380 --> 00:37:02,299
But what it's really

1152
00:37:02,299 --> 00:37:04,250
good at is representing
information

1153
00:37:04,250 --> 00:37:05,810
such as someone's
mental model

1154
00:37:05,810 --> 00:37:07,279
in a way that is

1155
00:37:07,279 --> 00:37:11,285
visual and that
is relative,

1156
00:37:11,285 --> 00:37:12,875
comparatively simple.

1157
00:37:12,875 --> 00:37:15,469
So they do this in

1158
00:37:15,469 --> 00:37:17,989
a pretty compact
and precise way

1159
00:37:17,989 --> 00:37:19,519
using a notation and

1160
00:37:19,519 --> 00:37:21,935
the causal
diagramming notation

1161
00:37:21,935 --> 00:37:25,429
makes information
combinable with

1162
00:37:25,429 --> 00:37:26,779
other kinds of information.

1163
00:37:26,779 --> 00:37:28,730
So from this perspective,

1164
00:37:28,730 --> 00:37:30,319
causal loop
diagrams illustrate

1165
00:37:30,319 --> 00:37:31,669
relationships that
people see in

1166
00:37:31,669 --> 00:37:33,589
the world as

1167
00:37:33,589 --> 00:37:36,364
sort of a roadmap of
possible outcomes.

1168
00:37:36,364 --> 00:37:38,120
And this is a shift

1169
00:37:38,120 --> 00:37:40,190
towards seeing causal
loop diagrams as

1170
00:37:40,190 --> 00:37:42,199
representations
of knowledge or

1171
00:37:42,199 --> 00:37:43,880
information or sort of

1172
00:37:43,880 --> 00:37:45,244
a translation

1173
00:37:45,244 --> 00:37:48,740
of a certain type
of information.

1174
00:37:48,740 --> 00:37:50,764
In this case,
qualitative information.

1175
00:37:50,764 --> 00:37:52,849
Rather than the,
rather than just as

1176
00:37:52,849 --> 00:37:55,130
blueprints are the bones

1177
00:37:55,130 --> 00:37:57,210
of a simulation model.

1178
00:37:57,310 --> 00:37:59,869
So another contribution
I wanted to

1179
00:37:59,869 --> 00:38:01,880
mention is that
in this research,

1180
00:38:01,880 --> 00:38:03,919
I did a lot of comparing
and integrating

1181
00:38:03,919 --> 00:38:05,990
approaches from
different fields.

1182
00:38:05,990 --> 00:38:07,579
And I noticed a distinction

1183
00:38:07,579 --> 00:38:09,499
between the systems
approaches and

1184
00:38:09,499 --> 00:38:11,269
the more standard
approaches in

1185
00:38:11,269 --> 00:38:12,410
program evaluation and

1186
00:38:12,410 --> 00:38:14,569
also an implementation
science.

1187
00:38:14,569 --> 00:38:17,659
So the more common
approaches,

1188
00:38:17,659 --> 00:38:18,604
the centered,

1189
00:38:18,604 --> 00:38:20,719
the intervention
and the analysis.

1190
00:38:20,719 --> 00:38:22,249
And by intervention
I mean anything

1191
00:38:22,249 --> 00:38:23,930
that's done to address or

1192
00:38:23,930 --> 00:38:25,099
solve a problem such

1193
00:38:25,099 --> 00:38:26,839
as part of a social program

1194
00:38:26,839 --> 00:38:28,129
with being evaluated

1195
00:38:28,129 --> 00:38:29,840
or a clinical intervention.

1196
00:38:29,840 --> 00:38:32,330
So any characteristics of

1197
00:38:32,330 --> 00:38:33,500
the people or the sighting

1198
00:38:33,500 --> 00:38:35,915
involved in the
intervention or

1199
00:38:35,915 --> 00:38:38,794
kind of considered
local contexts.

1200
00:38:38,794 --> 00:38:39,529
But there's still sort

1201
00:38:39,529 --> 00:38:41,375
of outside the
intervention.

1202
00:38:41,375 --> 00:38:43,129
And then there's a
sort of broader,

1203
00:38:43,129 --> 00:38:45,770
more social or
regulatory context.

1204
00:38:45,770 --> 00:38:47,870
For example, when
these aspects

1205
00:38:47,870 --> 00:38:49,894
of context are
externalized and are

1206
00:38:49,894 --> 00:38:51,739
only really look that
through the lens of

1207
00:38:51,739 --> 00:38:55,160
how the influence
that intervention.

1208
00:38:55,160 --> 00:38:57,980
So now this way

1209
00:38:57,980 --> 00:39:00,139
of looking at things
isn't wrong per se,

1210
00:39:00,139 --> 00:39:01,640
but centering the
intervention in

1211
00:39:01,640 --> 00:39:03,500
this way might get in

1212
00:39:03,500 --> 00:39:07,654
the way of seeing how
existing systems work.

1213
00:39:07,654 --> 00:39:09,845
So in systems modeling
on the other hand,

1214
00:39:09,845 --> 00:39:11,779
what's central to
the analysis is

1215
00:39:11,779 --> 00:39:13,789
a complex understanding of

1216
00:39:13,789 --> 00:39:16,310
how different parts
of a system interact

1217
00:39:16,310 --> 00:39:18,575
to produce a system,
a certain behavior.

1218
00:39:18,575 --> 00:39:20,270
So variables that we might

1219
00:39:20,270 --> 00:39:22,189
consider contexts such as

1220
00:39:22,189 --> 00:39:24,589
the characteristics
of the people

1221
00:39:24,589 --> 00:39:26,780
or the setting involved
in the intervention,

1222
00:39:26,780 --> 00:39:28,489
are included in
the model based on

1223
00:39:28,489 --> 00:39:29,329
how they interact with

1224
00:39:29,329 --> 00:39:30,484
other parts of the system.

1225
00:39:30,484 --> 00:39:32,720
So after you
develop a model of

1226
00:39:32,720 --> 00:39:35,090
how you understand
the system to work,

1227
00:39:35,090 --> 00:39:37,220
you can identify
possible leverage points

1228
00:39:37,220 --> 00:39:39,319
which are pizza
places in the system.

1229
00:39:39,319 --> 00:39:41,689
An intervention
might be effective.

1230
00:39:41,689 --> 00:39:44,030
Interventions are
seen as acting on

1231
00:39:44,030 --> 00:39:46,549
that preexisting system and

1232
00:39:46,549 --> 00:39:47,750
centering the problem like

1233
00:39:47,750 --> 00:39:49,670
this rather than
the intervention,

1234
00:39:49,670 --> 00:39:52,339
might open the door to
interventions that are

1235
00:39:52,339 --> 00:39:56,299
more closely tailored
to local needs.

1236
00:39:56,299 --> 00:39:58,639
So thinking about
the potential

1237
00:39:58,639 --> 00:39:59,719
impact of this research,

1238
00:39:59,719 --> 00:40:00,409
there are a
couple of things

1239
00:40:00,409 --> 00:40:01,895
I want to highlight.

1240
00:40:01,895 --> 00:40:03,890
Together. I think
these papers

1241
00:40:03,890 --> 00:40:05,614
contribute to a
broader effort

1242
00:40:05,614 --> 00:40:06,829
toward more rigor and

1243
00:40:06,829 --> 00:40:08,030
transparency for causal

1244
00:40:08,030 --> 00:40:09,814
loop diagram development.

1245
00:40:09,814 --> 00:40:11,720
And other thing
is that capturing

1246
00:40:11,720 --> 00:40:12,859
the mental model of

1247
00:40:12,859 --> 00:40:14,150
an interview
participant and

1248
00:40:14,150 --> 00:40:16,729
translating it into a
causal loop diagram.

1249
00:40:16,729 --> 00:40:18,470
Essentially putting
qualitative

1250
00:40:18,470 --> 00:40:19,580
data into a format

1251
00:40:19,580 --> 00:40:20,840
that can be integrated

1252
00:40:20,840 --> 00:40:22,505
with other data sources.

1253
00:40:22,505 --> 00:40:25,100
So for example, these
diagrams could be

1254
00:40:25,100 --> 00:40:26,420
used to integrate findings

1255
00:40:26,420 --> 00:40:28,040
from scientific literature,

1256
00:40:28,040 --> 00:40:30,635
local or stakeholder
knowledge and practice,

1257
00:40:30,635 --> 00:40:32,629
practitioner
knowledge held by

1258
00:40:32,629 --> 00:40:34,160
people doing an
intervention.

1259
00:40:34,160 --> 00:40:35,779
So this opens the door to

1260
00:40:35,779 --> 00:40:38,885
new kinds of decision
support tools.

1261
00:40:38,885 --> 00:40:41,060
The biggest limitation of

1262
00:40:41,060 --> 00:40:43,400
this research is that
the interview based

1263
00:40:43,400 --> 00:40:45,200
method modelling approach
that I outlined in

1264
00:40:45,200 --> 00:40:48,005
the third paper has not
been fully piloted.

1265
00:40:48,005 --> 00:40:50,779
I've used many of the
strategies mentioned in

1266
00:40:50,779 --> 00:40:51,949
the interviews that I

1267
00:40:51,949 --> 00:40:53,090
conducted for paper too,

1268
00:40:53,090 --> 00:40:54,724
but it wasn't a
formal pilot.

1269
00:40:54,724 --> 00:40:57,680
And also the methods and

1270
00:40:57,680 --> 00:40:58,850
paper two also should

1271
00:40:58,850 --> 00:41:01,475
be explored more as well.

1272
00:41:01,475 --> 00:41:03,410
Also, I drew from

1273
00:41:03,410 --> 00:41:05,299
several different fields
from this research,

1274
00:41:05,299 --> 00:41:06,590
which means there
were multiple

1275
00:41:06,590 --> 00:41:08,210
literatures to draw from.

1276
00:41:08,210 --> 00:41:10,849
And many different types of

1277
00:41:10,849 --> 00:41:12,515
similar sort of

1278
00:41:12,515 --> 00:41:15,590
mapping systems or
interview strategies.

1279
00:41:15,590 --> 00:41:17,570
Traditions of every
interviewing and

1280
00:41:17,570 --> 00:41:19,954
so exploring every
possible angle.

1281
00:41:19,954 --> 00:41:21,529
Literature review
wasn't feasible

1282
00:41:21,529 --> 00:41:22,595
in the time I had.

1283
00:41:22,595 --> 00:41:23,719
But there's,
there's probably

1284
00:41:23,719 --> 00:41:24,949
a lot more opportunity from

1285
00:41:24,949 --> 00:41:26,480
learning from
different perspectives

1286
00:41:26,480 --> 00:41:28,400
and traditions.

1287
00:41:28,400 --> 00:41:30,589
So I see this research is

1288
00:41:30,589 --> 00:41:32,150
forming the basis for

1289
00:41:32,150 --> 00:41:33,995
a lot of future research.

1290
00:41:33,995 --> 00:41:35,689
For example, Best Practices

1291
00:41:35,689 --> 00:41:37,219
for causal loop
diagrams for

1292
00:41:37,219 --> 00:41:38,390
program theory could be

1293
00:41:38,390 --> 00:41:39,650
developed based on

1294
00:41:39,650 --> 00:41:41,434
their review and paper won.

1295
00:41:41,434 --> 00:41:43,700
The methods from paper
was 23 should be

1296
00:41:43,700 --> 00:41:45,560
tested and refined further.

1297
00:41:45,560 --> 00:41:46,940
And like I mentioned, in

1298
00:41:46,940 --> 00:41:48,020
multiple settings

1299
00:41:48,020 --> 00:41:50,060
and guidance should
be developed

1300
00:41:50,060 --> 00:41:52,339
regarding when to
use these methods

1301
00:41:52,339 --> 00:41:55,729
versus other
existing approaches.

1302
00:41:55,729 --> 00:41:58,254
And finally, the use
of these methods,

1303
00:41:58,254 --> 00:41:59,579
sizing different kinds of

1304
00:41:59,579 --> 00:42:02,775
information should
be explored.

1305
00:42:02,775 --> 00:42:04,619
So I'd like to
leave you with

1306
00:42:04,619 --> 00:42:06,839
a closing thought
from Donella Meadows,

1307
00:42:06,839 --> 00:42:08,429
who is a system
scientist who's

1308
00:42:08,429 --> 00:42:10,755
inspired me and a
lot of my work.

1309
00:42:10,755 --> 00:42:13,079
So people who are raised in

1310
00:42:13,079 --> 00:42:15,509
the industrial world
and get hoot and who

1311
00:42:15,509 --> 00:42:16,710
get enthused by systems

1312
00:42:16,710 --> 00:42:17,700
thinking are likely to

1313
00:42:17,700 --> 00:42:20,265
make a terrible
mistake. They like.

1314
00:42:20,265 --> 00:42:22,305
They're likely to
assume that here.

1315
00:42:22,305 --> 00:42:23,609
And Systems Analysis

1316
00:42:23,609 --> 00:42:24,869
in the interconnection and

1317
00:42:24,869 --> 00:42:27,210
complication and the
power of the computer

1318
00:42:27,210 --> 00:42:28,499
here at last is the key

1319
00:42:28,499 --> 00:42:30,100
to prediction and control.

1320
00:42:30,100 --> 00:42:33,019
This mistake is likely
because the mindset of

1321
00:42:33,019 --> 00:42:34,700
the industrial world
assumes that there

1322
00:42:34,700 --> 00:42:36,875
is a key to prediction
and control.

1323
00:42:36,875 --> 00:42:38,569
But social systems are

1324
00:42:38,569 --> 00:42:40,969
the external
manifestations of

1325
00:42:40,969 --> 00:42:42,350
cultural thinking patterns

1326
00:42:42,350 --> 00:42:44,240
enough profound
human needs,

1327
00:42:44,240 --> 00:42:46,460
emotions, strengths
and weaknesses.

1328
00:42:46,460 --> 00:42:48,019
We can't control systems

1329
00:42:48,019 --> 00:42:48,874
are figuring them out,

1330
00:42:48,874 --> 00:42:50,360
but we can dance with them.

1331
00:42:50,360 --> 00:42:52,490
Living successfully
in a world of systems

1332
00:42:52,490 --> 00:42:53,719
requires more of us

1333
00:42:53,719 --> 00:42:55,264
than our ability
to calculate,

1334
00:42:55,264 --> 00:42:57,019
requires our full humanity,

1335
00:42:57,019 --> 00:42:59,190
our rationality, our
ability to sort out

1336
00:42:59,190 --> 00:43:01,370
truth from falsehood
or intuition,

1337
00:43:01,370 --> 00:43:04,769
our compassion, our
vision, and our morality.

1338
00:43:05,020 --> 00:43:07,190
And I'd like to say
thank you again

1339
00:43:07,190 --> 00:43:08,630
for everyone for

1340
00:43:08,630 --> 00:43:10,309
joining me today and I look

1341
00:43:10,309 --> 00:43:13,079
forward to having
some discussion.

1342
00:43:14,400 --> 00:43:16,719
Well here my
clap, because I,

1343
00:43:16,719 --> 00:43:18,490
I unmuted my back.

1344
00:43:18,490 --> 00:43:21,070
All clapping. That
was terrific.

1345
00:43:21,070 --> 00:43:23,079
And I love that
quote at the end.

1346
00:43:23,079 --> 00:43:24,459
That was really
a nice way to

1347
00:43:24,459 --> 00:43:26,799
cap off your presentation.

1348
00:43:26,799 --> 00:43:30,714
Yeah. And so I'm going
to open it up now.

1349
00:43:30,714 --> 00:43:32,964
First to committee members.

1350
00:43:32,964 --> 00:43:35,169
And I guess I

1351
00:43:35,169 --> 00:43:36,609
would like to see is
someone who I would

1352
00:43:36,609 --> 00:43:37,749
like to lead off and then

1353
00:43:37,749 --> 00:43:39,280
we'll take a
little while to

1354
00:43:39,280 --> 00:43:40,749
give kind of a little bit

1355
00:43:40,749 --> 00:43:42,789
of a priority to
committee members.

1356
00:43:42,789 --> 00:43:43,480
And then we'll open it up

1357
00:43:43,480 --> 00:43:44,379
to the audience because

1358
00:43:44,379 --> 00:43:47,150
we'd love to have your
interaction as well.

1359
00:43:57,520 --> 00:44:00,739
I can jump in errand.

1360
00:44:00,739 --> 00:44:03,889
And it's more of a
speculative question.

1361
00:44:03,889 --> 00:44:04,625
You know, we had

1362
00:44:04,625 --> 00:44:06,230
the opportunity to
ask you a lot of

1363
00:44:06,230 --> 00:44:07,310
clarifications on

1364
00:44:07,310 --> 00:44:09,499
the exact
dissertation content.

1365
00:44:09,499 --> 00:44:10,999
But one of the things I've

1366
00:44:10,999 --> 00:44:13,099
always appreciated
about your work

1367
00:44:13,099 --> 00:44:14,450
coming from an
implementation

1368
00:44:14,450 --> 00:44:15,530
science perspective

1369
00:44:15,530 --> 00:44:16,609
is that focus on

1370
00:44:16,609 --> 00:44:18,650
centring head of
the problem or

1371
00:44:18,650 --> 00:44:21,694
the system versus centering
the intervention.

1372
00:44:21,694 --> 00:44:23,389
And in your presentation

1373
00:44:23,389 --> 00:44:25,160
today when you talked about

1374
00:44:25,160 --> 00:44:27,410
those to you and
the opportunity

1375
00:44:27,410 --> 00:44:29,690
that Taylor the
intervention,

1376
00:44:29,690 --> 00:44:31,279
it made me
actually think we

1377
00:44:31,279 --> 00:44:33,139
haven't talked
about the role

1378
00:44:33,139 --> 00:44:35,359
implementation
strategies might play

1379
00:44:35,359 --> 00:44:37,220
in bridging those two.

1380
00:44:37,220 --> 00:44:41,659
Um, and so this is
totally beyond the scope,

1381
00:44:41,659 --> 00:44:42,725
but do you want to

1382
00:44:42,725 --> 00:44:44,540
talk about that or
how you might think

1383
00:44:44,540 --> 00:44:46,459
about integrating
that concept into

1384
00:44:46,459 --> 00:44:47,689
some other future work you

1385
00:44:47,689 --> 00:44:49,400
might move forward in

1386
00:44:49,400 --> 00:44:51,500
terms of the activities

1387
00:44:51,500 --> 00:44:52,684
at team might do to help

1388
00:44:52,684 --> 00:44:54,319
change or address a problem

1389
00:44:54,319 --> 00:44:56,720
and an align that to
the intervention?

1390
00:44:56,720 --> 00:44:58,760
Sure. That's a
good question.

1391
00:44:58,760 --> 00:45:00,439
So I mean,

1392
00:45:00,439 --> 00:45:01,999
I think there's a few
things about that.

1393
00:45:01,999 --> 00:45:03,799
I think the
first thing that

1394
00:45:03,799 --> 00:45:06,110
comes to mind is
something that I

1395
00:45:06,110 --> 00:45:09,199
think is I don't

1396
00:45:09,199 --> 00:45:10,730
know about common
knowledge or it's a,

1397
00:45:10,730 --> 00:45:13,160
it's a lesson in
system science,

1398
00:45:13,160 --> 00:45:14,990
which is to know
your own place

1399
00:45:14,990 --> 00:45:16,399
in the system as well.

1400
00:45:16,399 --> 00:45:19,549
And to understand when

1401
00:45:19,549 --> 00:45:21,920
you're trying to
understand a system,

1402
00:45:21,920 --> 00:45:27,680
to try to understand
what your influences,

1403
00:45:27,680 --> 00:45:30,995
how your perspective shapes

1404
00:45:30,995 --> 00:45:32,299
your understanding of it.

1405
00:45:32,299 --> 00:45:34,789
And then also how

1406
00:45:34,789 --> 00:45:37,580
you're positioned
to understand how

1407
00:45:37,580 --> 00:45:38,719
your position to
then be able to

1408
00:45:38,719 --> 00:45:41,690
understand how your where

1409
00:45:41,690 --> 00:45:43,520
your potential points
of leverage are or

1410
00:45:43,520 --> 00:45:45,860
where you can
have influence.

1411
00:45:45,860 --> 00:45:47,270
And so, I don't know,

1412
00:45:47,270 --> 00:45:49,490
I see implementation
strategies as sort of

1413
00:45:49,490 --> 00:45:52,054
part of that
conversation as part of,

1414
00:45:52,054 --> 00:45:53,914
you know, first of all,

1415
00:45:53,914 --> 00:45:55,430
when you're
really trying to

1416
00:45:55,430 --> 00:45:57,679
understand what
that system is,

1417
00:45:57,679 --> 00:45:59,990
is to try to, you know,

1418
00:45:59,990 --> 00:46:03,079
if this is a implementation
science study

1419
00:46:03,079 --> 00:46:05,119
where you have practice
facilitators working

1420
00:46:05,119 --> 00:46:07,129
with clinics to try to put

1421
00:46:07,129 --> 00:46:10,580
those people into your
understanding of okay,

1422
00:46:10,580 --> 00:46:11,809
not just how to
clinics work,

1423
00:46:11,809 --> 00:46:13,430
but how does that clinic

1424
00:46:13,430 --> 00:46:15,934
practice facilitator
relationship work

1425
00:46:15,934 --> 00:46:17,300
or how does the
relationship,

1426
00:46:17,300 --> 00:46:18,845
if you're looking
at a nonprofit

1427
00:46:18,845 --> 00:46:20,869
working in a
certain community,

1428
00:46:20,869 --> 00:46:22,565
how does that, how,

1429
00:46:22,565 --> 00:46:23,959
where is that non-profit

1430
00:46:23,959 --> 00:46:25,670
situated in that community,

1431
00:46:25,670 --> 00:46:26,960
and how do they,
how can you

1432
00:46:26,960 --> 00:46:28,309
build on those
relationships?

1433
00:46:28,309 --> 00:46:32,479
And so I would see
that intervention,

1434
00:46:32,479 --> 00:46:34,189
implementation
strategies is kind of

1435
00:46:34,189 --> 00:46:36,590
coming out of that
understanding of those,

1436
00:46:36,590 --> 00:46:39,710
that embeddedness
and then also trying

1437
00:46:39,710 --> 00:46:43,399
to kind of
reassess and see.

1438
00:46:43,399 --> 00:46:44,809
When you when you have

1439
00:46:44,809 --> 00:46:46,594
an intervention
finding court,

1440
00:46:46,594 --> 00:46:48,365
you incorporate that
into your model,

1441
00:46:48,365 --> 00:46:49,130
you know, because that is

1442
00:46:49,130 --> 00:46:50,405
part of your system now.

1443
00:46:50,405 --> 00:46:51,380
And then trying to

1444
00:46:51,380 --> 00:46:52,955
constantly reassess as you,

1445
00:46:52,955 --> 00:46:54,259
as you go along
and hopefully

1446
00:46:54,259 --> 00:46:56,705
having that kind of
feedback loop of

1447
00:46:56,705 --> 00:47:02,029
constant
improvement at Air.

1448
00:47:02,029 --> 00:47:03,380
And I was going to
suggest if you want

1449
00:47:03,380 --> 00:47:04,864
to unshare for a second,

1450
00:47:04,864 --> 00:47:06,020
David will have the gallery

1451
00:47:06,020 --> 00:47:07,189
view and we can kinda all

1452
00:47:07,189 --> 00:47:08,449
see each other
and sometimes

1453
00:47:08,449 --> 00:47:09,709
that's a good way to yeah,

1454
00:47:09,709 --> 00:47:10,759
I wasn't sure
if I would have

1455
00:47:10,759 --> 00:47:12,230
to switch back to my sides,

1456
00:47:12,230 --> 00:47:13,459
but I can always do that.

1457
00:47:13,459 --> 00:47:15,200
Of course, this way people

1458
00:47:15,200 --> 00:47:16,159
can sort of see each other

1459
00:47:16,159 --> 00:47:17,269
who was in the room door.

1460
00:47:17,269 --> 00:47:18,860
Yeah, that's okay.

1461
00:47:18,860 --> 00:47:22,290
Other questions
from the community?

1462
00:47:22,510 --> 00:47:27,470
Back over. Comment. Any
technical question?

1463
00:47:27,470 --> 00:47:29,975
So I think my my comment

1464
00:47:29,975 --> 00:47:31,220
is what I really like about

1465
00:47:31,220 --> 00:47:32,660
your work is that you are

1466
00:47:32,660 --> 00:47:34,819
taking your using
system dynamics.

1467
00:47:34,819 --> 00:47:36,800
I think in all honesty,
you should say, well,

1468
00:47:36,800 --> 00:47:38,899
very much have the mindset

1469
00:47:38,899 --> 00:47:41,150
of a systems dynamics
model or you're looking

1470
00:47:41,150 --> 00:47:43,159
for archetypes and
you're looking for

1471
00:47:43,159 --> 00:47:44,539
the language of system

1472
00:47:44,539 --> 00:47:45,979
dynamics and
those interviews,

1473
00:47:45,979 --> 00:47:47,300
you're not out
there getting

1474
00:47:47,300 --> 00:47:49,954
mental models like
psychologists do this.

1475
00:47:49,954 --> 00:47:51,680
You're a systems
dynamics model,

1476
00:47:51,680 --> 00:47:54,965
but what you do is you
are really thinking of

1477
00:47:54,965 --> 00:47:56,869
people contributing
knowledge that's

1478
00:47:56,869 --> 00:47:59,225
relevant about the
systems. It's not this.

1479
00:47:59,225 --> 00:48:01,400
The model will tell
you all the answers,

1480
00:48:01,400 --> 00:48:03,604
but the people will tell
you all the answers.

1481
00:48:03,604 --> 00:48:05,149
And I, and I think what

1482
00:48:05,149 --> 00:48:06,485
was missing a little bit

1483
00:48:06,485 --> 00:48:08,720
is clearly the integration

1484
00:48:08,720 --> 00:48:10,114
happens to the
model, right?

1485
00:48:10,114 --> 00:48:11,299
Because you have all these

1486
00:48:11,299 --> 00:48:13,640
different mental models.

1487
00:48:13,640 --> 00:48:16,070
And to do program
evaluation,

1488
00:48:16,070 --> 00:48:17,689
you have to smoosh
them together in

1489
00:48:17,689 --> 00:48:18,980
some way or form through

1490
00:48:18,980 --> 00:48:19,999
the systems dynamics.

1491
00:48:19,999 --> 00:48:20,839
Well, and I really love

1492
00:48:20,839 --> 00:48:21,829
that because I think that,

1493
00:48:21,829 --> 00:48:24,035
that respects the
knowledge of,

1494
00:48:24,035 --> 00:48:26,180
of people in
social systems,

1495
00:48:26,180 --> 00:48:28,925
what system dynamics
sometimes does.

1496
00:48:28,925 --> 00:48:31,174
A great job.

1497
00:48:31,174 --> 00:48:32,600
I think that's
super awesome.

1498
00:48:32,600 --> 00:48:36,110
My Indian reveal
technical question is,

1499
00:48:36,110 --> 00:48:37,520
so atlas TI has

1500
00:48:37,520 --> 00:48:40,519
the capabilities to
define relationship.

1501
00:48:40,519 --> 00:48:41,930
You can say this
concept has

1502
00:48:41,930 --> 00:48:43,385
a relationship
with this concept.

1503
00:48:43,385 --> 00:48:44,810
Yet you seem to
have gone to

1504
00:48:44,810 --> 00:48:47,870
a table format
and using your,

1505
00:48:47,870 --> 00:48:50,779
your kind of
hand-drawn things.

1506
00:48:50,779 --> 00:48:52,519
Why that why did

1507
00:48:52,519 --> 00:48:53,810
you choose to
do it this way?

1508
00:48:53,810 --> 00:48:55,939
Yeah. So from what
I understand,

1509
00:48:55,939 --> 00:48:57,079
I a I played

1510
00:48:57,079 --> 00:48:58,820
around with that
feature and atlas TI,

1511
00:48:58,820 --> 00:49:00,320
but from what I understand,

1512
00:49:00,320 --> 00:49:03,410
and I've seen
some research use

1513
00:49:03,410 --> 00:49:06,680
similar features I
think, and NVivo.

1514
00:49:06,680 --> 00:49:10,745
But what I understand
those concepts link,

1515
00:49:10,745 --> 00:49:12,470
you can only link codes.

1516
00:49:12,470 --> 00:49:13,729
From what I understand is

1517
00:49:13,729 --> 00:49:15,214
that you can make codes,

1518
00:49:15,214 --> 00:49:17,300
which then
obviously codes are

1519
00:49:17,300 --> 00:49:19,489
tied to certain
sections of text.

1520
00:49:19,489 --> 00:49:22,489
But it was what

1521
00:49:22,489 --> 00:49:24,889
I was playing
around with that it

1522
00:49:24,889 --> 00:49:30,410
seemed to what I
was interested in,

1523
00:49:30,410 --> 00:49:33,379
I think was, um,

1524
00:49:33,379 --> 00:49:35,899
so linking codes like

1525
00:49:35,899 --> 00:49:38,150
that I felt was
interesting,

1526
00:49:38,150 --> 00:49:39,589
but it's a different
type of inquiry

1527
00:49:39,589 --> 00:49:40,609
from when I was doing,

1528
00:49:40,609 --> 00:49:43,189
I felt that the what

1529
00:49:43,189 --> 00:49:45,530
was missing in some of
this prior research

1530
00:49:45,530 --> 00:49:48,110
was kind of giving
credence to

1531
00:49:48,110 --> 00:49:49,280
the role of the modeler

1532
00:49:49,280 --> 00:49:51,169
in their interpretive
abilities.

1533
00:49:51,169 --> 00:49:53,480
And this is something
that I feel like the

1534
00:49:53,480 --> 00:49:54,769
reach the writing

1535
00:49:54,769 --> 00:49:56,209
on qualitative
research does

1536
00:49:56,209 --> 00:49:58,910
well and is sort of giving

1537
00:49:58,910 --> 00:50:03,020
respect or kind of
recognizing that the,

1538
00:50:03,020 --> 00:50:05,929
that the analyst
brings with them.

1539
00:50:05,929 --> 00:50:07,429
Some things that
are important

1540
00:50:07,429 --> 00:50:08,540
for interpretation.

1541
00:50:08,540 --> 00:50:13,174
And so for the way
I had, that's the,

1542
00:50:13,174 --> 00:50:16,940
this system setup is
that by looking at

1543
00:50:16,940 --> 00:50:21,004
a quotation and having the,

1544
00:50:21,004 --> 00:50:22,970
the source, you
know, so the,

1545
00:50:22,970 --> 00:50:24,139
the quotation number is

1546
00:50:24,139 --> 00:50:25,489
connected to a
certain quotation.

1547
00:50:25,489 --> 00:50:26,570
So it's maybe
a paragraph of

1548
00:50:26,570 --> 00:50:27,769
text if someone telling,

1549
00:50:27,769 --> 00:50:29,674
explaining
something, you know.

1550
00:50:29,674 --> 00:50:32,750
And so it's like
it's associated

1551
00:50:32,750 --> 00:50:34,295
with like a whole thought

1552
00:50:34,295 --> 00:50:36,050
or a collection
of thoughts.

1553
00:50:36,050 --> 00:50:37,430
And so within
that there can be

1554
00:50:37,430 --> 00:50:38,945
different things going on.

1555
00:50:38,945 --> 00:50:41,929
You know, there can
be different codes,

1556
00:50:41,929 --> 00:50:44,599
there can be different
levels of things.

1557
00:50:44,599 --> 00:50:45,799
And so what I wanted

1558
00:50:45,799 --> 00:50:47,060
to do was to be
able to create

1559
00:50:47,060 --> 00:50:50,420
a system where the
modeller can track, okay,

1560
00:50:50,420 --> 00:50:53,884
here's a meaningful
section of text and then

1561
00:50:53,884 --> 00:50:55,430
hear them and apply

1562
00:50:55,430 --> 00:50:57,080
their understanding and to

1563
00:50:57,080 --> 00:50:58,279
be able to see, okay.

1564
00:50:58,279 --> 00:50:59,119
Especially when
you're looking

1565
00:50:59,119 --> 00:51:01,730
at that implied
information,

1566
00:51:01,730 --> 00:51:02,960
if someone makes a like

1567
00:51:02,960 --> 00:51:04,460
an offhand reference,
for example,

1568
00:51:04,460 --> 00:51:06,080
to something being
a vicious cycle,

1569
00:51:06,080 --> 00:51:07,219
but then doesn't say

1570
00:51:07,219 --> 00:51:08,570
anything more about that.

1571
00:51:08,570 --> 00:51:10,939
You know, it wouldn't
necessarily come up if

1572
00:51:10,939 --> 00:51:13,985
all your coding
for is just,

1573
00:51:13,985 --> 00:51:15,845
you know,
individual concepts

1574
00:51:15,845 --> 00:51:17,150
or something like that.

1575
00:51:17,150 --> 00:51:18,980
So I wanted to be
able to see, okay,

1576
00:51:18,980 --> 00:51:20,270
to have the analyst

1577
00:51:20,270 --> 00:51:21,410
to be able to look at that,

1578
00:51:21,410 --> 00:51:22,849
look at the context and

1579
00:51:22,849 --> 00:51:24,559
then be able to
pull out, Oh,

1580
00:51:24,559 --> 00:51:26,689
this is what they mean
because I wanted to be

1581
00:51:26,689 --> 00:51:29,225
able to, you know,

1582
00:51:29,225 --> 00:51:33,140
rely on the, the
intelligence of the analyst

1583
00:51:33,140 --> 00:51:34,940
to be able to interpret

1584
00:51:34,940 --> 00:51:37,220
what the person's saying.

1585
00:51:37,220 --> 00:51:38,869
And so I think
there's a lot

1586
00:51:38,869 --> 00:51:41,149
of room for improvement.

1587
00:51:41,149 --> 00:51:42,050
And what I would love to

1588
00:51:42,050 --> 00:51:43,340
see is to be
able to leverage

1589
00:51:43,340 --> 00:51:46,190
the power of that software,

1590
00:51:46,190 --> 00:51:49,054
to be able to, you know,

1591
00:51:49,054 --> 00:51:50,870
do not have to do

1592
00:51:50,870 --> 00:51:52,309
the manual
spreadsheet like that

1593
00:51:52,309 --> 00:51:54,154
was kind of, you know,

1594
00:51:54,154 --> 00:51:56,360
it's a lot more clunky
than it needs to be if

1595
00:51:56,360 --> 00:51:57,679
the software would
allow you to

1596
00:51:57,679 --> 00:51:59,419
take a section
of text and say,

1597
00:51:59,419 --> 00:52:02,690
okay, heres the
causal structure

1598
00:52:02,690 --> 00:52:05,270
that is tied to this.

1599
00:52:05,270 --> 00:52:06,770
I think that
would be awesome.

1600
00:52:06,770 --> 00:52:08,900
But from what I understand,

1601
00:52:08,900 --> 00:52:10,610
the way codes, the

1602
00:52:10,610 --> 00:52:12,169
code connections are
built like that.

1603
00:52:12,169 --> 00:52:13,310
It's like this higher level

1604
00:52:13,310 --> 00:52:19,500
those relationships.
Thank you.

1605
00:52:22,630 --> 00:52:25,640
Have a couple of questions.

1606
00:52:25,640 --> 00:52:27,349
So first era and I just, I

1607
00:52:27,349 --> 00:52:28,430
just want to say
thank you very

1608
00:52:28,430 --> 00:52:29,509
much for the
presentation and

1609
00:52:29,509 --> 00:52:30,995
for the excellent work.

1610
00:52:30,995 --> 00:52:32,120
I also want to tell you

1611
00:52:32,120 --> 00:52:32,989
that this is
probably one of

1612
00:52:32,989 --> 00:52:35,119
the best dissertation
research

1613
00:52:35,119 --> 00:52:36,229
presentations
that i've seen.

1614
00:52:36,229 --> 00:52:38,689
So congratulations on doing

1615
00:52:38,689 --> 00:52:39,949
an excellent job all

1616
00:52:39,949 --> 00:52:42,004
around. I'm
super impressed.

1617
00:52:42,004 --> 00:52:43,669
I have a couple of

1618
00:52:43,669 --> 00:52:45,560
different kinds
of questions

1619
00:52:45,560 --> 00:52:49,070
related to the program
evaluation piece,

1620
00:52:49,070 --> 00:52:50,659
the program's theory
piece of things being

1621
00:52:50,659 --> 00:52:53,479
the program evaluator
of the group.

1622
00:52:53,479 --> 00:52:55,609
And one of the
questions I want to

1623
00:52:55,609 --> 00:52:58,280
ask you about
is this idea of

1624
00:52:58,280 --> 00:53:02,870
where this approach
to developing

1625
00:53:02,870 --> 00:53:05,030
diagrams and to
developing program theory

1626
00:53:05,030 --> 00:53:07,870
fits within the idea of,

1627
00:53:07,870 --> 00:53:10,204
of how generally program
theory is developed.

1628
00:53:10,204 --> 00:53:11,659
Because I think that what

1629
00:53:11,659 --> 00:53:12,950
you're describing here is

1630
00:53:12,950 --> 00:53:15,604
an incredibly
important approach to

1631
00:53:15,604 --> 00:53:16,940
making explicit a lot

1632
00:53:16,940 --> 00:53:18,320
of assumptions and
mental models,

1633
00:53:18,320 --> 00:53:19,339
both about problems and

1634
00:53:19,339 --> 00:53:22,385
interventions
that as you know,

1635
00:53:22,385 --> 00:53:23,810
we've talked about
are typically

1636
00:53:23,810 --> 00:53:25,310
not made explicit by people

1637
00:53:25,310 --> 00:53:26,660
who are engaged in a lot of

1638
00:53:26,660 --> 00:53:28,589
public and social
service work.

1639
00:53:28,589 --> 00:53:30,260
That people don't
have the time

1640
00:53:30,260 --> 00:53:31,730
or don't take
the time to make

1641
00:53:31,730 --> 00:53:33,169
explicit a lot of
these assumptions

1642
00:53:33,169 --> 00:53:33,889
about what they're doing

1643
00:53:33,889 --> 00:53:35,045
and how they're doing it.

1644
00:53:35,045 --> 00:53:37,370
But in the developmental
program theory from,

1645
00:53:37,370 --> 00:53:38,870
from a program
evaluation model

1646
00:53:38,870 --> 00:53:43,265
is that asking
stakeholders in

1647
00:53:43,265 --> 00:53:45,200
programs to make

1648
00:53:45,200 --> 00:53:48,380
explicit their
assumptions and

1649
00:53:48,380 --> 00:53:51,560
their models
about the program

1650
00:53:51,560 --> 00:53:52,160
and what they're doing and

1651
00:53:52,160 --> 00:53:52,670
how they're doing it

1652
00:53:52,670 --> 00:53:54,109
as incredibly
important piece

1653
00:53:54,109 --> 00:53:55,010
of program theory and

1654
00:53:55,010 --> 00:53:56,270
developing program theory.

1655
00:53:56,270 --> 00:53:58,445
But so also is

1656
00:53:58,445 --> 00:54:00,349
that juxtaposition
with what people

1657
00:54:00,349 --> 00:54:01,279
think is happening or

1658
00:54:01,279 --> 00:54:02,209
where they assumed to be

1659
00:54:02,209 --> 00:54:03,259
happening and what their

1660
00:54:03,259 --> 00:54:04,594
experience tells
them as happening.

1661
00:54:04,594 --> 00:54:07,385
With existing research that

1662
00:54:07,385 --> 00:54:09,169
has been done about
larger issues in terms of

1663
00:54:09,169 --> 00:54:10,489
how we understand
the problem being

1664
00:54:10,489 --> 00:54:11,839
addressed and how
other kinds of

1665
00:54:11,839 --> 00:54:13,429
interventions that
may be similar

1666
00:54:13,429 --> 00:54:15,110
have addresses
these problems.

1667
00:54:15,110 --> 00:54:16,250
And so I'm wondering within

1668
00:54:16,250 --> 00:54:17,750
your model here about

1669
00:54:17,750 --> 00:54:19,160
how this helps to make

1670
00:54:19,160 --> 00:54:20,285
the stakeholder positions

1671
00:54:20,285 --> 00:54:21,800
so much more explicit.

1672
00:54:21,800 --> 00:54:23,119
How is that then

1673
00:54:23,119 --> 00:54:25,369
juxtaposed with
existing research that,

1674
00:54:25,369 --> 00:54:28,490
that Ben is juxtaposed

1675
00:54:28,490 --> 00:54:30,079
together to develop a
program theory that

1676
00:54:30,079 --> 00:54:31,595
is that has been formed by

1677
00:54:31,595 --> 00:54:33,440
both the larger issues as

1678
00:54:33,440 --> 00:54:34,820
well as those
localized issues as

1679
00:54:34,820 --> 00:54:37,369
you've spoken so
eloquently about?

1680
00:54:37,369 --> 00:54:39,109
Yeah. No, that's
a great question.

1681
00:54:39,109 --> 00:54:41,360
Let me share again
this slide that I

1682
00:54:41,360 --> 00:54:42,979
didn't share
early and that I

1683
00:54:42,979 --> 00:54:45,630
think speaks to this.

1684
00:54:50,850 --> 00:54:53,980
So this is from

1685
00:54:53,980 --> 00:54:58,270
a proposal for a K
award that I have.

1686
00:54:58,270 --> 00:55:00,624
I'm still waiting
to hear background,

1687
00:55:00,624 --> 00:55:02,080
so I don't know if
I'm going to do

1688
00:55:02,080 --> 00:55:03,730
this research, but it's,

1689
00:55:03,730 --> 00:55:06,250
the idea here is
that I'm using

1690
00:55:06,250 --> 00:55:09,249
this coding analysis
to describe

1691
00:55:09,249 --> 00:55:12,685
stakeholder mental
models and then

1692
00:55:12,685 --> 00:55:16,644
integrating those
diagrams with

1693
00:55:16,644 --> 00:55:18,805
different forms of data.

1694
00:55:18,805 --> 00:55:21,369
So I think a
strength of this,

1695
00:55:21,369 --> 00:55:23,799
of the causal loop
diagramming form

1696
00:55:23,799 --> 00:55:25,375
in particular
is that because

1697
00:55:25,375 --> 00:55:28,809
it's ultimately
kinda decomposable

1698
00:55:28,809 --> 00:55:30,939
to spreadsheet form, right?

1699
00:55:30,939 --> 00:55:33,460
And so it's, it's, it's,

1700
00:55:33,460 --> 00:55:36,849
you can, because it

1701
00:55:36,849 --> 00:55:38,619
provides a sort of
common language.

1702
00:55:38,619 --> 00:55:40,600
And so in this example,

1703
00:55:40,600 --> 00:55:43,689
we're integrating
were I proposed to

1704
00:55:43,689 --> 00:55:45,340
map stakeholder
perspectives

1705
00:55:45,340 --> 00:55:47,320
from patients
and clinicians.

1706
00:55:47,320 --> 00:55:50,169
This is an implementation
science study.

1707
00:55:50,169 --> 00:55:52,239
And then
integrating it with

1708
00:55:52,239 --> 00:55:54,624
meeting transcripts
that they have.

1709
00:55:54,624 --> 00:55:56,200
Clinic contact
logs, which is

1710
00:55:56,200 --> 00:55:58,240
another type of
qualitative information.

1711
00:55:58,240 --> 00:56:00,294
Scientific
literature as well,

1712
00:56:00,294 --> 00:56:02,229
participatory review,

1713
00:56:02,229 --> 00:56:04,194
you also engaging
people in it,

1714
00:56:04,194 --> 00:56:05,980
and then outcomes
data as well.

1715
00:56:05,980 --> 00:56:07,930
So using even the
quantitative data

1716
00:56:07,930 --> 00:56:09,594
to help form,

1717
00:56:09,594 --> 00:56:11,109
to help shape okay,

1718
00:56:11,109 --> 00:56:14,080
where, which
parts of a system

1719
00:56:14,080 --> 00:56:17,329
is maybe more relevant.

1720
00:56:17,329 --> 00:56:19,234
So I see a lot of,

1721
00:56:19,234 --> 00:56:20,839
this is where I think
there's a lot of

1722
00:56:20,839 --> 00:56:25,610
promising research in this,

1723
00:56:25,610 --> 00:56:28,385
for this research proposal
and talking about

1724
00:56:28,385 --> 00:56:31,384
customizing evidence-based
interventions

1725
00:56:31,384 --> 00:56:32,750
using these diagrams.

1726
00:56:32,750 --> 00:56:34,699
But I think the,
the main idea

1727
00:56:34,699 --> 00:56:36,184
being that, you know,

1728
00:56:36,184 --> 00:56:38,554
I would love for
there to be more

1729
00:56:38,554 --> 00:56:40,159
and for me that I'd

1730
00:56:40,159 --> 00:56:41,539
love to be involved
in this research,

1731
00:56:41,539 --> 00:56:43,549
but for there to be
more research on how

1732
00:56:43,549 --> 00:56:48,530
to translate different
types of information.

1733
00:56:48,530 --> 00:56:50,644
So information
from literature,

1734
00:56:50,644 --> 00:56:52,790
for example, into,
into diagrams.

1735
00:56:52,790 --> 00:56:54,575
And so there's,
there's some research

1736
00:56:54,575 --> 00:56:58,880
out there already on.

1737
00:56:58,880 --> 00:57:03,740
I'm using either
automated AI type methods

1738
00:57:03,740 --> 00:57:06,695
to kind of understand

1739
00:57:06,695 --> 00:57:07,880
causal information or

1740
00:57:07,880 --> 00:57:09,754
identify causal
information.

1741
00:57:09,754 --> 00:57:11,959
I think there's, you know,

1742
00:57:11,959 --> 00:57:13,339
there's maybe some
pros and cons to

1743
00:57:13,339 --> 00:57:15,125
that line of inquiry,
but there's,

1744
00:57:15,125 --> 00:57:17,419
there's some research
out there about how to

1745
00:57:17,419 --> 00:57:20,330
use how to use

1746
00:57:20,330 --> 00:57:22,774
causal loop diagrams in

1747
00:57:22,774 --> 00:57:26,570
systematic reviews of
scientific literature.

1748
00:57:26,570 --> 00:57:28,159
And so I think
there's, because

1749
00:57:28,159 --> 00:57:30,154
these diagrams are,

1750
00:57:30,154 --> 00:57:32,209
it can kinda
put information

1751
00:57:32,209 --> 00:57:34,009
on the same plane.

1752
00:57:34,009 --> 00:57:36,289
E. I think there's
a lot of room for

1753
00:57:36,289 --> 00:57:38,210
development of those
methods as far as

1754
00:57:38,210 --> 00:57:40,040
like what that might
actually look like

1755
00:57:40,040 --> 00:57:43,519
in a evaluation, I think,

1756
00:57:43,519 --> 00:57:47,015
you know, I think
we would, it would,

1757
00:57:47,015 --> 00:57:48,919
we'd have to kind of

1758
00:57:48,919 --> 00:57:53,180
balance the use of
methods like this

1759
00:57:53,180 --> 00:57:56,479
for sort of being
systematic and

1760
00:57:56,479 --> 00:57:58,729
being maybe a
bit more precise

1761
00:57:58,729 --> 00:58:01,655
with other methods
to meet other needs.

1762
00:58:01,655 --> 00:58:03,530
Because maybe it would be

1763
00:58:03,530 --> 00:58:05,390
blended with some
group modelling to

1764
00:58:05,390 --> 00:58:07,309
kind of engage people
and make them feel

1765
00:58:07,309 --> 00:58:09,649
more ownership of it.

1766
00:58:09,649 --> 00:58:12,604
Or maybe there's, you know,

1767
00:58:12,604 --> 00:58:14,989
you can incorporate
components

1768
00:58:14,989 --> 00:58:16,459
of lit review that then are

1769
00:58:16,459 --> 00:58:17,900
summarized in certain areas

1770
00:58:17,900 --> 00:58:18,679
and you could be able to

1771
00:58:18,679 --> 00:58:23,555
trace where different
information comes from.

1772
00:58:23,555 --> 00:58:25,580
So yeah, I don't
think it needs

1773
00:58:25,580 --> 00:58:27,020
to take obviously
take place

1774
00:58:27,020 --> 00:58:30,319
within the larger
question of OK,

1775
00:58:30,319 --> 00:58:31,969
what's right for
this argument?

1776
00:58:31,969 --> 00:58:32,449
You know, what's right

1777
00:58:32,449 --> 00:58:33,679
for that
organization, right?

1778
00:58:33,679 --> 00:58:35,120
Or what's right for?

1779
00:58:35,120 --> 00:58:37,130
What are your, your
evaluation questions,

1780
00:58:37,130 --> 00:58:38,480
what are your
research questions?

1781
00:58:38,480 --> 00:58:40,969
And so I'm hoping this
research kinda fits

1782
00:58:40,969 --> 00:58:44,225
within adding
to the toolbox.

1783
00:58:44,225 --> 00:58:45,664
But it doesn't necessary

1784
00:58:45,664 --> 00:58:47,270
definitely doesn't replace

1785
00:58:47,270 --> 00:58:49,970
the traditions
and the systems

1786
00:58:49,970 --> 00:58:51,695
that are already there.

1787
00:58:51,695 --> 00:58:53,509
Thank you. That's
super helpful.

1788
00:58:53,509 --> 00:58:54,290
And actually, I
think what you

1789
00:58:54,290 --> 00:58:55,550
did was just lead into

1790
00:58:55,550 --> 00:58:56,855
my next question which is

1791
00:58:56,855 --> 00:58:57,770
and I don't think I

1792
00:58:57,770 --> 00:58:58,790
could like
Melinda question.

1793
00:58:58,790 --> 00:59:00,170
It's it's a little
bit you know,

1794
00:59:00,170 --> 00:59:01,730
it's a little bit
outside of the scope

1795
00:59:01,730 --> 00:59:04,610
of what you've done here,
which is fantastic.

1796
00:59:04,610 --> 00:59:07,070
But whenever I kind of
thought you've given

1797
00:59:07,070 --> 00:59:10,249
to implementing that's not

1798
00:59:10,249 --> 00:59:11,299
just in a research context,

1799
00:59:11,299 --> 00:59:13,159
but actually in practice
in terms of working

1800
00:59:13,159 --> 00:59:16,610
with public and
non-profit organizations

1801
00:59:16,610 --> 00:59:18,049
to implement more
of this approach.

1802
00:59:18,049 --> 00:59:19,099
You know, at the
beginning you talk

1803
00:59:19,099 --> 00:59:20,929
about this idea

1804
00:59:20,929 --> 00:59:22,310
of introducing
more complexity

1805
00:59:22,310 --> 00:59:23,929
into programs, into
program theory.

1806
00:59:23,929 --> 00:59:25,579
And for someone like me

1807
00:59:25,579 --> 00:59:27,800
who works a lot with
non-profits and

1808
00:59:27,800 --> 00:59:30,349
does a lot of work
around trying

1809
00:59:30,349 --> 00:59:31,880
to introduce more
evaluative thinking

1810
00:59:31,880 --> 00:59:33,409
into organizations.

1811
00:59:33,409 --> 00:59:36,680
That's hard.
And introducing

1812
00:59:36,680 --> 00:59:37,909
even more complexity into

1813
00:59:37,909 --> 00:59:39,289
the Universe in a lot

1814
00:59:39,289 --> 00:59:41,135
of these non-profits and
public organizations

1815
00:59:41,135 --> 00:59:42,200
exist in a universe

1816
00:59:42,200 --> 00:59:46,849
where the easiest answer
is that the best?

1817
00:59:46,849 --> 00:59:48,919
Like people like
simple answers,

1818
00:59:48,919 --> 00:59:50,149
especially when it comes

1819
00:59:50,149 --> 00:59:51,965
to entrenched
social problems

1820
00:59:51,965 --> 00:59:53,660
and actually investing
in addressing

1821
00:59:53,660 --> 00:59:55,414
those kinds of issues
is incredibly hard.

1822
00:59:55,414 --> 00:59:56,570
Everybody want
simple answers and

1823
00:59:56,570 --> 00:59:57,769
so I'm just curious

1824
00:59:57,769 --> 00:59:58,969
what kind of
thinking you've

1825
00:59:58,969 --> 01:00:00,110
done in terms of like,

1826
01:00:00,110 --> 01:00:02,885
how would you bring this
approach to funders?

1827
01:00:02,885 --> 01:00:04,039
Like how would you
bring us approach

1828
01:00:04,039 --> 01:00:06,075
to organizations to

1829
01:00:06,075 --> 01:00:07,880
add even more complexity
into something

1830
01:00:07,880 --> 01:00:10,100
that they already
find kind of complex?

1831
01:00:10,100 --> 01:00:12,020
Yeah. No, it's a
good question.

1832
01:00:12,020 --> 01:00:12,979
Um, I think there's

1833
01:00:12,979 --> 01:00:14,929
a few different
answers here.

1834
01:00:14,929 --> 01:00:21,410
One is that there's
an existing practice.

1835
01:00:21,410 --> 01:00:23,524
I feel the practice
in literature around

1836
01:00:23,524 --> 01:00:24,709
group model building that I

1837
01:00:24,709 --> 01:00:26,585
think does a good job of.

1838
01:00:26,585 --> 01:00:29,479
It. You know, coming up

1839
01:00:29,479 --> 01:00:31,939
with ways to engage groups,

1840
01:00:31,939 --> 01:00:34,115
especially coalitions
of groups,

1841
01:00:34,115 --> 01:00:38,000
in these modeling
exercises.

1842
01:00:38,000 --> 01:00:39,829
And where a goal is

1843
01:00:39,829 --> 01:00:42,230
not just coming up
with a good model,

1844
01:00:42,230 --> 01:00:46,009
but it's having this sort

1845
01:00:46,009 --> 01:00:47,629
of shared group process and

1846
01:00:47,629 --> 01:00:49,774
a learning process where in

1847
01:00:49,774 --> 01:00:52,250
engaging in this
modelling and having

1848
01:00:52,250 --> 01:00:55,880
a a kind of artifact model

1849
01:00:55,880 --> 01:00:59,179
there to reference
is a way to

1850
01:00:59,179 --> 01:01:00,679
build shared
understanding within

1851
01:01:00,679 --> 01:01:02,779
a group and, and
that kind of stuff.

1852
01:01:02,779 --> 01:01:05,104
And so I think there's,

1853
01:01:05,104 --> 01:01:07,100
you know, and but yeah,

1854
01:01:07,100 --> 01:01:09,950
there's definitely
a high investment

1855
01:01:09,950 --> 01:01:12,559
of time and energy
and all of that.

1856
01:01:12,559 --> 01:01:14,555
And I think that's

1857
01:01:14,555 --> 01:01:17,254
where I think some of this,

1858
01:01:17,254 --> 01:01:18,649
where some of the
motivation for

1859
01:01:18,649 --> 01:01:20,615
my research comes in is

1860
01:01:20,615 --> 01:01:24,650
wanting to bring in
people into that process.

1861
01:01:24,650 --> 01:01:26,810
But also recognize that,

1862
01:01:26,810 --> 01:01:28,144
you know, people don't,

1863
01:01:28,144 --> 01:01:29,540
can't necessarily
show up for

1864
01:01:29,540 --> 01:01:31,730
a series of 12
modelling sessions,

1865
01:01:31,730 --> 01:01:33,409
synchronous, you know,

1866
01:01:33,409 --> 01:01:35,675
in person, all of that.

1867
01:01:35,675 --> 01:01:38,269
And so it's, you know,

1868
01:01:38,269 --> 01:01:41,989
I think there's different
methods out there,

1869
01:01:41,989 --> 01:01:46,504
but I think there's I
think there's also,

1870
01:01:46,504 --> 01:01:49,474
I think groom for
development of,

1871
01:01:49,474 --> 01:01:52,039
um, you know, I see some

1872
01:01:52,039 --> 01:01:54,380
of this as maybe
under the hood too.

1873
01:01:54,380 --> 01:01:58,850
Like where people
can, you know,

1874
01:01:58,850 --> 01:02:00,469
not everybody in

1875
01:02:00,469 --> 01:02:02,329
an organization
necessarily has to

1876
01:02:02,329 --> 01:02:05,900
be involved in
the same way,

1877
01:02:05,900 --> 01:02:06,919
which I think is something

1878
01:02:06,919 --> 01:02:09,289
that traditionally a
group model building.

1879
01:02:09,289 --> 01:02:09,950
It's like, okay, let's get

1880
01:02:09,950 --> 01:02:10,985
everyone in the room.

1881
01:02:10,985 --> 01:02:12,349
We have a white
board handhold,

1882
01:02:12,349 --> 01:02:14,344
kind of do this together.

1883
01:02:14,344 --> 01:02:17,539
Whereas I think with,

1884
01:02:17,539 --> 01:02:19,235
with this you can

1885
01:02:19,235 --> 01:02:21,830
do some short
interviews with people.

1886
01:02:21,830 --> 01:02:22,820
You can think of

1887
01:02:22,820 --> 01:02:24,290
other ways of
engaging people.

1888
01:02:24,290 --> 01:02:26,315
Maybe some people
are just involved.

1889
01:02:26,315 --> 01:02:28,850
The final review stage
or maybe sunlight,

1890
01:02:28,850 --> 01:02:30,139
you know, so I think
there's different

1891
01:02:30,139 --> 01:02:32,015
ways you can approach it.

1892
01:02:32,015 --> 01:02:34,160
As far as the
broader question

1893
01:02:34,160 --> 01:02:36,080
of People don't
want complexity.

1894
01:02:36,080 --> 01:02:38,570
I mean, I think there's,

1895
01:02:38,570 --> 01:02:41,209
you know, that's
that's true.

1896
01:02:41,209 --> 01:02:44,585
But I feel HC there's
also, you know,

1897
01:02:44,585 --> 01:02:45,800
people don't
want complexity

1898
01:02:45,800 --> 01:02:46,760
until they
realized that they

1899
01:02:46,760 --> 01:02:49,100
needed it from the
start, you know,

1900
01:02:49,100 --> 01:02:51,979
and that hopefully
there can

1901
01:02:51,979 --> 01:02:55,190
be a shift sort of more

1902
01:02:55,190 --> 01:02:58,249
broadly towards
understanding the value

1903
01:02:58,249 --> 01:03:01,519
of really kind of
understanding the system.

1904
01:03:01,519 --> 01:03:03,140
Doing a needs assessment of

1905
01:03:03,140 --> 01:03:05,344
where what you do,

1906
01:03:05,344 --> 01:03:06,814
what your problem is,

1907
01:03:06,814 --> 01:03:10,264
where your intervention
is going to come in

1908
01:03:10,264 --> 01:03:12,620
and then trying to be

1909
01:03:12,620 --> 01:03:14,659
mindful of that
relationship.

1910
01:03:14,659 --> 01:03:16,324
And that doesn't have to be

1911
01:03:16,324 --> 01:03:18,319
the sprawling
complex model.

1912
01:03:18,319 --> 01:03:19,745
It can be pretty simple and

1913
01:03:19,745 --> 01:03:22,369
that sort of basic
exercise of understanding.

1914
01:03:22,369 --> 01:03:24,320
Okay, what problem
we're trying to solve.

1915
01:03:24,320 --> 01:03:25,864
What are the key
dynamics there?

1916
01:03:25,864 --> 01:03:27,680
What buttons are we
trying to push by

1917
01:03:27,680 --> 01:03:29,780
doing this? I think
there's value.

1918
01:03:29,780 --> 01:03:32,600
A little bit of that
can go a long way in

1919
01:03:32,600 --> 01:03:34,190
helping people understand

1920
01:03:34,190 --> 01:03:36,215
what goes wrong, leader.

1921
01:03:36,215 --> 01:03:37,699
And then one
final thought on

1922
01:03:37,699 --> 01:03:39,229
that is I think
there's a lot of

1923
01:03:39,229 --> 01:03:43,955
interest were some at
least within funders,

1924
01:03:43,955 --> 01:03:45,260
you know, a lot
of the when I

1925
01:03:45,260 --> 01:03:47,284
was doing this
research, I found.

1926
01:03:47,284 --> 01:03:49,639
A lot of guides
to this sort of

1927
01:03:49,639 --> 01:03:51,859
diagramming or guides
to system dynamics.

1928
01:03:51,859 --> 01:03:53,030
Or there's a lot
of pleasure like

1929
01:03:53,030 --> 01:03:56,450
USA ID or different folks
have been, I think,

1930
01:03:56,450 --> 01:03:59,195
recognize the value
in some of this

1931
01:03:59,195 --> 01:04:02,690
and try to put some
of this out there as,

1932
01:04:02,690 --> 01:04:04,745
you know, here's something
that could help.

1933
01:04:04,745 --> 01:04:06,229
I think there's
a long way to

1934
01:04:06,229 --> 01:04:07,909
go and helping
kind of integrate

1935
01:04:07,909 --> 01:04:09,049
the best practices from

1936
01:04:09,049 --> 01:04:11,330
system dynamics or the
fields and kind of

1937
01:04:11,330 --> 01:04:15,035
really design strategies
that are good.

1938
01:04:15,035 --> 01:04:16,309
You know, instead
of just borrowed

1939
01:04:16,309 --> 01:04:18,980
from modeling
to really have

1940
01:04:18,980 --> 01:04:20,810
things that are
customized to

1941
01:04:20,810 --> 01:04:23,330
the needs of evaluators
and evaluation.

1942
01:04:23,330 --> 01:04:25,489
So I see this as a
starting point that,

1943
01:04:25,489 --> 01:04:27,019
you know, there's
a lot of ways,

1944
01:04:27,019 --> 01:04:28,549
yeah, a long way to go to

1945
01:04:28,549 --> 01:04:31,279
make things
actually usable.

1946
01:04:31,279 --> 01:04:36,320
And yeah, so thanks.
That's a good answer.

1947
01:04:36,320 --> 01:04:39,259
I might just add that while

1948
01:04:39,259 --> 01:04:41,810
people want to be sure

1949
01:04:41,810 --> 01:04:43,865
that they are not
overlooking things that

1950
01:04:43,865 --> 01:04:45,350
complexity perspective

1951
01:04:45,350 --> 01:04:47,360
would help, help them with.

1952
01:04:47,360 --> 01:04:50,119
They also would
love to know

1953
01:04:50,119 --> 01:04:52,699
that you can use
these methods to

1954
01:04:52,699 --> 01:04:55,370
make the whole process
manageable to address

1955
01:04:55,370 --> 01:04:59,195
complexity without
having to use tricky,

1956
01:04:59,195 --> 01:05:00,829
hard to understand
things at

1957
01:05:00,829 --> 01:05:02,449
the basic language
is pretty

1958
01:05:02,449 --> 01:05:03,890
straightforward
and by using

1959
01:05:03,890 --> 01:05:05,824
it in a thoughtful way,

1960
01:05:05,824 --> 01:05:07,910
you are addressing
the complexity,

1961
01:05:07,910 --> 01:05:09,770
but in a way that
we can think

1962
01:05:09,770 --> 01:05:12,020
about in process
and digest.

1963
01:05:12,020 --> 01:05:13,220
So and I think you

1964
01:05:13,220 --> 01:05:15,719
said that your
presentation as well.

1965
01:05:16,420 --> 01:05:18,575
I'd like to open it up for

1966
01:05:18,575 --> 01:05:19,250
questions from

1967
01:05:19,250 --> 01:05:20,690
everyone and
committee members.

1968
01:05:20,690 --> 01:05:21,949
Feel free to
continue asking

1969
01:05:21,949 --> 01:05:23,959
questions as well.

1970
01:05:23,959 --> 01:05:26,300
And I'll just go ahead

1971
01:05:26,300 --> 01:05:28,800
and take the mic if
you have a question.

1972
01:05:29,770 --> 01:05:32,119
Okay. Thank you.
Minus took quarter.

1973
01:05:32,119 --> 01:05:33,889
Thank you. Brought
her presentation.

1974
01:05:33,889 --> 01:05:36,890
There are really
interesting

1975
01:05:36,890 --> 01:05:39,515
and a totally new area

1976
01:05:39,515 --> 01:05:41,959
of study ever seen before.

1977
01:05:41,959 --> 01:05:44,839
But my first question was,

1978
01:05:44,839 --> 01:05:47,209
have you applied this

1979
01:05:47,209 --> 01:05:53,670
to a needs discovery
for marketing?

1980
01:05:54,640 --> 01:05:58,144
For marketing? I
personally have.

1981
01:05:58,144 --> 01:06:01,205
I feel like
there's, you know,

1982
01:06:01,205 --> 01:06:03,859
the field of system science

1983
01:06:03,859 --> 01:06:06,724
and causal loop diagramming
and all of that.

1984
01:06:06,724 --> 01:06:09,350
In generally,
people use things

1985
01:06:09,350 --> 01:06:12,380
for all different areas.

1986
01:06:12,380 --> 01:06:13,804
And so I'm sure
someone has done it.

1987
01:06:13,804 --> 01:06:14,959
I know there's a lot of

1988
01:06:14,959 --> 01:06:16,399
people who apply these

1989
01:06:16,399 --> 01:06:18,319
sort of modeling
with economics

1990
01:06:18,319 --> 01:06:19,490
and that kind of stuff.

1991
01:06:19,490 --> 01:06:21,349
But I personally haven't

1992
01:06:21,349 --> 01:06:24,090
gone in that direction.

1993
01:06:25,390 --> 01:06:28,099
Okay. Thank you. What might

1994
01:06:28,099 --> 01:06:30,359
be profitable now?

1995
01:06:30,880 --> 01:06:36,139
Can be high.

1996
01:06:36,139 --> 01:06:38,885
Erin, nice to meet you.

1997
01:06:38,885 --> 01:06:40,940
My name is Kathleen conti.

1998
01:06:40,940 --> 01:06:43,175
I do as soft
systems research,

1999
01:06:43,175 --> 01:06:44,900
systems methodologies

2000
01:06:44,900 --> 01:06:46,730
and participatory methods

2001
01:06:46,730 --> 01:06:48,784
for public
health. Go great.

2002
01:06:48,784 --> 01:06:51,170
And I loved your
presentation.

2003
01:06:51,170 --> 01:06:53,690
I'm, it's a fabulous work

2004
01:06:53,690 --> 01:06:55,025
and I hope that

2005
01:06:55,025 --> 01:06:56,899
there's an opportunity
to use it.

2006
01:06:56,899 --> 01:06:59,449
Be thinking about that
as I move forward with

2007
01:06:59,449 --> 01:07:00,830
my research and planning

2008
01:07:00,830 --> 01:07:03,050
and hopefully we
can connect on it.

2009
01:07:03,050 --> 01:07:06,920
My question for
you stems from

2010
01:07:06,920 --> 01:07:10,834
my experience doing
group model building and

2011
01:07:10,834 --> 01:07:12,215
working with mental models

2012
01:07:12,215 --> 01:07:14,465
and thinking
about how one of

2013
01:07:14,465 --> 01:07:17,030
the really important
elements of

2014
01:07:17,030 --> 01:07:20,480
that work is
surfacing assumptions

2015
01:07:20,480 --> 01:07:24,050
and recognizing and
resolving where there's

2016
01:07:24,050 --> 01:07:26,599
disagreement or
conflict and

2017
01:07:26,599 --> 01:07:29,240
how people see a
problem or an issue.

2018
01:07:29,240 --> 01:07:30,980
And I think this method

2019
01:07:30,980 --> 01:07:33,170
is really interesting in

2020
01:07:33,170 --> 01:07:34,850
that there are a lot
of opportunities here

2021
01:07:34,850 --> 01:07:36,589
to really go in
depth and away,

2022
01:07:36,589 --> 01:07:37,850
probably couldn't and

2023
01:07:37,850 --> 01:07:40,625
that group model
building situation.

2024
01:07:40,625 --> 01:07:42,050
But I also see that

2025
01:07:42,050 --> 01:07:43,669
there's some
opportunity there for

2026
01:07:43,669 --> 01:07:45,020
maybe identifying some of

2027
01:07:45,020 --> 01:07:46,849
those friction points.

2028
01:07:46,849 --> 01:07:47,929
And so I wondered if you

2029
01:07:47,929 --> 01:07:49,970
had been doing and
he's thinking or I've

2030
01:07:49,970 --> 01:07:52,460
done any work around how to

2031
01:07:52,460 --> 01:07:56,015
pull these disparate
models together,

2032
01:07:56,015 --> 01:07:57,560
maybe to identify some of

2033
01:07:57,560 --> 01:07:59,240
those points and how

2034
01:07:59,240 --> 01:08:01,085
you might use
that information.

2035
01:08:01,085 --> 01:08:02,960
Yeah. I think it's
a great question,

2036
01:08:02,960 --> 01:08:04,099
so yeah, I mean,

2037
01:08:04,099 --> 01:08:06,109
there's definitely
trade-offs

2038
01:08:06,109 --> 01:08:08,119
between using
this approach,

2039
01:08:08,119 --> 01:08:10,610
for example, and using
group model building.

2040
01:08:10,610 --> 01:08:12,379
There's different
things that

2041
01:08:12,379 --> 01:08:14,225
the two strategies
are good at.

2042
01:08:14,225 --> 01:08:16,849
I mean, I think
there's a lot of room

2043
01:08:16,849 --> 01:08:20,900
for combining them
to using this,

2044
01:08:20,900 --> 01:08:22,760
this sort of incorporating

2045
01:08:22,760 --> 01:08:24,215
from individual interviews

2046
01:08:24,215 --> 01:08:26,090
either as a precursor

2047
01:08:26,090 --> 01:08:27,844
to group model building,

2048
01:08:27,844 --> 01:08:28,820
where as a way

2049
01:08:28,820 --> 01:08:30,560
to incorporate
perspectives from

2050
01:08:30,560 --> 01:08:32,689
people for people who

2051
01:08:32,689 --> 01:08:34,174
can't be in the room

2052
01:08:34,174 --> 01:08:36,510
with group model building.

2053
01:08:36,760 --> 01:08:39,109
And, but yeah, I

2054
01:08:39,109 --> 01:08:40,685
think with group model
building the string,

2055
01:08:40,685 --> 01:08:41,810
one of this main
strengths is

2056
01:08:41,810 --> 01:08:43,549
that dynamic kind of,

2057
01:08:43,549 --> 01:08:46,370
you know, let's
the, the, the,

2058
01:08:46,370 --> 01:08:48,049
the cycles of iteration or

2059
01:08:48,049 --> 01:08:49,849
back and forth
is much tighter

2060
01:08:49,849 --> 01:08:51,109
when you have someone

2061
01:08:51,109 --> 01:08:52,340
in front of you
and can have

2062
01:08:52,340 --> 01:08:53,690
a whole conversation right

2063
01:08:53,690 --> 01:08:56,359
there with within
that group.

2064
01:08:56,359 --> 01:08:58,894
And so what I
tried to do is

2065
01:08:58,894 --> 01:09:01,969
capture some of
the strengths of

2066
01:09:01,969 --> 01:09:04,459
that and try to codify
some of it into

2067
01:09:04,459 --> 01:09:05,570
some of that guidance

2068
01:09:05,570 --> 01:09:07,429
for conducting
the interview.

2069
01:09:07,429 --> 01:09:09,739
So try to you I have

2070
01:09:09,739 --> 01:09:12,080
the interviewer
B, you know,

2071
01:09:12,080 --> 01:09:12,859
because you don't have

2072
01:09:12,859 --> 01:09:14,359
that group providing that

2073
01:09:14,359 --> 01:09:16,100
those checks and balances

2074
01:09:16,100 --> 01:09:17,795
are trying to get at that,

2075
01:09:17,795 --> 01:09:21,050
that the main idea,
the inner mark,

2076
01:09:21,050 --> 01:09:23,509
there's more onus on
the interviewer to

2077
01:09:23,509 --> 01:09:27,185
provide that that function.

2078
01:09:27,185 --> 01:09:29,210
And so I think it's

2079
01:09:29,210 --> 01:09:30,980
that's where I
think it's it's

2080
01:09:30,980 --> 01:09:34,940
definitely a
it's a practice

2081
01:09:34,940 --> 01:09:36,050
that I think there's
a lot of room

2082
01:09:36,050 --> 01:09:37,250
for development on,

2083
01:09:37,250 --> 01:09:40,969
you know, how do
you actually get at

2084
01:09:40,969 --> 01:09:42,514
the bottom of the

2085
01:09:42,514 --> 01:09:44,989
person's mental model
in an interview,

2086
01:09:44,989 --> 01:09:47,044
you have to, I think ice,

2087
01:09:47,044 --> 01:09:48,679
this research
kind of starts

2088
01:09:48,679 --> 01:09:50,540
that conversation of Here's

2089
01:09:50,540 --> 01:09:51,619
some things to think about,

2090
01:09:51,619 --> 01:09:52,729
but I think
there's room for a

2091
01:09:52,729 --> 01:09:54,725
lot more development and

2092
01:09:54,725 --> 01:09:56,420
in developing
kind of a craft

2093
01:09:56,420 --> 01:09:58,190
around, around that.

2094
01:09:58,190 --> 01:10:00,680
And so, you know,

2095
01:10:00,680 --> 01:10:02,630
this, yeah, I
think there's,

2096
01:10:02,630 --> 01:10:04,580
there's strengths
and weaknesses.

2097
01:10:04,580 --> 01:10:07,310
I think it's one thing
that I really tried.

2098
01:10:07,310 --> 01:10:10,459
I need that I think
this fills that

2099
01:10:10,459 --> 01:10:11,600
I was felt like

2100
01:10:11,600 --> 01:10:12,710
the limitation with

2101
01:10:12,710 --> 01:10:14,539
group model building
in some ways is,

2102
01:10:14,539 --> 01:10:17,074
I feel like for
some people,

2103
01:10:17,074 --> 01:10:19,414
a one-on-one interview
might be a bit

2104
01:10:19,414 --> 01:10:22,549
more comfortable
if a situation

2105
01:10:22,549 --> 01:10:24,859
or someone might feel more

2106
01:10:24,859 --> 01:10:28,189
open to sharing what
their perspective is.

2107
01:10:28,189 --> 01:10:30,109
Cuz I feel like with

2108
01:10:30,109 --> 01:10:31,789
the group
modelling because.

2109
01:10:31,789 --> 01:10:33,050
You know, strength is

2110
01:10:33,050 --> 01:10:34,160
getting people to kind of

2111
01:10:34,160 --> 01:10:36,770
coalesce around
a similar view.

2112
01:10:36,770 --> 01:10:37,730
But then the trade-off

2113
01:10:37,730 --> 01:10:38,689
with that is that
you don't get

2114
01:10:38,689 --> 01:10:40,610
that diversity and
perspectives, right?

2115
01:10:40,610 --> 01:10:42,349
And so this could maybe

2116
01:10:42,349 --> 01:10:44,149
be something
where you do in

2117
01:10:44,149 --> 01:10:45,739
an individual
interviews first with

2118
01:10:45,739 --> 01:10:47,539
people and you kind of get

2119
01:10:47,539 --> 01:10:49,520
segments of models
maybe or maybe

2120
01:10:49,520 --> 01:10:50,929
you kind of do

2121
01:10:50,929 --> 01:10:52,639
the individual
models and then see,

2122
01:10:52,639 --> 01:10:54,440
okay, what's kind of

2123
01:10:54,440 --> 01:10:56,539
diversity of perspectives
that you have.

2124
01:10:56,539 --> 01:10:59,690
And then bring people
together and kind of

2125
01:10:59,690 --> 01:11:01,279
have them engage
in that we're at

2126
01:11:01,279 --> 01:11:02,959
where we're doing
a study right now,

2127
01:11:02,959 --> 01:11:04,250
where we're doing
some of that.

2128
01:11:04,250 --> 01:11:05,989
We did individual
interviews with folks,

2129
01:11:05,989 --> 01:11:09,619
were doing some
some mapping and

2130
01:11:09,619 --> 01:11:11,569
then bringing them
together to try

2131
01:11:11,569 --> 01:11:13,955
to to reflect on that.

2132
01:11:13,955 --> 01:11:16,279
So yeah, I'd
love to talk to

2133
01:11:16,279 --> 01:11:19,830
talk more. Thank you.

2134
01:11:22,660 --> 01:11:26,270
I Kinsey, My name
is Nicola and that

2135
01:11:26,270 --> 01:11:27,320
internal evaluator for

2136
01:11:27,320 --> 01:11:29,509
the Native American
Youth and Family Center.

2137
01:11:29,509 --> 01:11:31,205
I'm really excited
to actually

2138
01:11:31,205 --> 01:11:32,899
use your work in my work.

2139
01:11:32,899 --> 01:11:34,759
So I look forward
to reading

2140
01:11:34,759 --> 01:11:35,539
your dissertation with

2141
01:11:35,539 --> 01:11:37,250
its upon cardiac scholar.

2142
01:11:37,250 --> 01:11:38,929
I was just wondering, it's

2143
01:11:38,929 --> 01:11:40,249
okay if you don't
have more details,

2144
01:11:40,249 --> 01:11:41,630
but if you just tell
us a little bit

2145
01:11:41,630 --> 01:11:43,250
more about how
you're able to so

2146
01:11:43,250 --> 01:11:44,809
wonderfully braid

2147
01:11:44,809 --> 01:11:46,549
constructivist and realist

2148
01:11:46,549 --> 01:11:48,754
interview methodologies

2149
01:11:48,754 --> 01:11:51,515
like background and such.

2150
01:11:51,515 --> 01:11:53,899
Yeah. I mean, I think I

2151
01:11:53,899 --> 01:11:57,289
just I think it came down

2152
01:11:57,289 --> 01:12:01,579
to was when I was
trying to figure

2153
01:12:01,579 --> 01:12:06,499
out where what's sort

2154
01:12:06,499 --> 01:12:08,299
of like what would make
when asking myself,

2155
01:12:08,299 --> 01:12:11,555
okay, what would make
a good interview

2156
01:12:11,555 --> 01:12:13,489
designed for
modelling, right?

2157
01:12:13,489 --> 01:12:16,880
I the the qualitative
training and me,

2158
01:12:16,880 --> 01:12:19,819
he kind of originally
went to OK,

2159
01:12:19,819 --> 01:12:20,599
we need to have

2160
01:12:20,599 --> 01:12:22,940
this more
constructivist approach

2161
01:12:22,940 --> 01:12:24,230
that kind of gives

2162
01:12:24,230 --> 01:12:29,269
more respect or was

2163
01:12:29,269 --> 01:12:33,545
steering capacity
to the participant.

2164
01:12:33,545 --> 01:12:34,699
And I think there's a lot

2165
01:12:34,699 --> 01:12:35,900
of validity in that,

2166
01:12:35,900 --> 01:12:38,149
but then I
realized that that

2167
01:12:38,149 --> 01:12:40,864
can only really go so far.

2168
01:12:40,864 --> 01:12:42,259
And that when you

2169
01:12:42,259 --> 01:12:46,279
were modeling what I
like about and there's,

2170
01:12:46,279 --> 01:12:49,129
there's preexisting
research out there within

2171
01:12:49,129 --> 01:12:52,294
the system dynamics
realm around,

2172
01:12:52,294 --> 01:12:53,420
you know, sort of what are

2173
01:12:53,420 --> 01:12:56,119
the philosophical
assumptions there?

2174
01:12:56,119 --> 01:12:59,510
And I think a
realist comes up,

2175
01:12:59,510 --> 01:13:01,790
realism comes up
there because,

2176
01:13:01,790 --> 01:13:03,710
you know, it's,
you don't want to,

2177
01:13:03,710 --> 01:13:04,940
when you're making
a model, you don't

2178
01:13:04,940 --> 01:13:06,319
want to pretend that you as

2179
01:13:06,319 --> 01:13:10,100
a modeler don't
have perspective,

2180
01:13:10,100 --> 01:13:12,184
are you, that you're
not part of it?

2181
01:13:12,184 --> 01:13:14,059
I feel like there's a lot

2182
01:13:14,059 --> 01:13:16,624
of room for
improvement within

2183
01:13:16,624 --> 01:13:18,995
the system
science field for

2184
01:13:18,995 --> 01:13:21,440
better acknowledging
what the role

2185
01:13:21,440 --> 01:13:22,580
of the modeler is.

2186
01:13:22,580 --> 01:13:24,139
And that there,

2187
01:13:24,139 --> 01:13:26,479
there's a tiny
little assumptions

2188
01:13:26,479 --> 01:13:27,980
that you're making
when you're building

2189
01:13:27,980 --> 01:13:30,740
a model and a ton of
little decisions.

2190
01:13:30,740 --> 01:13:33,410
And I think a lot

2191
01:13:33,410 --> 01:13:34,670
of times that gets glossed

2192
01:13:34,670 --> 01:13:35,749
over in the process.

2193
01:13:35,749 --> 01:13:37,039
I think, especially

2194
01:13:37,039 --> 01:13:38,659
with simulation modelling,

2195
01:13:38,659 --> 01:13:39,830
because it looks so

2196
01:13:39,830 --> 01:13:41,989
quantitative and it
has math behind it.

2197
01:13:41,989 --> 01:13:43,774
I think it's easy to say,

2198
01:13:43,774 --> 01:13:46,130
oh, that is the system.

2199
01:13:46,130 --> 01:13:48,050
But I think that
really glosses over

2200
01:13:48,050 --> 01:13:50,465
how that's a creation.

2201
01:13:50,465 --> 01:13:52,444
That it's something that's

2202
01:13:52,444 --> 01:13:54,095
crafted by a modeler.

2203
01:13:54,095 --> 01:13:55,670
And so I think.

2204
01:13:55,670 --> 01:13:58,010
Where this
research comes in,

2205
01:13:58,010 --> 01:13:59,569
is that what I
wanted to do is

2206
01:13:59,569 --> 01:14:03,409
really be intentional
and try to track,

2207
01:14:03,409 --> 01:14:06,260
Okay, what comes from
the purchase paint,

2208
01:14:06,260 --> 01:14:07,670
what comes from a modeler?

2209
01:14:07,670 --> 01:14:09,995
How can we understand where

2210
01:14:09,995 --> 01:14:12,560
those assumptions the
modeler comes in?

2211
01:14:12,560 --> 01:14:14,179
And so I felt
like there was

2212
01:14:14,179 --> 01:14:16,205
a need for, you know,

2213
01:14:16,205 --> 01:14:19,085
having that constructivist
orientation

2214
01:14:19,085 --> 01:14:20,390
to start kinda get

2215
01:14:20,390 --> 01:14:24,454
the purview and
then to say, okay,

2216
01:14:24,454 --> 01:14:26,360
the modeler does have

2217
01:14:26,360 --> 01:14:29,390
that role and that you
need to be able to

2218
01:14:29,390 --> 01:14:31,519
acknowledge that they're
coming to it with

2219
01:14:31,519 --> 01:14:34,339
their own hypotheses and

2220
01:14:34,339 --> 01:14:35,210
you need to have

2221
01:14:35,210 --> 01:14:37,530
a different
perspective there.

2222
01:14:41,220 --> 01:14:44,379
I have one more question.

2223
01:14:44,379 --> 01:14:46,509
The more you talk,

2224
01:14:46,509 --> 01:14:47,619
the clearer it becomes to

2225
01:14:47,619 --> 01:14:48,805
me that we are all kind of

2226
01:14:48,805 --> 01:14:52,030
hostages of our own
artifacts of our model.

2227
01:14:52,030 --> 01:14:54,160
And, and I think
to a large,

2228
01:14:54,160 --> 01:14:55,719
it's as a lot of the
things you're doing,

2229
01:14:55,719 --> 01:14:56,830
I'm doing with a technique

2230
01:14:56,830 --> 01:14:58,210
called fuzzy
cognitive maps.

2231
01:14:58,210 --> 01:15:00,910
And because fuzzy
cognitive maps happened to

2232
01:15:00,910 --> 01:15:02,200
be something that you

2233
01:15:02,200 --> 01:15:03,835
combine very, very easily.

2234
01:15:03,835 --> 01:15:05,350
While of course
we say, well,

2235
01:15:05,350 --> 01:15:07,929
let's do individual
interviews or even less.

2236
01:15:07,929 --> 01:15:10,240
And then combine all
this into a model.

2237
01:15:10,240 --> 01:15:11,275
And because they look a lot

2238
01:15:11,275 --> 01:15:12,309
like neural networks there,

2239
01:15:12,309 --> 01:15:14,170
let's try that with
artificial intelligence.

2240
01:15:14,170 --> 01:15:15,610
So I think a lot of where

2241
01:15:15,610 --> 01:15:17,260
we're going with
this simply

2242
01:15:17,260 --> 01:15:20,550
based on the
modelling technique.

2243
01:15:20,550 --> 01:15:22,880
And I'm seeing the
same in your work and,

2244
01:15:22,880 --> 01:15:25,820
and I've seen many
feel now duplicating

2245
01:15:25,820 --> 01:15:27,229
a lot of the same thing

2246
01:15:27,229 --> 01:15:28,955
with the same intentions,

2247
01:15:28,955 --> 01:15:31,145
but they're stuck on
their modeling method.

2248
01:15:31,145 --> 01:15:33,020
How do we get over this?

2249
01:15:33,020 --> 01:15:35,599
What, what would
you suggest we do?

2250
01:15:35,599 --> 01:15:38,225
Well after Kobe,
Michigan together and,

2251
01:15:38,225 --> 01:15:41,824
you know, I mean,

2252
01:15:41,824 --> 01:15:43,279
I think it's a
really good question

2253
01:15:43,279 --> 01:15:44,540
and I feel like
that was something

2254
01:15:44,540 --> 01:15:47,000
that one

2255
01:15:47,000 --> 01:15:48,800
of the hardest things
about this research

2256
01:15:48,800 --> 01:15:51,334
was feeling parallelize
that I couldn't

2257
01:15:51,334 --> 01:15:54,454
explore all of those
possible roads.

2258
01:15:54,454 --> 01:15:56,090
I would, yeah, I
really wanted to

2259
01:15:56,090 --> 01:15:57,964
do a deep dive into

2260
01:15:57,964 --> 01:16:00,319
fuzzy cognitive mapping
and these other

2261
01:16:00,319 --> 01:16:02,569
sorts of mapping methods.

2262
01:16:02,569 --> 01:16:07,234
And so it was,
you're right,

2263
01:16:07,234 --> 01:16:09,170
like there is a lot
out there and I think

2264
01:16:09,170 --> 01:16:12,274
there needs to be

2265
01:16:12,274 --> 01:16:16,954
much more dialogue in

2266
01:16:16,954 --> 01:16:18,140
real life and in the

2267
01:16:18,140 --> 01:16:19,400
literature around like how

2268
01:16:19,400 --> 01:16:22,354
do we learn from
each other?

2269
01:16:22,354 --> 01:16:26,150
And how do we yeah.

2270
01:16:26,150 --> 01:16:30,440
Combine or or blend
methods when needed.

2271
01:16:30,440 --> 01:16:33,170
So yeah, I totally agree.

2272
01:16:33,170 --> 01:16:35,569
Someday we'll have
the Vulcan mind

2273
01:16:35,569 --> 01:16:37,589
meld to accomplish that.

2274
01:16:38,080 --> 01:16:41,825
I just wanted to ask
you for the record.

2275
01:16:41,825 --> 01:16:43,459
Because I think
that might be

2276
01:16:43,459 --> 01:16:44,960
students watching this in

2277
01:16:44,960 --> 01:16:47,509
future years,
maybe in months.

2278
01:16:47,509 --> 01:16:50,000
Even. What advice would you

2279
01:16:50,000 --> 01:16:52,819
have are two
classes of people.

2280
01:16:52,819 --> 01:16:56,539
People in a room of
which could be full-on.

2281
01:16:56,539 --> 01:16:57,620
Experienced practitioners

2282
01:16:57,620 --> 01:16:58,760
will want to use
this method.

2283
01:16:58,760 --> 01:17:00,110
What sort of
advice would you

2284
01:17:00,110 --> 01:17:01,879
have for those people?

2285
01:17:01,879 --> 01:17:03,080
And in particular,
what advice

2286
01:17:03,080 --> 01:17:04,805
would you have for students

2287
01:17:04,805 --> 01:17:06,800
about how you found

2288
01:17:06,800 --> 01:17:10,049
your way to a very
successful dissertation.

2289
01:17:12,630 --> 01:17:15,099
I mean, I think
for this method,

2290
01:17:15,099 --> 01:17:16,119
for practitioners,

2291
01:17:16,119 --> 01:17:18,580
I may think This isn't
an original thought,

2292
01:17:18,580 --> 01:17:19,869
but I think it really
just comes down

2293
01:17:19,869 --> 01:17:21,699
to what are your
research questions?

2294
01:17:21,699 --> 01:17:23,560
What are you trying to get?

2295
01:17:23,560 --> 01:17:26,335
What do you really,

2296
01:17:26,335 --> 01:17:29,740
I think having
a dialogue and

2297
01:17:29,740 --> 01:17:33,985
a critical understanding
of what yeah,

2298
01:17:33,985 --> 01:17:35,364
what are your questions?

2299
01:17:35,364 --> 01:17:36,579
What are the strategies

2300
01:17:36,579 --> 01:17:39,459
needed to answer
those questions?

2301
01:17:39,459 --> 01:17:41,800
What is, what is sort of

2302
01:17:41,800 --> 01:17:43,989
the least strategy needed

2303
01:17:43,989 --> 01:17:45,685
to answer those
questions to?

2304
01:17:45,685 --> 01:17:49,840
I think this research
fills a certain niche,

2305
01:17:49,840 --> 01:17:51,219
but it's, you know,

2306
01:17:51,219 --> 01:17:54,049
it's not, it, It's
a specific niche.

2307
01:17:54,049 --> 01:17:55,970
You know, I think
there's a lot

2308
01:17:55,970 --> 01:17:57,890
of situations
where standard

2309
01:17:57,890 --> 01:18:00,259
qualitative interviews
or be just great or

2310
01:18:00,259 --> 01:18:01,909
modelling that's
more modeler

2311
01:18:01,909 --> 01:18:03,304
lead, but I think there's,

2312
01:18:03,304 --> 01:18:06,079
there needs to be more kind

2313
01:18:06,079 --> 01:18:08,389
of operationalization
or guidelines

2314
01:18:08,389 --> 01:18:11,779
developed for how to

2315
01:18:11,779 --> 01:18:14,179
use these different
tools in the toolbox.

2316
01:18:14,179 --> 01:18:16,099
So I think this is
a starting point,

2317
01:18:16,099 --> 01:18:18,740
but I think and
I think that's,

2318
01:18:18,740 --> 01:18:22,489
it's the onus of
developing that is

2319
01:18:22,489 --> 01:18:24,260
on people like us

2320
01:18:24,260 --> 01:18:27,260
who developed methods
are proposed methods.

2321
01:18:27,260 --> 01:18:29,539
I think we need to.

2322
01:18:29,539 --> 01:18:32,059
I really tried to
think about the,

2323
01:18:32,059 --> 01:18:33,394
the end user and,

2324
01:18:33,394 --> 01:18:36,209
and try to develop
some of that.

2325
01:18:36,280 --> 01:18:39,649
I mean, as far as advice

2326
01:18:39,649 --> 01:18:43,795
for dissertation enemy at

2327
01:18:43,795 --> 01:18:45,680
first try not to do
it during a pandemic.

2328
01:18:45,680 --> 01:18:49,459
I think that and if you

2329
01:18:49,459 --> 01:18:54,785
do yeah. Yeah.
I don't know.

2330
01:18:54,785 --> 01:18:58,295
I think it was yeah.

2331
01:18:58,295 --> 01:19:02,600
I think I'm trying
to set boundaries

2332
01:19:02,600 --> 01:19:04,369
for yourself and trying to

2333
01:19:04,369 --> 01:19:05,539
realize that you can't go

2334
01:19:05,539 --> 01:19:06,980
down on all of those roads.

2335
01:19:06,980 --> 01:19:09,305
It was important for me.

2336
01:19:09,305 --> 01:19:12,104
And try understanding that

2337
01:19:12,104 --> 01:19:13,749
doing research is

2338
01:19:13,749 --> 01:19:15,249
a learning process
and you don't

2339
01:19:15,249 --> 01:19:18,520
need to necessarily stick

2340
01:19:18,520 --> 01:19:20,500
to where your original
starting point wasn't

2341
01:19:20,500 --> 01:19:22,690
because you change and

2342
01:19:22,690 --> 01:19:24,160
you are understanding
changes and

2343
01:19:24,160 --> 01:19:27,009
so your your research

2344
01:19:27,009 --> 01:19:28,060
should change with that.

2345
01:19:28,060 --> 01:19:29,140
And as long as
you're kind of

2346
01:19:29,140 --> 01:19:32,410
being engaging or
committee and engaging,

2347
01:19:32,410 --> 01:19:34,659
kind of thinking
critically about that,

2348
01:19:34,659 --> 01:19:36,175
that you know, to,

2349
01:19:36,175 --> 01:19:38,020
to try to use this
as a period of

2350
01:19:38,020 --> 01:19:40,704
evolution in your
own understanding.

2351
01:19:40,704 --> 01:19:42,910
Which I, that's what I

2352
01:19:42,910 --> 01:19:43,570
thought was one of

2353
01:19:43,570 --> 01:19:45,640
the biggest
benefits for me.

2354
01:19:45,640 --> 01:19:49,819
Great. Thanks. Any
other questions?

2355
01:19:54,930 --> 01:19:58,149
Go ahead. I don't
have a question,

2356
01:19:58,149 --> 01:19:59,710
but I think maybe adding

2357
01:19:59,710 --> 01:20:02,364
some compliments on
would be valuable.

2358
01:20:02,364 --> 01:20:03,879
And I think, and in

2359
01:20:03,879 --> 01:20:05,470
relation to your
question Wayne

2360
01:20:05,470 --> 01:20:07,329
about advice for students

2361
01:20:07,329 --> 01:20:08,829
what maybe era and isn't

2362
01:20:08,829 --> 01:20:10,480
articulating as she had

2363
01:20:10,480 --> 01:20:13,869
a very different plan
about nine months ago.

2364
01:20:13,869 --> 01:20:15,669
And so I think

2365
01:20:15,669 --> 01:20:17,230
sometimes as we
enter this work,

2366
01:20:17,230 --> 01:20:20,949
we might have a plan

2367
01:20:20,949 --> 01:20:22,899
and not be open to
the opportunities.

2368
01:20:22,899 --> 01:20:25,240
And Aaron just did an
amazing job of really

2369
01:20:25,240 --> 01:20:27,880
responding to
the dynamics of

2370
01:20:27,880 --> 01:20:29,724
what was happening during

2371
01:20:29,724 --> 01:20:32,289
coded and then
finding a way to

2372
01:20:32,289 --> 01:20:34,390
move forward that
still allowed her to

2373
01:20:34,390 --> 01:20:37,484
advance her methods and
advance your skills.

2374
01:20:37,484 --> 01:20:39,574
And not getting stuck.

2375
01:20:39,574 --> 01:20:41,330
I think the other thing I

2376
01:20:41,330 --> 01:20:43,729
might comment on and
this relates to Billy,

2377
01:20:43,729 --> 01:20:45,769
your question about
how do we maybe

2378
01:20:45,769 --> 01:20:47,060
Bridge and how do we bring

2379
01:20:47,060 --> 01:20:48,665
this to communities is,

2380
01:20:48,665 --> 01:20:50,105
is I think we also need

2381
01:20:50,105 --> 01:20:51,289
to think about training.

2382
01:20:51,289 --> 01:20:53,660
So Erin brings like

2383
01:20:53,660 --> 01:20:55,489
the beauty that
sometimes you

2384
01:20:55,489 --> 01:20:57,559
also find in
biostatisticians,

2385
01:20:57,559 --> 01:20:59,030
which is the ability to do

2386
01:20:59,030 --> 01:21:01,790
complex work and
rigorous work.

2387
01:21:01,790 --> 01:21:03,799
But when you're
communicating with partners,

2388
01:21:03,799 --> 01:21:06,890
simplify it in a way
or focus attention

2389
01:21:06,890 --> 01:21:08,330
in a way that the end-user

2390
01:21:08,330 --> 01:21:10,445
can actually absorb,

2391
01:21:10,445 --> 01:21:12,559
unlike act on
the information.

2392
01:21:12,559 --> 01:21:14,390
So I guess I
would just say,

2393
01:21:14,390 --> 01:21:15,679
I think that's another

2394
01:21:15,679 --> 01:21:17,059
really important aspect

2395
01:21:17,059 --> 01:21:18,290
of this work is errands,

2396
01:21:18,290 --> 01:21:20,509
ability to distill
complexity

2397
01:21:20,509 --> 01:21:21,979
so that we can all

2398
01:21:21,979 --> 01:21:26,269
enter in and understand
the value of it.

2399
01:21:26,269 --> 01:21:28,219
So it's been a pleasure

2400
01:21:28,219 --> 01:21:29,630
and I would just
commend you for

2401
01:21:29,630 --> 01:21:32,929
kinda those softer
skills that may be

2402
01:21:32,929 --> 01:21:34,595
also are part of what

2403
01:21:34,595 --> 01:21:37,415
has allowed you to
be so successful.

2404
01:21:37,415 --> 01:21:41,960
Thinking. You
didn't really say

2405
01:21:41,960 --> 01:21:43,400
specifically to much about

2406
01:21:43,400 --> 01:21:44,840
your publishing plan to

2407
01:21:44,840 --> 01:21:46,550
get you at the very
highest level.

2408
01:21:46,550 --> 01:21:47,839
Just talked
about where this

2409
01:21:47,839 --> 01:21:49,280
might get published and

2410
01:21:49,280 --> 01:21:50,720
kind of the timeframe
that you sort of

2411
01:21:50,720 --> 01:21:52,269
have mindsets putting on,

2412
01:21:52,269 --> 01:21:53,950
they'll spotted
all just just talk

2413
01:21:53,950 --> 01:21:54,549
about a little bit

2414
01:21:54,549 --> 01:21:56,049
from a publishing
perspective.

2415
01:21:56,049 --> 01:21:57,429
Sure, yeah, I mean,

2416
01:21:57,429 --> 01:21:58,780
the first paper with

2417
01:21:58,780 --> 01:22:00,190
the systematic
review I hope

2418
01:22:00,190 --> 01:22:02,080
to publish in a journal

2419
01:22:02,080 --> 01:22:04,705
for an evaluation audience.

2420
01:22:04,705 --> 01:22:06,939
The second, third papers,

2421
01:22:06,939 --> 01:22:08,290
while the second paper,

2422
01:22:08,290 --> 01:22:13,015
the the causal structure
mapping method,

2423
01:22:13,015 --> 01:22:17,530
where I'm the
the interviews

2424
01:22:17,530 --> 01:22:21,130
with the the practice
facilitators.

2425
01:22:21,130 --> 01:22:23,964
There's this initial paper

2426
01:22:23,964 --> 01:22:27,219
which I think I think
I'm planning to copy.

2427
01:22:27,219 --> 01:22:29,290
I think both those
papers are 23

2428
01:22:29,290 --> 01:22:31,570
are more for a methods
audience are sort

2429
01:22:31,570 --> 01:22:34,035
of more of a systems
methods audience

2430
01:22:34,035 --> 01:22:35,660
kind of within
the tradition

2431
01:22:35,660 --> 01:22:37,639
of that Kim
Anderson paper and

2432
01:22:37,639 --> 01:22:38,690
the other papers that

2433
01:22:38,690 --> 01:22:39,770
have followed from that.

2434
01:22:39,770 --> 01:22:41,270
So I feel like that
kind of fits within

2435
01:22:41,270 --> 01:22:43,430
that tradition.

2436
01:22:43,430 --> 01:22:45,109
But then there's, Oh,

2437
01:22:45,109 --> 01:22:47,644
I'm also planning to,
at the end of that,

2438
01:22:47,644 --> 01:22:49,160
the orphan study with

2439
01:22:49,160 --> 01:22:50,989
the practice
facilitators will

2440
01:22:50,989 --> 01:22:54,499
do a longitudinal analysis

2441
01:22:54,499 --> 01:22:56,600
of the different map,

2442
01:22:56,600 --> 01:22:57,935
the maps from every,

2443
01:22:57,935 --> 01:22:59,450
the interviews from
every six months.

2444
01:22:59,450 --> 01:23:03,590
And so that will have
I figure we'll have a,

2445
01:23:03,590 --> 01:23:06,109
well at least one
publication that's more for

2446
01:23:06,109 --> 01:23:08,689
a practice facilitation
audience and,

2447
01:23:08,689 --> 01:23:10,730
you know, really more
digs into, okay,

2448
01:23:10,730 --> 01:23:11,840
what were the findings

2449
01:23:11,840 --> 01:23:13,850
of or the content
of some of

2450
01:23:13,850 --> 01:23:15,470
those maps and what can

2451
01:23:15,470 --> 01:23:17,599
that field learn
from some of that?

2452
01:23:17,599 --> 01:23:18,860
So I think it's,

2453
01:23:18,860 --> 01:23:21,799
I definitely, when
developing this work,

2454
01:23:21,799 --> 01:23:24,410
I kept kind of
switching between

2455
01:23:24,410 --> 01:23:25,609
these multiple hats and

2456
01:23:25,609 --> 01:23:27,469
the different
potential audiences.

2457
01:23:27,469 --> 01:23:29,119
I think it's something
and system science

2458
01:23:29,119 --> 01:23:31,729
we always kinda have
to ask is okay,

2459
01:23:31,729 --> 01:23:33,350
is this first systems
audiences is for

2460
01:23:33,350 --> 01:23:35,900
the sort of subject
matter audience.

2461
01:23:35,900 --> 01:23:37,909
And yeah, so I,

2462
01:23:37,909 --> 01:23:40,940
I'm looking forward
to kind of trying to

2463
01:23:40,940 --> 01:23:45,300
sort that out a little
bit more TO thank you.

2464
01:23:46,750 --> 01:23:49,669
Well, if there aren't
additional questions,

2465
01:23:49,669 --> 01:23:51,979
I'm gonna ask the
committee to move

2466
01:23:51,979 --> 01:23:55,340
into a breakout room.

2467
01:23:55,340 --> 01:23:57,509
I haven't set up.
I'm just going.