1
00:00:02,760 --> 00:00:08,780
All right, well, welcome everybody to
the system science writing Seminar Series.

2
00:00:08,780 --> 00:00:12,805
Today we have a double graduate
of system science or rainmaker,

3
00:00:12,805 --> 00:00:16,022
here with our silver and
what's it called though?

4
00:00:16,022 --> 00:00:18,402
It's called the research-
>> Regional Research-

5
00:00:18,402 --> 00:00:19,643
>> Regional Research Institute.

6
00:00:19,643 --> 00:00:21,310
>> Regional Research Institute.

7
00:00:21,310 --> 00:00:22,880
>> There it is right on the screen.

8
00:00:22,880 --> 00:00:24,940
So I'm gonna let her introduce herself,

9
00:00:24,940 --> 00:00:29,230
any further issue you might want to and
then just right into the topic.

10
00:00:30,450 --> 00:00:34,036
>> Yeah, so I was gonna say that realist
evaluation is something that I know very

11
00:00:34,036 --> 00:00:37,581
little about, I'm just learning but
there was room who knew a lot about it.

12
00:00:37,581 --> 00:00:41,912
So am hoping that you can
correct me if I misstep,

13
00:00:41,912 --> 00:00:46,139
I do know a fair amount
about interventions, so

14
00:00:46,139 --> 00:00:49,326
[LAUGH] I can guide you to some parts.

15
00:00:49,326 --> 00:00:54,474
So I am a research assistant professor
over at regional Research Institute for

16
00:00:54,474 --> 00:00:59,232
Human Services under the School of
Social Work, but I am an alumni here so

17
00:00:59,232 --> 00:01:00,792
I'm really, really,

18
00:01:00,792 --> 00:01:05,573
being back in this building makes me
feel super warm and fuzzy [LAUGH] I do.

19
00:01:05,573 --> 00:01:07,614
>> [LAUGH]
>> [LAUGH] [CROSSTALK] [LAUGH]

20
00:01:07,614 --> 00:01:10,006
>> It feels like coming home.

21
00:01:10,006 --> 00:01:11,691
I really like this equipment.

22
00:01:11,691 --> 00:01:16,551
So I'm a public health
social services intervention

23
00:01:16,551 --> 00:01:22,059
research so
everything I do is direct collaboration

24
00:01:22,059 --> 00:01:27,300
with populations who
are affected by the services.

25
00:01:27,300 --> 00:01:33,760
I have a pretty strong disability justice,
meaning I'm inside a researcher.

26
00:01:33,760 --> 00:01:38,690
So I'm going to bring up something that I
research and I'm pretty into science for

27
00:01:38,690 --> 00:01:40,070
social justice.

28
00:01:40,070 --> 00:01:46,660
Kind of PPC for my T shirt,
and I was a toddler.

29
00:01:48,060 --> 00:01:54,225
So these are two of my main projects and
my biggest project is a spy.

30
00:01:54,225 --> 00:01:58,375
And we have a an intervention
that's just healthcare toolkit.

31
00:01:59,605 --> 00:02:04,725
It's a psycho information about
health services and a tool for

32
00:02:04,725 --> 00:02:10,610
healthcare providers to better accommodate
autistic adults in healthcare settings.

33
00:02:10,610 --> 00:02:13,160
And I'm just working right
now in the formative

34
00:02:13,160 --> 00:02:15,970
stages of an intervention on top
of the mid skilled employment.

35
00:02:15,970 --> 00:02:20,070
I ended up not talking a whole lot about
that for the rest of the talk, but talk

36
00:02:20,070 --> 00:02:24,410
a lot more about the early assessments for
the Alliance Visa Connections Project.

37
00:02:24,410 --> 00:02:27,940
Which is an intervention
that I collaborated with

38
00:02:27,940 --> 00:02:29,600
the early psychosis community.

39
00:02:29,600 --> 00:02:36,152
So these are young adults who have
experienced first episode psychosis and

40
00:02:36,152 --> 00:02:42,185
we created a peer developed,
peer driven, online intervention for

41
00:02:42,185 --> 00:02:46,657
people who are new to
the program to welcome them and

42
00:02:46,657 --> 00:02:52,170
do make them kind of destigmatized,
psychosis a little bit and

43
00:02:52,170 --> 00:02:58,953
kind of help them gain entry into
the program by peer delivered information.

44
00:02:58,953 --> 00:03:00,558
What is an intervention research?

45
00:03:00,558 --> 00:03:06,749
This is always a very wide
group of people, wide areas,

46
00:03:06,749 --> 00:03:13,620
everything from mathematical
abstraction to acupuncture.

47
00:03:14,780 --> 00:03:18,669
So, is intervention research kind
of a new word for most people?

48
00:03:18,669 --> 00:03:20,192
Or is this, okay, so

49
00:03:20,192 --> 00:03:25,960
I'm gonna give just a little bit of a
background on what I'm talking about here.

50
00:03:25,960 --> 00:03:32,160
So, intervention research is research
that is meant to explore development

51
00:03:32,160 --> 00:03:37,330
test programs that are designed to
affect some kind of a specified outcome.

52
00:03:37,330 --> 00:03:39,790
So it can be really straightforward.

53
00:03:39,790 --> 00:03:43,040
If you think about current trials,
those are interventions for

54
00:03:43,040 --> 00:03:44,900
some kind of a medical thing.

55
00:03:44,900 --> 00:03:48,310
But they could also be really complex,
which is the kind of stuff I do research

56
00:03:48,310 --> 00:03:53,550
to explore developing testing program to
improve long term career outcomes for

57
00:03:53,550 --> 00:03:59,060
autistic professionals which has all
of the real world complexity to it.

58
00:03:59,060 --> 00:04:02,440
Either way, intervention research
is really about using science

59
00:04:02,440 --> 00:04:03,950
to do a thing to change the world.

60
00:04:03,950 --> 00:04:06,809
So what you're trying to do is
affect some kind of a change.

61
00:04:09,620 --> 00:04:10,415
So we're good.

62
00:04:10,415 --> 00:04:15,281
Frequently cover some classical
approaches to intervention research,

63
00:04:15,281 --> 00:04:19,673
and this is what I learned before
I got really excited about realist

64
00:04:19,673 --> 00:04:23,608
approaches to give something
to kind of contrast against.

65
00:04:23,608 --> 00:04:27,780
It's a little clearer that way at least,
yeah at this stage.

66
00:04:27,780 --> 00:04:32,133
So we develop these theories of change and
logic models when we're trying to think

67
00:04:32,133 --> 00:04:34,856
about how are we gonna
actually affect something?

68
00:04:34,856 --> 00:04:36,165
How do we understand it?

69
00:04:36,165 --> 00:04:39,739
We have this idea of inputs and
then things we're gonna do or

70
00:04:39,739 --> 00:04:45,550
things we're gonna activate in order to
create these outputs or these outcomes.

71
00:04:45,550 --> 00:04:48,120
This desired effects are very linear.

72
00:04:50,050 --> 00:04:53,440
Here's a couple filled out for
the EASA Connections Project.

73
00:04:55,630 --> 00:04:59,410
Our inputs are these resources and
we're gonna make them available and

74
00:04:59,410 --> 00:05:03,190
then we've got these determinants,
we're gonna activate people's competence

75
00:05:03,190 --> 00:05:08,059
relatedness and autonomy in order to
have the self determination outcome.

76
00:05:09,100 --> 00:05:11,890
This was the logic model.

77
00:05:11,890 --> 00:05:15,450
So we get the inputs and then
the activities are what are you doing?

78
00:05:15,450 --> 00:05:16,110
What are you gonna do?

79
00:05:16,110 --> 00:05:19,640
What are the intervention activities?

80
00:05:19,640 --> 00:05:22,650
What falls out of that?

81
00:05:22,650 --> 00:05:27,160
And then you've got outcomes or
short term or intermediate

82
00:05:27,160 --> 00:05:31,260
outcomes at may impact your impact,
is the big thing that you want to impact.

83
00:05:33,390 --> 00:05:37,090
Again, this very linear notion of what

84
00:05:37,090 --> 00:05:40,710
you're gonna do in a certain
evaluation steps along the way.

85
00:05:40,710 --> 00:05:46,000
So after you've developed it and
put it together, you do pilot tests.

86
00:05:46,000 --> 00:05:50,300
So you create a program based on your
theory of change in your logic model.

87
00:05:50,300 --> 00:05:52,361
You see if you can do it, is it feasible?

88
00:05:52,361 --> 00:05:55,645
You'll see if people like it,
as if they don't.

89
00:05:55,645 --> 00:05:56,861
That's it.

90
00:05:56,861 --> 00:05:57,945
Is it acceptable?

91
00:05:57,945 --> 00:06:02,214
You test your data collection protocols,
fidelity,

92
00:06:02,214 --> 00:06:07,985
can people consistently deliver this
intervention based on a protocol?

93
00:06:07,985 --> 00:06:13,152
Test your outcome measures because
you're gonna ultimately have to measure,

94
00:06:13,152 --> 00:06:17,718
did the things that we thought we're
gonna move at this point in time?

95
00:06:17,718 --> 00:06:23,995
Did we actually have
a increased self-esteem,

96
00:06:23,995 --> 00:06:29,515
so you need some measures for that custom.

97
00:06:29,515 --> 00:06:33,376
Often mixed methods of
quantitative qualitative mix,

98
00:06:33,376 --> 00:06:38,714
pre/post study design sometimes get
broad sketches, it's not about we can

99
00:06:38,714 --> 00:06:44,410
change it at this stage, but do we think
this is this looking like it might work?

100
00:06:44,410 --> 00:06:49,480
And then if it looks like it might work,
you go into the efficacy testing stage and

101
00:06:49,480 --> 00:06:53,260
randomized control trials are considered
the gold standard for that.

102
00:06:53,260 --> 00:06:57,960
So, efficacy testing you're asking did
the program really changed the specified

103
00:06:57,960 --> 00:07:00,155
outcomes under controlled conditions?

104
00:07:00,155 --> 00:07:04,463
So you're gonna use up a couple of
measures, something to compare.

105
00:07:04,463 --> 00:07:09,449
You're going to use a control through, so
you're gonna compare it to a population

106
00:07:09,449 --> 00:07:13,453
that's otherwise similar but
did not receive the intervention.

107
00:07:13,453 --> 00:07:18,647
So you can use out what was
the intervention versus what wasn't,

108
00:07:18,647 --> 00:07:23,474
and what's really the intervention
that impacted people.

109
00:07:23,474 --> 00:07:27,890
Yeah, RTC's are considered the gold
standard of strong evidence.

110
00:07:27,890 --> 00:07:30,206
There's a statistical
aggregation of results.

111
00:07:30,206 --> 00:07:35,702
So, in the end,
we're gonna have some thing that says,

112
00:07:35,702 --> 00:07:41,099
X percentage of people had
the dial blue on the outcome.

113
00:07:41,099 --> 00:07:45,772
And it makes sense for research, does
the medication reduce blood sugar level?

114
00:07:45,772 --> 00:07:48,820
The very black and white question.

115
00:07:50,690 --> 00:07:54,620
But there are a lot of issues
with these approaches.

116
00:07:54,620 --> 00:07:56,470
Where do you get into real world.

117
00:07:56,470 --> 00:08:00,911
Complex stuff, so
I'm giving this talk to you [LAUGH].

118
00:08:00,911 --> 00:08:04,505
So, real world implementation
is super messy.

119
00:08:04,505 --> 00:08:08,959
You can't control conditions, the
conditions are a constantly moving target.

120
00:08:08,959 --> 00:08:13,770
Because the environment is always
changing, there's turnover in staff.

121
00:08:13,770 --> 00:08:17,574
You don't know what's gonna happen,
contamination that's difficult to avoid.

122
00:08:17,574 --> 00:08:21,674
So if you've got a social program,
like the early psychosis program,

123
00:08:21,674 --> 00:08:23,941
you have people who talk to each other.

124
00:08:23,941 --> 00:08:27,185
They're gonna run into each other in
the hall, hey, I did this cool thing.

125
00:08:27,185 --> 00:08:28,403
Have you done the cool thing?

126
00:08:28,403 --> 00:08:33,469
And so, it's really hard to kinda
contain your intervention and

127
00:08:33,469 --> 00:08:36,710
keep it separate from the control group.

128
00:08:36,710 --> 00:08:39,590
Many influences on,
there's stuff happening

129
00:08:39,590 --> 00:08:43,899
in your life that might be moving the dial
that's separate from the intervention.

130
00:08:44,900 --> 00:08:46,818
There's feedback everywhere.

131
00:08:46,818 --> 00:08:50,662
Aggregation is kind of a problem because
every intervention works for someone and

132
00:08:50,662 --> 00:08:52,009
doesn't work for someone.

133
00:08:52,009 --> 00:08:54,613
So if we're gonna take
a binary percentage,

134
00:08:54,613 --> 00:08:57,772
are you actually throwing
away something important?

135
00:08:57,772 --> 00:09:00,308
Because the people that worked for

136
00:09:00,308 --> 00:09:04,250
you really didn't make
a difference in their lives.

137
00:09:04,250 --> 00:09:07,210
And there are also some issues with,
in my opinion,

138
00:09:07,210 --> 00:09:10,254
measuring some complex and
subjective constructs.

139
00:09:10,254 --> 00:09:15,123
Like, can a measure really tell you
whether you've reduced internalized

140
00:09:15,123 --> 00:09:15,748
stigma?

141
00:09:15,748 --> 00:09:20,175
Is there a measure that's good
enough to capture that construct?

142
00:09:20,175 --> 00:09:24,462
And there are multiple approaches
to solving these problems.

143
00:09:24,462 --> 00:09:29,280
But for
once I'm gonna speculate [INAUDIBLE].

144
00:09:29,280 --> 00:09:32,510
The Realist Evaluation does not ask,
what works?

145
00:09:32,510 --> 00:09:36,840
But it asks, what works for whom,
in what circumstances and why?

146
00:09:37,960 --> 00:09:44,660
So it's a different, it's a very different
core research question from the start.

147
00:09:44,660 --> 00:09:47,010
It provides a non-dichotomous paradigm for
evaluation.

148
00:09:47,010 --> 00:09:51,975
So in those traditional approaches
we're looking for a positive result,

149
00:09:51,975 --> 00:09:55,376
it worked, or a negative result,
it did not work.

150
00:09:55,376 --> 00:09:59,203
But that makes no sense
in terms of evaluation.

151
00:09:59,203 --> 00:10:04,530
It tests an intervention to kind of
understand its structure, contexts.

152
00:10:04,530 --> 00:10:07,522
What mechanisms inside of it
lead to various outcomes?

153
00:10:07,522 --> 00:10:10,494
And it's trying to better
target population, or

154
00:10:10,494 --> 00:10:12,554
create more effective programs.

155
00:10:12,554 --> 00:10:15,075
It's evaluating the results,

156
00:10:15,075 --> 00:10:20,498
like what do we know about what's
going on here based on the results?

157
00:10:20,498 --> 00:10:25,920
So some underlying assumptions and
framework.

158
00:10:25,920 --> 00:10:32,275
So, an intervention is an idea that
somebody has for creating change.

159
00:10:32,275 --> 00:10:35,040
It's a theory that we can use
scientific method to test.

160
00:10:35,040 --> 00:10:39,370
So the idea is that an intervention
isn't some program sitting here.

161
00:10:39,370 --> 00:10:43,230
It's something that we're gonna
test with scientific method.

162
00:10:43,230 --> 00:10:50,200
An intervention, it assumes exists inside
of a larger social context, cuz it does.

163
00:10:50,200 --> 00:10:53,250
It thinks about the results of
an intervention as being created

164
00:10:53,250 --> 00:10:57,900
by an active relationship between
participants in the intervention resource.

165
00:10:57,900 --> 00:11:00,030
It's not something you do to a person.

166
00:11:00,030 --> 00:11:05,120
It's something that is done
actively with someone.

167
00:11:05,120 --> 00:11:07,880
And it thinks of interventions
as open systems.

168
00:11:07,880 --> 00:11:11,240
They're actively entangled
in their environment, and

169
00:11:11,240 --> 00:11:12,790
everything is always changing.

170
00:11:12,790 --> 00:11:15,530
It doesn't pretend that you
can create a laboratory

171
00:11:15,530 --> 00:11:17,628
condition under which that isn't true.

172
00:11:17,628 --> 00:11:20,950
It says okay, this is the real world,
this is what it's like.

173
00:11:20,950 --> 00:11:26,273
So we're gonna start by accepting
this as if it's [INAUDIBLE].

174
00:11:26,273 --> 00:11:30,824
So three core components of
realist evaluation are mechanisms,

175
00:11:30,824 --> 00:11:32,569
contexts and outcomes.

176
00:11:35,019 --> 00:11:36,677
So mechanisms are,

177
00:11:36,677 --> 00:11:42,247
what about the program do we think
is going to bring about an effect?

178
00:11:42,247 --> 00:11:47,110
There are processes by which people
interact with the intervention.

179
00:11:47,110 --> 00:11:51,840
They're different from those determinants
that we saw in the change model.

180
00:11:51,840 --> 00:11:55,700
It's not like some construct
that's going to be activated.

181
00:11:55,700 --> 00:12:00,292
It's an active thing that is
the thing that is happening.

182
00:12:00,292 --> 00:12:04,754
Interventions don't work
the customary searches and protocols.

183
00:12:07,291 --> 00:12:10,116
The intervention itself
isn't that what works.

184
00:12:10,116 --> 00:12:12,656
What works is the resources and
the protocols,

185
00:12:12,656 --> 00:12:15,334
in conjunction with
a person's participation.

186
00:12:15,334 --> 00:12:19,770
Again, it's this sort of very active
process is what will work or not.

187
00:12:19,770 --> 00:12:24,444
And there are many of them,
so I took examples.

188
00:12:24,444 --> 00:12:28,641
I've ripped them straight
from a really cool document.

189
00:12:28,641 --> 00:12:34,604
And my last slide I've got links on, and
I can send you all slides if you email me,

190
00:12:34,604 --> 00:12:39,168
where you can get this document
that I pulled these examples,

191
00:12:39,168 --> 00:12:40,776
in whole cloth from.

192
00:12:40,776 --> 00:12:45,041
It's a really great foundational
document that I think,

193
00:12:45,041 --> 00:12:47,224
on a person [INAUDIBLE] [LAUGH].

194
00:12:47,224 --> 00:12:51,070
So, this is just an example of mechanisms.

195
00:12:51,070 --> 00:12:55,658
So the intervention was to
improve classroom attendance by

196
00:12:55,658 --> 00:12:57,592
doing a breakfast club.

197
00:12:57,592 --> 00:13:01,660
So, one mechanism is
providing extra nutrition.

198
00:13:01,660 --> 00:13:05,010
Another is to prevent
misbehaving before class,

199
00:13:05,010 --> 00:13:07,500
so people aren't revved
up when they walk in.

200
00:13:07,500 --> 00:13:11,490
Another mechanism is that it
burns off activity and energy.

201
00:13:11,490 --> 00:13:14,101
It makes school seam less stiff and
formal.

202
00:13:14,101 --> 00:13:16,369
It gives teachers more time to prepare,
and

203
00:13:16,369 --> 00:13:18,956
it enables parents to
connect with school staff.

204
00:13:18,956 --> 00:13:25,270
So those are some possible mechanisms
of that breakfast club intervention.

205
00:13:25,270 --> 00:13:28,353
So contexts, they aren't a place,

206
00:13:28,353 --> 00:13:33,531
they're in what conditions is
the intervention operating?

207
00:13:33,531 --> 00:13:36,723
And they are pretty much
just all of the elements or

208
00:13:36,723 --> 00:13:40,228
circumstances that
are impacting the intervention.

209
00:13:40,228 --> 00:13:46,525
They could facilitate, but they could
also be barriers, or they could be both.

210
00:13:46,525 --> 00:13:48,440
And again, there will be many of them.

211
00:13:48,440 --> 00:13:53,076
So this is much less directly
ripped because I actually had some

212
00:13:53,076 --> 00:13:56,321
social justice issues
with the [INAUDIBLE].

213
00:13:56,321 --> 00:14:01,936
[LAUGH] So this is a little bit modified,
but it's still mostly from the paper.

214
00:14:01,936 --> 00:14:06,563
So it is an intervention to improve
post-prison outcomes through a prisoner

215
00:14:06,563 --> 00:14:07,851
education program.

216
00:14:07,851 --> 00:14:12,816
So some contexts, the participants
being tired of being in prison,

217
00:14:12,816 --> 00:14:15,307
them having future aspirations.

218
00:14:15,307 --> 00:14:20,804
The culture within the prison and of
the prisoners, the culture of the guards.

219
00:14:20,804 --> 00:14:25,239
Having a stable home outside of prison or
something to return to.

220
00:14:25,239 --> 00:14:27,707
The culture of the community outside,

221
00:14:27,707 --> 00:14:31,835
the general social attitudes
toward ex-prisoners in society.

222
00:14:34,319 --> 00:14:38,197
So outcomes, contexts and
mechanisms are multiple, so

223
00:14:38,197 --> 00:14:40,474
one expects multiple outcomes.

224
00:14:40,474 --> 00:14:45,563
This is, they are not connected
to pass fail outcome measures.

225
00:14:45,563 --> 00:14:47,336
There's this idea of outcome patterns.

226
00:14:47,336 --> 00:14:51,930
So it's the intended and unintended
consequences of the intervention.

227
00:14:51,930 --> 00:14:55,061
And it's based on the combinations
of the contexts and mechanisms.

228
00:14:55,061 --> 00:15:02,693
So these mechanisms for people inside of
these contexts produce these outcomes,

229
00:15:02,693 --> 00:15:07,126
is kinda the way it's
[INAUDIBLE] fit together.

230
00:15:07,126 --> 00:15:12,120
And it provides a much more nuanced
understanding of the intervention's

231
00:15:12,120 --> 00:15:14,063
results than a pass/fail.

232
00:15:15,350 --> 00:15:19,944
So here are some examples,
these are more direct.

233
00:15:19,944 --> 00:15:29,600
So, The intervetion is a closed-circuit
TV to reduce crime in parking lots.

234
00:15:29,600 --> 00:15:33,310
One outcome is people spending
less time in parking lots.

235
00:15:33,310 --> 00:15:38,370
Another outcome is a reduction in
crime rate in the parking lots.

236
00:15:38,370 --> 00:15:41,302
You can have secondary outcomes or
more nuanced outcomes.

237
00:15:41,302 --> 00:15:46,170
Well maybe in that outcome pattern,

238
00:15:46,170 --> 00:15:49,830
crime is not dropping at busy times,
but it is dropping.

239
00:15:49,830 --> 00:15:50,992
[INAUDIBLE] times,

240
00:15:50,992 --> 00:15:55,936
so that's sort of given you a much greater
understanding of what's happening.

241
00:15:58,169 --> 00:16:00,900
So I kind of alluded to this and

242
00:16:00,900 --> 00:16:06,940
put it all together into
a context-mechanism-outcome model.

243
00:16:06,940 --> 00:16:09,670
So we have a model for

244
00:16:09,670 --> 00:16:14,707
how interventions activate mechanisms
within contexts to generate an outcome.

245
00:16:15,980 --> 00:16:18,720
So the model will answer what works for
whom, and

246
00:16:18,720 --> 00:16:22,630
in what conditions, going back to
that realist evaluation question.

247
00:16:24,280 --> 00:16:29,840
And then so yeah, so a realist evaluation
then becomes something that is

248
00:16:29,840 --> 00:16:36,640
going to scientifically test your model.

249
00:16:36,640 --> 00:16:40,880
And there's a quote, a fun quote,
that I just quoted because I like it.

250
00:16:40,880 --> 00:16:43,460
The sign of a good intervention is
that it is able to explain the complex

251
00:16:43,460 --> 00:16:45,230
signature of the outcome.

252
00:16:47,480 --> 00:16:52,430
I really like that as a way of thinking
about intervention as opposed to

253
00:16:52,430 --> 00:16:53,800
it's a thing that's going to work or
not work.

254
00:16:55,040 --> 00:16:59,372
At least when we're in
the complex social intervention

255
00:16:59,372 --> 00:17:03,897
world not in a world of [INAUDIBLE]
change your blood level.

256
00:17:06,895 --> 00:17:13,130
Realist research methods are not special,
regular old science applies.

257
00:17:13,130 --> 00:17:18,496
Although, in my opinion,
mixed methods may be particularly

258
00:17:18,496 --> 00:17:23,666
well suited to realist
evaluation's research question.

259
00:17:23,666 --> 00:17:27,845
Gonna break away for a moment, just
give general advice on research methods,

260
00:17:27,845 --> 00:17:31,709
is that the research methods you choose,
whether it's quantitative,

261
00:17:31,709 --> 00:17:35,380
like a survey or qualitative like
you're going to do interviews, or

262
00:17:35,380 --> 00:17:38,419
mixed methods,
where you're going to do some of both and

263
00:17:38,419 --> 00:17:42,130
in the kind of synergy between them,
you're going to learn a thing.

264
00:17:42,130 --> 00:17:45,590
Your research methods always need
to match your research question,

265
00:17:45,590 --> 00:17:50,510
because methods are really
tied to what you want to know.

266
00:17:50,510 --> 00:17:54,180
You can't answer an I need to understand
this with a quantitative measure.

267
00:17:54,180 --> 00:17:58,870
And you can't answer something where
you need an aggregate result for

268
00:17:58,870 --> 00:18:03,380
a mathematical model with
qualitative interviews,

269
00:18:03,380 --> 00:18:05,830
you need sometimes quantitative things.

270
00:18:05,830 --> 00:18:10,531
So I am a mixed methods researcher,
is what I do data on most,

271
00:18:10,531 --> 00:18:15,253
which means that I'm not very
good at either quadrant call.

272
00:18:15,253 --> 00:18:20,938
[LAUGH] But
I love how the interaction between,

273
00:18:20,938 --> 00:18:26,040
and because the question here is a complex

274
00:18:26,040 --> 00:18:31,276
one that, It's

275
00:18:31,276 --> 00:18:35,010
kind of triangulating information from
a lot of different data sources, right?

276
00:18:35,010 --> 00:18:37,709
Your crime rate is a quantitative thing,
but

277
00:18:37,709 --> 00:18:42,537
then you might wanna talk to some people
who regularly park in the lot to find out,

278
00:18:42,537 --> 00:18:45,178
get your sense of what's going on changed.

279
00:18:45,178 --> 00:18:46,374
Did you notice the camera?

280
00:18:46,374 --> 00:18:49,212
There might be qualitative
questions like that.

281
00:18:49,212 --> 00:18:54,144
The study design can also include
rival theories to the intervention

282
00:18:54,144 --> 00:18:57,780
in order to understand what's happening.

283
00:18:57,780 --> 00:19:03,580
One of the things that gets lost by
not putting in a controlled situation,

284
00:19:03,580 --> 00:19:08,660
and accepting that this is an open system,
is that we really don't know if

285
00:19:08,660 --> 00:19:11,700
it's the intervention that's affecting
something versus something else.

286
00:19:11,700 --> 00:19:16,329
So a suggestion that I read about
is to just include in your CMO

287
00:19:16,329 --> 00:19:21,090
some alternate theories,

288
00:19:21,090 --> 00:19:25,480
but within limits,
because it's an open system.

289
00:19:25,480 --> 00:19:29,260
So we need to draw boundaries around
the system of interest somewhere to those

290
00:19:29,260 --> 00:19:37,670
things that you think are most
plausibly going to be happening.

291
00:19:37,670 --> 00:19:41,874
This idea of it being hypothesis testing,

292
00:19:41,874 --> 00:19:48,082
this is the scientific method
cycle with realist evaluation.

293
00:19:48,082 --> 00:19:52,052
You have your theory, your CMO,
you collect data, you analyze it,

294
00:19:52,052 --> 00:19:57,380
does this match what we thought was
happening inside of this intervention?

295
00:19:57,380 --> 00:20:00,074
Adjust the hypothesis, and start again.

296
00:20:00,074 --> 00:20:05,299
I found this more
concrete diagram of that,

297
00:20:05,299 --> 00:20:11,240
that has some of the CMO
language put inside of it.

298
00:20:13,910 --> 00:20:17,800
And it breaks down into phases,
but it's kind of the same diagram.

299
00:20:17,800 --> 00:20:24,910
So you come up with your original program
idea based on your first half of your CMO.

300
00:20:24,910 --> 00:20:29,629
Develop what you're going to test,

301
00:20:29,629 --> 00:20:34,043
which then you collect data to test

302
00:20:34,043 --> 00:20:38,922
this to test your idea about your CMO.

303
00:20:38,922 --> 00:20:43,831
Again, it's so new that the method
should be appropriate to what

304
00:20:43,831 --> 00:20:47,960
questions you're asking,
what's inside your CMO.

305
00:20:47,960 --> 00:20:52,812
Analyse your data, then based on what
you learned, you refine your CMO,

306
00:20:52,812 --> 00:20:56,116
and then you can use that
to refine that feedback.

307
00:20:56,116 --> 00:21:00,622
And, [INAUDIBLE] a little loss, but
there's an arrow that then feeds back in,

308
00:21:00,622 --> 00:21:03,780
and you could potentially
adjust your intervention.

309
00:21:03,780 --> 00:21:07,210
You could target it more cleanly
to who it's working for,

310
00:21:07,210 --> 00:21:09,660
you could do a lot of
things at that point.

311
00:21:10,660 --> 00:21:13,497
It's not linear like your logic model,

312
00:21:13,497 --> 00:21:17,537
it's delightfully difficult and
keep that [INAUDIBLE].

313
00:21:17,537 --> 00:21:22,170
[LAUGH] So, I really wanted to be able to
give you a really great example from my

314
00:21:22,170 --> 00:21:28,020
research, to start wrapping it up.

315
00:21:28,020 --> 00:21:32,638
But I sorta ran out of time so, but
you get my messy research notes.

316
00:21:32,638 --> 00:21:38,390
So this is the EASA Connections project.

317
00:21:38,390 --> 00:21:42,960
I really think that this project is
right now in its final data analysis.

318
00:21:42,960 --> 00:21:45,470
It was not designed as
a realist evaluation,

319
00:21:45,470 --> 00:21:48,260
it was not designed for a realist.

320
00:21:48,260 --> 00:21:53,529
I actually have a, did I put it here?

321
00:21:53,529 --> 00:21:56,040
I guess I didn't.

322
00:21:56,040 --> 00:21:59,460
I actually have a, no,
I did put it in here.

323
00:21:59,460 --> 00:22:01,843
[INAUDIBLE] the number we had to add.

324
00:22:03,583 --> 00:22:04,810
This was the, whoa!

325
00:22:04,810 --> 00:22:08,808
I probably shouldn't have done this, okay.

326
00:22:08,808 --> 00:22:12,618
Yeah, so
we have a traditional change model and

327
00:22:12,618 --> 00:22:16,724
a traditional logic model for
this intervention.

328
00:22:16,724 --> 00:22:20,040
So this was not realist from the start.

329
00:22:20,040 --> 00:22:26,182
However, as I've gotten into
the data that we collected on it,

330
00:22:26,182 --> 00:22:30,161
we have a lot of there's a pretty small n,

331
00:22:30,161 --> 00:22:36,450
there were about 32 people who
went through the intervention.

332
00:22:36,450 --> 00:22:39,692
And you can't do anything statistically
with that, we didn't design it to.

333
00:22:39,692 --> 00:22:41,170
It was a pilot study.

334
00:22:41,170 --> 00:22:43,320
It was not an advocacy study.

335
00:22:43,320 --> 00:22:45,650
We did test measures.

336
00:22:45,650 --> 00:22:50,207
There wasn't statistically significant
change anywhere except for

337
00:22:50,207 --> 00:22:54,309
the stigma measure,
the internalized stigma measure moved.

338
00:22:54,309 --> 00:22:58,478
We also did interviews
with 10 participants, and

339
00:22:58,478 --> 00:23:03,688
we purposefully sampled them to be
a mix of people who had really,

340
00:23:03,688 --> 00:23:10,070
really loved the intervention and gotten
a lot out of it and people who didn't.

341
00:23:10,070 --> 00:23:13,810
And then the intervention
also had contact with a peer.

342
00:23:13,810 --> 00:23:18,742
So it was primarily this web-based
resource that people go look at

343
00:23:18,742 --> 00:23:20,159
on their own time.

344
00:23:20,159 --> 00:23:24,796
But there was also somebody who
had been through the program,

345
00:23:24,796 --> 00:23:29,344
who graduated from the program,
who introduced them to it,

346
00:23:29,344 --> 00:23:35,254
navigated around, told them parts that
were of interest personally to them.

347
00:23:35,254 --> 00:23:40,005
And then contacted them periodically
throughout the next couple of months to

348
00:23:40,005 --> 00:23:44,440
see if they had looked at it, and
to give them some encouragement.

349
00:23:44,440 --> 00:23:47,933
So there was actually
a one on one contact.

350
00:23:47,933 --> 00:23:52,446
With here throughout this.

351
00:23:52,446 --> 00:23:54,837
So we had asked them about the peer and

352
00:23:54,837 --> 00:23:58,323
then the peer also collected
a bunch of information.

353
00:23:58,323 --> 00:24:01,600
So the peer,
we have fidelity data from the peer,

354
00:24:01,600 --> 00:24:05,950
like did they deliver the intervention
as the protocol said.

355
00:24:05,950 --> 00:24:10,823
But they also made notes about their
interaction with the participant and

356
00:24:10,823 --> 00:24:16,259
things that the participant may told them
about how they were using the resource.

357
00:24:16,259 --> 00:24:20,860
So we have a big mush of quantitative and
qualitative data.

358
00:24:20,860 --> 00:24:22,960
We also talk to clinicians so

359
00:24:22,960 --> 00:24:27,640
we have a lot of different stakeholder
perspectives involved in it.

360
00:24:27,640 --> 00:24:32,317
And I've started kinda combing through

361
00:24:32,317 --> 00:24:36,723
this data and got really interested.

362
00:24:36,723 --> 00:24:40,476
The realist evaluation stuff went for
me, thought this is cool,

363
00:24:40,476 --> 00:24:44,041
I should look into it to this
going on to Zoom during that time.

364
00:24:44,041 --> 00:24:50,995
And now I'm really looking at these
connections data and wondering if I can.

365
00:24:50,995 --> 00:24:56,731
Apparently, you are allowed
to use realist valuation

366
00:24:56,731 --> 00:25:01,360
in a post [INAUDIBLE],
they help synthesis.

367
00:25:01,360 --> 00:25:04,109
I don't know,
maybe the synthesis of something else.

368
00:25:04,109 --> 00:25:08,389
But I'm kinda thinking about,
can I apply some of the realist ideas and

369
00:25:08,389 --> 00:25:11,611
the CMO models to understanding
this intervention.

370
00:25:11,611 --> 00:25:16,293
Particularly because we really do
have data from people in the and

371
00:25:16,293 --> 00:25:19,045
a lot of information about their context.

372
00:25:19,045 --> 00:25:22,329
We know what agency that
they were working out of,

373
00:25:22,329 --> 00:25:24,760
we know who their counsellors were.

374
00:25:24,760 --> 00:25:30,593
We know their demographics,
we know how long you're in program.

375
00:25:30,593 --> 00:25:37,077
And then we have a lot of subjective
information from the peer navigator nodes,

376
00:25:37,077 --> 00:25:41,934
peer navigator discussions and
these kind of interviews.

377
00:25:41,934 --> 00:25:47,090
So it had started with this, actually,
as I was going through the data.

378
00:25:47,090 --> 00:25:51,286
I started making these piles
of who this was useful for and

379
00:25:51,286 --> 00:25:53,305
who it wasn't useful for.

380
00:25:53,305 --> 00:25:56,245
And then looked at it and
was like, wait a minute,

381
00:25:56,245 --> 00:25:59,536
what I'm writing down here
are contexts [LAUGH] I think.

382
00:25:59,536 --> 00:26:02,686
I think I'm writing down context.

383
00:26:02,686 --> 00:26:05,093
And then I started thinking back,
I'm like,

384
00:26:05,093 --> 00:26:07,264
well, what are the mechanisms for this?

385
00:26:07,264 --> 00:26:08,716
What are the outcomes?

386
00:26:08,716 --> 00:26:13,485
We had a self-determination
outcome from the logic model, but

387
00:26:13,485 --> 00:26:18,961
I honestly think the most interesting
part of this is the anti-stigma and

388
00:26:18,961 --> 00:26:21,840
the peer connectedness pieces.

389
00:26:21,840 --> 00:26:26,510
So, I don't know,
you'd probably get people.

390
00:26:26,510 --> 00:26:30,362
I remember from my seminar days people
would come into with these beautifully

391
00:26:30,362 --> 00:26:32,021
completed projects and be like,

392
00:26:32,021 --> 00:26:34,763
I am now going to tell you
the findings from my science.

393
00:26:34,763 --> 00:26:38,322
So this is a look into
before we get to that stage.

394
00:26:38,322 --> 00:26:41,551
The kind of mass and wreckage of hike.

395
00:26:41,551 --> 00:26:42,461
>> Hikings.
>> [LAUGH]

396
00:26:42,461 --> 00:26:44,748
>> What is this, what are you?

397
00:26:44,748 --> 00:26:50,820
Let's look at that,
how can we use it to better target?

398
00:26:50,820 --> 00:26:54,260
So I'm already kind of thinking ahead to,

399
00:26:54,260 --> 00:26:59,129
well, if it is the case that
the people who this works best for

400
00:26:59,129 --> 00:27:03,444
are people who are disengaged
from the main program.

401
00:27:03,444 --> 00:27:09,649
And who are very nervous about
communication, and who needs support but

402
00:27:09,649 --> 00:27:17,180
don't like going directly to people to get
it and are feeling very uncomfortable.

403
00:27:17,180 --> 00:27:22,983
And if this is really, if we are starting
to see certain contexts that

404
00:27:22,983 --> 00:27:28,096
the mechanisms of our intervention
are really working for.

405
00:27:28,096 --> 00:27:32,914
Then maybe as the ISA program thinks
ahead to how they're gonna implement this

406
00:27:32,914 --> 00:27:37,522
further and integrate it further into
their own programming and systems.

407
00:27:37,522 --> 00:27:41,107
This might be something that
they make a point to offer to

408
00:27:41,107 --> 00:27:44,322
those people who are not
coming to your meetings.

409
00:27:44,322 --> 00:27:47,715
Or it has potential implications for

410
00:27:47,715 --> 00:27:53,412
changes to program delivery that
I think is much more useful.

411
00:27:53,412 --> 00:27:58,964
Than looking at this and saying,
well, yes, people liked it,

412
00:27:58,964 --> 00:28:03,542
and it was feasible, but
it did not move our outcomes.

413
00:28:03,542 --> 00:28:10,033
Okay, so we're doing that efficacy
trial without making modifications.

414
00:28:10,033 --> 00:28:12,330
>> Well,
it goes to inclusion criteria too.

415
00:28:12,330 --> 00:28:14,892
>> Yeah.
>> If it doesn't work for certain reasons,

416
00:28:14,892 --> 00:28:18,362
then don't try to solve the problem for
that particular.

417
00:28:18,362 --> 00:28:20,799
>> Yeah, yeah, and
don't try to push it at them.

418
00:28:20,799 --> 00:28:24,881
>> Yeah, yeah.
>> Or give everyone the information that

419
00:28:24,881 --> 00:28:29,543
the people who really
need to get together up

420
00:28:29,543 --> 00:28:34,331
here to connect up with
this are the people who

421
00:28:34,331 --> 00:28:39,520
are introverts or
whatever comes up out of this.

422
00:28:39,520 --> 00:28:44,069
So yeah, I mean, this is my question
right now, is can I create and

423
00:28:44,069 --> 00:28:47,992
test this demo with the data right now,
I think?

424
00:28:47,992 --> 00:28:54,089
And next Wednesday I'm doing this trip and
I'm gonna ask this question [LAUGH].

425
00:28:54,089 --> 00:28:57,561
My data, and
when it comes to that [INAUDIBLE],

426
00:28:57,561 --> 00:29:00,151
think something that I can offer.

427
00:29:00,151 --> 00:29:03,950
>> Can I just interrupt you, just say,
how many of you know what trip is?

428
00:29:05,470 --> 00:29:06,751
Well, of course I know some of you do.

429
00:29:06,751 --> 00:29:07,545
>> [LAUGH]
>> But

430
00:29:07,545 --> 00:29:10,220
it's scholarly research in progress.

431
00:29:10,220 --> 00:29:14,820
It's a weekly opportunity to share
research designs, preliminary or

432
00:29:14,820 --> 00:29:18,975
very well-developed to get
feedback from peers, peer issue.

433
00:29:18,975 --> 00:29:24,476
And it's a wonderful resource tool
that just isn't very well-publicized,

434
00:29:24,476 --> 00:29:27,160
I guess in this room in particular.

435
00:29:27,160 --> 00:29:30,011
>> The focus is on social
determinants of health research.

436
00:29:30,011 --> 00:29:34,967
So you are gonna get, it's all human
subjects research pretty much,

437
00:29:34,967 --> 00:29:36,735
so it's human subject.

438
00:29:36,735 --> 00:29:41,653
But it does run a huge gamut
from clinical team way much

439
00:29:41,653 --> 00:29:46,694
with their stuff and
understanding gentrification.

440
00:29:46,694 --> 00:29:50,332
I mean, it's all over,
it's all over the map, but

441
00:29:50,332 --> 00:29:53,842
it is within
>> So you mean if you wanna know more?

442
00:29:53,842 --> 00:29:55,816
>> Yeah.
>> I just wanted that-

443
00:29:55,816 --> 00:29:57,441
>> Not forget to do that.

444
00:29:57,441 --> 00:29:59,480
>> You wanna see the mess
that people make.

445
00:29:59,480 --> 00:30:04,355
The point behind it is,
is that you can bring your research at

446
00:30:04,355 --> 00:30:08,958
any stage of development but
it needs to not be finished.

447
00:30:08,958 --> 00:30:11,461
>> [LAUGH]
>> So you get people who come in with

448
00:30:11,461 --> 00:30:15,438
like, I have some ideas but
I don't know if any of them are good or

449
00:30:15,438 --> 00:30:18,251
any of these are good for
a research project.

450
00:30:18,251 --> 00:30:21,899
And then you have people like the one
that I'm doing next week, it's like,

451
00:30:21,899 --> 00:30:24,936
we've already done the project,
we've collected the data.

452
00:30:24,936 --> 00:30:30,291
And now I'm trying to figure
out how this data can

453
00:30:30,291 --> 00:30:37,487
have the most impact on the field
as I put it together, it's fun.

454
00:30:37,487 --> 00:30:42,502
>> [INAUDIBLE] you seem to have a criteria

455
00:30:42,502 --> 00:30:47,363
in which in here and stand.

456
00:30:47,363 --> 00:30:49,937
So you can use that criteria
to buck your sample and

457
00:30:49,937 --> 00:30:52,651
see if you have the sample
worked on that [INAUDIBLE].

458
00:30:52,651 --> 00:30:57,499
>> Yeah, I tend to do that because we have

459
00:30:57,499 --> 00:31:01,439
a bunch of evaluation data, so

460
00:31:01,439 --> 00:31:06,752
one of the questions is how useful was it?

461
00:31:06,752 --> 00:31:15,125
So we can certainly look at the population
that said that it's useful and

462
00:31:15,125 --> 00:31:20,765
the population that said
that it's not useful.

463
00:31:20,765 --> 00:31:26,135
>> I don't know if that your own criteria
would also specify the output for

464
00:31:26,135 --> 00:31:27,489
this [INAUDIBLE].

465
00:31:27,489 --> 00:31:33,840
>> I don't have [COUGH] I
have in the qualitative.

466
00:31:33,840 --> 00:31:37,412
I mean,
I don't know if they were outgoing people,

467
00:31:37,412 --> 00:31:41,158
I guess I'm gonna set that out,
not well thought out.

468
00:31:41,158 --> 00:31:46,140
But in their interviews,
there were people who said,

469
00:31:46,140 --> 00:31:49,814
I would rather talk to some Someone.

470
00:31:49,814 --> 00:31:52,669
I would rather have
a conversation with someone, and

471
00:31:52,669 --> 00:31:55,342
those were the people who
didn't find it useful.

472
00:31:55,342 --> 00:31:59,731
And the people who did find it useful
were saying things like, yeah,

473
00:31:59,731 --> 00:32:05,101
these were questions, I was afraid that,
I couldn't talk to my, I was embarrassed

474
00:32:05,101 --> 00:32:09,890
to bring this up to my counselor, or
I was afraid to tell this to my parents.

475
00:32:09,890 --> 00:32:12,787
So I was able to just print it out and
give it to them.

476
00:32:12,787 --> 00:32:18,580
And so I'm making a judgment about
whether they were outgoing or not.

477
00:32:18,580 --> 00:32:21,465
It was more like how
they like to interact.

478
00:32:21,465 --> 00:32:27,309
[INAUDIBLE] separate from
the [INAUDIBLE] as well.

479
00:32:27,309 --> 00:32:30,132
Is that [INAUDIBLE]?

480
00:32:30,132 --> 00:32:32,889
>> Yeah.

481
00:32:32,889 --> 00:32:37,225
So if I'm understanding it, sorry,

482
00:32:37,225 --> 00:32:41,980
outgoing or not would be a context and so.

483
00:32:41,980 --> 00:32:46,589
>> [INAUDIBLE], at my limited
understanding of realist evaluation.

484
00:32:46,589 --> 00:32:47,355
>> Yeah.
>> Yes.

485
00:32:47,355 --> 00:32:50,408
[LAUGH]
>> It seems like more, yeah,

486
00:32:50,408 --> 00:32:52,180
who it works for.

487
00:32:53,410 --> 00:32:55,400
Something that going forward.

488
00:32:55,400 --> 00:33:01,950
>> People who prefer to get information,
By themselves.

489
00:33:01,950 --> 00:33:07,652
>> Yeah, but I mean, that would be
more of a context than a mechanism.

490
00:33:07,652 --> 00:33:11,833
>> Yeah I believe so
I mean correct me if I-

491
00:33:11,833 --> 00:33:13,014
>> [LAUGH]

492
00:33:13,014 --> 00:33:14,390
>> I don't know if you've been cringing.

493
00:33:14,390 --> 00:33:16,841
>> [LAUGH]
>> You're doing fine.

494
00:33:16,841 --> 00:33:20,420
>> [LAUGH]
>> But yeah, I mean in my limited

495
00:33:20,420 --> 00:33:25,893
understanding right now,
the mechanisms are more like,

496
00:33:25,893 --> 00:33:32,385
what is the thing that's actually
happening in this intervention.

497
00:33:32,385 --> 00:33:38,060
Right, so
the mechanisms that I kind of brainstormed

498
00:33:38,060 --> 00:33:43,104
at midnight last night
were that peer navigator

499
00:33:43,104 --> 00:33:47,658
contact, relatableness of the content.

500
00:33:47,658 --> 00:33:52,383
Because there was a lot of peer,
there were videos that there's a need.

501
00:33:52,383 --> 00:33:58,250
That anytime resource delivery, like these
are things that the intervention did.

502
00:33:58,250 --> 00:34:04,492
They were what the activities
of the intervention

503
00:34:04,492 --> 00:34:09,406
that people were actually engaged in.

504
00:34:09,406 --> 00:34:12,832
User control, I think, is a mechanism.

505
00:34:14,314 --> 00:34:20,404
Could be, again, this is all in my
limited first pass understanding.

506
00:34:20,404 --> 00:34:24,011
But my impression from what I've read so

507
00:34:24,011 --> 00:34:28,892
far is that the contexts
are about the characteristics

508
00:34:28,892 --> 00:34:33,486
that are present within the-
>> More than

509
00:34:33,486 --> 00:34:34,970
characteristics of the subject.

510
00:34:34,970 --> 00:34:36,520
>> It's the characteristics
of the subjects but

511
00:34:36,520 --> 00:34:41,490
it's also the characteristics of the place

512
00:34:41,490 --> 00:34:44,506
at the time and the pressures that-
>> Delivery mechanisms.

513
00:34:44,506 --> 00:34:47,008
[CROSSTALK] and things like that.

514
00:34:47,008 --> 00:34:54,500
And then the mechanisms are like what
is the intersection actually doing?

515
00:34:54,500 --> 00:34:56,960
It's giving you early morning nutrition,

516
00:34:56,960 --> 00:35:01,989
it's giving you user controlled
information about psychosis.

517
00:35:04,030 --> 00:35:06,811
>> Alerts versus having to ask for
information, maybe,

518
00:35:06,811 --> 00:35:09,050
would be a different mechanism.

519
00:35:09,050 --> 00:35:14,774
>> Yeah, actually the peer motivational,

520
00:35:14,774 --> 00:35:21,492
reminders from peers,
might be the mechanism.

521
00:35:21,492 --> 00:35:26,654
So strengths, strong explanatory
capability, ability to understand and

522
00:35:26,654 --> 00:35:31,830
refine interventions to better
target them or meet specific needs.

523
00:35:31,830 --> 00:35:38,274
It doesn't sacrifice so many potentially
important skills that activation does.

524
00:35:38,274 --> 00:35:42,413
Avoids a one size fits all solutions
because you can have something that works,

525
00:35:42,413 --> 00:35:45,027
you can have different
classes through your CMO.

526
00:35:48,197 --> 00:35:51,358
It might present a more
useful set of findings for

527
00:35:51,358 --> 00:35:55,996
real world social science interventions
than that binary [INAUDIBLE].

528
00:35:55,996 --> 00:36:00,941
In my opinion, it's also a potentially
useful tool in systems thinking for

529
00:36:00,941 --> 00:36:06,275
understanding generative structures
because the CMOs is a terrific structure,

530
00:36:06,275 --> 00:36:09,618
and it comes from a strong
system's perspective.

531
00:36:09,618 --> 00:36:14,324
Limitations and considerations,
you might not need it or

532
00:36:14,324 --> 00:36:18,655
want it,
straight-forward drug types [INAUDIBLE].

533
00:36:18,655 --> 00:36:23,497
It ain't broke,
it is complex to conceptualize and

534
00:36:23,497 --> 00:36:29,455
implement, but we're in this room so
that might make us happy.

535
00:36:29,455 --> 00:36:33,300
It does not give a definitive
universal answer to does it work.

536
00:36:33,300 --> 00:36:36,350
So if you have a funder or
somebody who wants the answer,

537
00:36:36,350 --> 00:36:38,250
it's not going to give it.

538
00:36:38,250 --> 00:36:43,330
It instead gives complex answers,
but they do like that.

539
00:36:43,330 --> 00:36:45,010
And it's not well known to the US and

540
00:36:45,010 --> 00:36:47,560
we've had some conversations
about why that might be.

541
00:36:47,560 --> 00:36:51,450
But there might be a little
bit of extra work you'll

542
00:36:51,450 --> 00:36:55,880
have to do to convince someone that
this is a valid thing, that yes,

543
00:36:55,880 --> 00:37:00,397
you are going to get
scientific results from this.

544
00:37:00,397 --> 00:37:02,734
So that's it, and this is the [INAUDIBLE].

545
00:37:02,734 --> 00:37:04,930
And I want to mention a couple things and

546
00:37:04,930 --> 00:37:07,790
then we'll just go to discussion for
the rest.

547
00:37:07,790 --> 00:37:13,264
So one is I thank you so
much and a rock star for

548
00:37:13,264 --> 00:37:18,640
starting, kicking off PSU
realist evaluations study group.

549
00:37:18,640 --> 00:37:23,100
Which you can talk to us,
our next meeting is actually next Tuesday.

550
00:37:24,370 --> 00:37:27,130
These are some links and

551
00:37:27,130 --> 00:37:32,237
resources that have largely come from Ana,

552
00:37:32,237 --> 00:37:36,939
and I can send you the slides [INAUDIBLE].

553
00:37:36,939 --> 00:37:39,349
>> And I can post it if you're
willing to post this presentation.

554
00:37:39,349 --> 00:37:39,922
>> Yes, please.

555
00:37:39,922 --> 00:37:41,767
>> I can just put it up on our website.
>> Yes, please post it, please post it.

556
00:37:41,767 --> 00:37:42,430
>> Okay.

557
00:37:42,430 --> 00:37:45,408
>> And you had a question
like a hundred years ago.

558
00:37:45,408 --> 00:37:47,260
>> We kinda got into to, but

559
00:37:47,260 --> 00:37:52,643
it seems like the main difference to
me between the realist evaluation,

560
00:37:52,643 --> 00:37:57,430
what you presented as the classical
approach, is the context.

561
00:37:57,430 --> 00:38:01,692
It seems like mechanisms to me and
what mediums and determinants,

562
00:38:01,692 --> 00:38:04,517
there's at least some translation there.

563
00:38:04,517 --> 00:38:05,990
I mean.

564
00:38:05,990 --> 00:38:10,180
>> I don't feel like there is, so

565
00:38:10,180 --> 00:38:15,606
the methods,
>> Maybe

566
00:38:15,606 --> 00:38:18,546
it was just the particular mediums
in the slide that you showed.

567
00:38:18,546 --> 00:38:19,868
I could have got the wrong impression.

568
00:38:24,086 --> 00:38:27,777
So the mechanisms are actually,
I think the mechanisms,

569
00:38:27,777 --> 00:38:32,550
in my eyes, my thinking, are actually
a little closer to the activities.

570
00:38:32,550 --> 00:38:33,200
>> Yeah, okay.

571
00:38:33,200 --> 00:38:35,882
>> In a logic model, but not really.

572
00:38:35,882 --> 00:38:41,550
They're not, the activities are the thing

573
00:38:41,550 --> 00:38:46,680
you do to a person, but the mechanisms
do away with that kind of thinking.

574
00:38:46,680 --> 00:38:51,798
You're not doing a thing to a person,
there's something that,

575
00:38:51,798 --> 00:38:57,104
there's an active engagement
between something you're doing and

576
00:38:57,104 --> 00:38:59,628
the person who is engaged in it.

577
00:38:59,628 --> 00:39:00,961
>> Yeah.

578
00:39:00,961 --> 00:39:04,483
>> So I don't know that it analogs,
I mean,

579
00:39:04,483 --> 00:39:09,639
in some there are I mean, you're still-
>> Connected.

580
00:39:09,639 --> 00:39:15,919
>> It's,yeah, and the determinants in
these types of models, they're usually

581
00:39:15,919 --> 00:39:22,210
some kind of construct that you're gonna
use a complicated measure to get at.

582
00:39:22,210 --> 00:39:29,317
So we actually had quantitative scales
that we used to get those things,

583
00:39:29,317 --> 00:39:34,792
and the idea is, well,
we believe that if you stimulate

584
00:39:34,792 --> 00:39:40,153
someone's competence,
relatedness and autonomy,

585
00:39:40,153 --> 00:39:44,700
that based on DC's Self
determination theory that

586
00:39:46,740 --> 00:39:50,220
the outcome of that is, that you're
gonna have increased self determination.

587
00:39:50,220 --> 00:39:53,376
So, we're gonna create
an intervention that is going to

588
00:39:53,376 --> 00:39:56,216
activate people's
confidence with knowledges.

589
00:39:56,216 --> 00:40:00,786
And they're relatedness to peers and
the program, and their autonomy and

590
00:40:00,786 --> 00:40:04,080
self directing their own treatment plans.

591
00:40:04,080 --> 00:40:10,291
So, that's not the same thing as some-
>> Yeah,

592
00:40:10,291 --> 00:40:13,813
I just mean that, it seems to be the max.

593
00:40:13,813 --> 00:40:16,791
You can imagine if you've got a deviant,

594
00:40:16,791 --> 00:40:21,705
or you say, what mechanisms
do you think are effective?

595
00:40:21,705 --> 00:40:23,664
And so, it at least kind of appears, and

596
00:40:23,664 --> 00:40:28,090
it seems like the context though is
something that doesn't appear here at all.

597
00:40:28,090 --> 00:40:33,024
And so they are asking a question,
who gets what worked for isn't something

598
00:40:33,024 --> 00:40:37,582
that shows up in the classical [CROSSTALK]
>> Yes, it's mostly absent,

599
00:40:37,582 --> 00:40:39,990
I mean, it's not totally absent.

600
00:40:39,990 --> 00:40:45,175
There's a more business action
model thing that I didn't show for

601
00:40:45,175 --> 00:40:48,650
reasons of unnecessary complexity.

602
00:40:48,650 --> 00:40:54,114
But there are actually models in
some of those cluster intervention,

603
00:40:54,114 --> 00:40:58,840
evaluation, Things that
do look at content.

604
00:41:00,190 --> 00:41:02,490
Except that it's more of
an abbreviate context.

605
00:41:02,490 --> 00:41:10,560
It's like what the context would be
the EASA early psychosis program.

606
00:41:11,620 --> 00:41:17,559
It wouldn't be, A liking

607
00:41:17,559 --> 00:41:21,532
of on demand information delivery.

608
00:41:21,532 --> 00:41:29,230
So, I think it's there, but
it's there on a kind of a higher level.

609
00:41:29,230 --> 00:41:33,560
It's not considered in the same way,
they're considered as like

610
00:41:33,560 --> 00:41:38,950
a container that you're identifying or
something that's impacting it.

611
00:41:38,950 --> 00:41:44,590
But it's not,
The mapping just really breaks down.

612
00:41:44,590 --> 00:41:47,640
This is really totally,
totally different, and

613
00:41:47,640 --> 00:41:49,480
maybe I made a mistake
to show the classical.

614
00:41:49,480 --> 00:41:53,075
But one of the reasons why I had done that
was because, I wanted to show that this is

615
00:41:53,075 --> 00:41:58,045
just a different way of
thinking about the entire day.

616
00:41:58,045 --> 00:41:58,939
>> Before creating a catalog-
>> [LAUGH]

617
00:41:58,939 --> 00:42:01,239
>> A couple of voices that I mind,

618
00:42:01,239 --> 00:42:03,975
give the people either a chance.

619
00:42:03,975 --> 00:42:07,982
>> I also wanna give them
a chance to interact-

620
00:42:07,982 --> 00:42:08,780
>> No, no, don't-

621
00:42:08,780 --> 00:42:10,137
>> Let me give that a minute, yeah,

622
00:42:10,137 --> 00:42:12,656
I just like to try to represent them,
cuz it's much harder for

623
00:42:12,656 --> 00:42:14,020
them to get a voice into the room.

624
00:42:14,020 --> 00:42:19,788
Although it does work, it's just harder,
so what I had to do is.

625
00:42:19,788 --> 00:42:23,953
Okay, so Catherine says you've already
answered her question [INAUDIBLE] follow

626
00:42:23,953 --> 00:42:24,630
a comments.

627
00:42:24,630 --> 00:42:28,054
But there's a comment from Nick saying,
the systems evaluation topic

628
00:42:28,054 --> 00:42:31,308
of interest group of the American
Evaluation Association released

629
00:42:31,308 --> 00:42:32,460
a high level document.

630
00:42:32,460 --> 00:42:38,478
And the chat provided a link to that
which might be complimentary, so.

631
00:42:38,478 --> 00:42:39,979
>> Cool.

632
00:42:39,979 --> 00:42:43,041
>> I'll make sure that
capture this chat window, and

633
00:42:43,041 --> 00:42:45,186
that way we could maybe figure out.

634
00:42:45,186 --> 00:42:47,213
I don't know how to share everything, but

635
00:42:47,213 --> 00:42:49,368
we'll do the best we can
to share what we can.

636
00:42:49,368 --> 00:42:54,264
>> You can probably copy and
paste the link, and send [INAUDIBLE]

637
00:42:54,264 --> 00:42:56,804
>> Yeah, so, please, go ahead.

638
00:42:56,804 --> 00:43:00,083
[LAUGH]
>> I think to your point,

639
00:43:00,083 --> 00:43:06,468
I'm trying to think what the most
[INAUDIBLE] I think that They,

640
00:43:07,631 --> 00:43:16,322
Mechanism that one of the primary
differences between the sort of trials and

641
00:43:16,322 --> 00:43:22,310
we're gonna tried our positive approach,
right?

642
00:43:23,790 --> 00:43:29,608
And that's another way, then we talk about
this, is realists are realists, right?

643
00:43:29,608 --> 00:43:30,213
On specified to this [INAUDIBLE]

644
00:43:30,213 --> 00:43:30,933
>> [CROSSTALK]
>> But

645
00:43:30,933 --> 00:43:31,568
there is realist philosophy [INAUDIBLE].

646
00:43:31,568 --> 00:43:36,220
>> So one way of thinking
about the difference is,

647
00:43:36,220 --> 00:43:40,524
positivist trials approaches RCTs, say,

648
00:43:40,524 --> 00:43:44,142
did you get the intervention or not?

649
00:43:44,142 --> 00:43:48,640
And then could it work or not, right?

650
00:43:48,640 --> 00:43:53,515
Real Estate,
let's stop talking about interventions,

651
00:43:53,515 --> 00:43:57,600
let's stop talking about
what the program did.

652
00:43:57,600 --> 00:44:01,661
Let's stop talking about what
happened to the person instead,

653
00:44:01,661 --> 00:44:05,350
let's talk about the program theory,
right?

654
00:44:05,350 --> 00:44:06,687
What is it that we think is going on?

655
00:44:06,687 --> 00:44:12,590
And that then creates the possibility,
the likelihood, right?

656
00:44:12,590 --> 00:44:16,190
Of that outcome occurring.

657
00:44:16,190 --> 00:44:21,290
So context in that, so for one, it's a
very different sort of level of analysis.

658
00:44:21,290 --> 00:44:25,993
They talk about realism as
having ontological, right?

659
00:44:25,993 --> 00:44:29,998
So people like me who do program
evaluation, we just measure whether or

660
00:44:29,998 --> 00:44:32,414
not somebody has a program or not, right?

661
00:44:32,414 --> 00:44:37,229
So that's the top of
the iceberg in [INAUDIBLE] and

662
00:44:37,229 --> 00:44:41,701
real estate, no,
we need to go much deeper.

663
00:44:41,701 --> 00:44:46,981
And start testing what it is I think
is really going on when we make this,

664
00:44:46,981 --> 00:44:51,031
whatever the program is
available to people, right?

665
00:44:51,031 --> 00:44:54,634
So they said, and you could go all the way
down to measuring cortisol levels, right?

666
00:44:54,634 --> 00:44:59,623
Or you could have other measures of
motivation, and you've got to pick the one

667
00:44:59,623 --> 00:45:04,320
that has much more purchase given
whatever it is your context is, right?

668
00:45:04,320 --> 00:45:08,502
But so, for realist,
the mechanism is really,

669
00:45:08,502 --> 00:45:13,405
I like to talk about it in terms of,
what is the resource?

670
00:45:13,405 --> 00:45:16,955
What is the thing that's being
made available to someone or

671
00:45:16,955 --> 00:45:21,320
that's being taken away as
a result of the intervention?

672
00:45:21,320 --> 00:45:24,930
So, I think for some of yours,
if you're talking about the peer to peer

673
00:45:24,930 --> 00:45:29,170
interaction, I mean, that sort of works,
but really, what do you mean?

674
00:45:29,170 --> 00:45:34,757
Is it that somebody's getting
information about something

675
00:45:34,757 --> 00:45:39,584
from somebody that they can identify with,
right?

676
00:45:39,584 --> 00:45:44,447
So you get pretty specific
about what that resource is,

677
00:45:44,447 --> 00:45:52,120
what are the characteristics of that
thing that are actually causal, right?

678
00:45:52,120 --> 00:45:56,360
And are likely to have the response,
right?

679
00:45:56,360 --> 00:45:58,701
On the part of the program recipient.

680
00:45:58,701 --> 00:46:06,180
So, we've got resources, and you can
either say, reasoning or responses.

681
00:46:06,180 --> 00:46:13,050
So, what's causing that change,
and what do we mean?

682
00:46:13,050 --> 00:46:14,510
>> In a specific context.

683
00:46:14,510 --> 00:46:19,314
>> Well, I'm getting to context-
>> [LAUGH]

684
00:46:19,314 --> 00:46:20,447
>> Now in context.

685
00:46:20,447 --> 00:46:25,914
I mean, there are people hate context,
there is so

686
00:46:25,914 --> 00:46:32,409
much time spent on context on
the realist list serve within.

687
00:46:32,409 --> 00:46:34,987
Context is not necessarily setting,

688
00:46:34,987 --> 00:46:38,620
it's not necessarily the conditions even,
right?

689
00:46:38,620 --> 00:46:44,008
Context is something
that can either trigger,

690
00:46:44,008 --> 00:46:52,496
either facilitate or
inhibit the mechanism from firing, right?

691
00:46:52,496 --> 00:46:59,120
And so, often, the reason is not
the setting, is because if you say, well,

692
00:46:59,120 --> 00:47:05,445
this was a classroom and not a prison,
that explains nothing, right?

693
00:47:05,445 --> 00:47:10,600
You have to say what it is, and
connect it in a logical way, right?

694
00:47:10,600 --> 00:47:12,220
>> To-
>> To the mechanism.

695
00:47:12,220 --> 00:47:15,703
So, I think that's one of
the key differences again,

696
00:47:15,703 --> 00:47:20,795
between most of the work that I do is,
we don't bother with explanation at all.

697
00:47:20,795 --> 00:47:24,665
We have all these proxies, that oftentimes
I think are really does lead me, right?

698
00:47:24,665 --> 00:47:29,293
We measure race and ethnicity, and gender,
and income, and we do a [SOUND] piss for

699
00:47:29,293 --> 00:47:30,377
job on all of that.

700
00:47:30,377 --> 00:47:33,740
And then we analyze and then we say,
but we've got these correlations.

701
00:47:34,800 --> 00:47:37,737
So then let's design programs.

702
00:47:37,737 --> 00:47:41,660
Just because there's a statistically
significant correlation does,

703
00:47:41,660 --> 00:47:44,677
They don't say I don't care
about the [CROSSTALK].

704
00:47:44,677 --> 00:47:47,676
>> And just cuz there isn't
a statistically significant correlation

705
00:47:47,676 --> 00:47:49,154
doesn't mean anything either.

706
00:47:49,154 --> 00:47:53,396
I mean, and I wasn't expecting
statistically significant correlations

707
00:47:53,396 --> 00:47:57,233
from any of our measures,
because people used the intervention for

708
00:47:57,233 --> 00:47:59,860
one month and it wasn't designed for that.

709
00:47:59,860 --> 00:48:02,270
At the same time,
just because it didn't show up,

710
00:48:02,270 --> 00:48:07,070
you listen to people talking about
their experience and the fears.

711
00:48:07,070 --> 00:48:09,660
You listen to these interviews and
people are like,

712
00:48:09,660 --> 00:48:14,950
this really did have a huge impact and
you can't capture

713
00:48:14,950 --> 00:48:20,660
that with the measure.

714
00:48:20,660 --> 00:48:24,860
>> Right, I mean, there is a lively
debate in the realist community about

715
00:48:24,860 --> 00:48:28,014
whether there's such
thing as a realist in.

716
00:48:28,014 --> 00:48:31,070
Right, again,
because most RCTs are not realists.

717
00:48:31,070 --> 00:48:34,780
They're not looking,
there's no ontological depth, right?

718
00:48:35,840 --> 00:48:40,605
They're not testing theories in
any kind of meaningful way, right?

719
00:48:40,605 --> 00:48:42,247
They're, did you get it or not.

720
00:48:42,247 --> 00:48:47,422
And now we catch our p value,
it's either thumbs up or thumbs down.

721
00:48:47,422 --> 00:48:53,794
Could you design an RCT that
is fundamentally realist?

722
00:48:53,794 --> 00:48:56,830
Well, I'll give you the link to
the website, you can read that [LAUGH].

723
00:48:56,830 --> 00:48:59,797
Or there's a great video actually
of a [CROSSTALK] conference.

724
00:48:59,797 --> 00:49:03,969
>> [CROSSTALK] with the video on.

725
00:49:03,969 --> 00:49:05,228
Is it one of the ones?

726
00:49:05,228 --> 00:49:08,579
>> I don't think it is, cuz it's Leads.

727
00:49:08,579 --> 00:49:12,617
>> Would it be on some level just for

728
00:49:12,617 --> 00:49:17,852
every context I can
prepare doing an RCT or

729
00:49:17,852 --> 00:49:23,060
something like it to try and see
>> [LAUGH]

730
00:49:23,060 --> 00:49:25,436
>> I don't think that's feasible [LAUGH].

731
00:49:25,436 --> 00:49:26,005
I mean these are really hard to do.

732
00:49:26,005 --> 00:49:29,275
>> Realists also reject
the hierarchy of evidence, right?

733
00:49:29,275 --> 00:49:33,806
Which is one of my big things is, this
whole push to evidence based practice and

734
00:49:33,806 --> 00:49:37,526
sort of saying that it's all
about statistical significance.

735
00:49:37,526 --> 00:49:44,630
I mean, I think we afford way,
way more to that design than we should.

736
00:49:44,630 --> 00:49:52,432
What happens then, realists results
then feel less mushier, right?

737
00:49:52,432 --> 00:49:56,366
But I would argue, you're only
uncomfortable with that because you drank

738
00:49:56,366 --> 00:49:58,814
the Kool Aid about the RCTs
in the first place.

739
00:49:58,814 --> 00:50:02,231
>> But could you bring random assignment
into the notion of realist evaluation?

740
00:50:02,231 --> 00:50:04,577
>> Realist or agnostic,
when it comes to methods?

741
00:50:04,577 --> 00:50:07,316
What is the question
you're trying to answer?

742
00:50:07,316 --> 00:50:11,547
>> [LAUGH] [CROSSTALK]
>> Is that the appropriate method for

743
00:50:11,547 --> 00:50:14,410
the question that you're asking?

744
00:50:14,410 --> 00:50:18,919
>> Yeah, and again,
I want to say let's turn that on,

745
00:50:18,919 --> 00:50:25,686
don't just use that to attack our
qualitative methods or different sorts of,

746
00:50:25,686 --> 00:50:31,450
let's really look at some of those
RCTs and say, do they deserve?

747
00:50:31,450 --> 00:50:36,155
I mean in my field, if you had two
published studies that met the RCT

748
00:50:36,155 --> 00:50:40,021
criteria, you make it onto
the clearing house list and

749
00:50:40,021 --> 00:50:45,752
the federal government will pay for
those programs with child welfare dollars.

750
00:50:45,752 --> 00:50:47,652
Okay, if you look at the clearinghouse,

751
00:50:47,652 --> 00:50:50,056
there's two studies that
found positive effects.

752
00:50:50,056 --> 00:50:52,090
Let's not even get into effects studies.

753
00:50:52,090 --> 00:50:57,761
Let's not even get into whether or
not a p value of 0.05 makes sense, right?

754
00:50:57,761 --> 00:51:02,295
But if there are two,
even if there are 69 that found no effect,

755
00:51:02,295 --> 00:51:05,768
just as long as we have
none that showed real harm.

756
00:51:05,768 --> 00:51:09,549
I mean, my point there is just that,
we're giving RCTs,

757
00:51:09,549 --> 00:51:13,420
we're giving trials,
a huge intellectual pass, right?

758
00:51:15,910 --> 00:51:18,571
And in fact, and especially for-
>> [LAUGH]

759
00:51:18,571 --> 00:51:19,214
>> [CROSSTALK] I personally don't

760
00:51:19,214 --> 00:51:20,755
think that RCTs are actually appropriate
for complex social [INAUDIBLE].

761
00:51:20,755 --> 00:51:25,422
>> Absolutely.
>> And I think that's part of the problem

762
00:51:25,422 --> 00:51:32,957
is that RTCs come through this
very positivist look at the world,

763
00:51:32,957 --> 00:51:37,073
and that works for certain things.

764
00:51:37,073 --> 00:51:41,251
You want that or gravity, and
you want that for the pill and

765
00:51:41,251 --> 00:51:43,880
you want that for certain things.

766
00:51:43,880 --> 00:51:48,420
But when you're taking something
out of those contexts where things

767
00:51:48,420 --> 00:51:55,022
are controllable and bound by physics,
and you go into the wicked problem

768
00:51:55,022 --> 00:52:00,240
zone of social problems and social service
systems and are trying to apply science.

769
00:52:02,310 --> 00:52:05,796
I feel like there was a big push in
social sciences to be like, well,

770
00:52:05,796 --> 00:52:10,030
we're gonna do real science, and to prove
it, we're gonna be like this is this.

771
00:52:10,030 --> 00:52:13,557
But the real world is
really way too messy.

772
00:52:13,557 --> 00:52:18,367
I mean, that's part of what
excites me about [INAUDIBLE] is

773
00:52:18,367 --> 00:52:22,990
that it starts giving me
different ways to think about.

774
00:52:22,990 --> 00:52:28,269
How do you do analytic scientific
work in something where you can't

775
00:52:28,269 --> 00:52:33,548
break things down, where they're
not linear, where they're not

776
00:52:33,548 --> 00:52:38,840
going to be clean the way need they
need to be clean for an [INAUDIBLE].

777
00:52:38,840 --> 00:52:42,309
>> I'm gonna say that we should thank-
>> [LAUGH]

778
00:52:42,309 --> 00:52:43,518
>> [INAUDIBLE] for this talk right now and

779
00:52:43,518 --> 00:52:45,124
then continue the discussion afterwards,
so.

780
00:52:45,124 --> 00:52:50,574
>> [APPLAUSE]
>> Thank you.

781
00:52:50,574 --> 00:52:53,789
If you need to [CROSSTALK] screwing
around, that's okay, we'll keep talking.

782
00:52:53,789 --> 00:52:55,456
>> I'll leave the recorder on for
a little longer.

783
00:52:55,456 --> 00:53:00,631
>> You guys talked about very
positivists versus realists but

784
00:53:00,631 --> 00:53:03,435
he mentioned constructivism.

785
00:53:03,435 --> 00:53:04,939
How does that fit into it?

786
00:53:04,939 --> 00:53:07,461
How does the constructivists approach?

787
00:53:07,461 --> 00:53:10,812
>> [INAUDIBLE]
>> [LAUGH] [INAUDIBLE] think

788
00:53:10,812 --> 00:53:11,964
it's good when we both talk.

789
00:53:11,964 --> 00:53:14,596
>> [INAUDIBLE]
>> I actually like [INAUDIBLE].

790
00:53:14,596 --> 00:53:19,532
I like the idea of
co-constructive realities and

791
00:53:19,532 --> 00:53:27,172
that's what is always going on is that
we're engaged in reality creation and

792
00:53:27,172 --> 00:53:32,124
meaning making with all
that we're countering.

793
00:53:32,124 --> 00:53:38,927
When I do qualitative research and when I
approach my analysis, I can [INAUDIBLE].

794
00:53:38,927 --> 00:53:42,288
Constructivists, that was
kind of my starter part, but

795
00:53:42,288 --> 00:53:45,584
I haven't worked through it
[INAUDIBLE] more recently.

796
00:53:45,584 --> 00:53:53,882
I like the idea that there
are many different realities and

797
00:53:53,882 --> 00:53:59,310
that we're going to allow them this.

798
00:53:59,310 --> 00:54:03,777
I'm not the best person to talk to you
about startups and ontology, though,

799
00:54:03,777 --> 00:54:05,574
because I'm very pragmatic.

800
00:54:05,574 --> 00:54:09,493
And at the end of the day,

801
00:54:09,493 --> 00:54:15,852
what I really want to do is this.

802
00:54:15,852 --> 00:54:18,560
So I see this as like ways
to not fall into traps.

803
00:54:18,560 --> 00:54:25,193
There are ways to, I think, help keep
examining and reflecting on biases and

804
00:54:25,193 --> 00:54:30,413
mechanics inside of analysis and
data collection.

805
00:54:30,413 --> 00:54:35,506
But then there are people who really

806
00:54:35,506 --> 00:54:42,310
going into the kind of
metaphysics of science.

807
00:54:42,310 --> 00:54:44,610
That's never gonna be me [LAUGH].

808
00:54:44,610 --> 00:54:47,309
[CROSSTALK] They all kind
of take the paradigm that,

809
00:54:47,309 --> 00:54:50,467
I think about paradigms the same
way I think about methods.

810
00:54:50,467 --> 00:54:54,987
Is take the paradigm that is going to get
you the most useful answer to your search

811
00:54:54,987 --> 00:54:56,009
question [LAUGH].

812
00:54:56,009 --> 00:54:57,737
So I mean,
sometimes that is positivist, right?

813
00:54:57,737 --> 00:55:01,984
Again, if you're talking about physics,

814
00:55:01,984 --> 00:55:05,895
then you don't wanna, or maybe you do.

815
00:55:05,895 --> 00:55:10,128
You wanna think about multiple realities
if you're going into a quantum level

816
00:55:10,128 --> 00:55:10,775
but [LAUGH].

817
00:55:10,775 --> 00:55:15,807
I'm kind of in favor of like all
paradigms being just lens and

818
00:55:15,807 --> 00:55:21,036
ways that we are filtering or
focusing our biases or our views,

819
00:55:21,036 --> 00:55:25,280
or the way that we're
approaching our analysis and

820
00:55:25,280 --> 00:55:28,061
our interpretation [INAUDIBLE].

821
00:55:28,061 --> 00:55:30,336
No, you were about to speak when I
kinda interrupted you with the process.

822
00:55:30,336 --> 00:55:33,094
>> It could be that I'm no
longer good at [INAUDIBLE].

823
00:55:33,094 --> 00:55:35,818
>> [LAUGH]

824
00:55:35,818 --> 00:55:40,781
>> So I have to ask this Okay,

825
00:55:40,781 --> 00:55:46,631
is it commendable to focus the [INAUDIBLE]

826
00:55:46,631 --> 00:55:51,971
because I have a conventional view.

827
00:55:51,971 --> 00:55:57,739
I would say the heart of science is
the possibility of false conclusions.

828
00:55:57,739 --> 00:56:03,763
I think that's as bottom line
as things get in science and

829
00:56:03,763 --> 00:56:08,795
if you can't falsify anything,
that I think.

830
00:56:08,795 --> 00:56:14,375
>> Well, I think that I mean,
I haven't seen someone who's not actually

831
00:56:14,375 --> 00:56:19,777
done with this, I can imagine in my
head that your CMO might not test.

832
00:56:19,777 --> 00:56:25,299
If you're testing your GMO as a theory,
it might not cast doubt.

833
00:56:25,299 --> 00:56:29,955
There might be pathways in that you
don't find [CROSSTALK] evidence over

834
00:56:29,955 --> 00:56:32,373
the data that the whole thing could.

835
00:56:32,373 --> 00:56:34,317
I couldn't see it not.

836
00:56:34,317 --> 00:56:39,386
>> You could just accidentally falsify,
you have to design for that, I think.

837
00:56:39,386 --> 00:56:40,357
>> Well.

838
00:56:40,357 --> 00:56:43,358
>> Well I don't know,
people might not have actually,

839
00:56:43,358 --> 00:56:45,167
everybody could've hated it.

840
00:56:45,167 --> 00:56:50,149
>> What you're picturing, as,
for me a very complex model, and

841
00:56:50,149 --> 00:56:57,011
you're doing one intervention, right on
the email, the intervention many people.

842
00:56:57,011 --> 00:57:02,050
Many people are involved, still it's
one intervention that whow could you

843
00:57:02,050 --> 00:57:07,089
possibly tease out all the different
connections of multiple mechanisms,

844
00:57:07,089 --> 00:57:10,092
multiple contexts, multiple algorithms.

845
00:57:10,092 --> 00:57:10,911
>> You can.

846
00:57:10,911 --> 00:57:14,310
>> But again, you can't.

847
00:57:14,310 --> 00:57:19,370
And so given that, maybe you could
say overall this thing failed this,

848
00:57:20,400 --> 00:57:22,810
but you will never know what about it?

849
00:57:24,680 --> 00:57:25,410
>> That's not true.

850
00:57:25,410 --> 00:57:30,956
I think that's exactly your critique of
the trials approach is it could fail and

851
00:57:30,956 --> 00:57:35,535
you could know next to nothing
about what it was about it, right?

852
00:57:35,535 --> 00:57:39,610
But I think another issue here and
we don't have time and I'm definitely not

853
00:57:39,610 --> 00:57:44,147
able to articulate this very well, but the
positivity literature does a great job.

854
00:57:44,147 --> 00:57:48,381
Is the other another difference
between realism, the positives and

855
00:57:48,381 --> 00:57:52,060
the constructivist is their
understanding of pausality.

856
00:57:52,060 --> 00:57:56,016
And that's really foundational
approach to their realist approach and

857
00:57:56,016 --> 00:57:58,114
to your question, but can't go there.

858
00:57:58,114 --> 00:57:58,704
I gotta go.

859
00:57:58,704 --> 00:58:00,699
[CROSSTALK]
>> [LAUGH]

860
00:58:00,699 --> 00:58:02,299
>> And I think there's also,

861
00:58:02,299 --> 00:58:05,082
I mean speaking in
the positivity literature,

862
00:58:05,082 --> 00:58:07,041
there's a big thing about that.

863
00:58:07,041 --> 00:58:11,666
And it's like it gets you down your system
because you can always come up with some

864
00:58:11,666 --> 00:58:15,014
other reason and
you can get down into this little nuance.

865
00:58:15,014 --> 00:58:20,853
And if they need, you can go on and on
forever, but there are some like larger,

866
00:58:20,853 --> 00:58:25,660
more likely things that need to
be examined at something point.

867
00:58:25,660 --> 00:58:31,456
Those bits are statistically inactive way,

868
00:58:31,456 --> 00:58:34,939
but in a reality way not useful.

869
00:58:34,939 --> 00:58:38,223
>> [CROSSTALK]
>> Like what someone had for breakfast,

870
00:58:38,223 --> 00:58:44,085
maybe relate to something but you make
a decision at some point to bound your

871
00:58:44,085 --> 00:58:49,110
system that you're not going to go and
data on what people had for

872
00:58:49,110 --> 00:58:52,950
breakfast because it's
unlike to be that huge.

873
00:58:52,950 --> 00:58:53,881
>> Well.
[CROSSTALK]

874
00:58:53,881 --> 00:58:54,896
>> Go ahead.

875
00:58:54,896 --> 00:58:59,616
>> It seems to me that
you're proposing a model in

876
00:58:59,616 --> 00:59:04,586
the beginning, it through your study.

877
00:59:04,586 --> 00:59:08,826
You're learning to study
among other things

878
00:59:08,826 --> 00:59:12,781
division that would give
you not one results, but

879
00:59:12,781 --> 00:59:18,070
a set of percentage is now close
to that initial beginning effect.

880
00:59:19,470 --> 00:59:24,163
Now, if you call that
a positivist hypothesis,

881
00:59:24,163 --> 00:59:31,151
I need to be given a different sample and
you use that for the set of tests.

882
00:59:31,151 --> 00:59:38,485
You would then be able to apply statistics
to the new hypothesis versus your sample.

883
00:59:38,485 --> 00:59:41,067
>> [CROSSTALK] to the question
that you're asking.

884
00:59:41,067 --> 00:59:41,817
>> Right,
I think when you were talking about.

885
00:59:41,817 --> 00:59:45,140
>> Science questions.

886
00:59:45,140 --> 00:59:50,134
>> In science too is, I mean we live
in the world of we get paid to do

887
00:59:50,134 --> 00:59:54,183
research and
evaluate social programs, right?

888
00:59:54,183 --> 00:59:58,586
Which is a very different set of
questions that realist don't say

889
00:59:58,586 --> 01:00:01,718
all the positives you're not so
stupid here.

890
01:00:01,718 --> 01:00:02,390
That's not it at all.

891
01:00:02,390 --> 01:00:09,727
It's about what is appropriate given what
it is that we're trying to understand.

892
01:00:09,727 --> 01:00:11,801
>> Yes, but I'm not suggesting that.

893
01:00:11,801 --> 01:00:16,167
What I'm suggesting is
your original method let

894
01:00:16,167 --> 01:00:20,007
teaches you something about population.

895
01:00:20,007 --> 01:00:21,567
>> Yeah, you could do it like that.

896
01:00:21,567 --> 01:00:25,292
>> And
then you begin to find the population and

897
01:00:25,292 --> 01:00:30,123
see if your method works on
one group versus another, but

898
01:00:30,123 --> 01:00:34,869
you can't do but
you do it on the second step.

899
01:00:34,869 --> 01:00:39,808
And that second step is a perfectly
valid way of making you look at

900
01:00:39,808 --> 01:00:42,104
where you'll find.

901
01:00:42,104 --> 01:00:43,968
>> And that's part of that
derivative [CROSSTALK].

902
01:00:43,968 --> 01:00:49,963
>> That gets a plus viable process.

903
01:00:49,963 --> 01:00:55,081
>> Well again, I think I realist
would say falsifiable as applied to

904
01:00:55,081 --> 01:01:01,050
these sorts of questions is, I mean
we're playing fast and lose with that.

905
01:01:01,050 --> 01:01:04,121
With what we willing defined as false and

906
01:01:04,121 --> 01:01:09,281
again that comes down to what realist
understand about [INAUDIBLE].

907
01:01:09,281 --> 01:01:14,010
>> [INAUDIBLE] specific way,
having one group work with another.

908
01:01:14,010 --> 01:01:15,276
That can be tested.

909
01:01:15,276 --> 01:01:20,124
>> Right, but that's a somewhat
different notion of falsified delay and

910
01:01:20,124 --> 01:01:24,417
that is this worth it or do we get a,
You're gonna not [CROSSTALK]

911
01:01:24,417 --> 01:01:27,043
>> It is not what you're paid to do

912
01:01:27,043 --> 01:01:30,679
whether or
not we wanna believe it or not and

913
01:01:30,679 --> 01:01:36,250
that's what they want you do to and
that's what their paid to do.

914
01:01:36,250 --> 01:01:41,858
[CROSSTALK]
>> Yeah, and after 25 years we have.

915
01:01:41,858 --> 01:01:49,597
[CROSSTALK]
>> Thanks for having me.

916
01:01:49,597 --> 01:01:52,483
>> Thanks for coming and
I'm gonna stop the recording and-

917
01:01:52,483 --> 01:01:58,043
>> [CROSSTALK]

918
01:01:58,043 --> 01:02:01,032
>> [LAUGH]