tag:blogger.com,1999:blog-68848821873482423752024-03-14T05:19:19.666-07:00EvolvingA blog about evolving humans and evolving psychological methodsUnknownnoreply@blogger.comBlogger10125tag:blogger.com,1999:blog-6884882187348242375.post-67503193413044633732019-05-27T22:04:00.000-07:002019-05-27T22:06:27.764-07:00The Big Thing Relationships Researchers Don’t Study<br />
<i><b>You might think that relationship researchers would investigate how relationships form. But we don’t—largely because relationship formation is surprisingly difficult to study.</b></i><br />
<br />
<div class="MsoNormal">
When Eli Finkel and I conducted our speed-dating studies
over a decade ago, we were hopeful that we would see our participants go on to
form actual romantic relationships. That is, we thought we would be able to
follow participants from their very first impressions of each other through the
formation of a dating relationship.</div>
<div class="MsoNormal">
<br /></div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div class="MsoNormal">
About a third of our speed-daters went on to have something like a coffee
date with someone they met at speed-dating. But in the weeks and months following the initial event, only about <b>5%</b> of participants reported having a casual or serious
dating relationship with one of their fellow speed-daters.</div>
<br />
<div class="MsoNormal">
Was this low percentage something weird about speed-dating?
Maybe Northwestern University undergrads have terrible social skills? Well, in
2008, Eli and I were part of a co-ed kickball league in Chicago, and ~150 twenty-
and thirty-somethings from this league got together on a weekly basis to compete,
eat, and imbibe a few alcoholic beverages.
We administered a survey to try to get a sense of how often people were
forming relationships across this league.</div>
<div style="text-align: right;">
</div>
<div class="MsoNormal">
<br /></div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxdeSHCh05Gqe1zti3jpjFTFKm2ypuEbIh8e2n_DaQ2n-CWgePzVHPlMHsyF_yQLDykDwaLMKRQNUGxMJfQcSGrzFcxwYWkVFjdXNP13J7yogkrFKblsRItGtl_521y5lb2ksoNDn6QRB8/s1600/TheGap.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="415" data-original-width="670" height="247" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxdeSHCh05Gqe1zti3jpjFTFKm2ypuEbIh8e2n_DaQ2n-CWgePzVHPlMHsyF_yQLDykDwaLMKRQNUGxMJfQcSGrzFcxwYWkVFjdXNP13J7yogkrFKblsRItGtl_521y5lb2ksoNDn6QRB8/s400/TheGap.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The period of time between the moment two people meet and the<br />
formation of a committed relationship is empirically hazy.</td></tr>
</tbody></table>
<div class="MsoNormal">
Was relationship formation more common in this group? Yes… a
whopping <b>7%</b>. That is, 7% of single people who took part in
the kickball league formed a relationship with someone else in the league over
the course of a 2-3 month season. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In the years since, there haven’t been many more attempts
to capture relationship formation as it happens; I could probably count these
studies on one hand. The path from strangers to relationship partners is
extremely hard to study. And, in my view, it remains one of the greatest untapped
reservoirs of interesting psychological phenomena.</div>
<div class="MsoNormal">
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;"></span><br />
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;"></span></div>
<div class="MsoNormal">
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;">================</span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Close relationship scholars are pretty good at studying
initial attraction between strangers, and we are really good at studying people
who agree that they are currently romantic partners. <b>But what about the time period between initial attraction and "real relationship" status? It’s more or less missing entirely from our literature. </b></div>
<div class="MsoNormal">
<br /></div>
<div style="text-align: right;">
</div>
<div class="MsoNormal">
As Eli Finkel, Jeff Simpson, and I argue in this recent (open
access) <a href="http://pauleastwick.com/s/EastwickFinkelSimpson2019PsychInquiry.pdf" target="_blank">Psych Inquiry article</a>, this gap in the literature is a big problem. Why? Three reasons:</div>
<div class="MsoNormal">
<br /></div>
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://media.giphy.com/media/1r8SIOqBC1J06SHwBm/giphy.gif" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="258" data-original-width="480" height="172" src="https://media.giphy.com/media/1r8SIOqBC1J06SHwBm/giphy.gif" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ethan Hawke might have been able to pick up a stranger on a<br />
train, but most people form romantic relationships with<br />
acquaintances and friends, not strangers.</td></tr>
</tbody></table>
<div class="MsoNormal">
<span style="text-indent: -0.25in;">1. This period of time is not short. The average is
about a year. There are exceptions, of course, but people typically form
romantic relationships by drawing from their network of preferred-sex friends
and acquaintances. Successfully chatting up a stranger a la Be<i>fore Sunrise</i> is not the norm.</span><span style="text-indent: -0.25in;"> <b>So we are missing out on about a year's worth </b>of presumably important psychological processes. </span></div>
<div class="MsoNormal" style="text-indent: 0px;">
<span style="text-indent: -0.25in;"></span><br />
<span style="text-indent: -0.25in;"></span></div>
<div class="MsoNormal" style="text-indent: 0px;">
<span style="text-indent: -0.25in;">2. There are many studies that examine whether
individual differences predict relationship outcomes. But very rarely do these studies get
measures of individual differences that are uncontaminated by a current
relationship (i.e., measured <i>before </i>the current relationship had the chance to shape them).</span><span style="text-indent: -0.25in;"> </span><span style="text-indent: -0.25in;">Sure, some studies recruit
participants right as they start dating each other, but even these studies are
not capturing the true beginning of the relationship. If you just started dating someone who has been your friend for the past year, she could have been boosting your self-esteem or exacerbating your attachment
anxiety for that entire time. This means that <b>even though we think we’re studying the
effect of individual differences on relationship processes, we may actually be studying the effect of relationship processes on relationship processes</b>.</span></div>
<div class="MsoNormal" style="text-indent: 0px;">
<span style="text-indent: -0.25in;"></span><br />
<span style="text-indent: -0.25in;"></span></div>
<div class="MsoNormal" style="text-indent: 0px;">
<span style="text-indent: -0.25in;">3. The fields of evolutionary psychology and close
relationships both inspire a lot of work on romantic relationships, yet remain surprisingly disconnected given this shared focus. The mystery period between initial attraction and acknowledged romantic relationship might be hindering integration across these two fields:
Most evolutionary psychological studies resemble studies of initial attraction
(e.g., participants evaluate a stranger depicted in a photograph), whereas close relationships studies often focus on existing relationships (e.g., participants report on a dating partner
over time).<b> </b>Sometimes, scholars suggest that studies of initial attraction
capture short-term mating whereas studies of established relationships capture
long-term mating, but this suggestion imbues a methodological distinction with
intense theoretical weight (i.e., are you capturing two theoretically distinct mating strategies or simply measuring two points along a normative arc?). <b>By filling in the missing time period between initial attraction and relationship formation, we may be able to shed better light on the true distinctions between short-term and long-term relationships</b>, since it very well might take </span><a href="http://pauleastwick.blogspot.com/2018/02/intuitive-and-impossible-what-do-short.html" style="text-indent: -0.25in;">weeks
or months</a><span style="text-indent: -0.25in;"> after an initial interaction before people figure out whether
someone is friend material, hookup material, or bring-home-to-meet-grandma
material.</span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<o:p></o:p></div>
<div class="MsoListParagraphCxSpLast">
<br /></div>
<div class="MsoNormal">
<a href="http://pauleastwick.com/s/EastwickFinkelSimpson2019PsychInquiry.pdf" target="_blank">Our article</a> offers a meta-theoretical framework for thinking
about time across the entirety of a romantic relationship, from the moment two
people actually meet. You can also read several very thoughtful commentaries on our article from close relationships, sexuality, and evolutionary
psychological scholars who are deeply committed to studying these issues as
well (see <a href="https://www.tandfonline.com/doi/full/10.1080/1047840X.2019.1585731" target="_blank">here</a>, <a href="https://www.tandfonline.com/doi/full/10.1080/1047840X.2019.1577073" target="_blank">here</a>, <a href="https://www.tandfonline.com/doi/full/10.1080/1047840X.2019.1577096" target="_blank">here</a>, and <a href="https://www.tandfonline.com/doi/full/10.1080/1047840X.2019.1585733" target="_blank">here</a>, and see <a href="http://pauleastwick.com/s/EastwickFinkelSimpson2019PsychInquiryReply.pdf" target="_blank">here </a>for our reply). <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;">================</span></div>
<div class="MsoNormal">
<br /></div>
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6884882187348242375.post-89717054730728349992019-05-16T06:08:00.002-07:002019-05-16T06:09:54.821-07:00Testing the replicability of claims about a sex difference: A brief update<i>A public commitment to update my own beliefs in response to a planned analysis I haven’t seen yet (part 3)</i><br />
<br />
Over the last year or so, several people have asked me if I had any updates about the data sharing quandary that I covered in <a href="http://pauleastwick.blogspot.com/2018/03/going-on-record-via-preregistration.html" target="_blank">part 1</a> and <a href="http://pauleastwick.blogspot.com/2018/03/testing-a-sex-difference.html" target="_blank">part 2</a> of this series. Now, I do.<br />
<br />
To recap: As part of an effort to make sure I was willing to update my beliefs in response to empirical data (even if the data disconfirmed those beliefs), I preregistered an <a href="https://osf.io/nqt6r/" target="_blank">analysis plan</a> in March of 2018 to test the replicability of a sex difference in the effect of attractiveness on marital satisfaction. The relevant data would have been the published data columns from a recent <a href="https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000127" target="_blank">JPSP article</a> (which reported a different sex difference and only tangentially mentioned the one I would be testing in the Discussion). The owner of the data, Dr. McNulty, felt that my analysis was too close to a graduate student’s paper that was in the works and asked that I wait until the student’s paper was published, and so that’s what we did.<br />
<br />
In February 2019, the graduate student’s <a href="https://journals.sagepub.com/doi/full/10.1177/0146167219832337" target="_blank">paper </a>was published online in PSPB. I wrote to the authors, and after some back and forth, they sent me the data on April 24, 2019.<br />
<br />
Importantly, they sent me the data on the condition that I would only use the data to verify the exact analysis published in the PSPB article. Dr. McNulty explained that he wants to add additional data to the PSPB dataset before reporting the sex difference in the effect of attractiveness on marital satisfaction.<br />
<br />
<div class="MsoNormal">
<b>So, I am permitted to report that I have verified the published analysis that is reported in the PSPB (a three-way interaction that moderates the two-way interaction that I am interested in). </b>But I am not permitted to report the underlying two-way interaction (i.e., the analyses that could assess the replicability of the sex difference tested in Meltzer, McNulty, Jackson, & Karney, 2014, and in Eastwick, Luchies, Finkel, & Hunt, 2014).*</div>
<div>
<br />
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;">================</span><br />
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;"><br /></span></div>
<div>
<div class="MsoNormal">
This has all been a strange foray into the complexities of data sharing. These issues are thorny, people have strong opinions on both sides, and I don’t want to spend my time right now trying to push things further with PSPB or any other relevant governing body.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Nevertheless, I do want to reiterate that this whole episode wasn’t originally about data sharing. It was actually about preregistration and being willing to update a strongly held scientific belief in light of new data that could have gone either way. It was about the usefulness of declaring publicly what results would be persuasive, and what results would not be persuasive. It was about specifying an analysis plan as a means of improving one’s ability to differentiate signal from noise, and vice versa.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
When it comes to increasing our understanding of sex differences in the appeal of attractiveness in marriages, a persuasive contribution would have been enabled by a preregistered analysis plan that constrained the many researcher degrees of freedom (e.g., stopping rules for data collection, planned covariates) that happen to characterize this research area. It’s a missed opportunity that these hard-to-collect data won’t be able to do that.</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
* In the 3-way interaction analysis in the PSPB, the covariates are not quite the same as what is reported in the original Meltzer et al. (2014) article. I would need to remove some covariates and shift around some others to reproduce the Meltzer et al. (2014) analysis; Dr. McNulty has asked me not to do this.</div>
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;"></span><br />
<div class="MsoNormal">
<br /></div>
</div>
<div>
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;"><br /></span></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6884882187348242375.post-35148563672989024202018-08-07T09:57:00.000-07:002018-08-07T09:57:11.963-07:00What is a Mate Preference?<i>If you study nonhuman animals, there is one answer to this question. But if you study humans, there are two.</i><br />
<br />
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPKV0rJ3Pgp1rRvyJUKqaFVZXfs6WK7fecNnjMYD5WAur9-nbuJchOPtryomH0bLIJfYVzonEKGZY_B0DrWcCkrhGsSl5UNBCTJn2_ByN_O-y3ZYlfTGzMI9knhAhdieYSeHA1zizcqm7H/s1600/Doublebowerbird.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="439" data-original-width="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPKV0rJ3Pgp1rRvyJUKqaFVZXfs6WK7fecNnjMYD5WAur9-nbuJchOPtryomH0bLIJfYVzonEKGZY_B0DrWcCkrhGsSl5UNBCTJn2_ByN_O-y3ZYlfTGzMI9knhAhdieYSeHA1zizcqm7H/s1600/Doublebowerbird.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Top</b>: Interior decorating skills - a highly desirable <br />
attribute in male bowerbirds<br />
<br />
<b>Bottom</b>: One operationalization of the preference <br />
for "nest fanciness" in bowerbirds, <i>r </i>= .54 <br />
(Borgia, 1995). x-axis = # of decorations;<br />
y-axis = # of copulations.</td></tr>
</tbody></table>
Here is a male bowerbird. Like other members of his species, he is a natural interior decorator: He uses pieces of colorful plastic and glass to make his nest look as fancy as possible.<br />
<br />
Now, imagine you wish to test the hypothesis that females have a preference for males who are able to construct fancier nests. You might examine the extent to which (a) the nest fanciness of several male bowerbirds predicts (b) the extent to which female bowerbirds want to mate with them. In fact, this association is fairly strong (see graph at right), suggesting that bowerbird females have a strong preference for the attribute “nest fanciness” in their mates.<br />
<br />
Meanwhile, if you study humans rather than bowerbirds, you might want to test the hypothesis that women have a parallel mate preference for fanciness of abode. You could examine the extent to which (a) the apartment fanciness of several men predicts (b) the extent to which women want to date each of them.<br />
<div style="text-align: right;">
</div>
<br />
I could be wrong, but I don’t think anyone has conducted this study with humans. And that might be because there is a second, completely different way of studying preferences for attributes when working with human participants: You just ask them. After all, asking is much easier: Humans (unlike bowerbirds) can simply rate the appeal of the attribute “fancy apartment” on a 1-9 scale. [1]<br />
<br />
But here’s the problem: Asking humans about their mate preferences assesses their ideas about the attributes they like and dislike, rather than the extent to which an attribute actually drives their mate preferences in real life. In other words, these two mate preferences are not the same construct. Rather than providing a quick-and-easy measurement shortcut to the human analogue of bowerbird mate preferences, rating scales provide a tool for measuring a different—and perhaps uniquely human—type of mate preference.<br />
<br />
<span style="background-color: white; color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif; font-size: 13.2px;">=================</span><br />
<br />
As <a href="http://pauleastwick.com/s/LedgerwoodEastwickSmithPSPRInPress.pdf" target="_blank">this new paper</a> discusses, these two types of preferences are distinct enough that they deserve different names. As a nod to the animal behavior literature, we call the first one—the association between (a) the level of an attribute in each of a series of targets and (b) liking for each of those targets—<i>a functional preference</i>. [2] We call the second one a <i>summarized preference</i> because it reflects a person’s evaluative summary of the attribute as an overall concept. A functional preference for an attribute is the extent to which the attribute drives liking for a set of targets (e.g., the extent to which the intelligence of a potential partner drives the extent to which you like them). A summarized preference for an attribute is the extent to which a person likes the attribute itself, as a concept (e.g., your evaluation of the attribute “intelligence in a romantic partner”).<br />
<br />
The paper linked above reviews how functional and summarized preferences fundamentally differ in a number of ways. For one, they have different evolutionary origins. Functional preferences should exist in any species that possesses an evaluative mechanism (e.g., these food sources are good, and these are bad). But a summarized preference requires an organism to be able to evaluate an attribute as an abstraction—an evolutionarily much more recent ability—and it seems plausible that only humans can do this. What’s more, functional and summarized preferences tend to be biased by different sources of information, and they may have different downstream consequences.<br />
<br />
In the existing mate preferences literature, the summarized preference is the construct you will see most often. That’s okay if you intend to study people’s ideas about the attributes they like. But it’s not a shortcut to studying functional preferences: Depending on the context, the correlation between a summarized and functional preference for the same attribute ranges from <i>r</i> = ~.00 (if people are evaluating live interaction partners) to <i>r</i> = ~.20 (if people are evaluating photographs). In other words, although both constructs are interesting, they’re not the same thing.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhFTTHqXcG7LANQZU9uiS7nkGuljddj_x96VqnNNDSTWlG5pU9lP19pM_pAvSOyjEEW4y9S_N2ETzcnshcCO9jl-W-QCRIt5kHDARRojizYgAyDmMcMWU0fnOC8zDaMvWXFw90e3722SmT/s1600/Sevenstudies.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="320" data-original-width="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhFTTHqXcG7LANQZU9uiS7nkGuljddj_x96VqnNNDSTWlG5pU9lP19pM_pAvSOyjEEW4y9S_N2ETzcnshcCO9jl-W-QCRIt5kHDARRojizYgAyDmMcMWU0fnOC8zDaMvWXFw90e3722SmT/s1600/Sevenstudies.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Seven studies in the mating domain examining the correlation between<br />
summarized and functional preferences for the same attribute. Correlations<br />
approximate <i>r </i>= .20 when people rate partners depicted in photographs and<br />
<i>r </i>= .00 when people rate partners face-to-face.<br />
<br />
Adapted from Ledgerwood, Eastwick, & Smith (in press, <i>PSPR</i>)</td></tr>
</tbody></table>
<br />
And if we want to understand mate preferences in humans, we have to stop conflating summarized and functional preferences; <b>overlooking this distinction creates a construct validity nightmare</b>. For example, many scholars take inspiration from the animal literature on mating to generate new predictions about human mating. But they then test these predictions with summarized preferences—preferences that have no conceptual parallel in nonhuman animals and would not have been subject to the same evolutionary pressures. If you are generating predictions about human mate preferences with evolutionary relevance, your construct—nine times out of ten—is the functional preference.<br />
<br />
So if you work with humans, you can choose to study how strongly attributes predict evaluative outcomes (i.e., functional preferences), or you can study people’s ideas about the attributes they like (i.e., summarized preferences), or you can study both and the relationship between them. Just be mindful of and clear about which construct(s) you are studying. If you work with nonhuman animals, you are almost surely studying functional preferences...unless you have figured out how to get bowerbirds to fill out a pen-and-paper survey.<br />
<br />
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<br /></div>
<div style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<hr align="left" size="1" width="33%" />
<br /></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;">Borgia, G. (1995). Complex male display and female choice in the spotted bowerbird: Specialized functions for different bower decorations. <i>Animal Behaviour, 49</i>, 1291-1301.</span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;"><br /></span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;">Fletcher, G. J., Simpson, J. A., Thomas, G., & Giles, L. (1999). Ideals in intimate relationships. <i>Journal of personality and social psychology, 76</i>, 72-89.</span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;"><br /></span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;">Ledgerwood, A., Eastwick, P. W., & Smith, L. K. (in press). Toward an integrative framework for studying human evaluation: Attitudes towards objects and attributes. <i>Personality and Social Psychology Review</i>. </span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;"><br /></span></span></div>
<div class="MsoPlainText" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;">Wood, D., & Brumbaugh, C. C. (2009). Using revealed mate preferences to evaluate market force and differential preference explanations for mate selection. Journal of personality and social psychology, 96, 1226-1244.</span></span></div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #222222;"><br /></span></div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<br /></div>
<div class="MsoNormal" style="background-color: white;">
<span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13.2px;">[1] </span></span><span style="background-color: transparent; font-size: 13.2px;"><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;">“Nice house or apartment” is, in fact, one item on a very popular mate preference scale (Fletcher et al., 1999) – a scale that I myself use all the time.</span></span></div>
<div class="MsoNormal" style="background-color: white;">
<span style="background-color: transparent; font-size: 13.2px;"><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><br /></span></span></div>
<div class="MsoNormal" style="background-color: white;">
<span style="background-color: transparent; font-size: 13.2px;"><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;">[2] </span></span><span style="background-color: transparent; font-size: 13.2px;"><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;">Wood and Brumbaugh (2009) offer the most comprehensive prior treatment of this construct, which they called a “revealed preference.” We shied away from the “revealed preference” label primarily because behavioral economists use this term to refer to an observable behavior. We mean something far more specific (i.e., the association between an attribute and liking within a set of targets).</span></span></div>
<div>
<br /></div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-6884882187348242375.post-82505123590059769242018-06-18T10:11:00.000-07:002018-06-18T10:11:17.436-07:00How Critical Are We?One perennial issue in the best-practices discussion is
whether or not our discipline is overly critical or not critical enough. When
we evaluate other people’s research, should we be increasing our focus on the
positive aspects or the negative aspects?<br />
<br />
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
My current view is “yes, both,” and the moderator [1] is
whether we are talking about criticism that takes place pre- or
post-publication. Pre-publication, I think we need to dial up the positivity;
post-publication, I think we need to dial up the criticism.<br />
<br /><o:p></o:p>
=================<br />
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Pre-publication
peer-review: We can afford to emphasize the positives</b></div>
<div class="MsoNormal">
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<b style="mso-bidi-font-weight: normal;"></b><br />
<b style="mso-bidi-font-weight: normal;"></b></div>
<div class="MsoNormal">
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhua1wemsWMKrpTd3-VTbenUbH0SjZdhPgzoggtyLvH6GKm4SKS827qUR4XaEfjNnWuUSN96J9ybL4sN3zC8mde6ZXAD5ZQPX1sWqINKujeLoauteRn7uC79s_Vf6Z3uvrlOA9dhue0HKoS/s1600/Criticism.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="306" data-original-width="578" height="169" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhua1wemsWMKrpTd3-VTbenUbH0SjZdhPgzoggtyLvH6GKm4SKS827qUR4XaEfjNnWuUSN96J9ybL4sN3zC8mde6ZXAD5ZQPX1sWqINKujeLoauteRn7uC79s_Vf6Z3uvrlOA9dhue0HKoS/s320/Criticism.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lopsided criticism in peer-review</td></tr>
</tbody></table>
Here are two ways of envisioning the reviewer’s job. One way: The reviewer is the firewall that protects the world from weak manuscripts by pointing out all their flaws. A second way: The reviewer is a knowledgeable colleague who has been asked to offer input on ways to strengthen the manuscript.<br />
<br />
As an editor, I find that—nine times out of ten—the latter approach ultimately makes for a better published literature. Here are two specific reviewer tactics that help tip the pre-publication peer review balance in a more constructive direction:</div>
<div class="MsoNormal">
<span style="text-indent: -0.25in;"></span><br />
<span style="text-indent: -0.25in;"></span>
<span style="text-indent: -0.25in;">1. Reviews are especially helpful when they explain what
a particular subfield can gain from the manuscript. Given that the manuscripts
I handle are rarely (if ever) examining a topic that I myself study, I need
reviewers to tell me about the value that the manuscript will have for them and
their colleagues. Does the manuscript help to define or organize a problem?
Will the findings be useful for other scholars when planning their own studies?
Does the manuscript properly situate the findings in the literature they are
trying to inform? It is extraordinarily informative when a reviewer says “Wow, my subfield really needs an article that does what this manuscript is trying to do.”</span><br />
<span style="text-indent: -0.25in;"><br /></span>
<span style="text-indent: -0.25in;">2. Reviews are especially helpful when they avoid getting
hung up on small imperfections and inconsistencies that make a story less
pretty and glossy. These small imperfections (e.g., a simple main effect was
not significant on one of the five tests, different meta-analytic publication bias analyses reveal different conclusions) are very real parts of science, and
all articles have some. Indeed, I would argue that the picture-perfect (but
impossible) articles of the past emerged because authors and reviewers pushed
each other to scrub away the imperfections. [2]</span><span style="text-indent: -0.25in;"> As an editor, I prefer
reviews that focus in depth on a few big picture concerns, if they exist (e.g.,
missing a large segment of the literature in a review, using the incorrect statistical
test, drawing a conclusion that is not supported by the data). And if there are
no big picture concerns, the reviews should so.</span><br />
<span style="text-indent: -0.25in;"><br /></span></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<o:p></o:p></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Post-publication
peer-review: We can afford to be more critical</b><o:p></o:p><br />
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
I think we have a natural tendency to assume that findings
enter an “official canon” when they are accepted for publication. Canon is for
fiction, like Star Wars and Marvel. As scientists, we must fight the urge to
canonize.<br />
<br />
<div style="text-align: right;">
</div>
</div>
<div class="MsoNormal">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoiK9VV3TyamHojeNPYUOcB1kRR70qdujjKGhnr19tkmvmzz8esQFPvhTsU15YvAOUXNdjnTPPM4KN7OZKmrKibT1kZTN_BnQXdsah4J8e4lNxlNSi4HD9PK9e3Ex5R6gAZ7VN9rbNzs50/s1600/Hulks.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="298" data-original-width="409" height="145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoiK9VV3TyamHojeNPYUOcB1kRR70qdujjKGhnr19tkmvmzz8esQFPvhTsU15YvAOUXNdjnTPPM4KN7OZKmrKibT1kZTN_BnQXdsah4J8e4lNxlNSi4HD9PK9e3Ex5R6gAZ7VN9rbNzs50/s200/Hulks.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fictional scientist Bruce Banner fights the urge to<br />
transform into the Hulk in two eponymous movies,<br />
but only the second one is canon and "counts."<br />
<br />
Real scientists have to fight the urge to canonize<br />
articles just because they have crossed the<br />
threshold from unpublished to published..</td></tr>
</tbody></table>
After all, good science should spark debates. I study the psychology
of mating and relationships because I think much of this literature is debatable and debate-worthy. I <a href="http://pauleastwick.com/s/IdealsBestPractices_InPress.pdf">criticize</a>
others’ approaches; others <a href="http://pauleastwick.com/s/SchmittCommentPBull2014.pdf">criticize</a> mine. And I have served as a reviewer of productive <a href="http://psycnet.apa.org/PsycARTICLES/journal/bul/140/5">back-and-forth
debates</a> between other scholars. These experiences were sometimes stressful, but in my view, these
criticisms all served to advance the science.<br />
<br /></div>
<div class="MsoNormal">
I find it bewildering that some journals and editors are
reluctant to devote page space to debates and criticism of previously published
work. <o:p></o:p>I have heard people express the opinion that criticism belongs only in the review process; if an article survives this “due process,” it earns a shield against any further published criticism. This attitude has a perverse effect: It prevents debates from moving forward openly for all to evaluate and confines them to a closed review process. I would posit that blogs, Facebook, and Twitter have become popular means of scientific criticism and debate in part because journals do not commonly offer opportunities for the ongoing, post-publication peer-review that is an essential part of science.<br />
<div>
<br /></div>
</div>
<div class="MsoNormal">
I would love to see journals embrace post-publication
criticism—especially the thoughtful and productive kind of criticism that could
even merit publication on its own. Indeed, I would love to see all journals
operate like Behavioral and Brain Sciences or PNAS, where post-publication criticism
is encouraged (or even solicited) shortly after the initial release of an
article. If we create additional avenues for post-publication peer-review, I think we will see a much needed shift in the balance of criticism in our field.<br />
<div class="MsoPlainText" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<br /></div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<br /></div>
<div style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<hr align="left" size="1" width="33%" />
</div>
</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
[1] Hidden?<o:p></o:p></div>
<div class="MsoNormal">
[2] I have always liked <a href="http://journals.sagepub.com/doi/pdf/10.1177/1745691614528215">this piece</a>
about how changes in our scientific practices require changes on the part of
reviewers, too.<span style="mso-spacerun: yes;"> </span><o:p></o:p></div>
<div style="mso-element: comment-list;">
<!--[if !supportAnnotations]-->
<br />
<div style="mso-element: comment;">
<div class="msocomtxt" id="_com_1" language="JavaScript">
<!--[if !supportAnnotations]--></div>
<!--[endif]--></div>
</div>
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6884882187348242375.post-44758663349516265982018-05-16T13:52:00.000-07:002018-05-17T09:15:16.225-07:00Improvements in Research Practices: A Personal Power Ranking<br />
<div class="MsoNormal">
Science is about shifting consensus (Grene, 1985; Longino,
1990). At a given point in time, scientists in a field might believe one
thing, and later, they believe something else. For this reason, persuasion is
the fuel that powers the scientific enterprise. <span style="mso-spacerun: yes;"> </span><o:p></o:p></div>
<div class="MsoNormal">
<span style="mso-spacerun: yes;"><br /></span></div>
<div class="MsoNormal">
The conversation about best practices is our field over the
last decade is no different. It is a persuasion process: Scientists who believe
that direct replications or statistical power or preregistration will improve
the quality of our science attempt to convince scientists who do not hold this
belief to change their views. When confronted with strong evidence,
argumentation, and logic, skeptics should be willing to change their beliefs
(or else they aren’t really practicing science).<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
I have been persuaded about many things. Sometimes I was
persuaded when I simply learned more about a topic. Other times, I was
persuaded because I learned that my previous views were incorrect in some way.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In celebration of scientific persuasion, I thought I would
offer my own personal Top-5 power ranking. Relative to ten years ago, I have
been persuaded about the value of all of these practices. What follows <span style="mso-spacerun: yes;"> </span>is a list of the top 5 improvements in
research practices—ranked in the order that I have found them valuable for my
own research. [1]<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
==================<sup><o:p></o:p></sup></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><u>Improving my own
Research Practices: Top 5 Power Ranking<o:p></o:p></u></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><u><br /></u></b></div>
<div style="text-align: right;">
</div>
<div class="MsoPlainText">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikhDNi3wwv02BElQ_qW28LDVtofHZ1cqY82Esnmqy60N5SpaTk9LMunsbr7_BL4AT2is1KmPV7JXF9QL8NjblmbHQmqnJ1EbXXvpy7mLxSC0Eg9oFQAk3fbTtJ8kqk6_4jEqOzoD9tyXwu/s1600/28188286432_3353c1c7ed_b.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="768" data-original-width="1024" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikhDNi3wwv02BElQ_qW28LDVtofHZ1cqY82Esnmqy60N5SpaTk9LMunsbr7_BL4AT2is1KmPV7JXF9QL8NjblmbHQmqnJ1EbXXvpy7mLxSC0Eg9oFQAk3fbTtJ8kqk6_4jEqOzoD9tyXwu/s200/28188286432_3353c1c7ed_b.jpg" width="200" /></a>5. <b style="mso-bidi-font-weight: normal;"><u>Use social
media for scientific conversation</u></b>: It is remarkable that scholars of
all ranks can turn to social media to learn about research practices, share
their knowledge, and debate scientific issues. When I was in graduate school, debates
and critical discussions were largely confined to conferences and took place
once or twice a year. Now, these conversations happen multiple times a day,
with contributions from a diverse set of voices. In this way, social media has made
civil scientific critique and debate a normal, everyday activity.</div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
<u>Why not higher?</u> I still think that editors serve
an extremely important role in curating scientific criticism and keeping the
debate focused on the substantive issues. For reasons I can’t quite fathom,
some journals are reluctant to give page space to debates about previously
published work, so naturally social media stepped in to fill this void. Nevertheless,
I would love to see journals play a larger role in post-publication peer review,
perhaps by offering something like the PNAS “letters” <a href="http://blog.pnas.org/iforc.pdf">format</a>.<o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
4. <b style="mso-bidi-font-weight: normal;"><u>Conduct</u></b><u>
<b style="mso-bidi-font-weight: normal;">direct replications</b></u>: I now
routinely build direct replications into my work. For example, if we want to
see whether an effect of Study 1 is moderated in Study 2, I might ensure that
the effect in the Study 2 control condition ALSO functions as a direct
replication of Study 1. I continue to conduct conceptual replications, of
course, but I have certainly shifted my emphasis over the past few years. I now
routinely assess the direct replicability of my findings before building on
them, especially when I’m doing something new, and I no longer assume that
other findings are directly replicable if they have only been demonstrated
once. <o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
<u>Why not higher?</u> If we were to over-prioritize
direct replications, we could be at risk for enshrining particular operationalizations
in lieu of the conceptual variables we really care about. For example, in my
home topic area, many findings in the literature on stated mate preferences for
traits are directly replicable, but they have ambiguous connections to the
conceptual variables of interest: It’s very easy to replicate the finding that
men and women say they want different things in a partner, but it’s not clear
the extent to which what people SAY they want maps onto what they ACTUALLY want
when interacting with real potential partners (see <a href="http://pauleastwick.blogspot.com/2018/03/going-on-record-via-preregistration.html">this
earlier post</a>). We should not become so focused on direct replications that
we forget to care about what our variables are actually measuring. <o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLCC21IzDAb-XXw0dLxSfoPMzPt7uVnmUUF3oQfbgBH-Rp-NyRJD7quU3tE__zQ1WELi-2rr6irYw8YbGoIVRh9BcXMLcwrTFHQR44btQcwhd67z4SPVdXQuCESjaW6SVJ2aZXHwlrdtE4/s1600/Error-bar_graphs_of_outdoor_education_effect_sizes_from_Neill_2008.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="402" data-original-width="543" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLCC21IzDAb-XXw0dLxSfoPMzPt7uVnmUUF3oQfbgBH-Rp-NyRJD7quU3tE__zQ1WELi-2rr6irYw8YbGoIVRh9BcXMLcwrTFHQR44btQcwhd67z4SPVdXQuCESjaW6SVJ2aZXHwlrdtE4/s320/Error-bar_graphs_of_outdoor_education_effect_sizes_from_Neill_2008.png" width="320" /></a>3. <b style="mso-bidi-font-weight: normal;"><u>Focus on
effect sizes (rather than significance)</u></b>: In graduate school, my programs
of research often lived and died by <i style="mso-bidi-font-style: normal;">p</i>
< .05. I am overjoyed that this trend is shifting; when I focus on effect
sizes and confidence intervals rather than an arbitrary black-and-white
decision rule, I learn much more from my data. This is especially true when
comparing across studies: We used to think “This study was significant but this
one was not…what happened?” When we focus on effect sizes, these comparisons
take place on a continuum and do not rely on arbitrary cut-offs, and our
attention shifts instead to the extent to which effect size estimates are
consistent across studies. <o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
<u>Why not higher?</u> I am receptive to the argument
that, in many experimental contexts, the effect size “doesn’t really matter” in
the sense that the manipulation is not intended for use in an applied context.
Nevertheless, even when I run experiments, I still find it extremely useful to
compare effect sizes across similar operationalizations, so that I can develop
a sense of how confident I should be in a set of results (more confident if the
effect sizes are similar across experiments using similar manipulations; less
confident if the effect sizes seem to be all over the place).<o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
2.<b style="mso-bidi-font-weight: normal;"><u> Promote and
participate in registered reports</u>: </b>As I noted in a <a href="http://pauleastwick.blogspot.com/2018/01/two-lessons-from-registered-report.html">prior
post</a>, I am a big fan of registered reports. I love how they function to get
both reviewers and authors alike to agree that the results of a particular
study will be informative however they turn out. I now think that our studies
are generally stronger when we design them with this kind of informative
potential from the beginning. I have largely stopped conducting the “shoot the
moon” studies that are counterintuitive and cool if they “work” but wouldn’t
really change my mind if they don’t.<o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
<u>Why not higher?</u> If registered reports became the
norm, what would happen to large pre-existing datasets that are not eligible?
Would people stop investing in large-scale efforts going forward? I hope that
we develop a registered report format that can make use of pre-existing data
(e.g., perhaps in combination with meta-analytic approaches).<b style="mso-bidi-font-weight: normal;"><u><o:p></o:p></u></b></div>
<div class="MsoPlainText">
<br /></div>
<div style="text-align: right;">
</div>
<div class="MsoPlainText">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2FBq-PYqvKJghLtjW7cD3MoGbRz3XqhnVSMl3WqIMs1WB7ZXMNHuYjWHUl8c_kEcZ7-c4E6YtS_Y53luaqrrTBoZSGj_Td1QdCm5dLtiAhzOBYEvcV0Wb4sWkj3PdsOslcr56FH5-4c9q/s1600/Statistical_Power.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="278" data-original-width="652" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2FBq-PYqvKJghLtjW7cD3MoGbRz3XqhnVSMl3WqIMs1WB7ZXMNHuYjWHUl8c_kEcZ7-c4E6YtS_Y53luaqrrTBoZSGj_Td1QdCm5dLtiAhzOBYEvcV0Wb4sWkj3PdsOslcr56FH5-4c9q/s320/Statistical_Power.JPG" width="320" /></a>1. <b style="mso-bidi-font-weight: normal;"><u>Improve power</u>:
</b>My studies are more highly powered than they once were. And as a result, I
feel as though I have been going on fewer wild goose chases: If I see a medium
effect size with several hundred participants in Study 1, I would bet money
that I am going to see it again in my direct replication in Study 2. In cases
where I do make the decision to chase a small effect, that decision is now
conscious and careful (i.e., I will decide if it’s really worth it to invest
the resources to have adequate power to detect the effect if it is there), and
if I decide I do want to chase it with a highly powered study, I learn
something from my data no matter what happens.</div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
Even though I ranked this #1, I still see a potential downside.
For example, I am still running labor-intensive designs (e.g., confederate
studies involving one participant at a time), but they take much longer, and so
I am running fewer of them. But I have considered this tradeoff, and my
assessment is that I am better off running a few highly powered versions of
these studies than many underpowered ones. <o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoPlainText">
Will this be my top 5 power ranking forever?<sup> <span style="mso-spacerun: yes;"> </span></sup>Probably not. [2]<sup> </sup>I look
forward to future research practice improvements, and to having my mind changed
yet again.<o:p></o:p></div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<br /></div>
<div style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.2px;">
<hr align="left" size="1" width="33%" />
<br /></div>
<div class="MsoPlainText">
Grene,
M. (1985). Perception, interpretation, and the sciences: toward a new
philosophy of science. In <i>Evolution at a
crossroads: The new biology and the new philosophy of science</i>.</div>
<div class="MsoPlainText">
<br /></div>
<div class="MsoNormal">
<span style="background: white; color: #222222;">Longino, H. E. (1990). <i>Science as
social knowledge: Values and objectivity in scientific inquiry</i>. Princeton
University Press.</span></div>
<div class="MsoNormal">
<span style="background: white; color: #222222;"><br /></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
[1] Note that this is not a Top-5
list of what developments convinced me that the field as it existed circa 2010
“had a problem” or “was in crisis.” I have been persuaded on that front, too,
but that would be a different list. <o:p></o:p></div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoPlainText">
[2] If you’re curious, here were four honorable mentions
that did not quite make the top 5 for me, in no particular order: Preregistered
analysis plans, transparency in reporting methods, selection methods for assessing
publication bias, open data. <o:p></o:p></div>
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6884882187348242375.post-28686509963199568822018-03-26T11:25:00.000-07:002018-03-26T11:26:26.376-07:00Testing the replicability of claims about a sex difference: A regrettable delay <b><i>A public commitment to update my own beliefs in response to a planned analysis I haven’t seen yet (Part 2)</i></b><br />
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><i><br /></i></b></div>
<div class="MsoNormal">
In <a href="http://pauleastwick.blogspot.com/2018/03/going-on-record-via-preregistration.html" target="_blank">Part 1</a> of this series, I
tried to make some headway in the debate over sex differences in the appeal of
attractiveness in established relationships by putting my own beliefs on the
line, <a href="https://osf.io/nqt6r/" target="_blank">pre-registering</a> an analysis plan to see if a prior result would
replicate, and publicly committing to update my beliefs regardless of how the
results turned out. Unfortunately, this test will have to wait.</div>
<div class="MsoNormal">
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Although I assumed that it
would be easy to obtain the data from a <a href="http://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000127" target="_blank">just-published manuscript</a>, I was
incorrect: Dr. McNulty has informed me that there will be a “regrettable delay”
of unknown duration in sharing the data underlying the published manuscript
until his team finishes working on and successfully publishes a second
manuscript analyzing the same columns of data. Once the second manuscript is
successfully published, he will be happy to share the data associated with the
first manuscript, but he has no guess about how long that might take. Our full email
exchange is included below, with Dr. McNulty’s permission. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
I think it is fair to say
that he and I are reading the APA ethical principle on data-sharing differently.
In light of the field’s growing appreciation of the importance of openly and
transparently sharing the data that is used in published manuscripts, I wonder
if the language in the APA principle needs to be clarified or updated to
reflect current standards in the field. (Indeed, the most surprising element to
me of our whole exchange was Dr. McNulty noting that one of his colleagues had
advised him against ever sharing the data associated with his published
manuscript. Clearly, scholars have very different views about whether and when the
data behind published papers should be shared with other researchers, and it
seems crucial that our societies and journals provide clear guidance to authors
going forward.)<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In light of the indefinite
and regrettable delay, any claims that this particular sex difference is robust
seem premature. I have posted below the results of the <a href="http://psycnet.apa.org/record/2013-35735-001" target="_blank">Meltzer et al. (2014)</a> 28-covariate
analysis, as well as the <a href="http://psycnet.apa.org/buy/2014-07574-002" target="_blank">Eastwick et al. (2014)</a> unsuccessful replication
attempt, so that readers can get a sense of the existing evidence for this sex
difference. I have also left a blank space for the eventual inclusion of a
direct replication from the new <a href="http://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000127" target="_blank">McNulty et al. (2018 online publication)</a>
dataset. I will fill it in once the data from those <i>N</i> = 233 couples are shared with me and I can conduct the
preregistered analyses. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
I’ll close with an exhortation to other scholars: Future tests of this idea should examine it in a confirmatory way (i.e., with a detailed analysis plan that is written ahead of time, before seeing the data). My post did not end the debate, but I do hope that this approach will set a standard that helps researchers come together to address this question with strong methods going forward. </div>
<div class="MsoNormal">
<br /></div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZ5fiRni0Z3s9a6O9A9wo1UZJTE4P-VG3sreSBcbpvSxSXnC1x4rd58UIyJHqajFqSEWYrUpuMoLy2YeU93pzfa_cXbqkwhLI-bsB63-DwIyFIX9ZWZ4IXwrsbmrPtoR42TwnkVxfRK6R/s1600/SexDifFigure2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="466" data-original-width="642" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZ5fiRni0Z3s9a6O9A9wo1UZJTE4P-VG3sreSBcbpvSxSXnC1x4rd58UIyJHqajFqSEWYrUpuMoLy2YeU93pzfa_cXbqkwhLI-bsB63-DwIyFIX9ZWZ4IXwrsbmrPtoR42TwnkVxfRK6R/s1600/SexDifFigure2.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Results of the 28-covariate analysis proposed by Meltzer et al. (2014) and the one direct replication to date (Eastwick et al., 2014). Meltzer et al. (2014) concluded that the association of coder-rated attractiveness with relationship satisfaction is stronger for men than for women (see first Intercept test). I will update the figure when the data for McNulty et al. (2018 online publication) are made available. <br />
Bars indicate 95% CIs. Y axis is effect size <i>q </i>(interpretable like <i>r</i>).</td></tr>
</tbody></table>
<br />
My preferred approach to testing this sex difference is as follows: a random effects meta-analysis examining the effect of coder-rated attractiveness on relationship evaluations (e.g., satisfaction) in established (i.e., dating and/or married) relationships. That meta-analytic effect (k = 11, N = 2,976), which includes both the Meltzer et al. (2014) and Eastwick et al. (2014) data analyzed above, is shown here:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDSxUWNFUinJAFENuvGDzLg2ygwpqHAyrq6_p2RTOJ4PizUjQt5jHz0iFyWlMO_wPTSn9RZEOy67dIPHO8CKmbyWHTsttRYav8UG3eZZRO-MKB2_5Cx-Wh6dQW2AVLFAltoqZZMPmWCVDe/s1600/SexDiffsMeta.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="405" data-original-width="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDSxUWNFUinJAFENuvGDzLg2ygwpqHAyrq6_p2RTOJ4PizUjQt5jHz0iFyWlMO_wPTSn9RZEOy67dIPHO8CKmbyWHTsttRYav8UG3eZZRO-MKB2_5Cx-Wh6dQW2AVLFAltoqZZMPmWCVDe/s1600/SexDiffsMeta.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Bar indicates 95% CI. Y axis is effect size <i>q</i> (interpretable like <i>r</i>).</td></tr>
</tbody></table>
<br />
<div class="MsoNormal">
<br /></div>
<div>
<hr align="left" size="1" width="33%" />
<br />
<div id="ftn1">
<div class="MsoFootnoteText">
Emails reprinted here, with permission:<br />
<br />
<b>March 7, 2018</b><br />
<b><br /></b>
Hello Jim,<br />
<br />
I hope you enjoyed SPSP this year – it was good to run into you briefly. I am writing to request the data associated with your new paper, which looks really interesting: <a href="http://psycnet.apa.org/record/2018-05467-001?doi=1">http://psycnet.apa.org/record/2018-05467-001?doi=1</a><br />
<br />
In addition to the covariates in Table 2 and income (mentioned on p. 4), I would be very appreciative if you would also include extraversion if you have it. But I also recognize that, technically speaking, you are under no obligation to share extraversion given that it wasn’t mentioned in the published article.<br />
<br />
My intention is simply to conduct <a href="https://osf.io/nqt6r/" target="_blank">this preregistered analysis plan</a>. If you are curious, I also have written a <a href="http://pauleastwick.blogspot.com/2018/03/going-on-record-via-preregistration.html" target="_blank">blog </a>post about the relevant interpretive issues – if you and/or Andrea would like to comment on the second part (once I write it), I would be happy to include your response on the blog.<br />
<br />
Regards,<br />
<br />
Paul<br />
<br />
===========<br />
<br />
<b>March 8, 2018</b><br />
<b><br /></b>
Hi, Paul.<br />
<br />
I enjoyed SPSP and it was good to run into you. It was astute of you to realize we have some more data to address our debate. I would be happy to share them with you eventually, but one of Andrea’s doctoral students is currently working on a manuscript that addresses this exact effect. They have been working on it off and on for some time now, but, as is typical, other priorities keep interfering. I fear it could undermine her project to share these valuable data with you and the world right now. That said, I do appreciate complete transparency, as well as your attempts to shed more light on this issue, and I would be happy to share all the data with you once her project is complete. Does that sound okay? I wish I had a good guess as to when that would be, but for some reason I still haven’t figured out how to predict how reviewers will feel about a particular paper. Haha.<br />
<br />
Best,<br />
Jim<br />
<br />
===========<br />
<br />
<b>March 9, 2018</b><br />
<b><br /></b>
Hi Jim,<br />
<br />
I totally understand wanting to make sure that your student will be able to publish his/her paper. And I realize that my email might not have been clear: I was only suggesting that I would report the results on the blog, not a journal article. You should of course be able to carve up the remaining dataset for journal articles as you see fit – I’m only requesting the data that were used in the in press publication (plus extraversion if you had it and were willing to share it -- but of course, I understand that you are under no obligation to do so since it’s not in the published article). I wouldn’t anticipate that a blog post on this particular analysis would interfere with your student’s ability to report and build off of it in a future article.<br />
<br />
Regards,<br />
<br />
Paul<br />
<br />
===========<br />
<br />
<b>March 16, 2018</b><br />
<br />
Hi Jim,<br />
<br />
I just wanted to follow up with you on the message I sent last week requesting the data from your in press JPSP. I’m still excited to take a look, and I want to reiterate that my plan is only to share the results of the preregistered analyses on a blog (i.e., not a journal publication). In case it helps mitigate the concerns you articulated about wanting to publish analyses based on these data in a separate article, I had an idea: What if I only post the effect sizes and confidence intervals associated with the three sex difference tests that I preregistered (i.e., no other statistical information or detailed descriptives)?<br />
<br />
I really hope that we can navigate these data sharing complexities ourselves in a friendly way – I am committed to making some progress on the sex difference question by conducting and reporting the analyses I preregistered on my blog however they turn out, and you of course should be able to publish additional analyses in the future off of these published data. I do think it’s important to keep in mind that the data I am requesting are now published, and that this means that ethically, they must be made available to “other competent professionals” (APA, 8.14, 2010). But I’d much rather do this in a friendly and informal way over email rather than going through the journal or APA or something.<br />
<br />
If I don’t hear from you by next Friday (the 23rd), I’ll go ahead and update my blog to indicate that you declined to share the data, and we’ll go from there.<br />
<br />
Regards,<br />
<br />
Paul<br />
<br />
===========<br />
<b><br /></b>
<b>March 20, 2018</b><br />
<b><br /></b>
Paul,<br />
<br />
I understand that you do not plan to pursue publication of the data you requested. And I believe you are probably correct that a blog will not interfere with a future publication. However, I must admit that the blogosphere is extremely foreign to me and I perceive that it seems to have some traction. I also have no idea what the future holds. I see no reason to risk even an unlikely negative outcome for one of our students. I’m not sure I was clear in my original email, but the student is not simply working with these data; she is working on a manuscript describing the sex difference in the association between partner attractiveness and marital satisfaction—the precise effect in question. I have received advice from two colleagues who are unattached to this debate and they tell me not to share the data yet (one says don’t share it at all).<br />
<br />
Regarding any ethical obligation to share the data with you, my read of the APA ethics statement on this issue is that I am only obligated to share with “other competent professionals” who intend to replicate the result in question. APA Ethical Principles specify that "after research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals <b>who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose</b>, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release" (Standard 8.14).”Retrieved: http://www.apa.org/pubs/journals/psp/index.aspx?tab=4. You left out of your email the critical qualifier that I bolded above. It is quite clear from your email, and from the fact that you preregistered a completely unrelated analysis of my covariates, that you have no intentions to verify our substantive claims but instead want to capitalize on our covariates to address your own research goals.<br />
<br />
To be honest with you, Paul, what is frustrating to me about your latest email that threatens to post on your blog that I declined your request and potentially take up this issue with APA is that I did not decline your request. As I said in my original email, I will give you the data after the student working on this exact effect is finished, even though I do not believe I am obligated to do so, because I too am committed to science and understanding this sex difference. If you post anything on your blog about this other than the fact that there will be a regrettable delay in getting the data from us, please also post this entire string of emails so people can decide for themselves if I am being unethical.<br />
<br />
Jim<br />
<br />
===========<br />
<br />
<b>March 22, 2018</b><br />
<br />
Hi Jim,<br />
<br />
Thanks for your reply. It seems like we have different interpretations of the APA data-sharing principle (at least as it applies in this case). I thought it was self-evident that my proposed analysis was addressing a “substantive claim” of your published manuscript: You tested and reported a sex difference in the partner attractiveness-infidelity association, and concluded the following on pp. 15-16: “This latter sex difference is consistent with evidence that partner attractiveness is more important to men than it is to women (Li et al., 2013; McNulty, Neff, & Karney, 2008; Meltzer et al., 2014a, 2014b), and thereby challenges the idea that the importance of partner attractiveness is equivalent across men and women (see Eastwick & Finkel, 2008).” You had the opportunity to conduct the same analysis that you and your colleagues have argued is the best test of this sex difference (Meltzer et al., 2014a; this is the analysis I proposed in my blog post) to see if the Meltzer et al. (2014a) findings would replicate in this new dataset. Although you did not report this analysis, you claimed in the Discussion of your paper to have supported those findings anyway.<br />
<br />
In my blog post, I proposed to reanalyze the data from your published paper in order to test the claim that “partner attractiveness is more important to men than it is to women” (p. 16). To me, it seems like the APA data-sharing principle (as well as the field’s current norms about the importance of openness and transparency) applies here. Nevertheless, I agree that multiple interpretations of the APA principle are possible and I appreciate your willingness to engage with me on this issue.<br />
<br />
I’m disappointed that there will be a regrettable delay (as you note) in your sharing of these data. I’m also sad to hear that, in this day and age, your colleagues are advising you to delay or avoid sharing the data behind a published paper. I appreciate your willingness to allow me to post our email exchange, and I apologize if you worried that I would misrepresent you – that was definitely not my intention, and I agree with you that it is important to post the exchange for transparency’s sake.<br />
<br />
Regards,<br />
<br />
Paul<br />
<br />
PS: Despite all this, I really do think the new paper is cool. One of the questions it addresses had come up a few days beforehand in my grad class.<br />
<div>
<br /></div>
</div>
</div>
</div>
<div>
<br /></div>
Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6884882187348242375.post-77291708170861349442018-03-07T09:48:00.000-08:002018-03-26T11:32:43.105-07:00Going on the record via preregistration<b><i>A public commitment to update my own beliefs in response to a planned analysis I haven’t seen yet (Part 1)</i></b><br />
<b><i><br /></i></b>
<b>Update, 3/26/18: Unfortunately, my request for the data behind this recently published JPSP paper <a href="http://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000127" target="_blank">(McNulty, Meltzer, Makhanova, and Maner, 2018 online publication)</a> was unsuccessful. Dr. McNulty has informed me that there will be a “regrettable delay” of unknown duration in sharing these now published data until his team writes up and successfully publishes a second manuscript on these same data columns. <a href="http://pauleastwick.blogspot.com/2018/03/testing-a-sex-difference.html" target="_blank">Part 2</a> of this blog post is here, along with our email exchange about the data sharing question. </b><br />
<br />
In my <a href="http://pauleastwick.blogspot.com/2018/02/intuitive-and-impossible-what-do-short.html" target="_blank">previous post</a>, I talked about how essential it is that we, as scientists, remain open to the possibility of having our intuitions disconfirmed.<br />
<br />
Now let’s see if I can put my money where my mouth is.<br />
<br />
If I take my own admonishment seriously, I need to be willing to have my own intuitions and beliefs disconfirmed—even when those beliefs have developed through years of researching a particular topic.<br />
<br />
Here’s one of my own findings in which I have a high degree of confidence. In a <a href="http://pauleastwick.com/s/Eastwick2014PBull.pdf" target="_blank">meta-analysis</a> I conducted about five years ago, we examined whether a partner’s attractiveness was more romantically appealing to men than to women. We acquired a large collection of published and unpublished datasets (<i>k</i> = 97, <i>N</i> = 29,780) that spanned a variety of paradigms in which men and women reported on partners they had (at a minimum) met face-to-face. Overall, we found that the sex difference in the appeal of attractiveness was not significantly different from zero, and it did not matter whether the study examined initial attraction (e.g., speed-dating, confederate designs) or established relationships (e.g., dating couples, married couples).<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Here is a hypothetical illustration of this finding: If a man’s satisfaction in a given relationship is predicted by his female partner’s attractiveness at <i>r</i> = .08, we might find that a woman’s satisfaction is predicted by her male partner’s attractiveness at about <i>r</i> = .03. Meta-analytically, the sex difference is about this size: <i>r</i>(difference) = .05 or smaller. You can interpret this <i>r</i>(difference) like you would interpret <i>r</i> = .05 in any other context—really small, hard to detect, and probably not practically different from zero.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-BRcpk7Ytv9YyVD5Iz-nVm45j2D6Hp6doOULxp12fKf9oeOLy3d5eOepAobdtzT_kZQ0ST6z7EtcNZBdb5LVnOTOT1Ghgs-qIQcFEnPAsxtf_ffXeRoLD6izeIRpZ_IXuu6aYdNe9DOki/s1600/SexDifferenceTable3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="132" data-original-width="663" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-BRcpk7Ytv9YyVD5Iz-nVm45j2D6Hp6doOULxp12fKf9oeOLy3d5eOepAobdtzT_kZQ0ST6z7EtcNZBdb5LVnOTOT1Ghgs-qIQcFEnPAsxtf_ffXeRoLD6izeIRpZ_IXuu6aYdNe9DOki/s1600/SexDifferenceTable3.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">However you slice the meta-analytic data, it is hard to find a sex difference in the appeal<br />
of attractiveness in paradigms where participants have met partners face-to-face.<br />
(<i>p </i>refers to the <i>p </i>value of the sex difference test statistic <i>Qsex.</i>) From <a href="http://pauleastwick.com/s/EastwickComment2014JPSP.pdf" target="_blank">here</a>.</td></tr>
</tbody></table>
<div style="text-align: left;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Interestingly, the sex difference in attractiveness is much larger when you ask men and women to use a rating scale to indicate how much they think they like physical attractiveness in a partner. The size of this “stated preference” sex difference is about <i>r</i> = .25 (see Table 1 in this <a href="http://psycnet.apa.org/record/1991-12468-001" target="_blank">paper</a>). [1]<br />
<br />
<b>In other words, an <i>r</i> = .25 effect when people make judgments about what they think they like drops to <i>r</i> = .05 when people are responding to partners who they have actually met in real life. </b><br />
<br />
I find this “effect size drop” deeply fascinating. It opens two interesting questions that have guided much of my research:<br />
<br />
1.<span style="white-space: pre;"> </span>If men and women truly differ in the extent to which they believe attractiveness to be important in a partner, what factors interfere with the application of these ideals when they evaluate partners in real life?<br />
<br />
2.<span style="white-space: pre;"> </span>If there is essentially no difference between men and women in how much they actually prefer attractiveness in a real life partner, what sorts of social-cognitive biases might produce the sex difference in how much people think they prefer attractiveness in a partner?<br />
<br />
I have spent considerable time and effort in the last decade examining these two questions in my research. We’ve found some answers, and yet there’s still a long way to go in this topic area.<br />
<br />
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF728LzZFMHqG-bBLWIBqqv9Zs1MH9Nthx8DBrBeklzsjLr5JZIRmr7JSJMTZpS7VPGqA3_1x0YaxcIhOgQ36PxCxFcpWe35XgM8KRvif_XVEFXkR0aKWtyXUhpnKH7-nAc0IABK8T98jm/s1600/SexDifEstimates4.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="116" data-original-width="439" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF728LzZFMHqG-bBLWIBqqv9Zs1MH9Nthx8DBrBeklzsjLr5JZIRmr7JSJMTZpS7VPGqA3_1x0YaxcIhOgQ36PxCxFcpWe35XgM8KRvif_XVEFXkR0aKWtyXUhpnKH7-nAc0IABK8T98jm/s1600/SexDifEstimates4.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All effect sizes are coded so positive values mean that attractiveness receives higher<br />
ratings/is a larger predictor for men than for women. I am prepared to update the<br />
table after I examine the new McNulty et al. (in press) data according to my<br />
preregistered analysis plan.</td></tr>
</tbody></table>
But back to my belief that I am putting on the line in this blog post: I believe that the sex difference is about <i>r</i> = .05 (or smaller) when people evaluate real-life partners. I feel pretty confident about this belief, given all the evidence I have seen. But there are other scholars who believe something entirely different.<br />
<br />
================<br />
<br />
Since we published the meta-analysis, two empirical articles have taken a strong stance against our conclusion that the sex difference in the appeal of attractiveness is small or nonexistent. I discussed one of them (Li et al., 2013) in an <a href="http://pauleastwick.blogspot.com/2018/01/a-confederate-is-not-condition.html" target="_blank">earlier post</a>; given the tiny effective sample size of that study, I won’t discuss it further here. Instead, let’s talk about the second one: <a href="https://www.researchgate.net/profile/Andrea_L_Meltzer/publication/257837689_Sex_Differences_in_the_Implications_of_Partner_Physical_Attractiveness_for_the_Trajectory_of_Marital_Satisfaction/links/00463535815ce808ab000000/Sex-Differences-in-the-Implications-of-Partner-Physical-Attractiveness-for-the-Trajectory-of-Marital-Satisfaction.pdf" target="_blank">Meltzer, McNulty, Jackson, & Karney (2014)</a>.<br />
<br />
This paper found the expected sex difference in a sample of N = 458 married couples. In brief, they found that women’s attractiveness predicted men’s satisfaction at <i>r</i> = .10, whereas men’s attractiveness predicted women’s satisfaction <i>r</i> = -.05. That’s an <i>r</i>(difference) of .15—still pretty small, but not zero (<i>p</i> = .046).<br />
<br />
One unusual element of this paper is that the authors only present this sex difference in one analysis, and it included a large number of covariates. Twenty-eight of them, to be exact. Another element worth noting is that there were actually two ways that the sex difference could have emerged—on the intercept of satisfaction or the slope of satisfaction. The effect that the authors focused on was the intercept; slope effects did not differ for men and women, <i>r</i>(difference) = .02.<br />
<br />
Personally, I don’t believe that this analysis provides an accurate depiction of the sex difference. It’s hard for me to buy into the idea that you need twenty-eight covariates in this analysis, and even then, the sex difference only emerges in one place and not the other. In fact, we conducted an <a href="http://pauleastwick.com/s/EastwickComment2014JPSP.pdf" target="_blank">identical analysis</a> on some of our own data that had the same variables, and we didn’t find a hint of the sex difference (if anything, the slope effect trended in the opposite direction).<br />
<br />
Nevertheless, for the past five years, this debate gets distilled to “Team X says no sex difference, but Team Y says yes.” If someone wants to cite evidence for the absence of the sex difference, they have it; if someone wants to cite evidence for the presence of the sex difference, they can do that, too. This does not seem to be a good scientific recipe for getting closer to the truth.<br />
<br />
I’m pretty confident in my belief that the sex difference here is tiny or nonexistent. But you know what? Maybe I’m wrong. If I want to call myself a scientist, I have to be open to that possibility. I have to be willing to say: Here are the data that would convince me to change my belief.<br />
<br />
So here it is: I will update my belief if a preregistered test, using the same 28-covariate analysis in a new dataset, replicates the sex difference on the intercept found in Meltzer et al. (2014).<br />
<br />
You may be thinking, it’s easy for me to say that, so long as no dataset of the kind exists. But in fact, just the other day, I saw this new published paper (<a href="http://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000127" target="_blank">McNulty, Meltzer, Makhanova, & Maner, in press</a>). It primarily examines a different (and totally fascinating!) research question, and it uses a new sample of <i>N</i> = 233 couples. But buried in the descriptions of the covariates in that paper are all of the key variables and all but one of the covariates required to directly replicate the earlier sex difference analysis reported in Meltzer et al. (2014).<br />
<br />
Here is what I am committing to, publicly, right now: I have written up a <a href="https://osf.io/nqt6r/" target="_blank">preregistered analysis plan</a> that provides the test I outline above. I will email Jim McNulty for the data they used in this new published manuscript, which I am confident that he will share with me. I will run the preregistered analysis on these data, and I will describe the results as a “Part 2” of this blog post. <b>If the key finding from Meltzer et al. (2014) replicates—that is, if the sex difference on the intercept is significant—then I need to seriously consider the possibility that I am wrong, and I need to update my beliefs accordingly. If it is not, I hope that those scholars who believe in this particular sex difference will be willing to update their beliefs and/or conduct a highly powered test of their prediction. </b><br />
<br />
Either way, we’ll be getting closer to the truth rather than being stuck in an endless circle around it.<br />
<br />
<div class="MsoNormal">
<br /></div>
<div>
<hr align="left" size="1" width="33%" />
<br />
<div id="ftn1">
<div class="MsoFootnoteText">
<span class="MsoFootnoteReference"><span class="MsoFootnoteReference"><span style="font-family: "calibri" , "sans-serif"; font-size: 10pt; line-height: 15.3333px;">[1]</span></span></span> When people talk about the “robust literature” showing that attractiveness matters more to men than to women, they could be talking about one of two things. First, they could be talking about this stated preference sex difference. Second, they might be talking about findings showing that, in <b>hypothetical </b>settings (e.g., viewing photographs), attractiveness tends to matter more to men than to women. In fact, we preregistered a study examining this context and found the sex difference! As I described in this <a href="http://pauleastwick.blogspot.com/2018/01/two-lessons-from-registered-report.html" target="_blank">earlier post</a>, the size of the sex difference that we found in a very highly powered design was <i>r</i> = .13. </div>
</div>
</div>
<div>
<br /></div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-6884882187348242375.post-63047551585222972312018-02-20T13:32:00.000-08:002018-02-20T13:32:33.035-08:00Intuitive and Impossible: What do Short-Term and Long-Term Relationships Look Like?People have long-term relationships and short-term relationships. In what ways do these two kinds of relationships differ?<br />
<br />
You may find the answer to be extremely intuitive—or extremely counterintuitive—depending on your lay theories about relationships, or depending on which segment of the literature on human mating is more familiar to you.<br />
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<br />
<div style="text-align: right;">
</div>
<div style="text-align: right;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoB_Sm-OaAZH59IrlpVnKWrz-86O2qrHYgj_s-SNK8PjfphTg9yN0LJ7FvG-ZItzqL6Q3-qbCHOq_lOfiarEDFo3eL4_E_SzD8Mg5GQ8DRdEvkfil5vaqp4b17ZbebZ2EpXaGt0vYfD1u9/s1600/ReCAST2.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="1023" height="211" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoB_Sm-OaAZH59IrlpVnKWrz-86O2qrHYgj_s-SNK8PjfphTg9yN0LJ7FvG-ZItzqL6Q3-qbCHOq_lOfiarEDFo3eL4_E_SzD8Mg5GQ8DRdEvkfil5vaqp4b17ZbebZ2EpXaGt0vYfD1u9/s400/ReCAST2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The ReCAST Model. Double lines are long-term relationships,<br />
and the single line is a short-term relationship.</td></tr>
</tbody></table>
In a recent <a href="http://pauleastwick.com/s/ReCAST_InPress.pdf" target="_blank">paper</a>, we collected data on people’s real-life relationships over time—beginning at the first moment they met a partner—to compare the relationships that people think of as “long-term” and “short-term.” There is a vast literature that asks people what they want in these kinds of relationships, but there is far less data on people’s real life experiences with short-term and long-term relationships and partners. We wanted to know: How exactly do these types of relationships differ, and when do these differences become apparent? It took us about 4 years to collect and publish these data, and they helped us inform and develop something we call the ReCAST model.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: right;">
</div>
Perhaps the most important finding was this one: Differences did not emerge right away. That is, it took a considerable period of time—typically weeks or months—for short-term and long-term relationships to diverge. Put another way: You can’t tell, early on, whether a relationship is short-term or long-term; the trajectories only pull apart once you’ve known someone for quite awhile.<br />
<br />
We have a high degree of confidence in these findings.[1] But here is today’s question: <b>Are these findings intuitive and obvious? </b><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR-GS7dOV8yrxhnGS2FceYex9pdC73fI14trjg5mpDOo1a9duTugvmQ5QrmyEQ8pYzbAEiPWRFOUAdypdmlaLpjlxOlZ-0q9Hz8ap2kiUmxMSZzW7Wnw8UzgaUnrBu5706fsA7F9cKu2NQ/s1600/RomanticInterest2a.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="725" data-original-width="468" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR-GS7dOV8yrxhnGS2FceYex9pdC73fI14trjg5mpDOo1a9duTugvmQ5QrmyEQ8pYzbAEiPWRFOUAdypdmlaLpjlxOlZ-0q9Hz8ap2kiUmxMSZzW7Wnw8UzgaUnrBu5706fsA7F9cKu2NQ/s400/RomanticInterest2a.png" width="257" /></a>According to one type of reviewer (we had two reviewers like this), these data are extremely intuitive. These reviewers said: Researchers studying close relationships already know that relationships unfold gradually over time. Of course you cannot predict how long a relationship will last until two people have a chance to interact, assess interpersonal chemistry, and (preferably) have a few make-out sessions. These assumptions are built into the fabric of everything we have done for the past 30 years. Why would you try to test or publish something so obvious?<br />
<br />
To another type of reviewer (we had four reviewers like this), these results were highly implausible. These reviewers said: Researchers studying evolved strategies know that people approach relationships very differently depending on whether that relationship is short-term or long-term. For example, women can view a photograph of a man and know from his chiseled features that he is good for a short-term but not a long-term relationship. Your data are at odds with the assumptions that are built into the fabric of everything we have done for the past 30 years. You can’t possibly be testing these predictions correctly—if your methods were right, you would have gotten different results. Therefore, these data shouldn’t be published.<br />
<br />
Together, these reviews characterized our data as simultaneously obvious and implausible. And this juxtaposition highlights the risk of drawing on intuition when making scientific critiques.<br />
<br />
=================<br />
<br />
Here is a short history of the Pendulum of Intuitiveness in psychological journals.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2iSGDIwDxF1dSB9IyorkLm3_BOiAeTcYg14FrvY_QUSJLqPPh4JYK0SNtKQr95k7Nz3A9YdubhUeF0QGpvFlQIG6lpGuMH-dHTpEE8HRDV2sJSirIDjxpTE8YTjjlSW6HxfxUrRIBA1LI/s1600/Pendulum.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="786" data-original-width="885" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2iSGDIwDxF1dSB9IyorkLm3_BOiAeTcYg14FrvY_QUSJLqPPh4JYK0SNtKQr95k7Nz3A9YdubhUeF0QGpvFlQIG6lpGuMH-dHTpEE8HRDV2sJSirIDjxpTE8YTjjlSW6HxfxUrRIBA1LI/s200/Pendulum.png" width="200" /></a>When I was in graduate school in the early-mid 2000s, the easiest way to get rejected from a journal was to try to publish something that felt obvious and familiar. One way that people would try to combat this pressure: Find a result that was counterintuitive. Hopefully, very counterintuitive. Like “wow, can you believe it?!” counterintuitive.<br />
<br />
Sometimes, though, that counterintuitive finding didn’t emerge from a deep dive into two theories to discover where they made divergent predictions. Rather, the finding was something flashy—something a lay person wouldn’t have expected. Conducting data analysis felt more like gambling than detective work; ten obvious p < .05s were worth a lot less than one shocking (and perhaps “lucky”) p < .05. These pressures and strategies probably led to the publication of some counterintuitive findings that would be tough to replicate over some intuitive but easily replicable ones.<br />
<br />
But within the last few years, terms like “counterintuitive” have become radioactive in the wake of recent methodological advances in our field. In other words, if a result seems surprising to you, now there is reason to suspect that it might be “too good to be true.”<br />
<br />
The counterintuitive backlash makes sense. But it’s not a sufficient place to stop: Unless we want to keep swinging with the pendulum, we have to remember to continually question our intuitions at the same time. If we’re not willing to test our intuitions and publish the results—whether those results are themselves intuitive or counterintuitive—we sound more like advocates for “stuff we already know” than scientists asking questions about the world.<br />
<br />
So intuition may be great for inspiring study ideas and informing your own personal Bayesian priors about whether a study is likely to work or <a href="http://www.pnas.org/content/112/50/15343" target="_blank">replicate</a>. But it is not a substitute for actual empirical research. And if that research is appropriately-powered, theoretically grounded, and well conducted, the findings have value regardless of whether they happened to confirm or disconfirm your intuitions. After all, one scholar’s intuitive may be another scholar’s impossible.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
---------------------------<br />
<br />
[1] Please, please replicate us! The materials and preregistration can be found <a href="https://osf.io/2pnhm/" target="_blank">here</a>. And don’t hesitate to email me if you have questions.<br />
<div>
<br /></div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-6884882187348242375.post-57407051799505647932018-01-22T08:55:00.000-08:002018-01-22T08:55:43.385-08:00A Confederate is not a Condition<div class="MsoNormal">
I made a mistake. I equated a person with an experimental
condition. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In Study 1 of <a href="http://pauleastwick.com/s/Eastwicketal2009PSPB.pdf">this article</a>, we
introduced <i>N </i>= 54 men to both a White
and a Black female confederate in two separate face-to-face interactions. These
two confederates – we’ll call them “Hannah” and “Kiara” (not their real names)
– played their roles superbly and never forgot their lines. The study was a model
of experimental control.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
But the inferences I drew from these data were incorrect
because of a statistical issue I did not appreciate at the time.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3EvSCDBZXLyElNGpn7BgUibswqMYFJbPaGy4eMI9xzfkVSH_iRRKv9jPfP5qoc81fg0OcS2nwRDI6Un78N1Jkkz7cX3g2hPPCNSgpf0tv5MK-CwEtiDDWcd5ZMIrxMBzdtvYdj7Z3ID2J/s1600/2_1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="380" data-original-width="473" height="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3EvSCDBZXLyElNGpn7BgUibswqMYFJbPaGy4eMI9xzfkVSH_iRRKv9jPfP5qoc81fg0OcS2nwRDI6Un78N1Jkkz7cX3g2hPPCNSgpf0tv5MK-CwEtiDDWcd5ZMIrxMBzdtvYdj7Z3ID2J/s320/2_1.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">How would you label this pair of "conditions"?</td></tr>
</tbody></table>
</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
What we found was this: The men in our study (all of them
White) tended to like the White confederate to the extent that they were
politically conservative, but the men liked the Black confederate to the extent
that they were liberal. I drew the inference that political orientation was
associated with whether the men were attracted to members of their racial ingroup
(i.e., the White partner) or outgroup (i.e., the Black partner).</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<div style="text-align: right;">
</div>
But a logically equivalent description of these results
reveals my inferential overreach: The men in our study liked Hannah more to the
extent that they were politically conservative, but they liked Kiara more to
the extent that they were liberal. The results might have been attributable to
the women’s race…or to any of the other myriad differences between these two particular
women.[1]</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
This is why you sample stimuli as well as participants. Arguably, my sample size was not <i>N </i>= 54 (the number of participants), but
<i>N </i>= 2 (the number of stimuli). <o:p></o:p></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
===============</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The above example may seem pretty straightforward to you,
but the same issue frequently turns up in subtler—but equally problematic—forms.
Let’s say I hypothesize that attractiveness inspires romantic desire more for
men than for women in a face-to-face, heterosexual interaction. This makes
intuitive sense…anecdotally, men seem to talk more about how hot women are than
vice versa. Perhaps surprisingly, then, this sex difference does not emerge in
speed-dating contexts where people meet a slew of opposite-sex partners who
naturally vary in attractiveness (see <a href="http://pauleastwick.com/s/EastwickFinkel2008JPSP.pdf">here</a> and direct
replication <a href="http://journals.sagepub.com/doi/abs/10.1177/2158244015605160">here</a><span class="MsoHyperlink">)</span>. But maybe it would emerge with a <i>manipulation</i> of attractiveness: If men
and women each met an attractive and an unattractive partner, maybe this within-subjects
attractiveness manipulation would inspire romantic desire more for men than for
women?<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2yYZQV9mKi7Ze5p_wQxPQiF1LtuJJU59yUihH4Jxm2ji71Mhc5XcfeYI7MRN7sYWzvKNjdwRpoMp5JHzOJGUmciP2ia6cPkHv1Q8l8gH9tyOoXKUdhFwPxjh87r4LIiZQQ5lqJ_YL8uWu/s1600/2_2.bmp" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="437" data-original-width="682" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2yYZQV9mKi7Ze5p_wQxPQiF1LtuJJU59yUihH4Jxm2ji71Mhc5XcfeYI7MRN7sYWzvKNjdwRpoMp5JHzOJGUmciP2ia6cPkHv1Q8l8gH9tyOoXKUdhFwPxjh87r4LIiZQQ5lqJ_YL8uWu/s320/2_2.bmp" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From Li et al. (2013). Each bar was <br />
generated by 42-51 raters but only 2 targets.</td></tr>
</tbody></table>
<div class="MsoNormal">
<a href="http://psycnet.apa.org/record/2013-27492-001">Here’s
a study</a> that used exactly this approach to test the hypothesis that attractiveness
will matter more for inspiring romantic desire in men than in women. It seems to find—and is frequently cited as
showing—evidence for the hypothesized sex difference: In the figure on the right,
one can clearly see that men differentiated the attractive and unattractive
confederates much more strongly than women did. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
But notice that this study has
the same serious flaw that I described above with my confederate study. To see
why, let’s once again use (fake) names: The men desired Rachel and Sally much
more than Amanda and Liz, whereas women desired Brian and Karl just a bit more
than James and Dan. The results certainly tell us something about the
desirability of these particular confederates. But with such a small N (only 2
confederates per condition), we cannot generalize these findings to say
anything meaningful about <i>attractive and
unattractive targets in general</i>. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSxMbZPs6D6ctRwphHwMLXST_ygasff_ZgQk0HlddI58WlVZVWNcklqzaWeeRLABUaUvMmDZbcLOzWIAlSsjU44rWGYfrKRRnPuw9atSf0c4q5jh6ZsEU2_26i3k1yjY6KEKyeJ5VHk6qb/s1600/2_3.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="712" data-original-width="660" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSxMbZPs6D6ctRwphHwMLXST_ygasff_ZgQk0HlddI58WlVZVWNcklqzaWeeRLABUaUvMmDZbcLOzWIAlSsjU44rWGYfrKRRnPuw9atSf0c4q5jh6ZsEU2_26i3k1yjY6KEKyeJ5VHk6qb/s320/2_3.png" width="296" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">What is the N of this design: 93 or 8?</td></tr>
</tbody></table>
The problem here is that stimuli (in this case, confederates)
are <a href="https://www.researchgate.net/profile/Charles_Judd/publication/308756220_Experiments_with_More_Than_One_Random_Factor_Designs_Analytic_Models_and_Statistical_Power/links/57f5503608ae91deaa5c79bf/Experiments-with-More-Than-One-Random-Factor-Designs-Analytic-Models-and-Statistical-Power.pdf">nested
within condition</a>, just like participants are nested within condition in a
between-subjects design. In order to generalize our results beyond the specific
people who happen to be in our sample, we have to treat participant as a random
factor in our designs. The same logic applies to stimuli: When they are nested
within condition, we need to treat stimuli (e.g., confederates) as random
factors because we want to generalize the beyond the 2 or 4 or 8 confederates who
happened to be part of our study. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div style="text-align: right;">
</div>
<div class="MsoNormal">
What happens if you regularly equate confederate with
condition and use small samples of stimuli? Your effect size estimates will tend
to be <i>extremely</i> unstable. Consider <a href="http://www.sciencedirect.com/science/article/pii/S1090513810000759">this
study</a>, which used N = 389 participants but only 10 male and 11 female
confederates. They found an enormous sex difference <u>in the <i>opposite</i> direction from the study
described above</u>: Confederate attractiveness affected women’s romantic
desire much more strongly than men’s. If you were including this study in <a href="http://pauleastwick.com/s/Eastwick2014PBull.pdf">a meta-analysis</a>, it
would be more appropriate to assign it a N of 21 rather than 389 to reflect the
imprecision of this particular sex-difference estimate.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
So what to do? Power calculations with these designs <a href="https://jakewestfall.shinyapps.io/two_factor_power/">are complex</a>, but
a good start would be to use at least N = 40 or 50 stimuli per condition and
treat stimuli as a random factor. Then, any incidental differences between the experimental
stimuli would likely wash out, and we could be reasonably confident that any effects
of the “manipulation” were truly due to attractiveness. Yes, that’s probably
too many stimuli for a study involving live confederates, so you may need to
get creative—for example, many speed-dating studies provide this kind of
statistical power. [2]<o:p></o:p></div>
<div class="MsoNormal">
<span class="MsoFootnoteReference"><span class="MsoFootnoteReference"><span style="font-family: "calibri" , "sans-serif"; font-size: 11.0pt; line-height: 115%;"><br /></span></span></span></div>
<div class="MsoNormal">
It’s easy to get tripped up by this issue, especially when
you have confederates that you’ve carefully selected to differ in an obvious
way. But don’t make the mistake. If a confederate is nested within condition in
your design, you likely need to reconsider your design.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
For more information about stimulus sampling challenges, see
detailed discussions by <a href="http://journals.sagepub.com/doi/abs/10.1177/01461672992512005">Wells and
Windschitl</a> and <a href="http://jakewestfall.org/publications/JWK.pdf">Westfall
and colleagues</a>, as well as <a href="http://pauleastwick.com/s/EastwickSmithCRSPFinal.pdf">this paper</a> that
describes stimulus sampling challenges when studying sex differences in
particular.<o:p></o:p></div>
<div class="MsoNormal">
</div>
<div>
<!--[if !supportFootnotes]--><br clear="all" />
<hr align="left" size="1" width="33%" />
<!--[endif]-->
<br />
<div id="ftn1">
<div class="MsoFootnoteText">
<span class="MsoFootnoteReference"><!--[if !supportFootnotes]--><span class="MsoFootnoteReference"><span style="font-family: "calibri" , "sans-serif"; font-size: 10.0pt; line-height: 115%;">[1]</span></span><!--[endif]--></span>
Study 2 of the same paper replicated this interaction using N = 2,781 White participants
and N = 24,124 White and Black targets, which allows us to have more confidence
in the inference that this interaction is about race rather than peculiarities
of particular stimuli. Nevertheless, I assure you that at the time, I would
have tried to publish the two-confederate study on its own had I not had access
to this larger Study 2 sample. <o:p></o:p></div>
<div class="MsoFootnoteText">
<br /></div>
</div>
<div id="ftn2">
<div class="MsoFootnoteText">
<span class="MsoFootnoteReference"><!--[if !supportFootnotes]--><span class="MsoFootnoteReference"><span style="font-family: "calibri" , "sans-serif"; font-size: 10.0pt; line-height: 115%;">[2]</span></span><!--[endif]--></span>
Alternatively, you could manipulate the attractiveness of a single confederate
(e.g., using makeup and clothing); at least one study has successfully done so
(see Figure 1 <a href="http://www.sciencedirect.com/science/article/pii/S1057740810000306">here</a>),
although we have found executing such a manipulation to be challenging in our
lab.<o:p></o:p></div>
</div>
</div>
<div>
<div id="ftn2">
<div class="MsoFootnoteText">
<o:p></o:p></div>
</div>
</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6884882187348242375.post-77758704525887254622018-01-09T10:41:00.000-08:002018-01-09T12:12:35.577-08:00Two Lessons from a Registered Report<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1F1aqJ49iC0s9x4MihySxglDuSI5U0eeRXO967iYui0utECaVJG88rfTda-zSibFhNywpI2GQTVxy0Y-fQ2EAumLmRTL3gRSHfq6ZzAHLuzKgpDyWoStZiTmZMgg40xXm3DW-jOGm3qso/s1600/RRSP.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="715" data-original-width="500" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1F1aqJ49iC0s9x4MihySxglDuSI5U0eeRXO967iYui0utECaVJG88rfTda-zSibFhNywpI2GQTVxy0Y-fQ2EAumLmRTL3gRSHfq6ZzAHLuzKgpDyWoStZiTmZMgg40xXm3DW-jOGm3qso/s320/RRSP.jpg" width="222" /></a>Long ago and far away, in Chicago, in 2006, I submitted one of my first papers as a graduate student. The topic was controversial, and so we were not particularly surprised, when the reviews came back, to see that the reviewers were skeptical of the conclusions we drew from our findings. They wanted more (as JPSP reviewers often do). They thought maybe we had overlooked a moderator or two…in fact, they could think of a whole laundry list of moderators that might produce the effect they thought we should have found in our data. So we ran 1,497 additional tests.<br />
<br />
No, seriously. We counted. 1,497 post-hoc analyses to make sure that we hadn’t somehow overlooked the tests that would support Perspective X. We conducted them all and described them in the article (but there was still no systematic evidence for Perspective X).<br />
<br />
If your work involves controversy, you’ve probably experienced something like this. It’s been standard operating procedure, at least in some areas of psychology.<br />
<br />
Now, fast forward to 2017. My student Leigh Smith and I are about to launch a new study in the same controversial topic area, and it’s likely that we’ll get results that someone doesn’t like, one way or another. But this time, before we start conducting the study, we write up an analysis plan and submit it to Comprehensive Results in Social Psychology (CRSP), which specializes in registered reports. The analysis plan goes out for review, and reviewers—who have <b>the luxury of not knowing </b>whether the data will support Perspective X or Y or Z—thoughtfully recommend a small handful of additional analyses that could shed better light on the research question.<br />
<br />
The analysis plan that emerges is one that everyone agrees should offer the best test of the hypotheses; importantly, the tests will be meaningful however they turn out. We run the study and report the tests. We submit the paper.<br />
<br />
And then, instead of getting a decision letter back asking for 1,497 additional suggestions that someone thought would surely show support for Perspective X…the paper is simply published. The data get to stand as they are, with no poking and prodding to try to make them say something else.<br />
<br />
There’s a lot to like about this brave new world.<br />
<br />
<a href="http://pauleastwick.com/s/EastwickSmithCRSPFinal.pdf">Our new paper in CRSP</a> addresses whether attractiveness (as depicted in photographs of opposite-sex partners) is more appealing to men than to women. I, like most other evolutionary psychologists, had always assumed that the answer to this question was “yes.”<br />
<br />
But you know what? Those prior studies finding that sex difference in photograph contexts? Most of them were badly underpowered by today’s standards. Our CRSP paper used a sample that was powered to detect whether the sex difference was q = .10 (i.e., a small effect) or larger (using a sample of N = ~1,200 participants and ~600 photographs). These photographs came from the <a href="http://faculty.chicagobooth.edu/bernd.wittenbrink/cfd/index.html">Chicago Face Database</a>, and we used the ratings in the database of the attractiveness of each face (based on a sample of independent raters).<br />
<br />
<div style="text-align: left;">
</div>
The paper has two take-home lessons that are relevant to the broader discussion of best practices:<br />
<div style="text-align: left;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: left;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKl84pVYkOS5OBBEmFeI3AQPRj6eEI5IepmZDtJpG1_2Q9Ze7_N_up_GccYDt0NSezHl0iYnT-Hi3gBcviyNNCAtkmzadgq12p2erLTZ5hWg0EyxSQct07Iwvwj5BvI9bJveuAZCxiCTz7/s1600/Faces.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="497" data-original-width="350" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKl84pVYkOS5OBBEmFeI3AQPRj6eEI5IepmZDtJpG1_2Q9Ze7_N_up_GccYDt0NSezHl0iYnT-Hi3gBcviyNNCAtkmzadgq12p2erLTZ5hWg0EyxSQct07Iwvwj5BvI9bJveuAZCxiCTz7/s320/Faces.png" width="225" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Is attractiveness more appealing to men<br />than to women when people look at photographs?<br />Yes, although the effect is quite small, and<br />there's little evidence of hidden moderators.</td></tr>
</tbody></table>
1. Even though prior studies of this sex difference were underpowered, the sex difference was there in our new study: r(Men) = .41, r(Women) = .28, q = .13, 95% CI (.18, .08). There is no chance that the prior studies were powered to find a sex difference as small as what we found. But it was hiding in there, nevertheless.[1]<br />
<br />
<b>Lesson #1: Perhaps weakly powered studies in the published literature can still manage to converge on truth.</b> At least, perhaps this happens in cases where the presence or absence of p < .05 is/was not a hard criterion for publication. Sex differences might be one such example. (Still no substitute for a high powered, direct test, of course.)<br />
<br />
2. In this literature, scholars have posited many moderators in an attempt to explain why some studies show sex differences and some do not. For example, sex differences in the appeal of attractiveness are supposed to be bigger when people imagine a serious relationship, or when people evaluate potential partners in the low-to-moderate range of attractiveness. Sometimes, sex differences are only supposed to emerge when 2 or 3 or 4 moderators combine, like the Moderator Avengers or something. That wasn’t the case here: These purported moderators did not alter the size of the sex difference in the predicted manner, whether alone or in Avenger-mode combination. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<b>Lesson #2: Perhaps we should be extremely skeptical of moderators that are hypothesized, frequently post hoc, to explain why Study X shows a significant finding but Study Y does not. </b>Moderators within study? I’m on board. Moderators across studies? I’ll believe it when I see it meta-analytically.<br />
<br />
For every single research question I dream up going forward, I will consider whether it could be a good candidate for a registered report. When I think about an idealized, all-caps form of SCIENCE that stays untethered from prior perspectives or ideology, that CRSP experience pretty much captures it. [2]<br />
<br />
Notes:<br />
<br />
[1] This statement may shock some who think of me as some sort of sex-differences naysayer. Rather, my perspective is that this sex difference is larger in photograph contexts than live face-to-face contexts. Indeed, q = .13 is about 2-4 times larger than <a href="http://pauleastwick.com/s/Eastwick2014PBull.pdf">meta-analytic estimates</a> of the same sex difference in initial attraction contexts or established close relationships (which are q = .05 or smaller). (Does it make me a naysayer to suggest that the sex differences here are extremely small, and that prior single studies are unlikely to have been powered to detect them?)<br />
<br />
[2] And did I mention fast? This project went from “vague idea” to “in press” in less than 11 months. My prior best time for an empirical piece was probably twice as long.
Unknownnoreply@blogger.com1