Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p lt .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior t...o conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr =.002 or.014, depending on analytic approach). The median effect size for the revised protocols (r =.05) was similar to that of the RP:P protocols (r =.04) and the original RP:P replications (r =.11), and smaller than that of the original studies (r =.37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r =.07, range =.00–.15) were 78% smaller, on average, than the original effect sizes (median r =.37, range =.19–.50).
Keywords:
reproducibility / replication / Registered Reports / preregistered / peer review / open data / metascience
Source:
Advances in Methods and Practices in Psychological Science, 2020, 3, 3, 309-331
Publisher:
SAGE Publications Inc.
Funding / projects:
French National Research Agency (ANR-15-IDEX-02; H. IJzerman)
Netherlands Organization for Scientific Research (NWO; 016.145.049; H. IJzerman)
National Institute on Alcohol Abuse and Alcoholism (F31AA024358; M. H. Bernstein)
Social Sciences and Humanities Research Council of Canada (149084; M. Inzlicht)
Economic and Social Research Council (United Kingdom; ES/ L01064X/1, H. Rabagliati)
TY - JOUR
AU - Ebersole, C.R.
AU - Mathur, M.B.
AU - Baranski, Erica
AU - Bart-Plange, D.-J.
AU - Buttrick, N.R.
AU - Chartier, Christopher R.
AU - Corker, K.S.
AU - Corley, M.
AU - Hartshorne, J.K.
AU - IJzerman, Hans
AU - Lazarević, Ljiljana
AU - Rabagliati, H.
AU - Ropovik, I.
AU - Aczel, Balazs
AU - Aeschbach, L.F.
AU - Andrighetto, L.
AU - Arnal, Jack D.
AU - Arrow, H.
AU - Babincak, Peter
AU - Bakos, B.E.
AU - Banik, Gabriel
AU - Baskin, Ernest
AU - Belopavlović, R.
AU - Bernstein, M.H.
AU - Białek, M.
AU - Bloxsom, Nicholas G.
AU - Bodroža, Bojana
AU - Bonfiglio, D.B.V.
AU - Boucher, L.
AU - Brühlmann, F.
AU - Brumbaugh, C.C.
AU - Casini, E.
AU - Chen, Y.
AU - Chiorri, C.
AU - Chopik, William J.
AU - Christ, O.
AU - Ciunci, A.M.
AU - Claypool, H.M.
AU - Coary, S.
AU - Čolić, M.V.
AU - Collins, W.M.
AU - Curran, P.G.
AU - Day, C.R.
AU - Dering, B.
AU - Dreber, Anna
AU - Edlund, J.E.
AU - Falcão, F.
AU - Fedor, A.
AU - Feinberg, L.
AU - Ferguson, I.R.
AU - Ford, M.
AU - Frank, Michael C.
AU - Fryberger, E.
AU - Garinther, A.
AU - Gawryluk, K.
AU - Ashbaugh, K.
AU - Giacomantonio, M.
AU - Giessner, S.R.
AU - Grahe, Jon E.
AU - Guadagno, R.E.
AU - Hałasa, E.
AU - Hancock, P.J.B.
AU - Hilliard, R.A.
AU - Hüffmeier, J.
AU - Hughes, S.
AU - Idzikowska, K.
AU - Inzlicht, Michael
AU - Jern, A.
AU - Jiménez-Leal, W.
AU - Johannesson, Magnus
AU - Joy-Gaba, Jennifer
AU - Kauff, M.
AU - Kellier, D.J.
AU - Kessinger, G.
AU - Kidwell, Mallory C.
AU - Kimbrough, A.M.
AU - King, J.P.J.
AU - Kolb, V.S.
AU - Kołodziej, S.
AU - Kovacs, M.
AU - Krasuska, K.
AU - Kraus, S.
AU - Krueger, L.E.
AU - Kuchno, K.
AU - Lage, C.A.
AU - Langford, E.V.
AU - Levitan, C.A.
AU - de Lima, T.J.S.
AU - Lin, H.
AU - Lins, Samuel
AU - Loy, J.E.
AU - Manfredi, D.
AU - Markiewicz, Ł.
AU - Menon, M.
AU - Mercier, B.
AU - Metzger, M.
AU - Meyet, V.
AU - Millen, A.E.
AU - Miller, Jeremy K.
AU - Montealegre, A.
AU - Moore, D.A.
AU - Muda, R.
AU - Nave, G.
AU - Nichols, A.L.
AU - Novak, S.A.
AU - Nunnally, C.
AU - Orlić, Ana
AU - Palinkas, A.
AU - Panno, A.
AU - Parks, K.P.
AU - Pedović, I.
AU - Pękala, E.
AU - Penner, M.R.
AU - Pessers, S.
AU - Petrović, B.
AU - Pfeiffer, T.
AU - Pieńkosz, D.
AU - Preti, E.
AU - Purić, Danka
AU - Ramos, T.
AU - Ravid, J.
AU - Razza, T.S.
AU - Rentzsch, K.
AU - Richetin, J.
AU - Rife, S.C.
AU - Rosa, A.D.
AU - Rudy, K.H.
AU - Salamon, J.
AU - Saunders, B.
AU - Sawicki, P.
AU - Schmidt, Kathleen
AU - Schuepfer, K.
AU - Schultze, T.
AU - Schulz-Hardt, S.
AU - Schütz, A.
AU - Shabazian, A.N.
AU - Shubella, R.L.
AU - Siegel, A.
AU - Silva, R.
AU - Sioma, B.
AU - Skorb, L.
AU - de Souza, L.E.C.
AU - Steegen, Sara
AU - Stein, L.A.R.
AU - Sternglanz, R.W.
AU - Stojilović, D.
AU - Storage, Daniel
AU - Sullivan, G.B.
AU - Szaszi, B.
AU - Szecsi, Peter
AU - Szöke, O.
AU - Szuts, A.
AU - Thomae, M.
AU - Tidwell, N.D.
AU - Tocco, C.
AU - Torka, A.-K.
AU - Tuerlinckx, Francis
AU - Vanpaemel, Wolf
AU - Vaughn, Leigh Ann
AU - Vianello, Michelangelo
AU - Viganola, D.
AU - Vlachou, M.
AU - Walker, R.J.
AU - Weissgerber, Sophia C.
AU - Wichman, A.L.
AU - Wiggins, B.J.
AU - Wolf, D.
AU - Wood, M.J.
AU - Zealley, D.
AU - Žeželj, Iris
AU - Zrubka, M.
AU - Nosek, Brian A.
PY - 2020
UR - http://reff.f.bg.ac.rs/handle/123456789/3138
AB - Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p lt .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr =.002 or.014, depending on analytic approach). The median effect size for the revised protocols (r =.05) was similar to that of the RP:P protocols (r =.04) and the original RP:P replications (r =.11), and smaller than that of the original studies (r =.37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r =.07, range =.00–.15) were 78% smaller, on average, than the original effect sizes (median r =.37, range =.19–.50).
PB - SAGE Publications Inc.
T2 - Advances in Methods and Practices in Psychological Science
T1 - Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
EP - 331
IS - 3
SP - 309
VL - 3
DO - 10.1177/2515245920958687
ER -
@article{
author = "Ebersole, C.R. and Mathur, M.B. and Baranski, Erica and Bart-Plange, D.-J. and Buttrick, N.R. and Chartier, Christopher R. and Corker, K.S. and Corley, M. and Hartshorne, J.K. and IJzerman, Hans and Lazarević, Ljiljana and Rabagliati, H. and Ropovik, I. and Aczel, Balazs and Aeschbach, L.F. and Andrighetto, L. and Arnal, Jack D. and Arrow, H. and Babincak, Peter and Bakos, B.E. and Banik, Gabriel and Baskin, Ernest and Belopavlović, R. and Bernstein, M.H. and Białek, M. and Bloxsom, Nicholas G. and Bodroža, Bojana and Bonfiglio, D.B.V. and Boucher, L. and Brühlmann, F. and Brumbaugh, C.C. and Casini, E. and Chen, Y. and Chiorri, C. and Chopik, William J. and Christ, O. and Ciunci, A.M. and Claypool, H.M. and Coary, S. and Čolić, M.V. and Collins, W.M. and Curran, P.G. and Day, C.R. and Dering, B. and Dreber, Anna and Edlund, J.E. and Falcão, F. and Fedor, A. and Feinberg, L. and Ferguson, I.R. and Ford, M. and Frank, Michael C. and Fryberger, E. and Garinther, A. and Gawryluk, K. and Ashbaugh, K. and Giacomantonio, M. and Giessner, S.R. and Grahe, Jon E. and Guadagno, R.E. and Hałasa, E. and Hancock, P.J.B. and Hilliard, R.A. and Hüffmeier, J. and Hughes, S. and Idzikowska, K. and Inzlicht, Michael and Jern, A. and Jiménez-Leal, W. and Johannesson, Magnus and Joy-Gaba, Jennifer and Kauff, M. and Kellier, D.J. and Kessinger, G. and Kidwell, Mallory C. and Kimbrough, A.M. and King, J.P.J. and Kolb, V.S. and Kołodziej, S. and Kovacs, M. and Krasuska, K. and Kraus, S. and Krueger, L.E. and Kuchno, K. and Lage, C.A. and Langford, E.V. and Levitan, C.A. and de Lima, T.J.S. and Lin, H. and Lins, Samuel and Loy, J.E. and Manfredi, D. and Markiewicz, Ł. and Menon, M. and Mercier, B. and Metzger, M. and Meyet, V. and Millen, A.E. and Miller, Jeremy K. and Montealegre, A. and Moore, D.A. and Muda, R. and Nave, G. and Nichols, A.L. and Novak, S.A. and Nunnally, C. and Orlić, Ana and Palinkas, A. and Panno, A. and Parks, K.P. and Pedović, I. and Pękala, E. and Penner, M.R. and Pessers, S. and Petrović, B. and Pfeiffer, T. and Pieńkosz, D. and Preti, E. and Purić, Danka and Ramos, T. and Ravid, J. and Razza, T.S. and Rentzsch, K. and Richetin, J. and Rife, S.C. and Rosa, A.D. and Rudy, K.H. and Salamon, J. and Saunders, B. and Sawicki, P. and Schmidt, Kathleen and Schuepfer, K. and Schultze, T. and Schulz-Hardt, S. and Schütz, A. and Shabazian, A.N. and Shubella, R.L. and Siegel, A. and Silva, R. and Sioma, B. and Skorb, L. and de Souza, L.E.C. and Steegen, Sara and Stein, L.A.R. and Sternglanz, R.W. and Stojilović, D. and Storage, Daniel and Sullivan, G.B. and Szaszi, B. and Szecsi, Peter and Szöke, O. and Szuts, A. and Thomae, M. and Tidwell, N.D. and Tocco, C. and Torka, A.-K. and Tuerlinckx, Francis and Vanpaemel, Wolf and Vaughn, Leigh Ann and Vianello, Michelangelo and Viganola, D. and Vlachou, M. and Walker, R.J. and Weissgerber, Sophia C. and Wichman, A.L. and Wiggins, B.J. and Wolf, D. and Wood, M.J. and Zealley, D. and Žeželj, Iris and Zrubka, M. and Nosek, Brian A.",
year = "2020",
abstract = "Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p lt .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr =.002 or.014, depending on analytic approach). The median effect size for the revised protocols (r =.05) was similar to that of the RP:P protocols (r =.04) and the original RP:P replications (r =.11), and smaller than that of the original studies (r =.37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r =.07, range =.00–.15) were 78% smaller, on average, than the original effect sizes (median r =.37, range =.19–.50).",
publisher = "SAGE Publications Inc.",
journal = "Advances in Methods and Practices in Psychological Science",
title = "Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability",
pages = "331-309",
number = "3",
volume = "3",
doi = "10.1177/2515245920958687"
}
Ebersole, C.R., Mathur, M.B., Baranski, E., Bart-Plange, D.-J., Buttrick, N.R., Chartier, C. R., Corker, K.S., Corley, M., Hartshorne, J.K., IJzerman, H., Lazarević, L., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L.F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., Bakos, B.E., Banik, G., Baskin, E., Belopavlović, R., Bernstein, M.H., Białek, M., Bloxsom, N. G., Bodroža, B., Bonfiglio, D.B.V., Boucher, L., Brühlmann, F., Brumbaugh, C.C., Casini, E., Chen, Y., Chiorri, C., Chopik, W. J., Christ, O., Ciunci, A.M., Claypool, H.M., Coary, S., Čolić, M.V., Collins, W.M., Curran, P.G., Day, C.R., Dering, B., Dreber, A., Edlund, J.E., Falcão, F., Fedor, A., Feinberg, L., Ferguson, I.R., Ford, M., Frank, M. C., Fryberger, E., Garinther, A., Gawryluk, K., Ashbaugh, K., Giacomantonio, M., Giessner, S.R., Grahe, J. E., Guadagno, R.E., Hałasa, E., Hancock, P.J.B., Hilliard, R.A., Hüffmeier, J., Hughes, S., Idzikowska, K., Inzlicht, M., Jern, A., Jiménez-Leal, W., Johannesson, M., Joy-Gaba, J., Kauff, M., Kellier, D.J., Kessinger, G., Kidwell, M. C., Kimbrough, A.M., King, J.P.J., Kolb, V.S., Kołodziej, S., Kovacs, M., Krasuska, K., Kraus, S., Krueger, L.E., Kuchno, K., Lage, C.A., Langford, E.V., Levitan, C.A., de Lima, T.J.S., Lin, H., Lins, S., Loy, J.E., Manfredi, D., Markiewicz, Ł., Menon, M., Mercier, B., Metzger, M., Meyet, V., Millen, A.E., Miller, J. K., Montealegre, A., Moore, D.A., Muda, R., Nave, G., Nichols, A.L., Novak, S.A., Nunnally, C., Orlić, A., Palinkas, A., Panno, A., Parks, K.P., Pedović, I., Pękala, E., Penner, M.R., Pessers, S., Petrović, B., Pfeiffer, T., Pieńkosz, D., Preti, E., Purić, D., Ramos, T., Ravid, J., Razza, T.S., Rentzsch, K., Richetin, J., Rife, S.C., Rosa, A.D., Rudy, K.H., Salamon, J., Saunders, B., Sawicki, P., Schmidt, K., Schuepfer, K., Schultze, T., Schulz-Hardt, S., Schütz, A., Shabazian, A.N., Shubella, R.L., Siegel, A., Silva, R., Sioma, B., Skorb, L., de Souza, L.E.C., Steegen, S., Stein, L.A.R., Sternglanz, R.W., Stojilović, D., Storage, D., Sullivan, G.B., Szaszi, B., Szecsi, P., Szöke, O., Szuts, A., Thomae, M., Tidwell, N.D., Tocco, C., Torka, A.-K., Tuerlinckx, F., Vanpaemel, W., Vaughn, L. A., Vianello, M., Viganola, D., Vlachou, M., Walker, R.J., Weissgerber, S. C., Wichman, A.L., Wiggins, B.J., Wolf, D., Wood, M.J., Zealley, D., Žeželj, I., Zrubka, M.,& Nosek, B. A.. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. in Advances in Methods and Practices in Psychological Science
SAGE Publications Inc.., 3(3), 309-331.
https://doi.org/10.1177/2515245920958687
Ebersole C, Mathur M, Baranski E, Bart-Plange D, Buttrick N, Chartier CR, Corker K, Corley M, Hartshorne J, IJzerman H, Lazarević L, Rabagliati H, Ropovik I, Aczel B, Aeschbach L, Andrighetto L, Arnal JD, Arrow H, Babincak P, Bakos B, Banik G, Baskin E, Belopavlović R, Bernstein M, Białek M, Bloxsom NG, Bodroža B, Bonfiglio D, Boucher L, Brühlmann F, Brumbaugh C, Casini E, Chen Y, Chiorri C, Chopik WJ, Christ O, Ciunci A, Claypool H, Coary S, Čolić M, Collins W, Curran P, Day C, Dering B, Dreber A, Edlund J, Falcão F, Fedor A, Feinberg L, Ferguson I, Ford M, Frank MC, Fryberger E, Garinther A, Gawryluk K, Ashbaugh K, Giacomantonio M, Giessner S, Grahe JE, Guadagno R, Hałasa E, Hancock P, Hilliard R, Hüffmeier J, Hughes S, Idzikowska K, Inzlicht M, Jern A, Jiménez-Leal W, Johannesson M, Joy-Gaba J, Kauff M, Kellier D, Kessinger G, Kidwell MC, Kimbrough A, King J, Kolb V, Kołodziej S, Kovacs M, Krasuska K, Kraus S, Krueger L, Kuchno K, Lage C, Langford E, Levitan C, de Lima T, Lin H, Lins S, Loy J, Manfredi D, Markiewicz Ł, Menon M, Mercier B, Metzger M, Meyet V, Millen A, Miller JK, Montealegre A, Moore D, Muda R, Nave G, Nichols A, Novak S, Nunnally C, Orlić A, Palinkas A, Panno A, Parks K, Pedović I, Pękala E, Penner M, Pessers S, Petrović B, Pfeiffer T, Pieńkosz D, Preti E, Purić D, Ramos T, Ravid J, Razza T, Rentzsch K, Richetin J, Rife S, Rosa A, Rudy K, Salamon J, Saunders B, Sawicki P, Schmidt K, Schuepfer K, Schultze T, Schulz-Hardt S, Schütz A, Shabazian A, Shubella R, Siegel A, Silva R, Sioma B, Skorb L, de Souza L, Steegen S, Stein L, Sternglanz R, Stojilović D, Storage D, Sullivan G, Szaszi B, Szecsi P, Szöke O, Szuts A, Thomae M, Tidwell N, Tocco C, Torka A, Tuerlinckx F, Vanpaemel W, Vaughn LA, Vianello M, Viganola D, Vlachou M, Walker R, Weissgerber SC, Wichman A, Wiggins B, Wolf D, Wood M, Zealley D, Žeželj I, Zrubka M, Nosek BA. Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. in Advances in Methods and Practices in Psychological Science. 2020;3(3):309-331.
doi:10.1177/2515245920958687 .
Ebersole, C.R., Mathur, M.B., Baranski, Erica, Bart-Plange, D.-J., Buttrick, N.R., Chartier, Christopher R., Corker, K.S., Corley, M., Hartshorne, J.K., IJzerman, Hans, Lazarević, Ljiljana, Rabagliati, H., Ropovik, I., Aczel, Balazs, Aeschbach, L.F., Andrighetto, L., Arnal, Jack D., Arrow, H., Babincak, Peter, Bakos, B.E., Banik, Gabriel, Baskin, Ernest, Belopavlović, R., Bernstein, M.H., Białek, M., Bloxsom, Nicholas G., Bodroža, Bojana, Bonfiglio, D.B.V., Boucher, L., Brühlmann, F., Brumbaugh, C.C., Casini, E., Chen, Y., Chiorri, C., Chopik, William J., Christ, O., Ciunci, A.M., Claypool, H.M., Coary, S., Čolić, M.V., Collins, W.M., Curran, P.G., Day, C.R., Dering, B., Dreber, Anna, Edlund, J.E., Falcão, F., Fedor, A., Feinberg, L., Ferguson, I.R., Ford, M., Frank, Michael C., Fryberger, E., Garinther, A., Gawryluk, K., Ashbaugh, K., Giacomantonio, M., Giessner, S.R., Grahe, Jon E., Guadagno, R.E., Hałasa, E., Hancock, P.J.B., Hilliard, R.A., Hüffmeier, J., Hughes, S., Idzikowska, K., Inzlicht, Michael, Jern, A., Jiménez-Leal, W., Johannesson, Magnus, Joy-Gaba, Jennifer, Kauff, M., Kellier, D.J., Kessinger, G., Kidwell, Mallory C., Kimbrough, A.M., King, J.P.J., Kolb, V.S., Kołodziej, S., Kovacs, M., Krasuska, K., Kraus, S., Krueger, L.E., Kuchno, K., Lage, C.A., Langford, E.V., Levitan, C.A., de Lima, T.J.S., Lin, H., Lins, Samuel, Loy, J.E., Manfredi, D., Markiewicz, Ł., Menon, M., Mercier, B., Metzger, M., Meyet, V., Millen, A.E., Miller, Jeremy K., Montealegre, A., Moore, D.A., Muda, R., Nave, G., Nichols, A.L., Novak, S.A., Nunnally, C., Orlić, Ana, Palinkas, A., Panno, A., Parks, K.P., Pedović, I., Pękala, E., Penner, M.R., Pessers, S., Petrović, B., Pfeiffer, T., Pieńkosz, D., Preti, E., Purić, Danka, Ramos, T., Ravid, J., Razza, T.S., Rentzsch, K., Richetin, J., Rife, S.C., Rosa, A.D., Rudy, K.H., Salamon, J., Saunders, B., Sawicki, P., Schmidt, Kathleen, Schuepfer, K., Schultze, T., Schulz-Hardt, S., Schütz, A., Shabazian, A.N., Shubella, R.L., Siegel, A., Silva, R., Sioma, B., Skorb, L., de Souza, L.E.C., Steegen, Sara, Stein, L.A.R., Sternglanz, R.W., Stojilović, D., Storage, Daniel, Sullivan, G.B., Szaszi, B., Szecsi, Peter, Szöke, O., Szuts, A., Thomae, M., Tidwell, N.D., Tocco, C., Torka, A.-K., Tuerlinckx, Francis, Vanpaemel, Wolf, Vaughn, Leigh Ann, Vianello, Michelangelo, Viganola, D., Vlachou, M., Walker, R.J., Weissgerber, Sophia C., Wichman, A.L., Wiggins, B.J., Wolf, D., Wood, M.J., Zealley, D., Žeželj, Iris, Zrubka, M., Nosek, Brian A., "Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability" in Advances in Methods and Practices in Psychological Science, 3, no. 3 (2020):309-331,
https://doi.org/10.1177/2515245920958687 . .
