step-by-step guide to performing experiments online

As it stands, there are very few places on the interwebs that allow online field experiments in organizations. I wanted to change that with my r3ciprocity.com project. The r3ciprocity project is not only focused on helping graduate students (i.e., Masters and PhD-level students) and faculty make better decisions, get feedback on their writing, and connect these people with great editors / proofreaders. (Check out this blog post I wrote on getting feedback on your writing.) A central tenet of the mission of the r3ciprocity.com project is to aid in the construction and development of valid organizational science. In other words, a central reason why I created the platform was to help scientists do better science. In this blog post, I am going to detail how the r3ciprocity.com platform is open to serious academics and researchers who want to perform online experiments on the platform. I am going to provide a brief overview of the problem of valid science, provide advantages and disadvantages of these field experiments, detail some of the ethical concerns, and provide a step-by-step guide if you want to contact me about performing experiments on the platform. Note that the focus on this blog post details my thoughts on the subject and is not meant to be an academic article, nor is it representative of my institution. It reflects my role as the CEO / Founder of the r3ciprocity project, and is meant for an audience that is ‘nerdy’ (like me :-) ) but not not necessarily academic. By the way, I have been discussing this idea for well over 4 years, so I am pretty well acquainted with its issues, but I am still learning. The Problem With Valid Organizational Science. Organizational science has been going through a large transition in the last decade, just like in science. Many more people are interested in the validity of the findings scientific articles. The primary reason has to do with the reproducibility problem that we have. Most people just assumed that our findings are reproducible, until researchers tried to reproduce past scientific findings (See the work of Brian Nosek, for example). We found that it was difficult to reproduce some of these findings. Don’t get me wrong - the lack of reproducibility is by no means a reason to disregard scientific findings altogether. This is not due to people cheating the academic system (perhaps just a small number of people would do that, or at least intentionally), or people being devious, but from my perspective, the lack of reproducibility is due to the simple fact that is difficult to measure social constructs and the performance of the social constructs. Social constructs are often interrelated in non-linear ways. The real world is messy and it is difficult to measure this mess. As somebody who considers themselves a social scientist, let me tell you that it is difficult to observe these social constructs, and we do spend years trying to uncover these constructs and the relationships between constructs. Almost all of my papers have taken years of my life to explore social phenomenon. However, this still does not mean that we cannot improve the validity organizational science. I am forever looking for ways to become a better scientist, and that is why I am attracted to the idea of management experiments. Management Experiments. Interestingly, despite the fact that most of us are trained in the scientific method in the second and third grade, experimental research about organizations and the social constructs that occur within organizations is still rather lacking. From my perspective, management experiments involve using randomized controlled trials (RCTs) to uncover different aspects of management and organization. The primary benefit is that by having RCTs, you can rule out many other factors. For example, years ago, I worked at a plastics plant and we were testing the burn rates of different forms of plastic. By randomizing the conditions (say oxygen levels), we were able to get a good understanding of the burn rates of the different plastics. Essentially, by using RCTs, you increase the validity of the findings and the research study. Management experiments is not the same as simply trying something and seeing what happens. It involves having a randomized control and treatment group. There is a whole field called the design of experiments which I won’t get into here. Virtually, this is unheard of in most companies because it is difficult to do and expensive. However, some companies (i.e., Google) are actively trying to implement experimental methodologies within their organization. Thus, it comes as no surprise that experimental methodologies were overlooked for a long time in management research. If you are interested in what I term the experimental economy - how experimental techniques are becoming more important to business and innovation, you can watch my YouTube video here: https://youtu.be/m2EBNTpoSoU I also did a video where I tried something novel: Barefoot running for a month in Florida. You can watch the lessons from this 'experimentation' (not a true experiment) below. Let's just say that running with no shoes in 100 degree (30 degree C) weather on pavement is bit ouchy. While I did not stick with the technique, I did completely change my running style and have not looked back since. https://youtu.be/ZnecY8910kE Design Of Online Experiments In Organizations. After building out the r3ciprocity.com project for about six months, I began to realize that this could be an important platform to help improve the validity of organizational sciences. By having an understanding in software development and engineering, I began to realize that we can open up the back-end to researchers that might be interested in performing online experiments through A/B testing. A/B testing is now standard practice on online platforms to ensure the validity of software insights and improve the user experience. This is relatively advantageous because there are many existing tools that can be easily implemented. Further, I think most people are getting used to the idea of experimentation on webpages. Step-by-Step Guide To Performing Experiments On The Platform. While I have not implemented standard operating procedures and how to deal with opening up online experimentation just yet, I am very much interested in collaborating with researchers who are interested in performing online experiments. If you are interested in performing online experimentation on the r3ciprocity.com platform, please send a one-page document detailing your experiment with hypotheses to contact@r3ciprocity.com. Here are the requirements: One-page document detailing your experiment with hypotheses. Must include: Research background and theory. Proposed hypotheses. Expected findings. An attached CV, detailing your affiliation. A statement certifying that you will get IRB approval from your university before the study begins. A statement that r3ciprocity.com is not legally responsible for any complications and legal expenses that are the result of your study. You have to be an active researcher in either the behavioral or organizational sciences with a PhD or a PhD student from a legit university, and you will have to get IRB (ethics) approval from your university. My goal is to eventually make this a free service for anyone that is doing research from a legitimate university in the US. How To Set Up A Scientific Advisory Board For Online Experiments? I need help with setting up an advisory board. If you want to help me set up a not-for-profit to manage this process, and some sort of scientific advisory board, please send a note to contact@r3ciprocity.com, attach your CV, and include a reference that I might check. Please only apply if you are nice - Life is too short to deal with a lot of drama. Also note, that the scientific advisory board should be voluntary or set up in the same way as an editorial board on a good journal to prevent conflict of interests. We will have to at least look into how to build out this potential capability. I know that there are many questions and concerns that people would have to address before it can occur. This will likely be a few months (even a few years away), but I think it is important to start building these capabilities early so that others may find it useful. So, feel free to give me a shout. Advantages And Disadvantages Of These Online Field Experiments I think there are many advantages of performing online field experiments. Validity - Using randomized controlled trials can dramatically improve the validity of the insights organizational science. This, combined with pre-registering (https://osf.io/), can provide insights that are difficult to capture with archival data, interview data, or survey data. For example, archival data tends to show what happened, but we cannot be sure why it happened. Interview and survey data tends to focus on people’s thoughts about why they thought it happened, but sometimes intentions are different than behaviors. They don’t capture behaviors. Ease and expense - Compared to normal lab experiments, online experimentation tends to be less expensive and easier to implement. The major problem is having a large enough audience that people will want to participate in an online experiment. Availability of tools. There are many tools available to capture different behaviors in the insights (ie. Google Optimize). There are also disadvantages of performing online field experiments. Building An Audience. The biggest problem from my perspective it is having a large enough audience that would be interested in participating. It is extremely difficult to build up an online audience and takes years of dedication to the project. (Websites typically are penalized in the search engines if they do not have enough content or links to their material). Ethical Concerns in Experimental Research. The ethical concerns of experimental research are real, and I worry about these ethical concerns on a regular basis. Anytime that you are providing different conditions to people, you may be disadvantaged in one of these groups. Clearly, I will need an advisory board and arms-length organization (ie., a not for profit) that separates the income generating side of r3ciprocity.com from the scientific side of r3ciprocity.com. See above if you are qualified, nice, and want to help. My answer with ethical concerns is to always be nice, and the golden rule: To do onto others as I would have done onto me. Finding a Relevant Context. I think a major reason why search online experiments have not being performed in the past is that it is difficult to get access to managers, or people that are willing to see the value research. Because I am a professor, I greatly value research and see what science can do for the greater good. You might also want to read a few more posts: Don't say this to PhDs! Building an online writing community for PhDs and faculty. A proofreading / editing site for good. Anyway, I hope this blog post helps, and please check out the r3ciprocity.com project. Got to go hang out with my kids. Take care!