Consider the quality of a research article

Experts will tell you there are many factors to consider about the quality of research before relying on the tested intervention. For those unfamiliar with research design or statistical analysis, consider the following basic questions when examining overall quality of a research study:

1. Is it clinically significant? Knowledge about research-based massage techniques is helpful in practice, but which methods are best to employ for a given condition? Those with some research knowledge are trained to give more credibility to modalities presented in statistically significant studies (usually p<0.05), however statistical significance does not always mean that an intervention was clinically effective. As we learned from Betty Chou's post, statistical significance and clinical significance are not the same thing. Statistical significance allows you to draw conclusions about a population from the sample, but clinical significance is a measure of how effective the intervention was in the sample. Let’s say a study was done on a new treatment for improvement in shoulder ROM and it was found to be statistically significantly better than the old treatment. That sounds good. But when we find out that the improvement was only five degrees of motion more than the old treatment, it isn’t very clinically significant. Simply put, meaningful results can be statistically insignificant. As the practitioner you have the power to decide if the treatment’s effect is strong enough to apply to your practice. If you decide to implement an intervention with clinically meaningful results, you’ll still want to investigate the proposed treatment’s potential interactions with all of your patient’s therapies and existing conditions. It could also help to ask your patient what his/her prior experience is with the proposed therapy.

2. Did researchers select subjects without bias? In other words, was the participant selection random?

3. Were the study groups comparable? Researchers typically provide a table of subject demographics and should ensure this information is not significantly different across groups. If the average age of participants in the control group versus the treatment group was 50 years different, one group would naturally exhibit better therapeutic outcomes than the other based on age alone. If significant differences at baseline exist, the researcher may control for this statistically.

4. Was bias controlled for in the research design? Researchers can control for bias on many levels; the highest level is a double-blinded study where researchers and subjects are unaware of the group to which they belong. True double-blinding is nearly impossible in massage research, but efforts are made nonetheless.

5. Were intervening factors described? It’s critical that researchers acknowledge the potential outside factors that can influence treatment outcomes. These could include events like a participant’s fall or accident during the treatment series that impacted outcomes, or a change in medication, or some missed appointments. These extraneous variables can be controlled for statistically or in research design.

Research is an extremely useful tool in the development of your practice and quality of patient care. You have the authority to decide if a proposed intervention is valuable to your clients. But if you determine a treatment’s worth by statistical significance alone, you might overlook some research that could greatly improve your clients’ outcomes.

References

Man‐Son‐Hing, M., Laupacis, A., O’Rourke, K., Molnar, F. J., Mahon, J., Chan, K. B., & Wells, G. (2002). Determination of the clinical importance of study results. Journal of general internal medicine, 17(6), 469-476.

Statistical significance and clinical importance. Retrieved from http://www.med.uottawa.ca/sim/data/Statistical_significance_importance_e.htm

Walker, I. (2008). Null hypothesis testing and effect sizes. Retrieved from http://staff.bath.ac.uk/pssiw/stats2/page2/page14/page14.html