Hello, Colleagues -
You might be interested in either reading or participating in an on-line discussion on orthotic efficacy research for children with CP. Shortly after reading the research report by Sandra A. Radtka and others in the April, 1997 issue of Physical Therapy (Volume 77(4):395-408 ), I wrote a detailed critique which has been revised and reduced by half, and still will apparently not be published by the Physical Therapy journal in any form. I've decided not to cut it down further.
The latest version of the critique (10 web pages including 15 references) is posted on my web site <http://www.gaitways.com/radtka.html>. I've invited the authors to reply to this critique, and will add their replies to my paper. If you wish to check back later on this winter, you might find their comments at the same web site.
If you want more explanation as to it's existence, and as to my diligence in putting it out to my colleagues in pediatric rehabilitation, read on...
I wrote the response paper to alert the researchers involved in this project, and future researchers addressing similar topics, to:
I also wrote it to try to inspire the editorial review team at Physical Therapy to apply more rigorous academic standards to their review of such projects in the future. By way of example, this paper not only features inaccurate data (a discovery made by Judy Carmick, PT and her husband, Al Shepherd, as the numbers did not "compute" logically in their highly analytical minds - the data will be corrected in the March issue of Physical Therapy, I am told.) Furthermore, it contains an illustration of only 1 of the 2 orthoses studied, cites different body landmarks of reference to compare the shaft lengths of the two devices, and lacks any detail regarding the "clinical evaluation" that prompted the authors to prescribe one orthosis over the other in a clinical setting.
I think we can expect better than this.
If, having read the research paper, you, too, have concerns, I hope you will voice them in a letter to the editor of Physical Therapy, Jules Rothstein, PhD, PT, at < ptjournal@apta.org >, and that he will consent to print it.
My thanks for your attention to this message, and my apologies in advance for any inconvenience it's arrival might have caused those of you who do not work clinically with children with CP.
Beverly Cusick, MS, PT 970/728-0270 FAX: 970/728-3304 PO Box 600 Placerville, CO 81430
bcusick@gaitways.com http://www.gaitways.com
Dear Beverly and others interested in the AFO discussion,
Happy new year to all. Thanks to Beverly for starting with this controversial but welcome topic of AFO prescription.
I have just had a look at the paper in question, and I have to say I completely agree with Beverly that it is very suspect indeed. However, my own concerns are related to the statistical analysis, which I'm surprised Beverly didn't mention. I don't think the paper is alone in the rehab field in it's abuse of statistics, so I think it's worth discussing further.
The main thing that struck me was the enormous number of tables and statistical results reporting on a sample of 10 subjects! I believe this common phenomenon merits some sort of eponymous ratio, such as the inverse-Fisher (IF) coefficient (number of statistics reported divided by number of significant findings). I believe this paper would score highly...
It's remarkable that the authors take the trouble to address this issue:
"For all non-significant dependant variables in the study, the effect sizes were less than 0.38 and the power values were less than 0.45. Because these values were low, inferences regarding the non-significant findings were prone to a Type II error. A larger sample size would be needed to increase the power of the tests".
Yet this observation does not seem to inhibit them from concluding that "No differences were found for the gait variables when comparing the two orthoses."
My statistician friend tells me that statistics ought never even to be applied to samples smaller than about 30 subjects. Even that number is stretching the assumptions made to the limit. My own view is that modern stats have become so complicated that it is often very difficult to know what they mean. For this reason, amongst others, we have just submitted a new method for the calculation of "consistency" to the Journal of the Royal Statistical Society (A Suggestion For An Alternative Coefficient Of Consistency. Sommer JP & Kirtley C). We believe that clinical measurements ought to stand up to the same standards of accuracy as do items of measuring equipment, such as force platforms etc., with a clearly understood accuracy tolerance (+/- 5% etc.). I believe that if our method had been used with the data in question, it would have prevented the researchers from making such ill-judged conclusions.
If I had my way, I think I would rather go back to an earlier part of this century (pre-Gosset & Fisher) when modern statistics weren't yet invented. This didn't seem to prevent such greats as Sherrington and AV Hill from doing good work! However, if we have to have them (since the journals seem to require them) then we need to all learn to use them properly!
As a final comment, I'd like to refer those interested to a wonderfully cynical destruction of the whole concept of hypothesis testing:
An Applied Statistician's Creed (1996) Marks R Nester, Applied Statistics (Journal of the Royal Statistical Society, Series C) 45 (4): 401-410. I'm CCing this to him, hoping he'll comment.
Look forward to a lively debate - thanks again to Beverly for initiating it!
Chris -- Dr. Chris Kirtley (Kwok Kei Chi) MD PhD Assistant Professor Department of Rehabilitation Sciences The Hong Kong Polytechnic University
Dr. Kirtley,
I read Beverly's article and her concerns seem to be justified. However, to be fair to the original authors I should state that I have not looked at their paper. Also, I am not familiar with this medical area so I do not possess the necessary background knowledge and experience which is also very important in formulating judgments in these kinds of situations. However, I do know a bit about statistics.
-------------------------------------------------
I do not necessarily have a problem with a large number of tables. If all of the tables would have been justified for a large number of subjects then they may be worthwhile presenting even for a small number of subjects.
-----------------------------------------------
It's remarkable that the authors take the trouble to address this issue: "For all non-significant dependant variables in the study, the effect sizes were less than 0.38 and the power values were less than 0.45. Because these values were low, inferences regarding the non-significant findings were prone to a Type II error. A larger sample size would be needed to increase the power of the tests". Yet this observation does not seem to inhibit them from concluding that "No differences were found for the gait variables when comparing the two orthoses."
Yes, this does worry me. You obviously already know why I am concerned.
--------------------------------------------
My statistician friend tells me that statistics ought never even to be applied to samples smaller than about 30 subjects. Even that number is > stretching the assumptions made to the limit.
I disagree with this. It is unwise to nominate 30 as a mininum number of subjects. It is possible that 10 subjects may be sufficient for a particular investigation, depending inter alia on study objectives and sampling variation.
Also, alternative methods of statistical analysis may require less stringent assumptions.
------------------------------------------
I would appreciate a reprint of your paper, if that is possible.
-------------------------------------------
If I had my way, I think I would rather go back to an earlier part of > this century (pre-Gosset & Fisher) when modern statistics weren't yet > invented. This didn't seem to prevent such greats as Sherrington and AV > Hill from doing good work! However, if we have to have them (since the > journals seem to require them) then we need to all learn to use them > properly!
Then you may appreciate the sequel paper to my creed. Unfortunately the gist of it is still residing in my head and I just haven't had time to put pen to paper and do it justice.
-------------------------------------------
As a final comment, I'd like to refer those interested to a wonderfully cynical destruction of the whole concept of hypothesis testing: An Applied Statistician's Creed (1996) Marks R Nester, Applied Statistics (Journal of the Royal Statistical Society, Series C) 45 (4): 401-410.
I enjoy being acknowledged as a cynic.
Regards Marks Nester
Marks R. Nester
Senior Biometrician Queensland Forestry Research Institute Department of Primary Industries M.S. 483 Fraser Road Gympie 4570 Australia Telephone: 07 5482 0887 FAX: 07 5482 8755 email: nesterm@qfri1.se2.dpi.qld.gov.au