Blog

Note: The report is posted on the Chimes, Inc. website. See:http://www.chimes.org/news/20150622.htm. 

 

Funding for this study is very likely to have influenced the methodology used and conclusions in the published report. This study and the published report was funded by the Chimes Foundation, a foundation supporting Chimes, Inc. which is a very large, multi-million dollar provider – not located in Maine – that relies heavily on the sheltered workshop model for its revenue and business operations.

 

As clearly stated in the Chimes news release on its website that links to the study, “Neither the authors, nor Chimes, considers this paper a definitive answer on where we should be going, or the best ways to get there.” Given this, it is irresponsible of ACCSES or any other organization to suggest that this study “can serve as a guidepost for policymakers in other states.” We find this study to have many weaknesses in methodology and it fails to provide sufficient evidence to counter the body of peer-reviewed literature that exists which consistently suggests that transitioning people out of segregated sheltered workshops to integrated employment and other integrated support services is good public policy.

 

Chimes itself makes reference to “the limited sample available to Dr. Phoenix” but both Chimes and ACCSES fail to acknowledge that such a limited approach was Dr. Phoenix’s choice. The “sample of convenience” approach taken by Dr. Phoenix was fraught with weaknesses and led to many Maine providers that had closed their workshops not being fully represented in the report, if at all. As a result, in many ways, this report is based on “anecdotal evidence rather than hard fact,” placing it in a category with other studies Chimes criticizes (although there is no mention of which specific studies Chimes may be referring to).

• Only a handful of providers who formerly operated sheltered workshops in Maine were involved in the study: just seven (7) out of a total of seventeen (17) providers. The providers were not randomly selected. Instead the authors report they were referred by a provider trade association or fellow providers, creating the very strong possibility of bias in this small sample. As well, some of the seven (7) providers chose not to become approved providers of supported employment services; thus they were not in a position to even attempt to assist the individuals they served to transition successfully to integrated supported employment. Further, some of the providers who participated in this study do not currently offer employment services. It is questionable whether any of these providers could offer a valid perspective on individuals’ current employment status and how individuals have fared as a result of the policy change since they are not – and in some cases never were – providers of supported employment services.

• Only five (5) individuals (out of an estimated 220 people) who were working in sheltered workshops in Maine when the policy change took place were interviewed as part of this study. According to the report authors, these individuals were recommended by just two (2) of the seventeen (17) former sheltered workshop providers. They were not randomly selected from the entire population of individuals who were working in sheltered workshops in Maine when the policy change took place. This is not a representative sample in terms of size or selection method and therefore cannot be used to draw any conclusions about how the group of individuals transitioned view the impact and outcomes of these transitions.

Conclusions are based on very limited data, the validity of which was not addressed.

The authors collected “available data” from provider employment records. This data came from just four (4) former sheltered workshop providers and no description of how the data was validated can be found in the report.

• Employment status data came from just three (3) sheltered workshop providers.
• Data on comparative wages and days (not hours) worked per week data came from just one (1) sheltered workshop provider.
• No data was obtained from providers regarding actual hours worked per week.

Such limited data is being used to draw conclusions on the statewide impact of Maine’s policy change, which simply does not meet any reasonable standard for rigorous research. Without question, this data is totally insufficient for Chimes to conclude “the approach as implemented in Maine did not serve all people well” and for ACCSES to draw any of the conclusions it makes in its press release.

 

Conclusions drawn from other data sources are incorrect and misleading.

For example, the authors quotes a published data source as reporting people with intellectual disabilities in Maine work an average of only 12 hours per week in 2011, the lowest in the nation (page 4). In fact, this data relates only to thirty-two (32) individuals placed into competitive integrated employment by the Maine vocational rehabilitation agency during program year 2012. It is not accurate to report that 12 hours per week is the average hours worked for all individuals with intellectual disabilities working in integrated employment in Maine. In fact, 2012 data tracking individuals with intellectual disabilities receiving Medicaid waiver services (including those who left sheltered workshops as a result of Maine’s policy change) shows Maine is above the national average in terms of numbers receiving integrated employment services, outpacing 39 other states.[1]

 

Having Institutional Review Board approval does not guarantee a valid and reliable study. In its press release, ACCSES clearly misunderstands research protocols, implying that approval to conduct a study by an Institutional Review Board (IRB) assures the study’s validity and reliability. It does not. IRB is in place to protect human subjects in research. The purpose of the IRB is to ensure that all human subject research be conducted in accordance with all federal, institutional, and ethical guidelines. The weaknesses in research design, sample size, and data collection/analysis are not negated by IRB approval.

 

Documentation of Maine’s systems change effort is severely lacking in the report. The process of transformation actual began in 1999, with significant investment in building blocks Public Law Chapter 101. Very little of the extensive history is included in the report.

 

 

 

[1] Butterworth, J., Smith, F. A., Hall, A.C., Migliore, A., Winsor, J., & Domin, D. (2014). StateData: The national report on employment services and outcomes. Boston, MA: University of Massachusetts Boston, Institute for Community Inclusion. See page 20.