Development and implementation of developmental screening tools in primary care
Estimates suggest that 15-20 percent of children in the United States are affected by developmental and behavioral disabilities (Boyle et al., 2011; Merikangas et al., 2010). Despite evidence for the importance of early intervention, less than one third of children with such disabilities are diagnosed before they enter school (Sand et al., 2005). As a result, use of evidence-based screening instruments (typically in the form of brief questionnaires for parents) are widely recommended in order to improve detection by pediatricians (Council on Children with Disabilities, 2006), and surveys of physicians suggest that use has increased markedly in the past decade (Radecki, Sand-Loud, O'Connor, Sharp & Olson, 2011).
To be useful to pediatricians, screening tools must be valid indicators of increased risk of developmental and behavioral disabilities, but they must also be feasible in a primary care context (Sheldrick & Perrin, 2009). Ideally, a screening tool should accurately address a range of domains of interest, such as developmental milestones, internalizing and externalizing behaviors, autism symptoms and family risk factors. Moreover, it should be short and easy to administer and to score, possible for parents with varying education levels to complete, as well as low-cost to pediatricians and amenable to flexible administration, including via computers or paper-and-pencil formats. Ideally the instrument should allow for repeated administrations over the course of recurring health supervision visits. Notably, several of these requirements stand in tension with one another: a fully comprehensive screening tool that accurately detects all forms of developmental-behavioral problems is likely to be long, but length limits feasibility. Historically, different screening tools have addressed this tension in different ways. For example, the Ages & Stages Questionnaire, 3rd edition (ASQ-3; Squires, Bricker & Potter, 2009) is a well-validated developmental screening tool, but it requires that parents have access to materials such as blocks and crayons to administer correctly. Moreover, screening social/emotional problems requires administration of a different form known as the Ages & Stages Questionnaire: Social/Emotional (ASQ:S/E; Squires, Bricker & Twombly, 2002). Because each form includes approximately 40 questions, the two are seldom administered together. In contrast, the Parents' Evaluation of Developmental Status (PEDS; Glascoe, 1997) emphasizes feasibility. Ten questions about parents' concerns are designed to screen for both behavioral and developmental problems. Responses are then categorized following a scoring guide, and additional assessment is recommended in some cases.
Development of a new screening tool: The Survey of Wellbeing for Young Children (SWYC)
After surveying available developmental-behavioral screening instruments for children under five years of age, our research team concluded that there was a need for an instrument that struck a different balance between comprehensiveness and feasibility — i.e., a survey that included specific sections relevant to each domain of interest, each of which was short enough to be administered concurrently. To address this gap, we created and recently completed initial validation and replication of the Survey for Wellbeing for Young Children (SWYC). The SWYC is comprehensive in that it includes sections on behavior problems (Sheldrick et al., 2012; Sheldrick et al., 2013), developmental milestones (Sheldrick & Perrin, in press), autism symptoms (Smith, Sheldrick & Perrin, 2012), and family risk factors. Age-specific forms were developed for standard well child-visits recommended by the American Academy of Pediatrics (i.e., 2, 4, 6, 9, 12, 15, 18, 24, 30, 36, 48 and 60 months). Including all sections, each form is presented on two sides of one sheet of paper and includes between 33 and 46 questions, depending on which specific sections are relevant at each age. All forms are freely available on the SWYC website.
In developing the SWYC, feasibility was central to our thinking. Our goal was to maximize the amount of information that could be elicited from parents of young children (birth to 5 years) before pediatric well-child care visits. To maximize completion rates, candidate questions were written at a low reading level, and final forms were limited to those questions for which there was solid evidence of reliability and validity. To facilitate efficient use by pediatric staff, we created simple scoring rules whenever possible.
To allow for flexible administration, no images were included or testing materials required; thus, although the SWYC was designed as a paper-and-pencil instrument, administration by computer or even telephone is possible in the future. Administration by computer also raises the possibility of integration with electronic health records (EHR). Feasibility will be greatly enhanced if SWYC results can be directly incorporated into EHRs. To date, electronic communication between different EHR and personal health record (PHR) systems has been a significant barrier to incorporating evidenced-based screening instruments into existing medical records, in part because protocols and standards for such communication are still emerging (Trotter & Uhlman, 2013). Because SWYC questions are freely available, they can be incorporated into a range of software applications that best meet the needs of individual users, for example, via stand-alone websites designed to communicate with EHR systems, or directly through the EHRs themselves. In addition to our focus on feasibility, we made every attempt to maximize validity. We made extensive use of latent variable modeling and item response theory to identify appropriate questions. Before publication, each section of the SWYC was tested on at least two independent samples. Because our funding precluded inclusion of “gold standard” clinical tests and interviews, initial validation of the SWYC utilized criterion measures such as other validated screening instruments and parents' reports of developmental and behavioral diagnoses. Recently we have received funding from the National Institute of Child Health and Human Development (NICHD) to compare the SWYC and two other prominent screening instruments (the Ages and Stages Questionnaires and the PEDS) against a “gold standard” assessment battery of developmental and social-emotional criterion measures.
Although our commitment to offer the SWYC at no cost has contributed greatly to its initial success, this choice has presented challenges. Development of any assessment instrument is expensive and time-consuming, as is distribution to support implementation. In the case of the SWYC, we were very fortunate to receive early support from the Commonwealth Fund, and NICHD funding will allow us to compare the accuracy of the SWYC to other prominent screening instruments. However, foundations and government agencies that fund research have historically been reluctant to support the development of new assessment instruments. Typically, further research and validation is supported through sales by publishers. A notable exception — and a model for the SWYC — is the Pediatric Symptom Checklist (PSC). After extensive research by a wide array of investigators for over two decades, the PSC was recently selected as a national quality measure (Zima et al., 2013). Adopting a similar model, we are now beginning work with several independent investigators to continue research on the SWYC.
Implementation of screening tools in pediatric settings
As stated above, surveys of physicians suggest that use of screening instruments has increased markedly in the past decade (Radecki, Sand-Loud, O'Connor, Sharp & Olson, 2011). Meanwhile, initiatives by various states have encouraged or even mandated screening. For example, as a result of a policy based on a class action court case, over 70 percent of Medicaid-eligible children in the state of Massachusetts now receive a behavioral screening instrument at every pediatric visit. In North Carolina, the success of the Assuring Better Child Health and Development (ABCD) program led to adoption of a statewide Medicaid policy requiring developmental screening at designated well-child visits (Earls & Hay, 2006). Universal screening for specific disorders such as autism has also been demonstrated in several community-based pediatric clinics (Miller et al., 2011; Pierce et al., 2011).
Despite these successes, it is clear that significant barriers remain to appropriate referral and care, over and above increasing identification rates through implementation of screening instruments. Identification of risk by regular screening is only the first step in ensuring a good outcome. For example, a recent trial that randomly assigned children to either receive standard pediatric care or standard care plus developmental screening found that although screening led to more children being identified with and referred for developmental problems, 42 percent of children who screened positive were not referred for services, while some children who screened negative (4.7 percent) were referred (Guevara et al., 2013). In that pediatricians appear to depart from strict adherence to recommended thresholds for screening instruments, these findings are consistent with previous research on pediatric care. Many pediatricians are reluctant to follow evidence-based protocols (Lorenz et al., 2005), and many have been found to depart from expert screening recommendations for other health problems, including diabetes (Rhodes et al., 2006), obesity (Klein et al., 2010), and cardiac problems (Bensky, Covitz & DuRant, 1999). Moreover, developmental screening represents only one element of pediatric surveillance, which also includes direct examination of the child and discussion with parents (Dworkin, 1989). Pediatricians are encouraged to use the full range of information available to them when making treatment and referral decisions; thus, their decisions are not and should not be guided solely by the results of a single screening instrument.
Pediatricians' use of information beyond the results of screening tools can be conceptualized as altering the threshold of the screening instrument. Typically, screening instruments are designed to yield continuous scores. To score the instrument, thresholds (also known as “cut scores”) are then set to differentiate a positive range (often labeled “clinical range”) from a negative range (often labeled “normative range”). Data based on these thresholds are used to calculate the sensitivity and specificity of the instrument. When pediatricians apply their clinical judgment to a screening result, they are in effect revising the test's stated threshold and therefore the sensitivity (i.e., the proportion of children with disabilities who are accurately identified) and specificity (i.e., the proportion of children without disabilities who are accurately identified). A recent systematic review found that when pediatric providers identify behavioral disorders in general practice, specificity typically far exceeds sensitivity (Sheldrick, Merchant & Perrin, 2011), indicating that they are more confident and successful in identifying normal development than in identifying deviations from normal. This tendency may reflect a greater reluctance to make erroneous diagnoses (false positive errors) than to miss true diagnoses (false negative errors). It appears, therefore, that pediatricians would favor a much higher threshold than those that are typically recommended for screening instruments. These findings suggest a possible mismatch between screening recommendations and the realities of primary care.
There are many possible explanations for why pediatricians may adopt a relatively high clinical threshold that favors specificity over sensitivity. Many developmental-behavioral disorders are not acute but chronic problems, thus making a “wait and see” approach seem reasonable. Referrals for mental health care are often difficult and time-consuming, thus placing significant burdens on pediatric staff. Moreover, developmental-behavioral problems are widely seen as stigmatizing. Therefore, many pediatricians are reluctant to identify and thereby label children until they are certain of their diagnosis and of the possibility of appropriate care.
To better understand the implications of this phenomenon, we recently published a theoretical model suggesting that physicians set their clinical thresholds based on their perceptions of past clinical encounters (Sheldrick, Polk & Chan, 2013). The model rests on the observation that in the absence of perfect diagnostic accuracy, errors are unavoidable. Some children will be identified and/or referred who should not be (i.e., false positives), and other children with not be identified and/or referred who should have been (i.e., false negatives). Our model posits that physicians are influenced by two primary feedback loops. On the one hand, regret about false positive results causes physicians to increase the level of clinical certainty they require before making a referral. On the other hand, regret about false negative results causes physicians to decrease the level of clinical certainty required for referrals. These two feedback loops work in opposition to one another to form a dynamic balance.
Based on this theory, we developed a simple system dynamics (SD) model to make more specific predictions. For example, our model predicts that identification of developmental-behavioral problems will be enhanced by the implementation of screening instruments, but that this effect will be limited because pediatricians will not follow published scoring rules and will instead effectively raise their clinical threshold to favor specificity over sensitivity. This prediction is consistent with the systematic review of pediatricians' identification of developmental problems cited above (Sheldrick, Merchant & Perrin, 2011) and with the clinical trial described above, which found that a large proportion of children who scored positive on screening instruments were not referred (Guevara et al., 2013). Moreover, our model predicts that physicians' willingness to identify and refer children at risk for developmental disabilities may be influenced by a range of other interventions that have been proposed in the research literature, including systematic feedback regarding patients' clinical outcomes (Bickman, 2008) and co-locating providers of mental health care in pediatricians' offices (Perrin & Sheldrick, 2012; Kolko et al., 2012). For example, if co-locating quality mental health services increases the convenience of making referrals while reducing parents' reluctance to follow through with the referrals, pediatricians may feel less regret regarding referral decisions that ultimately prove to be unwarranted. In effect, this lowers pediatricians' clinical thresholds, thus leading to increased identification of children with developmental-behavioral disorders.
In summary, we believe that new screening tools like the SWYC have the potential to improve the identification of developmental and behavioral disorders in primary care settings. However, our optimism is tempered by the complex realities of primary care. Successful implementation is critical, and our initial work suggests that such efforts must go beyond encouraging strict adherence to screening protocols. In the absence of perfect diagnostic accuracy, false positive and/or false negative results are unavoidable. A well-designed health care system must therefore strike an appropriate balance between the two types of errors, minimizing the risks associated with each. In this light, we believe that implementation of developmental-behavioral screening is likely to be most successful if situated in the context of a medical home that facilitates communication with parents and other providers and provides direct (preferably on-site) access to needed services.
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the Academy of Child and Adolescent Psychiatry, 47 , 1114-1119.
Boyle, C.A., Boulet, S., Scheive, L.A., Cohen, R.A., Blumberg, S.J., Yeargin-Allsop, M., Visser, S., Kogan, M.D. (2011). Trends in the prevalence of developmental disabilities in US children, 1997-2008. Pediatrics, 127, 1034-1042.
Dworkin PH (1989). British and American recommendations for developmental monitoring: the role of surveillance. Pediatrics. 84 :1000–1010.
Earls MF, Hay SS. (2006). Setting the stage for success: Implementation of developmental and behavioral screening and surveillance in primary care practice. Pediatrics , 118 (1), e183–e188.
Guevara, J.P., Gerdes, M., Localio R., Huang Y.V., Pinto-Martin, J., Minkovitz, C.S., Hsu, D., Kyriakou, L., Baglivo, S., Kavanagh, J., Pati, S. (2013). Effectiveness of Developmental Screening in an Urban Setting. Pediatrics, 131, 30-37.
Kolko DJ, Campo JV, Kilbourne AM, Kelleher K. Doctor-office collaborative care for pediatric behavioral problems: a preliminary clinical trial [published online November 7, 2011]. Arch Pediatr Adolesc Med . 2012;166(3):224-231.
Lorenz, K.A., Ryan, G.W., Morton, S.C., Chan, K.S., Wang,S. & Shekelle, P.G. (2005). A qualitative examination of primary care providers' and physician managers' uses and views of research evidence. International Journal for Quality in Health Care, 17, 409-414.
Merikangas KR, He JP, Burstein M, et al. Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication–Adolescent Supplement. J Am Acad Child Adolesc Psychiatry. 2010;49(10):980 –989.
Miller JS, Gabrielsen T, Villalobos M, et al. (2011). Each Child Study: Systematic screening for autism spectrum disorders in a pediatric setting. Pediatrics, 127 (5), 866-871.
Perrin, E.C., Sheldrick, R.C. (2012). The challenge of mental health care in pediatrics. Archives of Pediatrics & Adolescent Medicine, 166, 287-288. DOI:10.1001/archpediatrics.2011.202; PMID 22064872.
Pierce K, Carter C, Weinfeld M, et al. (2011). Detecting, studying, and treating Autism early: the one-year well-baby checkup approach. Journal of Pediatrics , 159(3), 458-465.
Radecki, L., Sand-Loud, N., O'Connnor, K.G., Sharp, S. & Olson, L.M. (2011). Trends in the use of standardized tools for developmental screening in early childhood: 2002-2009. Pediatrics, 128 , 14-19.
Sheldrick, R.C. & Perrin, E.C. (in press). Evidence-based milestones for surveillance of cognitive, language and motor development. Academic Pediatrics . DOI: 10.1016/j.acap.2013.07.001
Sheldrick, R.C., Chan, K., Polk, D. (2013). Decision thresholds in developmental-behavioral screening: Gaining insight through system dynamics modeling. Conference Proceedings of The 31st International Conference of the System Dynamics Society, Cambridge, Massachusetts USA, July 21 – July 25, 2013. Available online at: http://www.systemdynamics.org/conferences/2013/proceed/papers/P1164.pdf (PDF, 664.41KB)
Sheldrick, R.C., Henson, B.S., Merchant, S., Murphy, M.J., Neger, E., Perrin, E.C. (2013). The Baby Pediatric Symptom Checklist (BPSC): Development and initial validation of a new social/emotional screening instrument for very young children. Academic Pediatrics, 13, 72-80 . DOI: 10.1016/j.acap.2012.08.003; PMID: 23092547.
Sheldrick, R.C., Henson, B.S., Merchant, S., Murphy, M.J., Neger, E., Perrin, E.C. (2012). The Preschool Pediatric Symptom Checklist (PPSC): Development and initial validation of a new social/emotional screening instrument. Academic Pediatrics, 12, 456-467. DOI: 10.1016/j.acap.2012.06.008.
Sheldrick, R.C., Perrin, E.C. (2009). Surveillance of Children's Behavior and Development: Practical Solutions for Primary Care. Journal of Developmental and Behavioral Pediatrics, 30, 151-3. DOI: 10.1097/DBP.0b013e31819f1bfb; PMID 19363367.
Smith, N., Sheldrick, R.C. & Perrin, E.C. (2012). An abbreviated screening instrument for autism spectrum disorders. Infant Mental Health Journal, 34, 149-155 . DOI: 10.1002/imhj.21356.
Squires, J., Bricker, D. & Potter, L. (2009). Ages & Stages Questionnaires, Third Edition (ASQ-3) User's Guide. Baltimore, MD: Paul H. Brookes Publishing.
Squires, J., Bricker, D., Twombly, E. (2002). The ASQ:SE User's Guide for the Ages & Stages Questionnaires: Social-Emotional: A parent-completed, child monitoring system for social-emotional behaviors. Baltimore, MD: Paul H. Brookes Publishing.
Glascoe, F.P. (1997). Parents' Evaluations of Developmental Status: A Method for Detecting and Addressing Developmental and Behavioral Problems in Children. Nashville, TN: Ellsworth & Vandermeer Press.
Trotter, F. & Uhlman, D. (2013). Hacking Healthcare: A Guide to Standards, Workflow, and Meaningful Use. O'Reilly Media: Sebastopol, CA.
Zima, B.T., Murphy, M., Scholle, S.H., Hoagwood, K.E., Sachdeva, R.C., Mangione-Smith, R., Woods, D., Kamin, H.S., Jellinek, M. (2013). National Quality Measures for child mental health care: Background, progress, and next steps. Pediatrics, 131 , S38. DOI: 10.1542/peds.2012-1427e
Ellen C. Perrin, MD, is a developmental-behavioral pediatrician and a professor of pediatrics and Director of Research in the Division of Developmental-Behavioral Pediatrics at the Floating Hospital for Children at Tufts Medical Center. She has participated in the work of the Academic Pediatric Association, the American Academy of Pediatrics, and the Society for Developmental and Behavioral Pediatrics, of which she is a past president. Her clinical work, teaching and research have focused on children with complex developmental disabilities, children whose parents are gay or lesbian, and children with chronic physical health conditions. Her recent research interest is in implementation of developmental-behavioral screening and interventions for developmental-behavioral disorders in primary care pediatric contexts. She and her colleagues have recently developed a comprehensive screening instrument for use in primary care settings to identify children at risk for developmental-behavioral disorders.
R. Christopher Sheldrick, PhD, is a research psychologist and an assistant professor in the Division of Developmental-Behavioral Pediatrics at the Floating Hospital for Children at Tufts Medical Center. His research focuses on screening and intervention for young children in pediatrics. Together with Ellen Perrin and colleagues, he has worked to create the Survey for Wellbeing for Young Children (SWYC), a freely available screening instrument for children under 5 years of age. His recent interests include the use of systems dynamics modeling to conceptualize and guide implementation and dissemination of evidence-based screening and treatment models.