This June 15 webinar introduced the Pediatric Integrated Care Survey (PICS), a validated instrument that assesses family experience of care integration. The family perception of care integration is essential in identifying opportunities to improve care coordination and care management. The PICS was developed by Richard Antonelli, MD, MS, Medical Director of Integrated Care at Boston Children's Hospital, and his team. The survey asks family respondents to identify the members of their child's/ youth’s care team and report on their experiences with integration across disciplines, institutions, and communities. The webinar provided an overview of the tool and highlighted how one hospital is using it to make improvements.
- Hannah Rosenberg, MSc – Project Manager, Integrated Care Program, Boston Children's Hospital
- Rebecca Baum, MD – Division Chief of Developmental and Behavioral Pediatrics, Nationwide Children's Hospital
Where or how can we access the actual survey questions?
What statistical analyses did you conduct to identify the 19 validated questions of PICS?
Hannah: Please refer to Pediatrics Paper.
When you say validated, do you mean by the study group? Or were these measures tested and validated in a larger scale?
Hannah: They were tested and validated through statistical analyses. The Pediatrics paper covers validation in detail.
Was validation only for administration once a year or more often than that?
Hannah: The questions that were used in the validation used a reference time frame of 12 months, lending itself to a yearly administration. However, we are currently using the PICS in a quality improvement project with questionnaire administration more than once a year to evaluate if we are able to measure change. The quality improvement project involves the use of an action grid as part of shared care planning and is being implemented using rapid cycle quality improvement.
I was wondering about the reliability scores from the paper. Nunnally said .70 was an acceptable alpha for early in the research stage but that .70 was not as development went on, .90 is his recommended target. What is being done to increase the reliability?
Hannah: Current efforts are focused on validating the Spanish version of PICS. Further refinements of the questions that might increase test-retest reliability is a potential for future work.
If you're experienced with the Family Experience of Coordination of Care instrument, could you compare/contrast with PICS? Especially in their application to large populations of CYSHCN, such as CCS.
Hannah: PICS is a measure of care integration. It measures overall experience as opposed to one visit. It is not linked to a PCP or provider, but is a measure of integration across providers.
I have developed a REDCap version of the PICS for use in a small QI project at my institution, Gillette Children’s. Is it OK to share this version?
Hannah: If you have developed your own adapted PICS, it is completely fine to share. We recognize that REDCap is proprietary, so the decision would be within your own institution. Any software package can be used to collect PICS responses.
How many surveys were initially sent out and are you following up to encourage completion? Have you had to add staff to accomplish this, and if so, how are those positions supported?
Rebecca: We sent out approximately 350 surveys (one-quarter of our cohort). We started with a “low tech” approach by mailing letters. We had hoped to use a call center that was collecting data for another project, but this proved to be too costly. Around one-quarter of our cohort is involved in high risk case management through our Accountable Care Organization, so their care coordinator can help encourage survey completion. If our response rate is low, we may try using an electronic link, which would allow us to send reminders e-mails, too. Unfortunately, only a small percentage of our cohort was enrolled in our EHR’s secure messaging portal, so this was not an option. We are also working with a health literacy expert because of concerns that we could be missing responses from caregivers with low literacy given our current methodology.
We are using existing staff to analyze the results. The Navigate My Care (NMC) program has a Quality Improvement Service Line Coordinator who helps with the program, and she has been tasked with the survey logistics and scoring. Her FTE is about 0.2 on the project.
What was the total cost of sending out the survey, (from start to finish)? I'm wondering what budget would look like for staff time, material, etc.
Rebecca: Our hospital’s marketing department helped with mailing and absorbed the cost into its usual operating expenses. We are using existing staff to analyze the results. The Navigate My Care (NMC) program has a Quality Improvement Service Line Coordinator who helps with the program, and she has been tasked with the survey logistics and scoring. Her FTE is about 0.2 on the project.
Did you send out the surveys in paper form, or did you send electronically?
Hannah: At Boston Children’s Hospital, we have used both paper and electronic surveys. This is the same for our implementation partners. We use REDCap, but are aware that other groups have used survey monkey.
Has the instrument been, or will will it be, piloted in Title V programs? Especially since many programs are divesting of direct care services and moving towards systems-building activities.
Hannah: Yes, very applicable to work of Title V programs. In Wisconsin, Title V has started work with PICS and in several other states, Title V Programs are looking into using PICS measures.
Where is this tool being utilized besides Nationwide Children's Hospital?
Hannah: PICS is being utilized in several different types of organizations across the country and nationally. At the National Center for Care Coordination Technical Assistance, we encourage others to share their work but do not promote work of others without permission.