OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES OCTOBER 12, 2021
TABLE OF CONTENTS
EXECUTIVE SUMMARY|910|
CONCLUSION|910|FINDINGS|910|RECOMMENDATIONS|910|CFD RESPONSE :|910|BACKGROUND|910|
DISPATCH PROCESS|910|FIRE AND EMS RESPONSE TIME STANDARDS|910|
National Fire Protection Association|910|Commission on Fire Accreditation International|910|Illinois Department of Public Health 9
C. . PRIOR OIG AUDIT OF CFD RESPONSE TIMES 9
FINDINGS AND RECOMMENDATIONS 10
FINDING 1: CFD HAS NOT IMPLEMENTED PERFORMANCE MANAGEMENT PRACTICES THAT
WOULD ALLOW THE DEPARTMENT TO EVALUATE ITS FIRE AND EMS RESPONSE
TIMES 10
Response Time Reporting 10
Average and Percentile Measurements 10
FINDING 2: CFD HAS NOT DOCUMENTED RESPONSE TIME PERFORMANCE GOALS OUTSIDE OF
ITS STATE-REQUIRED EMS PLAN 12
FINDING 3: CFD'S DATA IS NOT ADEQUATE TO RELIABLY MEASURE KEY COMPONENTS OF
RESPONSE TIME 16
OBJECTIVES, SCOPE, AND METHODOLOGY 18
OBJECTIVES 18
SCOPE 18
METHODOLOGY 18
STANDARDS 19
AUTHORITY AND ROLE 19
ACRONYMS
CAD Computer Aided Dispatch
CFAI Commission on Fire Accreditation International
CFD Chicago Fire Department
EMS Emergency Medical Services
GPS Global Positioning System
IDPH Illinois Department of Public Health
NFPA National Fire Protection Association
OEMC Office of Emergency Management and Communications
OIG Office of Inspector General
OPSA Office of Public Safety Administration
PAGE 1
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
PAGE 2
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
EXECUTIVE SUMMARY
The Office of Inspector General (OIG) conducted a second audit of the Chicago Fire Department's (CFD) fire and emergency medical response times. The first audit was published in 2013.1
The objectives of the audit were to determine if,
CFD has goals for fire and emergency medical services (EMS) response times consistent with state and national standards; and
CFD response times meet state and national standards.
CONCLUSION
OIG concluded that CFD has not implemented performance management strategies that would allow it to evaluate fire and EMS response times in alignment with best practices. Nor has the Department remedied data issues identified in 2013.
FINDINGS
OIG found that CFD has not implemented best practices for measuring response times. CFD does not produce annual department-wide reports that would allow it to evaluate response times, and it does not measure turnout and travel as separate components of response time or use industry-standard percentile measures.2 Notably, we recommended in 2013 that CFD correct these issues.
OIG further found that CFD has not documented response time performance goals outside of its state-required EMS plan. CFD documented its overall EMS response time goal as required by state law. However, the Department has not documented fire response time goals. Contrary to best practices, CFD has not set goals for turnout or travel time at the industry-standard 90th percentile. Here, again, we recommended in 2013 that CFD set and document such goals.
OIG also found that CFD's data is not adequate to allow reliable measurement of response time. We analyzed records for emergency events from January 1, 2018, to November 30, 2020, finding that only 705,061 of 937,446 (75.2%) included data for all categories necessary to calculate turnout and travel times for the first arriving unit. CFD acknowledges that it has been aware of data reliability issues since at least 2013, but has not remedied them.
|109|City of Chicago Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit," October 18, 2013, .|109|Turnout begins when first responders are notified of an emergency incident and ends when one or more units are ' en route to the scene. Travel begins when a unit is en route and ends when it arrives at the scene. National Fire Protection Association, "NFPA Standard 1710: Standard for the Organization and Deployment of Fire Suppression Operations, Emergency Medical Operations, and Special Operations to the Public by Career Fire Departments," 9, Quincy, Massachusetts: National Fire Protection Association, 2020.
PAGE 3
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
RECOMMENDATIONS
OIG recommends that CFD management acknowledge the importance of department-wide quantitative performance measures and begin public annual reporting on its response time performance. CFD management should establish and document department-wide turnout, travel, and total response time goals at the 90th percentile for both fire and EMS. If CFD management believe NFPA recommended turnout and travel times are unachievable in Chicago, they should conduct a systematic evaluation of local factors affecting response times and set reasonable goals for turnout, travel, and total response times accordingly. CFD should also identify, monitor, and remedy the cause of gaps in its data, and should consider hiring an internal data specialist to improve data quality. Finally, CFD should ensure that any external partners it engages to analyze departmental data conduct a full assessment of that data's completeness and reliability.
CFD RESPONSE
In response to our audit findings and recommendations, CFD stated that it "acknowledges the importance of department-wide quantitative performance measures" and will implement OIG's recommendations. CFD stated that it has "engaged Urban Labs at the University of Chicago, in part, to aid the department in analyzing its response time performance" and would work with OBM and DHR to hire additional data analytics staff. CFD stated that it would analyze its data to identify "causative factors and or trends" and "perform a complete and reliable measure of response time by each component piece and in total, reported as a percentile measure." CFD stated it would determine a reasonable percentile goal "as the completeness of data elements improves." Finally, CFD agreed to work with OEMC to improve and monitor data in the existing and new CAD systems.
The specific recommendations related to each finding, and CFD's response, are described in the "Findings and Recommendations" section of this report.
PAGE 4
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
II. BACKGROUND
The Chicago Fire Department (CFD) provides emergency fire suppression and medical services.3 From 2018 through 2020, CFD responded to more than 300,000 emergency events each year, the vast majority of which were calls for emergency medical services (EMS). More detailed data on the types and locations of CFD's emergency calls is available on OIG's Information Portal.
FIGURE 1: Over 80% of CFD's emergency calls from 2018 through 2020 were for EMS events
322,307 emergency calls
(19%) 1
318,920 emergency calls
iiii
321,151 emergency calls
Q Fire D EMS
2018
Source: OIG visualization of 911 data.
A. DISPATCH PROCESS
As shown in Figure 2, there are generally three phases of an emergency response: 911 call processing, turnout, and travel. First, 911 receives an emergency call, and the Office of Emergency Management and Communications (OEMC) dispatches first responders using a Computer Aided Dispatch (CAD) system. Second, CFD first responders press a button in the firehouse to acknowledge the call, marking the beginning ofthe "turnout time" phase (as well as the CFD total response time). Third, the first responders press another button inside their vehicle to show they are en route to the scene, beginning the "travel time" phase. When they arrive, they press the same button a second time, marking the end of the CFD total response time. The CAD system records timestamps for each of these milestones. If a CFD member does not press a button or there is a technical error, OEMC dispatchers may manually update the CAD record.
3 CFD is primarily responsible for emergency response within Chicago's city limits, an area of approximately 228 square miles and 2.7 million residents. The Department also responds outside the city limits if requested by the Mutual Aid Box Alarm System, a resource-sharing agreement with nearby jurisdictions.
PAGE 5
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
FIGURE 2: Emergency event responses have three phases—CFD performs the last two
Jjf —> g$ —> 0$
OEMC receives 911 CFD acknowledges call First CFD unit leaves First CFD unit arrives
call and dispatches ("Acknowledge") ("En Route") ("On Scene")
first responders
911 Call Processing Time Turnout Time Travel Time
x CFD Total Response Time
Source: NFPA Standard 1710 Figure A.3.3.64.6, adapted to Chicago.
Chicago implemented its CAD system in 1995. CFD and OEMC stated that it was designed as a resource-tracking system to determine whether vehicles were available—not to measure response times. In January 2020, OEMC announced that the City entered into an agreement to replace the current CAD system.4 The new system is scheduled to go live in November 2022.
B. FIRE AND EMS RESPONSE TIME STANDARDS
1. National Fire Protection Association
The National Fire Protection Association (NFPA) publishes a set of management and operations best practices for fire departments. NFPA Standard 1710 is widely accepted as a national benchmark for fire and emergency response times.5 Figure 3 summarizes NFPA's recommendations for turnout and travel time. The total fire response time goal for turnout and travel combined is 5 minutes and 20 seconds, and the total EMS response time goal is 5 minutes.
FIGURE 3: NFPA's recommendations divide response time goals into two components
ffurhdut^'-- 1*80'seednds:biMess
Travel 240;secondsor less
JftrhotJtjv .60 seconds or iess • ^ Travel "'; 240 seconds or less
Source: NFPA 1710 2020 Edition, Section 4.1.2.1.
|109|City of Chicago, "Chicago OEMC announces new State-Of-The-Art Dispatch System and Software for City's 9-1-1 Center to improve emergency responses," January 14, 2020, accessed May 3, 2021, en/depts/oem/provdrs/emerg mang/news/2020/januarv/chicago-oemc-announces-new-state-of-the-art-dispatch-svstem-and-.html,|109|National Fire Protection Association, "NFPA Standard 1710: Standard for the Organization and Deployment of Fire Suppression Operations, Emergency Medical Operations, and Special Operations to the Public by Career Fire Departments," 9, Quincy, Massachusetts: National Fire Protection Association, 2020.
PAGE 6
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
NFPA Standard 1710 states that fire departments may choose to modify these goals for their own jurisdictions. NFPA Standard 1710 also emphasizes that, "Documenting the benchmarks and objectives that make up NFPA 1710 is crucial to capturing and tracking data that would be helpful in ensuring the necessary allocation of resources."6 NFPA provides examples of how to conduct a community-wide risk assessment to inform modifications to these goals.
AVERAGE VERSUS PERCENTILE MEASUREMENTS
NFPA recommends using percentiles to measure response times, rather than averages. An average represents a typical value in a set of data. Simple averages are inadequate for measuring response times because very short or very long response times can skew results. This may give community members an inappropriate expectation of when help will arrive. Percentiles, in contrast, show exactly how often a department meets a response time goal. NFPA recommends setting response time goals and assessing performance at the 90th percentile. This means, for example, that a fire department following NFPA's recommendations would strive to ensure that at least 90% of EMS responses achieve a turnout time of 60 seconds or less and a travel time of 240 seconds or less.
Using a hypothetical dataset of fire response events, Figure 4 illustrates the difference between an average and a 90th percentile measure of response time. In this example, 90% of responses were at or below the NFPA travel time goal of 4 minutes (240 seconds). The average travel time is 2 minutes 35 seconds (155 seconds). However, many of these hypothetical responses took longer than 2 minutes 35 seconds and a few outliers took more than 9 minutes. The 90th percentile provides a clearer picture of the length of time within which the vast majority of responses occur.
5 NFPA 1710 uses the terms "goals," "benchmarks," and "objectives" in describing how departments determine the response times they intend to meet. OIG's audit report uses "goals" to describe these intended times.
PAGE 7
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
FIGURE 4: The 90th percentile represents an appropriate expectation for performance
ID
Kj 2:35 Average travel time
|TJ^ But 90% arrived within 4:00
||.
.¦'I I 1«
¦ 4
|1010|Travel Time (in minutes) Source: OIG illustration of hypothetical response times.
2. Commission on Fire Accreditation International
The Commission on Fire Accreditation International (CFAI) publishes a set of management best practices for fire departments.7 CFAI recommends "a process of agency self-assessment" to ensure departments are "community-focused, data-driven, outcome-focused, strategic-minded, well organized, properly equipped, and properly staffed and trained."8 Per CFAI, this process supports performance-based budgeting through measurement and analysis. The International City/County Management Association, International Association of Fire Chiefs, Insurance Services Organization, and International Association of Fire Fighters have contributed to the CFAI model.9
Like NFPA, CFAI recommends basing response time goals on a documented community risk assessment. It holds that departments should measure response times in component pieces (turnout and travel time) at the 90th percentile, as NFPA also recommends. CFAI's voluntary accreditation process requires departments to create a strategic plan that uses response time performance indicators.
|109|Center for Public Safety Excellence, Quality Improvement for the Fire and Emergency Services, 11, Chantilly, Virginia: Center for Public Safety Excellence, 2020.|109|Center for Public Safety Excellence, "Accreditation Overview," accessed April 28, 2021, accreditation/.|10 9|International Association of Fire Fighters is the union that represents most CFD firefighters and paramedics.
PAGE 8
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
3. Illinois Department of Public Health
The EMS Systems Act sets minimum standards for Illinois hospitals and EMS providers.10 CFD is required to meet strict reporting requirements and submit written plans to the Illinois Department of Public Health (IDPH) which include "a commitment to optimum [EMS] response times up to six minutes in primary coverage areas."11 CFD's internal Medical Administration and Regulatory Compliance division monitors CFD's compliance with the Act. IDPH is authorized to take regulatory action if CFD does not meet those requirements, such as imposing fines or creating a corrective action plan.
C. PRIOR OIG AUDIT OF CFD RESPONSE TIMES
In 2013, OIG published an audit of CFD's fire and EMS response times.12 The audit determined that CFD was not meeting the goals the Department had claimed to meet or exceed, and that its internal reports lacked the elements necessary to assess whether CFD was meeting its standards. A follow-up report published in 2015 found that the Department had not taken corrective actions to address OIG's findings.13
In response to the 2013 audit, CFD stated that the data in its CAD system was not reliable enough to measure response times. CFD argued that NFPA response time standards were guidelines rather than stringent rules and declined to document a fire response goal unless legally required to do so. CFD also stated that setting response time goals would pose public safety hazards because it could encourage reckless driving by first responders. CFD defended its use of a weekly average measurement as reliable and appropriate for fire response times. CFD also expressed opposition to performing any response time analysis by ward or community area. CFD repeated these objections in its response to OIG's 2015 follow-up.
As discussed in Finding 3 of this report, CFD continues to assert that the CAD system is not reliable enough to measure response times. CFD has stated that studies on this subject conducted by CBS Chicago were based on inaccurate data.14
10See 210 ILCS 50/1, etseq.
11 Illinois Administrative Code title 77, § 515.810 (2018).
12 City of Chicago Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times
Audit," October 18, 2013, .
13 City of Chicago Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times
Follow-Up Inquiry," March 3, 2015,
incident-response-times/.
14 CBS Chicago published a series regarding CFD's response times beginning in March 2019 and found that 19% of
calls took longer than 7 minutes for an ambulance or fire truck to arrive. CFD disagreed and said that it was working
on its own analysis, but did not provide such analysis to CBS. See
fire-department-ambulance-response-times/and
ambulances-but-if-you-need-one-will-it-be-on-time/.
PAGE 9
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
CFD has not adopted best practices to measure its response time performance. CFD does not conduct department-wide evaluations, use percentile measurements, or evaluate response time components. As discussed in Findings 2 and 3, CFD has not formally documented its response time goals or remedied issues with its data reliability. OIG first recommended each of these changes in 2013.
1. Response Time Reporting
CFD does not produce annual department-wide response time reports. The National Fire Protection Association (NFPA) and the Commission on Fire Accreditation International (CFAI) recommend annual evaluations to assess performance and opportunities for improvement.
CFD's chief administrative officer formerly received a bi-weekly Computer Aided Dispatch (CAD) report of overall average fire and EMS response times from OEMC. However, OEMC stopped sending the report in July 2020 after the chief administrative officer left CFD.
CFD managers have independently designed ad hoc response time reports for internal use. These reports use inconsistent metrics and different response time standards to assess performance. For example, CFD's EMS chief receives a daily report ofthe top 10 longest ambulance response times for the previous day, a weekly report of ambulance runs that took longer than 8 minutes, and a comprehensive list of the previous month's ambulance runs. In contrast, CFD's Medical Administration and Regulatory Compliance division creates "scorecard" reports that compare CFD's average EMS response times against the Illinois Department of Public Health's (IDPH) 6-minute standard, and advanced life support ambulance-only arrival time against a 90% percentile 9-minute NFPA benchmark.
CFD has not shared response time reports with the public. Since OIG's 2013 audit, CFD management has not demonstrated a commitment to transparency with regard to measures of department performance. Without transparent reporting, CFD cannot be fully accountable to the public for its response time performance.
2. Average and Percentile Measurements
CFD stated that it uses averages to measure its EMS and fire response times. Both NFPA and CFAI recommend using a percentile method because it more accurately reflects the amount of time within which most responses occurred.
PAGE 10
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
OIG first recommended that CFD use a percentile approach in 2013 and again in 2015. In response to OIG's 2013 audit, CFD argued, "due to the large number of each week's sample size, an average of response times is an appropriate measure of total performance." CFD "strongly believefd] that its measurement of average response times is a reliable and appropriate measure in line with methods employed by other major cities."15
CFD is correct that a large dataset will make an average more consistent. However, regardless of the number of fire and EMS runs being analyzed, the percentile approach better reflects expected performance while allowing for unusual cases.
RECOMMENDATION
1. CFD management should acknowledge the importance of department-wide quantitative performance measures and begin public annual reporting on response time performance. The reports should include the component pieces of turnout and travel time and should use percentile measurements. Further, consistent with NFPA 1710, the reports should provide geographic analysis that identifies areas of the city where CFD is not meeting its response time goals.
MANAGEMENT RESPONSE
1. "CFD management acknowledges the importance of department-wide quantitative performance measures. CFD currently does not have the staff available to conduct such a review or to issue such a report. However, CFD has been taking steps to resolve this. CFD has engaged Urban Labs at the University of Chicago, in part, to aid the department in analyzing its response time performance. In addition, the Office ofthe First Deputy Fire Commissioner is looking to hire additional staff to assist the department on data analytics.
It is the Fire Department's expectation that when it hires the additional staff and can expand the role of Urban Labs, it will be able to review and implement department-wide quantitative performance measures consistent with the recommendations made herein."
Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit."
PAGE 11
NFPA and CFAI best practices stress the importance of documenting and measuring response time goals. However, CFD has not documented its goals outside of its state-required EMS System Plan.
In an email, CFD told OIG it has a goal to "meet and if possible, exceed the [EMS response time] requirements" ofthe EMS Systems Act. The EMS Systems Act requires "a commitment to optimum response times up to 6 minutes in primary coverage areas, 6 to 15 minutes in secondary coverage areas, and 15 to 20 minutes in outlying coverage areas."16 CFD has documented this goal in its IDPH EMS System Plan, as required by the state.
CFD told OIG that the Department also "has a goal... to meet or exceed [the fire response time] requirements" of NFPA Standard 1710.17 CFD quoted the NFPA requirements for turnout and travel time, and stated, "these performance objectives should be met 90% ofthe time." However, CFD's fire response time goal is not officially documented and CFD does not measure its response time in percentiles. NFPA Standard 1710 states, "the fire department organizational statement shall provide service delivery objectives, including specific time objectives for each major service component... and objectives for the percentage of responses that meet the time objectives."18
NFPA and CFAI stress the importance of documenting and measuring total response time, turnout time, and travel time.19 As noted in the background of this report and shown in Figure 5, CFD's portion of response time has two components, beginning when it acknowledges an emergency dispatch from OEMC and ending when the first unit arrives on scene.
16 Illinois Administrative Code title 77, § 515.810 (2018).
17 Furthermore, in Mayor Emanuel's 2019 Mayoral Transition Report to the Lightfoot administration, CFD stated its
"culture and goal" was that "every fire has a unit on scene within four minutes."
18 NFPA Standard 1710 Section 4.1.2.
19 NFPA defines turnout time as "the time interval that begins ... by either an audible alarm or visual annunciation
or both and ends at the beginning point of travel time," and travel time as "the time interval that begins when a unit
is en route to the emergency incident and ends when the unit arrives at the scene." NFPA Standard 1710 Section
3.3.64.8.
PAGE 12
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
FIGURE 5: CFD's total response time includes turnout time and travel time components
Total CFD Response Time
—» •* —> mS
CFD acknowledges dispatch First CFD unit leaves First CFD unit arrives ("On
from OEMC ("En Route") Scene")
("Acknowledge")
• i i i
Turnout Time Travel Time
Source: NFPA Standard 1710 Figure A.3.3.64.6, adapted for Chicago.
Measuring these component pieces would allow CFD management to track staff performance and plan future improvements. Other fire departments have used such data to make process improvements. For example, the Miami-Dade County Fire Rescue Department told OIG that, because it analyzes component pieces, it was able to improve its alerting system. This improvement reduced the response time for structure fires.
CFD asserts that NFPA's guidelines are unachievable goals that are not suited to Chicago's unique needs. In response to OIG's 2013 audit, CFD wrote that NFPA is an "arbitrarily set time goal," "there are no official national standards for fire department response times," and "NFPA ... recognizes that its response standards must include flexibility and allow for differences in each fire department."20 CFD stated that when responding to EMS events, firefighters must still dress in protective firefighting gear because they could always be redirected to a fire event while out in the field. Further, CFD stated that establishing official performance metrics could cause unwanted competition between firehouses or encourage reckless driving by first responders.
OIG spoke to industry consultants and peer fire departments who agree that NFPA standards are not always achievable. However, as discussed in the background of this report, CFAI and NFPA allow for flexibility in response time standards. Fire departments or those with statutory authority may adjust response time goals and percentiles to match their risk tolerances and resources.21
20 Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit," 10-12.
21 NFPA defers to what it calls the Authority Having Jurisdiction, defined as the "organization, office, or individual
responsible for enforcing the requirements of a code or standard, or for approving equipment, materials, an
installation, or a procedure." This may be the fire department, public health officials, or "others having statutory
authority." NFPA Standard 1710 3.2.2 and A.3.2.2.
PAGE 13
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES OCTOBER 12, 2021
NFPA and CFAI recommend conducting risk assessments to identify hazards and potential impacts on a community. These assessments also help departments identify performance gaps and develop process improvements to close those gaps. A reasonable response time goal takes into account the resources available to departments and community expectations.
RECOMMENDATIONS
2. CFD management should establish and document department-wide turnout, travel, and total response time goals for both fire and EMS at the 90th percentile. If CFD management believe NFPA recommended turnout and travel times are unachievable in Chicago, they should conduct a systematic evaluation of factors affecting response times and set reasonable goals for turnout, travel, and total response times accordingly.
MANAGEMENT RESPONSE
2. "Unlike fire responses, EMS is strictly regulated by the EMS Systems Act, which is enforced by the Illinois Department of Public Health (IDPH). As referenced in the Audit, EMS meets "strict reporting requirements and submit(s) written plans" to the IDPH. EMS is bound by the requirements set forth by the EMS Systems Act and IDPH, not by the non-binding recommendations made by the NFPA which focuses on fire protection rather than EMS.
Further, due to the data gaps discussed in the Audit, the OIG expressed concerns about the reliability ofthe results should it have conducted its own percentile analysis. At the same time, OIG recommends that CFD adopt response time goals at the 90th percentile. CFD acknowledges that it can set higher standards than what is required and approved by IDPH, and CFD continually strives to improve its responses times across the board, but it would be imprudent to set a specific percentile goal without first having the ability to fully and accurately analyze the underlying data.
Until such time that the data gaps can be fully addressed, CFD management will utilize an after-action reporting and improvement planning format to achieve those ends.
First, as observed by the audit, the data elements necessary to definitively measure the department's response time performance are incomplete. The incomplete data elements will be analyzed by common elements for the identification of causative factors and or trends.
In the absence of fully automated time stamps for each component piece of response time and until such data capture can occur, CFD management will utilize the continuous quality process to resolve for those factors that prohibit a complete collection of response time data points.
Second, CFD will look to perform a complete and reliable measure of response time by each component piece and in total, reported as a percentile measure.
PAGE 14
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES OCTOBER 12, 2021
As the completeness of data elements improves, the reliability of response time measures will increase and CFD management can begin to formulate reasonable percentile response time goals."
PAGE 15
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
As discussed in Finding 2, NFPA and CFAI recommend that departments analyze turnout and travel times as separate components of response times. CFD's CAD system data is incomplete to the point that the Department cannot accurately measure these key components. As discussed in Findings 1 and 2, CFD's inability to accurately measure those components significantly limits its performance management capacity.
OIG determined that between January 1, 2018 and November 30, 2020, only 705,061 of 937,446 emergency events (75.2%) contained all necessary timestamps in the proper chronological order to measure turnout and travel times. We consulted with a statistics expert from the Government Accountability Office about whether a percentile analysis is possible with this amount of missing data. They stated that department management should assess whether there is a pattern to the missing data. If there is a pattern (for example, if all events missing timestamps occurred at night or came from the same firehouse), management should remedy the problem. If management do not know if there is a pattern, they cannot analyze the whole population reliably. CFD has not conducted such an assessment.
CFD stated that it does not have the technical capacity to analyze its response times or the underlying data. The Department signed a contract with the University of Chicago Urban Labs to conduct a series of analytical tasks. However, CFD has not tasked Urban Labs with analyzing CAD data reliability or conducting a formal or comprehensive response times analysis.
CFD acknowledged that the CAD data was not complete in 2013, when OIG released its first audit ofthe Department's response times.22 In response to that audit, CFD stated that "the data is subject to human error" and "depends on drivers pressing an 'on scene' button."23 In response to the current audit, CFD stated that these issues persist, in part because first responders can fail to press the appropriate button due to a rush to respond to an event or due to a lack of training. CFD also stated that there are geographic areas of the city with poor radio transmission. As noted in the background of this report, CFD is in the process of upgrading its CAD system. The new system is scheduled to go live in November 2022.
OIG spoke to OEMC about the CAD system's capabilities. OEMC confirmed that the data can show unusually long response times if a first responder failed to press the appropriate button prior to leaving their vehicle. OEMC agreed that while CAD is designed as a resource management tool, the system is capable of recording response times. OEMC stated that there can occasionally be issues with radio transmission due to the geography ofthe city, but this happens infrequently. The Office of Public Safety Administration (OPSA) provided evidence that in 2014, a CFD deputy commissioner discovered a systemic Global Positioning System (GPS) issue
Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit," 13-14. Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit," 12.
PAGE 16
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
preventing CAD from locating 11-30% of CFD vehicles each day. CFD worked with OEMC to correct the problem. OPSA estimated that currently GPS problems arise with only approximately 2% of vehicles each day.
RECOMMENDATIONS
CFD should work with OEMC to assess the root causes of data gaps and address these issues moving forward so that these gaps are not recreated in the new CAD system. Specifically, CFD should pursue system capabilities that will allow it to analyze turnout and travel time, thereby giving the Department a better understanding of potential issues in its response process.
While waiting for the new CAD system to become operational, CFD should consider collaborating with OEMC and OPSA to leverage existing GPS technology in CFD vehicles to address blank and inaccurate time fields.
CFD management should work with OEMC to continuously monitor the number of blank and inaccurate time fields in its existing and new CAD systems, and work toward achieving completeness and accuracy in all data fields.
CFD should ensure that its data analysis partners conduct a full assessment of its data completeness and reliability, including an assessment of any trends in missing data. The Department should then use the results of this assessment to address any operational errors that led to missing data, thereby allowing the Department or its partners to perform comprehensive response times analyses. CFD should also consider working with the Office of Budget and Management to create a position for an internal data analyst, who could combine their operational expertise with technical skills to improve data quality.
MANAGEMENT RESPONSE
"CFD accepts OIG's recommendation and will endeavor to work with OEMC to accomplish the goals stated herein."
"As discussed above, CFD agrees that a full assessment of the response time data must be done to ensure completeness and reliability. While CFD has engaged with Urban Labs for other analyses, it will explore whether it can also include such an assessment in its work.
CFD is also working with OBM and DHR to create and fill a position that would be responsible for data analytics like that recommended in this audit."
PAGE 17
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
IV. OBJECTIVES, SCOPE, AND METHODOLOGY
OBJECTIVES
The objectives of the audit were to determine if,
CFD has goals for fire and emergency medical services (EMS) response times consistent with state and national standards; and
CFD response times meet state and national standards.
SCOPE
OIG reviewed records of fire and EMS events between January 1, 2018, and November 30, 2020.
OIG did not review OEMC call processing and transfer times. Nor did we review CFD fire or EMS staffing, analyze the distribution of ambulances and fire trucks across the city, or compare demand to vehicle availability.
METHODOLOGY
For all objectives, OIG interviewed CFD management, OEMC staff and management, and OPSA management.
To determine whether CFD's fire and EMS response time goals met state and national standards, OIG compared documentation provided by CFD to NFPA standards and IDPHrequirements.
To determine whether CFD has documented response time performance goals, OIG interviewed CFD management in its Legal, Administrative, Communications, Medical Administration and Regulatory Compliance, and EMS divisions. To understand CFD's responsibilities within the Illinois EMS Systems Act, we spoke with representatives from IDPH and EMS Region 11 management.
To understand the technical aspects of CFD's response time data, OIG interviewed CFD management, OEMC data analysts and management, and OPSA management. To determine whether CAD system data was complete and reliable enough to measure key components of response time, we joined individual event data with the CFD units that responded to each event. Then, we filtered out non-dispatched events, non-emergency events, and cancelled events. We counted the number of fire and EMS events where timestamps for "dispatch," "acknowledge," "en route," and "on scene" were present in the correct chronological order for the first arriving unit. We also interviewed a subject matter expert from the Government Accountability Office to verify the proper method to handle missing data, and listened to four OEMC radio dispatch recordings to determine whether they might be used to supplement the dispatch data.
PAGE 18
OIG FILE #20-0567
SECOND OIG AUDIT OF CFD'S FIRE AND EMERGENCY MEDICAL RESPONSE TIMES
STANDARDS
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
AUTHORITY AND ROLE
The authority to perform this audit is established in the City of Chicago Municipal Code § 2-56-030 which states that OIG has the power and duty to review the programs of City government in order to identify any inefficiencies, waste, and potential for misconduct, and to promote economy, efficiency, effectiveness, and integrity in the administration of City programs and
operations.
The role of OIG is to review City operations and make recommendations for improvement.
City management are responsible for establishing and maintaining processes to ensure that City programs operate economically, efficiently, effectively, and with integrity.
PAGE 19
The City of Chicago Office of Inspector General (OIG) is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of City government. OIG achieves this mission through,
administrative and criminal investigations by its Investigations Section;
performance audits of City programs and operations by its Audit and Program Review Section;
inspections, evaluations and reviews of City police and police accountability programs, operations, and policies by its Public Safety Section; and
compliance audit and monitoring of City hiring and human resources activities by its Compliance Section.
From these activities, OIG issues reports of findings and disciplinary and other recommendations to assure that City officials, employees, and vendors are held accountable for violations of laws and policies; to improve the efficiency, cost-effectiveness government operations and further to prevent, detect, identify, expose and eliminate waste, inefficiency, misconduct, fraud, corruption, and abuse of public authority and resources.
OIG's authority to produce reports of its findings and recommendations is established in the City of Chicago Municipal Code §§ 2-56-030(d), -035(c), -110, -230, and -240.
PROJECT TEAM
Matthew Jacobson, Performance Analyst Claire Kruschke, Senior Performance Analyst Ben Spies, Chief Performance Analyst Lise Valentine, Deputy Inspector General Andrew Clark, Forensic Data Analyst
PUBLIC INQUIRIES
Communications: (773) 478-8417 | communications@igchicago.orR
TO SUGGEST WAYS TO IMPROVE CITY GOVERNMENT
Visit: igchicago.org/contact-us/help-improve-city-government
TO REPORT FRAUD, WASTE, AND ABUSE IN CITY PROGRAMS
Call OIG's toll-free hotline: (866) 448-4754 / TTY: (773) 478-2066 Or visit: igchicago.org/contact-us/report-fraud-waste-abuse/
Cover image courtesy ofCanva. Alternate formats available upon request.