WILLIAM MARBACK INTERIM INSPECTOR GENERAL
CITY OF CHICAGO OFFICE OF INSPECTOR GENERAL 740 NORTH SEDGWICK STREET, SUITE 200 CHICAGO, ILLINOIS 60654 TELEPHONE: (773) 478-7799 FAX: (773) 478-3949
VIA ELECTRONIC MAIL
Nick Lucius Chief Data Officer Office ofthe Mayor 121 N. LaSalle St. Chicago, IL 60602
Dear Chief Data Officer Lucius:
The City of Chicago's operations increasingly rely on collecting and utilizing high-quality data. Through our audit and investigation work, the City of Chicago Office of Inspector General (OIG) has observed many issues impacting data objectivity, utility, and integrity. The inconsistent quality ofthe City's data hinders it from effectively allocating resources, measuring performance, and achieving objectives. To support the chief data officer's (CDO) role in improving decisionmaking and management through data analysis, we summarize our observations below.
I. QUALITY DATA IS ESSENTIAL FOR EFFICIENT AND EFFECTIVE GOVERNMENT
Local governments fulfill a wide variety of missions by managing people, capital assets, and money.1 These tasks generate large amounts of data, which the CDO helps City departments understand and use. In recent years, governments have ramped up their use of data to improve resource allocation, measure success, and increase efficiency.2 The Organisation of Economic Cooperation and Development found that governments that actively use their data improve their ability to develop better long-term plans by anticipating constituent needs and trends affecting operations.3 Governments that actively use data can evaluate the success of public service
|109|Mahesh Kelkar, Peter Viechnicki, Sean Conlin, "Mission Analytics," Deloitte Center for Government Insights, 2016, accessed December 6, 2021, decision-making-in-government.html.|109|Kil Huh, "Using Data To Improve Policy Decisions: Insights To Help Governments Address Complex Problems: Insights To Help Governments Address Complex Problems," The Pew Charitable Trusts, August 14, 2018, accessed December 6, 2021, policv-decisions.|109|The Organisation for Economic Co-operation and Development, "The Path to Becoming a Data-Driven Public Sector," November 28, 2019, accessed December 6, 2021, driven-public-sector-059814a7-en.htm.
IGCHICAG0.ORG | OIG TIPLINE: (866) 448-4754 | TTY: (773) 478-2066
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
delivery and engage in continuous improvement. Accordingly, data is a key strategic governmental asset. Yet, data can only serve its purposes if it is accurate and reliable.4
According to the Federal Office of Management and Budget, data quality is an overarching concept incorporating the objectivity, utility, and integrity of information.5 The Federal Committee on Statistical Methodology (FCSM) built upon this concept, developing a more detailed framework for ensuring quality data.6 Although FCSM defines objectivity, utility, and integrity as distinct, in practice they are related. A data quality problem in one domain can negatively influence another. Figure 1 shows the specific dimensions of FCSM's three domains of data quality.
FIGURE 1: Data quality includes the objectivity, utility, and integrity of information7
integrity
Scientific Integrity
Credibility
Computer and Physical Security
•Confidentiality
Source: Federal Committee on Statistical Methodology.
II. DATA QUALITY ISSUES HINDER THE CITY'S OPERATIONAL EFFECTIVENESS
Figure 2 lists the data quality issues and operational impacts OIG has recently observed in its work within City departments and programs. The figure also identifies the relevant dimensions of data quality from FCSM's framework. The audits and reviews cited below did not necessarily
|109|Huh, "Using Data To Improve Policy Decisions."|109|Office of Management and Budget, "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies," Federal Register Vol. 67, No. 36 (February 5, 2002), 8453, accessed December 6, 2021, .|109|U.S. Federal Committee on Statistical Methodology, "A Framework for Data Quality," September 2020, 2, accessed December 6, 2021, A Framework for Data Quality.pdf.|109|U.S. Federal Committee on Statistical Methodology, p. 6.
PAGE 2
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
focus on data quality issues, but all illustrate OIG's encounters with those issues in our work. The reports summarized in Figure 2 are accessible via the accompanying footnotes.
FIGURE 2: OIG audits and reviews have identified data quality issues across City departments
Data Quality Issue
A 2021 audit of the Chicago Fire Department (CFD) found that the Department's data was not adequate to allow reliable measurement of emergency response times.8 Only 75.2% of records for events from January 1, 2018 to November 30, 2020, included all the data necessary to calculate turnout and travel times for first arriving units. The remaining records contained date-time milestones that were blank or out of sequence (e.g., they did not sequentially present the times a unit was dispatched, was en route, and arrived on the scene). This prevented CFD from identifying disparities in service provision and limited its performance management capabilities. Furthermore, the Department acknowledged that it had been aware of data reliability issues since at least 2013 but had not remedied them.9
A 2021 notification to the Department of Human Resources (DHR) identified data quality and management issues with employee records appearing in the Chicago Integrated Personnel and Payroll Systems (ChlPPS).10 As of August 2020,13% of employee leave records either lacked information about the basis for the leave or showed a reason that was not authorized under Personnel Rules. The records also listed employees on leave who were likely no longer active, including employees in departments that no longer existed. Some records lacked essential information such as gender, race, and bargaining unit, while others had invalid zip codes or addresses, sometimes due to typos. This happened in part because DHR relied on other departments to update personnel records in ChlPPS. Inaccurate and incomplete employee information makes it difficult for the City to make well-informed personnel decisions,
Data Quality Dimensions
Objectivity -
Accuracy/Reliability, Coherence
Objectivity -
Accuracy/Reliability, Coherence
|109|City of Chicago Office of Inspector General, "Second OIG Audit of the Chicago Fire Department's Fire and Emergency Medical Response Times," October 12, 2021,16-17, accessed December 6, 2021, wp-content/uploads/2021/10/Second-OIG-Audit-of-the-Chicago-Fire-Departments-Fire-and-Emergency-Medical-Response-Times.pdf.|10 9|This was the second audit of CFD's fire and medical incident response times. In the first, OIG found that 9% of fire incidents and 6% of medical incidents lacked data elements needed to determine response times. City of Chicago Office of Inspector General, "Chicago Fire Department Fire and Medical Incident Response Times Audit," October 18, 2013,13, accessed December 6, 2021, Audit-Report.pdf.
10 City of Chicago Office of Inspector General, "First Quarter Report 2021," April 15, 2021, 25-27, accessed
December 6, 2021, .
PAGE 3
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
identify hiring inequities, and enforce leave restrictions per the City's Personnel Rules.
A 2020 advisory concerning the Department of Assets, Information and Services' (AIS) management of municipal license plates registered to City vehicles revealed that its inventory of municipal license plates did not match the Illinois Secretary of State's database.11 AIS did not regularly audit the City's inventory and did not validate its data. It also maintained multiple disjointed datasets and lists to track plates, some of which contained conflicting information. These issues prevented AIS from accounting for approximately 7,000 municipal license plates issued by the Secretary of State. Unaccounted-for license plates and vehicles created the risk of abuse and misconduct with the City's fleet. Objectivity - Accuracy/Reliability, Coherence
A 2020 audit of the Department of Streets and Sanitation's (DSS) enforcement of commercial and high-density residential recycling requirements found that the Mobile E-Ticket system did not allow for citations under Section 11-5-030 ofthe Municipal Code of Chicago.12 This section requires businesses and residences of five units or more to contract with private haulers for recycling services. Ward superintendents who conducted recycling inspections were therefore unable to cite building owners for failure to provide recycling services. They instead used other, inapplicable code sections to issue citations. DSS also did not maintain a list of buildings subject to the ordinance and did not consistently record the outcomes of inspections. Utility - Relevance Objectivity - Accuracy/Reliability, Coherence
A 2020 audit of DSS' weed-cutting program found that the Department could not measure performance or meet its timely weed-cutting goals, for a number of reasons.13 First, DSS did not have an accurate list of City-owned lots. The process of identifying lots requiring maintenance relied on paper records, which were sometimes lost. Additionally, staff did not consistently and accurately enter these paper records into the service request tracking system, and fields in the system did not have consistent definitions. For example, the "work order completed date" field Utility - Relevance, Accessibility, Timeliness Objectivity -Accuracy/Reliability, Coherence
11 City of Chicago Office of Inspector General, "Advisory Regarding the Department of Assets, Information, and
Services' Management of Municipal License Plates Registered to City Vehicles," June 15, 2021, 2-6, accessed
December 6, 2021,
of-Municipal-License-Plates-Registered-to-Citv-Vehicles.pdf.
12 City of Chicago Office of Inspector General, "Department of Streets and Sanitation Commercial and High-Density
Residential Recycling Enforcement Audit," December 2, 2020,12-13, accessed December 6, 2021,
igchicago.org/wp-content/uploads/2020/12/DSS-Commercial-and-High-Densitv-Recvcling-Audit.pdf .
13 City of Chicago Office of Inspector General, "Department of Streets and Sanitation Weed-Cutting Program Audit,"
July 23, 2020,9-13, accessed December 6, 2021,
Cutting-Program-Audit.pdf.
PAGE 4
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
reflected when the work order was entered into the system, not when the lot was mowed and the order was actually completed. The system also did not interface with DSS' ticketing system. This prevented the Department from monitoring work orders from start to finish.
A 2020 audit ofthe Chicago Police Department (CPD) and the Department of Family and Support Services' administration ofthe Juvenile Intervention and Support Center found that the program's case management contractor did not keep accurate and consistent records.14 The contractor used three different tracking systems to record program data, each of which was incomplete and contained inaccuracies. This made it impossible to determine whether the program was achieving positive or negative outcomes for the over 3,000 youths it served each year. Objectivity - Accuracy/Reliability, Coherence
A 2020 review of CPD's management and production of records revealed that the Department's processes could not ensure it met its constitutional and other legal obligations.15 CPD could not determine what records existed for any case or incident. It also did not track its production of records and was unable to determine where it stored its paper records. This made it impossible to know whether the Department had identified and produced all records relevant to a subpoena or other request. Utility - Relevance, Accessibility Integrity-Credibility, Physical security
A 2019 review of CPD's gang database showed that the Department lacked controls over the generation, maintenance, and sharing of data designating members of the public as gang members.16 CPD did not require users to provide evidence supporting a gang designation, nor any formal review or approval. Birth dates, gang affiliations, and reasons for gang designation within the data contradicted each other or were missing entirely. Additionally, the Department was unable to provide a complete list of records and systems containing information on gang designations. CPD was also unable to provide a complete list of external agencies with access to this information. Lastly, there was no appeal process to remove an individual's gang designation from these records unless formally expunged through the courts. These issues undermine public trust Utility - Relevance, Accessibility Objectivity -Accuracy/Reliability, Coherence Integrity - Credibility, Computer Security
14 City of Chicago Office of Inspector General, "Audit of the Chicago Police Department and Department of Family
and Support Services' Administration of the Juvenile Intervention and Support Center," February 25, 2020, 20-23,
accessed December 6, 2021, .
15 City of Chicago Office of Inspector General, "Review ofthe Chicago Police Department's Management and
Production of Records," June 10, 2020, 21-28, accessed December 6, 2021,
uploads/2020/06/OIG-Review-of-CPDs-Management-and-Production-of-Records.pdf.
16 City of Chicago Office of Inspector General, "Review of the Chicago Police Department's 'Gang Database,'" April
11, 2019,43-49, accessed December 6, 2021,
Data base-Review, pdf.
PAGE 5
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
in the police and can cause significant legal and social consequences for individuals and communities.
A 2019 audit ofthe Chicago Low-Income Housing Trust Fund's housing quality inspections found the Trust Fund did not maintain complete documentation of known lead hazards, City Building Code violations, and local court action against subsidized properties.17 It therefore could not ensure that all subsidized properties were safe. The Trust Fund also used a single, unsecured, and manually updated spreadsheet to track and report on active subsidy allocations. Formula and labeling mistakes in the spreadsheet led to shifting, duplication, and exclusion errors. As a result, the Trust Fund could not determine the amount of funding it had allotted to subsidized properties, and it published inaccurate and incomplete quarterly reports from 2014 through 2018. It under-reported its total funding commitments by $295,680 in 2017 alone. While the Trust Fund operates as an independent entity, it conducts its work in this area on behalf of the City. The City and the CDO, therefore, have a strong interest in its ability to use data for program decision making and management. Objectivity - Accuracy/Reliability Integrity - Computer Security
A 2019 audit ofthe Chicago Department of Transportation's (CDOT) driveway billing process found that incomplete and inaccurate permit data resulted in annual revenue loss between $1.1 million and $1.5 million.18 Furthermore, CDOT did not know whether it had recorded all relevant driveways in its driveway permit system. It was therefore likely that the City was forgoing an unknown amount of additional revenue by not billing all relevant property owners. OIG conducted two follow-ups to this audit. The latest, published in August 2021, reported that while CDOT had completed the migration to a new data management system, it had not made necessary corrections to ensure all existing data was complete and accurate, nor begun to identify and record undocumented driveways.19 Objectivity - Accuracy/Reliability, Coherence
17 City of Chicago Office of Inspector General, "Chicago Low-Income Housing Trust Fund Housing Quality Inspections
Audit," December 12, 2019, 3-15, accessed December 6, 2021,
08/Chicago-Low-lncome-Housing-T rust-Fund-Housing-Qualitv-lnspections-Audit.pdf.
18 City of Chicago Office of Inspector General, "Chicago Department of Transportation Commercial Driveway Billing
Audit," July 1, 2019,4, accessed December 6, 2021,
Drivewav-Billing-Audit.pdf.
19 City of Chicago Office of Inspector General, "Chicago Department of Transportation Commercial Driveway Billing
Audit Second Follow-Up," 3-5, accessed December 6, 2021,
OIG-CDOT-Drivewav-Billing-Second-Follow-Up.pdf.
PAGE 6
OIG FILE #21-1035
ADVISORY CONCERNING CITY DATA QUALITY
SUGGESTIONS
Quality data enables a government to manage resources efficiently, measure performance accurately, and achieve its objectives. We encourage the CDO to work with departments to develop a proactive culture of data quality management. For example, a uniform data quality framework could encourage departments to consider what their data needs will be and whether the quality of their current data is sufficient. Such a framework could also guide departmental trainings, policies, and processes for monitoring and improving data quality. As part of these efforts, the CDO could help identify interdepartmental data needs and facilitate requests from one department to another.
We further encourage the CDO to engage with departments and provide direction regarding the development of their data quality plans. Finally, we encourage the CDO to work with departments to provide public information on the quality of their data and its limitations to users. For its part, OIG's Audit and Program Review section has developed a new process to communicate data quality issues encountered during our work to the CDO. We will note this communication in the relevant public reports. The goal ofthe process is to ensure that the departments and the CDO are aware of existing data quality issues and to support collaborative efforts toward corrective actions.
OIG invites the CDO to respond in writing before January 14, 2022. Any such response will be made public together with this OIG Advisory.
Respectfully,
William Marback Interim Inspector General City of Chicago
Sybil Madison, Chief of Staff, Office of the Mayor
PAGE 7
The City of Chicago Office of Inspector General (OIG) is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of City government. OIG achieves this mission through,
administrative and criminal investigations by its Investigations Section;
performance audits of City programs and operations by its Audit and Program Review Section;
inspections, evaluations and reviews of City police and police accountability programs, operations, and policies by its Public Safety Section; and
compliance audit and monitoring of City hiring and human resources activities by its Compliance Section.
From these activities, OIG issues reports of findings and disciplinary or other recommendations to assure that City officials, employees, and vendors are held accountable for violations of laws and policies; to improve the efficiency and cost-effectiveness of government operations; and to prevent, identify, and eliminate waste, misconduct, fraud, corruption, and abuse of public authority and resources.
OIG's authority to produce reports of its findings and recommendations is established in the City of Chicago Municipal Code §§ 2-56-030(d), -035(c), -110, -230, and -240.
PROJECT TEAM
Kevin Smith, Chief Performance Analyst Justin Gutierrez, Performance Analyst Benjamin Spies, Chief Performance Analyst
PUBLIC INQUIRIES
Communications: (773) 478-8417 | communications@igchicago.org
TO SUGGEST WAYS TO IMPROVE CITY GOVERNMENT
Visit: igchicago.org/contact-us/help-improve-citv-government
TO REPORT FRAUD, WASTE, AND ABUSE IN CITY PROGRAMS
Call OIG's toll-free hotline: (866) 448-4754 / TTY: (773) 478-2066 Or visit: igchicago.org/contact-us/report-fraud-waste-abuse/
Cover image courtesy ofiStock. Alternate formats available upon request.
^^^^^^^
CHI
VIA ELECTRONIC MAIL
January 13, 2022
William Marback
Interim Inspector General
Office of Inspector General
740 North Sedgwick Street, Suite 200
Chicago, Illinois 60654
Interim Inspector General Marback,
This letter is in response to your December 14, 2021, letter and Advisory Concerning the City of Chicago's Data Quality. I appreciate your efforts to call attention to the need for quality data, and I welcome this opportunity to respond and to outline the significant initiatives in progress and in development to meet the recommendations provided in the Advisory. The effort will build the foundation for modern data and technology practices at the City of Chicago.
As noted in the Advisory, quality data is a prerequisite to assuring that government agencies make decisions based on useful, objective and reliable information. Data quality is the result of intentional practices, such as improving processes and workflow designs using modern software and technology. As noted by the federal government's technology consultancy 18F, the path to quality data begins with user-centered design, agile software development, product ownership, devOps, building with loosely coupled parts, and modular contracting (https://derisking-guide.18f.gov/state-field-guide ). These principles guide how the City is approaching improving the underlying systems that produce the data.
Accordingly, addressing data quality issues starts with reimagining how the City of Chicago approaches IT. Last summer, the City made an initial step with the release of the Department of Assets, Information and Services' 2021 IT Strategic Plan. The plan rightly calls out the need for citywide data governance, a data literacy program and a plan for managing data assets across the city. These recommendations accompany other initiatives that will improve data quality, such as a focus on user-centered digital service delivery practices and full-scale IT modernization of aging, legacy systems.
Beginning this month, the City will kick off multiple engagements with professional technology consultants to implement an aggressive 12-month plan to transform enterprise technology practices. Individual components of the plan include:
121 North LaSalle Street, Room 406, Chicago, Illinois 60602
Standing up an IT Transformation Office from multiple City departments, led by AIS and the Mayor's Office
Beginning the modernization of core, enterprise-wide IT systems and underlying processes
Building a shared data and analytics platform for better collaboration, sharing and automation of data insights across the City
Establishing a source-of-truth and publishing framework to build the baseline for evaluating data quality metrics
The steps above directly address the recommendations in the advisory related to developing a proactive culture of data quality management. Specifically, these steps will develop a uniform data quality framework, facilitate data requests between departments and give departments tools they do not yet have to develop data quality plans.
The Advisory also recommends working with departments to provide public information on the quality of their data and limitations to users. The City's data portal has developed a channel for communicating about datasets with metadata and quality management tools. Each dataset contains a "primer" page with an array of communication tools that are in some cases already in use, and in others could be more utilized. Strengthening the use of this platform will give departments a toolkit and best practice guide to help develop their internal efforts.
We see early signals of improvement in how Chicago manages City government with data. Last month, Bloomberg Philanthropies announced Chicago received silver certification from the What Works Cities program, which recognizes and celebrates U.S. local governments that are leading the nation in the use of data and evidence to improve services, create operational efficiencies and engage residents. This recognition has helped identify centers of excellence to build from as the City works toward a culture continuous improvement in data and technology practices.
I am excited as we work this year to increase the speed of the City's digital transformation to modern technology and data practices. I appreciate your collaboration and helpful suggestions in this area.
Sincerely,
Nick Lucius Chief Data Officer Office of the Mayor
121 North LaSalle Street, Room 406, Chicago, Illinois 60602