CITY OF.CHICAGO
-r*- ""NOVEMBER 2020. ; ¦-'¦[' ' ' .-'""^ v'.;.;,:v^,;'v-;
nit\/ r\xz nX-A \n'Ar^rs ^ • - ^ l
I ¦ B '.' ¦ B^^^H — ¦ PHWH I ft. ¦ \ ¦ • ¦HM^k. MHaM* Y ¦ ^>B^ ^^^^ . ... K1K :
GENERAL
DEPARTMENT OF ASSETS, INFORMATION AND * " SERVICES' MANAGEMENT OF INFORMATION'; . V TECHNOLOGY INVESTMENTS FOLLOW ;
inquiry" ... :,'
' 3OSE P HFERC'USON, ;| N S PECTO R*GENER ALFOR TH E CITY OF CHICAGO
CITY OF CHICAGO
OFFICE OF INSPECTOR GENERAL
740 NORTH SEDGWICK STREET, SUITE 200
CHICAGO, ILLINOIS 60654
JOSEPH M FERGUSON TELEPHONE' (773) 478-7799
INSPECTOR GENERAL FAX: (773) 478-3949
NOVEMBER 12, 2020
TO THE MAYOR, CITY COUNCIL, CITY CLERK, CITY TREASURER, AND RESIDENTS OF THE CITY OF CHICAGO:
The City of Chicago Office of Inspector General (OIG) has completed a follow-up to its December 2019 Audit of the Department of Innovation and Technology's (DolT) Management of Information Technology Investments. Based on its responses, OIG concludes that DolT—which has since been incorporated into the Department of Assets, Information and Services (AIS) as the Bureau of Information Technology— has fully implemented five out of eleven audit recommendations, substantially implemented four, and partially implemented two.
The purpose of the December 2019 audit was to determine whether DolT managed information technology (IT) investments in accordance with the U.S. Government Accountability Office's Information Technology Investment Management framework.1 Our audit found that DolT did not consistently adhere to best practices for project selection, thereby increasing the risks of projects delivering fewer benefits, costing more, and/or taking longer than expected to complete. In addition, DolT's data collection practices hampered effective monitoring and evaluation of project and portfolio performance, consequently limiting the Department's ability to identify opportunities for improvement.
Based on the results of the audit, OIG recommended that DolT follow internal policies and industry best practices with respect to project selection, monitoring, and evaluation. These recommendations included completing internal documents to guide these activities, requiring all project managers to follow these policies consistently, and empowering governing committees to meet mandates for project oversight. In its response to the audit, DolT described corrective actions it would take.
In August 2020, OIG inquired about the status of corrective actions taken by AIS. Based on the Department's follow-up response, OIG concludes that AIS has substantially implemented corrective actions. Specifically, AIS has,
' U.S Government Accountability Office, "Information Technology Investment Management A Framework for Assessing and Improving Process Maturity," March 2004, 2, accessed October 26, 2020, pdf.
IGCHICAGO.ORG | OIG TIP LINE: (866) 448-4754 | TTY (773) 478-2066
updated its Project Management Office (PMO) Handbook in key areas and required its project managers to follow it;
developed procedures for collecting more robust data for project selection;
implemented new project management tools;
implemented monthly Information Technology Governance Board (ITGB) meetings;
ensured that project managers and the PMO Director provide the data needed for ITGB to execute its oversight role; and
endeavored to fully staff its Information Security Office.
To fully satisfy the audit's recommendations, AIS should add full project life cycle costs (such as maintenance and ongoing support) to its project cost estimates, and work with the Office of Budget and Management and the Mayor's Office to .ensure that all City departments submit their IT project requests to ITGB for review and approval. Once fully implemented, OIG believes the corrective actions reported by AIS may reasonably be expected to resolve the core findings noted in the audit (i.e., ensure IT projects deliver expected benefits on time and within budget). Below, we summarize our three audit findings and recommendations, as well as the Department's response to our follow-up inquiry.
We thank the staff and leadership of AIS for their cooperation during the audit and responsiveness to our follow-up inquiries.
Joseph M. Ferguson Inspector General City of Chicago
IGCHICAGO.ORG | OIG TIPLINE. (866) 448-4754 | "ITY: (773) 478-2066
Respectfully,
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
FOLLOW-UP RESULTS
In August 2020, OIG followed up on its December 2019 Audit of the Department of Innovation and Technology's Management of Information Technology Investments (DolT).2 DolT—which has since been incorporated into the Department of Assets, Information, and Services (AIS) as the Bureau of Information Technology—responded by describing the corrective actions it has taken and providing supporting documentation. Below, we summarize OIG's three findings, the associated recommendations, and the status of AIS' corrective actions. Our follow-up inquiry did not observe or test implementation of the new procedures; thus, we make no determination as'to their effectiveness, which would require a new audit with full testing.
PR/StICES FOR.^ELEGTIC^"'l^c3EGj5:M/W;:^; DE LWERliFEM E E&B E NE'E ITSr GOST.llyl O R'Ef'AN D-h TAKE LONGER™AN EXREGTED.T© COMPIeTES'
OIG RECOMMENDATION 1:
OIG recommended DolT require all project managers to follow the Project Management Office (PMO) Handbook for project selection activities to promote consistent, repeatable performance of duties. In particular, we recommended DolT require project managers to use predefined criteria to rank all projects before selection.
STATUS OF CORRECTIVE ACTION: FULLY IMPLEMENTED
AIS updated its Handbook to include a rubric of standardized project selection criteria. The Department stated that it has instructed its project managers to score potential projects using these criteria, and provided an example illustrating how this is done. The example demonstrates the use of nine scoring criteria, compared to the six outlined in the Handbook. To maximize standardization and clarity, we encourage AIS to update the Handbook further to include all scoring criteria.
2 The December 2019 report is available on the OIG website: depart.ment-of-innovation-and-technoloqvs-manaaement-of-information-technoloav-invest.ment.s/.
PAGE 1
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
OIG RECOMMENDATION 2:
OIG recommended DolT develop procedures for collecting more robust cost/benefit and risk data to improve comparisons between potential projects. DolT may choose to base the level of rigor required on the relative cost and risk of the project. We further recommended that DolT work with the Office of Budget and Management (OBM) to budget for projects through their full life cycle—not only year-to-year—and improve its scoring tool by requiring reviewers to provide justifications for their scores.
STATUS OF CORRECTIVE ACTION: PARTIALLY IMPLEMENTED
AIS provided an example project assessment document that illustrates how projects are categorized and how expected benefits and risks are assessed in quantitative terms, in accordance with updated guidance in the PMO Handbook. Project-specific risk logs also rate the severity, probability, and impact of particular issues on project development, and provide mitigation plans. AIS stated that the costs, benefits, and risks analyzed during project assessment are incorporated into all selection decisions. AIS also stated that, as of summer 2020, its project managers have been required to provide written justifications for their assessment scores. A generic notes field in the assessment document is apparently used for this purpose. While this satisfies part of OIG's recommendation, we encourage AIS to make the requirement to provide written justifications explicit in future versions of the PMO Handbook and assessment template.
Regarding project life cycle costs, AIS acknowledged that support and maintenance costs are not always included as part of cost estimates in project requests. In some instances, the project manager adds these costs to the statement of work once the project is approved. OIG reviewed three statements of work provided by AIS as documentation. These statements—which function as agreements with third-party vendors as to project deliverables, schedule, and costs—included estimates related to the delivery of the product, but none related to ongoing maintenance and support beyond the project development timeframe. Documentation of AIS'discussions with OBM regarding project budgets likewise details project implementation costs, but does not account for ongoing costs to
PAGE 2
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
be expected over the useful lives of the IT assets. We encourage AIS to forecast asset life cycle costs beyond their initial development and implementation, and to incorporate them into its project cost estimates.
OIG RECOMMENDATION 3:
OIG recommended DolT work with OBM and the Mayor's Office to ensure that the Information Technology Governance Board (ITGB) continues to meet at least quarterly to perform its role in the selection process. Further, we recommended all City departments—including the Chicago Police Department (CPD), Chicago Fire Department (CFD), and Office of Emergency Management and Communications (OEMC)—be required to submit projects to ITGB for selection.
STATUS OF CORRECTIVE ACTION: PARTIALLY IMPLEMENTED
ITGB has been meeting approximately monthly since the release of the audit, as reported by AIS and supported by documented meeting agendas and minutes.
However, AIS stated that while CPD, OEMC, and the Civilian Office of Police Accountability (COPA) did submit projects for approval in 2019 and 2020, these were not processed by ITGB because they related to efforts undertaken pursuant to the police consent decree.3 The Department further stated that a project request from CFD will be included as part of a citywide human resources system upgrade. Finally, AIS stated it will work closely with an IT group within the new Office of Public Safety Administration to ensure that its projects line up with the City's priorities and resources. We encourage AIS to work with OBM and the Mayor's Office to ensure that these departments submit their IT project requests to ITGB for approval, as other City departments do.
3 https.//wvwvchicaao.aov/cit.v/ep/sites/police-reforrTi/home/consent-decree.html
PAGE 3
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
OIG RECOMMENDATION 4:
OIG recommended DolT require all project managers to follow the PMO Handbook, as stated in the previous finding, including monitoring cost/benefit and risk performance for all projects and submitting all projects to the Architecture Board and Information Security Office (ISO) for review.
STATUS OF CORRECTIVE ACTION: SUBSTANTIALLY IMPLEMENTED
AIS stated that it has instructed project managers to monitor project benefits per the Handbook, and provided an example status report showing how this is done. Project managers are likewise required to manage project budgets; AIS referred to project statements of work as its managers' tools for matching project milestone completion against invoices from the vendor. These statements are static budgets agreed upon by AIS and the vendors; they do not reflect active tracking of projects' expenses for their adherence to budget. However, AIS does track project budget performance in documents prepared for its ITGB meetings. Project managers use risk logs to identify, manage, and resolve risks throughout a project's development.
AIS has suspended regular meetings of the Architecture Board during the COVID-19 crisis—stating that it repurposed these meetings to address projects related to the crisis—but hopes to re-establish them soon. AIS provided examples of the presentations its project managers submit to the Architecture Board and ISO for review. The Department acknowledged, however, that these entities' notes and decisions are not yet documented for future reference. Moving forward—once it has reestablished these meetings—AIS intends to document these notes and decisions in meeting minutes. While we acknowledge the importance of addressing the City's IT needs related to COVID-19, the Architecture Board and ISO also fulfill important functions independent of this crisis. We encourage AIS to resume regular meetings as soon as possible.
PAGE 4
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
OIG RECOMMENDATION 5:
OIG recommended DolT update the PMO Handbook and/or the City's IT Governance Policy to define criteria for determining whether to terminate underperforming projects.
STATUS OF CORRECTIVE ACTION: FULLY IMPLEMENTED
AIS has updated its Handbook with a process for project cancellation and stated that the IT Governance Policy has been similarly updated. The Department provided examples of projects that had been terminated following the process.
OIG RECOMMENDATION 6:
OIG recommended DolT ensure that ITGB continues meeting on at least a quarterly basis and fully inhabits its role of providing project oversight. DolT and OBM should work with ITGB to ensure that project managers collect the relevant data to enable ITGB to perform these functions. We recommended that at a minimum DolT should provide data related to actual cost/benefit, risk, and schedule performance.
STATUS OF CORRECTIVE ACTION: FULLY IMPLEMENTED
As reported in the follow-up to recommendation 1, ITGB is now meeting on a monthly basis. AIS developed a report template for ITGB to review projects. The report contains fields to track project funding expended and available, issues needing resolution, development progress, estimated completion, and manager notes, among other data. AIS provided examples of recent reports, which appear to provide relevant data for ITGB to fulfill its oversight role.
OIG RECOMMENDATION 7:
OIG recommended DolT work with OBM to ensure that the ISO is adequately staffed.
STATUS OF CORRECTIVE ACTION: SUBSTANTIALLY IMPLEMENTED
AIS provided the details of a hiring plan developed in 2016, which included adding eight new positions to ISO. While these were all initially approved by OBM, budget considerations led OBM to restrict some of these positions from rehiring in 2019, which were
PAGE 5
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
then reapproved for hire in 2020. As of the time of OIG's follow-up inquiry (August 2020), AIS reported that four of these eight positions were filled, three had received approval for hire, and one had been cut from the 2020 budget. AIS stated that OBM also funded two ISO staff positions in addition to the eight above— which have been filled—but that it had requested an additional eleven positions which have not been funded. While AIS has demonstrated its ongoing work with OBM, fully staffing the ISO remains a challenge.
,DOIT DID NOT CONSISTEi^Tl^B^ATuATE^
nk^ PR03ECTS AND ADJUST ITSrlNVESTMEN .¦¦ RINDING 3: ' • -¦ ^ ¦ ¦ ¦¦¦^ ¦ ¦•-¦^ .--u. --^w;-• ^H®#^:^t- ¦, ;. ¦LESSONS LEARNED ^ '£ £ £'. _^,v MZ
OIG RECOMMENDATION 8:
OIG recommended DolT work with ITGB to define the processes and criteria for evaluating project and portfolio-level performance.
STATUS OF CORRECTIVE ACTION: SUBSTANTIALLY IMPLEMENTED
AIS has added guidance on project performance management to the PMO Handbook. The Department stated that project managers follow this guidance by updating performance data in a project management tool throughout the life of a project, and provided examples of how the tool works in the form of reports to ITGB. AIS stated that these project reports are run every week and can be run on demand.
The updated PMO Handbook likewise provides guidance for management at the portfolio level. AIS referenced the same project management tool in its statement on portfolio-level performance management, but did not provide evidence of having implemented the Handbook's direction to provide reports and dashboards on portfolio performance to senior staff and ITGB. While AIS' project management tool provides the ability to consider the performance of larger portfolio by reviewing the performance of each individual project, it does not provide a dashboard-level overview of portfolio performance.
PAGE 6
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
OIG RECOMMENDATION 9:
OIG recommended DolT fully implement its new project management tool and ensure its staff consistently records the performance data required by ITGB.
STATUS OF CORRECTIVE ACTION: FULLY IMPLEMENTED
AIS stated that its project management tool was fully implemented in the Spring of 2019. As noted above, the Department provided examples of the tool in use that show the data points collected and demonstrate that the tool is widely utilized. AIS further stated it ensures that information is being consistently and accurately entered into the project management tool by running reports from it on a weekly basis and having senior staff review the reports.
OIG RECOMMENDATION 10:
OIG recommended DolT ensure that project managers evaluate individual performance for all projects after implementation, and document their lessons learned.
STATUS OF CORRECTIVE ACTION: FULLY IMPLEMENTED
AIS has added detail on the project closeout process to the PMO Handbook, including documenting lessons learned. The Department stated that it has instructed its project managers to use a standardized template to document the lessons learned after each project, and provided examples that appear to track the lessons both in real time and after project completion.
OIG RECOMMENDATION 11:
OIG recommended DolT assign someone to ensure the information collected meets the needs of ITGB.
STATUS OF CORRECTIVE ACTION: SUBSTANTIALLY IMPLEMENTED
AIS stated that the PMO Director fulfills this role, and provided a list of this position's responsibilities, including,
scheduling ITGB's meetings;
preparing ITGB reports, agendas, and minutes;
monitoring project progress and expenses;
PAGE 7
OIG FILE #20-0923
DOIT IT INVESTMENT MANAGEMENT FOLLOW-UP
processing new project requests;
compiling new project reports;
following up with departments submitting new projects; and,
maintaining the ITGB membership list.
The updated PMO Handbook also contains some guidance on the PMO Director's responsibilities, but does not specifically list them. As previously noted, AIS provided documentation of the PMO Director's reports to ITGB in the form of its project management tool.
PAGE 8
MISSION
The City of Chicago Office of Inspector General (OIG) is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of City government. OIG achieves this mission through,
administrative and criminal investigations by its Investigations Section;
performance audits of City programs and operations by its Audit and Program Review Section;
inspections, evaluations and reviews of City police and police accountability programs, operations, and policies by its Public Safety Section; and
compliance audit and monitoring of City hiring and human resources activities and issues of equity, inclusion and diversity by its Diversity, Equity, Inclusion, and Compliance Section.
From these activities, OIG issues reports of findings and disciplinary and other recommendations,
to assure that City officials, employees, and vendors are held accountable for violations of laws and policies;
to improve the efficiency and cost-effectiveness of government operations; and
to prevent, detect, identify, expose, and eliminate waste, inefficiency, misconduct, fraud, corruption, and abuse of public authority and resources.
AUTHORITY
OIG's authority to produce reports of its findings and recommendations is established in the City of Chicago Municipal Code §§ 2-56-030(d), -035(c), -110, -230, and 240.
Cover image courtesy of iStock.
¦ ,' • ¦ ', PROJECT TEAM ,. - ~;V ¦ ^ ."';';V "
'" BEN SPIES, SENIOR PERFORMANCE ANALYST ¦*'/ • '? ¦ '¦ \''\CAMERON LAGRONE, CHIEF PERFORMANCE ANALYST '' >v .,,,. ; ; ¦ LISE VALENTINE, DEPUTY INSPECTOR GENERAL '\ ¦
, V ' PUBLIC INQUIRIES: , ; .'V^
NATALIE ArKURIATA: (773) 478-8417^NKURIATAaiGCHICAGO.ORG
TO SUGGEST WAYS TO IMPROVE CITY GOVERNMENT, VISIT:
IGCHICAGO.ORG/CONTACPOS/HELP-IMPROVE'-CITYJ:GOVERNMENT
TO REPORT FRAUD, WASTE, AND ABUSE IN CITY PROGRAMS: CALL OIG'S TOLL-FREE TIP LINE' (866) 448-4754?TTYV(773) 478-2066*
OR VISIT OUR WEBSITE
igghicago.org/contact-us/report^-fraud'-vvasteLabuse/;