JOSEPH M. FERGUSON INSPECTOR GENERAL
CITY OF CHICACO OFFICE OF INSPECTOR GENERAL 740 NORTH SEDGWICK STREET, SUITE 200 CHICAGO, ILLINOIS 60654 TELEPI IONE (773) 478-7799 FAX (773) 478 3949
DECEMBER 19, 2019
TO THE MAYOR, CITY COUNCIL, CITY CLERK, CITY TREASURER, AND RESIDENTS OF THE CITY OF CHICAGO:
The City of Chicago Office of Inspector General (OIG) has completed an audit ofthe Department of Innovation and Technology's (DolT) management of the City's investment in information technology. The objective ofthis audit was to determine if DolT manages information technology investments in accordance with best practices outlined in the United States Government Accountability Office's Information Technology Investment Maturity framework. Specifically, we examined how DolT ensures that the City selects the right technology projects, manages them effectively, and evaluates performance after completion.
Based on the audit results, OIG concluded that DolT did not consistently adhere to best practices for project selection, which increased the risk that projects may cost more, take longer to complete than expected, and not meet requirements. OIG also determined that DolT does not consistently and accurately monitor project performance, nor does it consistently evaluate performance after completion or use lessons learned to inform future projects.
It is critical that DolT fully implement a process for selecting projects that not only meet departments' needs and aligns with the City's strategic goals, but also allocates limited City resources in the most efficient manner possible. Moreover, the Department needs to provide effective project management to ensure that expected benefits are delivered on budget and on schedule. Finally, a consistent and rigorous approach to evaluating past performance is necessary to identify lessons learned and use those lessons to improve future projects. DolT agreed with our recommendations and has already begun implementing corrective actions to improve the City's project selection, management, and evaluation processes.
We thank DolT staff and management for their cooperation in this audit We also thank staff from various City departments for providing information regarding their experience with IT projects.
Respectfully,
IGCHICAGOORG | OIG UPLINE (866) 448-4734 | I IY (773) 478-2066
<3^—
Joseph M. Ferguson Inspector General City of Chicago
IGCHICAGOOPG | OIG UPLINE (866) 448-4734 | TTY (773) 478-2066
OKj f-ILE ,717-0638
AUDIT OP DOIT'S MANAGEMENT OF INFORMATION TECI INOLOGY INVESTMENTS DECEMBER 19, 2019
TABLE OF CONTENTS
I. EXECUTIVE SUMMARY|910|A CONCLUSION |910|B FINDINGS 3
C RECOMMENDATIONS|910|D DOIT RESPONSE |910|li. BACKGROUND|910|A PROTECT MANAGEMENT OFFICE|910|INFORMATION SECURITY OFFICE 7
CITY OF CHICAGO IT GOVERNANCE STRUCTURE 7
IT INVESTMENT BEST PRACTICES 9
FINDINGS AND RECOMMENDATIONS 12
OBJECTIVES, SCOPE, AND METHODOLOGY : 26
OBJECTIVE 26
SCOPE 26
METHODOLOGY 26
STANDARDS 27
AUTHORITY AND ROLE 27
APPENDIX A: IT GOVERNANCE POLICY 28
APPENDIX B: PROJECT DESCRIPTIONS 37
APPENDIX C: MANAGEMENT RESPONSE ATTACHMENTS 39
ACRONYMS
DolT Department of Innovation and Technology
OBM Office of Budget and Management
OIG City of Chicago Office of Inspector General
PMO Project Management Office
PACE i
OIG HLE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVEST MENTS DECEMBER 19, 2019
¦1 ¦ ' • :- •, ¦, City of Chicago . -V'
•*>»' Office of Inspector General *' t •
AUDIT OF THE DEPARTMENT OF ' „ INNOVATION AND-.TECHNOLOGY'S (DOIT) * MANAGEMENT OF INFORMATION "
• TECHNOLOGY INVESTMENTS": - >
DolT did not estimate complete cost, benefit, and risk information before selecting projects
DolT did not identify performance goals for projects
DolT did not consistently monitor project spending
5 out 6 projects reviewed took longer than scheduled (Gl to complete
PAGE 2
OIG I-ILL #17-0630
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
I. EXECUTIVE SUMMARY
The City of Chicago Office of Inspector General (OIG) has completed an audit ofthe Department of Innovation and Technology's (DolT) management ofthe City's investment in information technology. The objective ofthis audit was to determine if DolT manages information technology investments in accordance with best practices outlined in the United States Government Accountability Office's Information Technology Investment Maturity (ITIM) framework. Specifically, we examined how DolT ensures that the City selects the right technology projects, manages them effectively, and evaluates performance after completion.
Effective management of an IT portfolio requires consistent and repeatable organizational processes.1 While certain projects may succeed without consistent enterprise-wide management, such successes are more often attributable to exceptional individual efforts, rather than effective, efficient, and repeatable institutional processes.
To assess the consistency and repeatability of DolT's processes, OIG compared documentation of DolT's processes and the outcomes of projects, such as budget or schedule information, to GAO's ITIM framework. The framework describes five stages of process maturity. At the lowest level—Stage 1—organizations make IT investment decisions in an unstructured, ad hoc manner. This suboptimal approach may result from a lack of well-designed formal procedures, inconsistent implementation of such existing procedures, or a combination ofthe two. At the highest level—Stage 5— organizations have optimized their processes, and IT investments drive strategic organizational change. DolT is in Stage 1 and is working toward Stage 2
CONCLUSION
DolT did not consistently adhere to best practices for project selection, which increased the risk of projects delivering fewer benefits, costing more, and/or taking longer than expected to complete. In addition, DolT's data collection practices hamper effective monitoring and evaluation of project and portfolio performance, thereby limiting the Department's ability to identify opportunities for improvement
FINDINGS
DolT designed a scoring tool to assess projects on a common set of predefined criteria, with the goal of ranking projects and selecting those that would most benefit
' United States Government Accountability Office, "information Technology Investment Management A Framework for Assessing and Improving Process Maturity," March 2004, 2. accessed October II, 2019. http //yyyvw ciao Cioy/assets/6O/76790 ndf
PAGE 3
OIG FILE rfl? 0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
City operations. OIG review of eight projects started in 2016 and 2017 determined that DolT did not use the ranking process at all. Notably, DolT did not have a complete inventory of the projects initiated during.the years under review. Moreover, DolT completed the required assessment prior to selecting only three of the eight OIG-reviewed projects. As a result, the City may have selected projects that did not best meet the departments' specific and the City's overall needs. The Department did not consistently collect critical information needed to rank projects and make selection decisions. In addition, the Chicago Police Department, Chicago Fire Department, and Office of Emergency Management and Communications each declined to use the project selection process DolT developed. Therefore, DolT could not rank these departments' projects against those proposed by other departments for purposes of setting priorities for spending City resources.
DolT did not ensure that launched projects met performance goals and did not consistently monitor progress. Five ofthe six projects reviewed took longer than scheduled to complete, with two taking more than twice as long as originally planned. Moreover, DolT did not have a process or criteria for determining whether ongoing projects were meeting user department needs and should be continued or terminated.
DolT did not evaluate projects across its portfolio and, therefore, did not adjust its investment processes based on lessons learned. The Department did not consistently evaluate project, performance after project completion. Some project managers told us that, while they typically discuss lessons learned from projects, those discussions are not memorialized or used to improve project and portfolio management.
C. RECOMMENDATIONS
OIG recommends that DolT rank all proposed projects using predefined criteria. The Department should also develop procedures for collecting more robust cost, benefit, and risk data to facilitate comparative evaluation of the merits across departments, i.e., City-wide. DolT should work with the Office of Budget and Management (OBM) and the Mayor's Office to ensure that the various boards, groups, and other entities authorized to oversee IT strategy and spending are fully engaged in maximizing the return on the City's investments throughout the project lifecycle.
DolT should also set performance goals related to cost/benefit and risk for each project, monitor performance against those goals, and report on performance to the appropriate governance body. Finally, project oversight should include evaluation of outcomes and long-term performance. Taking a broad view ofthe City's portfolio of projects will improve the Department's decision making at the proposal stage.
PACE 4
OIG FILE w'17-0638
AUDIT OF DOM'S MANAGEMENT OF INFORMATION TECHNOLOGY INVEST MENTS DECEMBER 19, 2019
D. DOIT RESPONSE
In response to our audit, DolT agreed with OIGs recommendations and stated that it has undertaken changes that will address the findings. These changes include updating relevant policies, requiring project managers to adhere to all written policies for selection, monitoring and evaluation of projects, achieving full engagement by the IT Governance Board, and requiring all City departments to engage in the standardized IT oversight processes.
The specific recommendations related to each finding, and DolT's response, are described in the "Findings and Recommendations" section ofthis report. ¦
PAG F 5
OIG FILE -£17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
II. BACKGROUND
DolT is "responsible for ensuring that the City's technology infrastructure is robust and works with City departments to design and implement technology improvements."2 The Department also oversees the City's geographic information systems and data science programs, and sets information security standards through its Information Security Office.
A. PROJECT MANAGEMENT OFFICE
DolT's Project Management Office (PMO) bears primary responsibility for coordinating the design and implementation of technology improvements. As described on the City website, "the PMO,
assigns project managers to manage key IT projects;
sets project management standards and implements best practices;
provides project management process support to all staff members that manage projects;
provides transparency into the performance ofthe project portfolio; and
supports project portfolio management processes, including project ideation, selection, and prioritization."-'
PMO staff oversee software and project management contractors, serving as the point of contact for these vendor-provided projects. PMO's Charter states the Office "provides value to the City of Chicago by ensuring that,
scarce resources are invested in projects that align with the City's business and technology goals and strategies;
projects are managed in a repeatable, standardized manner using industry best practices; and
project objectives and outputs meet business needs and meet or exceed end users' expectations."
7 City ofChicago. Office of Budget and Management. "2019 Budget Overview", 66, accessed October 11, 2019,
h ps//ww'.y Chicago aov/cont en t/da m/city/depts/obmi/su pp info/7019 D u cl ci e t /7 019 B1.1 cl q e t O v e r v i e w pel f ' Cily of Chicago, Department of Innovation and Technology, "Planning, Policy and Management", accessed October Tl. 2019,
hups//www Chicago ciov/citv/en/deprs/iTo;t/nrovovs/bi.isiness developmentiTianaqementpino hi nil
PAGE 6
OIG FILE #17-0638
AUDI T OF DON'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENT'S DECEMBER 19, 2019
The Charter also includes a mission statement that states, "Through standardization and collaboration, we deliver quality projects efficiently, faster, and at minimal cost Lo our internal clients (City departments) and external clients (Chicago residents, businesses, and visitors)." This reflects DolT's appreciation ofthe value of selecting the most beneficial projects, carefully managing them, and evaluating their efficacy once implemented.
PMO also developed a Handbook that defines its policies and procedures and guides the work of its project managers. The Handbook is based on practices recommended. by the Project Management Institute, which generally align with CAO's ITIM framework.
INFORMATION SECURITY OFFICE
DolT created the Information Security Office (ISO) in 2013 to provide "enterprise security monitoring and response" across City departments.4 The responsibilities of ISO include "[developing and enforcing] an information security strategy, framework, policies and procedures that align City of Chicago business need, legislative and regulatory requirements and industry best practices."5 The PMO Handbook states that ISO,
reviews an initial security assessment for projects prior to approval;
monitors project adherence to the security requirements policies; and
provides a security testing process to ensure that projects involving sensitive data meet security requirements
As discussed below in Finding 2, DolT stated that ISO has been unable to fulfill these responsibilities on a consistent basis due to staffing shortages. According to DolT, hiring and retaining individuals in these positions has presented an ongoing challenge due to high industry demand for skilled employees.
CITY OF CHICAGO IT GOVERNANCE STRUCTURE6
The City's Information Technology Governance Policy "establishes a standard citywide process for requesting, prioritizing, and selecting proposed IT investments." The Policy
4 City of Chicago, Department of Innovation and Technology, "Information Security Office'', accessed October 11, 2019,
ntt os //vvvvw Chicago goy/aty/en/depts/coit/pi ovars/secunty .and . data manage me nr. hi ml
¦ City ofChicago. Department of Innovation and Technology, "Information Security Office", accessed
October 11, 2019,
ntt os //www Chicago ooy/citv/en/deots/cioil/oi ovd' s/secu; itv..and..dai a management him I ' See pages 3 A of the IT Governance Po-icy found in Appendix A for quotes of doscr.puons
PAGE 7
OIG HLE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
creates a Technology Strategy Group (TSG) that "is comprised of leadership from all City departments who will work to collaboratively set citywide digital strategy, and identify technologies that deliver community benefit, optimize resources, improve service delivery, reduce risk, and build capacity." DolT stated that the group has met once so far, but anticipates it meeting quarterly going forward. The full Information Technology Governance Policy is attached as Appendix A to this report.
The Policy also establishes an IT Governance Board (ITGB) which reviews all requests for new IT investments or expansions of existing projects and makes decisions that align with the strategy set by TSG. ITGB, "is comprised of staff from the Mayor's Office, the Office of Budget & Management, DolT and consulted by the Departments of Finance and Procurement Services." The board is responsible for approving "requests for funding, regardless of source, for new projects and services, as well as for subsequent phases to previously approved projects," and monitoring "project health and outcomes on a monthly basis." As discussed below in Finding 1, ITGB held its first meeting in August 2018.
The Information Technology Governance Policy assigns DolT's IT Architecture Board the role of "set[ting] enterprise technology standards" to ensure that project technologies are compatible across platforms. The PMO Handbook states that projects will not move forward to implementation without Architecture Board review for technological alignment. v
Finally, DolT's Project Management Office (PMO) supports ITGB and the governance process by scoring the financial impact of all projects prior to selection by ITGB and reporting to the Board on the health of ongoing projects. PMO "is responsible for reviewing all new project requests and associated business cases and integrates the decisions of the TSG and ITGB into new and ongoing programs and projects". Figure 1 illustrates the relationships between these various boards and offices.
PACE 8
OIG FILE #17 0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMAT ION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
FIGURE 1: CITY IT GOVERNANCE STRUCTURE
TSG
- . Sets;
ITGB
: Selects and.oversees projects in-alignment with TSG strategy
Reviews projectS;to ensure compliance with the City's technical standards
PMO
Manages projects and provides,ITGB with* data : to support oversight ;
Source OIG illustration based on City ofChicago Information Technology Governance Policy
D. IT INVESTMENT BEST PRACTICES
The United States Government Accountability Office "Information Technology Investment Management (ITIM): A Framework for Assessing and Improving Process Maturity" lays out a "model composed of five progressive stages of maturity that an agency can achieve in its IT investment management capabilities."7 This model states "just as ITIM can be used as a tool for organizational improvement, it can also be used as a standard against which to judge the maturity of an organization's IT investment management process."8
As illustrated in Figure 2, ITIM defines three fundamental phases of investment in IT projects: select, control, and evaluate. An organization moving through these phases answers the following fundamental questions'
How do you know that you have selected the best projects?
How are you ensuring that projects deliver benefits?
Are the systems delivering what you expected?
7 United States Government Accountability Office, "Infoi mation Technology Investment Managemem A Framework for Assessing and Impiov.ng Process Maturity/' March 2004, Highlights, accessed October II, 7019, http//www gao gov/asseis/80/76790 oof
''¦ United States Government Accountability Office "Infonnation Technology Investment Management A Framework for Assessing and Imorovmg Process Maturity," March 2004. 26, accessed October 11. 2019, hl.tp//www gao gov'assols/80/76790 ix.il'
PAG t 9
OIG Fll.h /H7-0638
AUDIT Of DOIT'S MANAGEMENT or INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
FIGURE 2" SELECT, CONTROL, AND EVALUATE PHASES OF IT INVESTMENT
Select phase
Screen
Rank
Choose
Evaluate phase
Apply lessons learned
Conduct interviews
Make adjustments
/Are.the systems
delivering.;whatm
^iifs¦¦¦¦¦¦ ¦ -'^m'" • you expected?
H.OTyjdo.you. know^ that you have selected the best
Source GAO ITIM9
During the select phase, the organization analyzes the risks and benefits of and ranks potential projects before committing significant funding to any of them. As a selected project progresses during the control (i.e., implementation) phase, the organization assesses whether the project remains likely to deliver the expected benefits on time and on budget, and makes any changes needed to ensure those outcomes. After project implementation, during the evaluate phase, the organization determines whether the investment is delivering the expected benefits or whether adjustments are necessary, and documents lessons learned to improve future projects.
ITIM frames an organization's maturity in terms of how well it performs in each phase. More mature organizations devise and follow repeatable, effective, and efficient processes. It is important that organizations engage in continual assessment, affirmatively choosing to reselect—i.e., continue to work on—or deselect projects based on whether they are providing sufficient value. Because it is often hard for organizations to halt a project once launched, even when the dedicated resources could be put to better use, the framework emphasizes the reselection and deselection processes. ITIM also stresses the importance of an organization developing its capabilities for portfolio management, and the key role that investment boards play in organizational IT governance. Figure 3 outlines the characteristics of each ITIM maturity stage.
'¦' United States Government Accountability Office, "Information Technology Investment Management A Framework for Assessing and Improving Process Maturity," March 2004, 8, accessed October 11, 2019. hrtp//vvvvw gao goy/a?;.ors/80/7G790 pdf
PACE 10
OIG h I LE #17-0638
AUDIT OF DOITS MANAGEMENT OE INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
FIGURE 3' FIVE STAGES OF MATURITY IN THE ITIM FRAMEWORK
• The organization has mastered the selection, control, and evaluation processes and now seeks to shape its strategic outcomes by benchmarking its IT investment processes relative to other "best-in-class" organizations.
The organization is focused on evaluation techniques to improve its IT investment processes and portfolio(s), while maintaining mature selection and control techniques.
Stage; 2: Building theinvestment foundation
• The organization has developed a well-defined IT investment portfolio using an investment process that has sound selection criteria and maintains mature, evolving, and integrated selection, control, and evaluation processes.
-.s.i"-
Basic selection capabilities are being driven by the development of project selection criteria, including benefit and risk criteria, and an awareness of organizational priorities when identifying projects for funding. Executive oversight is applied on a project-by-project basis.
• Ad hoc, unstructured, and unpredictable investment processes characterize this stage. There is generally little relationship between the success or failure df one project and the success or failure of another project.
Source GAO 111M
An initial indicator that an organization is maturing is the implementation of consistent, repeatable investment processes. This consistency should span all project types, and all project managers should follow the same processes to achieve consistent outcomes. Ad hoc or inconsistent project management are hallmarks of a less mature organization. The City is at Stage 1 the lowest stage of organizational maturity-- and is working toward Stage 2.
PACE 11
Although DolT's project selection processes generally aligned with ITIM best practices, the Department did not consistently follow those processes. Most importantly, DolT did not assess and prioritize all proposed IT projects using predefined criteria.
DolT designed a scoring tool to assess projects on a common set of predefined criteria, with the goal of ranking projects and selecting those that would most benefit City operations.10 OIG review of eight projects initiated in 2016 and 2017 determined that DolT did not use the ranking process at all. Notably, DolT did not have a complete inventory of the projects started during the years under review. Moreover, DolT conducted the required assessment prior to selecting only three of the eight OIC-reviewed projects." As a result, the City may have selected projects that did not best meet the departments' specific and the City's overall needs. The projects selected may deliver fewer benefits, cost more, and take longer than expected to complete. Figure 4 identifieseach project we reviewed. Detailed descriptions ofthe projects are provided in Appendix B.
'T lie scoring tool is incoi poiatecl into me i I Governance Policy enclosed in Appendix A DolT retroactively completed the scoring tool afier selecting a fourth project
PACE "12
OIG FILE fH7-0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVEST MENTS DECEMBER 19, 2019
FIGURE 4: EIGHT OIG-REVIEWED PROJECTS INITIATED IN 2016 AND 2017l?
Project Name
311 Modernization
Hyperion Budget System
Array of I hings ;-
Utility Tax.
¦ Chicago.Early Learning : ~
House:Share Registration
System (Phase 1)
House Shore Registration '.
System (Phase 2)
Citrix Enterprise Services
Upgrade • ¦ - ¦
Source. OIG review of DolT project
Description- ., . . . I
Replace legacy-system that supports:311
Replace legacy budget system
Install sensors to collect data for
" research and public: use - ¦*>
Integrate water and'sewer taxes into
City's water billing system . •
Create portal for early learning
enrollment - i't
Create system to register shared,-.
housing rental units
Add functionality not addressed in' '¦¦
Phase]ile:
Upgrade hardware and software
environment that hosts over 20 City,,
applications ..... ;'
documentation
:;.:°,st Mm0
Estimate,"
$;3 5,000,000. $5,731,514 $4,250,000/
$2,100,036
$1,000,000
5698,770
.:$¥9:$tobo
S362.393
Although the PMO Handbook required project managers to score and rank projects, DolT management did not enforce these requirements, instead allowing project managers to rely on their own experience. ITIM recommends that institutions establish an IT Investment Board to oversee IT investment management and ensure adherence with internal policies and procedures, including those related to project selection. The City's ITGB would have fulfilled this role, but it never met between 2015 and August 2018. Although ITGB began to meet in 2018, it did not provide Citywide oversight for purposes of setting the 2019 budget. The Chicago Police Department (CPD), Chicago Fire Department (CFD), and Office of Emergency Management and Communications (OEMC) declined to use the project selection process DolT developed,
selecting their own IT projects without ITGB CPD, CFD, and OEMC
review or approval for 2019 funding ]i declined tO USe ITGB
and selected their own projects
"¦' DolT did not have a complete inventory of projects mil lated in 2016 and 2017 OIG selected a targeted sample of eight projects from among those known to have launched during this time period v for example. CPD spent at. least $1 1 million on compute's, storage, and support from Dell in 2016, and $3 2 million on ShotSpotter hardware and software in 2017
PAGE 13
OIG FILE #'17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS
In addition, OIG determined that DolT did not consistently collect the information needed to accurately assess and rank proposed projects using cost/benefit and risk criteria.
According to GAO, informed investment decisions are best supported by quantitative data on cost/benefit and risk.1'' DolT did not estimate costs and benefits for any ofthe eight OIG-reviewed projects, and it assessed the risk for only one (by retaining a vendor for that purpose). Although the Department consistently estimated the cost of paying vendors for design and implementation, it did not estimate full lifecycle costs for any project. DolT never considered the cost of internal labor, equipment, or materials. Similarly, the Department estimated ongoing costs to maintain and support a system for only one of the eight projects.15 Figure 5 shows the cost information collected by DolT for each project OIG reviewed. Appendix B has detailed descriptions of the projects.
FIGURE 5: DOIT DID NOT ESTIMATE FULL PROJECT COSTS
( Project Name
|:'311;iMpdernization _ -
: Hyperion Budget System
.......„-_
|?A;rray.:of Things ... :- - . .-, ¦¦¦
Utility Tax
'¦ Chicago Early Learning House Share Registration System (Phase 1) House Share Registration System (Phase 2) Citrix Enterprise Services Upgrade
Source: OIG review of DolT project documentation
Costs '
No .-, No No l,-No
No ' '. No No . No
External : Internal Maintenance
and Support
Partial16
No
No
No
No
No
No
No
"* GAO acknowledges the value of qualitative measurements of benefits, noting "Benefits must be defined and quantitatively and qualitatively measured in outcome-oriented terms" United States Government Accountability Office. "Assessing Risks and Returns A Guide for Evaluating Federal Agencies' 11 Investment Decision-making," February 1997,13, accessed October II, 2019, http //www gao gov/'special pubs/ailO'113 pdf
:r' GAO notes that "the amount of rigor and types of analyses that are conducted will depend, in part, on
the size ofthe investment and the amount of risk " For example, a full cost-benefit analysis may not have
been warranted lor the relatively low-cost Citrix Enterprise Services Upgrade With that in mind, GAO
recommends defining tire level of analysis requited based on project type, cost, and risk
United States Government Accountability Office. "Assessing Risks and Returns A Guide for Evaluating
Federal Agencies' IT Investment Decision-making." February 1937. 42, accessed October 11. 2019,
http //www gao gov/spec ia I nu bs/a. iOil3 pd f
15 The estimate included the cost of maintaining the old 311 system during the transition to tho new system It did not include maintenance or support costs for the new system
PAGE 14
OIG FILE #17-0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 7019
Although DolT's current scoring tool aligns with CAO's recommendation to set predefined selection criteria, it does not require quantified estimates of costs, benefits, or risks. The tool asks reviewers to award up to TOO points across 9 categories One category relates to cost, two relate to benefits, and two relate to risk. However, all could be scored without any quantified estimate. For example, the "Expected Return" category instructs reviewers to award 10 points if they agree that, "the project will result in a product or service that will generate revenue." But reviewers are not required to reference supporting analysis or otherwise justify their score.
The City's unsuccessful effort to implement the Hyperion Budget System illustrates the potential effect of DolT's inconsistent selection process. In 2017, OBM identified funding for a new system, and launched a project to replace the City's budget system with Hyperion. OBM estimated that engaging vendors to design and implement the new system would cost the City $5.7 million. Flowever, neither DolT nor OBM performed a risk assessment or compared the costs and benefits ofthe project to those of other proposed projects prior to selection.
$5.4 million spent
$1.2 million in equipment can be repurposed
$4.2 million lost
In 2019, OBM terminated the Hyperion project after concluding that using the software could result in an incomplete, inaccurate, or unbalanced budget. According to OBM, ofthe $5.4 million spent on the project, just $1.2 million was used to buy equipment that the City can repurpose. Thus, the net loss was $4.2 million.17 In addition, the City now must continue to use its outdated budget
application, which is no longer supported and has limited reporting capabilities. OBM stated that although it initially believed Flyperion would meet its business needs, it discovered during implementation that the software's functionality did not live up to expectations.
As discussed below in Finding 2, terminating a project may be appropriate if it no longer meets business needs, introduces excessive risk, or will exceed tolerable cost thresholds However, when DolT and OBM disagreed about whether to terminate the Hyperion project, OBM declined to meet with all project, stakeholders. As of July 3, 2019, DolT and OBM had not met to diagnose the root cause ofthe project failure OBM stated to OIG that if is not sure whether the loss was avoidable. Rigorous adherence to a consistent selection process may have avoided or mitigated the loss experienced by the City in this instance.
This assessment of the loss does not accouni foi internal City resources expended
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
RECOMMENDATIONS
To improve the project selection process, DolT should'
Require all project managers to follow the PMO Handbook for selection activities. Standardization will promote consistent, repeatable performance of duties. In particular; DolT should require project managers to use predefined criteria to rank all projects before selection.
Develop procedures for collecting more robust cost/benefit and risk data to improve comparisons between potential projects. DolT may choose to base the level of rigor required on the relative cost and risk ofthe project. The Department should work with OBM to budget for projects through their full life cycle, not only year-to-year, and improve its scoring tool by requiring reviewers to provide justifications for their scores.
Work with OBM and the Mayor's Office to ensure that ITGB continues to meet at least quarterly to perform its role in the selection process. Furthermore, all City departments—including CPD, CFD, and OEMC—should be required to submit projects to ITGB for selection.
MANAGEMENT RESPONSE18
7. "The project managers Have been following the PMO Handbook for selection activities since Spring of 2018. In the Spring of 2018, the ITGB established new selection criteria.
2 "The ITGB was re-established in the Spring of 2018. The IT Governance Board (ITGB) approves requests for funding, regardless of source, for new projects and services, additional investments and upgrades in existing products and services, as well as for subsequent phases to previously approved projects. This body ensures that requested investments align to strategies identified by the TSG (Technology/ Strategy Group) and reviews requests to scale successful pilots or modify purchasing-related policies. The ITGB also monitors project health and outcomes on a monthly basis, providing oversight and having the ability to cancel projects Lhat are not meeting established objective outlined in the Cancellation Process section below. The ITGB is comprised of staff from the Mayor's Office, the Office of Budget & Management, DolT and consulted by the Departments of Finance and Procurement Services. The group meets monthly in person and may meet virtually as needed. See Item 1 for the intake form that ask the requestor for cost, benefit, and risk information. Currently the criterion is outlined, and the project managers score the projects based on
Management Response At rachmenK can be found in Appendix C
DACE IS
OIG FILE #17 0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 20"I9
a defined numeric system. The PMO will work with the ITGB to discuss establishing text fields for the project managers to justify their scoring
3. "The ITGB was originally tasked to have quarterly meetings. Beginning March of2019, the ITGB has been meeting monthly due to the magnitude of project requests. Since March of 2019, the only month that a meeting didn't occur was in October, due to budget hearings.
All departments are required to submit their IT requests to the ITGB for selection."
PACE 17
OIG FILE #17-06.58
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19. 2019
Based on our analysis of a sample of six projects completed in 2016 and 2017, OIG found that DolT did not consistently monitor performance during project development and implementation.19 Figure 6 describes each project and provides the original cost estimate. Descriptions of these completed projects are included in Appendix B.
FIGURE 6: OIG REVIEWED SIX PROJECTS COMPLETED IN 2016 AND 2017
1 Project _
eProcurement ¦
Voice over Internet Protocol (Pnase 1) • Utility Tax
House Share'. Registration .:: System (Phase 1) ¦ Paperless . ." ,¦
WindyGrid 2.0
j^DescriptionCC v.
New systerrTto'allowall City departments to j manage^procurement electronically :jg Replace legacy phone system for 3,000 users"
Integrate-water and sewer taxes into water billing system
Create system toTegister shared-housing:. rental units
'New system to manage business license applications.online . F-f System that supports "situational awareness and incident monitoring and response"20
Budget *
$5,67,6,227 :
$3,000,000: $2,100,036 $698,700
%t $690,207 $249,480.
Source: OIG review of DolT project documentation
DolT did not consistently monitor the performance of these six projects. ITIM recommends that organizations monitor whether a project delivers expected benefits on schedule and on budget. In addition, organizations should track the extent to which any risks identified are managed. DolT's PMO Handbook required project managers to monitor compliance with budget and schedule targets and suggested a menu of ways to measure risk mitigation and benefit achievement. DolT tracked schedule adherence for all six projects but assessed only three for whether they stayed within budget. Additionally, the Department did not define performance measures, or monitor benefits delivery or risk management, for any of the six projects
DolT did not have a complete inventory of projects completed in .".'016 and 2017 OIG selected a targeted sample of six projects fiorn among those known to have concluded during this time period
The projoct included a puohc facit :g component, called OpenCno. that made some of WindyGncl's data and functionality publicly available at huos//poena;id 10/
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 7019
we reviewed. Hgure 7 summarizes the number of projects for which DolT completed five core monitoring activities.
FIGURE 7: DOIT DID NOT CONSISTENTLY MONITOR PROJECT PERFORMANCE
Oof 6
had
performance goals and measures
Oof 6
were assessed to determine if expected benefits were delivered
Oof 6
were reviewed to determine whether risks were addressed
3 of 6
were assessed to determine if actual costs matched the budget
6 of G
were assessed to determine if duration matched the schedule
Source OIG review of DolT project documentation
DolT provided reliable data for only one performance measure: schedule adherence. Five of the six projects took longer than scheduled to complete. Two took more than twice as long to complete as originally planned. Figure 8 compares actual duration to original schedule for each project.
FIGURE 8: MOST PROJECTS TOOK LONGER THAN PLANNED TO IMPLEMENT
eProcurcmon
Paperless
VoIP
WindyCiid 20 my
House Share Registration System |^w^>.j.;wi,),:.ti|
¦¦¦¦ J**'
10C
Planned Cuntior
Source OIG review of DolT project documentation
OIG FILE ifl'7 0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS
One ofthe projects that took more than twice as long than originally planned to complete was development and launch of the City's House Share Registration System. The system was needed to administer new registration and licensing requirements for short-term residential rental hosts and platforms, such asAirbnb, passed by City Council on June 22, 2016.-' Even as completed, however, that system did not deliver the full functionality defined in the original project scope. DolT tried to develop a single system by December of 2016 for all companies connecting guests and hosts via internet platforms. This system would have required the companies to send data to the City. According to DolT, one ofthe companies insisted instead that the City retrieve data from its system. To accommodate this company, the City built a custom interface, causing significant project delays. In August 2017, DolT delivered a system capable of registering home shares associated with the company. This partial completion, however, exhausted the funding budgeted for the entire project. As a result, DolT retroactively labeled the project "Phase 1" and issued a new task order request for $495,000 to build custom interfaces for other home sharing companies.
According to ITIM, collecting data on actual performance is critical because it allows
decision makers to consider whether to continue or terminate projects. In addition to
lacking the data needed to make informed decisions, DolT did not have a defined
process for considering whether existing projects were meeting goals, and whether it
should continue to fund the projects, correct the issues impeding progress, or
terminate the projects. The City's Information Technology Governance Policy
appropriately assigns to ITGB the responsibility for monitoring the health and
outcomes of existing projects and authorizes the . .
Board to terminate those "not meeting established objectives." However, at that time, ITGB had neither determined what criteria to use, nor ensured that project managers collect the necessary data to make those decisions.
—xx—
OIG also found that DolT did not consistently collect
information needed to identify and manage project risks. DolT did not ask its IT Architecture Board to review any of the six projects' compliance with the City's IT architecture;,; Omitting this review creates the risk that a project will not be fully compatible with the City's existing architecture or will inhibit efforts to move towards a target architecture in the future. Furthermore, DolT's Information Security Office
" City ofChicago, Business Affairs and Consumer Protection, 'Shared Housing and Accommodations Licensing." accessed October 21. 2019,
htt os //www chi c a o. o ci o v/c i ty/e n/d e pts/bacp/si j ppinfo/sharedhousincirjricJacc:(airio(Jaiionslicensina him I
According to the City's Information Technology Govei nance Policy, the Architecture Board is responsible for establishing standards—the I i architecture—to 'a'ign plaPorms. products, and services" with sti ategic goals
PACE 20
OIG FILE ffl7-OG38
AUDIT OF DOITS MANAGE MEN F OF INFORMATION I ECHNOLOCY INVESTMENTS DECEMBER 19, 2019
(ISO) performed a full security assessment for just one of the six projects we reviewed. This omission introduced the risk of security vulnerabilities going undetected. DolT stated that it has been unable to staff the ISO at the level needed to perform full security assessments for all projects.
RECOMMENDATIONS
To improve monitoring processes, DolT should:
Require that all project managers follow the PMO Handbook, as stated in the previous finding. Managers should,
monitor cost/benefit and risk performance for all projects; and
submit all projects to the Architecture Board and ISO for review.
Update the PMO Handbook and/or the City's IT Governance Policy to define criteria for determining whether to terminate underperforming projects.
Ensure that ITGB continues meeting on at least a quarterly basis and fully inhabits its role of providing project oversight DolT and OBM should work with ITGB to ensure that project managers collect the relevant data to enable ITGB to perform these functions. At a minimum, DolT should provide data related to actual cost/benefit, risk, and schedule performance.
Work with OBM to ensure ISO is adequately staffed.
MANAGEMENT RESPONSE23
"The PMs hove been required to follow the procedures outlined in the PMO Handbook since the Spring of 2018.
As part ofthe project process, the PMs keep track of the project risks. They also monitor the project performance and notify the PMO Director and or/the ClO/CTO/appropriate Program Manager if there is an issue that is impacting the project's performance/budget.
The Architecture Board meets bi-weekly. During this time, the PMs, Program Managers and /or the PMO Director discusses the projects. Often times additional meetings arc held depending on the project. The ISO review was added in the Spring of 2019. The new ISO provided guidelines and a process of when to engage them in projects.
One ofthe outcomes ofthe April 2019 ITGB meeting was to add language to the IT Governance Policy to address termination of underperforming projects. As a result, this language was added and disseminated to the Department
Management. Response Auaehrnents can bo found in Appendix
PACE
OIG hi Lb #17-0633
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
Commissioners ond ITSCs in the updated to IT Governance Policy in June of 2019 This language was also added to the PMO Handbook in April of 2019 and disseminated to ttie PMs. Please see Item 2 for the language that was added to the PMO Handbook and the IT Governance Policy.
"The ITGB was originally tasked to have quarterly meetings. Beginning March of 2019, the ITGB has been meeting monthly due to the magnitude of project requests Since March of 2019, the only month that a meeting didn't occur was in October, due to budget hearings. The ITGB created a report based on the information they want to review for the projects. This report is reviewed at each meeting. Please see Item 3 for the reporting fields.
"Security resource needs have been identified for ISO, and DolT will work with OBM to execute a hiring plan for the targeted resources."
PAGE 22
DolT did not consistently evaluate individual project performance after implementation. Indeed, the Department did not assess vendor performance or document lessons learned for any of the six projects OIG reviewed. Two project managers said that, while they typically discuss lessons learned, those discussions are not memorialized or used to improve project management.
During the audit, DolT collected incomplete project information in SharePoint, which DolT used to facilitate project management.2'' As of June 7, 2018, ofthe 271 projects in SharePoint, 168, or 62%, were missing budget information and 219, or 81%, were missing actual expenditure information/""' The site contains only budget and expenditure information for the current year; this prevents DolT from assessing whether it is meeting budget targets over the life of projects. In addition, the SharePoint site does not include actual end dates, which prevents DolT from ' calculating the extent to which projects finished late. DolT stated that the PMO was not yet fully capable of tracking performance because it lacked the necessary project management software. In 2019, the Department began to implement software to capture more complete performance data.
ITIM recommends assigning personnel to ensure that sufficiently detailed information to support decision making is available, understandable, and utilized by decision makers. DolT management has not assigned a specific individual to fulfill this role.
Because the City lacks information on project performance, it cannot take the next step- evaluation of portfolio-level performance. According to GAO, as organizations mature, they progress from managing individual projects into managing a well-rounded investment portfolio. ITIM states that, "taking a portfolio perspective enables the organization to consider its investments in a comprehensive manner, so that the
SharePoint is a web-based collaboration system used for document storage and organization :'f; A small number of these projects may have just begun and therefore correct ly did nol. reflect any expenditures 105. or 48% ol t he 219 records without any expenditures had been closed, and thus should have included expenditure data
OIG FILE #17-0638
AUDIT Or DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
investments address not. only the strategic goals, objectives, and mission ofthe organization, but also the impact that projects have on one another."20
Because ITGB did not meet in 2016 or 2017, it could not hold DolT accountable for collecting project and portfolio performance data. Moving forward, the Information Technology Governance Policy requires ITGB to engage in monthly project monitoring, but it does not specifically require evaluation of performance after -completion. The Policy also does not address portfolio level evaluation, identify outcomes ITGB would expect DolT to report, or describe how lessons learned should be used to improve future projects.
RECOMMENDATIONS
To improve project evaluation, DolT should:
Work with ITGB to define the processes and criteria for evaluating project and portfolio-level performance.
Fully implement its new project management tool and ensure its staff consistently records the performance data required by ITGB.
Ensure that project managers evaluate individual performance for all projects after implementation, and document their lessons learned.
Assign someone to ensure the information collected meets the needs of ITGB.
MANAGEMENT RESPONSE27
"The PMO has project performance criteria which is outlined in the PMO Handbook. This is established when the project manager is developing the project charter with the project requestor. This criterion is used throughout the lifecycle ofthe project. The PMO will review the criteria with the ITGB. See Item 4 for the criteria.
"The new project management tool was fully implemented in the Spring of 2019. As part ofthe implementation, the ITGB report was re-created. All ofthe data needed for the report is populated by the PMs as a part of their status reporting
"The project managers have been following the PMO Handbook for project close out activities which include lessons learned
¦ United States Government Accountability Office. "Information Technology Investment Management A Framework for Assessing and Improving Process Maturity," March 2004. 63. accessed October 11. 2019. hue//www gao ooy/asseis/80/76790 pdf Management Response Attachments can be found in Appendix C
OIG FILE #'17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
4. "The PMO Director collaborates with the ITGB to ensure the information collected meets their needs."
)
PAGE 25
OIG FILE ttIV-0638
AUDIT OF DOIT'S MANAGEMENT OP INFORMATION T ECHNOLOGY INVESTMENTS DECEMBER 19, 2019
IV. OBJECTIVES, SCOPE, AND METHODOLOGY
OBJECTIVE
The objective of the audit was to determine whether DolT manages information technology investments in accordance with the CAO's Information Technology Investment Management framework We focused on DolT's processes for selecting, monitoring, and evaluating IT projects.
SCOPE
The audit scope included DolT's processes for selecting, monitoring, and evaluating information technology projects that cost at least $250,000. We reviewed projects that were initiated or completed in 2016 and 2017.
METHODOLOGY
To assess DolT's selection, control, and evaluation processes, we first compared its documented policies to the processes identified in ITIM. As needed, we also interviewed DolT staff and asked follow-up questions to clarify our understanding of the policies and procedures.
To further evaluate DolT's project selection process, we examined a targeted sample of eight projects that were launched in 2016 and 2017, and reviewed project documentation to determine whether DolT adhered to its internal processes and best practices as defined in ITIM.-'3
To further evaluate DolT's project monitoring processes, we selected a targeted sample of six projects that were closed in 2016 and 2017, and reviewed project documentation to determine whether DolT adhered to its internal processes and best practices as defined in ITIM/'9 In addition, we assessed project, performance by comparing planned to actual schedules, and budgeted to actual costs. OIG assessed the reliability of DolT's cost numbers by comparing them with reports from the City's financial system, invoices, and other supporting documentation. DolT was unable to identify all invoices for all projects. Therefore, we determined that actual cost data provided by DolT was not reliable for further analysis.
To further evaluate DolT's project evaluation processes, we reviewed documentation for the targeted sample of six completed projects to determine whether DolT had
We limited the number of projects to eight due to tho volume of documentation associated with each project We selected a mix of protects designed to ensure review o( various functional areas within the City, both hardware and software projects, and projects of varying size
We selected the sample using the same criteria used to seleci oui pro/ect selection sample
PACE 26
OIG FILE #17-0633
AUDIT OF DOITS MANAGEMENT OF INFORMATION I ECFINOLOGY INVESTMENTS DECEMBER 19, 2019
assessed its own performance, vendor performance, or documented lessons learned. We also reviewed performance data recorded for all DolT projects to determine if was sufficiently complete to allow DolT to conduct portfolio-level analysis.
STANDARDS
We conducted this audit in accordance with generally accepted Government Auditing Standards issued by the Comptroller General ofthe United States. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
AUTHORITY AND ROLE
The authority to perform this audit is established in the City of Chicago Municipal Code § 2-56-030 which states that OIG has the power and duty to review the programs of City government in order to identify any inefficiencies, waste, and potential for misconduct, and to promote economy, efficiency, effectiveness, and integrity in the administration of City programs and operations.
The role of OIG is to review City operations and make recommendations for improvement.
City management is responsible for establishing and maintaining processes to ensure that City programs operate economically, efficiently, effectively, and with integrity
OIG FILE #17-0633
AUDIT OF DOITS MANAGEMENT OF INFORMATION TFCFINOLOGY INVES1 MENTS DECEMBER 19, 2019
APPENDIX A: IT GOVERNANCE POLICY
The following is the full text of the current City of Chicago Information Technology Governance Policy, last updated June 14/2019.
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
POLICY OVERVIEW
The City ofChicago Information Technology Governance I'olicy ("Policy") establishes a standard citywide process tor requesting, prioritizing, and selecting proposed IT investments. ITgovermmtv provides a framework for aligning Hie Lily's digital strategy with the City's business strategy. By following a formal framework, the City will
Align its investments to citywide strategy and goals, Minimize risk and duplication,
Better track and understand impact of its technology investments, and
Bring value to residents, businesses and visitors.
This Policy requires that all new technology projects and services as well as requests to fund additional investments and upgrades in existing products and services, regardless of the funding source must be submitted to DolT via a central request process and be reviewed and approved by the Information Technology Governance Board (ITGB).
Includes all requests for funding or other resources needed to complete new projects to create a new product or services, and modifications to existing products and services.
Requests must be submitted through the New ITGB Project form and entered into CBS to be considered for new funding on the dale of your department's budget submission is due each year.
In some cases, exception requests will need to be made outside of the budget cycle, ln such cases, the requests will be reviewed on an ad hoc basis with a consideration for available funding.
Small projects that may lie completed with no new funding .met internal resources do not require review and approval by the ITGB.
¦ Projects to be included in grant applications or funded by grants must be submitted to the ITGB. Urban Area Security Initiative (11AS1) Grant Program projects arc exempt from this process and shall be subject to Office of Budget & Management (OBM) review and approval. OBM will provide the ITGB with a list of projects funded by UAS1 as available.
The ITGB will also moniLor project health and outcomes on a moriLhly basis, providing oversight and may cancel projects that are not meeting established objectives. Templates will be provided.
While appropriate governance is needed, it should be aligned to a citywide strategy lhat has been set collaboratively by the organizational leadership. To that end, a new Technology Strategy Group (TSG) will be established. The TSG is chaired by the CIO and comprised of leadership from all City departments who will work to collaboratively set
Last Updated: June 14, 2019 1
QIC FILE #17-0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
citywide digital strategy, and identify opportunities that enalile technologies that deliver community benefit, optimize resources, improve service deliver/, reduce risk, and build capacity (i.e., through automation or business process reengineenng).
This Policy is established by the Office of the Mayor, the Office of Budget and Management (OBM), the Department of Innovation & Technology (DolT), consulted by the Departments of Finance and Procurement Services and will be reviewed at least annually.
List Updated: June 14, 2019 2
OIG FILL SI7-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION "1 ECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
POLICY SCOPE
This Policy applies lo requests lo fund new IT projects and services as well as requests lo fund additional investments and upgrades in existing products and services, regardless of funding source. This policy also applies to requests to leverage internal resources to complete requested work. Requests for maintenance and/or support of existing systems are exempted from this Policy (these cost should he included in the project requests"), but software and hardware maintenance is subject to the Technology Purchase Review and Approval (TPRA) policy.
This Policy does not supplant the City's TPRA policy. All purchases of hardware, peripheral devices, and software must adhere to that policy, including those contemplated for new IT projects covered by this policy. Further, requests made through TPRA must also already have adequate, available and approved funding sourc.e(s) in the fiscal year during which they are requested.
TPRA ensures that purchases are made according lo approved requests for funding. This IT Governance Policy governs how departments will request funding or resources for new IT project requests.
This Policy only applies to the selection of IT investments. Note that IT investments must be further prioritized against other funding requests and operational needs. There is no set amount of available funding for IT projects.
GOVERNANCE BODIES
The Technology Strategy Croup (TSG) will establish the City's digital roadmap, which will be used to inform the ITCB's decisions. The TSG is comprised of leadership from all City departments who will work to collaboratively set citywide digital strategy, and identify technologies that deliver community benefit, optimize resources, improve service delivery, reduce risk, and build capacity. This group will develop and maintain a digital roadmap that includes opportunities to test new technologies through pilots and optimization of business policies or processes to support the roadmap. This group meets quarterly in person or more frequently as needed, particularly during planning cycles.
The IT Architecture Board (Architecture Board) sets enterprise technology standards that align platforms, products, and services to the strategic digital roadmap. This group establishes IT standards and project/product management processes, and reviews details of each implementation to ensure compliance once funded. The Architecture Board is comprised of senior staff, architects, and project managers within DolT. This group meets weekly in person to respond to needs throughout the year.
Last. Updated: June 14, 2019
OIG FILE 417-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS • DECEMBFR 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
The DolT Project Management. Office (PMO) is responsible for reviewing all new project requests anil associated business cases, and integrates the decisions of the TSG and ITGB into new and ongoing programs and projects. The 1'MO may assist in the development of the request's business case.This group is an active member of all other groups to ensure coordination, and meets daily to keep requests moving through the pipeline and manage implementation.
The IT Governance Board (ITGB) approves requests for funding, regardless of source, for new projects and services, additional investments' and upgrades in existing products and services, as well as for subsequent phases to previously approved projects. This body ensures that requested investments align to strategies identified hy the TSG and reviews requests to scale successful pilots or modify purchasing-related policies. The ITGB will also monitor project health and outcomes on a monthly basis, providing oversight and having the ability to cancel projects that are not meeting established objective outlined in the Cancellation Process section below. The ITGB is comprised of staff from the Mayor's Office, the Office of Budget & Management, DolT and consulted by the Departments of Finance and Procurement Services. The group will meet monthly in person and may meet virtually as needed.
REQUEST REQUIREMENTS
All requests to initiate or fund new IT projects or additional investments in existing products and services must be entered via the "New ITGB Projects" form at httDs://chicagoeov.shnrepoint.com /sites /pwa/Lists/NewProiectlntakeForm/ltorn/ newifs.aspx and may be submitted at any time.
Requests must he submitted through the New ITGB Project form and entered into CBS to he considered for new funding on the date of your department's budget submission is due each year to be considered for the next funding cycle in subsequent years to be considered during the budgeting process for the following year, unless the request (its the criteria noted in the Exception Process section below.
Each department must vet and set their own priorities prior to submission of project requests. All requests must be approved by the Department Head—please attach this approval to your request.
Submissions are automatically routed for review. A DolT Project Manager will be assigned lo each request, and will follow-up with the department requester and identified business sponsor (if different). If the requesting department is unsure of costs, or need other assistance with the business case, the DolT Project Manager office will assist.
Once the DolT Project Manager has completed their initial review, the request will be submitted to the Architecture Board and the ITGB (or review.
Last. Updated: June 14, 2019 4
OIG FILT //17-0638
AUDIT Of DOIT'S MANAGEMENT OI~ INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
PRIORITIZATION & SELECTION PROCESS
Each request is initially scored Iry the Dol T PMs according to set criteria, and the PMO may update or override the PM's initial scoring to ensure a balanced portfolio.
, CRITERIA
STRATEGIC ALIGNMENT
STATEMENT
; The project is directly aligned with '. the City's strategic goals (e g., ¦ resiliency plan) or is mandated by ! regulation or ordinance
POINTS POSSIBLE
20
SCORING RUBRIC
0% - OO NOT AGREE
75% - SOMEWHAT AGREE
100% - AGREE
\2 1 ORGANIZATIONAL • SUPPORT & i AVAILABLE > RESOURCES
j The business owner (department ; head or deputy) is the sponsor and committed to providing the right people resources needed to meet project goals., including access to ', front line staff.
! 0% - DO NOT I AGREE
i
j 50% - SOMEWHAT | AGREE
1100% - AGREE
FUNDING/ESTIMATED ; For requests made during budget '' 10
COST & EFFORT
TIME TO COMPLETE : VS EFFORT
I cycle, the project has considered \ known costs within reason For : requests made outside the budget cycle, the score will be based on [ whether sufficient funds are j available to complete project.
10
I The project may be reasonably completed within the timeframe or
1 completior date specified In some instances, projects will be requred to be completed by a certain date to
I an ordinance, in these cases, the
J project may be force ranked if
I needed
0%, - DO NOT, AGREE
50% - SOMEWHAT AGREE
i 100% - AGREE
; 0% DO NOT ' AGREE
; 50% - SOMEWHAT
: AGREE
I
' 100%, ¦ AGREE
; The project will result in a product ¦ or service that will generate : revenue
; 0%, ¦ DO NOT ! AGREE
: 50% • SOMEWHAT
Last. Updated: June 14, 2019
OIG FILE #'17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
6 ORGANIZATIONAL i ; BENEFIT
ORGANIZATIONAL LOSS
18 i ARCHITECTURE & j ! STANDARDS ! | ALIGNMENT
The project provides non-moretary , 10 value to residents, businesses, or the organization. The project solves a real problem for end users The project contemplates engaging actual end users throughout the project to ensure success.
Not selecting this projea will impact ! 10 tho resident, businesses, or the organization negatively (i e., reduce ; revenue, incur additional costs, increase time in line, etc )
: 10
The project could leverage existing tools, systems, or technologies, reducing support costs and complexity ofthe City's enterprise architecture An inventory is j available to ITSCs
AGREE
1100% AGREE
j 0% - DO NOT ! AGREE
I 50% - SOMEWHAT I AGREE
, 100% AGREE
0% - DO NOT AGREE
50%-SOMEWHAT AGREE
•100%-AGREE
j 0% - DO NOT I AGREE
i
j50% SOMEWHAT AGREE
¦ 100% ¦ AGREE
The project is innovative or will improve a process or service delivery.
; 0% DO NOT , AGREE
' 50% SOMEWHAT AGREE
1 100% AGREE
The requests with the highest scores are more likely to be funded or assigned to internal resources to complete. The number of funded or green-lighted projects will vary by year according to the availability of funding and resources.
Steps and associated estimated times to complete follow. Note that less complex and mort complete requests will take less time to review and process, and more complex or less complete requests will require additional time.
,ast Updated: June 14, 2019
OIG FILE 417-0638
AUDIT OF DOIT'S MANAGEMENT! OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
OWNER STEP TARGET TIME TO COMPLETE
DolT - PMO Director Reviews and assign requests 1-3 BLsmess Days
DolT - Project Marager Contacts requester to obtain project details Provides a high level estimate of the project cost for departments budgeting purposes Scores the project based on the citywide IT strategy 1-5 Business Days
DolT-Architecture Board Reviews requests to ensure alignment with the City's policies and standards and determine if this project should use in-house resources or must leverage procurement processes 5-10 Business Days
DolT - Project Manager Contacts requester about next steps 1-3 business days
ITGB Reviews requests for new funding and may override the PMO's initial scoring to ensure a balanced portfolio Notifies project requesters based on results Quarterly, or more frequently curing budget cycle, and as needed in the case of ar. exception request
DolT - Project Maragor Contacts requester about next steps forfunded projects Within 1-3 days of ITGB review
EXCEPTION PROCESSES
On occasion, new projects are required to address an immediate emergency, an unknown and realized risk, or a new requirement resulting from legislation.
Last. Updated: June 14, 2019 7
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
In these cases, the requester should still submit their request via the SharePoint site littps.7/chicagogov.sha repoint.com/sites/cloit/ITProjectManatfL'iiient/.Sit.oPaRL'.s/Ho me.aspx.
The request must include a memo from the Department Head that explains the need for review outside ofthe annual process, the reason the exception is being requested, and when the new project must be completed.
The DolT PMO will follow the same, but an expedited process, and the request will be reviewed by the Architecture Board and ITGB outside ofthe normal annual cycle based on the situation presented by the requester. Approvals from the Architecture Board and ITGB may be done by email in these instances.
If salvage funds are identified that may be applied to requested, unfunded project requests, the department head must.
Notify their Budget Deputy prior to December 1, and request that: the ITGB review their request to reallocate these funds.
If approved by the Budget Deputy, the ITGB will review this request at a year-end ITGB meeting.
NOTE, projects using annually appropriated funds will he subject to the year-end procedures established by the Department of Finance. Expenditure deadlines will not be extended.
PROJECT MONITORING
The DolT PMO also monitors project health [relative to budget burn rate, schedule, quality) on a regular basis and is responsible for reporting this health to the Architecture Board, ITGB, and the TSG.
The ITGB will review project health and its progress against project goals and objectives for high priority and medium and large projects on a monthly basis, providing oversight. The ITGB may cancel projects that arc not meeting established objectives.
CANCELLATION PROCESS
If there is a need to cancel a project, an email notification needs to be sent to the CIO and Budget Director with the following information attached.
Memo from the Department Head that provides a detailed justification for the cancellation request
Business Case, Statement of Work, and/or the Requirements docunient(s) which were approved and signed by your departments' staff
Last Updated: June 14, 2019
OIG FILE ii 17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
CITY OF CHICAGO INFORMATION TECHNOLOGY GOVERNANCE POLICY
All cancellation requests will be reviewed by llie ITGB. If Hie ITGB approves Ihe cancellation, the Department Head will consult with the CPO or her designee about the appropriate communication method. The Department Head will draft a memo to the vendor(s) clearly communicating why the project is being cancelled and requesting a project close out meeting in which the vendor will deliver all project artifacts/code as outlined in the Statement of Work. Finally, the lessons learned form must be completed by the project team.
The ITGB may also suggest lhat the City consider cancelling a project if the project is not meeting the objectives outlined in the statement of work and/or business case. Should this occur, the Department Head(s) will be notified in writing with the rationale and request for additional information. The ITGB will set up one or more meetings with the project team to review the documentation and conduct appropriate due diligence before a decision is made.
Last Updated: June 14, 2019
OIG FILE if'17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENT'S
APPENDIX B: PROJECT DESCRIPTIONS
This appendix summarizes the projects OIG reviewed as described by DolT.
Hyperion Budget System: This project was intended to replace the City's legacy budget system. The City terminated the project in 2019 when it was determined to not meet the needs of OBM
311 Modernization: This project replaced a legacy Motorola system with Salesforce CRM system with the goal of improving departmental workflow tracking and request management and providing additional options for residents to enter and track requests.
Chicago Early Learning Phase 1: "Working with DFSS and CPS, in June 2016 DolT launched a universal early childhood portal designed to be a one-stop shop for all early learning information and enrollment for both school-based and out-of-school programs. Approximately 19,000 applications were collected, and more than 17,000 children were placed in pre-K programs."
Citrix: This project updated the City's Citrix Enterprise Services environment. The City hosts over 20 applications on this environment including applications that support the functions ofthe Department of Buildings, Fleet and Facilities Management, and the Department of Finance.
Array of Things (AoT): "AoT is an urban sensing project, a network of interactive, modular sensor boxes that will be installed around Chicago to collect real-time data on the city's environment, infrastructure, and activity for research and public use. AoT will essentially serve as a 'fitness tracker' for the city, measuring factors that impact livability in Chicago such as climate, air quality, and noise."
House Share Registration System Phase T This project created a system to identify, track and approve (or deny) shared-housing rental units marketed on Airbnb.
House Share Registration System Phase 2: This phase provided additional functionality to the House Share Registration System including accommodating additional companies that connect hosts and guests.
Paperless DolT and Business Affairs and Consumer Protection launched an online business licensing system that automated the process of small business license issuance and renewal.
WindyGrid 2.0: Launched in 2015, WindyGrid 2 0 is an enterprise system that supports Chicago's "situational awareness and incident monitoring and response" DolT developed the system internally using open source software. The project included a public facing component, called OpenGnd, that made some of WindyGrid's data and functionality publicly available.
PACE .37
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
Voice over Internet Protocol (Phase 1): Launched in 2017, Phase 1 replaced 3,000 legacy phones. This is a multi-phase project with a goal of replacing 24,000 phones.
eProcurement: Launched in 2015, this system created an online platform to increase efficiency and transparency in City procurements. This created a single platform for all departments to management procurement opportunities, track vendor and delegate agency payments, and enable the City to decommission the standalone grants system.
Utility Tax: "This project integrated water and sewer taxes into the existing Banner Utility Billing system "
PAGE 38
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
APPENDIX C: MANAGEMENT RESPONSE ATTACHMENTS
This appendix contains the attachments to the Management Response Form submitted by DolT.
Item 1: Project Intake Form
I: thii s new o< modified pi oce;: or scrvsc'
Proposed Project Timeframe:
Projected Stsrt Date '¦
[ JE«|: ^
Projetttd End Date-
; If")
Ttrncfiflme JoiCrficstion: 5
High Level Cost Ertimale:
High Level Cost Breakdown:
Do you have funding?
\ Funding Scrip ¦
rf io, funding Type:
*f srant, zttach the -nnt swird document'1
Cin The -rsrt cri c=ts go to the n«Vt grant'
Probability funding from your deparrrrtent
th^ r-pact t":i
A;:-Jn-.ptiefii
^-•y:K:';coi:icsiedt3 b
tr".t: jfi.s:* Dependency
C&nstisjnt
: t3.-r'i;r p---ci-5iT
Ajdier.ee/Er.d-Uiers Inis'fn'-r'Oisri
Procursmenr Method C-f 'ire-1
fi this mandatory*
o-Ci-;tc! driver ;nc
Why is th« mandator^7 fktur.i On Luiestment
Stilus (Complete fcy ?M0<
OIG FILE 1/17-0638
AUDIT OF DOITS MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
Item 2: Cancellation Process
If there is a need to cancel a project, an email notification needs to be sent to the CIO and Budget Director with the following information attached:
Memo from the Department Head that provides a detailed justification for the cancellation request
Business Case, Statement of Work, and/or the Requirements document(s) which were approved and signed by your departments' staff
All cancellation requests will be reviewed by the ITGB. If the ITGB approves the cancellation, the Department Head will consult with the CPO or her designee about the appropriate communication method. The Department Head will draft a memo to the vendor(s) clearly communicating why the project is being cancelled and requesting a project close out meeting in which the vendor will deliver all project artifacts/code as outlined in the Statement of Work. Finally, the lessons learned form must be completed by the project team.
The ITGB may also suggest that the City consider cancelling a project if the project is not meeting the objectives outlined in the statement of work and/or business case. Should this occur, the Department Head(s) will be notified in writing with the rationale and request for additional information The ITGB will set up one or more meetings with the project team to review the documentation and conduct appropriate due diligence before a decision is made.
PACE 40
OIG FILE #17-0638
AUDIT OF DOI I'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
Item 3: ITGB Report
Below are the reporting fields based on the information the ITGB wants to review
Project Number Project Name URL
Department Executive Sponsor
Description of Problem or Need Project Percent Completed Open/Closed Project Manager Project Manager Notes Funding Source Amount Encumbered Expended YTD Funds Still Available Project Finish Date
PACE 41
OIG FILE #17-0638
AUDIT OF DOIT'S MANAGEMENT OF INFORMATION TECHNOLOGY INVESTMENTS DECEMBER 19, 2019
Item 4: Project Performance Criteria
I he following is the project performance criteria used1 by the PMO. Execution Phase Project Performance Metrics
Percentage on time
Percentage on budget
Percentage on scope
Percentage on quality
Percentage of deliverables on schedule
Budget versus forecast
Number of requirements changes/Total number of requirements
Outstanding issues/Total issues
Risks .mitigated/Total risks
Percentage of user base trained
Client satisfaction
Number of help desk calls related to project
Number of vendor performance issues
Time to fulfill project change requests
Percentage of project resources devoted to reusable component development
Time to fix detected problems 1
Percentage compliance with architecture standards
MISSION
The City of Chicago Office of Inspector General (OIG) is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of City government. OIG achieves this mission through,
administrative and criminal investigations by its Investigations Section,
performance audits of City programs and operations by its Audit and Program Review Section;
inspections, evaluations and reviews of City police and police accountability programs, operations, and policies by its Public Safety Section; and
compliance audit and monitoring of City hiring and employment activities by its Hiring Oversight Unit.
From these activities, OIG issues reports of findings and disciplinary and other recommendations,
to assure that City officials, employees, and vendors are held accountable for violations of laws and policies;
to improve the efficiency and cost-effectiveness of government operations; and
to prevent, detect, identify, expose, and eliminate waste, inefficiency, misconduct, fraud, corruption, and abuse of public authority and resources.
AUTHORITY
OIG's authority to produce reports of its findings and recommendations is established in the City of Chicago Municipal Code §§ 2-56-030(d), -035(c), -110, -230, and 240.
Cover image courtesy of /Stock