E333EE
Table of Contents
11 Executive Summary|910|A | Conclusion|910|B | Finding|910|C | Recommendations|910|D | DFSS Response|910|| Background|910|A | The Commitment to Outcomes|910|B | Strategic Contracting Process|910|| Finding and Recommendations|910|Finding: DFSS Largely Applies Its Strategic Contracting Process in Accordance With the
Commitment to Outcomes, but There Is Room For Improvement|910|A | DFSS Generally Aligned RFPs With the Commitment to Outcomes, but Did Not
Include Some Key Elements in Certain Instances|910|B | DFSS Generally Aligned Its Tools for Evaluating RFP Applications With the
Commitment to Outcomes, but 13 of the 22 Questions Were Missing From 1 or
More Tools 10
C | Application Evaluators Inconsistently Applied Scoring Guidance 12
D | Most DFSS Contracts Were Consistent With the Corresponding RFPs, but Key
Elements Were Missing From Some 13
IV | Objective, Scope, and Methodology 17
A | Objective 17
B | Scope : 17
C | Methodology 17
D | Standards' 17
E | Authority and Role 18
Appendix A: Management Response 19
Acronyms
DFSS Department of Family and Support Services
OIG Office of Inspector General
RFP Request for Proposal
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
City of Chicago Office of Inspector General
D0f
The Department of Family and Support Services' (DFSS) Strategic Contracting process largely aligns with the Commitment to Outcomes—its internal Framework for achieving outcomes-based goals—but DFSS could strengthen the process to ensure that the framework is consistently applied.
DFSS' Request for Proposal (RFP) templates incorporate 13 key elements to help ensure that RFPs and their resulting programs align with the Commitment to Outcomes. DFSS included 7 of these elements in all 31 RFPs released between January 2020 and April 2021. It included another 3 elements in at least 90% of RFPs and the remaining 3 elements in less than 90% of RFPs.
DFSS generally aligned its evaluation
tools with the Commitment to Outcomes but 13 of 22 key questions were missing from 1 or more tools. These included the number of clients a program intended to serve, descriptions of the target population, and how the delegate agency would identify, recruit, and retain that population.
Application evaluators inconsistently
applied scoring guidance, which could hamper DFSS' ability to select quality delegate agencies in a fair and efficient manner.
Audit of DI-'SS' Strategic Contracting
City of Chicago Office of Inspector General
I | Executive Summary
The Office of Inspector General (OIG) conducted an audit of the Department of Family and Support Services' (DFSS) Strategic Contracting process for selecting delegate agencies. The objective of the audit was to determine whether DFSS' contracting processes align with outcomes-based goals, and also in line with the Department's Commitment to Outcomes, of which its Strategic Contracting process is a part.
A | Conclusion
OIG concluded that DFSS' Strategic Planning and Impact division's involvement in developing requests for proposal (RFPs) and evaluation tools—critical steps in the Strategic Contracting process—helps align those steps with the Commitment to Outcomes. However, OIG also determined that there is room for improvement. The division could provide more guidance for the evaluation of RFP applications. In addition, RFPs, evaluation tools, and contracts could be improved if the division ensured the inclusion of the Commitment to Outcomes' key elements.
B | Finding
DFSS developed RFPs, tools for evaluating RFP applications, and contracts that largely align with the Commitment to Outcomes, but the Department could strengthen its process by ensuring the inclusion of key elements that match the Commitment to Outcomes. Moreover, RFP application evaluators inconsistently applied scoring guidance.
C I Recommendations
OIG recommends that DFSS develop procedures to ensure that it includes the key elements of the Commitment to Outcomes in all future RFPs, evaluation tools, and contracts, and that evaluators consistently score applications according to the Department's scoring guidance. OIG also recommends that DFSS ensure that all divisions share an understanding of outcomes-based goals, outcome metrics, and the key elements of the Commitment to Outcomes. To assist in fulfilling these recommendations, the Strategic Planning and Impact division could provide additional guidance across the phases of the Strategic Contracting process.
D | DFSS Response
In response to OIG's audit findings and recommendations, DFSS stated that it would continue to provide guidance and training on the Commitment to Outcomes to program divisions through the Strategic Planning and Impact division. The Department will also continue to provide guidance and procedures to help divisions include relevant and appropriate elements in RFPs, evaluation tools, and related contracts. Finally, DFSS stated that it will improve the scoring guidance it provides to evaluators.
The specific recommendations related to the finding, and DFSS' response, are described in the "Finding and Recommendations" section of this report.
Audit of DI-'SS' Strategic Contracting
Cily of Chicago Office- of Inspector General
II | Background
The Department of Family and Support Services (DFSS) assists Chicagoans in need to resources covering a variety of categories, including senior health and wellness, housing, youth mentoring, and early childhood education. DFSS "works to promote the independence and well-being of individuals, support families and strengthen neighborhoods by providing direct assistance and administering resources to a network of community-based organizations, social service providers and institutions."1 DFSS enlists the services of organizations through requests for proposal (RFPs) in a competitive process. The Department evaluates applications to each RFP, selects delegate agencies for each program, and awards payment for their services through a contract. As of January 2022, the Department reported working with 350 such delegate agencies under 1,600 contracts, with a total annual community investment of S346 million. DFSS' services and programs include the following:
Children Services - provides children of all ages with access to early learning programs.
Community Service Centers - assists individuals and families with shelter, food, clothing, job training, scholarships for higher education, and other services.
Division on Domestic Violence - operates a 24-hour, toll-free, and confidential help line and provides counseling, legal, and advocacy services.
Senior Services - provides information and connections to assisted living, caregiving, dining programs, insurance counseling, and various other services.
Services for People Experiencing Homelessness - provides short-term financial assistance for rent and utilities, performs outreach services, funds shelters, and connects clients to services for behavioral health, substance abuse, and more.
Veterans Resources - supports veterans through programs including housing, entrepreneurship, employment, education, legal assistance, and health care.
Workforce Development and Ex-Offender Programs - provides resources to various workforce development initiatives to support disadvantaged Chicago residents and ex-offenders transitioning back into the workforce.
Youth Services - supports youth with enrichment activities after school, on weekends, and during school breaks and works with other government institutions, community-based organizations, and employers to offer employment and internship opportunities.
A | The Commitment to Outcomes
In 2016, DFSS launched the Commitment to Outcomes to clearly describe, measure, and report on the outcomes which the Department wants to achieve through its social service programs. Developed with partners and stakeholders, including the Civic Consulting Alliance and Harvard Kennedy School Government Performance Lab, DFSS describes the Commitment to Outcomes as an "outcome-oriented model that focuses on how many people leave better off after receiving DFSS' services, versus how-many people come through the door.";: The Commitment to Outcomes' ultimate goal is to achieve better results for vulnerable Chicagoans by refocusmg services on
' City of Chicago. Department of Family and Support Services, "Our Structure," accessed January VI, 2022. https //www. Chicago qov/citv/en/depts/fss/auto qenerated/fss our structure html.
Cily of Chicago. Department of Family and Support Services, "Commitment to Outcomes," accessed April 4, 2022. tit!RsV/www chicaqo.qov/citv/en/depts/fss/suop info/deoartmeni-strategic-frainework.html
Audit of DFSS' Strategic Contracting
Source' DFSS Commitment to Outcomes Overview.
Phase 1 of the Commitment to Outcomes, the Strategic Framework, "consists of a refreshed mission, priorities, and goals, along with a plan for how DFSS will measure, report on, and review them in the years to come; use them to make decisions; and drive greater collaboration within DFSS."3 Rather than short-term strategies, the Strategic Framework seeks to establish a long-term foundation for orienting DFSS' operations and decision-making toward clients' outcomes. The refreshed mission states, "Working with community partners, [DFSS] connect[s] Chicago residents and families to resources that build stability, support their well-being, and empower them to thrive."'1 In December 2017, DFSS reported that it had implemented Phase 1 across its program divisions, setting division-level outcome goals that aligned with the new Department-level mission and priorities.
Phase 2 of the Commitment to Outcomes, Strategic Contracting, works to ensure that DFSS' RFPs and contracts reflect the outcomes it seeks. The Department uses RFPs to solicit proposals from potential delegate agencies to provide direct services. Under Strategic Contracting, DFSS' RFPs ask applicants to identify target populations and their needs, evidence-based solutions, and outcomes-based success measures. DFSS updates its evaluation tools to aid the selection of
:i City of Chicago, Department of Family and Support Services. "Department Strategic Framework," June 2016, accessed November 17, 2021, httos7/www Chicago qov/content/dam/citv/depts/fss/supp info/Commiiment roOutcomes/ DFSSStrateqicFramework.pdf.
J Department of Family and Support Services, "Department Strategic Framework "
Audit of DFSS' Strategic Contracting Page 5
Ci ly of Chicago Office of Inspector General
delegate agencies that reflect its results-driven approach.b DFSS has implemented Phase 2 to varying degrees in each division. The Department stated that factors affecting implementation include grant cycles, staff capacity, data access, the willingness of divisions to adopt changes, and delegate agencies' capacities for change.
Phase 3 of the Commitment to Outcomes, Performance Improvement, seeks to identify priorities, performance improvement strategies, and resources in order to demonstrate progress toward outcomes. Divisions may apply various improvement strategies such as target population analysis, best practice research, and active contract management based on the nature of programs and availability of resources. DFSS initially planned to implement Phase 3 by June 2020. However, the Department stated that implementation had been delayed due to the global COVID-19 pandemic and the resource-intensive nature of making department-wide changes and actively managing contracts.15 DFSS is continuing the process of implementing Performance Improvement across its divisions.
DFSS is still developing Phase 4 of the Commitment to Outcomes. In Phase 4, the Department plans to implement a system-wide database that will make it easier to connect clients with services that meet their needs.
DFSS' Strategic Planning and Impact division leads the implementation of each of these phases. According to the Department, the Commitment to Outcomes will mean structural and cultural changes within both DFSS' program divisions and its delegate agencies. The Commitment to Outcomes was designed to be implemented over multiple years to allow time to make these substantial changes.
B | Strategic Contracting Process
All DFSS program divisions have begun Phase 2 of the Commitment to Outcomes, Strategic Contracting. As shown in Figure 2, this process begins when division staff draft an RFP.
Figure 2: The Strategic Contracting Process Begins With RFP Development and Fnds With an Executed Contract
Source' OIG visualization of information provided by DFSS
;' City of Chicago. Department of Family and Support Services, "DFSS's Commitment to Outcomes." January 2022, accessed February 14, 2022, htlps //www Chicago gov/conteni/dam/ciiv/depts/fss/suop info/CommitmentToOutcomes/ DFSSCommitmentloOutcomesOverview.pdf.
0 Department of family and Support Services, "DFSS's Commitment to Outcomes "
Audit of DFSS' Strategic' Contracting
ms^masa City ol Chicago Office: of Inspector General
The Strategic Contracting process involves the following steps:
DFSS' Strategic Planning and Impact division and its program divisions collaborate on RFP templates, which the program divisions use to develop each RFP. These templates contain 13 Strategic Contracting elements that are designed to encourage results-driven contracting. For example, RFPs should describe clear outcomes-based goals for their programs, as well as performance metrics to measure their success. With guidance from the Strategic Planning and Impact division, program divisions also develop evaluation tools for each RFP. These include application questions asking potential delegate agencies to demonstrate how they would achieve outcome goals described in the RFPs. Another tool includes guidance instructing evaluators to score applicants on their ability to achieve those goals. The Strategic Planning and Impact division encourages program staff to write each RFP and utilize the associated evaluation tools at the same time to ensure the questions are clearly linked to RFP content.
DFSS releases RFPs to the City's eProcurement website, inviting potential delegate agencies to submit proposals.7
Once the application period is over, evaluators score the applications based on the scoring guidance. Two evaluators score each application, and the scores are compared for consistency. If the scores differ by a pre-determined margin, a third evaluator scores the proposal. Although evaluators are usually program staff, DFSS sometimes solicits external help from universities and research institutes that possess relevant knowledge and experience. The Department will not assign more than one external evaluator to any one application. All evaluators complete Conflict of Interest forms and receive training on scoring. Management and program staff decide which applicants will receive grant awards based on their final scores, ability to serve targeted communities, available funding, and staff diversity and qualifications.
DFSS sends award letters and notifies the applicants who were not selected. Once a delegate agency accepts the award, program divisions collaborate with the Contracts and Finance division to finalize the contract, which is based on a boilerplate from the City of Chicago's Department of Law.
7 City of Chicago, Department of Procurement Services. "eProcurement," accessed February 23, 2022, https //www. Chicago qov/citv/en/depts/dps/provdrs/cprocurcment html
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
III | Finding and Recommendations
DFSS' Strategic Contracting process helps ensure that its programs align with the Commitment to Outcomes. The Strategic Planning and Impact division guided the creation of templates and guidance to craft RFPs, evaluation tools, and contract sections on scope of services. These were key to aligning programs with outcomes-oriented objectives. While DFSS included most of the elements needed to align these resources with the Commitment to Outcomes, some were missing key elements. This increases the risk that programs will not deliver DFSS' intended outcomes. The Strategic Planning and Impact division also developed written guidance for scoring RFP applications. In some instances, however, evaluators applied this scoring guidance inconsistently. This could leave DFSS unable to substantiate the fairness of its scoring and created the risk that the Department would select delegate agencies that could not actually deliver the desired program outcomes.
A | DFSS Generally Aligned RFPs With the Commitment to Outcomes, but Did Not Include Some Key Elements in Certain Instances
DFSS' RFP templates incorporate 13 key elements to help ensure that RFPs and their resulting programs align with the Commitment to Outcomes.8 As Figure 3 shows, DFSS included 7 of these elements in all 31 RFPs released between January 2020 and April 2021. It included another 3 elements in at least 90% of RFPs and the remaining 3 elements in less than 90% of RFPs.
" OIG reviewed DFSS' templates, RFPs, and other program documentation and identified the questions and pieces of information that relate and help align programs to the Commitment to Outcomes, referred to here as key elements I hose are as comprehensive as possible as relates to the Commitment to Outcomes, but omit unrelated items such as legal and administrative requirements OIG shared and discussed its results with DFSS throughout the audit, and removed some items from their initial consideration as key elements as appropriate. See Methodology section IV C
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
Figure 3: DFSS Included Over Half of the 13 Key Elements in All RFPs, but Did Not Include All Elements in Each RFP"
Element Percentage of RFPs with element present
How DFSS wants the program to improve Core outcome metrics Problem statement
Clear outcome-based goals of the program
Current state of the program
Core output metrics
Description of outcomes vs. outputs
Division priorities
Clear target population
Scope of services
Desired delegate competencies
Data reporting and performance requirements
Mention of pre-proposal webinar
Source: OIG analysis of DFSS RFPs released between January 2020 and April 2021.
Some of the missing elements would have enabled DFSS to maximize its collection of information on program outcomes. Notably, in some RFPs the Department listed output metrics—which measure actions that may contribute to what a program intends to achieve—instead of outcome metrics, which, measure what the program actually achieves. This suggests room for'improvement in the development of outcome metrics. For example, as an outcome metric for the Case Advocacy and Support for Vulnerable Older Adults program, DFSS listed a goal for the amount of time it should take for 80% of the target population to be served. While this is an important metric, it does not address whether the population served was better off after having received the service.
In addition, DFSS included some outcome metrics that did not fully reflect outcome goals outlined in the RFPs. For example, the 2021-2022 Resource and Information Advocacy Services for Victims of Domestic Violence program's goals were to increase both the safety of domestic violence survivors and their confidence in navigating and understanding the legal system. However, the performance measures for the program only address survivors' understanding of the legal system, not their safety or feelings of safety after participating in the program (as might be collected in a survey, for example) Without clear and complete outcome metrics in place, DFSS cannot measure the success of a program in terms of whether those it served were made better off.
'•' These results account for the fact that not all elements are applicable to every program. For example, questions about the current state of the program were not considered missing from new programs
Audit of DFSS' Strategic Contracting
City of Chicago Office: of Inspector (General
B | DFSS Generally Aligned Its Tools for Evaluating RFP Applications With the Commitment to Outcomes, but 13 of the 22 Questions Were Missing From 1 or More Tools
DFSS anticipates that its evaluation tool template will help its program divisions craft RFP application questions and scoring guidance. According to DFSS' program documents, the questions should reflect the Commitment to Outcomes and help applicants understand the goals and desired outcomes of the RFP. Potential delegate agencies apply for RFPs via the City's online procurement system, eProcurement. The application questions are also incorporated into evaluation tools along with scoring guidance for evaluators. The scoring guidance helps evaluators determine if a potential delegate agency's application aligns with the Commitment to Outcomes. This allows DFSS to select agencies capable of delivering the proposed services and achieving the desired outcomes.
The Strategic Planning and Impact division guided the Department's program divisions to include 22 questions in the evaluation tools and to add program-specific questions as needed. The questions were sorted into four sections:10
Strength of the Proposed Program
Performance Management and Outcomes
Organizational Capacity
Reasonable Cost, Budget Justification, and Leverage of Funds
As Figure 4 shows, the evaluation tools used to evaluate the 31 RFPs released from January 2020 to April 2021 generally aligned with the Commitment to Outcomes. However, some did not include questions addressing certain elements that reflect the Commitment to Outcomes.
;" DFSS stated that it has since added a fifth section on Community Involvement, containing equity-focused questions. Tools containing this section fell outside of this audit's time scope.
Audit of DFSS' Strategic Contracting
mttavr.rxerwmw.mn CilV l)f Chk'itgO Ol'ficC of lUSpCCtO!' G
Figure 4: Many Evaluation Tools Did Not Include Questions About the Number of Clients and the Proposed Services, As Well As Descriptions of the Target Population
% of Evaluation Tools with Question Present
Strength of Proposed Program
The numberof clients Proposed services Target population
Identification, recruitment, and retention of target population
Client-driven services
Evidence-based/best practices
Target population's needs and challenges
Coordination efforts
61% 68% 71% 77% 84% 94% 94% 94%
Performance Management
Data collection and storage capacity Past performance
Performance monitoring and management Identifying areas for improvement
97% 100%
jaikfj 100% ¦.Iii 100%
Organizational Capacity
Community engagement/DEI HR capacity
Appropriate personnel for oversight and management Expenditure monitoring and fiscal controls Organizational expertise for target population
94% 97%
•-.y;fcfV. 1 100%
100%
till 100%
Budget Justification
Funding from other government entities Matching funds/in-kind contributions Internal audit process Financial capacity Reasonable cost justification
90% 94% 100% ' ¦ I 100% 100%
Source- OIG analysis of DFSS evaluation tools used for RFPs released between January 2020 and April 2021.
Figure 4 shows that 13 of the 22 questions in the evaluation tool template were missing from one or more evaluation tools. Questions in the Strength of Proposed Program section were most likely to be missing. These included the number of clients a program intended to serve, descriptions of the target population, and how the delegate agency would identify, recruit, and retain that population. Additionally, nearly one-third of the questions about proposed services did not ask the potential delegate agency to tie those services to the outcome goals of the program, even though the scoring guidance asked evaluators to "award points for tying activities to the outcome goals of the RFP." Without making this inquiry, DFSS may not be able to determine whether delegate agencies are prepared to execute programs as intended. Specifically, such questions are necessary to clarify how an agency intends to achieve outcome goals through its proposed services.
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
C I Application Evaluators Inconsistently Applied Scoring Guidance
DFSS provides written guidance on using its evaluation tools to score applications. The guidance is unique to each evaluation tool and helps evaluators assign scores based on the potential delegate agencies' responses to each RFP question.
Evaluators did not always evaluate applications from potential delegate agencies according to this scoring guidance. For example,'35 evaluations collectively contained 81 instances where the evaluator provided no justification for deducting points. Additionally, in 34 evaluations the evaluators awarded points even where the applicant did not provide the required elements. Finally, although scoring is supposed to be independent to decrease bias, in two of the evaluations an evaluator appears to have copied comments and scores from other evaluators. Figure 5 summarizes the frequency of these various issues.
81 instances in 3S evaluations
78 instances in 16 evaluations
60 instances in ^^evaluations-'
20 instances in 18 evaluations
11 instances in 10 evaluations ,
9 instances in 8 evaluations
4 instances in^2 evaluations 3 instances in 3 evaluations
Figure 5: Evaluators Did Not Consistently Adhere to the Scoring Guidance
Type of Scoring Issue Frequency of Scoring Issue
¦Evaluator deducted pointswithout-providing justification: Deduction followed
scoring guidance, although the amount deducted did not. r > '
Evaluator did not complete scores and/or comments on evaluation worksheet.
Evaluator awarded points although applicant did not provide re'quired;Jfe;;"j.'
C|cments. ;'='/. • " ^ ¦
^valuator deducted points due to the lack of required elements but elements gwere present.
Evaluator deducted points due to the lack*of certain elements that were not- .
required by scoring guidance. : '; 5j£vs ' ;
Evaluator deducted points without providing justification. Deduction did not follow scoring guidance.
Evaluator copied another evaluators' comments or used another evaluators'
completed evaluation tools. . . '^'.^?'fc '5if^;
iEvaluator/tool inaccurately calculated total score.
Source: OIG analysis of completed evaluations.
In the above instances, the scoring guidance provided clear instructions which not all evaluators followed closely. In other instances, guidance may not have been clear enough for evaluators to apply consistently, leading to varying interpretations and inconsistent scoring. Some scoring issues—caused by both inconsistencies between scorers given the same guidance, and scorers failing to closely follow the guidance—are to be expected given the subjective nature of evaluations. Too many such issues, however, increase the risk of DFSS selecting delegate agencies that will not achieve program goals, or being unable to demonstrate the fairness of scoring decisions. Working towards more consistent and complete use of scoring guidance will help the Department achieve its goal of selecting quality delegate agencies for its outcomes-based programs in a fair and efficient manner.
Audit of DI-'SS' Strategic Contracting
»^«>i»ji City of Chicago Office of Inspector General
D | Most DFSS Contracts Were Consistent With the Corresponding RFPs, but Key Elements Were Missing From Some
Contracts form the agreements DFSS enters into with delegate agencies to pursue the program goals the Department outlined in its RFP. Including key elements from the RFPs in contracts helps DFSS advance its strategic priorities and orient delegate agencies towards clear outcome goals. In 16 contracts related to the 3 RFPs released in 2020, DFSS included most, but not all, of the key elements As Figure 6 shows, some contracts were missing outcomes-based goals of the program and data reporting and performance requirements.
Figure 6: Contracts Did Not Always Reflect the Key Elements of the RFPs
Outcome-based goals of the program
Data reporting and performance requirements
Target population and their needs :
Scope of services
Core outcome metrics
Core output metrics
Centralized Shelter Intake and Transportation (2 contracts)
Senior Legal Services (1 contract)
of
Service Coordination and Navigation for Youth (13 contracts)
J
Source: OIG analysis of DFSS contracts.
The Commitment to Outcomes guides DFSS to center its program divisions' outcomes-based goals throughout the entire Strategic Contracting process. If a contract does not contain specific information that matches the associated RFP, DFSS may not be able to ensure that delegate agencies achieve the intended outcome goals for the program.
Overall, DFSS has improved its alignment with the Commitment to Outcomes by providing training and guidance on RFP and evaluation tool development. However, program divisions may still lack a consistent understanding of the outcomes-based goals, metrics, and data requirements that are key elements of Strategic Contracting and the Commitment to Outcomes. Although DFSS' Strategic Planning and Impact division is heavily involved in developing RFPs and evaluation tools, it provides less guidance on application scoring and contract scope of services development. Inconsistent implementation of the key elements of the Commitment to Outcomes may lead DFSS to select delegate agencies that do not achieve the outcomes-based goals DFSS desires.
| Recommendations
1. DFSS should develop procedures to ensure that its program divisions understand the outcomes-based goals, outcome metrics, and data requirements that are key elements of the Commitment to Outcomes. DFSS should also ensure these procedures are implemented on an ongoing basis, to ensure understanding is maintained as time passes and personnel change.
2 DFSS should develop.procedures to ensure that its program divisions include all key elements of the Commitment to Outcomes in future RFPs and tools used to evaluate RFP applications.
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
3. DFSS should develop procedures to ensure that evaluators score applications according to the scoring guidance and include written justifications for their scores.
4 DFSS should develop procedures to ensure that divisions include all key RFP elements in program contracts.
Management Response11
1. "DFSS agrees that the Commitment to Outcomes contains elements that require ongoing training and guidance from the Strategic Planning and Impact Division to program divisions. The Strategic Planning and Impact Division has and will continue to provide ongoing training and guidance to DFSS program divisions as part of its core function and daily work
"DFSS strives to include performance metrics in its RFPs that best reflect outcome-based goals (i.e., 'how Chicagoans leave better off after receiving services'). However, the process for selecting performance metrics includes many considerations, including the nature of the program (i.e., are services point-in-time and/or transactional?), timeframe of the contract (i.e., will a particular impact be measurable within the timeframe?), availability of data (i.e., are health and/or criminal justice data required and accessible?), and population (i.e., should clients be identifiable in the data collected?). In some cases, DFSS must select metrics that may be more output-oriented due to one or more of these considerations.
"In referencing the Resource and Information Advocacy Services for Victims of Domestic Violence ('RIA') program, the OIG states: 'However, the performance measures for the program only address survivors' understanding of the legal system, not their safety or feelings of safety after participating in the program.'
"RIA provides on-site services to victims of domestic violence upon entering the courthouse building - it is an entry-point to ongoing engagement and services The program is designed as a point-in-time (i.e., 15-30 minutes), transactional, and place-based intervention that focuses on navigation within the courthouse (e.g., locating the correct hearing room, language translation, assistance obtaining and completing forms) and referrals to additional, external services
"In the case of survivors of domestic violence, DFSS does not collect client-level data (names or other identifiable information) that would be required to link to other databases (e.g., law enforcement, court documents, HIPAA) and potentially measure personal safety, as suggested by the OIG report. In addition, in the case of RIA, as a point-in-time, transactional program, DFSS would not expect a change in 'feelings of safety' potentially identifiable through a survey at the courthouse (survey-based measures of 'feelings of safety' are incorporated in other DFSS programs focused on victims of domestic violence, such as legal advocacy and counseling services).
"DFSS bases this decision in an ethical and victim-centered service approach, which is standard practice across the domestic violence services community. The metrics and
" The Department's full response is included in Appendix A
Audit of DFSS' Strategic Contracting
of Chicago Office of Inspector General
survey tools implemented by DFSS were developed in collaboration with the field of domestic violence services subject matter experts, including victims and practitioners. DFSS will not follow the recommendations of the Office of Inspector General auditors in this report over the field of domestic violence services practitioners and victims who inform program and performance measure design.12
"In referencing the Case Advocacy and Support for Vulnerable .Older Adults (CAS) program, the OIG report states that it '. does not address the more important question of whether the population served was better off after having received the service'.13 The CAS program is similarly designed as a point-in-time, crisis mitigation program that focuses on responding quickly, stabilizing the client, and providing them with connections to longer-term services with other agencies. The program is a triage mechanism to determine and access next steps for clients. The timeframe for the intervention does not lend itself to long-term outcome metrics and DFSS does not have access to the kinds of HIPAA-protected health data that would be required to inform any such long-term outcome metrics.
"While these program models are the focus of DFSS' response because they were referenced in the OIG report, there are other program models for which outcome-related metric also are not feasible or appropriate."
2. "The Commitment to Outcomes (CTO) was launched as a strategic framework in response to internal feedback from internal staff and external partners who identified concerns in the areas of impact measurement, decision-making, and coordination. It was developed and implemented in-house as a long-term change management initiative consisting of guidance, templates, tools, processes, and trainings for staff that are designed and implemented by the Strategic Planning and Impact Division. In essence, DFSS has, over time, set a series of organizational 'stretch'goals' for itself.
"DFSS is committed to including all relevant and appropriate elements in its program RFPs and RFP evaluation tools and will continue to ensure that procedures and guidance are in place to enable program divisions to make those determinations and include a comprehensive set of relevant and appropriate elements in program RFPs and evaluation tools.
"The CTO is referred to as a strategic framework, and the tools are referred to as guidance and templates, for a reason - they represent a theory of practice that is flexible and adaptive to the needs of each program, by design. Components, or key elements (e.g., evaluation questions and criteria), are intended to be adopted, adapted, or removed at the discretion of the subject matter experts, and this is clearly stated in the template materials.
"To 'ensure that its program divisions include all key elements of the Commitment to Outcomes in future RFPs and tools used to evaluate RFP applications' (emphasis added),
"¦' As noted on page 9 of this report. DFSS defined the safety of domestic violence survivors as a program goal but did not identify outcome measures related to that goal OIG's analysis is based on ihe DfSS-dcfmed goal and DFSS program documentation.
"' OIG's analysis was based on the outcome goals DFSS defined for the programs Rather than asserting that outcome metrics were inappropriate for this program, DFSS completed the section with output metrics
Audit of DI-'SS' Strategic Contracting
City of Chicago Office of Inspector General
as the OIG recommends within this report, is not aligned with this approach, and does not allow for the nuances of each program model.
"f-or example, two of the elements cited in the OIG report - 'How DFSS wants the program to improve' and 'Current state of the program' - would not be appropriate if a program is new.'' "'
"As another example, including the 'number of clients' as a question in the evaluation tool, as suggested in the OIG report, presumes that there is a directional correlation between the number of clients served and points awarded. '¦' In some cases, the number of clients served is prescribed, while in others, a larger client load would in fact decrease program effectiveness. In these cases, the number of clients would be collected for informational purposes, but not included as a question in the evaluation and given a value as part of the score."
"DFSS is exploring strategies for requiring that justifications be completed by evaluators. DFSS will also continue to improve the guidance provided to evaluators in the scoring rubrics "
"The Commitment to Outcomes is referred to as a strategic framevjork, and the tools are referred to as guidance and templates, for a reason - they represent a theory of practice that is flexible and adaptive to the needs of each program, by design. Components, or key elements are intended to be adopted, adapted, or removed at the discretion of the subject matter experts, and this is clearly stated in the template materials.
"To 'ensure that its program divisions include all key elements in program contracts' (emphasis added), as the OIG recommends within this report, is not aligned with this approach, and does not allow for the nuances of each program model.16
"However, DFSS is committed to including all relevant and appropriate elements in its program contracts and will continue to ensure that procedures and guidance are in place to confirm that the relevant and appropriate elements included in a given RFP are included in related contract."
'• As noted in footnote 9, on page 9 of this report, OIG analysis accounted for the fact thai not all elements were applicable to every program, including new programs
OIG assessed whether DFSS included a question regarding the number of clients based on DFSS' guidance within the evaluation tool
"; As noted in footnote 8, on page 8 of this report, OIG refers to key elements as the questions and pieces of information that relate and help align programs to the Commitment to Outcomes
Audit of DFSS' Strategic Contracting
Cily of Chicago Office of Inspector General
IV I Objective, Scope, and Methodology
A | Objective
The objective of the audit was to determine if DFSS' Strategic Contracting process for selecting delegate agencies aligns with the Commitment to Outcomes.
B | Scope
This audit's scope included all 31 RFPs DFSS released between January 2020 and April 2021 and the associated evaluation tools. It also included a sample of 29 applications in response to the 38 RFPs DFSS released in 2020, chosen randomly with a 95% confidence level and 10% margin of error, as well as all 61 of the evaluations conducted on those 29 applications. Finally, it included all 16 contracts executed for RFPs released in 2020.
C | Methodology
To determine if DFSS' RFPs, evaluation tools, and contracts aligned with the Commitment to Outcomes, OIG developed rubrics containing the Commitment to Outcomes elements found in the Department's templates, as well as guidance for those documents. Using DFSS' RFP templates and associated guidance, OIG identified 13 pieces of information, referred to as elements, that help align each RFP with the Commitment to Outcomes. These include, for example, a statement of the problem the RFP intends to address and core outcome metrics for the delegate agency. OIG's analysis excluded items not directly related to the Commitment to Outcomes, such as legal and administrative requirements. We then reviewed whether each RFP included these elements.
Using DFSS' evaluation tool template and associated guidance, OIG identified 22 questions, referred to as elements, that help align evaluations with the Commitment to Outcomes. These questions are asked of applicants, and their responses are scored on each evaluation tool. These include, for example, the applicant's strategy for identifying, recruiting, and retaining the program's target population, and its capacity for data collection and storage. OIG reviewed whether each evaluation tool included these elements.
Using DFSS' RFPs, OIG identified six key pieces of information, referred to as elements, that help align contracts to the Commitment to Outcomes, and then reviewed whether each contract contained these elements.
To determine if DFSS' evaluation of RFP applications followed its evaluation tools and scoring guidance, OIG examined a random sample of 29 RFP applications and 61 completed evaluations of those applications. OIG compared potential delegate agencies' applications to the scores they received, and reviewed the justifications for those scores provided by DFSS evaluators to determine if evaluators had followed DFSS' scoring guidance on each evaluation. OIG did not make determinations on each evaluators' judgments, only whether they had followed explicit guidance.
D | Standards
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient,
Audit of DFSS' Strategic Contracting
City of Chicago Of lieu of Inspector General
appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis lor our findings and conclusions based on our audit objectives.
E | Authority and Role
The authority to perform this audit is established in the City of Chicago Municipal Code § 2-56-030 which states that OIG has the power and duty to review the programs of City government in order to identify any inefficiencies, waste, and potential for misconduct, and to promote economy, efficiency, effectiveness, and integrity in the administration of City programs and operations.
The role of OIG is to review City operations and make recommendations for improvement.
City management are responsible for establishing and maintaining processes to ensure that City programs operate economically, efficiently, effectively, and with integrity.
Audit of DI-'SS Strategic Contracting
i,,,.,,,...,,,,,^ ¦„.^.i...»J..numtjummikunm City of Cliicago Office ol' Inspector General
Appendix A: Management Response
DFSS management response on: June 2022. Audit of the Department oj Family and Support Services' Strategic Contracting Draft. City of Chicago Office of Inspector General.
DFSS agrees with the OIG recommendation that the Strategic Planning and Impact division "could provide additional guidance across the phases of the Strategic Contracting process". The Commitment to Outcomes (CTO), of which the Strategic Contracting process is one part, was developed and implemented as a department-wide, continuous improvement process focused on outcomes-based impact measurement, data-informed decision-making, and service coordination. Indeed, one of the Strategic Planning and Impact division's primary functions is to identify the need for, develop, and provide guidance, templates, tools, processes, and trainings for staff. DFSS also agrees that additional measures should be developed "to ensure that evaluators score applications according to the scoring guidance and include written justifications for their scores". DFSS is exploring strategies that will ensure that evaluators complete justifications in the evaluation tool.
However, DFSS has fundamental disagreements with the OIGs recommendations related to. 1) the inclusion of "all key elements" of the Commitment to Outcomes and, 2) the use of "clear and complete outcome metrics" across all program model requests for proposals (RFPs) and contracts.
The CTO is referred to as a strategic framework, and the tools are referred to as guidance and templates, for a reason - they represent a theory of practice that is flexible and adaptive to the needs of each program, by design. Components, or key elements (e.g., evaluation questions and criteria), are intended to be adopted, adapted, or removed at the discretion of the subject matter experts, and this is clearly stated in the template materials. To "ensure that its program divisions include all key elements" (emphasis added), as the OIG recommends within this report, is not aligned with this approach and does not allow for the nuances of each program model. DFSS responses within the report provide more details on this matter.
DFSS strives to include performance metrics in its RFPs that best reflect outcome-based goals (i.e., "how Chicagoans leave better off after receiving services"). However, the process for selecting performance metrics includes many considerations, including the nature of the program (i e , are services point-in-time and/or transactional7), timeframe of the contract (i.e., will a particular impact be measurable within the timeframe7), availability of data (i.e., are health and/or criminal justice data required and accessible7), and [population (i.e., should clients be identifiable in the data collected7). In some cases, DFSS must select metrics that may be more output-oriented due to one or more of these considerations.
One RFP that the OIG cited, the Resource and Information Advocacy Services for Victims of Domestic Violence, is an example where outcome-based goals are not appropriate. A point-in-time (15-30 minute interaction), courthouse-based navigation program for victims of domestic violence could not possibly be a model where "clear and complete outcome metrics" could be established. The recommendation from the OIG that outcome-based goals be included in all RFPs is not something that DFSS will be able to implement across all program models, but will continue to implement where appropriate. DFSS responses within the report provide more details on this matter.
Audit of DFSS1 Strategic Contracting
City of Chicago Office of Inspector General
OFFICE OF INSPECTOR GENERAL City of Chicago
D'.-btvjh Wit: tun & insaccicr General
Management Response Form
Project Title. Audit of the Department of Family and Support Services' Strategic Project Number. #20-1629
Contracting
Department Name. Department of Family and Support Services Date July 20,2022
Department Head: Brandie Knazze
OIG Recommondation Agree/.'-.' Disagree ¦' Deportment's Proposw Action^.'. Implementation Target Date Party , Responsible^
1 DFSS .should develop procedures to ensi re that, its program division.-) unders/.and the outccmes-based goals, ou'.come nemos, ard data require-rents that :;rti key elements or the Commitment to Outccnes DFSS thaulJ also ensue Lhet>e orocerlures ."ire :r"p;eme->red on ar ongoing basis, to ensure understanding is mcntohc-ri as time posses and personref change Agree in part/ Disagree in part DFSS agrees that the Commitment to Outcomes contains elements that require ongoing training and guidance from the Strategic Planning and Impact Division to program divisions The Strategic Planning and Impact Division has and will continue to provide ongoing training and guidance to DFSS program divisions as part of Us core function and daily work. DFSS strives to include performance metrics in its RFPs that best reflect outcome-based goals {i e , "how Chicagoans leave better off after receiving seivices") However, the process for selecting performance metrics includes many considerations, including the nature of the program (i e., are services poinl-in-time and/or transactional?), timeframe of the contract (i.e., will a Ongoing OFSS Strategic Planning and Impact Division
Page lojO
Audit of DFSS' Strategic Contracting
City of Chicago Off ice of Inspector General
OIG Recommendation Agree/ Disagree Deportment's Piopojod Action implementation Target Date Party Responsible
¦ particular impact be measurable within the timeframe5), availability of data (i e , are health and/or (rtrninat justice data required and accessible5), and population (i e., should clients be identifiable in the data collected'). In some cases, DFSS must select metrics th.it may be moie output-oriented due to one or more of these considerations. In refencing the Resource and Information Advocacy Services for Victims of Domestic Violence ("RIA") program, the OIG states: "However, the performance measures for the program only address survivors' understanding of the legal system, not their safety or feelings of safety after participating in the program." RIA provides on-site services to victims of domestic violence upon entering the courthouse building - it is an entry-point to ongoing engagement and services. The program is designed as a point-in-time (i.e., 15-30 minutes), transactional, and place-based intervention that focuses on navigation within the courthouse (e.g., locating the correct hearing room, language translation, assistance obtaining and completing forms) and referrals to additional, external services In the case of survivors of domestic violence, DFSS does not collect client-level data (names or other identifiable information) that would be required to link to other databases (e.g, law enforcement, court documents, HIPAA) and potentially measure personal safety, as suggested by the OIG report In addition, in the case of RIA, as a point-in-time, transactional program, DFSS would not expect a change in "feelings of safety" potentially
Page 2 of 6
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
OIG RcKommendatiort Agroo/ D»ag ro9 Department's Proposed Action Implementation Target Date Party Responsible
identifiable through a survey at the courthouse (survey-based measures of "feelings of safety" are incorporated in other DFSS programs focused on victims of domestic violence, such as legal advocacy and counseling services). DFSS bases this decision in an ethical and victim-centered service approach, which is standard practice across the domestic violence services community. The metrics and survey tools implemented by DFSS were developed in collaboration with the field of domestic violence seivices subject matter experts, including victims and practitioners DFSS will not follow the recommendations of the Office of Inspector General auditors m this report over the field of domestic violence services practitioners and victims who inform program and performance measure design In referencing the Case Advocacy and Support for Vulnerable Older Adults (CAS) program, the OIG report states that it "...does not address the more important question of whether the population served was better off after having received the service" The CAS program is similarly designed as a point-in-time, crisis mitigation piogram that focuses on responding quickly, stabilizing the client, and providing them with connections to longer-term services with other agencies. The program is a triage mechanism to determine and access next steps for clients. ¦ The timeframe for the intervention does not lend itself to long-term outcome metrics and DFSS does not have access to the kinds of HIPAA-protected health data that would be required to inform any such long-term outcome metrics. While these program models are the focus of DFSS' response because they were referenced tn the OIG report.
Page 3 of 6
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
OIG Recommendation Agree/ Disagree Department's Proposed Action Implementation Target Date Party Res ponsible
thsre are other program models fur which outcome-related metric also are not feasible or appropriate.
1. DFSS should develop procedures to ensure that its program divisions include all key elements cf the Commitment to Outcomes m future RFPs and tools used to evaluate RFP applications Disagree The Commitment to Outcomes (CTO) was launched as a strategic framework in response to internal feedback from internal staff and external partners who identified concerns in the areas of impact measurement, decisionmaking, and coordination It was developed and implemented in-house as a long-term change management initiative consisting of guidance, templates, tools, processes, and trainings for staff that are designed and implemented by the Strategic Planning and Impact Division. In essence, DFSS lias, over time, set a series of organizational "stretch goals" for itself. DFSS is committed to including all relevant and appropriate elements in its program RFPs and RFP evaluation tools and will continue to ensure that procedures and guidance are in place to enable program divisions to make those determinations and include a comprehensive set of relevant and appropriate elements in program RFPs and evaluation tools The CTO is referred to as a strategic framework, and the tools are referred to as guidance and templates, for a reason - they represent a theory of practice that is flexible and adaptive to the needs of each program, by design. Components, or key elements (e.g., evaluation questions and criteria), are intended to be adopted, adapted, or removed at the discretion of the subject matter experts, and this is clearly stated in the template materials. N/A N/A
Page 4 of 6
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
OIG Recommendation Agree/ Disagree Dopartmont's Proposed Action Implementation Target Date Party Roiponsible
To "ensure that its program divisions include all key-elements of the Commitment to Outcomes in future RFPs. and tools used to evaluate RFP applications" (emphasis added), as the OIG recommends within this report, is not aligned with this approach, and does not allow for the nuances of each program model For example, two of the elements cited in the OIG report -"How DFSS wants the program to improve" and "Current state of the program" - would not be appropriate if a program is new. As another example, including the "number of clients" as a question in the evaluation tool, as suggested in the OIG report, presumes that there is a directional correlation between the number of clients served and points awarded. In some cases, the number of clients served is prescribed, while in others, a larger client load would in fact decrease program effectiveness. In these cases, the number of clients would be collected for informational purposes, but not included as a question in the evaluation and given a value as part of the score.
3. DFSS should develop procedures to ensure that evaluators score applications according to the scoring guidance and include written justifications for their scores. Agree DFSS is exploring strategies for requiring that justifications be completed by evaluators. DFSS will also continue to improve the guidance provided to evaluators m the scoring rubrics. January 2023 DFSS Strategic Planning and Impact. Division
Page 5 oj 6
Audit of DFSS' Strategic Contracting
City of Chicago Office of Inspector General
OIG Recommondalion Agroo/ Disagroa Department's Proposed Action Implementation Target Date Party Respom tble
-1 UrS;-
Page 6 oj 6
Audit of l)l'SS: Strategic Contracting
Darwyn Jones
Deputy Inspector General, Audit and Program Review
Zach McNealy
Performance Analyst
Jisung Shin
Senior Performance Analyst .
Benjamin Spies
Chief Performance Analyst
The City of Chicago Office of Inspector General is an independent, nonpartisan oversight agency whose mission is to promote economy, efficiency, effectiveness, and integrity in the administration of programs and operations of city government.
OIG's authority to produce reports of its findings and recommendations is established in the City of Chicago Municipal Code §§ 2-56-030(d), -035(c), -110, -230, and -240. For further information about this report, please contact the City of Chicago Office of Inspector General, 740 N. Sedgwick Ave., Suite 200, Chicago, IL 60654, or visit our website at iqchicago.org .
Suggest Ways to Improve City Government:
igchicago.org/contact-us/help-improve-citv-qovernment
Report Fraud, Waste, and Abuse in City Programs:
Call OIG's complaint hotline at (866) 448-4754 / TTY: (773) 478-2066 iqchicacio.org/contact-us/report-fraud-waste-abuse/
Cover photo courtesy of the Department of Assets, Information and Services. Alternate formats available upon request.
August 2022
OIG File #20-1629
Why We Did This Audit
DFSS works with approximately 350 delegate agencies and has issued 1,600 contracts, with a total annual community investment of $346 million, to provide social services for vulnerable Chicagoans. The performance of delegate agencies is key to the success of each program. We sought to determine whether DFSS' processes for selecting delegate agencies align with outcomes-based goals, in line with its Commitment to Outcomes.
Background
DFSS adopted the Commitment to Outcomes to transition to an "outcome-oriented model that focuses on how many people leave better off after receiving DFSS' services, versus how many people come through the door."
DFSS defined a new mission, and set new priorities and goals, in Phase 1 of the Commitment to Outcomes— the Strategic Framework. The Department developed results-driven requests for proposal and contracts in Phase 2 of the Commitment to Outcomes— Strategic Contracting. This audit focused on the selection of delegate agencies through Strategic Contracting.
We thank DFSS staff and management for their cooperation during the audit.
Audit of the Department of Family and Support Services' Strategic Contracting
OIG concluded that the Department of Family and Support Services' (DFSS) Strategic Planning and Impact division's involvement in developing requests for proposal (RFPs) and evaluation tools—critical steps in the Strategic Contracting process—helps align those steps with the Department's Commitment to Outcomes. The Strategic Planning and Impact division could provide more guidance for the evaluation of RFP applications. In addition, RFPs, evaluation tools, and contracts could be improved if the division ensured the inclusion of the Commitment to Outcomes' key elements.
Finding
DFSS developed RFPs, tools for evaluating RFP applications, and contracts that largely align with the Commitment to Outcomes, but it could strengthen its process by ensuring the inclusion of key elements that match the Commitment to Outcomes. Moreover, RFP application evaluators inconsistently applied scoring guidance.
Recommendations
OIG recommends that DFSS develop procedures to ensure that it includes key elements of the Commitment to Outcomes in all future RFPs, evaluation tools, and contracts, and that evaluators consistently score applications according to the Department's scoring guidance. OIG also recommends that DFSS ensure that all divisions share an understanding of outcomes-based goals, outcome metrics, and the key elements of the Commitment to Outcomes. To assist in fulfilling these recommendations, the Strategic Planning and Impact division could provide additional guidance across the phases of the Strategic Contracting process.
Department Response
In response, DFSS stated that it would continue to provide guidance and training on the Commitment to Outcomes to program divisions through the Strategic Planning and Impact division. The department will also provide guidance and procedures to help divisions include relevant and appropriate elements in RFPs, evaluation tools, and related contracts. Finally, DFSS stated it will improve the scoring guidance it provides to evaluators.
City of Chicago
Office of Inspector General
iachicaqo.org