One of the interesting tasks today has been a cost-benefit analysis for a policy evaluation. Policy design is interesting on all sorts of fronts. At the moment (2013) , much policy design is done without much formal understanding of how different factors shape the dynamics of outcomes.
In this sort of environment, everyone wants 'value for money' and so 'cost-benefit' analysis becomes central to the design of future policies .
Typically most new policies are based on identifying what went wrong with previous policies and trying to fix them. Much policy design seems to work like this:
- Policy designers and sponsors have subjective opinions about how factors cause particular outcomes
- Post-facto evaluation of the previous policies is used to identify what went wrong them and needs fixing
- The costs of the different parts of the previous policy implementation are calculated along with some assessment of the benefits that seem to have resulted from the policies
- Policy designers and sponsors use the evaluation and cost-benefit analysis to underpin their judgments about what to change to ensure: the chosen outcomes occur, benefits improve overall, and implementation is budgeted correctly
This kind of iterative design is reasonable in situations that straightforward rather than complex and the money flows are tightly linked to the policy project (implementing the policy comes out of a single bucket of cash that can be accounted).
Multi-agency policy design
Policy design intended to achieve outcomes that result from integrated activities of multiple agencies presents a different set of difficulties.
Multi-agency policy design typically has each agency contribute only part of their work to the integrated policy project. Commonly, however, such agencies are also operating in the same arena as the intended policy intervention.
This presents several problems in evaluation terms:
- It can be difficult to identify allocate the amount each agency's activities in the integrated intervention contribute to outcomes
- Agencies may also incidentally contribute to outcomes via their other outputs and work outside of the policy intervention
- Project output evaluation data may be shaped more by agency operational factors than the intervention context and this can be compounded in a multi-agency arrangement
- Delays in agency processes can compound in multi-agency policy interventions
- Feedback loops between agencies and between agencies and contextual factors can dominate output and outcome trajectories
- Power and communication issues can dominate multi-agency functioning.
- Multi-agency policy interventions require distributed data collection for evaluation
- Evaluation data collection and collation can be compromised by local sub-optimization by individual agencies
Role of Program Logic models
Appropriate Program Logic models can keep the relationships and processes clear and transparent. Cost benefit analyses can then be based on the structures of the program logic models.
Evaluating policy designs in complex multi-agency interventions - using systems models
Evaluation of multi-agency policy interventions in situations that that are intrinsically complex, and involve feedback and delays between causal factors, is not easily addressed by conventional cost-benefit analysis, cost-effectiveness analysis, or even cost analysis. The above difficulties are compounded by the confounding consequences of dynamic interactions between causal factors that shape outcomes. At this stage, it appears the only way forward is to ground cost analyses on a dynamic causal model of the policy intervention.