Program Evaluation Basics

What is program evaluation?

This phase involves tracking the progress of the program and making any necessary adjustments to keep the program on track. This includes monitoring budgets, schedules, and other metrics to ensure the program is meeting its goals.

Program execution and evaluation are closely related. They could be seen as one phase, but it is worth calling out the need to make space for evaluation.

Inputs and outputs

Let’s look at the inputs and outputs of this phase.

Inputs:

• Program goals from the charter

• Program progress from the plan

Outputs:

• Improvements on the program objectives or program plan

• Improvements on how we operate the program

• Confirmation that the program team is on the right track

• Communication to stakeholders

What are we evaluating?

There are multiple aspects of the program that you may want to evaluate. Ultimately, you want to get a feel for whether or not the investment into the program is returning enough value to justify continuing the program.

There are two main aspects to consider:

• Outcomes: These are usually lagging indicators of success. A lagging indicator of success is a metric that measures the outcome of a particular process or activity after it has occurred. Lagging indicators provide a historical view of performance and are used to measure the results of past actions or decisions.

• Outputs: These are leading indicators of success. A leading indicator of success is a metric that measures the progress or potential of a particular process or activity before it has occurred. Leading indicators provide a forward-looking view of performance and are used to identify potential issues and opportunities before they occur.

Leading and lagging indicators are often used in conjunction with each other to give both a future-facing, probabilistic view of success and a historical look back at the impact of work.

Outcomes

Outcomes are the intended impact from successful program execution.

Outcomes are typically tied to your initial key results outlined in your OKRs from the original program planning.

For example, the company might want to expand into new revenue streams. Your program is meant to build out one of those new revenue streams. In this case, the outcome would be increased revenue.

You would evaluate whether or not this new stream of revenue is generating what it was expected to.

In most companies, program outcomes are owned by the program sponsor, product management, or engineering. You support achieving these outcomes through driving successful outputs.

Outputs

Outputs are a function of how efficient the program is being run to drive the outcomes.

If you've got poor program outputs, you will likely not reach your desired outcomes.

For example, one output is delivering all requirements for a given milestone. If your program is having a hard time delivering milestones on time and within scope, that directly impacts how effectively your program can reach the program outcomes.

If your program is delivering all the necessary outputs, probability is high that the outcomes will follow.

However, a program that is delivering on all outputs but outcomes aren't being achieved is also very informative. It could mean that we need to go back to the drawing board with some of the initial requirements and pivot to a new strategy.

Your primary role as a technical program manager is to deliver on program outputs so that program outcomes are highly probable.

Gathering data

You should work with your partner teams, especially product management and engineering, to gather data on program implementation success.

Take time to evaluate success measured on both functional requirement delivery and nonfunctional requirement delivery.

For example:

• Nonfunctional success: Is latency within the acceptable range as described in the nonfunctional requirements?

• Nonfunctional success: Have we properly implemented data privacy protections for GDPR laws and regulations?

• Functional success: Did we ship that major feature on time? Did we have to compromise on scope in order to ship that feature on time?

• Functional success: Are we seeing an increase in user traffic utilizing this new feature?

Each of these questions should be supported by and primarily answered with data. This is what helps create trust across the program! The success is transparent and objective, not somebody's opinion of how well its going or how disastrous it is.

How often should we evaluate?

Let’s look at how often the program should be evaluated to ensure program success.

Continual evaluation

Waiting long periods of time to evaluate program success is the wrong way to do things. Program evaluation is not so much an event as it is a part of the program culture. This should be your baseline for performing program evaluation.

Previously, we discussed the necessity of program team meetings and status reports to all stakeholders. Through these methods, you'll drive regular program evaluation. The intent with continual evaluation is to prevent any majorly disruptive or surprising shifts in the program lifecycle. You can't completely avoid unexpected events impacting the program, but you can do quite a bit to lower that probability.

Scheduled evaluation

On a regular cadence, perhaps quarterly, you should hold a special meeting with primary program contributors to evaluate the progress of the program.

The primary discussion should be focused on progress towards the program OKRs.

Take a look at both outputs and outcomes. Outputs are definitely measurable at this stage of the program. Outcomes may or may not be measurable yet.

Come prepared with data from your partner teams. One of the major goals here is not to reveal new information to everyone, but rather to align all the teams on the multiple data sources supporting the evaluation.

What happens after evaluation?

There are typically only two options after program evaluation:

1. You loop back to program planning or execution with better information and the ability to course-correct if necessary. You would loop all the way back to planning if the program is off course and there is a need to completely stop to reevaluate. In most cases, program evaluation is a basic extension of program execution.

2. You close out the program because it has reached its desired objectives, or it is confirmed that it will not reach those objective