Applying Best Practice in Project Evaluation

Over a three year period SkillsTrain (the South-East Media Training Consortium) undertook an ambitious £600,000 sector development project, with support from the South-East Economic Development Agency and the European Social Fund. The project comprised four linked strands encompassing:

  • Development of a sector skills strategy
  • Consultancy and training support for businesses
  • Career development support for individuals
  • Development of an informational web-site resource

As independent consultants, we were asked to provide an evaluation of the overall project and also separately to evaluate the web-site strand. The scope of this work was to assess the quality of the project activities and outputs, using the UKRP Good Practice Guide and the SEEDA Project Evaluation Toolkit as benchmarks. Within this remit, we were asked to address issues of:

  • Impact
  • Value for money
  • Value to the industry
  • Operating context
  • Successes
  • Opportunities for improvement

We also sought to identify learning outcomes and were asked to make recommendations for taking the project forward.
In order to address these issues, we adopted a two-tier approach comprising both validation and evaluation. We first confirmed the extent to which the project’s intended objectives, outputs and impacts, as set out in its original terms of reference, had been achieved. We then measured the project’s achievements and processes against current industry best practice.
In order to validate the overall project’s achievements, we examined project documentation including:

  • Contracts
  • Terms of reference
  • Reports
  • Stakeholder correspondence

We also examined proprietary technology and other intellectual property that was developed as part of the web-site strand.
We then designed surveys and conducted over thirty hours of telephone and face-to-face interviews with twenty-six stakeholders including:

  • SkillsTrain board members
  • Funders
  • Participants
  • Contractors
  • Customers
  • Others affected by the project activities

In order to evaluate value for money and value to the industry, we obtained industry sector reports from third parties inside and outside the South-East UK area, consulted independent industry sector specialists who were involved in relevant projects beyond the South-East, and drew upon relevant research data from sources including:

  • Government departments
  • Public sector bodies
  • Industry institutes and societies

Finally, we submitted our findings in two written reports. The first presented these in respect of the web site strand. The second reported our evaluation of the overall project.

We found that both the web-site strand and the overall project were broadly successful in achieving their intended objectives, and represented a foundation from which significant prospective benefits to the industry could be derived. In particular, the web-site strand had delivered substantial intellectual property that could be further capitalised upon. We recommended that the project be taken forward.

We were also able to highlight learning points that will assist in increasing the project’s impact, value for money and value to the industry. We identified a number of activities to increase support for the sector, in particular relating to project planning, resourcing, implementation and evaluation. We further identified thirty improvements to the design and delivery of all four strands so that further publicly funded work can apply best practice and yield a high return on investment. A key finding was that more structured analysis of risks and options in the planning stage would ensure future projects could be designed, managed and delivered with minimum risk and maximum return.

In undertaking this evaluation, HirschWorks worked in partnership with Strategic Management Partners. Together, we were able to draw on over twenty years’ working experience in the media sector, encompassing strategy, implementation, management consultancy, standards and best practice, as well as involvement in a range of industry forums and initiatives. This has allowed us to widen our sector-practitioner perspective and offer a unique ‘outside view’.

“SkillsTrain are to be commended for investing in a rigorous evaluation based on a holistic approach. This enabled us to make benchmark comparisons outside the region and include best practice from other relevant professional bodies.” – Clive Bonny, Strategic Management Partners.

“This evaluation demonstrates how industry stakeholders can successfully work together to benefit the sector. We are very pleased to have been able to assist SEEDA in identifying the value of their investment and in deciding how to take the project forward.” – Jonathan Hirsch, HirschWorks.

Posted in Research and Evaluation