The NEXES Action aims to assess the impact of the adoption of Internet-enabled technologies by emergency services throughout Europe and even beyond Europe. Emergency services are quite different, due to their location and organisation.
How to assess any impact given very different subjects?
NEXES is a Research and Innovation Action about NEXt generation Emergency Services. Emergency services comprise fire brigades, police organisations, ambulance services and possibly other search and rescue organisations: services provided by the government for the protection, safety and security of its citizens. Emergency services situated in a city have goals such as an ambulance must arrive at the incident location within 10 minutes of calling. Emergency services situated in the countryside may have different goals, perhaps an arrival time of 20 minutes is required. Each country, within Europe and beyond, has a similar, yet different culture and organisation of its emergency services.
Emergency services that adopt new communication capabilities change. Part of the puzzle is being able to compare a ‘previous’ and ‘current’ emergency service. This difference within emergency services are a challenge for NEXES. NEXES strives to provide a set of key performance indicators – that is, evaluations of the performance – of emergency services that are about the results of adopting Internet-enabled technologies. These key performance indicators must assess ‘the impact’ of using the new technological possibilities.
And that’s where the challenge lies. How to assess the impact of the new technological possibilities, if the ‘subjects’ are similar, yet different? How to assess ‘before’ and ‘after’ the adoption of new technologies, and evaluate the gains? And how can key performance indicators help?
Key Performance Indicators (KPIs) are a well-known instrument to assess organisational performance. NEXES investigates the use of KPIs to assess the impact of its beyond state-of-the-art solutions for emergency services in different countries. Another issue with KPIs became apparent when exploring the application of KPIs within NEXES. Can we use one set of KPIs for all emergency services? This makes sense: by defining one (albeit) large set of KPIs, it becomes possible to assess the impact of the new Internet-enabled technologies, compare the new performances of emergency services and ultimately give compliments to emergency services that implement these new technologies.
However, soon it became apparent that the ‘one size fits all’ approach was not deemed useful at all by emergency services. As a result, the “one-size-fits-all” approach to KPIs was abandoned. A creative solution is needed, one that is fair, respectful and motivating.
The standard structure of one KPI is to measure a single effect accompanied by a certain target value, e.g. “arrival time of first responders at incident location should be less than 10 minutes”, or “average call waiting time should be less than 10 seconds”. The situation unfolds that traditional KPIs are not easily applied to multiple, heterogeneous organisations. Emergency services are organised differently and each has their own plan for implementation of NEXES solutions. So, how can a KPI be defined more flexibly so that it can accommodate the differences between emergency services?
NEXES solution lies in decoupling effects from KPIs, by describing a common denominator in the KPI which is to be specified in detail in the effect measurements by the single, different parties. A flexible KPI is in practice based on multiple effect measurements by the partaking organisations. And the relevant effect measurements can be tailored to their specific situation.
Within the NEXES Action, we used this idea to define flexible NEXES KPIs. First, a set of 11 KPIs is defined and grouped into categories ranging from Emergency Calls to the Coordination of Response Activities and Customer Satisfaction, where each KPI is mandatory. That means each emergency service must have a score for all KPIs. A KPI is not anymore defined as a single effect, rather each flexible KPI is dependent on multiple effect measurements, currently 58. The score of a KPI can be calculated by aggregating the scores for the relevant and tailored effect measurements.
The approach chosen by NEXES is thus to guarantee the autonomy of emergency services. Here, ‘autonomy’ refers to the ability that an emergency service governs itself, makes its own plans, and decides when and how to adopt specific NEXES solutions. Any approach that limits the autonomy of emergency services is destined to fail.
The autonomy of emergency services is guaranteed in two ways. First, by requiring any emergency service to select those effect measurements that are relevant to their situation. Second, by allowing an emergency service to tailor the relevant effect measurements to their specific situation. Furthermore, NEXES requires that all of these decisions and adaptations are made visible. Having these aspects visible makes it possible to compare emergency services’ adoption of NEXES solutions: the original reason for introducing KPIs.
Within NEXES, an effect measurement has a structured description. An emergency service is invited to ‘fill in the blanks’, thereby tailoring the effect measurement to their situation. For example, consider effect measurement #31, which measures Incident Location Accuracy. This effect measurement measures the discrepancy between the incident location determined by the emergency services’ operators versus the actual location of the incident. It is expected that using GPS-based location information from callers, the incident location becomes known more accurately than using the cellular base-stations triangulation algorithms, especially in more rural areas.
Furthermore, emergency services can specify how important certain effect measurements are when calculating their KPIs’ scores. The mechanism provided for this, is that the score of an effect measurement gets an associated importance before aggregation into the KPI score.
For each emergency service, we cannot ‘equalise’ the measurement methods: that would bring us back onto the “one size fits all” path. The measurement methods of the emergency services will be assumed to be different, although these may be similar. Based on the resulting measured values, the evaluation by emergency services is tailored to their own needs. This tailoring is what we can re-use: we can use the tailoring of any one emergency services and apply it to itself (e.g. after a change) or another emergency service (e.g., out of curiosity). So we can compare the KPIs’ scores from the perspective of any emergency service by applying tailoring to many emergency services including itself after a change. It also becomes possible to re-compute the scores for all emergency services, given new insights in e.g. the relative importance of specific measurement methods, or using similar reference values, or any other reason.
The flexibility of the KPIs and effect measurements has shown to be beneficial (1) to guarantee autonomy of emergency services in the application and tailoring of the KPIs, (2) to enable structured comparison of results of the before and after situation of an emergency service, and (3) to foster exploration and understanding of the NEXES impact by changing and re-computing scoring formula and scores of KPIs for (all) emergency services.
NEXES’s challenge was to find an alternative approach to replace the ‘one size fits all’ KPI-approach. NEXES adhered to the ‘design for change’ adage and created the principle of flexible KPIs. This flexibility is achieved through the tailoring of relevant effect measurements and scoring importance.
Is the current set of NEXES effect measurements, 58 in total, finished? Not likely. The NEXES Action is set to conduct a number of exercises in which these KPIs will be applied to emergency services in Slovenia, Romania and Turkey that implement some or all of the NEXES Internet-enabled technologies. These exercises will also yield insight in the usefulness and practical applicability of the current KPIs and effect measurements. It may even lead to changes to the KPIs: that is a welcome effect of accumulating progressive insight. The net result will be an improved set of flexible KPIs that is useable by emergency services throughout Europe and beyond.
Moving forward, NEXES will gain experience with applying flexible KPIs and explore the opportunities to improve the current NEXES KPIs and effect measurements to the benefit of next generation emergency services across the world.