DESIGN EVALUATION WITH CWA

CWA is uniquely suited to the evaluation of the impacts of system change due to its formative nature. That is, it enables the analyst to identify all of the possible effects on system functioning, not just those that might be expected or anticipated. It provides a structured thinking tool that can help to identify emergent behaviours, unintended consequences of change, potential risks introduced due to system changes and potential design requirements to address these.

In this research programme, evaluation with CWA was conducted using large hardcopy diagrams of the appropriate WDA model (i.e. passive or active rail level crossing) as a starting point for the evaluation. Evaluations were conducted by small teams, with one analyst documenting the changes to the WDA diagram and another documenting the process and resulting discussion using an electronic template. For two concepts, the team included a small number of stakeholders who participated in the design workshops, to familiarise stakeholders with the process to enable future applications to proposed design changes without assistance from the researchers. Due to the time required to conduct each evaluation, the involvement of stakeholder representatives in the evaluation process was not feasible for all five concepts and other evaluations were completed by the research team only.

To undertake the WDA evaluation, the following steps were undertaken (see Figure 7.2 for an example):

• • New physical objects present in the design concept were added to the bottom level of the WDA. For example, if the design concept included an in-vehicle warning device, then this was added to the physical object level of the WDA.
• • Existing physical objects enhanced in the proposed design were highlighted. For example, an improved rail level crossing warning sign would enhance the existing physical object ‘rail level crossing signage’.
• • For each new and enhanced object, the following was undertaken:
• • The existing object-related processes that it would support were highlighted. For example, the new object ‘In-vehicle warning device’ would enhance the object-related processes ‘Alert user to presence of rail level crossing’ and ‘Alert user to presence of train’.
• • Pathways following the means-ends links from these highlighted object-related processes up to the functional purposes of the system were highlighted.
• • Any new object-related processes that the new or enhanced object would afford were added into the abstraction hierarchy.
• • Pathways following the means-ends links from these new object- related processes up to the functional purposes of the system were highlighted.

As the changes were being overlaid onto the WDA, a standardised template was used to document the following information arising from discussions:

• • Assumptions being made about the design and the effect it will have on system functioning.
• • Key benefits that are apparent from reviewing the effects on the system.
• • Potential negative effects, such as new risks introduced by the new design.
• • Areas where further investigation or research is required to understand the impacts.
• • Suggested refinements to improve the design’s potential to achieve desired benefits or to minimise potential negative effects of the new design.
• • The frequencies of new nodes, nodes that are enhanced or supported, nodes that are appropriately restricted and nodes that are negatively influenced.

The final step in the WDA evaluation process was to assess the impact of each new object assessed by summing the following at each level of the abstraction hierarchy:

• • New nodes: For example, the new physical object ‘Optimal speed to avoid train in-vehicle display’ would ‘Communicate optimal speed’ and ‘Provide distance to rail level crossing’ notification.
• • Support for existing nodes: For example, the new physical object ‘In-vehicle warning display’ would provide support for the existing function of ‘Alert user to the presence of train’.
• • Appropriate restriction: For example, the new physical object ‘Default closed pedestrian gates’ would appropriately restrict (pedestrian) traffic flow, which in turn would support the function of ‘Maintain road and rail user separation’.
• • Negative influence: For example, the new physical object ‘Speed limit reduction signs’ would have the effect of slowing traffic through rail level crossings, which in turn may negatively influence the ‘Maximise efficiency’ value and priority measure.

Each of the five prioritised design concepts was evaluated in this manner. To illustrate, we present the evaluation of the Speed, Expectancy, Gap concept.