Challenges using electronic nursing routine data for outcome analyses: A mixed methods study

Renate Nantschev , Elske Ammenwerth

a UMIT - Private University for Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria

b Institute of Medical Informatics, Hall in Tirol, Austria

Keywords:Austria Health care quality indicators Nursing care plan Nursing informatics Routinely collected health data Secondary use Standardized nursing terminology

ABSTRACT Objectives: To explore the challenges of secondary use of routinely collected data for analyzing nursingsensitive outcomes in Austrian acute care hospitals.Method: A convergent parallel mixed methods design was performed.We conducted a quantitative representative survey with nursing managers from 32 Austrian general acute care hospitals and 11 qualitative semi-structured interviews with nursing quality management experts.Both results were first analyzed independently and afterward merged in the discussion.Results: On average, 76% of nursing documentation is already electronically supported in the surveyed Austrian hospitals.However, existing nursing data is seldom used for secondary purposes such as nursing-sensitive outcome analyses.This is due to four major reasons:First,hospitals often do not have a data strategy for the secondary use of routine data.Second,hospitals partly lack the use of standardized and uniform nursing terminologies,especially for nursing evaluation.Third,routine nursing data is often not documented correctly and completely.Fourth, data on nursing-sensitive outcomes is usually collected in specific documentation forms not integrated into routine documentation.Conclusion: The awareness of the possibilities for secondary use of nursing data for nursing-sensitive outcome analyses in Austrian hospitals is still in its infancy.Therefore, nursing staff and nursing management must be trained to understand how to collect and process nursing data for nursing-sensitive outcome analyses.Further studies would be interesting in order to determine the factors that influence the decision-making processes for the secondary use of nursing data for outcome analyses.

What is known?

· The potential of existing electronic nursing care plan data is seldom used for secondary research purposes, such as to evaluate nursing-sensitive outcome indicators.

· Data for nursing-sensitive outcome analyses are primarily gathered in specific reports.

· Secondary use of nursing care plan data requires certain data quality criteria: Data must be standardized, complete, and correctly documented.

What is new?

· Digitalization process in Austria is mainly done without a concrete strategy for secondary use of routine nursing data.

· Nursing staff and the nursing management still can not see the possibilities of secondary use of standardized nursing data and the associated nursing evidence development following nursing-sensitive outcome analytics.

· Nurses must be empowered to design documentation processes so that the potential of secondary use can be better used in the future.

1.Introduction

Digitalization in hospitals is increasing internationally as well as in Austria.However, recent research shows that the level of digitalization in Austrian hospitals is still relatively low, with an EMRAM score of 2.3 (on a scale from 0 to 7) [1].

With increasing digitalization, the availability of electronically documented routine data is also growing.This amount of electronically collected routine data offers new possibilities for secondary use.

Secondary use refers to using data related to patient care as a byproduct for purposes not originally intended [2].The original reason for collecting routine clinical data is to support patient care coordination [3].Secondary use or reuse of this data aims to generate new knowledge from this data, for example, to improve health care quality and patient safety [2].

In nursing documentation,much routine data is gathered during the mandatory documentation of the nursing care process in Austria.The nursing care process requires a step-by-step approach(assessment, diagnosis, outcome planning, intervention, and evaluation) to plan and evaluate nursing practice [4].

However, the data routinely collected during the nursing care process is seldom used for secondary purposes,such as evaluating and improving nursing quality [5-7].Instead, this data is mostly gathered in specific reports related to selected nursing-sensitive outcomes, such as pressure ulcers, falls, and falls with injury or nosocomial infection, required for evaluating nursing quality.In general, nursing-sensitive outcomes are defined as reliable indicators that capture patient outcomes most affected by nursing care and are widely used to evaluate the nursing care quality[8,9].

Using specific reports to collect data in order to analyze nursingsensitive outcomes instead of using routine nursing data is associated with some disadvantages.For example,the data collection is time-consuming and allows only descriptive analysis.It also supports only moderate improvement of nursing care quality because the data of the reports is not linked to the nursing care plan[5,6,10,11].

However, the secondary use of routine data, and thus also the use of nursing care plan data,requires certain quality criteria:Data must be standardized, complete, and correctly documented by nursing staff [12].

First, to support standardized nursing documentation, several standardized nursing terminologies are available, such as NANDA International (NANDA-I), Nursing Intervention Classification (NIC),Nursing Outcome Classification (NOC), or International Classification of Nursing Practice (ICNP).Digitalization now more easily offers the opportunity to adopt standardized nursing terminologies to support nursing documentation [6].However, the adoption of standardized nursing documentation is still limited.For example,health care leaders often lack knowledge about the value of standardized terminologies, so different terminologies are used in different organizations and a considerable amount of nonstandardized nursing terminology is applied [13].

Completeness is the second aspect of data quality for secondary use.The aim of completeness in terms of secondary use is how the available data matches the specific requirements for the particular research interest[14].For nursing documentation,completeness is mostly measured regarding the completeness of all categories in the nursing care plan and whether the legal requirement is met.For this intended task,assessment instruments such as Q-DIO(Quality of Nursing Diagnoses, Intervention, and Outcomes) or D-Catch are used for measuring nursing documentation in hospitals [15,16].However, these instruments might not be helpful to assess the completeness of routine data for secondary purposes such as nursing-outcome analyses.In terms of secondary use, the completeness of data must be assessed once the data quality features that are of interest to research have been defined [14].

Finally,correctness is the third requirement for secondary use of routine data.The correctness of the nursing data is influenced by the accuracy of how the nursing staff assesses and evaluates the patient situation, thus making the data suitable for secondary use[17].Several studies found that nursing documentation is often incorrect in capturing the patient situation [18].This might also limit their utility for nursing outcome analysis.

Summarized, these three requirements (standardization,completeness, and correctness) seem to be the most challenging and limiting factor of secondary use of routine clinical data.To more deeply understand the challenges in meeting these requirements for the secondary use of routine nursing data, we carried out this mixed methods study in Austria.We believe that we can only develop targeted steps for improvement if we better understand the current challenge in using routine data for quality analyses.Our results in Austria might also be relevant for other countries for opening up the best benefit from the secondary use of routine nursing data for nursing outcome analyses in the advancing digitalization process.Currently, no national strategy is available for secondary use of electronic nursing routine data in Austria.

In this study, we aimed to explore aspects influencing the challenges of secondary use of electronically routinely collected data to analyze nursing-sensitive outcomes in Austrian acute care hospitals.

We defined four research questions and addressed two of them(Q1, Q2) by the quantitative methods and two (Q3, Q4) by qualitative methods.

Q1: What is the current availability of electronically documented nursing data in Austrian acute care hospitals?

Q2: How are standardized nursing terminologies used in nursing documentation in Austrian acute care hospitals?

Q3: What challenges currently affect the utilization of routine clinical data in nursing care for nursing-sensitive outcome analysis?

Q4: What strategies of nursing-sensitive outcome analyses are currently applied in Austrian hospitals?

2.Methods

We performed a convergent parallel mixed methods study according to Creswell et al.[19]design using quantitative and qualitative methods undertaken with nursing experts from Austrian acute care hospitals.This convergent design involves both quantitative and qualitative data collection in parallel,given equal priority to both,analyzing both separately and then merging the results of each part.Collecting both quantitative and qualitative data can give a more comprehensive picture of the research field because each finding may be insufficient by itself [20].

To be more specific, both methods were thus complementary.The quantitative data captured the standardization in electronic documentation, and the qualitative data helped understand how nursing experts deal with routinely collected data to analyze nursing outcomes.Both methodological approaches were equally important to answer the research question.Both types were analyzed independently, and a side-by-side comparison with separate reports of the quantitative and qualitative findings is presented in the results section.Further comparative integration and merging of both results can be found in the discussion section.

The Good Reporting of a Mixed Methods Study (GRAMMS)guidelines was used as a framework to report the study design and findings [21].

2.1.Quantitative design and sample

A standardized, self-developed, online-based survey was performed to collect the quantitative data for this study.A selective cluster sample of all 90 Austrian general acute care hospitals was built.We aimed to include at least 30%of the hospitals.For this,the cluster sample method was chosen to achieve an even distribution across Austria, as shown in Table 1.

General acute care hospitals in Austria are defined as hospitals for people regardless of gender, age, or type of medical care and characterized by a length of stay of less than 18 days on average.These hospitals are non-profit and publicly funded by the Austrian Diagnosis-Related-Groups (DRG) system, the Austrian pay-forperformance hospital financing model.All hospitals offer a wide range of health-related services, including at least services in internal medicine and general surgery [22].

We aimed to recruit one representative person in a nursing management position (e.g., senior manager, nursing quality management expert,or similar)in the cluster sample(n=39)built from all general acute care hospitals as shown in Table 1.Each representative of the selected hospital in the cluster sample was contacted personally via telephone and invited to participate.The questionnaire was sent to each person via an online tool.Data collection was carried out in the period from January to March 2020.The response rate was 82% (n= 32).

2.2.Quantitative data collection and analysis

The survey we used in this study was developed based on a literature review.We also included modified questions from the annual national IT-Report [23].In total, the questionnaire contained 19 questions and 54 sub-questions and an opportunity for open comments.The survey’s major topics included the degree of electronic nursing documentation,the use of nursing classification systems for standardized documentation in nursing care planning,and the use of nursing-sensitive outcome indicators.After establishing the first draft,the questionnaire was tested by four nursing experts.Based on their comments,the questionnaire was modified for the final version.

Descriptive statistics were performed for the quantitative data using SPSS to describe the characteristic of the respondents and to answer the first and second research questions.

2.3.Qualitative design and sample

We conducted a qualitative study using a hermeneutic design based on the methodological framework developed by Kuckartz[24].A convenience sample of experts responsible for nursing quality management was recruited from 14 Austrian acute care hospitals.In most Austrian hospitals,at least one nursing expert is intended to be accountable for nursing quality management and control.One of their tasks is continuously preparing nursing quality analyses and nursing quality reports; dealing with nursingsensitive outcome indicators is therefore part of their tasks.

We aimed to recruit one or two experts at general hospitals from each of the nine Austrian federal states in order to obtain a comprehensive picture.

2.4.Qualitative data collection

In recruiting, we used the first-line personal relationship with experts in this field and recommendations within this network.Overall,we invited 14 experts by telephone and email to participate in the interview.In the end,11 experts agreed to participate in the interview.Qualitative data collection.

We developed a semi-structured interview guide using the methods recommended by Helfferich (2011), which is based on four main steps(brainstorming,classifying the questions,verifying for suitability, pretesting and refining [25].

The interview guide subsumed two main topics:The first topic covered seven open questions focused on relevance and benefits in analyzing nursing-sensitive outcome indicators.The second topic covered five open, and one standardized question focused on challenges in secondary use of routinely collected data in nursing care.We evaluated the interview guide with two nursing experts in a pilot study.The interviews were conducted personally and online in the period between January and March 2020 by one researcher.Participants’ informed consent was collected from each expert on the day of the interview.

The audio-recorded interviews took place at the participants’place of work and lasted 25-60 min.Recorded interviews were transcribed verbatim by the interviewer using the MAXQDA software.

2.5.Qualitative data analysis

We conducted qualitative content analyses following the phases described by Kuckartz[24].Two approaches of building categories were combined: deductive category development and inductive category development [24].

First,we built the main categories of literature-based theoretical consideration and the interview guide.Each main category was defined beforehand.We then deductively coded the interview transcripts based on these defined main categories.

In a second run through the text, we inductively built subcategories added to the main categories.We used MAXQDA 2018 for data analysis.

2.6.Ethical approval

This study was conducted in accordance with the standards laid out by the Research Committee for Scientific Ethical Questions(RCSEQ).The RCSEQ, an independent body of the Tyrolean Private University UMIT TIROL, examines planned research projects that,due to the legal provisions,do not fall under an ethics committee's jurisdiction.

3.Quantitative research results

We received responses from 32 different general acute care hospitals in Austria;this corresponds to 36%of the overall number of hospitals in Austria and 37% of overall hospital beds.Nineteen respondents(60%)were senior nursing managers or assistants,ten(31%) were nursing quality managers, and three (9%) were nurses responsible for nursing documentation development in their hospital (n= 32).Table 1 shows the cluster sample characteristics of the participating hospitals.

Table 1Characteristics of cluster sampling in our survey based on all general acute care hospitals in Austria.

3.1.Use of electronic documentation

The first part of the survey focused on using electronic documentation for different clinical enterprise functions.Participants were asked to estimate the percentage of electronic documentation on a scale between 0 and 100, for example, in medical documentation, nursing documentation, or medication prescription.The results show that the percentage of electronic nursing documentation in Austrian hospitals is between 0 and 100%,with a mean of 76%.Half of the examined hospitals (n= 32) have 90% or more electronic nursing documentation (Median= 90%).Fig.1 shows more details on the proposed enterprise function in the survey.

3.2.Standardized terminologies

Further, we surveyed respondents on the use of standardizednursing terminologies, self-developed standards, and free-text documentation.As shown in Fig.2, standardized terminologies are mostly used for nursing diagnoses (63%) and for nursing assessments (47%).For nursing outcome planning and nursing intervention, a mix of self-developed standards and standardized terminologies is usually used in nursing documentation.Free text is predominantly used in nursing evaluation.

We then asked which standardized nursing terminologies are used in the examined hospitals in Austria (n= 32).We found that NANDA-I is used by 29%of the hospitals,ICNP by 13%,and ENP also by 13%.Moreover,a number of hospitals are planning to introduce standardized nursing terminology in the future: 48% of the hospitals plan to introduce LEP for standardized nursing intervention,45% plan to introduce epCC? for standardized documentation of nursing assessments, and 13% plan to introduce NANDA-I or ENP.

3.3.Data collection for nursing-sensitive outcome analyses

Furthermore, we asked for the current approach of the documentation for nursing-sensitive outcome analyses.The proposed nursing-sensitive outcomes in the questionnaire were: falls, pressure ulcers, wound infection, malnutrition, pain, catheter-related urinary tract infection, delirium, and pneumonia.

Fig.3 shows the detailed responses.Nearly 100%of all hospitals indicated that they document falls, pressure ulcers, pain, and delirium.The outcomes are, however, documented in different forms:Falls and pressure ulcers are mostly documented in specific reports, while pain and delirium are recorded in the routine nursing documentation (nursing care plan).

4.Qualitative research results

This section presents the qualitative study results from the expert interviews in order to obtain more insight into the challenges facing experts when analyzing nursing-sensitive outcomes.The interviews were conducted with 11 participants.Three interviewed experts were senior nursing managers,and eight were in the position of nursing quality manager.

In the following, we describe the most important results obtained from this qualitative content analysis led by five main categories.To indicate verbal responses from the interviews, we are using the following structure“(x.Pos.:25-26)”(x =the first letter of the name of the transcript; Pos.= lines in the transcript).

4.1.Challenges of using routine nursing data

Interview results show that all experts realize the importance of routine data in nursing care.They are aware that the analysis of routine nursing data can open up new nursing topics.“With the electronically collected nursing data,the time also has come for a new era in nursing.” (V.Pos.:25-26) Nevertheless, the experts also said that added value could only arise if appropriate considerations are made explicitly in advance,which seems quite challenging to them.Three main challenges could be summarized from the expert interviews.

First,one of the challenges in using routine data lies in the entire digitalization strategy.Almost all experts complained that there was no data strategy for secondary use of routine data in the entire process of digitalization.They noted that it is crucial to involve everyone in the implementation process and concentrate only on important topics.These were key elements for most of the experts in order to succeed and achieve added value through digitalization.“A lot of knowledge has to be built up in the company, and that requires the cooperation of informatics specialists, organization management, and nurses.” (Gr.Pos.: 22-26)

Fig.1. Estimated percentages of using electronic documentation systems to support specific enterprise functions in Austrian acute care hospitals (n = 32).

Fig.2. Use of standardized nursing terminology in different categories of the nursing care plan in Austrian general acute care hospitals (n = 32).

Second, the experts also mentioned that the challenge is to support adequate nursing documentation that allows the secondary use of routine data.In the mindset of health professionals,clinical documentation is primarily still an instrument for treating patients and fulfilling the legal documentation requirements.“Nurses have for many years thought about the possibilities of nursing documentation,but they have never thought about the possible added value of this data.” (Ew.Pos.: 2-2) It is quite challenging to implement standardized documentation because nurses and other health professionals are concerned that they cannot fully describe the patients’ situation in a standardized form.The experts supposed that the documentation would have a different meaning for the nurses if they experienced the added benefit of routine nursing data.

Third, experts saw many technical issues challenging the secondary use of routine data.At the moment, only little data can be used from the electronic nursing care documentation to analyze,for example, nursing-sensitive outcomes.There often are few possibilities to extract data from the application systems.Another challenge is that different application systems are not compatible.Therefore, “additional electronic reports have to be created for data collection and evaluation.” (Ke.Pos.: 23-24)

4.2.Current strategies of data management for nursing-sensitive outcome analyses

All interviewed experts (n= 11) at the acute care hospitals in Austria explained that specific reports are used to record adverse events such as falls or pressure ulcers.For example, several parameters have to be recorded for each fall event.Pressure ulcers are usually recorded as prevalence surveys on specific reference dates.Both falls and pressure ulcers are only rarely recorded as part of nursing care plan documentation.Furthermore,if outcomes such as malnutrition or delirium are of interest for evaluation, the data is also collected in specific reports.Recording and analyzing nosocomial infection is mostly not perceived as part of nursing outcomes and is instead carried out by specialized hygiene teams to evaluate the infection rate.

To analyze data from the specific reports, almost all experts indicated that they had to manually transfer the data from the database to an Excel sheet.Data from routine nursing care plans was seldom used;if used,the data was also entered manually in an Excel sheet.Only one expert mentioned utilizing a tool for the preparation of the data and having the opportunity to integrate administrative data: “…and a semi-automated evaluation has now been linked to this new documentation.” (Ki.Pos.: 32-32)

4.3.Challenges of data quality for nursing-sensitive outcome analyses

In general,data quality in nursing documentation is a big issue.Some experts assumed that one-third of the data was incorrectly documented.Much relevant data is missing in the nursing care plan and the specific reports.According to the experts, lower data quality often depends on nurses’ ability to assess the clinical situation correctly.For example,one expert mentioned,“A lot of errors can be seen in the data, e.g., incorrectly classified pressure ulcers or that a lot of data is missing.” (Gr.Pos.: 26-27) Also, a lack of standardized terminologies and the use of self-developed standards complicate using routine data.This circumstance makes analyzing data very difficult.

Some experts pointed out that nursing ward managers heavily influence data quality.They play a key role in how well the nursing documentation processes have been implemented and how responsibly they deal with data quality assurance.“It depends on how managers make sure that everything is precisely documented.”(Cz.Pos.: 18-19)

Fig.3. Approach of documentation of the proposed nursing-sensitive outcomes in Austrian hospitals (n = 32).

Data quality also depends on nursing staff's awareness of the importance of this data.In some cases, nurses have already recognized the benefit taken by analyzing nursing data.However,as soon as hybrid documentation is necessary to record the nursing-sensitive outcomes, this deteriorates the data quality.

4.4.Challenges in analyzing nursing-sensitive outcomes

One aspect of the challenge in analyzing nursing-sensitive outcomes mentioned by the experts is that no national recommendations in Austria are available to record and analyze nursing data.Comparable target values for nursing-sensitive outcome indicators are missing.Each organization has its own database and its own target values.“So far, we were not able to think beyond the boundaries of the own area to measure uniformly comparable nursing quality and to allow for a national benchmark.”(Kap.Pos.: 29-31)

Also,a limiting factor in analyzing nursing-sensitive outcomes is the typically short length of patient stay in acute care hospitals,which limits the ability to generate sufficient data for further analysis.Due to this fact, specific nursing outcomes such as malnutrition cannot be meaningfully considered.

4.5.Challenges in reporting nursing-sensitive outcomes

To report specific nursing-sensitive outcomes,summaries of the key figures are made in all examined hospitals (n= 11) by the nursing quality management experts.These summaries are then provided to the ward managers and the nursing management,e.g.,fall rate or pressure ulcer prevalence.The experts complained that these retrospective data analyses require a lot of interpretation.To prepare the reports, nursing experts have to know the specific conditions at the individual ward that may influence the analyses.That includes, for example, knowing incorrect entries or duplicate entries.

Based on this kind of analysis, it is difficult to generate meaningful insight into the complex interrelationship of care.Also,ward managers have to explain and justify the outcome of what happened, for example, one year ago.The results always remain a personal subjective interpretation of the ward managers.“The problem with all of this is that I don’t measure the nursing process when measuring the outcome,but the quality of care is reflected in the process.” (Cz.Pos.: 32-34)

5.Discussion

In our discussion, we will now merge results from the quantitative results and the qualitative findings following our convergent mixed methods study design [19].

This mixed methods study sought to gain findings on the challenges currently facing the secondary use of routinely collected data to analyze nursing-sensitive outcomes in Austrian acute care hospitals.The study covered three main topics: availability of electronic nursing data and use of standardized nursing terminologies,challenges influencing the utilization of routine nursing data for nursing-sensitive outcome analysis, and the current strategies in nursing outcome analyses.

5.1.Availability of electronic nursing data and use of standardized nursing terminologies

Electronic documentation has already been introduced in most Austrian acute care hospitals.Increasing electronic documentation can be seen in recent years.In Austria, the proportion of fully introduced electronic nursing documentation increased from 59%in 2015 to 64% in 2020 [23,26].Our current representative study showed that, on average among Austrian hospitals, 76% of nursing documentation is done in electronic form.

Another focus of our study was the actual use of standardized nursing terminology in nursing documentation as a precondition for secondary use of data.In our survey,we found that standardized nursing terminology is mostly used to document nursing diagnoses and nursing assessment.The other elements of the nursing care process (e.g., nursing outcomes, nursing intervention) are mostly documented with self-developed standards or a mix of standardized terminology and self-developed standards.A recent nationwide representative study from the Netherlands with registered nurses also showed that only 56% of the respondents use standardized terminology in their electronic documentation system[27].However,the study did not differentiate for which element of the nursing care process standardized nursing terminology is used.

Our study’s findings from the interviews, merged with the survey, indicated that electronic documentation supports the implementation of standardized nursing terminology.The hospitals that we examined are planning to introduce standardized nursing terminologies in nursing documentation for various elements in the nursing care process.The interviewed experts mentioned that this process is driven by the entire digitalization process in the hospitals.

Our study also showed that many different standardized terminologies are used.In Austria, there is no national recommendation for a common nursing terminology.This is also the case in other countries such as the Netherlands, where different standardized nursing terminologies in nursing documentation are used.For example, in the Netherlands, Gordon’s Functional Health Patterns (Gordon) is used most (39%), in addition to NANDA-I, NIC,NOC (10%), the Omaha System and Resident Assessment Instrument(RAI)[27].Some countries,such as Finland,have developed a national standardized terminology system,in this case the Finnish Care Classification (FinCC) [28].This is not currently planned for Austria.

5.2.Challenges influencing the usability of routine nursing data for nursing-sensitive outcome analysis

The interviewed experts were aware that increased digitalization opens up new possibilities for nurses, making it necessary to consider new aspects in nursing documentation.Currently,nursing documentation is considered an instrument primarily for patient care.The experts interviewed in our study mentioned that nurses often feel that standardized terminologies hinder them in best describing the entire patient situation during care provision.They cannot yet see the benefits of using standardized terminologies,for example, for secondary purposes.This finding aligns with other research that the emphasis of clinical data collection is elementary for clinical use and reimbursement[29].Another study pointed out that nursing staff sees the most benefit from electronic documentation in supporting their activities of providing patient care[27].

However, the interviewed experts mentioned that the lack of standardized nursing terminology in nursing documentation makes using routine data more difficult.Still,they also pointed out that this is just one criterion for secondary use of routine nursing data and does not necessarily contribute to improving data quality.Nursing staff could bypass standardized documentation or essential data could not be recorded.

Another challenge for analyzing nursing-sensitive outcomes is the lack of completeness and correctness of electronic nursing data.International studies have shown that capturing the patients’needs is incomplete in almost all areas of nursing documentation [18].Our study’s findings indicated that the competence of the ward managers heavily influences the completeness and correctness of nursing documentation.Data quality also depends on the nursing staff's awareness of the importance of this data.For this, critical thinking seems to be an essential aspect for improving nursing documentation accuracy [18].

Furthermore, in the era of electronic nursing documentation,nursing staff needs to know the importance of accuracy and completeness of nursing documentation and the benefit from this data [16].The success of secondary use of routine clinical data depends on the perceived usefulness of the information derived from this data [30].

Several instruments have been developed so far to measure the accuracy of nursing documentation.For example,D-Catch or Q-DIO are valid and reliable instruments designed to evaluate the accuracy of nursing documentation in documentation structure,assessment on patient admission,nursing diagnoses,nursing intervention,patient outcome,and documentation legibility[15,16].However,these instruments were not developed to measure data quality for secondary use of routine nursing data.According to Johnson et al.(2016), a set of data quality measures must be developed for each desired task [17].In the view of secondary use, completeness and correctness are determined by how well the available data matches the specific requirements for this task.The requirements for data quality,for example,must be defined in such a way that they allow for the analysis of nursing-sensitive outcomes[14].

5.3.Current strategies in nursing outcome analyses

Currently, routinely collected electronic data from the nursing care process is seldom used for nursing-sensitive outcome analyses.The experts interviewed for our study complained that there is often no data strategy considered in digitalization for the secondary use of routine nursing data,for example,to evaluate nursing outcomes.

The most common strategy in the surveyed Austrian hospitals to evaluate nursing outcomes is gathering data with patient-specific reports, and mostly this is done to evaluate patient falls (91%)and pressure ulcers (88%).This means that nurses must perform additional documentation,a disadvantage that also harms the data quality.Previous studies indicate that analyses based on data collected using methods with specific reports might help to advance the understanding of nursing-sensitive outcomes.Still,these methods seem impractical for routine use in practice[31].

However, our study results also indicate that most of the nursing-sensitive outcomes proposed in this study are not relevant to the evaluation.For example, while the survey results showed that catheter-related urinary tract infection(47%),wound infection(41%) or pain (50%) are documented in nursing care plans, the interviewed experts note that this data is rarely used for nursing outcome analyses.

This study also showed that a large amount of nursing data is routinely collected but not analyzed and used to support clinical nursing practice by reporting nursing-sensitive outcomes.Nursing staff, nursing educators, and nursing scientists often lack the required competencies necessary to meaningfully use and understand the advantage of big nursing data [5,32].Also, the experts interviewed for our study pointed out that a lack of data integration possibilities from different electronic documentation systems hinders the use of routine data for nursing-sensitive outcome analyses.

5.4.Limitations

We purposely conducted a concurrent mixed methods study design [19].We started in parallel with a quantitative study and a qualitative study.Both results were analyzed independently and merged in the discussion.The limitation of this design is that the quantitative and qualitative studies do not influence each other.For this research, we might have also considered an exploratory sequential mixed methods design,first starting with the qualitative study and using these results to develop the questionnaire for the quantitative part of the mixed methods study,then carrying out the quantitative study as a second step.

In this mixed methods study,we used a convenience sample for both methods.We were able to gain representative results from the quantitative study for the Austrian general acute care hospitals.For the interviews, we included 11 experts for nursing quality management.Although the number of experts was small, the content was saturated.With a different group of experts for the qualitative sample, different results might possibly be expected.For example,IT experts would probably have seen the technical aspects differently.The inclusion of experts from management would possibly have brought a different view of the key figures for the nursing outcomes.

6.Conclusions

This study aimed to explore the challenges of using routine nursing data to analyze nursing-sensitive outcomes in Austrian acute care hospitals.Our results show that the utilization of routine nursing data from the nursing care plan in Austria is still strongly limited.We were able to illuminate some reasons for this.

Digitalization is continuously increasing in Austrian acute care hospitals.Still, we could see in our study that this digitalization process is mainly done without a concrete strategy for secondary use of routine nursing data.First, lower data quality makes the process of reusing routine nursing data difficult.Nursing documentation is currently not very standardized.Nursing terminologies are not used in the entire documentation of the nursing care plan.The most standardized part are nursing assessments and nursing diagnoses.The results of the interviews show that data quality is also limited by the correctness and completeness of nursing data.This often depends on the nurses’ability to assess the clinical situation correctly, but often they are also unaware of the importance of complete and standardized nursing documentation.

Second, another reason that hindered secondary use of routine nursing data is technical issues.For example, the opportunity for data integration is limited.The available electronic data can often not be extracted for further processing.e.g., to analyze nursingsensitive outcomes.

The current approach to analyzing and reporting only a few nursing-sensitive outcomes is to use data collected in patientspecific reports.The most reported indicators are fall rate and pressure ulcer prevalence.The results of this retrospective analysis need a lot of interpretation by the ward managers.The interviewed experts mentioned that this kind of report provides only limited support for the process of nursing quality development.Furthermore, automated real-time analyses with data from the entire nursing process would be more helpful for feedback on the nursing quality and would support nursing staff in reflecting the advantages of high data quality in the nursing documentation.

However, as a result of our study, nursing staff and the nursing management often seem to lack knowledge about the possibilities of secondary use of nursing data and the associated nursing evidence development following nursing-sensitive outcome analytics.

Therefore, nurses and nursing management need to be trained to understand the advantages and possibilities of routine nursing data for nursing-sensitive outcome analyses.Nurses and nursing management must be empowered to design documentation processes so that the potential of secondary use can be better used in the future, both in one as well as across several facilities.Future studies should determine the factors that influence the decisionmaking processes for secondary use of routine nursing data for outcome analyses in Austrian acute care hospitals in more detail.Furthermore,in the context of the advancing digitalization process in hospitals,strategies in processing nursing routine data should be developed.Based on these findings, the next aim of our research project is to develop a guide for data management strategies for hospitals.

Funding

This work is part-funded by the Tyrolian Science Fund [grant number TWF GZ: F.16633/5-2019].

Data availability statement

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

CRediT authorship contribution statement

Renate Nantschev:Conceptualization, Methodology, Validation, Formal analysis, Investigation, Resources, Data curation,Writing-original draft,Writing-review&editing,Visualization,Project administration.Elske Ammenwerth:Conceptualization,Methodology,Validation,Writing-review&editing,Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A.Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.ijnss.2021.11.003.