For which clinical indicator would the nurse question an order for a gastric lavage

Comput Inform Nurs. Author manuscript; available in PMC 2018 Dec 1.

Published in final edited form as:

PMCID: PMC5722656

NIHMSID: NIHMS881447

Elaine L Larson, RN, PhD, FAAN, CIC, Associate Dean for Research and Anna C. Maxwell Professor of Nursing Research, Bevin Cohen, MPH, Project Director, Jianfang Liu, PhD, Senior Data Analyst, Philip Zachariah, MD, Assistant Professor of Pediatrics, David Yao, PhD, Piyasombatkul Family Professor of Industrial Engineering and Operations Research, and Jingjing Shang, RN, PhD, Assistant Professor

Abstract

Although previous research has confirmed that nurse staffing affects patient outcomes, some potentially important factors have not been accounted for in tools to assess relationships between staffing and outcomes. The aim of this project was to develop and test a Nursing Intensity of Care Index using electronically available data from 152,072 patient discharges from three hospitals. Initially 1,765 procedure codes were reviewed; 69 were confirmed as directly increasing nursing work load by at least 15 minutes per shift. Two research staff independently reviewed a random sample of five patient days to assess inter-rater reliability of with complete scoring agreement. To assess face validity, eight nurse clinician experts reviewed factors included in the Nursing Intensity of Care Index to assess the accuracy of the nursing time estimates in the tool. To examine concurrent validity, Nursing Intensity of Care Index scores for a random sample of 28 patients from four clinical units were compared with assessments made by a unit-based clinical nurse (low/medium/high intensity) for the same patients on the same day with a Spearman’s correlation of 0.94. In preliminary testing, data for the Nursing Intensity of Care Index which accurately reflect nursing care intensity can be obtained electronically in real time. The next steps will be a discrete-event simulation model and large-scale field trials.

Keywords: Electronic health records, nursing workload, psychometrics, nursing care

Introduction

The prevalence of electronic health record (EHR) systems is increasing rapidly; as of 2010, only 11.9% of U.S. hospitals had an EHR and 17% had computerized order entry for medications,1 but by 2015, 96% of acute care hospitals had adopted a certified EHR2. Whether health records are electronic or paper-based, however, nurses spend approximately half of their work time interacting with documentation, which is increasingly relied upon to plan and implement downstream data-driven improvements to patient safety and quality of care. While recent substantial gains have been made in EHR adoption, there is still limited evidence that using EHRs improves patient care and captures sufficient aspects of nursing practice to provide useful information for care planning.

There continue to be substantial barriers to the full application of health information technology, and there is much work to be done to demonstrate how EHR data can and should be used to improve the quality and efficiency of patient care systems. Despite these barriers, recent Centers for Medicare and Medicaid Services legislation regarding reimbursement and public reporting for adverse events such as healthcare-associated infections or falls, now mandated in many states, require improved ability to track patient outcomes and risk factors over time and across settings. As noted by Brennan and Bakken3, electronically collected data now includes information that has been previously unmanageable in speed and size but can now contribute to health services research in ways not previously possible.

One potential area of research using electronic data is the assessment of nursing workload and the intensity of care nurses must provide. Such assessment is invaluable because patient and nurse outcomes are significantly associated with care intensity. High workloads are associated with lower job satisfaction and burnout and higher turnover among nursing staff4 and, more important, with compromised patient safety, reduced quality of care, and increased adverse patient outcomes and mortality5,6. Hence, the aim of this project was to develop and test a nursing intensity of care index (NICI) using electronically available data to incorporate information previously difficult to obtain.

Background

Nursing care demands and distractions and risk of hospital-acquired infection

Several studies have identified a link between distractions and nursing care demands and increased rates of adverse patient events, many of which have focused on medication errors7,8. In almost 3,500 ‘near misses’ reported by nursing students over a 3-year period, about one-fourth of near misses related to infection control practices9. Factors which contribute to distractions and care demands include the acuity of patient needs, unit staffing patterns, characteristics of the work environment, and technical demands such as numbers of invasive devices and procedures (e.g., ventilators, central lines, urinary catheters, feeding tubes, surgery, patient isolation). Patient isolation, as one example, is exceedingly time and cost intensive10. Technical demands, procedures, and numbers and types of medical devices not only require time-consuming nursing activities but also increase the patient’s risk for adverse outcomes. Similarly, patient movement, reflected by the number of admissions, transfers, and discharges on a unit, imposes a large demand on nursing time, but is a factor often ignored when assessing the relationship between adverse outcomes and nurse staffing and care intensity.

While many patient classification and nurse staffing systems exist, they generally require additional data collection and time/resource expenditure and may not capture specific patient- and/or unit-level distractions or be strongly correlated with actual workload11. Although many studies assess nurse staffing needs by using only the number of patients receiving care as the denominator, the intensity of nursing care in actual practice is multidimensional, affected not only by the number of patients on the unit but also their acuity, turnover rates, and other technical demands. In fact, nurse staffing rates which are not adjusted for patient turnover and patient severity have been found to underestimate nursing workload and overestimate nurse staffing levels12. Prior workload measurement systems have not been available using real-time data as an algorithm; these systems have sometimes been unsuccessful because they were retrospective, and the nurse entering the data did not witness any benefit.

Sicker patients often require more nursing care, and researchers must adjust for patient acuity using well established measures to accurately assess the relationship between nurse staffing and patient outcomes. Although there are validated tools for measuring the potential impact of patient acuity on nursing care requirements13,14, severity of illness is just one of a number of factors that predict nursing care demands15.

One potentially important factor which has not been considered in previous studies assessing the impact of nurse staffing on patient outcomes is the additional workload and stress associated with emergency planning, such as threatened or impending outbreaks or potential disasters. Some of these are local (e.g., following the destruction of the World Trade Center in New York City, or a terrorist threat during which time hospitals are on alert and staff are ‘on call’), and others may be regional or national (e.g., the Ebola threat). In the face of such events, emergency preparedness activities for the entire hospital system include developing educational materials and algorithms, convening multiple meetings, coordinating response with the health department and the Centers for Disease Control and Prevention (CDC), training and, most important, educating and supporting front-line staff. These preparedness activities are extremely costly16,17 and add considerably to front-line nursing responsibilities and workload due to educational requirements and increased patient acuity and/or volume18. Similarly, exposures to community onset conditions such as TB, norovirus, pertussis, or scabies also add considerably to front-line staff responsibilities, time, and workload19–21. Hence, we sought to include such events on the unit or institution level in the assessment of nursing care intensity.

Problems associated with assessing nurse staffing have been identified that could result in over-simplification and misinterpretation of data22, suggesting the need to rethink the relationship between staffing and patient outcomes, and identify patient- and unit-level (patient acuity, nursing care demands, distractions) as well as systems-level factors (staffing, time required to address emerging and epidemiologically important community infections) associated with care needs with much more granularity so that appropriate targeted interventions can be developed and tested. Although some exploratory work has been done, to our knowledge there are no tools currently available to accurately measure nursing care demands using electronic data, nor do current measures include some potentially modifiable factors to guide interventions. Hence, some important unit/institution-level factors which may have a significant impact on nursing care needs have not been currently accounted for in patient classification and staffing tools.

Methods

Sample and setting

With federal funding (R01NR010822 and R01HS024915), a dataset was extracted from various electronic databases from three hospitals, which are part of a large health care system in metropolitan New York City and housed in the system-wide Clinical Data Warehouse. The system is the largest hospital system in the largest metropolitan region in the United States with more than 2,000 beds and more than 100,000 patient admissions annually, including a community hospital, a free standing children’s hospital, and a tertiary/quaternary care hospital that provide care to a diverse patient population. The work to extract and link multiple data sources and construct variables of interest required less than 2 years, and is described more fully in previous publications23,24. Data from the years 2012 and 2013 from the three hospitals were used for the initial development and testing of this tool (Table 1). This project received expedited approval from the Columbia University Medical Center and Weill Cornell Institutional Review Boards.

Table 1

Numbers of patient discharges from three study hospitals

Initial development of the Nursing Intensity of Care Index

To determine the relevant factors that contribute to nursing care intensity, our research team including nurse and physician clinicians and clinical researchers in a series of weekly meetings developed an initial list of factors which contribute to the intensity of nursing care demands on a clinical unit. We first examined factors at patient level, including indicators of disease severity and specific procedural and technical demands for nursing care.

To identify procedural demands, nurses and clinical experts examined 51 pages of procedure codes available in our electronic database (N = 1,765 procedures) to identify procedures that might result in increased nursing workload. From that list we initially identified a total of 69 procedures which were then reviewed by an additional 11 full-time expert adult and pediatric clinical nurses from various types of units who confirmed which procedure codes met the following criteria: (1) included some nursing responsibility and (2) directly increased nursing work load by at least 15 minutes per shift. This time increment was selected arbitrarily after discussion with clinical staff as being sufficiently long to contribute to workload.

Additional information regarding time required for various nursing interventions were obtained from de Cordova, et.al.25, who used a Delphi consensus approach and focus groups to obtain nurse reported average times to complete selected nursing interventions. Using these data sources, procedures were coded in 15-minute increments for each 12-hour shift according to nursing time required, with scores ranging from 1 to 3 (1 = 15 to 30 minutes; 2 = 31 to 60 minutes; 3 = more than 61 minutes of required nursing time per shift). For example, insertion of an indwelling urinary catheter received a score of 1, isolation/barrier precautions scored 2, and extracorporeal membrane oxygenation (ECMO) and left ventricular assist device (LVAD) monitoring scored 3 points per shift. Since nursing responsibilities and patient populations vary by unit, many of these procedures are only relevant to certain practice settings and are therefore unit-specific. Other patient specific contributors to the NICI was a well validated index of adult patient mortality, the Charlson Co-morbidity Index26,27.

At a unit level we included patient movement (admissions, transfers, discharges) and nurse staffing, both available from our EHR system. Finally we incorporated institution-level factors such as times of outbreaks and emergency response needs (e.g., the Ebola preparations, measles outbreak) extracted using the institution’s electronic Infection Prevention Surveillance system and the mandatory NY State Department of Health Nosocomial Outbreak Reporting Application (NORA). The total possible NICI score was designed to be calculated separately for each unit on a daily basis combining patient, unit, and systems level factors. See Table 2 for a summary of the categories and factors used to develop the NICI. Table 3 describes the electronic data sources used to create the NICI.

Table 2

Sample data elements at the patient, unit, and systems levels used to develop Nursing Intensity of Care Index

Table 3

Electronic data sources used to develop the Nursing Intensity of Care Index

Confirmation of electronic data availability

Next we confirmed that data for these categories and factors were available in the existing electronic database for each individual patient on a daily basis by retrieving 1 month of daily data from four randomly selected units, including an exact patient count, turnover rates (e.g., admissions, transfers, discharges), numbers of mechanically ventilated patients, patients with indwelling devices (central lines, urinary catheters), patients requiring dialysis or have feeding tubes, and number of surgical procedures. In addition, two of our research staff independently reviewed a random sample of 5 patient days across several units and hospitals to assess inter-rater reliability of recordings of patient severity and technical demands and procedures from our database versus from chart review. There was complete agreement on their ratings, confirming that the data could be accurately retrieved electronically. Finally, we conducted a preliminary assessment of the frequency with which these technical demands and procedures occurred to confirm that there was variation across units and across time.

Subsequent psychometric testing of the Nursing Intensity of Care Index

To assess face validity, eight nurse clinician experts from these units reviewed the list of factors included in NICI to assess the accuracy of the nursing time estimates in the tool (Table 4). To examine concurrent validity, we independently calculated the scores for the Technical Demands and Procedures and Medication Administration sections of the NICI instrument (see Table 3) for a random sample of 28 patients from four clinical units (adult general medical-surgical unit, medical intensive care unit, and rehabilitation; pediatric intensive care and cardiology) and calculated the correlation between these scores with assessments made by a unit-based clinical nurses of the intensity of care needs (low/medium/high, scored as 1,2,3) for the same patients on the same day.

Table 4

Psychometric testing of Nursing Intensity of Care Index (NICI)

Results

The clinical nurse experts who examined the factors included in the NICI as well as the incremental time calculations and scoring completely concurred with the factors included and the nursing time allotted. They identified six additional factors not previously included in the tool that had an impact on nursing care intensity, and these factors were added to the tool. In terms of concurrent validity, the Spearman’s rho correlation coefficient between the NICI score and the clinical nurse scores for 28 patients was .94 (p<0.001). We also confirmed that there was variation across units and across time. For example, over a period of 1 month in one medical intensive care unit there were 239/358 (66.7%) patient days of isolation as compared with 188/1087 (17.3%) isolation days on one general medical unit.

Discussion

Extensive surveys to assess the relationship between staffing and patient outcomes have been conducted in numerous countries in the Americas, Europe, and Asia, and have consistently reported that lower nurse workloads and higher percentages of baccalaureate-prepared nurses are associated with lower mortality and readmissions rates5,6. A 2008 systematic literature review of the relationship between staffing and infections concluded that while methods and definitions varied, there was a consistent link between nurse staffing and healthcare-associated infections, with significantly increased rates associated with temporary nursing staff in four studies28. In a study of 2,675 infants admitted to neonatal intensive care units we found that a greater number of care hours provided by registered nurses was associated with a significant reduction in bloodstream infections (odds ratio: 0.21, 95% confidence intervals: 0.06–0.79)29. A scoping review of 45 studies regarding the relationship between nurse staffing and risk of infection was published in 2015, but because of a number of methodologic problems such as the use of imprecise administrative databases, how variables were measured, and time frames used to link staffing and infections, we found mixed results22. While many studies assess nurse staffing needs by using only the number of patients receiving care as the denominator, the intensity of nursing care in actual practice is multidimensional, affected not only by the number of patients on the unit but also their acuity, turnover rates, and other technical demands. In fact, nurse staffing rates which are not adjusted for patient turnover and patient severity have been found to underestimate nursing workload and overestimate nurse staffing levels12.

To further assess concurrent validity, we attempted to find other intensity of care tools with which we could correlate the NICI. In a literature review we found a few tools that had been developed with the goal of quantifying nursing care intensity. The Therapeutic Intervention Scoring System (TISS) developed in the 1970s classified nursing care intensity of patients in intensive care and has been subsequently adapted and simplified. This tool, however, may not be a reliable measure of nursing care needs at the individual patient level30,31. Several other scoring systems have been developed and tested but are often limited to intensive care32,33. A Finnish system for workforce planning seems to be rigorously tested but unfortunately is not available outside of Finland without a special license agreement and deployment project34. Other methods to measure nursing workload are available25,35, but none have used only electronically available data at the patient as well as the unit and/or systems level, nor do they make it possible to calculate scores with electronically available data in real time.

This project took advantage of natural variability in acute care practices and setting, that is, variability in nurse staffing and other organizational and systems factors, in a database which included >200,000 patient discharges from three hospitals. The parent database has been used to compare the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor infections in “real world” settings. Its purpose is to improve health outcomes and systems of care by developing and disseminating evidence-based information about the everyday effectiveness of interventions36–38. This is in contrast to designs such as randomized controlled trials in which the question is often whether the treatment is efficacious under highly controlled conditions. It is generally not feasible or ethical, however, to randomize patients to different interventions to reduce adverse events such as infection (e.g., randomize patients to barrier precautions or not or to receive care from staff with limited hand hygiene regimens). Because health services research is designed to examine actual conditions within systems of care, it often has greater external validity as well as being cost effective, and therefore holds great promise to advance the field of infection prevention and control.

Limitations and further considerations

Previous nursing workload formulas have been built on such theoretical underpinnings as nursing diagnosis, nursing intervention classifications, or time and motion studies13,25,39–41. In developing our current Index, however, we took a pragmatic approach with the goal of using data already available in the EHR to provide real-time estimates of nursing care intensity without adding to the nurse’s workload by requiring additional data collection or documentation.

There are clearly limitations to EHRs from the nursing perspective. While systems are designed to facilitate and speed documentation, fields are often limited to procedures or other biological parameters such as vital signs and ‘intake and output’. Many aspects of care provided by nurses such as patient and family education, discharge planning, and palliative care may be poorly captured and fail to reflect the time and resources required. In that regard, EHRs do not present a ‘holistic’ picture of nursing care requirements and may actually be detrimental if nurses assume that they should be spending their time primarily on what is required to be documented in pre-set data fields. Even when fields are available for recording such practices as palliative care or discharge planning, they may simply be recorded as a yes/no check box which provides no information about time requirements and little direction regarding patient needs. Hence, although a care intensity tool such as NICI is one step toward more accurately explicating nursing intensity, many important aspects of care are still missed in EHRs.

As with any electronic database, this research is limited by the retrospective nature of the data available and by the accuracy of what is recorded, hence the importance of carefully validated data sources. Another problem is that using data from such large and complex data sets requires sophisticated programming capabilities. While there are programmers working full-time in large healthcare systems, many smaller hospitals do not yet have the programming capability to complete the necessary data extraction and management.

Finally, while this testing included a large sample size, data were from just three hospitals in a single urban area and hence generalizability and representativeness may be of concern. Data collection was also limited to 2 years. Although the time tested (2012–13) was several years ago, it does not seem likely that the parameters included in the NICI have changed considerably. Despite this, items in any intensity tool would have to be re-evaluated on a regular basis as care practices evolve over time.

Conclusions

This project is an example of identifying ways to efficiently conduct research using administrative and other electronically available data already being collected for other purposes (i.e., no ‘extra’ data collection for the purpose of research). We describe the development and initial psychometric testing of the NICI designed to provide real-time data on nursing care needs and to incorporate individual patient-level data across a variety of acute care settings. The tool is now ready for further unit-specific and large-scale testing and is available to others upon request, although it will likely require modification based on the care needs and practices at individual institutions. The next steps will be to develop a discrete-event simulation (DES) model and conduct large-scale field trials to assess its accuracy and utility in this era when large volumes of electronically available data are being collected for billing and other clinical purposes can and should be efficiently used for multiple purposes. Finally, it is clear that EHRs are not currently capturing many important and time-consuming aspects of patient care which require considerable nursing time.

The application of information technology is rapidly expanding and EHRs are now the standard in the majority of acute care settings, but greater attention to accurate documentation of patient needs other than physical care and biologic maintenance is imperative. Nurses need to be actively involved in the development of EHRs rather than being ‘taught’ by them what is important to document.

Acknowledgments

This project was funded in part by the Agency for Healthcare Research and Quality, R01HS024915

Footnotes

Authors have no conflicts to disclose

Contributor Information

Elaine L Larson, School of Nursing, Professor of Epidemiology, Mailman School of Public Health, Columbia University.

Bevin Cohen, School of Nursing, Columbia University.

Jianfang Liu, School of Nursing, Columbia University.

Philip Zachariah, College of Physicians and Surgeons, Columbia University.

David Yao, Department of Industrial Engineering and Operations Research, Columbia University.

Jingjing Shang, School of Nursing, Columbia University.

References

1. Jha AK, DesRoches CM, Kralovec PD, Joshi MS. A progress report on electronic health records in U.S. hospitals. Health Aff (Millwood) 2010 Oct;29(10):1951–1957. [PubMed] [Google Scholar]

3. Brennan PF, Bakken S. Nursing Needs Big Data and Big Data Needs Nursing. J Nurs Scholarsh. 2015 Sep;47(5):477–484. [PubMed] [Google Scholar]

4. Aiken LH, Sloane DM, Clarke S, et al. Importance of work environments on hospital outcomes in nine countries. Int J Qual Health Care. 2011 Aug;23(4):357–364. [PMC free article] [PubMed] [Google Scholar]

5. Aiken LH, Sloane DM, Bruyneel L, et al. Nurse staffing and education and hospital mortality in nine European countries: a retrospective observational study. Lancet. 2014 May 24;383(9931):1824–1830. [PMC free article] [PubMed] [Google Scholar]

6. Cho E, Sloane DM, Kim EY, et al. Effects of nurse staffing, work environments, and education on patient mortality: an observational study. Int J Nurs Stud. 2015 Feb;52(2):535–542. [PMC free article] [PubMed] [Google Scholar]

7. Cohoon B. Causes of near misses: perceptions of perioperative nurses. AORN J. 2011 May;93(5):551–565. [PubMed] [Google Scholar]

8. Keers RN, Williams SD, Vattakatuchery JJ, et al. Medication safety at the interface: evaluating risks associated with discharge prescriptions from mental health hospitals. Journal of clinical pharmacy and therapeutics. 2015 Nov 3; [PubMed] [Google Scholar]

9. Geller NF, Bakken S, Currie LM, Schnall R, Larson EL. Infection control hazards and near misses reported by nursing students. Am J Infect Control. 2010 Dec;38(10):811–816. [PubMed] [Google Scholar]

10. Pereira PR, Isaakidis P, Hinderaker SG, et al. Burden of isolation for multidrug-resistant organisms in a tertiary public hospital in Southern Brazil. Am J Infect Control. 2015 Feb;43(2):188–190. [PubMed] [Google Scholar]

11. Gabbay U, Bukchin M. Does daily nurse staffing match ward workload variability? Three hospitals' experiences. International journal of health care quality assurance. 2009;22(6):625–641. [PubMed] [Google Scholar]

12. Unruh LY, Fottler MD. Patient turnover and nursing staff adequacy. Health services research. 2006 Apr;41(2):599–612. [PMC free article] [PubMed] [Google Scholar]

13. Rosenthal GE, Halloran EJ, Kiley M, Pinkley C, Landefeld CS. Development and validation of the Nursing Severity Index. A new method for measuring severity of illness using nursing diagnoses. Nurses of University Hospitals of Cleveland. Med Care. 1992 Dec;30(12):1127–1141. [PubMed] [Google Scholar]

14. Zhang Z, Chen K, Chen L. APACHE III Outcome Prediction in Patients Admitted to the Intensive Care Unit with Sepsis Associated Acute Lung Injury. PLoS One. 2015;10(9):e0139374. [PMC free article] [PubMed] [Google Scholar]

15. Navarra AM, Schlau R, Murray M, et al. Assessing nursing care needs of children with complex medical conditions: The Nursing-Kids Intensity of Care Survey (N-KICS) Journal of pediatric nursing. 2016 Jan 7; [PMC free article] [PubMed] [Google Scholar]

16. Kanerva M, Blom M, Tuominen U, et al. Costs of an outbreak of meticillin-resistant Staphylococcus aureus. J Hosp Infect. 2007 May;66(1):22–28. [PubMed] [Google Scholar]

17. The neglected dimension of global security: A framework to counter infectious disease crises. Washington DC: 2016. Commission on a Global Health Risk Framework for the Future; National Academy of Medicine. [Google Scholar]

18. Wilkinson AM, Matzo M. Nursing education for disaster preparedness and response. J Contin Educ Nurs. 2015 Feb;46(2):65–73. quiz 74-65. [PubMed] [Google Scholar]

19. Uppal N, Batt J, Seemangal J, McIntyre SA, Aliyev N, Muller MP. Nosocomial tuberculosis exposures at a tertiary care hospital: a root cause analysis. Am J Infect Control. 2014 May;42(5):511–515. [PubMed] [Google Scholar]

20. Chong C, Tan N, Yunos H, Lim S, Acharyya S, Thoon K. Temporal trend in the incidence of pertussis and exposures among healthcare workers: Descriptive report from a tertiary care hospital for children in Singapore. Journal of Hospital Infection. 2015;91(4):376–378. [PubMed] [Google Scholar]

21. Sfeir M, Munoz-Price LS. Scabies and bedbugs in hospital outbreaks. Curr Infect Dis Rep. 2014 Aug;16(8):412. [PubMed] [Google Scholar]

22. Shang J, Stone P, Larson E. Studies on nurse staffing and health care-associated infection: methodologic challenges and potential solutions. Am J Infect Control. 2015 Jun 1;43(6):581–588. [PMC free article] [PubMed] [Google Scholar]

23. Apte M, Neidell M, Furuya EY, Caplan D, Glied S, Larson E. Using electronically available inpatient hospital data for research. Clin Transl Sci. 2011 Oct;4(5):338–345. [PMC free article] [PubMed] [Google Scholar]

24. Neidell MJ, Cohen B, Furuya Y, et al. Costs of healthcare- and community-associated infections with antimicrobial-resistant versus antimicrobial-susceptible organisms. Clin Infect Dis. 2012 Sep;55(6):807–815. [PMC free article] [PubMed] [Google Scholar]

25. de Cordova PB, Lucero RJ, Hyun S, Quinlan P, Price K, Stone PW. Using the nursing interventions classification as a potential measure of nurse workload. J Nurs Care Qual. 2010 Jan-Mar;25(1):39–45. [PMC free article] [PubMed] [Google Scholar]

26. D'Hoore W, Bouckaert A, Tilquin C. Practical considerations on the use of the Charlson comorbidity index with administrative data bases. J Clin Epidemiol. 1996 Dec;49(12):1429–1433. [PubMed] [Google Scholar]

27. D'Hoore W, Sicotte C, Tilquin C. Risk adjustment in outcome assessment: the Charlson comorbidity index. Methods of information in medicine. 1993 Nov;32(5):382–387. [PubMed] [Google Scholar]

28. Stone PW, Pogorzelska M, Kunches L, Hirschhorn LR. Hospital staffing and health care-associated infections: a systematic review of the literature. Clin Infect Dis. 2008 Oct 1;47(7):937–944. [PMC free article] [PubMed] [Google Scholar]

29. Cimiotti JP, Haas J, Saiman L, Larson EL. Impact of staffing on bloodstream infections in the neonatal intensive care unit. Arch Pediatr Adolesc Med. 2006 Aug;160(8):832–836. [PMC free article] [PubMed] [Google Scholar]

30. Clermont G, Angus DC, Linde-Zwirble WT, Lave JR, Pinsky MR. Measuring resource use in the ICU with computerized therapeutic intervention scoring system-based data. Chest. 1998 Feb;113(2):434–442. [PubMed] [Google Scholar]

31. Dickie H, Vedio A, Dundas R, Treacher DF, Leach RM. Relationship between TISS and ICU cost. Intensive Care Med. 1998 Oct;24(10):1009–1017. [PubMed] [Google Scholar]

32. Debergh DP, Myny D, Van Herzeele I, Van Maele G, Reis Miranda D, Colardyn F. Measuring the nursing workload per shift in the ICU. Intensive Care Med. 2012 Sep;38(9):1438–1444. [PubMed] [Google Scholar]

33. Iapichino G, Mistraletti G, Corbella D, et al. Scoring system for the selection of high-risk patients in the intensive care unit. Crit Care Med. 2006 Apr;34(4):1039–1043. [PubMed] [Google Scholar]

34. Fagerstrom L, Lonning K, Andersen MH. The RAFAELA system: a workforce planning tool for nurse staffing and human resource management. Nurs Manag (Harrow) 2014 May;21(2):30–36. [PubMed] [Google Scholar]

35. Kwiecien K, Wujtewicz M, Medrzycka-Dabrowska W. Selected methods of measuring workload among intensive care nursing staff. International journal of occupational medicine and environmental health. 2012 Jun;25(3):209–217. [PubMed] [Google Scholar]

36. Report to the President and the Congress. Washington, DC: US Dept. of Health and Human Services; 2009. Federal Coordinating Council for Comparative Effectiveness Research (U.S.), United States. President., United States. Congress., United States. Dept. of Health and Human Services. [Google Scholar]

37. Iglehart JK. Prioritizing comparative-effectiveness research--IOM recommendations. N Engl J Med. 2009 Jul 23;361(4):325–328. [PubMed] [Google Scholar]

38. Volpp KG, Das A. Comparative effectiveness--thinking beyond medication A versus medication B. N Engl J Med. 2009 Jul 23;361(4):331–333. [PubMed] [Google Scholar]

39. Carmona-Monge FJ, Rollan Rodriguez GM, Quiros Herranz C, Garcia Gomez S, Marin-Morales D. Evaluation of the nursing workload through the Nine Equivalents for Nursing Manpower Use Scale and the Nursing Activities Score: a prospective correlation study. Intensive Crit Care Nurs. 2013 Aug;29(4):228–233. [PubMed] [Google Scholar]

40. Myny D, Van Hecke A, De Bacquer D, et al. Determining a set of measurable and relevant factors affecting nursing workload in the acute care hospital setting: a cross-sectional study. Int J Nurs Stud. 2012 Apr;49(4):427–436. [PubMed] [Google Scholar]

41. Myny D, Van Goubergen D, Limere V, Gobert M, Verhaeghe S, Defloor T. Determination of standard times of nursing activities based on a Nursing Minimum Dataset. J Adv Nurs. 2010 Jan;66(1):92–102. [PubMed] [Google Scholar]