Which factor would strongly influence the reliability of an EHRs data repository?

Stud Health Technol Inform. Author manuscript; available in PMC 2018 Jan 5.

Published in final edited form as:

PMCID: PMC5755623

NIHMSID: NIHMS928215

Abstract

There has been increasing recognition of the key role of social determinants like occupation on health. Given the relatively poor understanding of occupation information in electronic health records (EHRs), we sought to characterize occupation information within free-text clinical document sources. From six distinct clinical sources, 868 total occupation-related sentences were identified for the study corpus. Building off approaches from previous studies, refined annotation guidelines were created using the National Institute for Occupational Safety and Health Occupational Data for Health data model with elements added to increase granularity. Our corpus generated 2,005 total annotations representing 39 of 41 entity types from the enhanced data model. Highest frequency entities were: Occupation Description (17.7%); Employment Status – Not Specified (12.5%); Employer Name (11.0%); Subject (9.8%); Industry Description (6.2%). Our findings support the value for standardizing entry of EHR occupation information to improve data quality for improved patient care and secondary uses of this information.

Keywords: Occupations, Social Determinants of Health, Electronic Health Records

Introduction

The need to accurately capture occupation and other social history information is central for the use of this information in the provision of direct clinical care and for secondary uses like research and risk stratification of patients [1]. As electronic health record (EHR) system use increases within healthcare organizations, there is an opportunity for improved capture of this information at the point of care. Within the United States, the National Academy of Medicine (NAM) in their 2011 report “Incorporating Occupational Information in Electronic Health Records” advocates for inclusion of social factors in EHR systems, due to the important impact of social factors on health status and outcomes [2]. Similarly, Phases 1 and 2 of their 2014 reports “Capturing Social and Behavioral Domains and Measures in Electronic Health Records” [3, 4] further emphasize the importance of these factors. The National Institute for Occupational Safety and Health (NIOSH), a leader in efforts to qualify occupation information in the United States, has done extensive work to promote documentation of occupation information in a standard manner by creating the Occupational Data for Health (ODH) data model [5].

Prior work has been done to analyze broadly how social history information is captured in EHR systems [6, 7], public health datasets [8], and how occupation-related information appears within standards and within social history in the EHR [9, 11]. More specifically, using a top-down approach, Rajamani et al. examined reports, standards, surveys, and research measures to analyze the ODH model’s ability to provide coverage for occupation-related information [9], including the Health Level Seven Clinical Document Architecture content module of the Integrating the Healthcare Enterprise Patient Care Coordination Technical Framework, which incorporates the ODH model [10]. Aldekhyyel et al. examined the content and quality of entries within the free-text occupation field of the Fairview Health Services enterprise EHR social history module [11]. The results of these studies provided a comprehensive overview of the current state of representation and standardization of occupation information. Collectively, they highlight that the ODH model is robust in coverage of occupation-related information, and that the content and quality of occupation information within the EHR can be inconsistent and variable.

The main objective of this study was to build upon prior efforts of this group of authors by looking at a range of free-text clinical sources to further inform occupation representations leveraging the NIOSH ODH model. In performing our analysis, we anticipate additional refinements thus enabling future standardization and additional insights into language around occupation, ultimately aiding in standards refinement and future natural language processing (NLP) efforts around occupation information.

Methods

Data Sources

This study utilized six clinical document sources comprising a mix of both publically available and local sources to analyze free-text mentions of occupation and related information within notes. Information was unstructured, yielding a large variety of occupation-related information. The public domain note sources used came from 491 “Consult – History and Physical” notes from MTSamples (MTS) [12] and 200 de-identified “H&P” (History and Physical Examination) notes from the University of Pittsburgh Medical Center (UPMC) [13] obtained via a data use agreement. Four document types were analyzed from the University of Minnesota (UMN)-affiliated Fairview Health Services Epic EHR: (1) Social History Documentation, (2) Social Work notes, (3) Physical Therapy notes, and (4) Occupational Therapy notes available through the University of Minnesota Clinical Data Repository (CDR) from 2013.

Analysis of Clinical Text

Clinical text analysis consisted of three main parts: (1) text de-identification, (2) schema creation, and (3) text annotation. Sentences from Fairview Health Services were anonymized prior to annotation using the Safe Harbor Method [14]. Codes for anonymizing data came from previous work [15]. Further obfuscation was used where necessary to avoid personally identifiable data, and descriptors were created to provide specificity regarding employer type. These included descriptors for level of government employers, hospitality industry employers, healthcare employers, education employers, and Fortune 500 employers. Fortune 500 companies were determined using the Fortune 500 yearly ranking for 2016 [16]. Any remaining companies that did not fit into more specific descriptors were anonymized to “company.”

Annotations were made using the brat rapid annotation tool (BRAT) [17]. Annotation schema and guidelines were derived from the NIOSH ODH model categories and elements, providing the parent entities Occupational History, Usual Occupation, Employment Status, Occupational Injury, and Occupational Exposure and associated child entities. The Systemized Nomenclature of Medicine-Clinical Terms (SNOMED CT) [18] was examined, but it was determined that the ODH model was more relevant for our corpora, given its usage in previous work by this group of authors [9]. An initial set of 25 sentences was annotated with a group of annotators, and 50 overlapping sentences were individually annotated to evaluate the annotation schema and guidelines (Figure 1) by three annotators (EL, SR, NM). Additional entities were created to increase coverage, including Subject, Negation, Temporal, Occupational Conditions, and Occupation Status. The final schema had 9 parent entities and 32 child entities, for a total of 41 elements (Table 1).

Overview of Assessment of Occupation Information in Clinical Text

Table 1

Distribution of occupation entities across sources.

Annotators were instructed to annotate at the most specific level of detail. For example, in the sentence, “She is a registered nurse,” the text “registered nurse” is annotated as the child entity Occupation Description, rather than the parent entity Occupational History. Parent entities were used as general annotation categories when a child entity could not be specified. In another example, “Prior to retirement, pt worked as a civil engineer,” the sentence contains annotations relating to both clauses (i.e., “Prior to” is Temporal – Time Frame; “retirement” is Employment Status – Retired; “worked” is Employment Status – Not Specified; “civil engineer” is Occupational Description). Relationships were created to describe how entities are connected to each other. An overlapping set of 10% of sentences was annotated to calculate inter-rater reliability between three annotators (achieving a Cohen’s kappa of 0.76 and proportion agreement of 0.95). When inter-rater reliability was ascertained, the remaining sentences were annotated using the most expanded version of the schema. Following annotation process, annotations were extracted by element, generating a list of values for each element. These values were grouped by similar meaning for highest frequency elements. For example, for Occupation Description, the group ‘Legal Occupations’ represents annotations ‘paralegal,’ ‘attorney,’ and ‘tax attorney.’

Results

A total of 868 sentences from the six sources of clinical documents were annotated, yielding 2,005 annotations, which were mapped to 41 entities. The most frequent entities were: Occupation Description (17.7%), Employment Status – Not Specified (12.5%), Employer Name (11.0%), Subject (9.8%), and Industry Description (6.2%). Table 1 summarizes the representation of entities across source type. The Fairview Social History Documentation sentences have the greatest variety of occupation information, containing 39 (95%) of 41 possible entities. The sentences from the Fairview Physical Therapy Notes have the least diversity in occupation-related information, containing annotations for 13 (32%) of the 41 possible entities. No annotations were made for the entities Volunteer and Usual Occupation.

Tables 2 through ​5 provide summaries of value sets for the most commonly represented elements, demonstrating how elements were represented in clinical text. Table 2 summarizes the value sets for the most common entity, Occupation Description. Values are grouped according to the Bureau of Labor Statistics 2010 Standard Occupation Classification major groups [19]. Of the major groups in SOC, 22 of 23 are represented across sources. Military Specific Occupations is the only group not represented within these clinical sources, due to its small presence within the corpus, for Occupation Description annotations. Healthcare Practitioners and Technical Occupations are most common with 73 overall values (20.6%) and 51 unique values. Farming, Fishing, and Forestry Occupations are the least commonly represented, with 1 entry (0.3%).

Table 2

Distribution of values for Occupation Description element with grouping based on the 2010 Standard Occupation Classification groups [16]

Table 5

Distribution of values for Subject element

The entity Employer Status – Not Specified is a child entity of Occupation Status that is heavily used through the annotation process, because annotation guidelines specified to annotate all tenses of the verb “work” as Employment Status – Not Specified. This was done to capture work behaviors when part-time or full-time employment status is not directly stated (Table 3). Consequently, “Work” is the most frequent group in this data set, with 214 total values (86.3%) and only 6 unique values. The “Other” group included a variety of other terms indicating work status, including “released to regular work” and “has been on light duty.”

Table 3

Distribution of values for Employment Status – Not Specified element

Table 4 summarizes the value sets for the entity Employer Name. All values from Fairview Health Services Epic EHR in this set were anonymized prior to annotation, and are grouped accordingly. Educational Services was the most frequently occurring group with 83 entries (37.6%). Student status is a contributing factor to this result, as schools were annotated as Employer Name. Military employment is least commonly seen with 1 entry (0.5%). Five entries (2.3%) did not provide enough information in order to group by employer type.

Table 4

Distribution of values for Employer Name element

Table 5 summarizes the value sets for the entity Subject. This entity was added to the annotation schema after initial evaluation when a need for granularity was discovered. Parent group entries are most frequent with 137 total entries (74.5%) and 13 unique entries. Names of individuals and references to spouses were also common – de-identified prior to annotation as “Name” – 15.2% and 7.1% respectively.

Discussion

This study’s findings demonstrate the wide variety and range of occupation-related information within EHR clinical texts. A total of 2,005 annotations were made for occupation-related information in 868 sentences from six clinical document sources. This pervasiveness of occupation-related information within free-text notes points to the need for standardized entry within the EHR. In large part, the NIOSH ODH model proves appropriate for representing most of the occupation information, but several additions to the schema were necessary for comprehensive representation in our clinical text corpus. Specifically, the Subject entity is one of the most frequently occurring entities (9.8%), and was added during initial schema evaluation when need was assessed. Within this element, parent group entries are highly used, demonstrating a clinical interest in parental occupation and a prevalence of pediatric patients in this dataset. This correlation could also point to the relevance of a parent’s occupation on a child’s health outcomes. Similarly, the prevalence and interest in spousal occupations within this dataset points to the relevance of a spouse’s occupation to an individual’s health outcomes.

The Fairview Social History Documentation notes represent the largest portion of the dataset, with 553 sentences and 1,295 annotations. This source also presents the greatest coverage of occupation-related elements, with 39 of 41 possible elements across the dataset. Volunteer and Usual Occupation (parent entity) are the two elements not represented in the dataset; these elements were also not seen in any other source. The Fairview Physical Therapy Notes present the least coverage of elements with 13 of 41 possible elements. The Fairview Occupational Therapy Notes and Fairview Social Work Notes also present a more focused coverage of elements with 17 and 23 elements, respectively. This could be due to the specialty nature of these notes and more focused reasons for obtaining occupation information. The MTS sentences presented the greatest coverage of elements per sentence annotated with 35 elements represented in 171 sentences.

The addition of the element Occupational Conditions was useful in describing daily stressors of an individual’s job that have noted long-term effects on health outcomes, but do not necessarily constitute injury or exposure. Among these are items such as “Pt describes standing 12 hour days on concrete for his job,” “Patient works at a computer and on the phone all day,” and “He is employed in sales, which requires quite a bit of walking, but he is not doing any lifting.” While Occupational Conditions annotations comprise only 1.1% of total annotations, there were zero duplicate entries, indicating the scope of conditions – both physical and mental demands – individuals face in their occupations.

Occupation Description is the most frequent element seen in this dataset with 354 total values in 868 sentences. Among these values, 267 were unique, and all but one were categorized into 22 of 23 SOC major groups. The ability to classify a large number of unique entries into 22 groups proves that occupation descriptions could benefit from standardized entry of this information within the EHR. While value sets are wide reaching, many occupations fall into a smaller set of groups. Employer Name was also pervasive in the dataset with 221 total values. These were categorized into 7 groups based on anonymized codes.

We also observed that ambiguity was common within the Occupation Status related entities, such as Self-Employed and Homemaker/Housewife. For example, some individuals may hold more than one type of employment status (i.e., both a full-time position and a part-time position). Some individuals may have overlapping employment status (i.e., the term “stay at home mom” implies both a caretaker and homemaker role). Self-employment also presents ambiguity, as this could refer to either full-time or part-time employment status in addition to self-employment.

The fifth most frequent element within the dataset was Industry Description (6.2%). This value set was not grouped by any classification set, because a large majority of entries were unique and presented varying degrees of information presented. The issue of discrepancies in information is best represented by Industry Description, but was prevalent throughout the dataset. This underscores the lack of standardization currently in place in documenting occupation information within the EHR, and the need for further work to characterize the wealth and variety of occupation information that has potential impacts on health status and outcomes.

This work builds off of previous work [8, 9] to identify how occupation information is represented across the literature and various aspects of the EHR. Work could be done involving comparisons against occupation information in EHR systems from other vendors. Future work will focus on dissemination of this research on the Brown Digital Repository [20], creating granularity within the current model, and lead to better NLP techniques to analyze occupation information. This will contribute towards standardized entry of occupation information within the EHR, promoting data quality and ultimately improving patient care and secondary use of occupation information.

Conclusion

As EHR system use becomes more widespread, it will become imperative to have standardized entry of factors that influence health status and outcomes. Several respected groups have recognized occupation as a factor of health status – with this recognition comes the need to understand and standardize how occupation-related information is being captured within the EHR. This study analyzed free-text clinical notes from a variety of sources in order to characterize the state of occupation-related information within the EHR. The NIOSH ODH data model proved robust in characterizing information content, and additions were made to the annotation schema to provide additional granularity. This work has potential to lead to more detailed and knowledgeable standards, and provides a basis for creating a standardized entry system within the EHR and improved NLP techniques.

Acknowledgments

This work was supported in part by National Library of Medicine grant R01LM011364.

References

1. Behfourouz HL, Drain PK, Rhatigan JJ. Rethinking the Social History. N Engl J Med. 2014;371(14):1277–1279. [PubMed] [Google Scholar]

2. National Academy of Medicine (NAM) Washington, DC: 2011. [cited 2016 December 9]. Incorporating Occupational Information in Electronic Health Records: Letter Report [Internet] Available from: http://www.nap.edu/catalog.php?record_id=13207. [Google Scholar]

5. National Institute for Occupational Safety and Health (NIOSH) Structuring Patient Work Information in EHRs to Improve Patient Care and Public Health; Occupational Data for Health (ODH) Model, Public Health Informatics Conference; Atlanta, GA. 2014. [Google Scholar]

6. Chen ES, Manaktala S, Sarkar IN, Melton GB. A multi-site content analysis of social history information in clinical notes. AMIA Annu Symp Proc. 2011:227–36. [PMC free article] [PubMed] [Google Scholar]

7. Winden TJ, Chen ES, Lindemann E, Wang Y, Carter EW, Melton GB. Evaluating living situation, occupation, and hobby/activity information in the electronic health record. AMIA Annu Symp Proc. 2014:139. [Google Scholar]

8. Melton GB, Manaktala S, Sarkar IN, Chen ES. Social and Behavioral History Information in Public Health Datasets. AMIA Annu Symp Proc. 2012:625–634. [PMC free article] [PubMed] [Google Scholar]

9. Rajamani S, Chen ES, Aldekhyyel R, Wang Y, Melton GB. Validating the Occupational Data for Health Model: An Analysis of Occupational Information in Reports, Standards, Surveys, and Measures. AMIA Annu Symp Proc. 2016:115–116. [Google Scholar]

10. Health Level Seven (HL7) – IHE PCC Technical Committee. Integrating the Healthcare Enterprise (IHE) Patient Care Coordination (PCC) Technical Framework Supplement – CDA Content Modules: Trial Implementation Guide [Internet] [cited 2017 April 6]; Available from: http://www.ihe.net/uploadedFiles/Documents/PCC/IHE_PCC_TF_Vol1.pdf.

11. Aldekhyyel R, Chen ES, Rajamani S, Wang Y, Melton GB. Content and Quality of Free-Text Occupation Documentation in the Electronic Health Record. AMIA Annu Symp Proc. 2016:1708–1716. [PMC free article] [PubMed] [Google Scholar]

12. MTSamples. [cited 2016 December 9];MTSamples.com [Internet] Available from: http://mtsamples.com.

13. University of Pittsburgh NLP Repository. Department of Biomedical Informatics, University of Pittsburgh. [Internet] [cited 2016 December 9]; Available from: http://www.dbmi.pitt.edu/nlpfront.

17. Brat: Brat rapid annotation tool [Internet] [cited 2016 December 9]; Available from: http://brat.nlplab.org.

19. Bureau of Labor Statistics. The 2010 Standard Occupational Classification System [Internet] [cited 2016 December 9]; Available from: http://www.bls.gov/soc/major_groups.htm.