Prevalence of diabetes

Impacts of diabetes

II. Activities currently underway in developing diabetes quality of care measures

1. National Committee for Quality Assurance

2. American Diabetes Association

3. American Academy of Family Practice

4. HCFA diabetes work group

III. Measurement approach

The principles of the diabetes measurement recommendations

Description of the measurement areas

Optimal vs. good measures

Measures recommended for diabetes

Recommended protocols for data collection

IV. Conclusion

Appendix A

Appendix B

References

Prevalence of diabetes

Diabetes mellitus is a life-long chronic disease in which the body does not produce or respond to insulin. It is not currently curable. Over 14 million people in the United States (approximately five to six percent of the population) have diabetes, although only half this number are aware they have the disease. Non-Insulin Dependent Diabetes Mellitus (NIDDM) is the most prevalent type of metabolic disorder, affecting up to ten percent of the elderly population. Diabetes is also considered one of the most costly diseases in the country.(1)

Impacts of diabetes

Quality of life for persons with diabetes is affected by both their high blood sugar and the regimen required to control it. This regimen includes adherence to a plan of strict diet, exercise, and weight control, as well as use of medications or injections of insulin. Diabetes has both acute and long-term complications, including effects on kidney function, eyesight, cardiovascular function, and extremity disease. There is accumulating evidence that effective control of blood sugar can prevent or delay onset of some of these long-term complications.(2) Early and effective management of complications can retard progression of the disease.(3,4)

For purposes of this discussion, both Insulin Dependent Diabetes Mellitus (IDDM) and Non-Insulin Dependent Diabetes Mellitus (NIDDM) are included. Treatment for these two disorders is often similar, and complications are similar in kind, if not in incidence. Gestational diabetes is not included in this discussion, as it is a transitional state and identification of the population would follow a different path. In general, identification of persons with either IDDM or NIDDM needs to be done through a combination of administrative data (by diagnosis), laboratory test data, and pharmacy claims. Laboratory tests, such as Hemoglobin A1c (HbA1c), and pharmacy records of insulin or oral medications are fairly specific to diabetes. These sources can be supplemented by survey data (for example, for patients new to a plan or provider system). (See Appendix A for diagnostic criteria.)

II. Activities currently underway in developing diabetes quality of care measures

To evaluate the quality of care delivered to the American patient with this disease, several bodies are deliberating to develop measures:

1.  National Committee for Quality Assurance

The National Committee for Quality Assurance (NCQA) is developing a new version of the Health Plan Employer Data and Information Set (HEDIS).(5) A scientific and comprehensive framework will be used in selecting measures for HEDIS 3.0, based on the deliberations of its Technical Advisory Committee. Rating criteria are available from NCQA. This committee, led by Dr. David Eddy, has undertaken a critique of HEDIS 2.0, and has developed a method to score the measures along the three general dimensions of relevance, scientific support and feasibility, with multiple sub-dimensions of each of these general categories. HEDIS 2.0 and 2.5 contained one process measure relating to diabetic care; the annual rate of diabetic retinal exam. HEDIS 3.0 will not include a comprehensive set of measures for diabetes.(5)

2. American Diabetes Association

The American Diabetes Association (ADA) provides a comprehensive set of measures through its Provider Recognition Committee. Their measures derive from clinical opinion, reflect the thinking of the modal specialists, diabetologists, but are not as grounded as the HEDIS measures are in the quality of care principles that define a useful measure. Further, ADA measures emphasize physiologic indicators and concentrate less on recent trends guiding quality of care assessment in HMOs, such as a heavy reliance on the patient as rater and reporter, and using cost as well as effectiveness as a factor in the determination of what represents optimal quality of care.5

3. American Academy of Family Practice

Dr. Barbara Fleming of the Health Care Financing Administration (HCFA) has recently put together a diabetes policy initiative with the American Academy of Family Practice (AAFP). The orientation of the generalist physician is practical and focused on realistic expectations of what can be done in short office visits under considerable restraints.

The principles, perspectives and philosophies underlying each of the three groups (managed care, diabetologists and family practitioners) differ, but common ground exists.

4. HCFA diabetes work group

A fourth group, the Work Group on Standardization of Diabetes Definitions, Performance Measures, Quality of Care and Outcomes, also formed by Dr. Fleming of HCFA, will provide valuable summaries of diabetes quality of care measures.(7)

Each of these initiatives brings a slightly different perspective: HEDIS is a data set intended for measurement of quality of care provided by pre-paid health maintenance organizations (HMOs). The ADA represents patients as well as providers. The AAFP work has the priorities of primary care physicians as its focus. The HCFA work group concentrates on the older diabetic patient who is covered by Medicare.

This paper will attempt to adopt all the above perspectives in reviewing measures to determine the quality of care for patients with diabetes. It will embrace the HMO-oriented scientific approach of HEDIS but, at the same time, be comprehensive enough to capture the level of clinical detail required by experts in this field. A primary care perspective will also be evident, with a managed care orientation. Thus, instead of starting with measures, this paper will begin with guiding principles so that the agreements and disagreements over recommendations can be seen in light of the point of view presented, rather than be rooted in disciplinary protection and ownership. Because so much thoughtful work has already been done by these groups and others, the goodness of the recommendations herein will rest on a synthesis of points of view, rather than on the background research into each individual quality measure.

The principles of the diabetes measurement recommendations

The measures suggested in this paper have been selected on the basis of a number of guiding principles. These principles, which should themselves be debated because the inclusion of measures depends on their acceptance or rejection, are:

Principle one: different measures for different types of diabetes

There will have to be different measures for different types of patients with diabetes. There is a common set of processes and outcomes (e.g., glycemic control reduces microvascular complications in every type of diabetes), but the cost effectiveness of the processes and the strength of evidence varies for different subgroups. Thus, episodes of ketoacidosis are minimally relevant for non-insulin treated diabetics; frequent, yearly, eye exams are useful for Type I diabetics after 5 years, for high risk Type 11 diabetics (Hawaiians, Native Americans, possibly Afro-Americans, young patients less than 55 years) but can be much less frequent for elderly, late onset diabetics. Vijan and colleagues at the University of Michigan have explored the impact of lowering blood sugar levels in NIDDM patients to DCCT levels. They found that the benefits of very strict glucose control were significantly less in the older patient with NIDDM, due to competing risk from heart disease and other comorbidities.(8) The option to screen these patients less frequently and to treat emerging disease when it is detected should be considered.

Because of the unifying biology of diabetes, there has been a reluctance to define subgroups, but identifying them would greatly assist the process of selecting measures. The American Diabetes Association has recommended that the terms Insulin Dependent Diabetes Mellitus and Non Insulin Dependent Diabetes Mellitus be replaced by the terms Type I Diabetes Mellitus and Type 11 Diabetes Mellitus. This terminology is helpful clinically and will be used in some of the measurement descriptions in this paper.

We would propose an additional sub-division of adult diabetic patients on the basis of age. Age is a key factor in the management and outcome of diabetes. Because of competing comorbidities including macrovascular disease that may not be reduced by tight control in older patients, tight glucose control may be unjustified. These patients may have poor outcomes and quality of life that has greater dependence on their heart disease and other illnesses than on their diabetes.(9,10)

Therefore, a distinction between patients under 65 years of age and 65 years of age or older seems appropriate, particularly as this cut off matches the Medicare insurance eligibility age. There is also a distinct group of young diabetics, age up to around 45 years who, regardless of Type I or Type II Diabetes Mellitus, should be managed aggressively.

This leaves a middle group: patients aged from around 45 to 65 years. These patients often have comorbid conditions that may impact their general health and quality of life. Aggressive treatment may not be appropriate for this group who may never the less benefit from 'moderate' control. For the purposes of this paper this third group will be defined as those patients aged 45 to 64. This matches the criteria used by the National Diabetes Data Group to describe prevalence of diabetics of different ages (see Table 1 ) and the percent of the total represented by each age group (see Table 2). Treatment recommendations for this group are less well defined than for the young and elderly diabetics. In general they represent a middle ground between the other two groups.

This argument therefore creates the following three groups:

i) diabetic patients 18 to 44 years of age;

ii) diabetic patients 45 to 64 years of age;

iii) diabetic patients 65 years of age and older, many of whom will have considerable comorbidity and have low risks for microvascular disease over their remaining lifetime.

Measures should be reported by these three age groups, regardless of treatment, unless otherwise indicated. Age groups may be identified by a combination of ICD-9 codes and demographic information. Pharmacy information may be used to determine treatment where appropriate.

Table 1. Prevalence of diagnosed diabetes (thousands) according to type of diabetes, U.S., 1992.

Source: Harris M.I. Classification criteria, and screening for diabetes. In Diabetes in America, National Diabetes Data Group. Second Edition, National Institutes of Health, 1995.

Table 2. Percentage of total diagnosed diabetes according to type of diabetes, U.S., 1992.

Source: Harris M.I. Classification criteria, and screening for diabetes. In Diabetes in America, National Diabetes Data Group. Second Edition, National Institutes of Health, 1995.

Principle two: measures of treatment and prevention should be included

Since diabetes is a multi-system disease, each complication has to be treated separately, and often divided into two parts, treatment and prevention. This is a more important issue for NIDDM than IDDM, where sugar control is the common source of essentially all complications. In NIDDM, heart disease, kidney disease and other complications have multiple etiologies.

Principle three: consider short and long term outcomes

The short term has to be considered in providing high quality of care, as well as the long term. Often there has been a solitary focus on preventing the microvascular and macrovascular complications of diabetes, with less emphasis on day to day quality of life, compliance, satisfaction with care, and costs of care. Both short term and long term outcomes are important.

Principle four: multiple sources of data will be required

To implement a comprehensive set of measures, multiple sources of data will be required. These will include chart review, administrative data sets for reliable laboratory information and utilization, and patient reports of quality of life, satisfaction, symptoms, and certain processes, such as foot examinations, which are not found routinely in the records. It has recently been shown that patients can reliably report the severity, not just the presence, of diabetic complications and comorbidities.(9,10) The use of multiple data sources allows emphasis on both processes and outcomes, and outcomes of different types, including physiologic, functional and mortality.

Principle five: risk adjustment will be necessary

Some type of risk adjustment or severity adjustment will be necessary, not only for outcomes, such as functional status, which is now obvious, but also for processes. Greenfield et al., in their PORT study, have shown that blood sugar is not as well controlled in NIDDM patients with major comorbid disease as it is in those free of coexistent disease. Careful risk adjustment is necessary for any outcome comparison of patients with Type II diabetes.(11,12)

Principle six: process and outcome measures should be included

The dynamic tension and complementarily between processes and outcomes must be appreciated: it may be good practice to require four glycated hemoglobin (HbA1c) measurements per year, but it should be remembered that simply performing a test does not lead directly to good control. Similarly, patient education is a universal goal, but just transmitting information and urging patients to comply does not mean that the quality of the effort was high, nor that it leads to better compliance. Conversely, outcomes alone, especially over one to two years, would not reflect care for the long term complications of diabetes.

Principle seven: individual provider feedback may be possible

Although not emphasized in this report, it now may be possible to profile physicians at the individual or small group or clinic level based on research findings from the NIDDM PORT. The number of patients per physician, the necessary precision of the functional status, satisfaction measures and process measures, and the ability to perform case mix adjustment have opened the possibility of providing interpretable information at the individual physician level.

Description of the measurement areas

The dimensions that should be included in assessing quality of care for diabetics are:

• glycemic control (blood sugar levels, episodes of hyper and hypoglycemia, diet and weight, monetary and nonmonetary effects of treatment)
• eye disease
• kidney disease
• heart disease, lipid control, blood pressure control; peripheral vascular disease; peripheral neuropathy
• general health maintenance (such as smoking)
• costs of care
• satisfaction

Attention to all of these aspects of the disease would represent the highest quality of care consistent with the state of the art of science and health care. These will be addressed in turn below, drawing on the deliberations of the NCQA, of the ADA Provider Recognition Committee, the HCFA sponsored American Academy of Family Physicians work group and the Work Group on the Standardization of Diabetes Definitions, Performance Measures, Quality of Care and Outcomes.

The name of the measure, definition of the measure, the justification for the measure, the source of data to obtain it, and the values are included where possible. The source of much of the detail supporting the measures comes from the ADA Provider Recognition Committee and the HCFA Work Group on Standardization.

Optimal vs. good measures

The organization of the measures recommendation section will be as follows: for each dimension (glycemic control, kidney disease, etc.) there will be a rationale or justification followed by optimal measures (more expensive, less validated) and good measures (more accepted, more feasible). Optimal measures represent the ideal criteria that could be used to assess the quality of care. Many are outcome measures. It may not be feasible to use all of these in the short term. The good measures are those that are currently accepted, and are feasible for immediate use. The measure will be labeled regarding its appropriateness for reporting on the three age groups: young patients aged 18 to 44, patients 45 to 64, and older patients 65 years and over.

Hemoglobin A1c (HbA1c ) is the best measure of overall glycemia, integrating the level of control over the previous 8-12 weeks. Many studies, including the Diabetes Control and Complications Trial (DCCT), have shown that the mean HbA1 c over a period of time correlates closely with the rate of appearance and progression of microvascular and neuropathic complications.2 The HbA1 c can therefore be regarded as a valid intermediate endpoint. HbA1c values may be used to compare both within and between populations to identify the range and average glycemic control.

The requirement for tight control may vary among the three age groups outlined above. In both IDDM and NIDDM, glycemic control prevents microvascular complications but because of reduced life expectancy, comorbid disease, treatment induced reduction in quality of life, and costs, the salience of reducing HbA1c to DCCT levels has been challenged.9 There is no strong evidence that glycemic control reduces cardiac and other macrovascular complications. Since the macrovascular complications are four to six times or more as frequent in the elderly, and since many of the elderly will also have competing comorbid diseases, controversy swirls around the level of tight control necessary for the elderly patients. However, even for this group, which may represent up to 50% of all diabetic patients, there is agreement that high (for example HbA1c >10) levels do not reflect good quality of care.

Optimal:

A) Yearly HbA1C levels, adjusted for comorbidity, obesity, and socio-economic status.

National practice based norms are available.(11,12,13)

FACCT domain: Clinical outcome

Name: Most Recent HbAlC (or glycated hemoglobin) Value

Definitions: Most recent value for HbA1c, reported by subgroup. Optimally, two values would be available for each patient, one at the beginning, and one later in the enrollment or study period. Any improvement in control could then be detected by longitudinal follow up.

Location of data: Lab data or patient's chart

Attribute values: numeric

There is a national initiative underway to standardize all measurement of glycated hemoglobin to the HbA1c assay used in the DCCT (which had a normal range of 3.8-6.05%). Until this is in widespread use, plans may need to supply the normal range for their laboratories, along with the intra- and inter-assay coefficients of variation.

Lower adjusted levels should be achieved for group i (less than or equal to 7%), and group ii (less than or equal to 9.5%), than for group iii (less than or equal to 10%).

Subgroups: i-iii

Report: Distribution of values by age group or rate, i.e., % of diabetics with HbA1c above accepted normal range for each age group

B) Combined ER, hospital records, chart or survey data for episodes and symptoms of hypoglycemia.

FACCT domain: clinical outcome

Name: Frequency of Severe Hypoglycemia

Definitions: How often in the past year has the patient experienced an episode of hypoglycemia which was so severe that it required the help of someone else to treat the patient (for example, where a third party had to give glucagon, intravenous glucose, call 911, take patient to the ER, or where hospitalization was necessary)?

Location of data: Patient chart or patient interview.

Attribute values: 1 = Never; 2 = 0nce; 3 = Twice; 4 = Three or more times

Subgroups: i-iii

Report: frequency distribution of diabetics with scores of 1-4 for past year

C)  Frequency of Diabetic Ketoacidosis. Although this is only measuring the most extreme form of hyperglycemia, and misses the morbidity associated with lesser degrees of hyperglycemia, it is a major outcome resulting from poor glycemic control.

FACCT domain: clinical outcome

Name: Frequency of diabetic ketoacidosis

Definition: How often has the patient experienced an episode of diabetic ketoacidosis which was so severe that it required the help of someone else to treat the patient (for example, when a third party had to give insulin and intravenous fluids, and where patient was treated in the ER, or where hospitalization was necessary)?

Location of data: Patient chart

Attribute values: 1 = never; 2 = once; 3 = twice; 4 = three or more times

Subgroups: i and ii

Report: frequency distribution of diabetics with scores of 1-4 for past year

Good:

A) Chart review for performance of HbA1c (or glycated hemoglobin) four times/year (minimum of once a year), or fasting blood sugar three to four times/year.

FACCT domain: essential processes

Name: Measurement of HbA1c (or glycated hemoglobin) or fasting blood sugar

Definitions: Has HbA1c (or glycated hemoglobin) or fasting blood sugar been measured four times in the past year?

Location of data: Lab data or patient's chart

Attribute values: 1 = Never; 2 = 0nce; 3 = Twice; 4 = Three or more times; 5 = Four or more times

Subgroups: i-iii

Report: The frequency distribution of the laboratory tests, by age group

B) Evidence by patient report of self-blood glucose monitoring

FACCT domain: essential processes

Name: Frequency of Self Blood Glucose Monitoring (SBGM)

Definitions: How often do patients test their own blood glucose in a day?

Location of data: Patient's chart or patient interview

Attribute values: 1 = Never; 2 = Less than once per day; 3 = 0nce a day; 4 = Twice a day; 5 = Three times a day; 6 = Four or more times daily

Subgroups: group i - 3-4 times daily; groups ii and iii, insulin treated - at least once daily; group iii, non insulin treated - at least once weekly

Report: frequency distribution for patients in groups i and ii, frequency in insulin treated and non insulin treated patients in group iii.

C) Evidence by patient report of modifications of hypoglycemic medications based on monitoring data. This is another way to estimate how intensively patients are involved with their own glycemic control. Some studies have shown that HbA1C is lower in patients who use adjustable insulin algorithms with SBGM.

FACCT domain: essential processes

Name: Regimen Modification as a Result of Self Blood Glucose Monitoring (SBGM)

Definitions: Does patient alter hypoglycemic medication based on the results of SBGM data?

Location of data: Patient interview

Attribute values: 1 = Yes; 2 = No

Subgroups: i - iii

Report: % of patients in each age group who modify regimen as a result of SBGM

D) Patient reports of hypoglycemia

FACCT domain: clinical outcomes

Name: Patient self report of hypoglycemia

Definition: Does patient have low blood sugar episodes?

Location of data: Patient report

Sample question used in Type II Diabetes Port: On average, how often do you have symptoms of low sugar (such as sweating, weakness, trembling, or an "insulin reaction")?

Attribute values: 1 = Never; 2 = Less than once a month; 3 = About once a month; 4 = 2-3 times a month; 5 = Every week or more

Subgroups: i - iii

Report: frequency distributions for patients in each age group

2. Eye Disease

Detection and treatment of diabetic eye disease, proven to be overall cost effective in both Type I and Type II diabetes, may differ between subgroups in cost effectiveness. Different criteria are therefore suggested for these groups.(14)

The use of a visual function score, such as the VF-14, which asks questions about difficulty with activities of daily living due to visual impairment, allows assessment of the impact of sight problems on the individual patient. ]5 Such an instrument may be included as part of a patient survey. This does not preclude the need for dilated retinal examinations.

Optimal:

A) Dilated retinal exam or photograph of fundus yearly for group i after 5 years; yearly from onset in group ii; and every 2 or 3 years in group iii after an initial normal exam. Fundus photography is performed instead of direct retinal examination in a growing number of persons with diabetes; it is an acceptable alternative.

FACCT domain: essential processes

Name: Dilated Retinal Eye Exam or photography of fundus

Definitions: Has patient received a dilated retinal eye examination or fundus photograph in the time recommended for each group above?

Location of data: Optometry/Ophthalmology records

Attribute values: 1 = Yes; 2 = No

Subgroups: i-iii

Report: Could be reported as separate rates for each subgroup, or as an overall compliance rate with recommended measures for each group.

B)  Eye function by VF-14 Index of Visual Functioning patient questionnaire every other year after adjustment for age and baseline function, reflecting detection and management of retinopathy, macular edema, cataracts, and glaucoma as well as optimal refraction.(15)

FACCT domain: clinical outcome

Name: VF-14 Index of Visual Functioning

Definitions: functional status measure of patient's vision

Location of data: Patient questionnaire

Attribute values: VF-14 scores

Subgroups: i - iii

Report: mean scores in subgroups

Good:

A) Visual acuity yearly.

FACCT domain: clinical outcome

Name: Visual Acuity

Definitions: What is the most recent visual acuity?

Location of data: Ophthalmology/Optometry records.

Attribute values: The standard way to record this information is from 20/10 (the ability to read from 20 feet away what an "average" person can read from 10 feet) to 20/200 (the ability to read from 20 feet what an "average" person can read from 200 feet). If vision is worse than this, it is usually recorded whether the patient can still count fingers, can detect hand movements, can perceive light, or has no perception of light. Thus, the following exclusive data entries could be made: 1 =20/10; 2 = 20/20; 3 = 20/40; 4 = 20/60; 5 = 20/80; 6 = 20/100; 7 = 20/200; 8 = Patient can Count Fingers; 9 = Patient can detect Hand Movements; 10 = Patient can Perceive Light; 11 = Patient has no Perception of Light

Subgroups: i-iii

Report: frequency distribution by subgroup

3. Extremity Disease

Because of the effects of diabetes on peripheral nerves (neuropathy) and blood vessels, diabetics are prone to infections and ulcers of their feet and extremities. Patient education, prevention, and timely management can prevent complications. 4 The use of amputation alone as a marker of quality is not adequate because of the small numbers of cases that progress to this stage.

A)  Yearly rate of new foot ulcers and infections

FACCT domain: clinical outcome

Name: Foot ulcers and infections

Definition: The number of new foot ulcers and infections reported in the past year

Location of data: Patient's chart or self report

Sample questions for questionnaire:

During the past year have you been told by a physician that you have foot ulcers? 1 = No; 2= Yes

During the past year how often have you had sores or wounds on your feet that did not heal? (check one). 1= All of the time; 2= Most of the time; 3= Some of the time; 4= A little of the time; 5= None of the time

Further questions for patient self report of foot problems are included in the Total Illness Burden Index designed by Greenfield et al.(10) and used in the Type II Diabetes PORT Study.

Subgroups: i - iii

Report: rates of new ulcers and infections by subgroup

B) Yearly report of examination for ulcers and infections

FACCT domain: essential processes

Name: Frequency of Foot Examination

Definitions: Has the patient had a foot examination in the past year?

Location of data: Patient's chart

Attribute values: 1 = Some mention made of foot exam; 2 = No mention made of foot exam

Subgroups: i - iii

Report: rates of foot exam by subgroup

C)  Yearly review of charts and administrative data for reports of amputations.

FACCT domain: essential processes

Name: Amputation

Definitions: Has the patient had amputation of toe(s), part of feet, or limbs at any time in the past?

Location of data: Patient's chart, administrative database ICD-9 codes.

Attribute values: 1 = No parts amputated; 2 = Amputation of one or more toes; 3 = Partial amputation of foot; 4 = Below knee amputation; 5 = Above knee amputation

Subgroups: i - iii

Report: frequency distribution by subgroup

Good:

A) The fact that the patient recalls taking his or her shoes and socks off during a physician visit at least once a year is suggestive of, but not confirmation of, inspection of the feet.

FACCT domain: essential processes

Name: Patient recollection of foot inspection

Definition: Removal of shoes and socks/hose during at least one physician visit in the past year.

Location of data: Patient report

Sample questions (based on questions from the Type 11 Diabetes PORT study):

When you saw the doctor who treats your diabetes at his or her office or clinic, how often in the last year did he or she:

Ask you about numbness or tingling in your feet

Ask you about your foot care (that is, trimming your toenails, checking for infections, etc.)

Take your shoes and socks off and check your feet

Recommend that you see a foot doctor

Attribute value for each subquestion: 1 = once in the past year; 2= twice in the past year; 3= three times in the past year; 4= four or more times in the past year; 5= Never

Subgroups: i - iii

Report: distribution by subgroup

4. Lipids

Control of lipids in preventing heart disease is thought to be no less valuable in diabetic than in nondiabetic patients, and maybe more effective. Hyperlipidernia is a major risk factor for macrovascular disease in diabetics.46 The American Diabetes Association has made recommendations of normal values for lipids in diabetic patients; these are cited in the measures suggested below.(6)

Optimal:

A) Rates of abnormal levels of total cholesterol, HDL, LDL and triglyceride should be measured every year.

FACCT domain: clinical outcome

Name: Most Recent Total Serum Cholesterol Value

Definitions: Most recent value for total serum cholesterol recorded in the chart or lab database.

Location of data: Lab data in patient's chart.

Attribute values: numeric, in milligrams per deciliter. If no value in past year = missing. ADA specifies normal range as less than 200 mg/dL.

Subgroups: i, ii and iii (recent American College of Physicians recommendations have suggested relaxing the criteria for patients over 65).(16)

Report: rate of values outside normal range, and absent values, by subgroup

Name: Most recent LDL Cholesterol Value

FACCT domain: clinical outcome

Definitions: Most recent LDL cholesterol in patient's chart

Location of data: Lab data

Attribute values: numeric. ADA specifies normal range as less than 130 mg/dL.

Subgroups: i, ii and iii (recent American College of Physicians recommendations have suggested relaxing the criteria for patients over 65).

Report: rate of values outside normal range, and absent values, by subgroup

Name: Most recent LDL Cholesterol Value

FACCT domain: clinical outcome

Definitions: Most recent HDL cholesterol in patient's chart

Location of data: Lab data

Attribute values: numeric. ADA specifies normal range as greater than 35 mg/dL for men and 45 mg/dL for women.

Subgroups: i, ii and iii (recent American College of Physicians recommendations have suggested relaxing the criteria for patients over 65).

Report: rate of values outside normal range, and absent values, by subgroup

Name: Most Recent Fasting Serum Triglyceride Value

FACCT domain: clinical outcome

Definitions: Most recent value for fasting serum triglycerides

Location of data: Lab data

Attribute values: numeric. ADA specifies normal range as less than 2bo mg/dL.

Subgroups: i, ii and iii (recent American College of Physicians recommendations have suggested relaxing the criteria for patients over 65).

Report: rate of values outside normal range, and absent values, by subgroup

Lipid data could be reported in a table format. See Table 3.

Good:

A) Lipid levels have been drawn at least once over the past year

Name: Measurement of Serum Lipids

FACCT domain: essential processes

Definitions: Have serum lipids been measured in the past year?

Location of data: Lab data or patient's chart

Attribute values: 1 = Yes; 2 = No

Subgroups: i-iii

Report: rate of tests in past year by subgroup

5.  Kidney Disease

Kidney disease in NIDDM is partially preventable by normalizing blood pressure. There are two other factors that may prevent nephropathy. One, glycemic control, is often advanced on the assumption that the relationship between glycemic control and nephropathy in NIDDM is parallel to that in IDDM. The United Kingdom Prospective Diabetes Study and other studies suggest that nephropathy takes many years to develop, so that, in group iii, tight control may not be nearly as cost effective.

There is clear evidence that the presence of small amounts of protein in the urine (microalbuminuria) which are not detectable by the usual dipstick method identifies a subset of diabetic patients who (a) are at significantly increased risk of developing coronary artery disease, (b) are at significantly increased risk for sudden death, (c) are at significantly increased risk for developing diabetic nephropathy and End Stage Renal Disease (ESRD), (d) would benefit from treatment with an ACE inhibitor, even if they are normotensive.

There is mounting evidence that treatment of diabetic hypertensive patients with an ACE inhibitor will prevent nephropathy3 in patients with microalbuminuria. Thus, measure of microalbuminuria is useful in guiding treatment toward more intense glycemic control and use of ACE inhibitors.

Optimal:

A) Yearly rates of blood pressure control from medical records.

Name: Blood Pressure

FACCT domain: clinical outcome

Definitions: Most recent blood pressure recording in chart

Location of data: Patient's chart

Attribute values: numeric. For example, 162/102

Subgroups: i-iii

Report: rate of values outside normal range ( >140/90 mmHg) or absent values.(13)  Can be reported across all groups.

B)  Yearly testing for microalbuminuria and, if present at >30 mg/24 hours, treatment with ACE inhibitor and tight glycemic control (HbA1c <7.0).

Name: Measurement of Quantitative Urinary Protein

FACCT domain: essential processes

Definitions: Has patient had a 24 hour urinary protein estimate (or measurement of the urinary albumin:creatinine ratio ) in the past year?

Location of data: Lab data or patient's chart.

Attribute values: 1 = Yes; 2 = No

Subgroups: i-iii

Report: rate by subgroup

Name: Quantitative Urinary Protein

FACCT domain: clinical outcome

Definitions: Most recent value for quantitative urinary protein (expressed as milligrams per 24 hours, or as milligrams per milligram of creatinine).

Location of data: Lab data in patient's chart.

Attribute values: numeric

Subgroups: i-iii

Report: rate of values outside normal range, and absent values, by subgroup

c)  Yearly measure of creatinine and referral to nephrologist for dialysis consideration if creatinine >3.0.

Name: Measurement of Serum Creatinine

FACCT domain: essential processes

Definitions: Has patient had serum creatinine measured in the past year?

Location of data: Lab data in patient's chart.

Attribute values: 1 = Yes; 2 = No

Subgroups: i-iii

Report: rate by subgroup

Good:

A) Yearly measure of blood pressure (From Patient Survey)

Name: Annual blood pressure check

FACCT domain: essential processes

Definition: Blood pressure checked at least once during past year

Location of data: Patient report or chart

Attribute value: 1 = No; 2= Yes

Subgroups: i-iii

Report: combined rate for all subgroups

B) Yearly measure of microalbuminuria

Name: Annual microalbuminuria check

FACCT domain: essential processes

Definition: Microalbuminuria checked within past year

Location of data: Laboratory records or patient chart

Attribute value: 1 = No; 2=Yes

Subgroups: i-iii

Report: rate by subgroup

C) Yearly measure of creatinine

Name: Annual creatinine check

FACCT domain: essential processes

Definition: Serum creatinine checked within past year

Location of data: Laboratory records or patient chart

Attribute value: 1= No; 2= Yes

Subgroups: i-iii

Report: rate by subgroup

D)  Rate of treatment with ACE inhibitors. Unless there are compelling reasons for not using an ACE inhibitor, most patients who have hypertension or proteinuria should be receiving treatment of this sort.

Name: Treatment with ACE Inhibitor

FACCT domain: essential processes

Definitions: Is the patient with microalbuminuria or hypertension receiving treatment with ANY ACE inhibitor (e.g., captopril, enalapril, lisinopril, etc.)?

Location of data: Pharmacy data and patient chart

Attribute Values: 1 = No treatment with ACE inhibitor or other hypotensive drug; 2 = Treatment with hypotensive drug (but NOT ACE inhibitor); 3 = Treatment with ACE inhibitor

Subgroups: i-iii

Report: frequency distribution of diabetics with microalbuminuria hypertension treated with 1, 2, or 3 by subgroup

6. Hypertension

As described in kidney disease section

7. Heart Disease

There are two areas that need to be considered. One is prevention, which is covered under blood pressure, lipids, and possibly glycemic control. Another is treatment, which will not be considered here as it constitutes a separate diagnostic and therapeutic field that merits full exploration as a discrete disease entity.

8. Obesity

Lowering weight is desirable, but because of the degree to which it is dependent on patient factors, including socioeconomic variables, and the difficulty of risk adjusting these, it is not currently suggested as a quality of care measure.

Justification

In recent years, there has been a growing recognition that the practice of medicine involves tradeoffs, including ones that the patient can be involved in, as rater of the care (satisfaction), as reporter (symptoms and health-related quality of life [HRQOL]), and as participant. The patient acts as reporter of the global quality of care when he or she scores the self perceived functional status, or quality of life. The long term consequences of good preventive care for diabetes are reflected many years later in the reduction of eye and kidney disease, as noted earlier. However, the shorter term outcomes, or intermediate outcomes, such as blood sugar control, may not reflect the totality of care provided by the conscientious physician, clinic, nurse or system in which the care is carried out. Insulin reactions or hypoglycemia can cause great distress and reduction of function, as might multiple injections and multiple SBGM determinations. Blood pressure medications might be causing fatigue, and impotence. In the intense treatment of blood sugar, the patient's other medical conditions (arthritis, prostatism, heart disease, lung disease, back pain, etc.) might not receive equal attention. The patient's mental state may be neglected, without addressing depression related to having a great disease burden to contend with. HRQOL can serve to measure the tradeoffs, to capture the global care of the patient as well as the specific functional results of symptoms. Thus, there has been a move to include HRQOL, or functional status, as a short term outcome of care, to complement the physiological measures defined by the disease.

There are several widely used and tested measures that have been shown to be reliable and valid. Using these measures, if the patient says that quality of life is compromised, clinical examination confirms that because of illness the patient is compromised. Further, patients with poor HRQOL will have higher mortality than those who report that their HRQOL is good or excellent.(17) Thus, there are measures that are valid for use in this population.

However, as pointed out by Deyo and Patrick, Greenfield and Nelson and others, problems arise in the use of HRQOL status as a quality measure.(18,19) These barriers have been thoroughly reviewed and will only be briefly noted here. In medical care situations, such as the management of diabetes, the form of validity that is important for quality of care is how the measure reflects the quality and effectiveness of care. In particular: is the reported disability and dysfunction due directly to the medical care received (the drugs, the interpersonal care, the diagnostic tests, etc.)? This might be put another way: is the HRQOL specific to the treatment or is it due to problems beyond the control of the health system or physician? Also, sensitivity is important: is the measure sensitive? That is, if the physician helps the patient by reducing hypoglycemic reactions or angina attacks, will the measure reflect that result? From other areas of research, if a patient has a transurethral prostatectomy, for example, the seH reported physical function (climbing stairs, getting in and out of a car) does not change even though the patient's symptoms are much better or gone.(20)

Specificity can be increased by adjustment for case mix, i.e., by controlling for those factors (socioeconomic status, severity of diabetes, severity of comorbid diseases) that independently affect outcome, to make comparison groups equivalent. There are successful current efforts, for example, the Total Illness Burden Index, 9 '° that measure case mix well enough to be able to adjust, control or account for the factors beyond the doctor's control and thereby reduce the nonspecificity.

To increase the sensitivity, diabetes-specific measures can be employed, either to complement general measures, such as SF-36, or in some cases, supplant them.(21) These measures might reflect hassles, or dysfunction due to tight blood sugar control that may mean a lot to the patient but do not result in changes in ability to walk or have friends or carry out one's role in life. One of these specific measures was used in the DCCT. Recently, the work of Kaplan et al. has shown that is it possible to make these disease specific measures generalizable by employing universal dimensions of HRQOL that impact patients with all chronic disease, but draw the items or variables or disease factors from that particular disease. In the Type II Diabetes PORT study, the "HASSLES" measure differentiated between regimen effects (i.e., the burden associated with different treatments), but generic health status instruments did not.(22 ) The dimensions Kaplan has identified are: hassles (problems or difficulties in carrying out role functions while still doing them well enough, such as going out to dinner with friends for a patient with NIDDM, or opening a jar for a patient with rheumatoid arthritis); body image (obesity in NIDDM); self esteem (not being able to work as successfully as peers due to diabetes complications); and worry (will I be blind?). These dimensions are present to some degree in all diseases but have different forms and meaning in different diseases. Using some universal measures assures comparability across diseases to create a common metric, such as occurs with the general health status measures. At the same time using measures that are sensitive to high quality disease management in terms of immediate patient response assures that high quality of care will be recognized.

Thus, the ideal array of outcome measures would include physiologic measures, short and long term complications, general health status, and disease specific health status. This array would be expected to be sensitive to the optimal management of NIDDM patients because of its comprehensiveness. Providers could also address tradeoffs between optimizing certain kinds of outcomes and not others.

Optimal

Both generic and diabetes specific measures, at a baseline time and 1 to 3 years later, adjusted for socioeconomic status and case mix using a measure that goes beyond diagnoses alone.

FACCT domain: quality of life

Instruments recommended: MOS SF-3623 and 'HASSLES' Diabetes Specific Health Status Measure (from Diabetes PORT Study) and Total Illness Burden Index.(9,10) Subscales, or key questions that relate most closely to functional health in diabetic patients, may be refined from these instruments by further analysis of existing data.

Subgroups: i-iii

Report: risk adjusted mean scores, standard deviations, for age groups, and Type I vs Type II diabetics. (There are norms available from the Type II Diabetes PORT data. There may be norms available in the near future from the HCFA Project.)

Good

Generic functional status, adjusted for baseline and for the number of ICD-9 diagnoses.

Instruments recommended: MOS SF-36,23 risk adjusted by Ambulatory Care Groups, or similar weighted diagnosis case-mix adjustment.

Subgroups: i-iii

Report: adjusted mean scores, standard deviations for age group, and Type I and Type 11 diabetics.

Satisfaction

The issue of including patient satisfaction with care as an outcome is controversial. There are two fundamentally different approaches to patient satisfaction measurement. The first is the use of periodic surveys to provide a summary evaluation of medical care during a specific time period, for example, one year. The second approach narrows the focus to a specific medical care visit, or series of visits. Instruments are available for both types of evaluation.(24,25) There is new evidence that the processes that lead to satisfaction are more complex than originally understood, and are little dependent on treatment or the scientific basis for diabetic care. Patrick and colleagues, and Greenfield et al. determined that patient satisfaction with care is linked to patients' health status; those patients with greater disability and comorbidity may be less satisfied with care.(26,27) There is emerging evidence from the work of Kaplan and colleagues that patient passivity and physician participatory decision making style impact the providerpatient relationship, and thereby affect patient satisfaction with care.(28)

Therefore, the recommendation for consumer satisfaction is the careful use of a generic measure to evaluate,the overall satisfaction with care. The addition of diabetes specific questions may be considered. The FACCT satisfaction with care review may provide further information that will inform the choice of instrument.

FACCT domain: satisfaction

Instrument recommended: generic satisfaction with care instrument, reported by age group and Type I vs. Type II diabetics.

The economic impact of diabetes

Diabetes is a costly disease, and NIDDM is the most costly disease in the country. The effective detection and management of diabetic complications must therefore reduce health care costs, and also costs to the economy and individual through lost production and function. The prevention and detection of eye complications has been shown to reduce health care costs.(14) Further information on cost effectiveness has resulted from the work of the diabetes PORT study about use of antibiotics in diabetic foot infections.(29) The Type II Diabetes PORT has also provided utilization data on the yearly number of visits, hospitalization rates, laboratory tests and SBGM.(30) The PORT findings suggest that case mix adjustment is critical when comparing costs of care.(30)

Good

FACCT domain: economic

Name: economic impact of diabetes on patients

Location of data: patient report

The economic impact of diabetes on daily life and work can be assessed by direct patient questioning. Use of a standardized instrument to assess time off work and school, in combination with role function information from the SF36. (See Appendix B for sample questions.)

Attribute values: numeric (time off work/school), and scores (SF36)

Subgroups: i-iii

Report: mean scores and standard deviation or range, by age group and Type I vs. Type II diabetics.

As noted throughout, data will need to be merged from: patient surveys, administrative databases, medical records and computerized laboratory reports. Sources of certain data may vary depending on the information system available in different settings; for example, laboratory results may be available from computerized records or the patient's chart. At present most health plans will not be able to retrieve HbA1c or lipid levels from an automated laboratory system, as different laboratories are used by individual plan providers. However, as the prevalence of computerized records increases, the ease of collecting essential process and outcome data will increase.

Data to be collected directly from patients may be combined to form one questionnaire, which may contain several sub-sections including HRQOL, economic impacts, satisfaction, case-mix adjustment tool, symptoms, and process of care questions.

2. Identification of the sample

Adult diabetic patients can be identified from administrative data, laboratory and/or pharmacy data. Criteria for diagnosis are attached in Appendix A. Sampling may be stratified random samples of all three subgroups, or all diabetics in plan/group practice, stratified by subgroup. (Technique would vary depending on total number of members/patients enrolled.)

3. Sample size

The sample sizes required to compare groups of diabetic patients will vary depending on the measures used, the likely observed differences, variation in results, etc. The following sample size calculations are based on use of the PFI10 (physical function score of the MOS SF-36) and possible response patterns.

If two groups are being compared, samples of 154 in each group (total 308) are required to detect a difference of eight points in the PFI10 with power of 80%. To detect the same difference with 90% power, samples of 206 (total 412) are needed. To detect a six point difference between two groups, samples of 273 (total 546) are needed to achieve 80% power, and samples of 306 each (total 612) for 90% power.

4. Design

Patients should be followed longitudinally with a random sample of new diabetics added yearly to reflect annual enrollment and disenrollment figures.

5. Frequency of measurement

Measurement should be annual, unless otherwise specified in the measure description.

6. Case mix adjustment

To address the nonspecificity of death and functional status measures, which may be impacted by any aspect of diabetes, by any aspect of comorbid diseases, and by nonmedical factors, adjustment should be made for case mix. There is increasing recognition that even process measures, such as attempts to control blood sugar, may need to be adjusted for case mix differences. Some recent unpublished data indicate that older NIDDM patients with a great deal of comorbid disease have less well controlled sugars. The 72 year old obese patient who also has chronic lung disease, prostatism, heart disease and arthritis will have both poor glycemic control and lower HRQOL than a patient who is 44 years old and healthy. If two HMOs, two centers, or two offices have vastly different kinds of patients, comparisons may be spurious for both outcome and process measures.

For outcome measures, such as death and HRQOL, case mix adjustment is mandatory. Just as we cannot speak of mortality rates without using "risk adjusted" mortality, there is growing appreciation that we cannot use functional status measures without risk adjustment. An abundance of evidence indicates that all manner of other diseases (both mental and physical), besides the disease or the treatment being considered have impacts on quality of life.

Several recent publications address the generic issues of case mix adjustment for outpatient care.(31,32) The conventional way of assessing severity of illness (both the severity of the diabetes and the coexistent diseases) is to sum the diagnoses from the medical record or administrative data base and apply some weights to them. Ambulatory Care Groups(33) (ACG's) and other diagnostic groupings are used for this purpose. Questions have been raised about this approach because of the inaccuracies in coding in office or ambulatory practice, and because of a more fundamental consideration, whether a diagnosis, such as NIDDM, Coronary Artery Disease (CAD), or Chronic Obstructive Pulmonary Disease (COPD), captures the range of severity of patients with that diagnostic label. A patient with NIDDM, CAD, or COPD can be running marathons or be moribund. Two groups of patients may have the same diagnoses, but vary extremely in the amount of illness that would lead to death or poor HRQOL.

For this reason Greenfield et al., in the Diabetes PORT, have developed a new feasible and reliable measure that goes beyond diagnosis to encompass symptoms, and disease manifestation such as amputations, episodes of bronchitis, history of procedures such as percutaneous transluminal coronary angioplasty.(9,10) In this measure, the Total Illness Burden Index, information comes from patient report, using a self administered or interviewer administered form. This method has been shown to predict HRQOL and mortality better than lists of diagnoses, and to discriminate populations differing widely in severity, such as a poor vs. a middle-class sample of NIDDM patients.(9) Using this method, which summarizes the severity of 15 body systems, and classifies patients into one of 4 categories according to total disease burden (diabetes complications and comorbid diseases), the NIDDM group in the lowest category (least severe NIDDM and no comorbid diseases) had a physical function score from the SF-36 of 85, with the second group having a mean score of 75, the third 60, and the most severe group 47.(10)

To judge the quality of care of a group of diabetic patients compared to a group at another institution, the patients could be risk adjusted at the outset. Over a one or two or three year period, each severity subgroup would be expected to have a trajectory under circumstances of good care and a different trajectory under inadequate care. The comparison of the differences in trajectories, adjusted for baseline disease, could assess whether one group is receiving substandard quality of care.

7. Scale scoring

Scale scores were developed for each section of the Type II Diabetes PORT Study questionnaires. These are currently available in the form of computer programs.

8. Level of aggregation of data reporting

These measures are suitable for use in a diverse set of environments, from prepaid health plans to individual providers in a fee for service arrangement. External reporting of results is much harder at the individual provider level as statistical significance depends upon having sufficient patients of one subgroup per physician. Case-mix is also a bigger problem at the individual provider level as physicians tend to attract similar patients. Therefore reporting at the group, network or plan level would be more appropriate.

IV. Conclusion

In summary it is possible to identify a 'family' of valid and reliable quality measures for diabetes that can be used to measure quality of life, clinical outcomes, and essential processes. Some optimal measures may not be considered ready for immediate use. Their implementation may require more effort and refinement, but they deserve serious appraisal at this stage.

Acknowledgments

The authors would like to thank Lisa Sullivan Ph.D. for assistance with the sample size calculations.

1. Patient is identified as a diabetic by one of the following criteria in his/her medical record:

a. An ICD-9 code for diabetes mellitus (250.xx), diabetic polyneuropathy (357.xx), diabetic retinopathy (362.0 - 362.0X), or diabetic cataract (366.41);

b.  A test for glycated hemoglobin (CPT 83036);

c. Notation of prescribed use of insulin or oral medication for diabetes (e.g Orinase, Diabinase, Tolinase)

APPENDIX B

Burden of disease questions from Type II Diabetes PORT Study (used in addition to SF-36)

1. In the past 12 months, did you cut down on the things you usually do, such as going to work or working around the house, because of illness or injury?

No

Yes - How many days did you cut down on the things you usually do because of illness or injury? (enter number of days)

2. In the past 12 months. did you ever stay in bed because of an illness or injury?

No

Yes - How many days did you stay in bed at least half the day because of illness or injury? (enter number of days)

References

1. Javitt JC, Chiang YP. Economic impact of diabetes. In Diabetes in America, National Diabetes Data Group. Second Edition, National Institutes of Health, 1995; Chapter 30: 601-611.

2. DCCT Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin dependent diabetes mellitus. N Eng J Med 1993;329:977-86.

3. Molitch ME. ACE inhibitors and diabetic nephropathy. Diabetes Care.1994;17:756-774.

4. Caputo GM, Cavanagh PR, Ulbreacht JS, Gibbons GW, Kaerchmer AW. Assessment and management of foot disease in patients with diabetes. N Eng J Med. 1994;331:854-860.

5. National Committee for Quality Assurance. Health Plan Employer Data and Information Set and users manual. NCQA, 1996; Washington DC.

6. American Diabetes Association. Manual for Completing the American Diabetes Association Provider Recognition Program Application. Draft; ADA 1996.

7. Work Group on Standardization of Diabetes Definitions, Performance measures, Quality of Care and Outcomes. Diabetes Treatment and Outcome Data Set. December 1995.

8. Vijan S, Hofer T, Hayward R A. The benefits of glycemic control in Type II diabetes (Unpublished data).

9. Greenfield S. Underestimation of severity of illness using diagnosis vs. comprehensive measures. Abstract 1996, Drug Information Association, Third Annual Symposium of Contributed Papers on Quality of Life Evaluation.

10. Greenfield S, Sullivan L, Dukes KA, Silliman R, D'Agostino R, Kaplan SH. Development and testing of a new measure of case mix for use in office practice. Medical Care 1995;33(4): AS47-AS57.

11. Greenfield S, et al., The Type II Diabetes Patient Outcomes Research Team Study. (unpublished data)

12. Greenfield S, Rogers WH, Mangotich M, Camey MF, Tarlov AR. Outcomes of patients with hypertension and non insulin dependent diabetes mellitus treated by different systems and specialties: results from the Medical Outcomes Study. JAMA;274(18):1436-1444.

13. Hiss RG, Anderson RM, Hess GE, Stepien CJ, Davis W. Community diabetes care: a ten year perspective. Diabetes Care 1994;17(10):1124-1134.

14. Javitt JC, Aillo LP, Chiang Y, Ferris FL, Canner JK, Greenfield S. Preventive eye care in people with diabetes is cost saving to the Federal govemment. implications for health-care reform. Diabetes Care 1994; 17(8):909-917.

15. Steinberg EP, Tielsch JM, Schein OD, Javitt JC, Sharkey P, et al. The VF14 An index of functional impairment in patients with cataract. Archives of Ophthalmology 1994;112:630-638.

16. American College of Physicians. Guidelines for using serum cholesterol, high-density lipoprotein cholesterol, and triglyceride levels as screening tests for preventing coronary heart disease in adults. Annals of Intemal Medicine; 1996; 124(5): 515-517.

17. Dasbach EJ, Klein R, Klein BEK, Moss SE. Self-rated health and mortality in people with diabetes. American J of Public Health 1994; 84(11):1775-1779.

18. Deyo RA, Patrick DL. Barriers to the use of health status measures in clinical investigation, patient care, and policy research. Medical Care, 1989;27(3):S254-S268

19. Greenfield S., Nelson EC. Recent Development and Future Uses of Health Status Measurements in Clinical Practice. Medical Care.1992;30(5):MS23-MS41, Supplement May 1992.

20. Epstein RS, Deverka PA, Chute CG, et al. Validation of a new quality of life questionnaire for benign prostatic hyperplasia. J of Clinical Epidemiology. 1992;45(12):1431-1445.

21. Parkerson GR, Connis RT, Broadhead WE, Patrick DL, Taylor TR, Tse CJ. Disease-specific versus generic measurement of health-related quality of life in Insulin-dependent diabetic patients. Medical Care 1993;31(7):629-639.

22. Kaplan S, Sullivan L, Dukes K, Greenfield S. Disease-specific vs. general health status measures for evaluating treatment effects in NIDDM. Abstract, American Diabetes Association National Meeting, 1996.

23. Ware JE, Snow KK, Kosinski M, Gandek B. SF-36 Health Survey Manual and Interpretation Guide. Boston, Mass: The Health Institute, New England Medical Center Hospitals; 1993.

24. Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Medical Care 1988;26(4):393-402.

25. Davies AR, Ware JE. OHM's Consumer Satisfaction Survey. 2nd Edition, 1991, GHAA, Washington D.C.

26. Patrick DL, Scrivens E, Chariton JRH. Disability and patient satisfaction with medical care. Medical Care 1983;21(11):1062-1075.

27. Greenfield S, et al. Unadjusted patient satisfaction scores used in physician profiling result in spurious ranking [abstract]. Meeting of the Society of General Internal Medicine; 1996 May; Washington, DC.

28. Kaplan SH, Greenfield S, Gandek B, Rogers WH, Ware JE. Patient and visit characteristics related to physician's participatory decision-making style: results from the Medical Outcomes Study. Medical Care 1995; 33(12):1176-1187.

29. Eckman MH, Greenfield S, Mackey WC, Wong JB, Kaplan SH, Sullivan L, Dukes KA, Pauker SG. Foot infections in diabetic patients: decision and cost-effectiveness analyses. JAMA 1995;273(9):712-720.

30. Hayward RA, Manning WG, Kaplan SH, Wagner E, Greenfield S. The costs and effectiveness of insulin therapy in Type II Diabetes. [abstract]. National Meeting of the American Diabetes Association; 1996 June; San Diego.

31. Murray JF, Hasselblad V, Sands E. Sources of variation and risk adjustment for HEDIS performance measures. (Unpublished data).

32. Greenfield S, Sullivan L, Silliman RA, Dukes K, Kaplan SH. Principles and practice of case mix adjustment: applications to end stage renal disease. American Journal of Kidney Disease. 1994;24(2):298-307.