Validation of Chinese medicine syndrome differentiation in early breast cancer: a multicenter prospective clinical study.

1
Introduction
Breast cancer is the most common malignancy and a major cause of cancer-related deaths among women (1, 2). Research has indicated that breast cancer patients turn to complementary medicine to relieve side effects from cancer treatments and improve their quality of life (3–5). Chinese herbal medicine (CHM) is a frequently used form of complementary medicine, with its effectiveness dependent on accurate Chinese medicine (CM) syndrome differentiation (6). As such, there is a pressing need to standardize the CM syndrome differentiation and treatment of breast cancer to enhance therapeutic outcomes in clinical settings. An essential step in advancing the understanding of early breast cancer CM syndromes involves establishing a standard for CM syndrome differentiation and quantifying as well as standardizing the syndrome differentiation process.
In our clinical practice, we noted variations in CM syndromes in response to different treatments for breast cancer and pointed out the differentiation of CM syndromes across various treatment stages (7). Researchers have also indicated that syndrome differentiation by treatment stage in breast cancer exemplifies the integration of disease and syndrome concepts (8). As such, we further utilized a systematic review (9), a clinical study (10), and a Delphi study (11) to establish diagnostic criteria for CM syndrome differentiation among different treatment stages of early breast cancer. This diagnostic criterion clearly defines primary and secondary CM syndrome diagnostic conditions through the comprehensive use of multiple statistical methods, data mining, and expert consultation meetings. Clinical validation is crucial in assessing the clinical applicability of CM syndrome research findings. However, no confirmatory tests have been conducted in published studies on breast cancer CM syndrome, leading to limited credibility and acknowledgment of the research outcomes.
To rigorously verify the diagnostic efficacy of these criteria, this study conducted a prospective, multicenter, large-sample clinical investigation to gather clinical case information on early breast cancer. Diagnostic evaluation tests were then performed to offer evidence-based medical support in clinical practice and enhance the credibility of stage syndrome differentiation.

2
Materials and methods
This study was approved by the Ethics Committee of Guangdong Provincial Hospital of Chinese Medicine (approval no. ZE2022-142-01) before recruitment commenced. Other centers received ethics approval from their centers. All participants provided written informed consent. This study was registered with the Chinese Clinical Trial Registry under the registration number ChiCTR2200061022 on 15 June 2022.
2.1
Inclusion and exclusion criteria
Patients were selected based on the inclusion and exclusion criteria. The inclusion criteria included (1): histologically confirmed primary invasive breast cancer (Stage I–III, no distant metastases) (2); female patients aged 18 years or older (3); an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2; and (4) ability to understand the study and provide written informed consent. The exclusion criteria were (1): the presence of accompanying severe diseases, such as cardio-cerebrovascular or psychological disease (2); undergoing neoadjuvant therapy or receiving concurrent treatments among chemotherapy, radiation therapy, and endocrine treatment (3); complications arising from breast cancer treatment, such as postoperative wound infection, severe lymphoedema, neutropenic fever or radiation pneumonia; and (4) pregnant or breastfeeding women.

2.2
Study procedure and data collection
Based on a previously published clinical study used to develop diagnostic criteria (10), the case report form for this validation study was designed according to the research review of literature and textbooks. The form encompasses several key sections, including patient basic information, breast cancer history, as well as symptoms and signs.
Patients were screened to assess whether they met the inclusion and exclusion criteria. Researchers thoroughly explained the study’s objectives and procedures to ensure complete understanding and informed consent from the patients. Additionally, participants were informed that they would receive a questionnaire survey at each stage of treatment (including preoperative, postoperative, chemotherapy, radiotherapy, and endocrine therapy). Once the patient signed the informed consent form, the first questionnaire survey began and subsequent treatments were tracked. Follow-up questionnaires were administered upon each return to the hospital for treatment. In total, each patient completed up to five surveys corresponding to each of the treatment phases.
Researchers at each hospital underwent training on the standard operating procedures, which included a comprehensive explanation of the study’s methodology and detailed instructions on obtaining informed consent and accurately completing the case report form. This training ensured that all researchers thoroughly understood the procedures and could implement them correctly. Finally, the syndrome name of each cluster was determined in accordance with the Clinic Terminology of Traditional Chinese Medical Diagnosis and Treatment–Syndromes (12), following thorough discussion with the research team.

2.3
Sample size calculation
The sample size estimate for this diagnostic test should be based on the assumed test’s sensitivity and specificity, as well as permissible levels of error and test error. The formula used is: , , where and are the numbers of samples needed for the test standard and reference, respectively. Here, represents the allowable error, is the sensitivity, and is the specificity. Setting =0.05, =0.10, and we calculated =1.96. With estimated sensitivity and specificity at 0.90, the required sample size for each syndrome type in the diagnostic test is calculated. Given that the study includes nine different syndromes, the estimated sample size needed is 70 samples per syndrome (in total nine syndrome according to previous Delphi study) (11), resulting in 630 cases overall. The target number of cases per site was derived from each center’s 2021 early-stage breast cancer records.

2.4
Diagnostic test evaluation analysis
After collecting patient data, a diagnostic test evaluation was performed. The authenticity and clinical application value evaluation of diagnostic tests were divided into two groups: the reference standard group and the test standard group.
The diagnostic results are assessed by two physicians with associate senior titles based on their experience, and they are valid as a reference standard when their results are consistent. Initially, a preliminary trial is conducted to evaluate the consistency of results between the two doctors. They first differentiate 20 randomly selected cases. After a week, they repeat the process for the same cases, and their consistency is analyzed. A Kappa value above 0.4, calculated using SPSS software, indicates high consistency, allowing formal differentiation to proceed. The test standard group comprises two CM attending physicians specialized in breast cancer, performing syndrome differentiation based on diagnostic criteria (see Supplementary File) established according to previous studies (9–11). Similarly, these two doctors conducted preliminary tests using the clinical data of 20 selected patients. The formal study was initiated once the Kappa value reached 0.7 or higher, indicating sufficient consistency between the doctors’ assessments.
SPSS statistical software (IBM SPSS Statistics 26.0) was utilized for statistical analysis in this study. Descriptive statistics were applied to analyze the demographic information of the patients. Shapiro-Wilk tests were conducted to assess the normal distribution of continuous variables. Measurement data are presented using maximum, minimum, average values, and standard deviation. Additionally, rates, proportions, and frequencies were used to express categorical data.
Sensitivity measures the proportion of correctly identified positive patients, with higher sensitivity indicating fewer missed diagnoses. Specificity measures the proportion of correctly identified negative patients, with higher specificity indicating fewer false positives. Accuracy represents the proportion of true positive and true negative results out of all cases, reflecting the consistency between diagnostic tests and expert evaluations. The Youden index, calculated as sensitivity plus specificity minus one, assesses the overall effectiveness of a diagnostic test. Likelihood ratios reflect the sensitivity and specificity by comparing the probability of test results in patients versus non-patients.
Predictive values assess patient prevalence based on diagnostic test results. The positive predictive value indicates the proportion of true positives among those diagnosed positive, while the negative predictive value reflects the proportion of true negatives among those diagnosed negative. Higher values enhance the diagnostic standard’s discriminatory power and clinical relevance. The Kappa value measures diagnostic consistency: values above 0.75 indicate good consistency, values between 0.4 and 0.75 suggest moderate consistency, and values below 0.4 denote poor consistency. A higher Kappa value signifies more reliable diagnostic criteria.

3
Results
This study was carried out in multiple centers from June 2022 to October 2023, including the hospitals located in South China (Guangdong, Hainan), East China (Shanghai, Jiangsu, Zhejiang), Central China (Hunan), and Southwest China (Chongqing). A total of 665 patients were initially approached for participation in this study. Among them, 17 patients did not provide informed consent, and seven were excluded because they did not meet the inclusion criteria. Ultimately, 641 eligible patients were included in the study. A total of 1141 case report forms were collected, as illustrated in Figure 1; the distribution of cases across centers is detailed in Figure 2.
3.1
Patient characteristics
The demographic characteristics of the patients are presented in Table 1. The ages of the participants ranged from 25 to 88 years. Most patients were married, making up 93.76% of the total. The pathological stages were stage I (34.63%) and stage II (43.99%), with over 80% of patients diagnosed with invasive ductal carcinoma.

3.2
Diagnostic test evaluation results
The self-differentiation results of two doctors in the pre-experiment reference standard group were consistent. The Kappa values for the consistency tests for two doctors in the reference standard group were 0.725 and 0.673, respectively, exceeding the 0.4 threshold, indicating moderate consistency. Additionally, after conducting a formal consistency test for the syndrome differentiation results of the two doctors, a Kappa value of 0.535 was obtained, further supporting the moderate consistency of their assessments.
The syndrome differentiation results of the two doctors in the test standard group were tested for consistency, and the Kappa value was 0.837, which is greater than 0.7, indicating a high level of agreement. In cases where there were discrepancies in the syndrome differentiation results during the pre-experiment phase, the two doctors engaged in discussions and negotiations to reach a consensus on the syndrome differentiation before proceeding with the formal differentiation process.

3.3
Evaluation of diagnostic criteria
The reliability of the diagnostic criteria and the evaluation results are detailed in Table 2. Sensitivity rates for all identified syndromes ranged between 70.37% and 92.00%. Notably, the sensitivity of the Spleen and Stomach disharmony in the postoperative stage was highest, suggesting its effectiveness in accurately identifying patients. Specificity across the criteria varied from 86.84% to 98.89%. Except for the dual deficiency of qi and yin in the endocrine therapy stage, the specificity of other syndrome diagnostic criteria exceeded 90%, demonstrating a notably low rate of misdiagnosis across the five treatment stages.
Additionally, the overall accuracy of these criteria was observed to range from 84.25% to 94.45%. The positive predictive value for all diagnosed syndromes varied from 72.22% to 98.21%, with the positive predictive value of the remaining syndromes exceeding 80% except for the syndrome of yin deficiency and fluid depletion in the radiotherapy stage and disharmony of thoroughfare and conception vessels in the endocrine therapy stage. The negative predictive value ranged from 79.25% to 98.82%. Except for dual deficiency of qi and Blood, the negative predictive value of the remaining syndrome was greater than 80%.
The reliability evaluation results for the diagnostic criteria are presented in Table 3. The concordance rate for these criteria ranged from 90.91% to 96.45%, with the lowest rate observed for Spleen and Stomach disharmony during the chemotherapy stage and the highest for the dual deficiency of qi and yin in the postoperative stage. Kappa values, which measure inter-rater agreement, ranged from 0.747 to 0.926. The Kappa value for Blood stasis with phlegm in the preoperative period was 0.747, indicating moderate reliability for this syndrome’s diagnostic criteria. In contrast, the Kappa value for the other syndromes exceeded 0.750, suggesting good reliability for their diagnostic criteria.

4
Discussion
In this multicenter prospective clinical study, we collected epidemiological data on CM syndromes to evaluate diagnostic tests, specifically assessing the diagnostic efficacy of CM syndrome differentiation criteria for early breast cancer. To the best of our knowledge, this is the first publication that verifies the CM syndrome diagnostic criteria in the context of breast cancer. This study contributes significantly to the standardization of clinical practice and offers a crucial reference for future investigations into the diagnosis of breast cancer syndromes.
The primary goal of establishing diagnostic criteria for CM syndrome is to facilitate their application in clinical practice and enhance standardization (13, 14). The scientific validation of these criteria is essential for their adoption and critical in assessing their validity and reliability. For instance, Luo (15) conducted a diagnostic test evaluation on Blood stasis syndrome, demonstrating high diagnostic effectiveness and greater reliability. Similarly, Zhou (16) evaluated the diagnostic threshold for coronary heart disease’s phlegm and dampness syndrome, improving the criteria’s diagnostic efficacy, operability, and repeatability. Despite these advances, many studies in CM syndrome research lack further validation (17), undermining the authority and recognition of their findings. This issue also pertains to breast cancer syndrome research (18–20).
Consequently, this study performs diagnostic tests on the newly established criteria for early breast cancer stage differentiation, addressing previous limitations and confirming the criteria’s effectiveness and enhanced scientific credibility.
The results demonstrate a high level of reliability in the diagnostic standards employed for evaluating the syndromes. Sensitivity and specificity are essential metrics in assessing diagnostic accuracy (21), based on the test’s inherent attributes. Sensitivity indicates the ability to avoid missed diagnoses, while specificity relates to reducing misdiagnoses. It is crucial to evaluate both metrics when analyzing diagnostic tests. However, relying solely on sensitivity and specificity may not provide a complete assessment; additional factors like accuracy and the Youden index should also be taken into account. The Youden index and likelihood ratios are composite indices that reflect both sensitivity and specificity (22). The positive and negative predictive values are significantly influenced by the disease’s prevalence (23). With constant sensitivity and specificity, the positive predictive value increases as prevalence rises, whereas the negative predictive value decreases. Thus, in cases where the disease prevalence is low, even highly sensitive and accurate diagnostic criteria may yield a low positive predictive value.
Conducting multicenter clinical research is the optimal approach for collecting comprehensive diagnostic data (24), but should be careful about data interpretation bias (25). Wang (26) conducted a multicenter cross-sectional study on damp-heat syndrome in diabetic coronary heart disease, including a broad array of research subjects to ensure representativeness. Currently, many CM syndrome studies, including those on breast cancer, are limited to one research institution or one research region (18, 27, 28). This limitation leads to sample selection bias and inadequate sample sizes, compromising the objectivity and evidential strength of the research (29). Additionally, some multicenter studies are restricted to specific areas, lacking nationwide representation. To address these issues, this study undertakes a large-scale, multicenter clinical study across various regions to gather epidemiological data, thereby enhancing the reliability and applicability of the research findings.
In addition, due to objective differences in the scale and patient flow of the participating centers, there were disparities in the number of cases ultimately contributed by each center, resulting in an inherent imbalance in the data. The International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) E9 (Statistical Principles for Clinical Trials) guideline suggests that exploring center effects is necessary when the sample size in each center is sufficient. In this diagnostic test study, where inter−center sample sizes varied considerably and the primary aim was to verify the generalizability of the diagnostic criteria, we focused on evaluating the overall performance of the criteria in the pooled population from all centers. Therefore, we ultimately did not perform an inter−center heterogeneity analysis.
The scientific establishment of diagnostic criteria for CM syndromes constitutes a foundational element in realizing the standardization and regularization of CM diagnosis and treatment. Historically, the absence of well-defined and universally accepted protocols for the development of these criteria has significantly impeded their formulation and subsequent dissemination. In 2023, the World Federation of Chinese Medicine Societies addressed this gap by drafting the “Guidelines on establishing diagnostic criteria of Chinese medicine syndromes” (30), which was informed by the preexisting technical framework for CM syndrome standard development. The guidelines delineate a comprehensive development process that encompasses literature review, expert consultation, clinical investigation, formulation of syndrome diagnostic criteria, and validation. Our research project has already undertaken relevant preliminary studies and is currently advancing the final phase of clinical validation in strict adherence to contemporary standard research methodologies, thereby exemplifying a robust paradigm for the development of CM syndrome standards. Further, the field of CM syndrome research remains replete with areas of inquiry that warrant further exploration. These include the objective diagnosis of CM syndromes, the elucidation of their scientific connotations, and the integration of CM syndromes with artificial intelligent (31, 32). These emerging frontiers of research are poised to catalyze our continued advancement in this domain.
Some limitations should be acknowledged. The lack of a recognized gold standard for reference and reliance on expert experience introduces an inevitable subjective bias, and this shortcoming also mirrors the broader methodological constraints currently faced in validating CM syndromes. Further, the lack of clear decision thresholds for each diagnostic test index prevents objective and accurate assessment of the diagnostic criteria. In this study, patients who did not receive endocrine therapy were excluded from enrollment. This exclusion underscores a gap in CM syndrome research for this population, limiting the diagnostic criteria’s applicability. The suboptimal diagnostic performance observed for certain syndromes may be attributed to their relatively small sample sizes, which also precluded further subgroup analysis. This underscores the need to enlarge the overall sample and achieve a more balanced distribution across syndromes in future work, thereby enhancing statistical power and robustness.

5
Conclusion
This multicenter prospective clinical study aimed to validate the diagnostic criteria for CM syndrome among different treatment stages in early breast cancer. The study collected substantial clinical data and performed a diagnostic evaluation test on the criteria for syndrome diagnosis. The results demonstrated high reliability in the diagnostic standards. Ultimately, the diagnostic test confirmed the authenticity, reliability, and clinical application value of CM syndrome differentiation criteria for early breast cancer. This research addressed a gap in verifying CM syndrome diagnostic criteria for breast cancer and significantly contributed to standardizing clinical practice. Further, this study lays a valuable foundation for future studies and enhances the understanding and application of CM syndrome in the context of breast cancer.