SPSS Validity Analysis
Validity analysis is used to measure whether questionnaire items (quantitative data) are designed reasonably, verified through factor analysis (exploratory factor analysis). Researchers have expected relationships between variables and items. After factor analysis, the relationship between factors (variables, called factors in factor analysis) and items should be basically consistent with expectations, indicating good validity. Validity analysis is specifically designed for scale data only; non-scale questions such as multiple choice or single choice gender questions cannot undergo validity analysis.
What is Validity Analysis
Validity analysis evaluates whether questionnaire items are designed reasonably and effectively. It uses factor analysis to verify the relationship between expected variables and items. When the factor-item correspondence from analysis matches the expected relationships, it indicates good validity.
Feature Access
1. Navigate to the "Analyze Results" section of your questionnaire in the SurveyMars system.
2. Click on the "SPSS Analysis" option to access the analysis features.
3. Click on the "Analysis now" button and select "Validity Analysis" from the available analysis methods.
Setting Up Validity Analysis
1. Select the measurement items you want to analyze for validity.
2. If you expect the analysis items to be divided into several aspects (variables), you can manually set the number of factors (dimensions). If not set, the system will use eigenvalues greater than 1 as the criterion to determine the number of factors.
3. Ensure all selected items are scale-type questions (quantitative data with ordered response options).
4. Click the "Confirm" button to generate validity analysis results.
Understanding Factor Loading Coefficients
Factor loading coefficients represent the correlation between analysis items and factors. The absolute value of factor loading coefficients indicates the strength of the relationship. For each analysis item, there are multiple factor loading coefficient values corresponding to different factors.
Judgment Criteria:
- If the absolute value of a factor loading coefficient is greater than 0.4, the item should be assigned to that factor
- For each analysis item, find the factor with the highest absolute loading coefficient value (greater than 0.4) to determine which factor it belongs to
Example Factor Loading Coefficient Table:
| Analysis Item | Factor 1 | Factor 2 | Factor 3 | Communality (Common Factor Variance) |
|---|---|---|---|---|
| Analysis Item 1 | 0.765 | -0.066 | 0.093 | 0.598 |
| Analysis Item 2 | 0.676 | 0.081 | -0.017 | 0.464 |
| Analysis Item 3 | 0.657 | 0.207 | -0.205 | 0.517 |
| Analysis Item 4 | 0.645 | 0.271 | 0.089 | 0.497 |
| Analysis Item 5 | 0.501 | 0.457 | 0.085 | 0.467 |
| Analysis Item 6 | 0.311 | 0.697 | -0.005 | 0.583 |
| Analysis Item 7 | 0.226 | -0.669 | 0.130 | 0.516 |
| Analysis Item 8 | 0.191 | 0.644 | 0.046 | 0.453 |
| Analysis Item 9 | 0.476 | -0.187 | 0.542 | 0.555 |
| Analysis Item 10 | 0.001 | -0.048 | 0.968 | 0.939 |
Note: In the table above, "Analysis Item 5" has factor loading coefficients of 0.501 for Factor 1 and 0.457 for Factor 2, indicating that it has relatively high associations with both Factor 1 and Factor 2 (slightly higher with Factor 1). This is an example of an "entangled" item, which can be assigned to either Factor 1 or Factor 2. Similarly, "Analysis Item 9" is also "entangled."
Special Cases:
- "Entangled" Items: When an item has factor loading coefficients greater than 0.4 for multiple factors, it is considered "entangled." This is very common and usually does not require any treatment. However, if you have many items, you may consider removing entangled items. The specific handling depends on the researcher's comparison and selection of optimal results.
- "Misplaced" Items: When an item's factor assignment does not match the expected relationship (e.g., Item 1 should be with Items 9 and 10 in Factor 3, but Item 1 is placed under Factor 1), this is called "misplaced" and must be deleted and re-analyzed.
Deleting Unreasonable Items
There are three situations where items should be deleted:
1. Low Communality: If an analysis item's communality (common factor variance) value is less than 0.4, the corresponding item should be deleted.
2. Low Factor Loadings: If all absolute values of "factor loading coefficients" for an analysis item are less than 0.4, the item should be deleted.
3. Misplaced Items: If an analysis item's factor correspondence shows serious deviation (misplaced items), the item should be deleted and re-analyzed.
Key Validity Indicators
1. KMO Value:
- KMO > 0.9: Excellent
- KMO 0.8 - 0.9: Good
- KMO 0.7 - 0.8: Acceptable
- KMO 0.6 - 0.7: Mediocre
- KMO < 0.6: Unacceptable
2. Bartlett's Test of Sphericity:
- Should be significant (p < 0.05), indicating that the correlation matrix is suitable for factor analysis
3. Variance Explained Rate:
- Cumulative variance explained rate should generally be greater than 50%
- Individual factor variance explained rate should be greater than 10%
The following table explains key terms used in validity analysis:
| Term | Definition |
|---|---|
| Factor Loading Coefficient | Represents the correlation between an analysis item and a factor. The absolute value indicates the strength of the relationship. If the absolute value is greater than 0.4, the item should be assigned to that factor. |
| KMO Value (Kaiser-Meyer-Olkin) |
Measures sampling adequacy for factor analysis. KMO > 0.9 is excellent, 0.8-0.9 is good, 0.7-0.8 is acceptable, 0.6-0.7 is mediocre, and < 0.6 is unacceptable. |
| Bartlett's Test of Sphericity | Tests whether the correlation matrix is suitable for factor analysis. Should be significant (p < 0.05) to indicate that factor analysis is appropriate. |
| Variance Explained Rate | The proportion of total variance explained by each factor. Individual factor variance explained rate should be greater than 10%. |
| Cumulative Variance Explained Rate | The cumulative proportion of total variance explained by all factors. Should generally be greater than 50%. Normally less than 100%, but may exceed 100% if multicollinearity is severe. |
| Eigenvalue | A measure of the variance explained by each factor. Factors with eigenvalues greater than 1 are typically extracted, but factors with eigenvalues less than 1 can also be extracted based on professional judgment. |
| Factor | Also called a variable or dimension. In factor analysis, factors represent underlying constructs that explain the relationships among analysis items. |
Troubleshooting Low KMO Values
If KMO value is too low, it may be due to:
- Correlation coefficients between analysis items are too low (less than 0.2 or not significant), resulting in low information overlap and inability to effectively concentrate information
- Correlation coefficients between analysis items are too high (greater than 0.8), causing severe multicollinearity and possibly preventing KMO value output
- Correlation coefficients between analysis items should ideally be between 0.3 and 0.7
Solutions:
1. Check correlation relationships and remove items with excessively high correlation coefficients.
2. Increase sample size; it is recommended that the analysis sample size be greater than 5 times the number of analysis items
3. Check if categorical data has been included in the analysis; if so, remove it from the model first
Troubleshooting Validity Issues
Validity analysis requires comprehensive judgment based on multiple indicators, including KMO value, Bartlett's test, variance explained rate, cumulative variance explained rate, factor loading coefficients, and dimension-item correspondence.
Key Points:
1. Validity analysis often requires deleting items to make dimension-item correspondence match expectations
2. The most critical aspect is whether the dimension-item correspondence matches professional expectations;
3. Validity analysis may require multiple rounds of comparison and item deletion to find optimal results
4. If validity analysis consistently fails to meet standards, consider analyzing by individual dimension. If there are 3 dimensions, perform 3 separate validity analyses, then consolidate and standardize the 3 analysis results
Important Notes
- Validity analysis is specifically designed for scale data only; non-scale data generally cannot undergo validity analysis
- Validity analysis requires comprehensive judgment based on multiple indicators: KMO value, Bartlett's test, variance explained rate, factor loading coefficients, and dimension-item correspondence
- The most critical aspect is whether dimension-item correspondence matches professional expectations
- Items with communality < 0.4, all factor loadings < 0.4, or misplaced items should be deleted
- "Entangled" items (high loadings on multiple factors) are common and usually acceptable; "misplaced" items must be handled
Frequently Asked Questions (FAQs)
Q1: How do I handle "entangled" items?
A: "Entangled" items are very common in actual analysis and usually do not require any treatment. However, if you have many items, you may consider removing entangled items. The specific handling depends on the researcher's comparison and selection of optimal results. But "misplaced" items must definitely be handled.
Q2: Is it necessary to perform validity analysis with only two items?
A: If a dimension corresponds to only two scale items, the KMO value will always be 0.5. You can directly describe the loading coefficient values. When loading coefficients are all greater than 0.4, validity is indicated. It is recommended to have more than 2 items per dimension (preferably 4-7 items). Having only two items to represent a dimension may easily lead to reliability issues.
Q3: Can I use only KMO value for validity analysis?
A: Validity analysis does not have fixed standards. The simplest validity test is to directly check whether the KMO value is greater than the standard (generally 0.6). For specific validity analysis methods, refer to relevant literature.
Q4: Can I extract factors when eigenvalues are not greater than 1?
A: Analysis usually requires comprehensive judgment based on your professional knowledge and software results. Even if eigenvalues are less than 1, factors can still be extracted.
Q5: What should I do if cumulative variance explained rate exceeds 100%?
A: Normally, cumulative variance explained rate should be less than 100%. If data multicollinearity is too severe, variance explained rate may exceed 100%. In this case, perform correlation analysis to find items with excessively high correlation (e.g., correlation coefficient > 0.8), remove them from the analysis, and re-analyze. Additionally, if the sample size is too small, this problem may also occur; increase the sample size.
Q6: Should I keep items deleted during reliability or validity analysis in subsequent analyses?
A: If an analysis item has been determined to be unreasonable and should be deleted during validity analysis (or reliability analysis, or other analyses), all subsequent analysis methods should be consistent. Do not delete the data directly, but simply exclude that item from analysis.
