GeoDa functions covered

• Clusters > PCA
  • select variables
  • PCA parameters
  • PCA summary statistics
  • saving PCA results
• Clusters > MDS
  • select variables
  • MDS parameters
  • saving MDS results
  • spatial weights from MDS results

Variable Settings Panel

We select the variables and set the parameters for the principal components analysis through the options in the left hand panel of the interface. We choose the six same variables as in the multivariate analysis presented in Dray and Jombart (2011): Crm_prs (crimes against persons), Crm_prp (crimes against property), Litercy (literacy), Donatns (donations), Infants (births out of wedlock), and Suicids (suicides) (see also Anselin 2018, for an extensive discussion of the variables).⁴ These variables appear highlighted in the Select Variables panel, Figure 4.

The default settings for the Parameters are likely fine in most practical situations. The Method is set to SVD, i.e., singular value decomposition. The alternative Eigen carries out an explicit eigenvalue decomposition of the correlation matrix. In our example, the two approaches yield opposite signs for the loadings of three of the components. We return to this below. For larger data sets, the singular value decomposition approach is preferred.

The Transformation option is set to Standardize (Z) by default, which converts all variables such that their mean is zero and variance one, i.e., it creates a z-value as \[z = \frac{(x - \bar{x})}{\sigma(x)},\] with \(\bar{x}\) as the mean of the original variable \(x\), and \(\sigma(x)\) as its standard deviation.

An alternative standardization is Standardize (MAD), which uses the mean absolute deviation (MAD) as the denominator in the standardization. This is preferred in some of the clustering literature, since it dimineshes the effect of outliers on the standard deviation (see, for example, the illustration in Kaufman and Rousseeuw 2005, 8–9). The mean absolute deviation for a variable \(x\) is computed as: \[\mbox{mad} = (1/n) \sum_i |x_i - \bar{x}|,\] i.e., the average of the absolute deviations between an observation and the mean for that variable. The estimate for \(\mbox{mad}\) takes the place of \(\sigma(x)\) in the denominator of the standardization expression.

Other choices for the Transformation option are to use the variables in their original scale (Raw), or as deviations from the mean (Demean).

After clicking on the Run button, the summary statistics appear in the right hand panel, as shown in Figure 5. We return for a more detailed interpretation below.

Saving the Results

Once the results have been computed, a value appears next to the Output Components item in the left panel, shown in Figure 6. This value corresponds to the number of components that explain 95% of the variance, as indicated in the results panel. This determines the default number of components that will be added to the data table upon selecting Save. The value can be changed in the drop-down list. For now, we keep the number of components as 5, even though that is not a very good result (given that we started with only six variables).

The Save button brings up a dialog to select variable names for the principal components, shown in Figure 7. The default is generic and not suitable for a situation where a large number of analyses will be carried out. In practice, one would customize the components names to keep the results from different computations distinct.

Clicking on OK will add the principal components to the data table, where they become available for all types of analysis and visualization. As always, the addition is only made permanent after saving the table.

Explained variance

The first item lists the Standard deviation explained by each of the components. This corresponds to the square root of the Eigenvalues (each eigenvalue equals the variance explained by the corresponding principal component), which are listed as well. In our example, the first eigenvalue is 2.14047, which is thus the variance of the first component. Consequently, the standard deviation is the square root of this value, i.e., 1.463034, given as the first item in the list.

The sum of all the eigenvalues is 6, which equals the number of variables, or, more precisely, the rank of the matrix \(X'X\). Therefore, the proportion of variance explained by the first component is 2.14047/6 = 0.356745, as reported in the list. Similarly, the proportion explained by the second component is 0.200137, so that the cumulative proportion of the first and second component amounts to 0.356745 + 0.200137 = 0.556882. In other words, the first two components explain a little over half of the total variance.

The fraction of the total variance explained is listed both as a separate Proportion and as a Cumulative proportion. The latter is typically used to choose a cut-off for the number of components. A common convention is to take a threshold of 95%, which would suggest 5 components in our example. Note that this is not a good result, given that we started with 6 variables, so there is not much of a dimension reduction.

An alternative criterion to select the number of components is the so-called Kaiser criterion (Kaiser 1960), which suggests to take the components for which the eigenvalue exceeds 1. In our example, this would yield 3 components (they explain about 74% of the total variance).

The bottom part of the results panel is occupied by two tables that have the original variables as rows and the components as columns.

Variable loadings

The first table shows the Variable Loadings. For each component (column), this lists the elements of the corresponding eigenvector. The eigenvectors are standardized such that the sum of the squared coefficients equals one. The elements of the eigenvector are the coefficients by which the original variables need to be multiplied to construct each component. For example, for each observation, the first component would be computed by multiplying the value for Crm_prs by -0.0658684, add to that the value for Crm_prp multiplied by -0.512326, etc.

It is important to keep in mind that the signs of the loadings may change, depending on the algorithm that is used in their computation, but the absolute value of the coefficients will be the same. In our example, setting Method to Eigen yields the loadings shown in Figure 8.

For PC1, PC4, and PC6, the signs for the loadings are the opposite of those reported in Figure 5. This needs to be kept in mind when interpreting the actual value (and sign) of the components.

When the original variables are all standardized, each eigenvector coefficient gives a measure of the relative contribution of a variable to the component in question. These loadings are also the vectors employed in a principal component biplot, a common graph produced in a principal component analysis (but not currently available in GeoDa).

Substantive interpretation of principal components

The interpretation and substantive meaning of the principal components can be a challenge. In factor analysis, a number of rotations are applied to clarify the contribution of each variable to the different components. The latter are then imbued with meaning such as “social deprivation”, “religious climate”, etc. Principal component analysis tends to stay away from this, but nevertheless, it is useful to consider the relative contribution of each variable to the respective components.

The table labeled as Squared correlations lists those statistics between each of the original variables in a row and the principal component listed in the column. Each row of the table shows how much of the variance in the original variable is explained by each of the components. As a result, the values in each row sum to one.

More insightful is the analysis of each column, which indicates which variables are important in the computation of the matching component. In our example, we see that Crm_prp, Litercy, Infants and Suicids are the most important for the first component, whereas for the second component this is Crm_prs and Donatns. As we can see from the cumulative proportion listing, these two components explain about 56% of the variance in the original variables.

Since the correlations are squared, they do not depend on the sign of the eigenvector elements, unlike the loadings.

Visualizing principal components

Once the principal components are added to the data table, they are available for use in any graph (or map).

A useful graph is a scatter plot of any pair of principal components. For example, such a graph is shown for the first two components (based on the SVD method) in Figure 9. By construction, the principal component variables are uncorrelated, which yields the characteristic circular cloud plot. A regression line fit to this scatter plot yields a horizontal line (with slope zero). Through linking and brushing, we can identify the locations on the map for any point in the scatter plot.

To illustrate the effect of the choice of eigenvalue computation, the scatter plot in Figure 10 again plots the values for the first and second component, but now using the loadings from the Eigen method. Note how the scatter has been flipped around the vertical axis, since what used to be positive for PC1, is now negative, and vice versa.

In order to gain further insight into the composition of a principal component, we combine a box plot of the values for the component with a parallel coordinate plot of its main contributing variables. For example, in the box plot in Figure 11, we select the observations in the top quartile of PC1 (using SVD).⁵

We link the box plot to a parallel coordinate plot for the four variables that contribute most to this component. From the analysis above, we know that these are Crm_prp, Litercy, Infants and Suicids. The corresponding linked selection in the PCP shows low values for all but Litercy, but the lines in the plot are all close together. This nicely illustrates how the first component captures a clustering in multivariate attribute space among these four variables.

Since the variables are used in standardized form, low values will tend to have a negative sign, and high values a positive sign. The loadings for PC1 are negative for Crm_prp, Infants and Suicids, which, combined with negative standardized values, will make a (large) positive contribution to the component. The loadings for Litercy are positive and they multiply a (large) postive value, also making a positive contribution. As a result, the values for the principal component end up in the top quartile.

The substantive interpretation is a bit of a challenge, since it suggests that high property crime, out of wedlock births and suicide rates (recall that low values for the variables are bad, so actually high rates) coincide with high literacy rates (although the latter are limited to military conscripts, so they may be a biased reflection of the whole population).

Principal component maps

The visualization of the spatial distribution of the value of a principal component by means of a choropleth map is mostly useful to suggest certain patterns. Caution needs to be used to interpret these in terms of high or low, since the latter depends on the sign of the component loadings (and thus on the method used to compute the eigenvectors).

For example, using the results from SVD for the first principal component, a quartile map shows a distinct pattern, with higher values above a diagonal line going from the north west to the middle of the south east border, as shown in Figure 12.

A quartile map for the first component computed using the Eigen method shows the same overall spatial pattern, but the roles of high and low are reversed, as in Figure 13.

Cluster maps

We pursue the nature of this pattern more formally through a local Moran cluster map. Again, using the first component from the SVD method, we find a strong High-High cluster in the northern part of the country, ranging from east to west (with p=0.01 and 99999 permutations, using queen contiguity). In addition, there is a pronounced Low-Low cluster in the center of the country.⁶

Principal components as multivariate cluster maps

Since the principal components combine the original variables, their patterns of spatial clustering could be viewed as an alternative to a full multivariate spatial clustering. The main difference is that in the latter each variable is given equal weight, whereas in the principal component some variables are more important than others. In our example, we saw that the first component is a reasonable summary of the pattern among four variables, Crm_prp, Litercy, Infants and Suicids.

First, we can check on the makeup of the component in the identified cluster. For example, we select the high-high values in the cluster map and find the matching lines in a parallel coordinate plot for the four variables, shown in Figure 15. The lines track closely (but not perfectly), suggesting some clustering of the data points in the four-dimensional attribute space. In other words, the high-high spatial cluster of the first principal component seems to match attribute similarity among the four variables.

In addition, we can now compare these results to a cluster or significance map from a multivariate local Geary analysis for the four variables. In Figure 16, we show the significance map rather than a cluster map, since all significant locations are for positive spatial autocorrelation (p < 0.01, 99999 permutations, queen contiguity). While the patterns are not identical to those in the principal component cluster map, they are quite similar and pick up the same general spatial trends.

This suggests that in some instances, a univariate local spatial autocorrelation analysis for one or a few dominant principal components may provide a viable alternative to a full-fledged multivariate analysis.

Distance measures

The Parameters panel in the MDS Settings dialog allows for a number of options to be set. The Transformation option is the same as for PCA, with the default set to Standardize (Z), but Standardize (MAD), Raw and Demean are available as well. Best practice is to use the variables in standardized form.

Another important option is the Distance Function setting. The default is to use Euclidean distance, but an alternative is Manhattan block distance, or, the use of absolute differences instead of squared differences, shown in Figure 21. The effect is to lessen the impact of outliers, or large distances.

The result that follows from a Manhattan distance metric can seem slightly different from the default Euclidean MDS plot. As seen in Figure 22, the scale of the coordinates is different, but the identification of close observations can also differ somewhat between the two plots. On the other hand, the identification of outliers is fairly consistent between the two.

For example, using the linking and brushing functionality shown in Figure 23, we can select two close points in the Manhattan graph and locate the matching points in the Euclidean graph. As it turns out, in our example, the matching locations in the Euclidean graph appear to be further apart, but still in proximity of each other (to some extent, the seeming larger distances may be due to the different scales in the graphs).

On the other hand, when we select an outlier in the Manhattan graph (a point far from the remainder of the point cloud), as in Figure 24, the matching observation in the Euclidean graph is an outlier as well.

Power approximation

An option that is particularly useful in larger data sets is to use a Power Iteration to approximate the first few largest eigenvalues needed for the MDS algorithm. In large data sets with many variables, the standard algorithm will tend to break down.

This option is selected by checking the box in the Parameters panel in Figure 25. The default number of iterations is set to 100, which should be fine in most situations. However, if needed, it can be adjusted by entering a different value in the corresponding box.

At first sight, the result in Figure 26 may seem different from the standard output in Figure 20, but this is due to the indeterminacy of the signs of the eigenvectors.

Closer inspection of the two graphs suggests that the axes are flipped. We can clearly see this by selecting two points in one graph and locating them in the other, as in Figure 27. The relative position of the points is the same in the two graphs, but they are on opposite sides of the origin on the y-axis.

Linking a location and its neighbors

We can also select the neighbors explicitly in the map (provided there is an active spatial weights matrix, in our example, this is the queen contiguity) using the selection and neighbors option (right click in the map to activate the options menu and select Connectivity > Show Selection and Neighbors). This allows an investigation of the tension between locational similarity (neighbors on the map) and attribute similarity (neigbhors in the MDS plot) for any selection and its neighbors. In our example in Figure 31, we selected one department (the location of the cursor on the map) with its neighbors (using first order queen contiguity) and identify the corresponding points in the MDS plot. For the selected observation, there is some evidence of a match between the two concepts for a few of the points, but not for others.

Matching attribute and geographic neighbors

We can now investigate the extent to which neighbors in geographic space (geographic queen contiguity) match neighbors in attribute space (queen contiguity from MDS scatter plot) using the Weights Intersection functionality in the Weights Manager (Figure 35).

This creates a gal weights file (guerry_85_mds_queen.gal in our example) that lists for each observation the neighbors that are shared between the two queen contiguity files. As is to be expected, the resulting file is much sparser than the original weights. As Figure 36 illustrates, we notice several observations that have 0 neighbors in common. But in other places, we find quite a bit of commonality. For example, for location with dept=2, we have 3 neighbors in common, 60, 59, and 80.⁸

The full list of properties of the weights intersection are listed in the Weights Manager, as shown in Figure 37. The mean and median number of neighbors is much smaller than for the original weights. For example, the median is 1, compared to 5 for geographic contiguity and 6 for MDS contiguity. Also, the min neighbors of 0 suggests the presence of isolates, which we noticed in Figure 36.

Finally, we can visualize the overlap between geographic neighbors and attribute neighbors by means of a Connectivity Graph. In Figure 38, the blue lines connect observations that share the two neighbor relations (superimposed on a quartile map of the first principal component).

Several sets of interconnected vertices suggest the presence of multivariate clustering, for example, among the departments enclosed by the green rectangle on the map. Although there is no significance associated with these potential clusters, this approach illustrates yet another way to find locations where there is a match between locational and multivariate attribute similarity.