The Social Vulnerability Index (SoVI) measurement to assess social vulnerability is only able to describe conditions in general, without being able to show which factors dominate the score. Therefore, the aim of this research is to fill this gap by applying a correlational approach with a clustering method to characterize the dominant factors of social vulnerability at the district level in Java and surrounding areas. The clustering method used in this study is the K-Medoids algorithm. This method is more powerful when there are outliers in the dataset used. In this study, we considered the use of 3 different distance methods, namely Euclidean distance, Manhattan distance, and Minkowski distance. As a result, the K-Medoids algorithm using Manhattan distance provides the best value based on the Davies Bouldin Index. This research found that social vulnerability exists in every region of Java Island and its surroundings.