Performance Analysis of CT-Scan Covid-19 Classification Using VGG16-SVM

Abstract

In this study, the researcher will compare basic or manually built CNN architecture with CNN transferred learning methods using Visual Geometry Group (VGG) with 16 layers architecture and Support Vector Machine (SVM) as its classifier. The VGG16-SVM method is proposed to prevent models from mispredicting and obtain higher accuracy, because of the small amount of data we use to train the model. This study intended to use small datasets or emphasize the amount of dataset to reduce the computational burden while still achieving remarkable performance. Support Vector Machine (SVM) is a method included in supervised learning and also great for image classification [14]. In several previous studies, SVM can provide slightly higher accuracy compared to the basic methods [15], [16]. This study aims to analyze the performance of pre-trained model VGG16 and SVM as its classifier for COVID-19 detection based on CT images. Next, the model's performance is assessed by comparing it with the Plain CNN method to evaluate its performance and effectiveness.

B. Research Method a. Method
Methodology is an important component in research, as it outlines the entire process from initial stage to the research result. It provides a structured approach to ensure that the research is carried out in a systematic manner. In this study, we proposed comparing model algorithm methods to analyze the model performance on diagnosis COVID-19 disease based on lung CT-Scan images. The methodology starts from preparing datasets as splitting, standardizing image dimensions and performing image augmentation techniques. Then performed the learning process as feature extraction and model fitting. Lastly, evaluate the model results to compare its performance. The methodology flow is shown in Figure 1.

b. Dataset
This study used a binary class and multi class datasets from Kaggle platform. The binary class dataset is COVID-19 Lung CT Scans, and Large COVID-19 CT Scan Slice as a multi class dataset. Data composition is shown in Table 1. The binary class dataset contains two label which positive and negative images. Then the multi class datasets contains three label which positive, normal or negative and cap images. Figure 2 and 3 shows the image samples from two COVID-19 CT-Scans Datasets.

c. Data Preparation
Data preparation's primary objective is to prepare the raw data for further analysis. In this section, all image dimensions are set to 224 x 224 pixels, and the data is splitted into train and test with 80:20 percent ratio. We proposed to reduce the Large COVID-19 CT Scan Slice samples from 17104 to 3000 data using the random sample technique to get the random 3000 sample data from its original dataset. to analyze the model performance on small datasets and prevent crashes while training the data.

d. Image Augmentation
Overfitting is a problem when a machine learning models are considered as overly complex and perform overly well when fitting the data for training section, which leads to a lower performance on new data or test data. The application of image augmentation techniques is to overcome overfitting problems in machine learning models [17]. Image augmentation is a regularization technique used to reduce model complexity and ensure that the model can properly generalize to new data. We performed an Image augmentation technique to increase the data sample and prevent models from overfitting since the data sample we use in this study is small. Table 2 shows the data composition after applying image augmentation technique.  [18]. We will use the VGG16 pre-trained model as a feature extractor. In this study, all the 16 layers of VGG16 will not be used but only using the convolutional layers and discarding the dense layer. After several data preparations, we extract the data feature using VGG16 and normalize its shape.

f. Train Model SVM
SVM (Support Vector Machine) is used to classify extracted features from the VGG16 feature extraction stage. We selected SVM as our classifier based on its performance in accurately classifying datasets with high-dimensionality features [19]. In this stage, we performed model training on the extracted features from VGG16 using SVM linear kernel. Figure 4 shows the flow of training model SVM after the VGG16 feature extraction stage.

g. Plain CNN
In this study, the researcher propose to use a Plain CNN or normal CNN to compare its performance with the VGG16-SVM method. The Plain CNN architecture for binary and multi class dataset is slightly different. The architecture consists of 4 convolution and pooling layers, 3 fully connected layers and 1 dropout layer for the multi class dataset.

h. Train Model (CNN)
In this stage, after the Plain CNN model was created the next stage is training process. We used ADAM as an optimizer. The model will be trained using 50 epochs and 35 steps per epoch with 32 batch size.

i. Evaluate Model
Model evaluation is the last stage in this research. The model evaluation process is conducted to obtain information on the model's performance [20]. This last process will be comparing the performance of Plain CNN and CNN based VGG16-SVM Classifier. After the classifying process, the confusion matrix method will be used to calculate the number of TP (True Positive), FP (False Positive), TN (True Negative), and FN (False Negative), to get the accuracy, precision, recall, and f1score [21].

C. Result and Discussion
In this result and discussion stage, it describes the two model algorithm method performance on classifying binary and multiclass COVID-19 image datasets. The model result obtained by predicting the test data after the training section. After the training section, the machine will predict the new image from test data using the knowledge from the training section. The prediction results will be used to perform a comparative analysis between the VGG16-SVM method and the Plain CNN method. Comparison points consist of accuracy, precision, recall and f1-score. The Plain CNN and VGG16-SVM performance have a huge difference. Where the VGG16-SVM got higher score compared to the Plain CNN. The VGG16-SVM can have better performance because of its capability to work on complex data [22].

a. Confusion Matrix
Confusion matrix is a technique used to obtain information about the model's performance in classification. From the confusion matrix, the results can be used to calculate the model's accuracy, precision, recall, and f1-score value.

b. Performance
After the confusion matrix section, the confusion matrix value will be used to calculate the model value of accuracy, precision, recall, and f1-score.
a. Accuracy Accuracy is an evaluation metric that measures the percentage of correct classification that the model can successfully predict. Accuracy can provide information about the performance of the model in predicting the correct classification. The accuracy value obtained by calculating the number of correct prediction values divided by the total number of evaluation data.
c. Recall The classification report is in the form of accuracy, precision, recall, and f1-score results after calculating the confusion matrix values which are shown in Tables 3  and 4.  Table 3 provides information on the evaluation performance of the Plain CNN model in classifying two datasets, which are binary class and multiclass datasets. It shows that the performance of the Plain CNN models in classifying has a better value when using binary datasets. This model can achieve an accuracy rate of 0.52, precision 0.55, recall 0.50, and f1-score 0.53. The performance of the model decreases when using multiclass datasets. This is indicated by the accuracy value of 0.32, precision 0.32, recall 0.32, and f1-score 0.32.

c. Discussion
There are several things obtained after analyzing the performance of the Plain CNN and VGG16-SVM methods for detecting deathly disease named COVID-19 based on CT scan images after the research was conducted. For the Plain CNN method, the model has an optimal performance when classifying binary class datasets while the performance decreases when classifying multiclass datasets. The results of Plain CNN in this study show that the performance is better when classifying data that tends to be simple. This is evidenced by the performance when classifying on binary datasets, the model can get a higher evaluation value compared to the evaluation value on multiclass datasets. Furthermore, for the VGG16-SVM method, based on the research that has been done, it shows that the performance of the method shows an opposite result to the Plain CNN method. The Plain CNN method works better on binary datasets while the VGG16-SVM method shows better performance when performing classification on multiclass datasets. This is shown in the evaluation value obtained from the VGG16-SVM classification results on multiclass datasets which has a higher performance than binary class datasets.
This may occur perhaps due to the capability of SVM in performing classification on more complex data, in this case a multiclass dataset. A multiclass dataset contains three classes or labels for each image. The more labels contained in the dataset, the more varied the images will be. It can cause the machine to find it difficult to perform the classification process. This may be the reason why the performance of the Plain CNN method decreases when performing classification using the multiclass dataset. Since the data is more complex, the machine has difficulty in performing classification so that the machine makes prediction errors that cause its performance to decrease. In contrast to VGG16-SVM, the performance of this method increases compared to the Plain CNN method because the capability of SVM as a classifier that can work well on complex data. Therefore, the performance of VGG16-SVM on multiclass datasets is also higher than its performance on binary datasets.
According to the research findings, in order to provide further evidence regarding the performance of the proposed methods, this can be done by equalizing the amount of data used in binary class and multiclass datasets. This is intended to determine the performance of the Plain CNN and VGG16-SVM methods in classifying binary class and multiclass datasets. Whether the performance of Plain CNN really increases when using binary class datasets and decreases when using multiclass datasets or not. Likewise with VGG16-SVM, whether the performance of this method is really better when using multiclass datasets and decreases when using binary class datasets. Thus, the performance analysis process will achieve a better result.

d. Limitations and Future Research
In this study, researchers limit several aspects that will be studied. These several aspects include comparing the performance of models which in this case researchers propose to use the Plain CNN and VGG16-SVM methods. In addition, how to overcome resource limitations during the research process so that it can achieve the optimal results. The researcher aims to limit the problem so that the research is more focused and does not widen to other issues. The researcher focuses on analyzing the performance of the two proposed methods, which are the Plain CNN and VGG16-SVM in classifying coronavirus illness data according to lung CT scan images. The research was conducted using binary and multiclass datasets obtained from the public dataset platform named Kaggle. Total data from the binary class dataset is 764 images, while the total data from the multiclass dataset is 17.104 images.
In the beginning, the research was carried out by conducting an experiment on both datasets first. When implementing both proposed methods on binary class datasets, the process worked well but performed poorly. Later, when performing the same implementation on multiclass datasets, Plain CNN could work properly, but the model crashed when performing the process using VGG16-SVM. Therefore, researchers are trying to overcome problems related to the high computational load when performing the method implementation process on multiclass datasets by reducing the total amount of samples used. The amount of sample reduction is done by taking 1000 random samples from each COVID, NonCOVID, and CAP class. Image augmentation is applied after the reduction process of the amount of data samples that will be used for the binary and multiclass dataset to increase the variety of image samples so the model can make better predictions.
In the future, disease detection using radiology images, particularly CT scans image by applying methods such as those used in this study, which is Plain CNN or VGG16-SVM, can be further developed by using more and better resources so that the resulting model has better and more optimal performance. This study also can be developed by using several other image classification methods. So that the results of the research related to disease detection are able to help the healthcare practitioners in diagnosing diseases based on radiology images to be faster. Thus, medical action can be taken immediately and also can minimize the errors.

D. Conclusion
In this study, after comparing two algorithm methods for classifying the small COVID-19 image datasets. The researcher found that the performance of each method decreased and increased when classifying binary class and multiclass. According to the research results that have been carried out, it shows that the Plain CNN method has a better performance in classifying COVID-19 disease according to CT image using binary class datasets with 52% accuracy while for multiclass datasets it gets an accuracy of 33%, and for the VGG16-SVM method has a better performance in classifying using multiclass datasets with 96% accuracy while on binary class datasets obtains an accuracy of 89%. Both proposed methods have their advantages, the performance of Plain CNN decreases when classifying complex data but increases when the data is not too complex, and the performance of VGG16-SVM increases when classifying complex data and decreases when using data that tends to be simple.
The conclusion is that the VGG16-SVM is the better method to performed the COVID-19 classification using images from the lung CT scan. This method could be used to perform a detection on complex data such as COVID-19 CT images with more accurately. With 89% accuracy on binary class dataset, and 96% accuracy on multiclass dataset. In addition, this study can also conclude that small data resources and low hardware performance problems on image classification can be overcome by implementing this VGG16-SVM method.