diff --git a/recognition/s4627234_3710project/README.MD b/recognition/s4627234_3710project/README.MD new file mode 100644 index 0000000000..633875cf14 --- /dev/null +++ b/recognition/s4627234_3710project/README.MD @@ -0,0 +1,159 @@ + +# Using an U-NET to segment the ISIC dataset + + +## Author +Name: Jingming Dai + +Student number: s4627234 / 46272346 + +This project was completed for COMP3710 + + + +## Description +This project uses Improved UNet to split the ISIC dataset with a minimum Dice similarity coefficient of 0.8 for all labels on the test set. + +In image segmentation tasks, especially medical image segmentation, U-Net is undoubtedly one of the most successful methods. U-net uses encoder (down-sampling) and decoder (up-sampling) structural connections. This project applies a technique that uses the Dice loss to train the model. Compared with the common U-Net, this model has better segmentation effect and higher dice similarity coefficient. + + +## Data set description: +The ISIC package includes four folders, including Training, Testing, Validation and their Ground Truth files. After downloading the files, we can put them into the newly created data folder. The directory of the files needs to be arranged as follows. + +* data + * data_ISIC + * ISIC-2017_Training_Data + * ISIC-2017_Training_Part1_GroundTruth + * ISIC-2017_Test_v2_Data + * ISIC-2017_Test_v2_Part1_GroundTruth + * ISIC-2017_Validation_Data + * ISIC-2017_Validation_Part1_GroundTruth + +![image](./images/data_image_example.png) + + +## How it works: + +### UNet: +The structure of U-Net is shown in the following figure, the left part can be regarded as an encoder (down), and the right part can be regarded as a decoder (up). + +![image](./images/UNet.png) + +The encoder has four submodules, each submodule contains two convolution layers, and each submodule is followed by a down-sampling layer implemented by MaxPool2D. + +The decoder consists of four sub-modules, and the resolution is sequentially increased by up-sampling operations until it is consistent with the resolution of the input image. The decoder uses two 3x3 un-padded convolutions and . After four up-sampling, a final 1x1 convolution with a sigmoid activation function is applied. + + +### Improved_UNet : +The algorithm is a modified version of UNet created by Isensee and colleagues, and below is the improved UNet graph. The UNet() and UNet_imp() functions in modules.py build the UNet model before and after improvement. The model is built according to the following UNet architecture, but changing the output "softmax" to "sigmoid" at the end so the output is a mask of one channel. + +![image](./images/model_imp.png) + +#### Important parts of improved UNet: + +All the convolution, context, element-wise-sum in encoding are integrated into the up_imp() function. + +Part of upsampling, concatenate, and localization in decoding are integrated into the down_imp() function. + +__Context module:__ +Described by 2 convolution layers (all except the first one with stride 2) with a dropout layer (0.3) in between. + +__Upsampling module:__ +It consists of an upsampling layer and a convolution layer. + +__Localization module:__ +Completed by a 3x3 convolution and a 1x1 convolution. + +Instance normalization and Leaky reLU are used throughout the architecture. The model is compiled with dice coefficient loss and dice coefficient. + + +### Dice Similarity Coefficient (DSC): +is an ensemble similarity measure function, usually used to calculate the similarity of two samples. It is used as the loss function and validation segmentation image in this model. + + +## Install: +``` +git clone https://github.com/shakes76/PatternFlow.git +``` + + +## Using: +!!! Before running it, you must set path_data to the path of the data folder in the dataset.py folder. + +When using this data for the first time, you need to use dataset.load_dataset(data_reshape = True), this setting will convert all non-uniform data pattern sizes into 256*256 size patterns and save them in a new folder "data_Reshape" in the "data" folder ". The data format is as follows: + +* data_Reshaped + * Train + * Train_GT + * Test + * Test_GT + * Val + * Val_GT + +!!! Always remember to use (data_reshape = False) if reshaped data already exists + +__run train.py to training and save the model (model save in the current folder):__ +(The specific code needs to be changed according to the specific device environment) +``` +python3.9 train.py +``` + +__run predict.py to load the model and predict mask with validation data:__ +(The specific code needs to be changed according to the specific device environment) +``` +python3.9 predict.py +``` + +## Testing and Conclusion: +Train and test data are use on training and testing the model. + +Run for 10 epochs will get (this is a very low epochs value +, please select more in train.py if wants a better result): + +### model DSC: + +![image](./images/DSC.png) + + +### model loss: + +![image](./images/loss.png) + + +Evaluating the model through the val dataset, we can get the DSC data: +tf.Tensor(0.83824617, shape=(), dtype=float32) + +The image folder contains the first 50 original images, predict mask , and ground truth comparison. You can also change the output image number by changing the number_list in predict.py. + +### Some example of images: +![image](./images/output0.png) +![image](./images/output5.png) +![image](./images/output10.png) +![image](./images/output13.png) +![image](./images/output37.png) + + + +## Packages: +os, cv2, skimage, tensorflow-macos_version_2.9.2 + +SimpleITK_version_2.2.0 +-- SimpleITK is a simplified interface to the Insight Toolkit (ITK) for image registration and segmentation +(http://simpleitk.org/) + +numpy_version_1.23.1 +-- NumPy is the fundamental package for array computing with Python. +(https://www.numpy.org) + +pandas_version_1.4.4 +-- Powerful data structures for data analysis, time series, and statistics +(https://pandas.pydata.org) + +matplotlib_version_3.5.2 +-- Python plotting package +(https://matplotlib.org) + + +## Reference + +Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., & Maier-Hein, K. H. (2018, February 28). Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge. arXiv.org. Retrieved October 19, 2022, from https://arxiv.org/abs/1802.10508v1 \ No newline at end of file diff --git a/recognition/s4627234_3710project/commit-history1.png b/recognition/s4627234_3710project/commit-history1.png new file mode 100644 index 0000000000..d28c170807 Binary files /dev/null and b/recognition/s4627234_3710project/commit-history1.png differ diff --git a/recognition/s4627234_3710project/commit-history2.png b/recognition/s4627234_3710project/commit-history2.png new file mode 100644 index 0000000000..7986682df3 Binary files /dev/null and b/recognition/s4627234_3710project/commit-history2.png differ diff --git a/recognition/s4627234_3710project/commit_history3.png b/recognition/s4627234_3710project/commit_history3.png new file mode 100644 index 0000000000..3ca71820fc Binary files /dev/null and b/recognition/s4627234_3710project/commit_history3.png differ diff --git a/recognition/s4627234_3710project/dataset.py b/recognition/s4627234_3710project/dataset.py new file mode 100644 index 0000000000..3d326ccd5c --- /dev/null +++ b/recognition/s4627234_3710project/dataset.py @@ -0,0 +1,168 @@ + +import os +import cv2 as cv +import SimpleITK as sitk +import numpy as np +import pandas as pd +import matplotlib +import matplotlib.pyplot as plt + +import skimage +from skimage import io + +# ISIC data format using 2017 ISIC data + +# Because original data from ISIC has multiple different size, so we +# need to reshape them into a better size for doing ML +# if data_reshape is True, then the program will create a new data folder in +# the given direction and reshape all the ISIC images into given size +# (.csv will still in the original position and will not change) + +""" + Change the path to the path of the IRIS folder, + e.g. if the folder path is ./data/data_ISIC, use path_data = "./data" +""" +path_data = "/Users/davedai/Desktop/MySolution/data" + +def create_data(data_from, data_images, data_to, img_size = 256): + """ Create data image based on the given data image to the given direction. + new data should have the given image size. + + Args: + data_from (String): From direction + data_images (list): list of images + data_to (String): To direction + img_size (int, optional): image size to be transformed. Defaults to 256. + """ + for i in data_images: + img=sitk.ReadImage(os.path.join(data_from,i)) + img_array=sitk.GetArrayFromImage(img) + new_array=cv.resize(img_array,(img_size,img_size)) + data_name = i[:-4] # removing last four (.jpg/.png/...) + + io.imsave(data_to + data_name + '.png', new_array) + + +def reshape_data(): + """ + Reshape image into given size and save into the new file + """ + # training image path + train_path = path_data + '/data_ISIC/ISIC-2017_Training_Data/' + train = [fn for fn in os.listdir(train_path) if fn.endswith('jpg')] + train.sort() + + # training ground truth path + train_path_gt = path_data + '/data_ISIC/ISIC-2017_Training_Part1_GroundTruth/' + train_gt = [fn for fn in os.listdir(train_path_gt) if fn.endswith('png')] + train_gt.sort() + + # test image path + test_path = path_data + '/data_ISIC/ISIC-2017_Test_v2_Data' + test = [fn for fn in os.listdir(test_path) if fn.endswith('jpg')] + test.sort() + + # test ground truth images + test_path_gt = path_data + '/data_ISIC/ISIC-2017_Test_v2_Part1_GroundTruth' + test_gt = [fn for fn in os.listdir(test_path_gt) if fn.endswith('png')] + test_gt.sort() + + # validation image path + val_path = path_data + '/data_ISIC/ISIC-2017_Validation_Data' + val = [fn for fn in os.listdir(val_path) if fn.endswith('jpg')] + val.sort() + + # validation image path + val_path_gt = path_data + '/data_ISIC/ISIC-2017_Validation_Part1_GroundTruth' + val_gt = [fn for fn in os.listdir(val_path_gt) if fn.endswith('png')] + val_gt.sort() + + if not os.path.exists(path_data + '/data_Reshaped'): + os.mkdir(path_data + '/data_Reshaped/') + os.mkdir(path_data + '/data_Reshaped/Train') + os.mkdir(path_data + '/data_Reshaped/Train_GT') + os.mkdir(path_data + '/data_Reshaped/Test') + os.mkdir(path_data + '/data_Reshaped/Test_GT') + os.mkdir(path_data + '/data_Reshaped/Val') + os.mkdir(path_data + '/data_Reshaped/Val_GT') + + create_data(train_path, train, (path_data + '/data_Reshaped/Train/')) + create_data(train_path_gt, train_gt, (path_data + '/data_Reshaped/Train_GT/')) + create_data(test_path, test, (path_data + '/data_Reshaped/Test/')) + create_data(test_path_gt, test_gt, (path_data + '/data_Reshaped/Test_GT/')) + create_data(val_path, val, (path_data + '/data_Reshaped/Val/')) + create_data(val_path_gt, val_gt, (path_data + '/data_Reshaped/Val_GT/')) + + +def load_data(csv, path_image, path_image_gt): + """ load the data and its mask from the given path with the order in csv file + csv file only use for getting the names of the images + + Args: + csv (pandas.core.frame.DataFrame): csv file by using pandas to read + path_image (String): image path + path_image_gt (String): image ground truth path + + Returns: + numpy.ndarray, numpy.ndarray: return numpy.ndarray of images and it's mask + """ + x, y = [], [] + for _, i in csv.iterrows(): + image = sitk.ReadImage(path_image + i[0]+'.png') + image_array_ = sitk.GetArrayFromImage(image) + image_array = image_array_/255.0 + x.append(image_array) + + mask_ = cv.imread(path_image_gt + i[0]+'_segmentation.png') + mask = mask_/255.0 + y.append(mask) + + return np.array(x), np.array(y) + + +def load_dataset(data_reshape = False): + """ Load the dataset, if the data need to reshape(data_reshape = True) then reshape the dataset + + Args: + data_reshape (bool, optional): reshape the data if True. Defaults to False. + + Returns: + numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray: + return the image and its mask image for all training and testing data + """ + if data_reshape: + reshape_data() + + train_csv = pd.read_csv(path_data + '/data_ISIC/ISIC-2017_Training_Data/ISIC-2017_Training_Data_metadata.csv') + test_csv = pd.read_csv(path_data + '/data_ISIC/ISIC-2017_Test_v2_Data/ISIC-2017_Test_v2_Data_metadata.csv') + + path_train = path_data + '/data_Reshaped/Train/' + path_train_gt = path_data + '/data_Reshaped/Train_GT/' + + path_test = path_data + '/data_Reshaped/Test/' + path_test_gt = path_data + '/data_Reshaped/Test_GT/' + + train_x, train_y = load_data(train_csv, path_train, path_train_gt) + test_x, test_y = load_data(test_csv, path_test, path_test_gt) + + return train_x, train_y, test_x, test_y + + + + +def load_val(): + """ Load the dataset + + Returns: + numpy.ndarray, numpy.ndarray: return the image and its mask image for all val data + """ + val_csv = pd.read_csv(path_data + '/data_ISIC/ISIC-2017_Validation_Data/ISIC-2017_Validation_Data_metadata.csv') + + + path_val = path_data + '/data_Reshaped/Val/' + path_val_gt = path_data + '/data_Reshaped/Val_GT/' + + val_x, val_y = load_data(val_csv, path_val, path_val_gt) + + return val_x, val_y + diff --git a/recognition/s4627234_3710project/images/DSC.png b/recognition/s4627234_3710project/images/DSC.png new file mode 100644 index 0000000000..79842d5795 Binary files /dev/null and b/recognition/s4627234_3710project/images/DSC.png differ diff --git a/recognition/s4627234_3710project/images/UNet.png b/recognition/s4627234_3710project/images/UNet.png new file mode 100644 index 0000000000..d87587eb84 Binary files /dev/null and b/recognition/s4627234_3710project/images/UNet.png differ diff --git a/recognition/s4627234_3710project/images/data_image_example.png b/recognition/s4627234_3710project/images/data_image_example.png new file mode 100644 index 0000000000..645d708332 Binary files /dev/null and b/recognition/s4627234_3710project/images/data_image_example.png differ diff --git a/recognition/s4627234_3710project/images/loss.png b/recognition/s4627234_3710project/images/loss.png new file mode 100644 index 0000000000..6a42ca81ab Binary files /dev/null and b/recognition/s4627234_3710project/images/loss.png differ diff --git a/recognition/s4627234_3710project/images/model256.png b/recognition/s4627234_3710project/images/model256.png new file mode 100644 index 0000000000..096d2081a6 Binary files /dev/null and b/recognition/s4627234_3710project/images/model256.png differ diff --git a/recognition/s4627234_3710project/images/model_imp.png b/recognition/s4627234_3710project/images/model_imp.png new file mode 100644 index 0000000000..53979ecdbd Binary files /dev/null and b/recognition/s4627234_3710project/images/model_imp.png differ diff --git a/recognition/s4627234_3710project/images/output0.png b/recognition/s4627234_3710project/images/output0.png new file mode 100644 index 0000000000..1e97739357 Binary files /dev/null and b/recognition/s4627234_3710project/images/output0.png differ diff --git a/recognition/s4627234_3710project/images/output1.png b/recognition/s4627234_3710project/images/output1.png new file mode 100644 index 0000000000..5a1d72e43b Binary files /dev/null and b/recognition/s4627234_3710project/images/output1.png differ diff --git a/recognition/s4627234_3710project/images/output10.png b/recognition/s4627234_3710project/images/output10.png new file mode 100644 index 0000000000..50db94efc7 Binary files /dev/null and b/recognition/s4627234_3710project/images/output10.png differ diff --git a/recognition/s4627234_3710project/images/output11.png b/recognition/s4627234_3710project/images/output11.png new file mode 100644 index 0000000000..8deda91db3 Binary files /dev/null and b/recognition/s4627234_3710project/images/output11.png differ diff --git a/recognition/s4627234_3710project/images/output12.png b/recognition/s4627234_3710project/images/output12.png new file mode 100644 index 0000000000..70d343639b Binary files /dev/null and b/recognition/s4627234_3710project/images/output12.png differ diff --git a/recognition/s4627234_3710project/images/output13.png b/recognition/s4627234_3710project/images/output13.png new file mode 100644 index 0000000000..dbe6f75531 Binary files /dev/null and b/recognition/s4627234_3710project/images/output13.png differ diff --git a/recognition/s4627234_3710project/images/output14.png b/recognition/s4627234_3710project/images/output14.png new file mode 100644 index 0000000000..7b615b4922 Binary files /dev/null and b/recognition/s4627234_3710project/images/output14.png differ diff --git a/recognition/s4627234_3710project/images/output15.png b/recognition/s4627234_3710project/images/output15.png new file mode 100644 index 0000000000..18da2b3c47 Binary files /dev/null and b/recognition/s4627234_3710project/images/output15.png differ diff --git a/recognition/s4627234_3710project/images/output16.png b/recognition/s4627234_3710project/images/output16.png new file mode 100644 index 0000000000..ecc997a1b4 Binary files /dev/null and b/recognition/s4627234_3710project/images/output16.png differ diff --git a/recognition/s4627234_3710project/images/output17.png b/recognition/s4627234_3710project/images/output17.png new file mode 100644 index 0000000000..4aeed7c153 Binary files /dev/null and b/recognition/s4627234_3710project/images/output17.png differ diff --git a/recognition/s4627234_3710project/images/output18.png b/recognition/s4627234_3710project/images/output18.png new file mode 100644 index 0000000000..5d1594d354 Binary files /dev/null and b/recognition/s4627234_3710project/images/output18.png differ diff --git a/recognition/s4627234_3710project/images/output19.png b/recognition/s4627234_3710project/images/output19.png new file mode 100644 index 0000000000..8e5018f926 Binary files /dev/null and b/recognition/s4627234_3710project/images/output19.png differ diff --git a/recognition/s4627234_3710project/images/output2.png b/recognition/s4627234_3710project/images/output2.png new file mode 100644 index 0000000000..7f65a40e87 Binary files /dev/null and b/recognition/s4627234_3710project/images/output2.png differ diff --git a/recognition/s4627234_3710project/images/output20.png b/recognition/s4627234_3710project/images/output20.png new file mode 100644 index 0000000000..5e3b599ac5 Binary files /dev/null and b/recognition/s4627234_3710project/images/output20.png differ diff --git a/recognition/s4627234_3710project/images/output21.png b/recognition/s4627234_3710project/images/output21.png new file mode 100644 index 0000000000..6e564c1dc6 Binary files /dev/null and b/recognition/s4627234_3710project/images/output21.png differ diff --git a/recognition/s4627234_3710project/images/output22.png b/recognition/s4627234_3710project/images/output22.png new file mode 100644 index 0000000000..ac13a80608 Binary files /dev/null and b/recognition/s4627234_3710project/images/output22.png differ diff --git a/recognition/s4627234_3710project/images/output23.png b/recognition/s4627234_3710project/images/output23.png new file mode 100644 index 0000000000..3ad3de929d Binary files /dev/null and b/recognition/s4627234_3710project/images/output23.png differ diff --git a/recognition/s4627234_3710project/images/output24.png b/recognition/s4627234_3710project/images/output24.png new file mode 100644 index 0000000000..230c024b5c Binary files /dev/null and b/recognition/s4627234_3710project/images/output24.png differ diff --git a/recognition/s4627234_3710project/images/output25.png b/recognition/s4627234_3710project/images/output25.png new file mode 100644 index 0000000000..fb301e889e Binary files /dev/null and b/recognition/s4627234_3710project/images/output25.png differ diff --git a/recognition/s4627234_3710project/images/output26.png b/recognition/s4627234_3710project/images/output26.png new file mode 100644 index 0000000000..ff4c0ff489 Binary files /dev/null and b/recognition/s4627234_3710project/images/output26.png differ diff --git a/recognition/s4627234_3710project/images/output27.png b/recognition/s4627234_3710project/images/output27.png new file mode 100644 index 0000000000..3a9160cbc0 Binary files /dev/null and b/recognition/s4627234_3710project/images/output27.png differ diff --git a/recognition/s4627234_3710project/images/output28.png b/recognition/s4627234_3710project/images/output28.png new file mode 100644 index 0000000000..4a155b3949 Binary files /dev/null and b/recognition/s4627234_3710project/images/output28.png differ diff --git a/recognition/s4627234_3710project/images/output29.png b/recognition/s4627234_3710project/images/output29.png new file mode 100644 index 0000000000..6f5322ec2f Binary files /dev/null and b/recognition/s4627234_3710project/images/output29.png differ diff --git a/recognition/s4627234_3710project/images/output3.png b/recognition/s4627234_3710project/images/output3.png new file mode 100644 index 0000000000..0e544031ad Binary files /dev/null and b/recognition/s4627234_3710project/images/output3.png differ diff --git a/recognition/s4627234_3710project/images/output30.png b/recognition/s4627234_3710project/images/output30.png new file mode 100644 index 0000000000..c1b8f5a30f Binary files /dev/null and b/recognition/s4627234_3710project/images/output30.png differ diff --git a/recognition/s4627234_3710project/images/output31.png b/recognition/s4627234_3710project/images/output31.png new file mode 100644 index 0000000000..a299d81d01 Binary files /dev/null and b/recognition/s4627234_3710project/images/output31.png differ diff --git a/recognition/s4627234_3710project/images/output32.png b/recognition/s4627234_3710project/images/output32.png new file mode 100644 index 0000000000..fcf544ced9 Binary files /dev/null and b/recognition/s4627234_3710project/images/output32.png differ diff --git a/recognition/s4627234_3710project/images/output33.png b/recognition/s4627234_3710project/images/output33.png new file mode 100644 index 0000000000..12b5f535cf Binary files /dev/null and b/recognition/s4627234_3710project/images/output33.png differ diff --git a/recognition/s4627234_3710project/images/output34.png b/recognition/s4627234_3710project/images/output34.png new file mode 100644 index 0000000000..0a522aef2a Binary files /dev/null and b/recognition/s4627234_3710project/images/output34.png differ diff --git a/recognition/s4627234_3710project/images/output35.png b/recognition/s4627234_3710project/images/output35.png new file mode 100644 index 0000000000..c09c2f2a3d Binary files /dev/null and b/recognition/s4627234_3710project/images/output35.png differ diff --git a/recognition/s4627234_3710project/images/output36.png b/recognition/s4627234_3710project/images/output36.png new file mode 100644 index 0000000000..ab252f2da4 Binary files /dev/null and b/recognition/s4627234_3710project/images/output36.png differ diff --git a/recognition/s4627234_3710project/images/output37.png b/recognition/s4627234_3710project/images/output37.png new file mode 100644 index 0000000000..e57314c1df Binary files /dev/null and b/recognition/s4627234_3710project/images/output37.png differ diff --git a/recognition/s4627234_3710project/images/output38.png b/recognition/s4627234_3710project/images/output38.png new file mode 100644 index 0000000000..c0e0f4d062 Binary files /dev/null and b/recognition/s4627234_3710project/images/output38.png differ diff --git a/recognition/s4627234_3710project/images/output39.png b/recognition/s4627234_3710project/images/output39.png new file mode 100644 index 0000000000..14c6f6a11e Binary files /dev/null and b/recognition/s4627234_3710project/images/output39.png differ diff --git a/recognition/s4627234_3710project/images/output4.png b/recognition/s4627234_3710project/images/output4.png new file mode 100644 index 0000000000..51832ccea3 Binary files /dev/null and b/recognition/s4627234_3710project/images/output4.png differ diff --git a/recognition/s4627234_3710project/images/output40.png b/recognition/s4627234_3710project/images/output40.png new file mode 100644 index 0000000000..8483231208 Binary files /dev/null and b/recognition/s4627234_3710project/images/output40.png differ diff --git a/recognition/s4627234_3710project/images/output41.png b/recognition/s4627234_3710project/images/output41.png new file mode 100644 index 0000000000..51644907fa Binary files /dev/null and b/recognition/s4627234_3710project/images/output41.png differ diff --git a/recognition/s4627234_3710project/images/output42.png b/recognition/s4627234_3710project/images/output42.png new file mode 100644 index 0000000000..5ffc5a41ea Binary files /dev/null and b/recognition/s4627234_3710project/images/output42.png differ diff --git a/recognition/s4627234_3710project/images/output43.png b/recognition/s4627234_3710project/images/output43.png new file mode 100644 index 0000000000..880eb1e6c9 Binary files /dev/null and b/recognition/s4627234_3710project/images/output43.png differ diff --git a/recognition/s4627234_3710project/images/output44.png b/recognition/s4627234_3710project/images/output44.png new file mode 100644 index 0000000000..ebe9ad921a Binary files /dev/null and b/recognition/s4627234_3710project/images/output44.png differ diff --git a/recognition/s4627234_3710project/images/output45.png b/recognition/s4627234_3710project/images/output45.png new file mode 100644 index 0000000000..4d93f5499c Binary files /dev/null and b/recognition/s4627234_3710project/images/output45.png differ diff --git a/recognition/s4627234_3710project/images/output46.png b/recognition/s4627234_3710project/images/output46.png new file mode 100644 index 0000000000..97b4ab1441 Binary files /dev/null and b/recognition/s4627234_3710project/images/output46.png differ diff --git a/recognition/s4627234_3710project/images/output47.png b/recognition/s4627234_3710project/images/output47.png new file mode 100644 index 0000000000..5008b8f9f0 Binary files /dev/null and b/recognition/s4627234_3710project/images/output47.png differ diff --git a/recognition/s4627234_3710project/images/output48.png b/recognition/s4627234_3710project/images/output48.png new file mode 100644 index 0000000000..1aa2edef03 Binary files /dev/null and b/recognition/s4627234_3710project/images/output48.png differ diff --git a/recognition/s4627234_3710project/images/output49.png b/recognition/s4627234_3710project/images/output49.png new file mode 100644 index 0000000000..b7edb4ea93 Binary files /dev/null and b/recognition/s4627234_3710project/images/output49.png differ diff --git a/recognition/s4627234_3710project/images/output5.png b/recognition/s4627234_3710project/images/output5.png new file mode 100644 index 0000000000..20791ba088 Binary files /dev/null and b/recognition/s4627234_3710project/images/output5.png differ diff --git a/recognition/s4627234_3710project/images/output6.png b/recognition/s4627234_3710project/images/output6.png new file mode 100644 index 0000000000..5f009bbc4a Binary files /dev/null and b/recognition/s4627234_3710project/images/output6.png differ diff --git a/recognition/s4627234_3710project/images/output7.png b/recognition/s4627234_3710project/images/output7.png new file mode 100644 index 0000000000..377bae54ce Binary files /dev/null and b/recognition/s4627234_3710project/images/output7.png differ diff --git a/recognition/s4627234_3710project/images/output8.png b/recognition/s4627234_3710project/images/output8.png new file mode 100644 index 0000000000..ca35009d6d Binary files /dev/null and b/recognition/s4627234_3710project/images/output8.png differ diff --git a/recognition/s4627234_3710project/images/output9.png b/recognition/s4627234_3710project/images/output9.png new file mode 100644 index 0000000000..ed56db0087 Binary files /dev/null and b/recognition/s4627234_3710project/images/output9.png differ diff --git a/recognition/s4627234_3710project/modules.py b/recognition/s4627234_3710project/modules.py new file mode 100644 index 0000000000..55ec65c035 --- /dev/null +++ b/recognition/s4627234_3710project/modules.py @@ -0,0 +1,170 @@ + +# Some machine may need the following import statement to import (with .python.) +# May because of some version issue, however in most of the machine, (with .python.) will have a wrong result +# from tensorflow.python.keras.layers import Conv2D, MaxPool2D, Dense, Concatenate, UpSampling2D, Input +# from tensorflow.python.keras.models import Model + +import tensorflow as tf +from tensorflow.keras.layers import Conv2D, MaxPool2D, Dense, Concatenate +from tensorflow.keras.layers import UpSampling2D, Input, LeakyReLU, Add, Dropout +from tensorflow.keras.models import Model + +import tensorflow_addons as tfa +from keras import backend as k + + +def DSC (y_true, y_pred): + """Dice similarity coefficient function, this should achieve 0.8 on the test set when testing + + Args: + y_true (numpy.ndarray): true mask of the data + y_pred (numpy.ndarray): predict mask of the data + + Returns: + int: return dice similarity coefficient value between two image + """ + y_true_f = k.flatten(y_true) + y_pred_f = k.flatten(y_pred) + + intersection1 = k.sum(y_true_f*y_pred_f) + coeff = (2.0 * intersection1) / (k.sum(k.square(y_true_f)) + k.sum(k.square(y_pred_f))) + return coeff + +def DSC_loss (y_true, y_pred): + """ Loss function of dice similarity coefficient, + in other word this is for the not matching part + + Args: + y_true (numpy.ndarray): true mask of the data + y_pred (numpy.ndarray): predict mask of the data + + Returns: + int: return the loss value of dice similarity coefficient, + which is (1 - dice similarity coefficient) + """ + return 1 - DSC(y_true, y_pred) + + +#-- normal UNet --# + +def down(x, filters): + conv = Conv2D(filters, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(x) + conv = Conv2D(filters, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(conv) + conv = Concatenate()([x, conv]) + out = MaxPool2D((2, 2), (2, 2))(conv) + return conv, out + +def up(x, skip, filters): + us = UpSampling2D((2, 2))(x) + concat = Concatenate()([us, skip]) + conv = Conv2D(filters, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(concat) + out = Conv2D(filters, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(conv) + return out + +def UNet(): + f = 16 + inputs=Input((256,256,3)) + p0 = inputs + c1, p1 = down(p0, f) + c2, p2 = down(p1, f*2) + c3, p3 = down(p2, f*4) + c4, p4 = down(p3, f*8) + + conv = Conv2D(f*16, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(p4) + conv = Conv2D(f*16, kernel_size=(3, 3), padding="same", strides=1, activation="relu")(conv) + + u1 = up(conv, c4, f*8) + u2 = up(u1, c3, f*4) + u3 = up(u2, c2, f*2) + u4 = up(u3, c1, f) + + outputs = Conv2D(3, (1, 1), padding="same", activation="sigmoid")(u4) + model = Model(inputs, outputs) + return model + + +#-- improved UNet --# + +act = LeakyReLU(alpha = 0.01) + +def down_context_module(input, filters): + """ + A context module consisting of two 3x3 convolutions + with a dropout of 0.3 between them + """ + conv1 = tfa.layers.InstanceNormalization()(input) + conv1 = Conv2D(filters, (3, 3), padding = "same", activation = act)(conv1) + dropout = Dropout(0.3) (conv1) + conv2 = tfa.layers.InstanceNormalization()(dropout) + conv2 = Conv2D(filters, (3, 3), padding = "same", activation = act)(conv2) + return conv2 + +def down_imp(input, filters, stride): + """ + encode module with 3x3 convolution + """ + conv = Conv2D(filters, (3, 3), strides = stride, padding = "same")(input) + conv_module = down_context_module(conv, filters) + add = Add()([conv, conv_module]) + return add + + +def localization_module(input, filter1, filter2): + """ + A localization module consists of a 3x3 convolution + and a 1x1 convolution that halves the number of features + """ + conv1 = Conv2D(filter1, (3, 3), padding = "same", activation = act)(input) + conv2 = Conv2D(filter2, (1, 1), padding = "same", activation = act)(conv1) + return conv2 + +def up_imp(input, adds, filter1, filter2): + """ + decode module with 3x3 convolution + """ + # upsampling module + up = UpSampling2D((2, 2))(input) + up = Conv2D(filter1, (3, 3), padding = "same", activation = act)(up) + concat = Concatenate()([up, adds]) + + # localization module + local = localization_module(concat, filter2, filter2) + return local + +def UNet_imp(): + + inputs = Input(shape=(256, 256, 3)) + + # Encoder (down) + add1 = down_imp(inputs, 32, (1, 1)) # only first one with strides = (1, 1) + add2 = down_imp(add1, 32, (2, 2)) + add3 = down_imp(add2, 64, (2, 2)) + add4 = down_imp(add3, 128, (2, 2)) + add5 = down_imp(add4, 256, (2, 2)) + + + # Decoder (up) + local1 = up_imp(add5, add4, 128, 128) + + local2 = up_imp(local1, add3, 64, 64) + seg1 = Conv2D(3, (1, 1), padding = "same")(local2) + seg1 = UpSampling2D(size = (2, 2))(seg1) + + local3 = up_imp(local2, add2, 32, 32) + seg2 = Conv2D(3, (1, 1), padding = "same")(local3) + seg_add_1 = Add()([seg1, seg2]) + seg_add_1 = UpSampling2D(size = (2, 2))(seg_add_1) + + up = UpSampling2D((2, 2))(local3) # up-sampling + up = Conv2D(16, (3, 3), padding = "same", activation = act)(up) + + concat4 = Concatenate()([up, add1]) + conv4 = Conv2D(32, (3, 3), padding = "same")(concat4) + seg3 = Conv2D(3, (1, 1), padding = "same")(conv4) + seg_add_2 = Add()([seg_add_1, seg3]) + + outputs = Conv2D(3, (1, 1), activation = "sigmoid")(seg_add_2) + model = tf.keras.Model(inputs = inputs, outputs = outputs) + + return model + diff --git a/recognition/s4627234_3710project/predict.py b/recognition/s4627234_3710project/predict.py new file mode 100644 index 0000000000..5c1c884f5a --- /dev/null +++ b/recognition/s4627234_3710project/predict.py @@ -0,0 +1,45 @@ + +import os +import matplotlib.pyplot as plt +import tensorflow as tf +import train +import dataset +import modules + + +def main(): + # load data and model + val_x, val_y = dataset.load_val() + model = tf.keras.models.load_model('imp_unet_model.h5', + custom_objects={'DSC': modules.DSC, 'DSC_loss': modules.DSC_loss}) + + pred = model.predict(val_x) + gt = tf.convert_to_tensor(val_y, dtype=tf.float32) + print(modules.DSC(gt, pred)) + + number_list = range(50) # image number (any) + for each in number_list: + fig = plt.figure() + + fig.subplots_adjust(hspace = 0.4, wspace = 0.4) + ax = fig.add_subplot(1, 3, 1) + ax.imshow(val_x[each]) + ax.title.set_text("Testing Image") + ax.axis('off') + + result=pred[each]>0.5 + ax = fig.add_subplot(1, 3, 2) + ax.imshow(result*255, cmap="gray") + ax.title.set_text("Predict Image") + ax.axis('off') + + ax = fig.add_subplot(1, 3, 3) + ax.imshow(val_y[each]) + ax.title.set_text("Ground Truth") + ax.axis('off') + + plt.savefig('./images/output'+ str(each) +'.png') + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/recognition/s4627234_3710project/train.py b/recognition/s4627234_3710project/train.py new file mode 100644 index 0000000000..32306c2652 --- /dev/null +++ b/recognition/s4627234_3710project/train.py @@ -0,0 +1,64 @@ +import dataset +import modules +import tensorflow as tf +import matplotlib.pyplot as plt +from tensorflow.keras.optimizers import Adam + + +def training(data_reshape = False): + """ training data with model in modules.py with data from dataset.py + + Args: + data_reshape (bool, optional): true if wants to reshape data. Defaults to False. + + Returns: + model, history: return the model and trained history of data + """ + train_x, train_y, test_x, test_y = dataset.load_dataset(data_reshape) + + model = modules.UNet_imp() + + # fit the model with normal learning rate + model.compile(optimizer = Adam(0.0005), loss = modules.DSC_loss, metrics=['accuracy', modules.DSC]) + + history = model.fit(train_x, train_y, validation_data= (test_x, test_y), + batch_size=8,shuffle='True',epochs=50) + + return model, history + + +def plot_data(history, type): + """ plot DSC or DSC loss data and save into images + + Args: + history (keras.callbacks.History): history of training model + type (string): decide acc or loss + """ + plt.figure(figsize = (10, 5)) + + if type == 'acc': add = 'DSC' + else: add = 'loss' + + plt.plot(history.history['' + add], label='Training ' + add) + plt.plot(history.history['val_' + add], label='Validation ' + add) + plt.title('Test vs Validation ' + add) + plt.legend(loc='lower right') + plt.xlabel('Epochs') + plt.ylabel('' + add) + plt.savefig('./images/'+ add +'.png') + plt.show() + + +def main(): + """ + Training and save the model + """ + + model, history = training(data_reshape = False) + model.save("imp_unet_model.h5") + + plot_data(history, 'acc') + plot_data(history, 'loss') + +if __name__ == "__main__": + main() \ No newline at end of file