Unable to reproduce city2foggy results from paper

[FlorentF9]

After running your code for the city2foggy task, I am unable to reproduce the results from your paper (by a large margin).

For the source-only training, by running exactly the command indicated in this repo, I reach only 20 mAP (to be precise, 20.67 after 10 epochs), whereas the paper reports 24 mAP.

Then, I modified the code to integrate the style enhancement (as I mention in this issue, it is not used in the code). However, using the pre-trained weights provided for the style enhancement module, it only improves to around 26 mAP, very far from the 35.8 mAP in the paper.

I wonder if you could provide some help to reproduce the results, maybe I missed something ? Are the city2foggy results in the paper reported with the vgg16 backbone, or is it R101 ?

[Flashkong]

Thank you for your interest in our work!

I have uploaded our trained models, test data and test codes in the Trained_models directory, please check them out. For the inability to reproduce the experimental results, I give two suggestions in Trained_models/README.md, please refer to it.

The source only model was trained by me a long time ago, and I can’t remember the specific details of the training at that time (it may be slightly different from the code provided). But one thing can be guaranteed is that the source only model is completely based on Faster RCNN, the model has not undergone any modification. Since there are labels for supervision, you can try to train for a longer time, or adjust the learning rate to achieve better accuracy of the source only model.

[FlorentF9]

Thanks a lot for providing all this information, it's very helpful! The reason might be python 3, because the rest of our environment seems identical to yours. We will investigate!

[Flashkong]

I think you misunderstood. What I mean is that when testing the trained models we provide, you need to use the corresponding python version, otherwise the accuracies of the trained models may be inconsistent with what we reported in the paper (because we use python 2.7.12 to train the trained models).

We have not tried running the code we provided with a version other than python2.7.12. Different versions may also produce similar results. I guess your problem may be caused by too long training time. Good luck!