The visual system provides us an enormous amount of information. For processing it in real time to be able to survive, humans (and other animals) have developed an attention system that allows them to filter out non important portions of the scene by just focusing on the most salient parts of what is being observed.
I wrote a paper about using saliency as a new feature for object detection obtaining good results. It was accepted for publication in the Image and Vision Computing (IMAVIS) journal. You can download the manuscript from here. For calculating the new proposed features, I wrote a fine-grained saliency library. It requires OpenCV to run properly. You can download it from here. You can also download the source code which is released under a BSD license.
I updated the code to be used with the modern C++ OpenCV interface and CMake. You can get this code here.
UPDATE:This saliency method is now included in OpenCV contrib module. You can learn how to use it here.
Hello,
I’ve translated your code from C to Python:
http://www.arash-shafiei.com/?p=99
Thanks for providing a great resource :)
PS, You should URL-encode filenames uploaded on the online FGSG or else the results cannot be viewed for certain filenames.
Thank you very much for your code and paper.
I have learnt much form your research work.
Interesting… I read your paper… I also used a Frintrop-related bottom-up saliency in my master thesis… http://igitur-archive.library.uu.nl/student-theses/2010-0623-200147/thesis.pdf
I have a question though, doesn’t your described method calculate surrounds for fixed centers? Isn’t this different from calculating surrounds for different centers… (moving to a different scale should give a different center(x,y) and not the same i(x,y) value). Right?
Great question Medhat Riad.
I designed my system to be as similar as possible to the ideas behind Frintrop’s VOCUS, but with an increased resolution in the output.
Because of that, I decided to keep the original idea that the center will always be I(x, y), which is the minimum possible value that the surround can be compared to. If you enlarge the center area, you may miss fine grained details that are lost in the average of that area.
Having said that, these differences of sizes probably will not affect the final result in a significant manner. On a later paper, Frintrop also explored the idea of using circular areas instead of squared ones, but the results were very similar.
I hope I could help you.
How can I contact Arash Shafiei ?
I do not know PengPai, I do not have any contact information of him.
hi, the code does not seem to be available. could you fix the link please? I’m interested in testing your approach
Hi Matias,
You will be able to download it now.
Great, thank you!
Have you tried using more feature channels? such as color opponency, contrast and orientation? Is your approach capable of adding these channels?
Nice work!, What’s the licence of the code? Could you update the sources with the licence?
Thanks Peter,
It is released under a BSD license, and I just updated the sources to reflect this.
Hi Matias,
For the purpose of the publication only the intensity channel was used, but it is totally possible to use the same idea to calculate other channels. It will depend on the specific feature that you want, but things like color for example should be really simple to integrate using this method. Other features like orientation may need an alternative representation or an approximation to make use of this technique.
I updated the code to use the C++ interface (cv::Mat instead of IPLImage, etc…), and sent it to opencv contrib github. You can get it here:
https://github.com/Itseez/opencv_contrib/pull/122
which cpp file(s) to download from there? sorry i couldn’t find the files in that link
Well, it is a pull request for the official opencv_contrib repository, so no direct zip.
You can go to my github project and download the zip there though (click on Download Zip):
https://github.com/samontab/opencv_contrib
That will contain the entire opencv_contrib repository with my changes added(fine grained saliency). The files you are after are inside the folder modules/saliency. In particular, take a look at staticSaliencyFineGrained.cpp.
Hopefully in the future it will be included in OpenCV.
Thank you for the rapid reply. Do I have to build whole opencv to use the module? Isn’t there a simple way to just to include the files and get it running? Because in the website it tells me to rebuild the opencv library with the modules and etc, I do not want to rebuild the opencv since this isn’t my own computer, therefore I would like to avoid any possible damage that might arise from rebuilding opencv.
so in short, is it okay if i just build a project and include the stuff which are only in modules/saliency?
Hi remake,
Nah, you only need the staticSaliencyFineGrained.cpp file actually. The easiest would be to just download the version from my website to get it compiling with the old interface, and then just replace the code with the new C++ interface from that file.
Hi Mr. Montabone,
I am able to compile and run the old interface one, no problem with that. It works perfectly.
However, when I try the new interface one, I face problems since I do not have OpenCV 3.0 (and I guess some headers require that). I tried to put the new .cpp file ( staticSaliencyFineGrained.cpp) into the old interface one’s project but it didn’t work. I think it requires a header called precomp, and that header requires some other header and it goes like that, at the end I have all the files in that new module/saliency folder, but I do not know how to get this running. I tried to do it with cmake and all, but no luck. Here is the folder structure that I have:
-src
–main.cpp
–motionSaliency.cpp
–motionSaliencyBinWangApr2014.cpp
–objectness.cpp
–precomp.hpp
–saliency.cpp
–saliency_init.cpp
–staticSaliency.cpp
–staticSaliencyFineGrained.cpp
–staticSaliencySpectralResidual.cpp
-include
–opencv
—saliency
—-saliencyBaseClasses.hpp
—-saliencySpecializedClasses.hpp
–saliency.hpp
and optionally :
-CMakeLists.txt
This above setup does not even compile due to Opencv 3.0 issue, some header there requires 3.0 and I don’t have that. So I deleted that file (i guess it was objectness) since i will not use it. However then some other errors occurred, somehow the functions of the fine grain saliency class are not recognized, it treats them like they weren’t predefined.
I even tried deleting all irrelevant files like motion saliency and others, and just keeping fine grain one, but again, didn’t work.
I guess i will try to convert this old interface one into the new one in the project that already runs. Sweat work, but better than compilation errors.
Yeah, it is written for OpenCV 3.0, but it should be easy to solve the problems, there are minor changes in the syntax between 2.x and 3.x.
You could just change the method (functions) from the working old interface using the new one. If you just copy and paste the entire file it will not work, as it is written as a module for opencv.
I got what you mean now, and yes that is the easiest way. I did so and it worked, thanks a lot :)
Great to hear that it worked for you, remake
Hi samontab!
I found an example in the opencv_contrib, and I build it. But I still don’t know how to use that software even with your little tips. Could you explain it to me, thank you. Maybe you can add an example usage to that file.
I mean what’s the type of saliencyAlgorithm, how to input it correctly.
Hi wux.
The saliency type is FINE_GRAINED. You need to download the opencv_contrib code from my github account though, which is here:
https://github.com/samontab/opencv_contrib
Hi wux,
There is already an example for saliency algorithms in general. You just need to specify the saliency type you want to use.
Hi samontab,
I build your computeSaliency example and named it Saliency, so I call
./Saliency
and it says
This example shows the functionality of “Saliency “Call:
./example_saliency_computeSaliency
so I input the subtype FINE_GRAINED like this:
./Saliency FINE_GRAINED test.avi 3
and it says
***Error in the instantiation of the saliency algorithm…***
I don’t konw why
missing comments:
This example shows the functionality of “Saliency “Call:
./example_saliency_computeSaliency saliencyAlgorithmSubType video_name start_frame
Hi wux,
Probably you are using the latest opencv_contrib code, and not the fork on my github account.
This means that the FINE_GRAINED is not defined there
Make sure that you are using my code and not the latest code from opencv.
Hi wux,
I updated my github project, and tested it with OpenCV 3.0.0. It is working fine.
If you still have issues, I also re packaged it into a stand alone application including a CMake project file to make it really easy to use.
You can get the C++ interface of the fine-grained saliency library with an example usage here:
http://samontab.com/web/files/saliencyFineGrained.zip
Hi samontab,
I downloaded opencv_contrib from your github, but I can’t build it with my opencv code(also downloaded from itseez), really sad. So I downloaded opencv-3.0.0, but problems still exist.
Anyway, thank you.
I think I’ll back to the beginning, see whether these saliency api works or not.
didn’t see your latest comments, I’ll test it. thank you!!! :)
Hi, samontab!
Thank you for your update, now I can look into those saliency API.
You are welcome wux.
Sebastian: I am very interested in your masters work and the examples provided via video showing the algorithms in action. Please contact me on the email provided here to start a conversation. Thank you.