What is GSoC?
Google Summer of Code is designed to encourage student participation in open source development. Over the past 13 years, Google Summer of Code has brought together over 13,000 students and 608 open source projects to create over 33 millions of lines of code. Google will accept applications from student applications from March 12 to 27, 2018.
vitrivr at GSoC
In GSoC 2018, we want you to work with all components of vitrivr. Cineast and ADAMpro work jointly to provide the user with a sketch-based multimedia retrieval engine which can be interacted with via Vitrivr-NG. Watch the following video to see a demo of our running system. To see how you can set up the vitrivr stack on your system, visit our getting started page.
This is vitrivr’s second participation to Google Summer of Code. Visit our GSoC 2016 page to learn more about our participation to GSoC 2016.
Student Application
To apply for one of our projects, we ask you for the following information:
| Problem Statement State very briefly the problem you are trying to solve, the goal of the project and the expected outcome. Be specific and realistic about the outcome and about what you want to achieve in three months. |
| Methodology State the steps you think are required for solving the problem. Be specific, choose small doable parts that you can oversee and of which you can give good time estimate. |
| Project Plan We want to see a detailed weekly project plan with well-defined milestones and deliverables (and potential breaks or absences). |
In addition, we ask you to submit code to one of the following pre-tasks. For each of the tasks you are asked to write a piece of source code in your language of choice, in good style, no longer than 150 lines. Upload your code snippet anywhere and send the link to it to with subject “GSoC 2018 - Task X” (where X denotes the task you have solved). In case you have not yet submitted your application via the Google Summer of Code website, please provide a draft of your application (via link or attachment) in the email together with the solution of your chosen task.
It is important that you know well about vitrivr and about potential things to improve. Get in contact with the mentors to pick a good project that fits well to vitrivr and to you.
Ideas
Below, you can find some of our ideas on the directions in which we could push vitrivr together. Please consider them as starting points for your proposal. Of course, if you have other ideas, we would be very happy to hear them. Be creative and send us proposals that extend the ideas below – or that are advancing the vitrivr multimedia retrieval stack in another way. Feel free to contact us and get feedback on what you plan to do beforehand.
| Task | Software | Details | Technologies | Principal Mentor | Difficulty |
|---|---|---|---|---|---|
| vitrivr App | Cineast | vitrivr comes with a web UI which has the advantage that it can be accessed by a large variety of devices. However, as a disadvantage, the UI cannot make use of any possibly further available native functionality, such as touch input, a camera, GPS sensors for linking photos or videos to the real world, etc. Furthermore, it is not optimized for small screens. In this work, the goal is to implement a native Android-based vitrivr app that can be used on an Android phone or tablet. The app could also make use of the spatial retrieval capabilities of vitrivr to limit results to only those which are somehow relevant to your current location. This project can be done in collaboration with the Red Hen project. | Java, Kotlin, etc. | TBD | medium |
| [email protected] | vitrivr | While vitrivr is currently capable of processing the vast majority of media files commonly found in home media collections, its usability in these situations could still be vastly better. The aim of this project would be to make it easier to deploy vitrivr at home and use it to manage a home media collection. In order to do this, one could for example increase the capabilities of the UI in order to allow collection management or build plugins for other media management tools such as Plex to enable them to leverage the search capabilities of vitrivr. | Java and others | TBD | medium |
| bring cthulhu back | cthulhu | The distributed extraction management tool cthulhu which was created during Google Summer of Code 2016 sadly did not keep pace with the development of the other components of vitrivr and is therefore currently incompatible with the latest version of vitrivr. The aim of this project would be to bring cthulhu back to a point where it can communicate with vitrivr and add new functionality to it. | Java | TBD | medium |
| vitrivr and iiif | vitrivr | The international image interoperability framework describes a standard to interact with images which is being adopted by an increasing number of museums, archives and other institutions which store large collections of digital images. The aim of this project would be to make vitrivr compatible with the apis specified by iiif so it can be used to interact with any iiif capable data sources easily. | Java, web | TBD | hard |
| vitrivr and TensorFlow | Cineast | The staggering amount of progress in the field of deep learning bares many fruits, some of which could be beneficial for vitrivr as well. While there is a way to use TensorFlow in vitrivr, it uses an unofficial and outdated interface which makes it difficult to employ cutting edge models. The aim of this project would be to replace the current TensorFlow interface with the official one for Java and build the necessary structures for it to be able to use current TF-models. | Java, TensorFlow | TBD | medium |
| vitrivr in the cloud | vitrivr | While vitrivr can be and has been deployed in a cloud environment, it involves manual changes every time the scaling is to be changed. The aim of this project would be to make vitrivr cloud-aware by adding for example dynamic load balancing capabilities which would facilitate scale-out. Adding the possibility to feed vitrivrs vitals into for example a Grafana dashboard would also go in this direction. | Java | TBD | hard |
| Subtitle support | vitrivr | vitrivr currently supports a few clear-text formats of subtitles but there are many which are not supported yet. The aim of this project would be to add support for various subtitle formats to Cineast and Vitrivr-NG and maybe even build an interface to CCExtractor. This project will explore synergies from combining CCExtractor and vitrivr and will thus be done in collaboration of both organizations. | Java, web, etc. | TBD | medium |
| vitrivr and structured metadata | vitrivr | Add support for structured metadata on objects and the search within these structured metadata on the UI. | Java, web | TBD | easy |
| Art from Data | vitrivr | vitrivr does have some capabilities for feature visualization which are however currently not accessible via the APIs. The aim of this project would be to first re-enable the access to these visualization capabilities and subsequently expand among them. This project might also apply the visualization features to the large newscape video collection, in a collaboration between vitrivr and Red Hen. | Java, web, etc. | TBD | medium |
| vitrivr the explorer | vitrivr | While vitrivr works great if you know what you are looking for, its capacity for exploration is somewhat limited. The aim of this project would be to expand the browsing and exploration capabilities of vitrivr, for instance by offering visualizations leveraging semantic topologies. This can be done in collaboration with the Red Hen organization, on the basis of the newscape video collections and the semantic annotations. | Java, web, etc. | TBD | medium |
| Enhanced 3D retrieval | Cineast | vitrivr has the capabilities to search for 3d models based on 3d sketches or 3d examples. It currently lacks the capability to search 3d objects which can be seen in pictures or videos. It’s currently also not possible to search for videos or images in which a certain 3d model might be visible. The aim of this project would be to explore ways in which this gap between 3d and visual objects can be bridged. | Java, etc. | TBD | hard |
| Speech and Text from video | vitrivr | It has been proven useful when searching for video segments to be able to search using dialog or text which is shown on screen. The aim of this project would be to integrate state of the art OCR and Speech-to-Text methods into vitrivr in order to support such queries. | Java, etc. | TBD | medium |
| vitrivr the organizer | vitrivr | Especially in the context of video, vitrivr assumes that every video file contains only one piece of content. But what if this is not the case? In instances where a video contains several pieces of content, for example in the case of continuous television recordings, uncut film footage, etc., vitrivr is not currently able to reorganize this content into a more semantically meaningful form. The goal of this project would be to add such segmentation and organization capabilities – both manual and automatic – to vitrivr. This can be applied on a real world dataset for the detection of show boundaries in TV recordings using the newscape collection of Red Hen in a collaboration of both organizations. | Java, etc. | TBD | medium |
Mentors
This list has the contact information of some of our mentors. It will be updated over time.
| Mentor | Profile | Contact | Languages |
|---|---|---|---|
| Heiko Schuldt | http://dbis.dmi.unibas.ch/team/heiko-schuldt | English, German | |
| Ivan Giangreco | http://dbis.dmi.unibas.ch/team/ivan-giangreco |
|
English, German |
| Luca Rossetto | http://dbis.dmi.unibas.ch/team/luca-rossetto | English, German | |
| Ralph Gasser | http://dbis.dmi.unibas.ch/team/ralph-gasser | English, German | |
| Silvan Heller | http://dbis.dmi.unibas.ch/team/silvan-heller | English, German | |
| Mark J. Williams | http://film-media.dartmouth.edu/people/mark-j-williams | English | |
| Francis Steen | http://commstudies.ucla.edu/content/francis-steen-phd | English | |
| Carlos Fernandez | English |
All mentors are available on Google Hangout and skype.
Contact
Please feel free to contact us if you have any questions. To make it easier for us to organize our emails, please add “GSoC 2018” in your subject line. You can contact us via our email address or you can contact the mentors directly.
Mailing List
We have set up a mailing list for the vitrivr project. You can use the mailing list for questions regarding the code, bugs, etc. Note that the mailing list is publicly visible.