The project is funded under the PTE technology transfer program of the Romanian Authority for Scientific Research, via UEFISCDI (project number PN-III-P2-2.1-PED2021-3120, contract number 679PED from ) with title “VinEye: cartografierea colaborativă a viilor cu roboți autonomi.“. It takes place at the Automation Department of the Technical University of Cluj-Napoca, Romania, for a duration of 2 years and a total budget of 100000 EUR.
Contact:
- UTCN (CO): Levente.Tamas@aut.utcluj.ro, tel: 0264401586, Str. Memorandumului 28, Cluj-Napoca, Ro.
- UBB (P1): Sergiu Daraban, email: sergiu.darabant@ubbcluj.ro
- USAMV(P2): Attila Sandor, email: adsandor@gmail.com
Abstract
The major challenges in the vineyards are related to the diseases which affect the grapevine in different phases of its maturity. The detection of the malformations in the grape usually is done visually. Unless these diseases are detected in time, they can produce massive damage to up to 70-80 per cent of the entire wine production in a year. Thus, early detection of the malformations in the plant is crucial for the entire yearly wine production.
Some early-stage research works are focusing on aerial observation of the vineyard, however, they lack the close proximity observation which is usually available from ground-level robots. This project proposal aims to demonstrate the potential of fusing aerial and ground-level images in a common topological map for vineyard state evaluation. The fusion of different images captured from different view-point and scales is non-trivial, however, our in-house camera homography estimation algorithms can cope with this as suggested in our previous publications.
Adopting our camera image fusion solutions for the wine yard mapping would solve the early-stage disease detection offering real-time information on large-scale productions. With the support from the local agro R&D research centre and with the involvement of our latest technology autonomous robot fleet, this project would demonstrate the real benefits of the paradigm shift towards the use of robots in the era of Agriculture 4.0.
The impact of the VinEye project is multifold: environmental (optimization of viticultural activities), social (makes everyday labour easier), economic (reduces losses) and scientific (omnidirectional heterogeneous image fusion and detection). This will be ensured by the interdisciplinary project team formed by experts in the machine learning and robotics fields with support from a local R&D vineyard centre in validation and consultancy from the regional DIH hub in the agronomical robotics field.
Objectives
O1 – Georeferenced visual mapping of the vineyards with static and moving embedded cameras.
Building a georeferenced map using cameras mounted on fieldwork-performing tractors
O2 – Multi-scale change detection for early-stage disease detection. Detection of possible zones with diseases from the images, and active monitoring using the fused images in the georeferenced map.
O3 – Validation of the proposed setup in the real case study. Validating the proposed experimental setup in the vineyard and evaluating of the performance of the proposed methods on
Planned project activities
Phase 1: Fused map creation designs, develops, and tests the fusion techniques required to achieve O1
Phase 2: Disease detection designs, develops, and tests the VDD techniques required to achieve O2.
Phase 3: Case study requirement specifications, testing and validation in order to complete O3.
Outcomes of the planned activities
- (additional deliverable, first month): project website, updated periodically
- (m12): report + submitted publication on VDD.
- (m15): report + submitted publication on transfer learning for object detection.
- (m21): report + submitted publication on multi-view constrained UAV image mapping
- (m24): final report + case study also on ecological impact and corresponding publication.
Outcomes
Phase 1: Fused map creation designs, develops, and tests the fusion techniques required to achieve O1
Canopy segmentation:
Leaf detection (for VDD):
State-of-the-art survey papers (with pdfs):
Phase 2: Disease detection designs, develops, and tests the VDD techniques required to achieve O2.
This phase had as its main objective multi-scale change detection for early-stage disease detection. Detection of possible zones with diseases from the images, and active monitoring using the fused images in the georeferenced map.
VDD segmentation results are shown in an animation. The results of this research were published at IFAC2023 (see publication list below).
Segmentation
Demo video with segmentation using embedded devices. The results of this research were published at FEDCIS2023 conference and ESWA2023 journal.
Phase 3: Case study
Leaf orientation analysis (WIP)
Media coverage of the project:
Publications: