Avatud taotlusvooru "Nutikad lahendused noorsootöös" hindamiskomisjoni otsused

Martin Karro Riigi Tugiteenuste Keskus [email protected] Meie 10.08.2021 nr 2-6.3/2498-1 Avatud taotlusvooru "Nutikad lahendused noorsootöös" hindamiskomisjoni otsused Lugupeetud Martin Karro Palume Riigi Tugiteenuste Keskusel kui Euroopa Majanduspiirkonna ja Norra toetuste 2014– 2021 programmi „Kohalik areng ja vaesuse vähendamine“ rakendusüksusel langetada toetuse rahuldamise otsused avatud taotlusvooru „Nutikad lahendused noorsootöös“ raames laekunud projektitaotluste suhtes, mille on kinnitanud 3. augustil 2021. aastal kohtunud taotlusvooru hindamiskomisjon. Palun rakendusüksusel edastada Haridus- ja Noorteametile hindamiskomisjoni seatud tingimused ja kommentaarid, millega taotleja peab enne toetuse rahuldamise otsust täpsustama oma projektikirjeldust ja eelarvet (lisatud). Lugupidamisega (allkirjastatud digitaalselt) Kalle Killar e-teenuste arengu ja innovatsiooni asekantsler nutika arengu toetamise osakonna juhataja ülesannetes Lisa: Minutes of the SC meeting (August 3 2021)_FINAL Ülar Vaadumäe 626 9249 [email protected] Suur-Ameerika 1 / 10122 Tallinn / 626 9301 / [email protected] / www.sm.ee / registrikood 70001952 EEA and Norway Grants 2014-2021 Programme “Local Development and Poverty Reduction” Selection Committee meeting Date and time: August 3, 2021 from 10:00 to 12:20 (Estonian time) Venue: Cisco Meet web platform & MoSA, 7th floor, room Siil (Suur-Ameerika 1, Tallinn)  Open call for proposals „Smart Youth Work“ Selection Committee voting members:  Ms Katrin Olt – chief expert of the Department of Youth and Talent Policy, Ministry of Education and Research;  Ms Britt Järvet – chief expert of the Department of Primary and Basic Education, Ministry of Education and Research;  Mr Märt Aro – education innovation expert, director of Nordic EdTech Forum. . Observers:  Ms Gudrun Gudmundsdottir – Financial Mechanism Office, Senior Sector Officer, Health, Children and Youth, also representing the Financial Mechanism Committee;  Ms Joanna Kurvits – chief expert of the Department of Youth and Talent Policy, Ministry of Education and Research;  Ms Margit Aus – representative of the National Focal Point; State Shared Service Centre, substituting for Ms Laura Pikkoja;  Ms Kelly Poopuu – representative of the PO Implementing Agency, State Shared Service Centre;  Mr Ülar Vaadumäe – Ministry of Social Affairs, Smart Development Department, chief specialist, PO. Absent:  Ms Piret Marvet – representative of the donors, Royal Norwegian Embassy in Tallinn. Minutes of the meeting Mr Ülar Vaadumäe gave an overview of the goals, targets and criteria of the call for proposals. Also, Mr Vaadumäe introduced the submitted projects and explained the tasks of the Selection Committee. For more, please see Annex 1 (PO presentation). Altogether 2 project applications were received within the deadline (April 15, 2021 at 17:00 o’clock local time) set for the call. As a result, the total grant amount applied for is 2,6 MEUR, exceeding the available grant amount by approx. 871 000 euros. Programme Operator Implementing Agency (State Shared Service Centre) has carried out the assessment of the project applications to the eligibility and administrative criteria set in the call text. The projects met the administrative and eligibility criteria and were forwarded for evaluation by two impartial, independent and reliable experts. These experts were:  Evaluator 1: Ms Maarja Olesk, PhD, analyst at Institute of Public Studies;  Evaluator 2: Ms Heleri Reinsalu, adviser to the Strategy Unit of the Government Office.  There were no project applications with points difference over 30% between the scores given by experts, therefore a third evaluation was not necessary (point 20.5)  There were no project proposals receiving less than 50% of the total score, and therefore would have automatically received a negative decision (point 20.6). Please see also Annex 2 (Ranking list of experts). Discussion Mr Vaadumäe opened with a remark that the impartial evaluators have concluded the ranking list as follows: 1. Education and Youth Board project application „Digital Leap in the Field of Youth“, in partnership with University of Tartu, Tallinn University, the Association of Estonian Open Youth Centres and the Estonian Association of Youth Workers = average score of 78,0 points and 2. CTF-Tech Ltd project application “Cyber Training Programme for Youth and Youth Workers“, in partnership with the University of Tartu and CybExer Technologies Ltd = average score of 69,0 points. Mr Ülar Vaadumäe summarised the evaluators’ comments – for both projects the profile and the capacity of the project applicants (incl. partners) is strong and reliable to carry out the proposed activities. Still, both applications lack in ambition to reach target groups and engage them, especially local governments, as well as in methods of ensuring quality engagement with young people with special educational needs, young people with a migrant background and young people in the risk of exclusion (vulnerable and marginalised young people). CTF-Tech Ltd has focused on a rather narrow scope (raising cyber awareness among young people and youth workers) and lacks in outreach, so it cannot be hoped their intervention would bring about a wider improvement for the target group and their network. Education and Youth Board has – on the contrary – taken up quite a wide and ambitious scope with different digital services envisaged to be developed (e-participation platform, non-formal learning outcome log, e-youth centre, e-games, etc.). Mr Märt Aro explained the main principles for his opinion on good successful projects. It entails two main aspects: a) the scalability of the project as well as b) the sustainability of project results. Creating new digital platforms is quite expensive and there are different platforms already developed, so there might be a possibility to use existing platforms. IT development is an iterative process by design-feedback-redesign, i.e. the digital platforms really to be used must go through a multiple improvement phases with input from end-users. Ms Britt Järvet concluded that the project proposal from the Education and Youth Board is quite ambitious in its scope trying to fit most of the national strategic goals into one project. On the other hand, CTF-Tech Ltd falls short in regard to the core purpose of the call  increasing and growing the opportunities and capabilities of local governments in the use of smart youth work solutions. Ms Katrin Olt agreed that the scope of the project proposal by CTF-Tech Ltd is narrow, not engaging with all the young people but with those between the ages from 16 to 21, and only in one specific area (cyber awareness raising). The outreach to different target groups is also vague (i.e., young people with special educational needs, young people with a different linguistic background). Still, the proposal by CTF-Tech is more precise and thought through than the one by the Education and Youth Board, whose application quality was lower than expected whilst the organisation possesses the expertise needed. Mr Ülar Vaadumäe brought out that the evaluators had also comments regarding the cost- efficiency of the Education and Youth Board proposal as well as inadequate risk analysis carried out by the organisation on the potential IT development risks (potential failures in the development of new information systems and the acceptance of solutions by the target groups. What will be done if it proves difficult to establish a concrete IT solution that is compatible with the needs of different parties, if the developer is not able to deliver the solution in the planned schedule/budget or if the interest in key users turns out to be low (e.g. local governments do not have the motivation to use the e-participation platform to collect input)? Mr Märt Aro described that there are several methods to facilitate innovation projects. The US model constitutes several teams starting with “seed money” and governmental expertise and ideas with better potential are granted several additional rounds of financing to build up the ideas. In the end, there will be scaling down / focusing on the best that would be then fully developed. Another method would be incentivising the private sector with 50% of the cost being covered from the public sector. In case the IT procurement is oriented to the number of programming hours, the iterative process is difficult to be carried out. -------------------------------- NB! So, the Selection Committee has 3 bigger alternatives how to proceed: a) rely on the ranking list of individual experts and finance one higher ranking project; b) change the ranking list, in case there are objective and transparent justifications for it; c) financing both projects, what would mean cutting the current budget of the Education and Youth Board project proposal into half  from 1 742 870 euros to 871 435 euros. Although the budget of the Education and Youth Board could be made more detailed and comprehensive (the undetailed budget seems inflated, so there seems to be room in the budget for items to be reduced), then it is objectively unrealistic to cut their budget by 50%. The Selection Committee members agreed that taking into account the set-up and the purpose of the call, then the evaluators’ current ranking list of projects is fair and there is a potential to achieve a relatively large-scale intervention with good results in the youth field in case of the project applied by Education and Youth Board. Selection Committee decisions 1. The Selection Committee decided that the Education and Youth Board project application „Digital Leap in the Field of Youth“, is worthy of a project grant, but needs to adhere to the following conditions and suggestions before the grant decision is cast by the State Shared Service Centre: - The project application must point out and prioritise what are the biggest challenges facing youth work at present (prioritising them) and how to overcome them and set the project goals from that standpoint (to focus on e-participation platform, e-youth centre, non-formal learning outcome log, e-games, etc.). It would be quite a difficult task to  consider all aspects equally as this amounts to a huge effort that leaves doubts in the Selection Committee as to whether applicant is able to fulfil all the aspects of their role as required. A lower but better-balanced ambition and a more effective execution of activities (incl. sustainable and quality solutions) with a better resource management is preferred. - The applicant should elaborate in more depth on what would the concept and the value added of the new platforms be (e-participation platform, e-youth centre, non-formal learning outcome log, e-games), why the existing tools are not good enough to be used? For instance, what will the benefits of the new e-youth centre be compared to the virtual youth centre on Discord – a solution successfully used by some local governments during Covid-19 lockdown. - The applicant must elaborate on how to cooperate with, motivate and encourage a large number of local governments to be interested in putting all the developed smart youth solutions into real practice. This applies to the involvement of local governments already in the development phase of the platforms. - The applicant must elaborate on the outreach of difficult target groups as the equality of access is one of the most critical issues in the provision of youth work – youth at risk, young people with special educational needs (not only, but including young people with disabilities), adolescent and youth immigrants. Standard digital solutions are usually not a solution for these target groups as they do not have the digital know-how or cannot access or use them for some other reasons. - The project team should – in case it’s possible – take advantage of different tools already available (and not only focus on developing new platforms). o Triumph Health (mobile health game for children’s wellbeing) https://www.triumf.health/; o Kõneravi (platform to treat speech disorders) https://koneravi.ee/; o Infohunt (platform used by both youth workers and young people) https://www.infohunt.ee/et; o Grapho Game (an application, game and methodology for learning literacy skills) https://www.graphogame.com/index.html; o Evolytes (platform to teach mathematics) https://www.evolytes.com/. - The applicant should analyse the potential risks and their mitigation mechanisms. o Are there any project management (personnel) risks to be identified and how would they be mitigated? (the project manager doesn’t have experience running similar large-scale projects as mentioned by the evaluators). o What are the mitigation mechanisms for potential failures in the development of new information systems and the acceptance of solutions by the target groups. What will be done if it proves difficult to establish a concrete IT solution that is compatible with the needs of different parties? - The applicant should consider including the method of small project rounds (see description above by Mr Märt Aro) in developing the innovative digital solutions as well as hackathon – a method that has been carried out successfully in Estonia for some years already. - Taking into account all of the above the project application should be revised and, in doing so, the project budget should be more detailed. o personnel costs should be separated out by employees and specified by units of time, workload and pay for unit;  o all the costs should be separated out, also those of the project promoter and each project partner; o IT development procurement costs, preparatory and maintenance costs should be separated out by “service”; o the costs of the three events with an average of 26 000 euros per event should be cut. With public grant funds received, the applicant should adhere to the principles using public funds economically and purposefully. The result must be achieved with a reasonable price. The final grant amount to be awarded to the Education and Youth Board can be confirmed after the revisions in content and budget have been made. The Selection Committee stresses that the new solutions developed must be both: a) attractive to all young people and youth workers b) easily accessible to all young people and youth workers c) sustainable in the long term. 2. The Selection Committee decided to decline a project grant to CTF-Tech Ltd as the total amount of the call will be exhausted with only one project able to be financed. Mr Ülar Vaadumäe thanked everybody for their input and closed the meeting. Meeting chaired by (digitally signed) minutes taken by Katrin Olt (digitally signed) Ülar Vaadumäe  EEA and Norway Grants 2014-2021 Programme: Local Development and Poverty Reduction Selection Committee meeting August 3, 2021 Open call for proposals „ Smart Youth Work“  Overview Project max duration: 30 months Call amount: € 1,742,870 Project grant size: € 871,435 – 1,742,870 Project grant rate: 85%-100% Eligible applicants: all legal persons under public or private law, non-governmental organisation registered in Estonia. Open call is financed from the EEA Grants FM  Eligible activities:  developing smart youth work solutions. The innovative methodologies and information technology solutions developed must be based on the actual interests and needs of young people and be developed in cooperation with young people.  development work to increase the capacity of local governments and youth organizations and to implement the adoption of smart youth work solutions.  activities directed towards youth workers, which increase their capacity to use smart youth work solutions. Activities may include trainings, cooperation and networking meetings and seminars with partners, outreach activities for young people and other relevant stakeholders.  Targets (cumulative for the open call):  at least 50 specialists who work with young people and children trained to develop applications based on target group needs;  at least 48 local governments must participate in the project’s activities. Number Number Expected programme Indicator Baseline Baseline Target results value year value Share of local governments Outcome 1 Enhanced wellbeing of involved in programme 0 N/A 60% children and youth activities applying smart solutions in youth work Evaluation criteria of the open call Criterion Points % Need for the project and conformity with the goals of the 1 open call 25 27,17% 2 Substantive quality of intervention logic 25 27,17% 3 Strategic project management 20 21,74% 4 Profile and capacity of project promoters 10 10,87% 5 Project visibility and sustainability 10 10,87% 6 Cooperation with donor countries 2 2,17% TOTAL 92  Projects received within deadline:  Altogether 2 project proposals were submitted by deadline a) Education and Youth Board in partnership with University of Tartu, Tallinn University, the Association of Estonian Open Youth Centres and the Estonian Association of Youth Workers b) CTF-Tech Ltd in partnership with the University of Tartu and CybExer Technologies Ltd The total grant amount applied for is 2,6 MEUR, exceeding the available grant amount by approx 870 000 euros. Evaluation of independent experts  Evaluator 1: Ms Maarja Olesk, PhD, analyst at Institute of Public Studies  Evaluator 2: Ms Heleri Reinsalu, adviser to the Strategy Unit of the Government Office.  There were no project applications with points difference over 30% between the scores given by experts, therefore a third evaluation was not necessary (point 20.5)  There were no project proposals receiving less than 50% of the total score, and therefore would have received automatically a negative decision (point 20.6).  Ranking list of experts Education and CTF-Tech Ltd Youth Board Selection criteria No Criterion Max score Average score Average score Need for the project and conformity 1 with the goals of the open call 25 25,0 19,5 Substantive quality of intervention 2 logic 25 20,5 18,0 3 Strategic project management 20 14,5 15,5 Profile and capacity of project 4 promoters 10 9,5 9,0 5 Project visibility and sustainability 10 8,5 6,0 6 Cooperation with donor countries 2 0,0 1,0 TOTAL 92 78,0 69,0 Selection Committee members Voting members Ms Katrin Olt– chief expert of the Department of Youth and Talent Policy, Ministry of Education and Research; Ms Britt Järvet – chief expert of the Department of Primary and Basic Education, Ministry of Education and Research; Mr Märt Aro– education innovation expert, director of Nordic EdTech Forum Observers The representatives of the National Focal Point and Programme Operator Implementing Agency (both State Shared Service Centre) as well as the donor states (Norwegian Ministry of Foreign Affairs and Financial Mechanism Committee) shall be invited to participate as observers. Selection Committee tasks The goal to confirm the list of project applications suggested to be financed and the list of project applications not to be financed and forward its decision to the Programme Operator Implementing Agency (i.e. State Shared Service Centre) for formal decision making. The tasks  review the ranking list of project applications resulting from the mathematical average of individual expert evaluation scores;  confirm the list of project applications suggested to be financed and the list of project applications not to be financed;  modify the ranking list of the project applications or propose only partial financing of projects in justified cases based on transparent criteria recorded in the minutes of the committee meeting. After the Selection Committee  Chairman of the Selection Committee and the PO will sign the minutes of the meeting, where decisions are laid down in a transparent and accountable manner describing which project proposals shall be awarded a grant and which proposals will be rejected.  PO Implementing Agency will cast a decision to award a grant or decline awarding a grant based on the written recommendation of the Selection Committee. Discussion and decisions  Aitäh! Takk! Ülar Vaadumäe ([email protected])