Proposal for the Evaluation of Open Data Portals

Propuesta para la evaluación de portales de datos abiertos

Proposta para a avaliação de portais de dados abertos

Camila-Andrea Herrera-Melo¹

Sin institución

Juan-Sebastián González-Sanabria juansebastian.gonzalez@uptc.edu.co

Universidad Pedagógica y Tecnológica de Colombia, Colombia

This work is licensed under Creative Commons Attribution 4.0 International.

Open data helps government institutions disseminate information of interest to civil society in order to provide transparency and social control, and thus, empower citizens through information access, to the point that today this philosophy of openness has transcended to other areas such as academia and research institutes, who seek the development and improvement of services, plans, programs, projects and standards with the collaborative participation between state-citizen-company.

The “open” data must have technical and legal characteristics to be used, reused and redistributed by any person or entity, without any restriction; These parameters are stipulated in the International Open Data Charter [1].

In favor of this initiative, in some countries standards and portals have been implemented in order to contribute to its use; For example, in Colombia, Law 1712 of 2014 obliges all public entities to disclose their data, and since 2016, the nation adopted the principles established in the International Open Data Charter, making the Colombian State Data Portal available as a space for the dissemination of public information in the country [2]. Likewise, portals were created at the departmental and municipal levels, with the objective that each entity had its own space for data opening. In 2019, a total of 30 portals focused on the dissemination and access of open data were registered.

However, having quality open data portals implies that they fulfill a dynamic role in the data life cycle and that they establish a relationship between producers, publishers and data consumers, through interaction mechanisms that contribute to aspects such as identification of the demand for data, data publication of interest for specific users, the feedback of data sets and the portal, as well as the improvement of their quality.

At international level there are several proposals from experts in the area for the evaluation of open data portals, each with different dimensions, factors or aspects to carry out this process. Therefore, and given that the portal is the means by which the quality of the published data is guaranteed, the search is facilitated by the users, the data is available in usable formats and these are published so that respond to a specific demand in order to meet specific needs that generate value, for which they have an integral evaluation methodology with criteria and dimensions proposed by experts in preliminary work.

In order to have the necessary basis for the formulation of an evaluation proposal that covers the different perspectives, as well as to create a wider and complete evaluation mechanism, a documentary review of works and research was developed in the recent years for portals evaluation.

In most of the works, an implementation of the Tim Berners-Lee five-star model was found, where it is proposed to evaluate the opening of data from its accessibility and reuse through five levels, represented by stars, that evaluate: 1. If data is only published in any format under an open license, 2. If data is structured, 3. If they are in non-proprietary formats, 4. If URI is used to access specific data directly, and 5. If they are linked to other data generating context [3].

In the case of the Open Data portal of Barcelona, the authors evaluated the quality of the portal data according to its reuse, they complemented the five-star model with the proposal to include factors such as the frequency of updating and geolocation of the data and related the amount of download and themes, according to the number of stars obtained with the model [4]. Similar case to the evaluation of portals of the European Union, where relevance is given to the analysis of the state of the data sets and the standards in which they were published at the time for the implementation of recommendations and the general improvement of portals [5].

In the case of the Barcelona Open Data Portal, the authors evaluated the quality of the portal data according to its reuse, complemented the five-star model with the proposal to include factors such as the frequency of updating and geolocation of the data and related the amount of download and themes, according to the number of stars obtained with the model [4]. Similar case is the one of the European Union, portal evaluation, where relevance is given to the analysis of the state of the data sets and the standards in which they were published at the time for the implementation of recommendations and the general improvement of portals [5].

Although it is evidenced in other works that more robust portals evaluation models are proposed that complement, to a large extent, the model proposed by Berners-Lee, enriching aspects of data and portal quality [6], as well as the Using indicators proposed by organizations such as the Open Knowledge Foundation (OKF) for open data programs [7], it is considered that there are factors that are left out of the scope of the study or not covered in depth, but are necessary for the evaluation of the quality of data and portals, for example, the evaluation of metadata and communication channels offered by the portals.

As for the other studies, there are methodologies such as Meloda, which is used for the exclusive evaluation of data reuse [8]; the evaluation of metadata from its use, availability, completeness, openness and addressability [9]; the analysis of the structural composition of the portal based on its conformation and categorization [10], and the evaluation of national portals through the general characteristics of the portals and the data set [11].

Among the methodologies, models and standards of found evaluation, those presented in Table 1 stand out.

Table 1.
Methodologies, models and standards of found evaluation.

Table 2 shows a consolidation of the dimensions measured by each of the methodologies described in Table 1.

Table 2
Dimensions evaluated by the methodologies

Taking as reference the methodologies presented in Table 1, the evaluation of open data from two approaches is proposed: 1) Published data, covering quality, use and metadata, and 2) Portal, highlighting aspects of its structure, usability and communication mechanisms. Each dimension is composed of several factors, whose general criteria are explained in Table 3.

Table 3
Proposed methodology for portal evaluation

As part of the proposed methodology, a quantitative measurement system is proposed with the objective of scoring each of the presented criteria (Table 3). Each approach, portal and data has a maximum score of 100 points, distributed as shown in Table 4. The final score will be:

𝑆𝑐𝑜𝑟𝑒=(𝑑𝑎𝑡𝑎 𝑠𝑐𝑜𝑟𝑒∗0.6)+(𝑃𝑜𝑟𝑡𝑎𝑙 𝑠𝑐𝑜𝑟𝑒∗0.4)

That is, the score obtained when evaluating the data will be equivalent to 60% of the score, and the portal score will have an equivalence of 40%. Although some of the criteria proposed may have qualitative considerations, the methodology proposes a quantitative approach to the evaluation of factors, with the objective of responding to the use of indicators to evaluate open data initiatives, as organizations such as the World Wide Web Foundation with the Open Data Barometer, or the Open Knowledge Foundation with the Global Open Data Index.

Table 4
Score by criterion factor and dimension

The maximum score to be obtained in each criterion that makes up each factor is presented in the boxes in Table 4. These criteria are related to the data:

Quality:

1. 1. Availability:

   1. A) The set is available for viewing.

   2. B) The data set can be used without any restrictions.

   3. C) You have access to the data that was completed based on requests for completeness and improvement.

2. 2. Upgrade:

   1. A) There is a record of the updating periodicity of the data set.

   2. B) The data set is updated with time according to the subject and purpose of its publication.

3. 3. Accessibility:

   1. A) The data set is downloadable.

   2. B) It is possible to access the data set through an API.

4. 4. Visualization:

   1. A) Data is available in tables or other graphic representation that allows a better understanding of the whole.

   2. B) Data can be exported to different formats that allow its use.

5. 5. Publication formats:

   1. A) Data is in non-proprietary formats.

   2. B) Data is in machine-processable formats that allows its use.

6. 6. Completeness:

   1. A) Data set has a sufficient number of records for studies and analysis.

   2. B) It does not present empty or null fields.

   3. C) The fields are consistent with the objective of the columns, maintaining consistency with the whole set.

Use:

1. 1. Defined demand: it is clearly known to whom the data is directed.

2. 2. Number of visualizations: it is possible to determine the number of people who have visualized the data set.

3. 3. Download:

   1. A) The data set has been downloaded at least once.

   2. B) The data set has a significant average discharge.

4. 4. API: queries can be made through parameterizable addresses that allow obtaining specific fields of a data set.

5. 5. Resulting products:

   1. A) The data reference products and applications derived from the use of the set.

   2. B) The use of data set for the creation of products and services is in detail.

   3. C) Data reference graphs and reports made by users.

Metadata:

1. 1. Use: metadata is used to detail the characteristics of the data sets.

2. 2. Completeness:

   1. A) The public to whom the group is directed is explicitly defined, as well as its purpose.

   2. B) The purpose of the fields is defined in detail and without exceptions.

   3. C) The contact information of the data author is available.

3. 3. Recoverability:

   1. A) The topic to which the data belongs is specifically defined, allowing it to be related to similar sets or of the same categorization axis.

   2. B) Keywords are specified that allow the rescue of the data set in subsequent searches.

In relation to the portal:

Structure:

1. Categorization: data sets are consistent with respect to similarity with other sets that are classified in the same category.

Usability:

1. 1. Search

   1. A) Searches by entities or publishers are available.

   2. B) It is possible to search for data through themes, topics or categorization.

   3. C) You can search for periods that allow you to obtain data from a specific time.

2. 2. Navigability:

   1. A) The portal has a navigation map available to users, where the structure of the site is evidenced.

   2. B) The portal has a simple navigability that allows users to scroll through the portal and find information quickly.

   3. C) The portal implements different elements to facilitate navigability in the system, such as: help buttons, contact buttons, navigation bars, a general menu.

3. 3. Use / consumption / data download:

   1. A) The portal offers the possibility to visualize data in order to facilitate its analysis and understanding.

   2. B) It is possible to download the data from the portal in different formats that allow its versatility of use, without any restriction.

   3. C) The portal makes available to users at least one API that allows the consumption and consultation of data.

   4. D) The portal offers statistics about the users use of data.

Communication:

1. 1. Comments and discussion:

   1. A) It is possible to comment on the data sets at their place of publication.

   2. B) The portal has spaces where users can deal with topics related to the data available on the portal.

   3. C) The portal provides support mechanisms between users through forum-like spaces.

   4. D) A space is offered for users to view and learn about the resulting products from the use of data published on the portal.

2. 2. Source-user:

   1. A) The publisher is notified when comments are received about the data sets he has published.

   2. B) Users are notified when the data sets on which they showed interest are updated or modified.

   3. C) In the portal there is the contact information of the entities or publishers.

   4. D) The portal offers direct communication between the publisher and the end user, contributing to the improvement of data quality.

3. 3. Requests:

   1. A) Users can make direct requests for specific data sets through the portal.

   2. B) Users are notified when there is a response to their request.

In case the score gives a decimal value, it must be adjusted by rounding. Next, Table 5 shows the scores with their corresponding classification.

Table 5
Portal quality evaluation

If, when evaluating a portal, the data score was 50 points and that of the portal was 63 points, the following would be obtained:

𝑆𝑐𝑜𝑟𝑒=(50∗0.6)+(63∗0.4)

𝑆𝑐𝑜𝑟𝑒=30+25.2=55.2

According to the classification proposed in the methodology, the portal would have an acceptable quality.

With the aim of evaluating the methodology, it was applied in the open data portal provided by the Colombian government (https://www.datos.gov.co/), based on the experience of a group of users, both experts as inexperienced. The qualification obtained is presented in Table 6, which also summarizes the main aspects that justify the evaluation of each factor or criterion.

Table 6
Quality evaluation of the Colombia Open Data portal

All the above, gives the portal the following score:

𝑆𝑐𝑜𝑟𝑒=(48∗0.6)+(59∗0.4)

𝑆𝑐𝑜𝑟𝑒=52.4

Consequently, according to Table 5, the portal would have an acceptable rating, which indicates that, although it has different functionalities, it is necessary to add control points that provide greater satisfaction to the end user, eliminating sets that do not comply with minimum quality conditions or allowing to qualify a set by users.

The use of methodologies and models to determine the quality of the data contributes to the improvement of these, based on the identification of the status and flaws that may occur, also helps the continuation of the life cycle of open data, whose processes are in constant improvement.

Each methodology provides a different approach to the extent that its evaluation criteria is raised, which may lead to the studied element (portal or data) having different quality levels, depending on the used methodology. However, it is not unknown that the approach to a more real quality result is given by the combination and complement of methodologies and models that allow a greater number of aspects to be covered.

Open data portals play an important role in data opening initiatives, since they are the main point of access and availability of data, mainly published by government entities, which is why the quality of the data, of the structure of the data portal and the characteristics it provides to its users, can determine its level of use, impact and reputation; This is why the responsibility of the portals also lies in their constant improvement to offer users the highest possible quality.

When interacting with the Open Data portal of the Colombian State, it has been found that there are a large number of data sets available, but that many of them present inconsistencies or other flaws that hinder their use, which evidences the need to evaluate the portal with regarding its data and structure, since this type of aspects may raise the question about the use of portal resources.

Herrera-Melo carried out the process of documentary review and the evaluation of the state of the art of the evaluation methodologies. González-Sanabria validated the proposed methodology and applied it to the study case. Both authors wrote and validated the document.

The authors have declared that no competing interests exist.

Citation: C.-A. Herrera-Melo, J.-S. González-Sanabria, “Proposal for the Evaluation of Open Data Portals,” Revista Facultad de Ingeniería, vol. 29 (54), e10194, 2020. https://doi.org/10.19053/01211129.v29.n0.2020.10194