Writing your own data to Wikidata using spreadsheets: Part 2 - editing the real Wikidata

 For a video walk-through of the previous blog post and this one, see this page.

 In the previous post, I described how to create a Wikimedia bot password and use it to write spreadsheet data to the test Wikidata instance: https://test.wikidata.org/. The process required setting up a JSON metadata description file that mapped the CSV column headers to the RDF variant of the Wikibase data model. The VanderBot Python script used that mapping file to "understand" how to prepare the CSV data to be written to the Wikidata API. The script also recorded its interactions with the API by storing identifiers associated with the knowledge graph entities in the CSV along with the data.

This post will continue in the "do it yourself" vein of the previous post. In order to successfully complete the activities in this post, you must:

• have a credentials plain text file (prepared in the last post)
• have Python installed and know how to run a script from the command line (Python programming skills not required)
• have downloaded the VanderBot script to a directory on your local drive where you plan to work.
• understand that the edits you make are your responsibility just as if you had made them using the graphical interface. If you mess something up, you need to fix it -- mostly likely manually since VanderBot is designed to upload new data, not change or delete existing data.
• have practiced on the test Wikidata instance enough to feel comfortable using VanderBot to make edits. 

If any of these things are not true, then you need to go back and read the first blog post to prepare. 

Options when running the script

In the last post, we practiced with VanderBot using all of the settings defaults. However, you may want to change some of those defaults depending you your situation. The most obvious change is to suppress the display of the giant blobs of response JSON from the API that fly up the screen as the script runs. You can redirect most of the output to a log file using the --log option. The log file will record the JSON output and at the end will include a summary of known errors that occurred throughout the writing process. (The same error report will be shown on the console screen, too.) You may choose to ignore the log file most of the time -- it will simply be overwritten the next time the script is run. However, it may be useful if the script terminates due to an error. 

Most of the other options allow you to designate different file names or locations for the metadata description file and credentials file. It may be convenient to keep the credentials file in the same directory as the other files (the working directory option), but if you are using version control (e.g. GitHub) you should keep it elsewhere. You may wish to use different file names if you have multiple bot passwords or have different metadata description files for different CSVs.

The --update option is used to control whether the label and descriptions in the CSV will overwrite different values for existing items in Wikidata. It defaults to suppress updates, but we will talk later about when you might want to use a different option.

Options:

long form     short values                                  default

--log         -L    log filename, or path and appended     none

                      filename

--json        -J    JSON metadata description filename     "csv-metadata.json"

                      or path and appended filename

--credentials -C    name of the credentials file           "wikibase_credentials.txt"

--path        -P    credentials directory: "home",         "home"

                      "working", or path with trailing "/"

--update      -U    "allow" or "suppress" automatic        "suppress"

                      updates to labels and descriptions                      

Option examples:

Note: some installations of Python require using python3 instead of python in the command.

python vanderbot.py --json project-metadata.json --log ../log.txt

Metadata description file is called project-metadata.json and is in the current working directory. Progress and error logs saved to the file log.txt in the parent directory.

python vanderbot.py -P working -C wikidata-credentials.txt

Credentials file called wikidata-credentials.txt is in the current working directory.

python vanderbot.py --update allow -L update.log

Progress and error logs saved to the file update.log in the current working directory. Labels and descriptions of existing items in Wikidata are automatically replaced with local values if they differ. Notice that the long and short forms of the options can be mixed and are interchangeable.

Writing to the "real" Wikidata

Once you have set everything up, it is a simple matter to switch from writing from the test.wikidata.org API to the "real" www.wikidata.org API. All that is necessary is to change test to www in the first line of the configuration file:

endpointUrl=https://www.wikidata.org

The username and password lines can stay the same. 

However, we cannot use the same CSV and metadata description files as before because the property and item IDs are different in the real Wikidata. We also don't yet want to create new items until we are comfortable with making several edits in the real Wikidata. Fortunately, there are several items in Wikidata that are designated as "sandbox" items, i.e. their metadata can be change to anything by anyone without consequence. They are generally lightly used, so you can edit them and still have time to examine what you have done before someone changes them to something else. The first sandbox item (Q4115189) is better known than the other two, so we will use sandbox items 2 and 3 in our practice. 

 Wikidata sandbox items:

Q ID      Label

--------  ------------------

Q4115189  Wikidata Sandbox

Q13406268 Wikidata Sandbox 2

Q15397819 Wikidata Sandbox 3

With respect to etiquette regarding the sandbox items, I don't know that there are particular rules, but I would say that it would not be acceptable to change their labels, since that is the primary means by which users will know what they are. I would say that anything else, including descriptions and aliases is probably open to editing. 

I would avoid adding a large number of statements to the sandbox items and then just leaving them, although a few edits probably don't matter. Probably the best thing to do after you are done playing with editing the sandbox items is to go the the View history page and undo your change (if you've only made one edit) or restore the last previous version before you started playing with the item if you've made a lot of changes. I'll show how to do that when we get to that point.

After you are comfortable playing with the sandbox items, we will try adding new real items. 

Describing a CSV using simpler JSON format

We could use the web tool again to create a new metadata description file based on the real Wikidata properties, but I will tell you about another tool that you can use that requires many fewer button clicks. I created a simplified configuration file format that can be used to generate the standard metadata description file based on some rules about how to construct the column header names, assumptions about labels and descriptions, and one simplification of references. (The detailed specifications for the configuration file format are here.) The configuration file that we will be using has the default name config.json and can be viewed in this gist.  

It is not necessary for you to edit this file to use it for the practice exercise. You can simply download it (right-click on the Raw button and select Save file as...). Download it into a directory you can access easily from your home folder -- you can use the same one you used last time, although it might get a bit cluttered.

If you understand JSON, the file structure will make sense to you. Even if you don't, you can probably copy and paste parts of it to change it to fit your needs. (It includes examples of most of the object types including two we haven't used before: monolingual text and quantity.) If you copy and paste, you will mostly need to be careful about placement of commas. Indentation is optional in JSON and is only used to make the structure more apparent. 

For now, we can ignore the first three key:value pairs. The rest of the JSON after outfiles describes two CSV files that will be mapped by the metadata description file: artworks.csv and works_depicts.csv . artworks.csv contains data about statements involving 5 properties: P31 (instance of), P217 (inventory number), P1476 (title), P2048 (height), and P571 (inception). depicts.csv contains data about only one kind of statement: P180 (depicts). You may be wondering why I chose to put the depicts statements in a separate CSV. That is because all of the other properties will typically have only one value per item, while a particular artwork may depict several things. So in the first CSV, there will only be one row for each item, while in the second CSV there is an indefinite number of rows per item. 

{

"pid": "P571",

"variable": "inception",

"value_type": "date",

"qual": [

{

"pid": "P1319",

"variable": "earliest_date",

"value_type": "date"

},

{

"pid": "P1326",

"variable": "latest_date",

"value_type": "date"

}

],

"ref": [

{

"pid": "P248",

"variable": "statedIn",

"value_type": "item"

}

]

}

Each property has a pid (property ID), a column header name (variable) and a value_type. The value_type will determine the details of the number of data columns needed to represent that kind of value and the kind of data that will be stored in those columns. Each property can also have zero or more qualifier properties and zero or more reference properties associated with it.  In the snippet above, inception (P571) statements will have two associated qualifier (earliest date and latest date) properties and one reference property (stated in). The Wikibase model allows many references per statement, but this configuration file format restricts you to a single reference with as many properties as you want. 

The structure of the qualifier and reference properties are the same as the statement properties (ID, variable, and value type) with the only restriction being that you must use properties that are appropriate for use in qualifiers or references. 

{

"manage_descriptions": true,

"label_description_language_list": [

"en",

"es"

],

"output_file_name": "artworks.csv",

"prop_list": [

The situation with labels and descriptions is a little more complicated. If you have more than one data table, you probably only really want to manage the labels and descriptions in one of the tables. In this case, it would make the most sense to manage them in the artworks.csv table, since it has a row for every item and the other table may have zero or more than one row per item. So the manage_description value for the first table is set to true. 

{

"manage_descriptions": false,

"label_description_language_list": [

"en"

],

"output_file_name": "works_depicts.csv",

"prop_list": [

In the second table (works_depicts.csv)  the manage_descriptions value is set to false. In that table, there will be a label column, but it will be set to be ignored during CSV processing and will only be to help humans understand what is in the rows. The label_description_language_list value contains a list of the ISO language codes for all languages to be included. If manage_description is set to true for a table, there will be both a label and description in the table for every language. If it is set to false for a table, there will only be a label for the default language. The default language of the suppressed output labels is set by the --lang option (see below). Any languages supplied in the JSON (as in the example above) will be ignored.

Generating the metadata description file and CSV headers

To generate the metadata description files and the CSV files, we need to download another script from GitHub called convert_json_to_metadata_schema.py . Download it into the same directory where you downloaded the config.json file. At the command line, run the following command if you used the default name config.json

python convert_json_to_metadata_schema.py

If you saved the input configuration file with a different name, or if you want a different name than csv-metadata.json to be used for the output metadata description file, use the following command line options:

long form     short values                                     default

--config      -C    input configuration file path              config.json

--meta        -M    output metadata description fille path     csv-metadata.json

--lang        -L    language of labels when output suppressed  en

After you run the script, it will have generated the csv-metadata.json file and also variants of the two CSV files that were specified in the input config.json file: artworks.csv and works_depicts.csv . To prevent accidentally overwriting any existing data, the letter "h" is prepended to the file names of the generated CSVs (hartworks.csv and hworks_depicts.csv). So before you use the files, you need to delete the initial "h" from the file names. 

The generated CSV files contain only the column headers with no data. But you can still open them with your spreadsheet software to look at them. 

 

If you compare the columns in the created spreadsheet with the source JSON configuration file, you should see that the columns are in the order that they were designated in the JSON. The variable values are joined to any parent properties by underscores (e.g. earliest_date appended to inception to form inception_earliest_date). In cases where more than one column is required to describe a value node, the _nodeId, _val, etc. suffixes are added to the corresponding root column 

Since we chose true as the value of manage_descriptions for the artworks.csv file, the generated spreadsheet includes both labels and descriptions for the two languages we designated. The script automatically prepends label_ and description_ to the language codes to generate the column headers.

For the second spreadsheet, works_depicts.csv, the value of manage_descriptions is false, so only labels are generated. Since the label_en column is only for local use to make the identity of the rows clearer, I only bothered to generate it as English. The value of the labels in this spreadsheet will be ignored by the API upload script.

Adding data to the CSV files

Since we are still testing, we won't create new items yet in the real Wikidata. Instead, we will add statements to two of the sandbox Wikidata items. 

In the qid column of the artworks.csv file, add Q13406268 and Q15397819 to the first two rows after the header row.  For purposes of keeping the row identities clear, I added Wikidata Sandbox 2 and Wikidata Sandbox 3 as label_en values for those rows, although since we will be using the default to suppress updating labels, these values will have no effect. I also chose to use Q3305213 (painting) and Q860861 (sculpture) as values of instance_of since the the CSV file is supposed to be about artworks. 

If you want to see what other values I used in my test, you can look at or download this gist. You can use whatever values would amuse you as long as the types of the values match the types that are appropriate for properties specified in the configuration file. Leave all of the ID columns blank (those ending in _uuid, _hash, or _nodeId), since they will be filled in by the API upload script. For the dates, you can either use the abbreviated conventions discussed in the last post (in which case you MUST leave the _prec column empty). If you want to use dates that don't conform to those patterns (precisions less than year, BCI dates, or dates between 1 and 999 CE), you will need to use the long form values and provide an appropriate _prec value. See the VanderBot landing page for details.

If you looked at the configuration JSON carefully, you may have noticed that there were two new value types that we didn't see in the last post. The title property (P1476) has the type monolingualtext. Monolingual text values are required to have a language tag in addition to a provided string. Unfortunately, because of limitations of the W3C CSV2RDF Recommendation, the language tag has to be hard-coded in the metadata description file rather than being specified in the CSV. That's why the language is specified in the configuration JSON as the value of language for that property rather than as a column in the CSV table. 

The other new value type is quantity. Like dates, quantities have value nodes that require two columns in the CSV table to be fully described. The _val column contains a decimal number and the _unit column should contain the Q ID for an item that is an appropriate measurement unit for the number (e.g. Q11573 for meter).

The second spreadsheet, works_depicts.csv, describes only one kind of statement, depicts (P180). It is intended to have multiple rows with the same qid, since a work can depict more than one thing. Since I described the Sandbox 2 item as a painting with title "Mickey Mouse house", I decided to say that it depicts Mickey and Minnie Mouse. You can set the depicts values to any item.

Writing the data

Before writing the data in the CSVs, open the pages for Sandbox 2 and Sandbox 3 so that you can see how they change when you write. Make sure that the two CSV files, the csv-metadata.json file you generated from config.json, and a copy of vanderbot.py are together in a directory that can easily be accessed from your home directory. Make sure that you removed the "h" from the beginning of the CSV filenames as well.

Open your console application (Terminal on Mac, Command Prompt in Windows), navigate to the directory where the files are, and run VanderBot. Unless you changed default file names and locations, you can just enter

python vanderbot.py

(or use python3 if your installation requires that). If you want to save the API response in a log file, specify its name using the --log (or -L) option, like this:

python vanderbot.py -L log.txt

When you run the script, you should see something like this (with logging to file):

There are two episodes of writing to the API, one for each of the CSVs. If you refresh the web pages for the two items, you should see the changes that you made.

Click on the View history link at the top of the Wikidata Sandbox 2 page. You will see the revision history for the page.

Notice that on Sandbox 2, there were three edits listed. Each line in a spreadsheet resulted in one write to the API. The first larger one (4997 bytes) was an update consisting of all of the statements made in the first line of artworks.csv . The two later and smaller ones were from the two single-statement depicts writes in the works_depicts.csv table.

It is not a requirement to get rid of all of your edits to the sandbox, but to avoid causing the sandbox items to be hopelessly cluttered, you should probably delete your edits. If you made only a single change to the page, you can just click the undo link after the edit. If you made several edits and your edits were the last ones, you can revert back to the last version before your changes by clicking on the restore link after the last edit that was made prior to yours.

The restore dialog will show you all of the changes you made so that you can review them before committing to the restore. Give a summary, and click Publish changes.

Changing labels

VanderBot handles labels and descriptions differently from statements and references. 

Adding statements or references is controlled by the presence or absence of an identifier corresponding to the column(s) representing the statement or reference in the spreadsheet. The statements or references are only written if their corresponding identifier cell is empty. If you examine the CSVs after their data have been written to the API, you will see that identifiers have been added for all of the columns that contain statement values. That means that if you run the script again, nothing will happen because VanderBot will ignore the values -- they all have assigned identifiers. 

The behavior of labels and descriptions is different. When a new item is created, any labels or descriptions that are present will be added to the item. However, VanderBot will NOT make any edits to labels or descriptions of existing items unless the the --update (or -U) option is set to allow when the script is run. If updating is allowed, the existing labels and descriptions will be changed to whatever is present in the spreadsheet for that item. (The exception to this is when a label or description cell is empty. Empty cells will not result in deleting the label or description.) 

From the screenshot above, you can see that at the start of this experiment, Sandbox 2 had no descriptions in either Spanish or English. Also, the Spanish label isn't actually in Spanish. I'm going to use VanderBot to change that.

To make the changes, I started with the artworks.csv spreadsheet after my last edits. I deleted the line for Sandbox 3 since I didn't want to mess with it. I first made sure that my English label was exactly the same as the existing label so that it won't be changed. Then I added a Spanish label, and English and Spanish descriptions. I left the rest of the row the way it was since none of those statements would be written since they all had IDs. 

The following command will write the labels and log to a file:

python vanderbot.py -L log.txt --update allow

After the script finishes, checking the log file shows the changes made.

Since the English label was identical, there were no changes to it. The log also shows that there were no changes in the other CSV. 

Checking the web page shows

Since I'm done with the test, I'm going to delete the descriptions, but the Spanish label isn't any worse than what was there before, so I'll leave it. Checking the history, I can see that all of the labels and descriptions were changed in a single API write, so I'll have to delete them manually if I want to leave the Spanish label -- I can't undo the description changes without also undoing the new Spanish label.

The take-home message from this section is that you need to make sure that the existing labels and descriptions in the CSV match what is in Wikidata when label/description updates are allowed (unless you actually want to change them). This is particularly an issue if your data table is stale because you are coming back to work on it at a much later time after you initially wrote the data. If in the intervening time other users have improved the quality of the data by changing labels and descriptions, you would essentially be reverting their changes back to a worse state. That would be really irritating to someone who put in work to make the improvements. I will talk about strategies to avoid this problem in a later post.

Creating new items

At this point, you are hopefully comfortable with VanderBot enough to create or edit real items in the real Wikidata. For now, let's stick with creating new items since editing existing items has the issue of avoiding creating duplicate statements. We will address problem that in a future post. 

There are several issues that you should consider before creating new items for testing. One is that you really should only create items that meet some minimal standard of notability. The actual notability requirements for Wikidata are so minimal that you could theoretically create items about almost anything. But as a practical matter, we really shouldn't just create junk items that don't have some relatively useful purpose. One type of item that seems to be relatively "safe" is university faculty, since they generally have the potential to be authors of academic works that could potentially be references for Wikipedia articles. When I'm testing VanderBot, I often add faculty from my alma mater, Bluffton University, since none of them were in Wikidata until I started adding them. 

The second issue is that you should create items that have enough information that the item can actually be unambiguously identified. There are several really irritating categories of items that have been added to Wikidata without sufficient information. There are thousands of "Ming Dynasty person" and "Peerage person" items that have little but a name attached to them. They are pointless and just make it harder to find other useful items with similar labels. So, for example, if you add faculty to Wikidata, at a minimum you should include their university affiliation and field of work. 

The third issue is that you should make sure that you are not creating a duplicate item. In a future post I will talk about strategies for computer-assisted disambiguation. But for now, just typing the label in the search box in the Wikidata is the easiest way to avoid duplication. Try typing it with and without middle initials and also with and without periods after the initials to make sure you tried every permutation.

Configuring the properties

If you want to try my strategy of practicing by creating faculty records, you can start with this template configuration file. It contains the obligatory instance of (P31) that should be provided for every item, and sex or gender (P21), which despite its issues is probably the most widespread property assigned to humans. I did not provide reference fields for those two properties since they are commonly given without references. The other two properties are probably the minimal properties that should be supplied for faculty: employer (P108) and field of work (P101). One reason why I chose these two properties is because they can both be determined easily from a single source, the Bluffton University faculty web listing. The statements for these two properties should definitely have references.

I've done some querying to try to discover what the most commonly used properties are for references. A key reference property is retrieved (P813). All references should probably have this property. The other property is usually an indication of the source of the reference. Commonly used source properties are: reference URL (P854, used for web pages), stated in (P248, used when the source is a described item in Wikidata with a Q ID value), and Wikimedia import URL (P4656, used when the data have been retrieved from another Wikimedia project with a URL value). Unless you are working specifically on a project to move data from another project like Wikipedia to Wikidata, the first two are the ones you are most likely to use. Since all of my data are coming from a web page, I'm using P854 and P813 as the reference properties for both of the statement types that have references.

Use the convert_json_to_metadata_schema.py Python script and the config.json file you downloaded to generate the metadata description file csv-metadata.json and the faculty.csv CSV file (with header row only). 

Adding the data

I chose two of the faculty from the web page list and pasted their names into the Wikidata search box to make sure they weren't already existing items. I then added their names to the label_en column of faculty.csv and described them in the description_en column. See this gist for the examples. Q5 is the value for instance_of for all humans. Sex or gender options are given on the P21 property page. All of the faculty work at Bluffton University (Q886141). The trickiest value was their field of work, which I had to determine using the Wikidata search box. 

I was then able to copy and paste the URL for the faculty web listing,  https://www.bluffton.edu/catalog/officers/faculty.aspx, into the reference URL columns and today's date, 2021-03-07, into all of the _retrieved_val columns. I then saved the file. 

Writing the data to the API

Note: what would happen if you tried to use my example CSV file to write to Wikidata without changing its contents? Before the VanderBot script tries to write a new record to the real Wikidata, it checks the Wikidata Query Service to see if there are already any items with exactly the same labels and descriptions in any language in that row. If it finds a match, it logs an error and goes on to the next row. So since those items were already created by me, VanderBot will do nothing as long as no one has changed either the label or description for the two example items. If a label or description for either of them has been changed since I created the items, then the API will create duplicate items that will need to be merged later. So don't try running the script with my unmodified example files unless you first check that the labels and descriptions are still exactly the same on the items' Wikidata item pages.

I ran the vanderbot.py script with logging to a text file.

When writing the statements, the two rows were identified as new records. When the rows were later checked for any new unwritten references, the Q IDs were already known since they had been reported in the API response. 

To see how the faculty.csv file looked after its data were written to the API, see this gist.

I could check for the new item pages in Wikidata by either searching for the faculty names or by directly using the two new Q IDs. 

The new page contains all of the data from the CSV table in the appropriate place!

Although for only two items this work flow probably took longer than just creating the records by hand, it doesn't take many more items to make this process much faster, particularly if references are added (and they should be!). Adding references requires many button clicks on the graphical web interface, but because the same reference can be added to many rows of the spreadsheet with a single copy and paste, it is very efficient to add references using the VanderBot script. 

What's next?

In the next post, I'll talk about how you can determine what properties are most commonly used for various types of items. This is important information when you are planning your own projects that involve adding a lot of data to Wikidata.

Writing your own data to Wikidata using spreadsheets: Part 1 - test.wikidata.org

Warning: this blog post involves extreme hand-holding. If that irritates you and you want to try to figure out how to use VanderBot on your own without hand-holding, you can go straight to the VanderBot landing page and look at the very abbreviated instructions there. However, make sure that you understand your responsibilities as a Wikidata user. If they are unclear to you, read the "Responsibility and good citizenship" section below.

On the other hand, if you love extreme hand-holding, there is a series of videos that will essentially walk you through the steps in this post.

 It has been almost a year since I last wrote about my efforts to write to Wikidata using Python scripts. At that time, I was using a bespoke set of scripts for a very specific purpose: to create or upgrade items in Wikidata about researchers and scholars at Vanderbilt University. I was feeling pretty smug that I actually got the scripts to work, but at that point the scripts were pretty idiosyncratic. They were limited to a particular type of item (people), supported a restricted subset of property types, and used a particular spreadsheet mapping schema that wasn't easily modified. 

Since that time, I have been working to adapt those scripts to be more broadly usable and have been testing them on several other projects: WikiProject Vanderbilt Fine Arts Gallery and WikiProject Art in the Christian Tradition (ACT), and several smaller ones. The scripts and my ability to explain how to use them have now evolved to the point where I feel like they could be used by others. The goal of this series is to make it possible for you to try them out in a do-it-yourself manner. 

Background

This series of posts will not dwell on the conceptual and technical details except where necessary for you to make the scripts work. For those interested in more details, I refer you to previous things I've written:

• blog post dealing with the minutiae of Wikibase, the Wikimedia API, and authentication (June 2019)
• blog post describing how to retrieve data from Wikidata from the Query Service using HTTP and SPARQL (May 2019)
• blog post giving a general overview of the paradigm of writing to Wikidata using spreadsheets (February 2020)
• blog post with an overview of the Wikibase model and associated identifiers (February 2020)
• web page with somewhat overlapping overview of the Wikibase model but details about property labels (2019, revised 2020)
• blog post with a very brief overview of using the W3C CSV2RDF Recommendation to map spreadsheets to the Wikibase model and a discussion of issues related to timing of interactions with the API (February 2020)
• submitted manuscript with very technical description of using the W3C CSV2RDF Recommendation to map spreadsheets to the Wikibase model (submitted to Semantic Web Journal in December 2020, revised version submitted June 2021)
• blog post with overview of the workflow for the Vanderbilt scholar and researcher Wikidata project (February 2020)
• video of presentation at the 2020 LD4 Conference on Linked Data in Libraries: VanderBot: Using a Python script to create and update researcher items in Wikidata (July 2020)
• video of presentation to the Program for Cooperative Cataloging Wikidata Pilot group: VanderBot: A spreadsheet-based system for creating and updating items in Wikidata (February 2021)

It is not necessary to refer to any of this material in order to try out the system. But those interested in the technical details may find the links helpful.

Do I want to try this?

Before going any further, you should assess whether it is worth your time trying this out. 

Requirements:

• You need to know how to use the command line to navigate around directories and run a program. See this page for Mac or this page for Windows if you don't know how to open a console and issue basic commands. In particular, read the section on "Running a program using the command line".
• You need to have Python installed on your computer so that you can run it from the command line.  See this page for installation instructions. You do NOT need to know how to program in Python. I believe that the only module used in the script that is not part of the standard library is requests, so you may need to install that if you haven't already. 
• You need to have an application to open, edit, and save CSV files. The recommended application is LibreOffice Calc. Other alternatives are OpenOffice Calc and Excel, but there are situations where you can run into problems with either of them. For information on CSV spreadsheets and how to save them in Excel, see this video. For a deeper dive and description of the problems with Excel and OpenOffice Calc, see the first video in this lesson and the screenshots after the second video. 
• You must have a Wikimedia user account. The same user account is used across Wikimedia platforms, including Wikipedia, Wikidata, and Commons, so if you have an account any of those places you can use it here.
• You need to be familiar with the Wikidata graphical editing interface. I assume that every reader has already done enough editing to understand the important features of the Wikibase model (items, properties, statements, qualifiers, and references) and how they are related to each other. They will not be explained in this post, so if you don't already have experience exploring these features using the graphical interface, you are probably not adequately equipped to continue with this exercise.

Other alternatives you should consider

There are a number of good alternatives to using the VanderBot scripts to write to Wikidata. They are:

1. Use the graphical interface at https://www.wikidata.org/ to edit items manually. Advantage: very easy to use and robust. Disadvantage: slow and labor intensive. 
2. Use QuickStatements (https://www.wikidata.org/wiki/Help:QuickStatements). Advantage: very easy to use and robust, particularly when used as an integrated part of other tools like Scholia "missing". Disadvantage: there is a learning curve for constructing the input files from scratch. Users who aren't familiar with how CSV files work may find it confusing.
3. Use the Wikidata plugin with OpenRefine (https://www.wikidata.org/wiki/Wikidata:Tools/OpenRefine). Advantage: powerful and full-featured, I believe that some scripting is possible to the extent that scripting is generally possible in OpenRefine using GREL. Disadvantage: requires skill with using OpenRefine plus an additional learning curve for figuring out how to make the Wikidata plugin work, not sure whether it is possible to integrate OpenRefine with command line-based workflows involving other applications.
4. Use PyWikiBot (https://pypi.org/project/pywikibot/) or Wikidataintegrator (https://github.com/SuLab/WikidataIntegrator). Many (most?) Wikidata bots are built using one of these two Python frameworks. Advantage: powerful, full featured, robust. Disadvantage: you need to be a relatively experienced Python coder who understands object oriented programming with Python to use these libraries. There are a number of bot-building tutorials for PyWikiBot online. However, when starting out, I found the proliferation of materials on the subject confusing and for both of these platforms, when I couldn't get things to work, the libraries were so complex I couldn't figure out what was going on. Professional developers would probably not have that problem, but as someone who is self-taught, I was confused.

Factors that might make using VanderBot right for you

If any of the following situations apply to you, VanderBot might be useful for you:

1. Your data are already in spreadsheets or can be exported as spreadsheets and you would like to keep the data in spreadsheets for future reference, ingest, or editing using off-the-shelf applications like Libre Office or Excel.
2. You want to keep humans in the Wikidata data entry loop for quality assurance, but want to increase the speed at which edits can be made.
3. Over time, you are interested in keeping versioned snapshots of the data that you have written in a format that is suitable for archival preservation (CSV). 
4. You are interested in comparing what is currently in Wikidata with what you put into Wikidata to discover beneficial information added by the community or to detect vandalism by bad actors.
5. You want to develop a workflow based on command-line tools that can be scheduled and monitored by humans.

The last of these two features are not yet fully developed, but I'm trying to design the scripts I'm writing to make them possible in the future. 

If one or more of these factors applies to you and the other existing tools don't seem better suited for your purposes, then let's get started.

Responsibility and good citizenship

This "lesson" involves using the VanderBot API uploading script to write data to the test Wikidata API (application programming interface). In order to do that, you will need to create a bot password, but not a separate bot account. So let's clarify exactly what that means.

User account and bot password

When you create a bot password, you should be logged in under your umbrella Wikimedia account. That account applies across the entire Wikimedia universe: Wikipedia, Wikidata, Commons, and other Wikimedia projects. The bot password you create allows you to automate your interactions by using any Wikimedia API, but the edits that you make will be logged to your user account. That means that you bear the same responsibility for the edits that you make using the bot password as you would if you made them using the graphical interface or QuickStatements. Edits that you make using the bot password will show up in the page history just as if you had made them manually. If you make a mess using the bot password, you are responsible for cleaning it up just as you would be if you made errors using any other editing method. The whole point of scripting is to allow you to do things faster and easier, but the down side of that is that you can also make mistakes faster and easier as well. 

Because of the potential for disaster, we will start by using Wikidata's test instance: https://test.wikidata.org/ . It behaves exactly like the "real" Wikidata, except that the items and properties there do not necessarily correspond to anything real. If you make a mess in test.wikidata.org, you do NOT have to clean it up -- that's the whole point of it. So it is a place we can experiment without risk and once we feel comfortable, we can easily move to using the "real" Wikidata.

Distinction between a User-Agent and user account

In the Wikimedia world, typically when one creates an autonomous bot (one that works without human intervention), a separate bot user account is created. That account is used with a particular application (script, program) that carries out the bot's defined task. However, VanderBot is not an autonomous bot and has no particular defined task. It is a general-purpose script that can be used by any account to make human-mediated edits. So we need to draw a distinction between the application (technical term: User-Agent) and the user account. There actually is a user account called VanderBot (https://www.wikidata.org/wiki/User:VanderBot). It is operated by me and it shows up as the user who made the edits when I use it with the API-writing script. But you can't use it because you don't have the account credentials -- edits that you make will be made under your own user account. On the other hand, regardless of the user account responsible for the edits, the VanderBot Python script will identify itself to the API as the software that is mediating the interaction between you and the API. Software that manages communications between a user and a server is called a User-Agent.

You can think of this situation as similar to the difference between your web browser and you. Your web browser is not responsible for the actions that you take with it. If you use Firefox to write the world's best Wikipedia article, the Mozilla Foundation that created Firefox doesn't get credit for that. If you use Chrome to buy drugs or organize an assassination, Google, which created Chrome, does not take responsibility for that. On the other hand, if your browser has a bug that causes it to repeatedly hit a website and create a denial of service problem for a web server, the website may use the User-Agent identification for the browser to either block the browser or to contact the browser's developer to ask them to fix the bug. 

VanderBot, the User-Agent, has features that prevent it from doing "bad" things to the API, like making requests too fast or not backing off when the server says it's too busy. As the programmer, I'm responsible for those features. I am not responsible if you write bad statements, create duplicate items, or overwrite correct labels and descriptions with stupid ones. Those mistakes will be credited to your user account. On the other hand, if you significantly modify the VanderBot API-writing script (which you are allowed to do under its GNU General Public License v3.0), then you should change the value of its user_agent_header variable with your own URL and email address, particularly if you mess with its "good citizen" features and settings. 

Do you need a bot user account and flag?

Wikidata has a bot policy, which you can read about here. However, that policy defines bots as "tools used to make edits without the necessity of human decision-making". By that definition, VanderBot is not technically a bot since its edits are under human supervision (it's not autonomous).  That's good, because it means that you can use it without going through any bot approval process, just as if you had used QuickStatements or OpenRefine to make edits. 

However, not having bot approval also places rate limitations on interactions with the API. User accounts without "bot flags" (granted after successfully completing the approval process) are limited to 50 writes per minute. Writing at a faster speed without a bot flag will cause the API to block your IP address. This is the primary limitation on the speed of writing data with VanderBot and a delay (api_sleep) is hard-coded in the script. 

Note that bot approval and bot flags are granted to accounts, not User-Agents. If you use the VanderBot API-writing script along with other scripts as part of a defined automated process, you should set up a separate bot user account. You could then get a bot flag for that particular account and purpose, and remove the speed limitation from the script. I don't know if VanderBot is actually ready for that kind of use at this point. So you're on your own there.

The short answer to the overall question of whether you need a separate bot account is usually "no".

Generating a bot password

If you understand your responsibilities and have decided that experimenting with VanderBot is worth your time, let's get started on the DIY part. Because the bot password you create can be used across Wikimedia sites, I will illustrate the password creation process at https://test.wikidata.org/ since that is where we will first use it.

The test Wikidata instance looks similar to the regular one, except that the logo in the upper left is in monochrome rather than color. The functionality is identical. Click on the Special pages link in the left pane.

On the Special pages page, click on the Bot passwords link in the Users and rights section.

On the Bot passwords page, enter a name for the bot password. It is conventional to include "bot" or "Bot" somewhere in the name of a bot. However, since this password is actually going to be associated with your own user account and not a special bot account, including "bot" in the name is not that important. In the past when bot passwords were only associated with particular Wikimedia sites, it was more important to have mnemonic names to keep bots for different sites straight. However, since you can use the same bot across sites, this is no longer important. 

There are actually two reasons why you might want to use multiple, differently-named bot passwords. One is that different passwords can have different scope restrictions (see next step). So one password might only be able to perform certain actions, while another might be less restricted. The other reason is that if a particular password is being used "in production", you might want to have another one for testing. In the event you accidentally expose the credentials for the testing bot password, you could revoke those credentials without affecting the production bot password. However, for most purposes it would probably make sense to just have the "bot name" be the same as your username. 

After entering the name, click the Create button.

On the next page, select the rights that you want to grant to this bot password. I think the important ones are Edit existing pages, Create, edit, and move pages, and Delete pages, revisions, and log entries. However, just in case, I also selected High-volume editing, and View deleted files and pages as well. Leave the rest of the options at their defaults and click the Create button.

The resulting page will give you the username and passwords that you will need to write to the API. There are two variants: one where the bot name is appended to your username by @, and another where the username is used alone and the bot name is prepended to the password by @.  We will use the first variant (username@botname).

You need to create a plain text file that contains the username and password. To do this, you should use a text or code editor and NOT a word processor like Microsoft Word. Your computer should have a built-in text editor (TextEdit for Mac, Notepad for Windows). If you don't know what text and code editor are, see the first three videos on this page. If you are using a Mac, the second video explains how to ensure that TextEdit saves your file as plain text rather as rich text (which will cause an error in our situation) and to ensure that files are opened and closed using UTF-8 character encoding. 

Open a new document in the text editor. Create three lines of text similar to this:

endpointUrl=https://test.wikidata.org

username=User@bot

password=465jli90dslhgoiuhsaoi9s0sj5ki3lo

Be careful, since mistyping any character will cause VanderBot to not work. It's best to copy and paste rather than to try to type the credentials. (These are fake credentials, so you can't actually use them -- use your own username and password.) Do not leave a space between the equals sign (=) and the other characters. The first line specifies that we are going to use the test.wikidata.org API, so you can copy it exactly as written above. The username is the login name that includes the @ symbol (Baskaufs@BaskaufTestBot in the example above). The password is the password version that does not have the @ symbol in it. Double check that when you copied the username and password, you did not leave any characters off. Also, put the cursor at the end of each line and make sure that there are no trailing spaces after the text on the line. It does not matter whether the last line is followed with a newline (hard return) or not.

When you have entered the text, save the file as wikibase_credentials.txt in your home directory. In the next post, we will see how to use a different name or to change the location to somewhere else. Make sure that there is an underscore between "wikibase" and "credentials", not a dash or a space.  If you do not know what your home directory is, or where it is located on your computer, see the Special directories in Windows section of this page or the Special directories on Mac section of this page. In Finder on a Mac, you can select Home from the Go menu to get there.  In Windows File Explorer, start at the c: drive, then navigate to the Users folder. Your user folder will will be within the Users folder and have the same name as your username on the computer. 

Preparing the metadata description file and CSV headers

The VanderBot API upload script uses CSV files as its data source. Each row in the table represents data about an item. The columns of the table represent various aspects of metadata about the items, such as statements, qualifiers, and references. In order to transfer the data from the CSV to the Wikidata API, the columns of the CSV spreadsheet need to be mapped to the Wikibase data model (the model used by Wikidata). Since the Wikibase model can be represented as RDF, the W3C Generating RDF from Tabular Data on the Web Recommendation can be used to systematically map the CSV columns to the Wikibase model. The VanderBot script uses that mapping to determine how to construct the JSON required to transfer the CSV data to the Wikidata API.

Initially, I constructed the mapping file (known as the CSV's "metadata description file") by hand while referring to the W3C Recommendation and its examples. However, it is extremely difficult to build the mapping file by hand without making errors that were difficult to detect. Fortunately, my collaborator, Jessie Baskauf, created a web tool that allows a user to construct the mapping file using drop-downs that are organized in a structure that reflects that of the Wikidata graphical user interface. We will use that tool to create both the mapping file and the CSV header whose field names correspond to those used in the mapping file. 

The tool itself can be accessed online from this link. The Javascript that runs the tool runs entirely within the web browser, so it can be used offline by downloading the HTML, CSS, and Javascript files from GitHub into the same directory, then opening the HTML file in a browser. 

On the tool page, leave the Wikidata ID field at its default, qid. Use the Add label and Add description buttons to enter the names of each of those fields. I have been using the convention labelEn, labelDe, descriptionEs, etc. where I use lower camelCase and append the language code. However, you can use any name that makes sense to you. Select the appropriate language codes from the dropdown.

One thing to note is that there is no correspondence between property and item identifiers in test.wikidata.org (the test Wikidata implementation) and www.wikidata.org (the real Wikidata). So before we can add properties and item values of those properties, we need to look in the test Wikidata site to find properties and items that we want to play with. From the https://test.wikidata.org/ landing page, select Special pages from the left pane as you did before.  

Near the bottom of the Special pages page in the Wikibase section, click on the List of Properties link. In the real Wikidata instance, creation of properties is controlled by a community process. In the test Wikidata instance anyone can create, change, or delete properties. So although the properties used in this example may still be the same when you do this exercise, they also may have changed. Since we are practicing, you can substitute any other similar property for the ones shown in the examples. We want to chose a couple of properties that have different kinds of values in order to see how that affects the mapping file and CSV headers. So we are looking for a property that has an Item value and one that has a Point in time value. 

I picked P17 (country) and P18 (Date of birth) to use in the practice example. Clicking on the links shows that P17 has an Item value and P18 has a Point in time value.  

There are not necessarily items in the test Wikidata instance that correspond to those in the real Wikidata, so I searched for some countries to use as values of P17 in the test. I found Q346 (France) and Q53079 (Mexico). You can find your own, or create new items to use if you want.

I also wanted to select a property to use for a qualifier and another one to use for a reference. In the real Wikidata instance, many properties have constraints that indicate whether they are suitable to be used as properties in statements, qualifiers, or references. In the test Wikidata instance, most properties don't have any constraints. So I just picked a couple that seemed to make sense. I chose P87 (start date, having a Point in time value) as a qualifier property for P17 (country). (What does that mean? I don't know and it doesn't matter -- this is just a test.) I chose P93 (reference URL, having a URL value) as a reference property for P18 (Date of birth). Here is a summary of my chosen entities:

P17 country (Item value, used as a statement property)

P87 start date (Point in time value, used as a qualifier property for P17)

Q346 France (Item, used as a value for P17)

Q53079 Mexico (Item, used as a value for P17)

P18 Date of birth (Point in time value, used as a statement property)

P93 reference URL (URL value, used as a reference property)

Using the buttons and drop-downs, I selected the properties listed above on the web tool.

 The field names that you choose for the properties used in statements can be whatever you want. It is best to keep them short and do NOT use spaces. If you use multi-word names, I recommend lower camelCase, since dashes may cause problems later on and underscores are used by the tool to indicate the hierarchy of qualifier and reference properties.  The fields ending in _uuid and _hash are for statement and reference identifier fields and you should leave them at their defaults. When you create statement properties, by default the tool prefixes qualifier and reference properties their parent statement property names followed by an underscore. You can change these to shorten them if you want, but it's probably best to leave them at their defaults since when CSVs have many columns it becomes difficult to remember the structure without the prefixes. 

A statement can have multiple qualifier properties, but a statement can have both multiple references and multiple properties within a reference. For simplicity's sake, I recommend sticking with a single reference having one or more properties. 

Using the drop-downs, be sure to select a value type that is appropriate for the property. There is no quality control here at the point of the tool, but an error will be generated when writing to the API if the selected value type does not match with the value type specified for the property on the property page of the test Wikidata instance.

After you have entered all of the property information, scroll to the bottom and enter a filename in the box. Click on the Create CSV button. At this time, the script isn't sophisticated enough to actually generate the CSV file. (That is a possible future feature.) Rather it generates the header line for the CSV as raw text. Click the Copy to clipboard button, then open a new file using the same text editor that you used to generate the credentials file. 

Paste the copied text into the new file window.

Select Save or Save As... from an appropriate menu on your editor. The exact appearance of the dialog window will depend on your editor. The screenshot above is for TextEdit on a Mac. Be sure that you use exactly the same file name as you entered in the filename box in the web tool, with a .csv file extension. If your editor gives you a choice of text encoding, be sure to choose UTF-8. The directory into which you save the CSV file will be the one from which you will be running the upload script using the command line. So it is best to save it in some folder that is a subfolder of your home folder. Generally, Downloads, Documents, and Desktop are directly below the home folder, so if you use a subfolder of one of those folders, you should be able to navigate to that folder easily using the command line. 

Now click the Create JSON button. 

The metadata description JSON for the CSV file columns that you set up will be generated on the screen below the button. Click the Copy to clipboard button. Open a new file in the text editor that you used before and paste the copied text into it. Save the file using the name csv-metadata.json in the same directory where you saved the CSV file.

I like to paste the JSON into my favorite code editor (VS Code) because it will validate the JSON and display it using syntax highlighting, but that isn't really any better than using a vanilla text editor.

Preparing data to create new items

Now we will open the CSV file to add the data that we want to write to the test Wikidata instance. For this practice exercise, you can use Excel to edit the CSV if that is all you have, but if you are serious about using this system in the future, I highly recommend downloading and installing LibreOffice and using its Calc application to open and edit CSVs. I explain the reasons for this in the Skills required section at the top of this post. You can probably just double-click on the file in your file handling system (Finder on Mac or File Explorer in Windows), but if that doesn't work, open your spreadsheet application and open the file via Open in the File menu. 

When you open the CSV file, it should appear as a spreadsheet with the column names in the order that you created them with the web tool with empty rows below. You can now add data in the rows below the header. 

The screenshot of my example above is too small to easily see, but you can get a better look at it by going to this GitHub gist. You must use different labels and descriptions from the ones I used because if you use the same ones, the API will not allow them to be written (more details about this in the next blog post). As values for the country column, you can use the Q ID of any item. (I used Q53079 for Mexico). Notice that the birthDate column does not have a prefix, indicating that it is a statement property and not a child property of something else. The startDate column was prefixed with country_ by the web tool. That prefix and its position following the country column are clues that this column is a qualifier for the country column. The refUri column was prefixed by the web tool with birthDate_ and ref1_, indicating that it is a property of the first reference for the birthDate statement. Because the value type of the birthDate_ref1_refUrl column is URL, it must be a valid IRI starting with either http:// or https://.

The two date fields are a more complicated type. Dates, globe coordinates, and quantities are complex data types that cannot be represented by single fields. In the case of dates, they require one column field for the date string and another column filed to indicate the precision of the date (e.g. to year, to month, to century, etc.). There is a somewhat complicated system for representing dates in the Wikibase model (see this page for details). Fortunately, the VanderBot script will automatically convert dates that are formatted according to its conventions into the format required by the API. Those conventions are:

character pattern  example     precision

-----------------  -------     ---------

YYYY               1885        to year

YYYY-MM            2020-03     to month

YYYY-MM-DD         2001-09-11  to day

In the example spreadsheet, the country_startDate_val date value for the second item has precision to month, while the birthDate_val date values have precision to day. 

The dates should be placed in the corresponding column with name ending in _val. The script knows that it should make the conversion when the corresponding column with name ending in _prec is empty.  If the year has fewer than four digits, is BCE (a negative number), or has a precision lower than year (century, millennium, etc.), then a date string and precision integer properly formatted according to the Wikibase model must be provided explicitly. The script only provides minimal format checking (for the correct number of characters), so dates that are otherwise incorrectly formatted will result in an error that prevents the record from being written to the API. 

You should also notice that the example spreadsheet has a number of empty columns. These columns will contain identifiers for the various entities described by the data columns. For example, the qid column will contain the identifier for the item. The country_uuid and birthDate_uuid columns will contain the identifiers for the country and birthDate statements. The birthDate_ref1_hash will contain the identifier for the first reference for birthDate, which contains a reference URL. In all of these cases, the Wikidata API will assign those identifiers when the various entities are created and they will be recorded in the CSV file immediately after the item has been created. The VanderBot script uses the presence or absence of these identifiers to know whether the particular identified entity exists and therefore whether it needs to be written to the API or not. 

The situation with the two date columns whose names end in _nodeId is complicated. For technical reasons that I don't want to get into in this post, the node ID values are not assigned by the API, but rather are generated by VanderBot at the point of processing the dates. This is true for all of the properties with node value types (dates, globe coordinates, and quantities). All you need to know is that you should leave the columns ending in _nodeId blank and that the sets of three date-related columns that have the same first part (country_startDate_nodeId, country_startDate_val, and country_startDate_prec; birthDate_nodeId, birthDate_val, and birthDate_prec) represent complex values that can't be represented by a single column. 

Note that I did not fill in every cell in the table that could contain values. I did that because in a later step we will practice adding values to the item statements and references after the items have already been created. 

Be sure to close the CSV file before continuing to the next step. Failure to close the CSV will have different effects depending on the spreadsheet program you are using. I believe that both Excel and Open Office Calc place a lock on the file so that when the VanderBot script tries to write the API responses to the CSV file, it generates an error and crashes the script. Libre Office Calc will allow the changes to be written to the CSV file, but they will not show up unless the file is closed and re-opened. Libre Office Calc will warn you if you try to save an open file if it has been changed by the script while the file was open. In that case, close the file without saving and re-open it to see the changes.

Creating new items using the API

The last thing you need to have to actually write data to the API is the VanderBot Python script itself. Go to the code page on GitHub. Right-click on the Raw button in the upper right of the page. Select Save Link As..., navigate to the directory where you saved the csv-metadata.json file and the CSV file that you edited, and save the the vanderbot.py script there. 

If you have not previously installed the requests library, you may need to do that before you can run the script. If you have Anaconda installed on your computer, requests may already be installed. If you aren't sure, just try running the script as described below. If you get an error message saying that Python doesn't know about requests, then try entering:

pip install requests

If that doesn't work, try

pip3 install requests

If you use some non-standard package manager like brew or conda, install requests by whatever means you normally install packages.

Open the appropriate console program for your operating system (probably Terminal for Mac or Command Prompt for Windows). Use the cd command to navigate to the directory where you saved the file, then list the files to make sure you are in the right place and the files are all there (ls for Mac Linux or dir for Windows DOS). 

Depending on how you set up Python the command to run the script will probably either be

python vanderbot.py

or

python3 vanderbot.py

If things work correctly, the console should show the progress of writing to the API.

The first part of the output shows how VanderBot is interpreting the columns of the CSV based on the information from the csv-metadata.json column-mapping file. Then there is an indication that dates have been converted to the form required by the Wikibase model. As the script writes each row to the API, it displays the response of the API. The contents of the response don't matter as long as the end of the response contains "success".  After writing statements for each row, the script then checks whether there were any existing statements with added references. Since there were none, nothing was reported. Finally, there is a report of any errors that occurred that prevented particular rows from being written. Not every possible type of error is trapped and some will result in the script terminating before finishing all of the rows of the CSV. In that situation, the last response from the API may give clues about what went wrong. All information about identifiers received from the API prior to termination of the script should be saved in the CSV file, so once the error is fixed, you can just run the script again to try again to write the problematic line. 

If you re-open the CSV file, you should see results similar to this gist. All of the identifier columns in the table that are associated with value columns have now been filled in, indicating that those data now exist on Wikidata. Notice also that the dates have been converted into the more complicated format. 

If you search the test.wikidata.org site, you should see the record for the new item that you created. Because the birthDate_ref1_refUrl column had a value, a reference was created for the Date of birth statement. 

Because in the second row the country property column was followed by value columns for country_startDate that contained data, the country statement on the web page for the item displays a start date qualifier. The country_startDate_val column contained a value in the form 1986-02, so a precision of 10 (to month) was placed in the country_startDate_prec column of the table and therefore only the month is shown on the web page. In contrast, the birthDate_val column was given a value of 1982-02-03, so it was assigned a precision of 11 (to day) and the day is displayed on the web page for the date of birth statement. 

Editing existing items using the API

We can add information to the two new items that we just created by filling in parts of the CSV that we left blank before. In this gist, I added a country value (Q346, France) for item Q214621 and I added a reference value in the birthDate_ref1_refUrl for the birth date statement, which already existed, but did not previously have any references. After making sure that I closed the CSV file in the spreadsheet program, I ran the VanderBot script again.

In the first section of the output, the script detected that it needed to add a statement to an existing item (there was already a value in the qid column, but there was a value in the country column without a corresponding identifier value in the country_uuid column). It found no statements to add in the second row, so it did nothing. 

When it went through each row looking for new references for existing statements, it found one for the birth date reference URL column for the second record (there was an identifier in the birthDate_uuid column, but no identifier in the birthDate_ref1_hash column). It attempted to write the new data to the API, but an interesting thing happened. The server was too busy, and sent a message back to the script that it should wait a while and try again. In general, the script will keep trying with an increasing delay of up to 5 minutes, giving up after 10 tries. In this particular case, on the second retry the server was no longer too busy and the reference was successfully added. 

 When I reload the Juan Jose Garza page, I see that the date of birth statement now has a reference where there was none before.

Reloading Marie Gareau's page shows the new country statement. You may have noticed that I filled in the country value without giving any start date qualifier value for the statement in the country_startDate_val column.  The Wikidata Query Service treats qualifiers and references differently in that it assigns IRI identifiers to references, but does not assign them to qualifiers. Because VanderBot is designed to get information about specific metadata about items using the Query Service, it does not capture and store any identifier for qualifiers. Thus it is currently not possible to add a qualifier to a statement once the statement has been created. This behavior may be modified at some point in the future, but for now you should be aware of that limitation. 

Who's responsible for what just happened?

We can check the revision history of the Juan Jose Garza page to see how the edits we made were recorded. 

Notice that since the edits were made by a script using a bot password associated with my user account (Baskaufs), the edits were credited to me just as if I had made them by hand using the graphical interface. One difference is that the original item was created using a single API interaction. So even though it involved creating a label, a description, and two statements, it was recorded as a single edit instead of four. 

The benefit of editing as many parts of the item metadata at once as possible is that the interactions with the API are the rate-limiting factor when writing data to the API. VanderBot only makes one API call per row, even if the row contains many more columns than in this simple example. So it can make the edits much faster all at once than it could if it did them all separately.

Notice also that there is no record here that the VanderBot script was used. It identified itself to the API through its User-Agent HTTP header when it communicated with the server, and it was a "good citizen" by waiting to retry when the server reported that it was lagged. But there is no record of that interaction in the revision history.

To see the final state of the CSV file after all of the uploads shown here, see this gist.

What should you do next?

While you have the spreadsheet and JSON metadata description file set up, you should do a lot more experimenting. There is really nothing that you can "break" on either test.wikidata.org or VanderBot. In particular, you should try doing the following things to see what happens. Some of them are "wrong" things that produce bad results or aren't allowed by the API, while others are harmless or fine. If the script doesn't crash, reload the page or search for the new item in the graphical interface to see what happened. 

1. Create another row in the spreadsheet where the label and description are the same as an existing item. What happens? (When writing to the "real" Wikidata, there is code in VanderBot that tries to prevent this, but it doesn't work with the test instance.)
2. What happens if either the label or the description (but not both) is the same as an existing item? Can you create an item that is missing either a label or a description?
3. What happens if you delete the uuid identifier for a statement property and run the script again?
4. What happens if you delete the uuid identifier for a statement property, change the value, and run the script again?
5. What happens if you delete a reference hash identifier and replace the reference value with a different one?
6. What happens if you leave off part of a date (e.g. 1997-9-23 with no leading zero for the month)? You may need to change the cell format to "text" in order to be able to make this mistake in your spreadsheet program. 
7. What happens if you have the correct number of characters in a date, but the date is malformed (e.g. 199x-09-23)?
8. What happens if you change a value, but do not delete the corresponding identifier associated with the value?

In the next blog post, we will switch to writing to the "real" Wikidata, where we would prefer not to make these kinds of mistakes.

Answers are below.

.

,

,

,

,

,

,

,

,

,

,

,

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Answers:

1. The API responds with an error message and the script ends prematurely. 
2. A new item will be created with the same label (or description). You can also create items lacking either a label or a description (but not both).
3. A duplicate statement will be created. This is a bad practice.
4. A second value will be added for the property. This is perfectly fine as long as the second value is correct information.
5. The new reference gets added as a second reference for the same statement. This is perfectly fine.
6. The script does nothing and reports that there was an incorrectly formatted date in that row.
7. The script tries to write the value, but the API returns an error message saying that the date format is bad. The script then stops running.
8. Nothing happens. The script doesn't look at the value if it already has an identifier associated with it.

TDWG gets 5 Stars!

Photo from W3C https://www.w3.org/DesignIssues/LinkedData.html

TDWG IRIs are dereferenceable with content negotiation!

Yesterday was a happy day for me because after several years of work, the switch was flipped and all of the IRIs minted by TDWG under the rs.tdwg.org subdomain became dereferenceable with content negotiation in most cases.  For those readers who aren't hard-core Linked Open Data (LOD) buffs, I'll explain what that means.

An internationalized resource identifier (IRI; superset of uniform resource identifiers, URIs) is a globally unique identifier that generally looks like the well known URL. It usually starts with http:// or https://,  which implies that something will happen if you put it in a web browser. That "something" is dereferencing - the browser uses the IRI to try to retrieve a document from a remote server and if successful, a web page shows up in the browser. Because a browser's job is to retrieve web pages, when it dereferences an IRI, it asks for a particular "content type" (text/html) indicating that it wants an HTML web page.

But there are other kinds of software designed to retrieve documents that are readable by machines rather than by humans. When those applications dereference an IRI, they ask for other content types (like text/turtle or application/rdf+xml) that can be interpreted as structured data and be integrated with data from other sources. The same IRI can be used to retrieve different documents that provide the same information in different formats depending on the content type that is requested. The process of determining what kind of document to return to the requesting application is called content negotiation.

In the past, the behavior of TDWG IRIs were inconsistent. Some IRIs like those of Darwin Core terms would retrieve a web page in a browser and provide machine-readable RDF/XML when requested. Other IRIs like those of Audubon Core terms would retrieve a web page, but no machine-readable formats. Obsolete IRIs like those of old versions of Darwin Core and the defunct TDWG ontology did nothing at all. Then there were many TDWG resources, such as old standards documents, that didn't even have IRIs.

In an earlier blog post, I described the IRI patterns that I established in order to be able to denote all of the kinds of TDWG standards components that were described in the TDWG Standards Documentation Specification. Those patterns made it possible to use IRIs to refer to things like vocabularies, term lists, and documents in a consistent way. Just creating the IRI patterns and using them to assign IRIs to vocabularies and documents provided a way to uniquely identify those resources, but did not create the "magic" of actually making it possible to use those IRIs to retrieve information. That's what happened yesterday.

What happens when the IRIs are dereferenced?

The action that takes place when an rs.tdwg.org IRI is dereferenced depends on the category of the resource and the content type that's requested.  There are four categories of behavior that vary primarily on how they deliver human-readable content.

1. "Living" TDWG vocabulary terms. When a term from one of the actively maintained TDWG vocabularies (currently Darwin Core and Audubon Core) is dereferenced, the browser is redirected to the most helpful reference document for that vocabulary (the Quick Reference Guide for Darwin Core and the Term List document for Audubon Core). You can try this with dwc:recordedBy, http://rs.tdwg.org/dwc/terms/recordedBy and ac:caption, http://rs.tdwg.org/ac/terms/caption.

2. Obsolete TDWG vocabulary terms, vocabularies, term lists, and special categories of resources. When terms in these categories are dereferenced, a generic web page is generated by a script that provides vanilla information about the term. The same is true for some special categories like Executive Committee decisions.  Try it with an obsolete term http://rs.tdwg.org/dwc/curatorial/Disposition, a decision http://rs.tdwg.org/decisions/decision-2011-10-16_6 and a term list http://rs.tdwg.org/ac/xmp/.

3. TDWG-maintained standards documents. The maintenance of TDWG standards documents is idiosyncratic and their location depends on where their maintainers happened to have stashed them. The URLs used to retrieve the documents might change if they are put into different places or if their format changes (e.g. changed from PDF to Markdown).  To provide a stable way to denote those documents, the IRIs minted in rs.tdwg.org subdomain redirect to whatever current URL delivers that particular document. If the document moves or the access URL changes for some reason, the stable IRI will redirect to the new access URL. Try it with the TDWG Vocabulary Maintenance Specification http://rs.tdwg.org/vms/doc/specification/, the Audubon Core Structure document http://rs.tdwg.org/ac/doc/structure/,  and the TAPIR Protocol Specification http://rs.tdwg.org/tapir/doc/specification/.

4. Non-TDWG-maintained standards documents. A lot of the old TDWG standards were not actually published by TDWG, and their maintenance is carried out by organizations whose websites are not under TDWG control. So we will just try to keep the TDWG-issued document IRIs pointing at whatever the access URL is currently for the document. Examples: Economic Botany Data Collection Standard specification http://rs.tdwg.org/ebdc/doc/specification/, Taxonomic Literature : A Selective Guide to Botanical Publications and Collections with Dates, Commentaries and Types (Second edition, vol. 1) http://rs.tdwg.org/tl/doc/v1/, and Index Herbariorum http://rs.tdwg.org/ih/doc/book/.

Machine-readable metadata
For these categories, the machine readable metadata is delivered in the same way: generated by script from the data in the rs.tdwg.org Github repository. To access the content through content negotiation, you can dereference any of the IRIs above using software like Postman that will allow you to specify an Accept header for the machine-readable content type that you want (text/turtle or application/rdf+xml). To access the machine-readable documents directly, drop any trailing slashes and append .ttl or .rdf to access RDF/Turtle or RDF/XML respectively. Examples: http://rs.tdwg.org/dwc/terms/recordedBy.ttl, http://rs.tdwg.org/dwc/terms/recordedBy.rdf, and http://rs.tdwg.org/tl/doc/v1.ttl.

There are also a number of legacy XML schemas that are still being retrieved by some applications and they are made available by just redirecting from the rs.tdwg.org IRI to wherever the schema lives. Example: http://rs.tdwg.org/dwc/text/tdwg_dwc_text.xsd .

How this happens
The script that handles all of these many variations of IRIs is written in XQuery (a functional programming language designed to process XML) and runs on a BaseX server instance. A second XQuery script generates the vanilla HTML web pages that are generated from the same data as the machine-readable metadata. I've written more extensively about this approach in an earlier post, so I won't say more about it here.

There was a lot of concern about maintaining a server that is based on a programming language that is not well-known among IT professionals. So it's likely that in the future the XQuery-based system will be replaced by something else. I'd like to use something based on the W3C Generating RDF from Tabular Data on the Web Recommendation, since the source data live as CSV files on Github. But for now, this is what we have.

5 Stars???

The title of this post says that TDWG now gets 5 stars. What does that mean? In 2010, Tim Berners-Lee promoted a 5 star system to rate the extent to which data sources are freely available in machine-readable form. The TDWG standards metadata have been available online in structured form under an open license (stars 1 through 3), but failed to achieve 5 stars since standards-based machine readable data (RDF) couldn't be acquired by dereferencing the IRIs (star 4) and the resources weren't linked to others in the machine-readable metadata (star 5). As of yesterday, we can tick off stars 4 and 5, so the TDWG standards metadata are now fully compliant with Linked Open Data best practices. Congratulations TDWG!

Special thanks to Matt Blissett of GBIF for working out the technical details of setting up the server and production protocol and to Tim Robertson of GBIF for his support in getting this done. Thanks also to Cliff Anderson and the XQuery Working Group of the Vanderbilt University Heard Library for introducing me to BaseX server.