Enriching Collections Data with Linked Art
\n", "\n", "The Linked Art reconciliation exemplar provides a step through the process of reconciling geographical place names that occur in the title of artworks by the artist, John Ruskin, to enrich collections data with an additional representation of the geographical coordinates of the place depicted in the artworks. \n", "\n", "From Wikipedia:\n", "\n", "\"John Ruskin (8 February 1819 – 20 January 1900) was an English writer, philosopher, art critic and polymath of the Victorian era. He wrote on subjects as varied as geology, architecture, myth, ornithology, literature, education, botany and political economy.\"\n", "\n", "John Ruskin travelled extensively in Europe and was a prolific artist, creating drawings of paintings whose titles often included place names for the locations depicted. \n", "\n", "#### Artwork Title contains Place Name\n", "The title of the artworks has been recorded in the title field in many of the collection data records , and this has been used as the basis for the reconciliation process shown here.\n", "\n", "#### OpenRefine Tool to Reconcile Data\n", "The place names are reconciled with the Getty Thesaurus of Geographic Names (TGN), using the Open Refine tool.\n", "\n", "#### The Getty Thesaurus of Geographic Names (TGN)\n", "Reconciliation with the Getty Thesaurus of Geographic Names (TGN) has allowed additional information to be associated with the artwork: \n", "- an authoritative global identifier for the geographical location depicted \n", "- geographical coordinates\n", "\n", "#### Input Data Files\n", "\n", "The input files are Linked Art files created with the `01-06-Transform-John-Ruskin` Jupyter notebook.\n", "\n", "\n", "### Further Reading\n", "\n", "- The Getty Thesaurus of Geographic Names® Online (TGN) http://www.getty.edu/research/tools/vocabularies/tgn\n", "- John Ruskin Wikipedia entry https://en.wikipedia.org/wiki/John_Ruskin\n", "\n" ] }, { "cell_type": "markdown", "id": "348d9614", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Reconciliation Process\n", "\n", "1. Create CSV file from Linked Art JSON-LD\n", "2. Identify place name in title\n", "3. Use OpenRefine to reconcile place names\n", "4. Define geolocation representation in Linked Art\n", "5. Add place name and coordinates into Linked Art JSON-LD files" ] }, { "cell_type": "markdown", "id": "aec2095c", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 1. Create CSV file from Linked Art JSON-LD\n", "\n", "To reconcile the place names in the artwork titles\n", "- create a CSV file from the JSON-LD Linked Art files\n", "- CSV contains `id` and `_label` properties\n", "\n", "The script gets a list of all files in a selected directory using `os.listdir()` and iterates over them.\n", "\n", "- `json.load` is used to deserialize the Linked Art JSONLD file to a Python dictionary object. \n", " - json.loads uses the following conversion table https://docs.python.org/3/library/json.html#json-to-py-table \n", "\n", "Finally, the script uses `csv.DictWriter` \n", "- to create an object that maps the Python dictionary onto output rows. \n", "- `Dictwriter.writeheader()` writes a row with the field names (as specified in the constructor) to the writer’s file object. \n", "- `Dictwriter.writerows()` writes all elements in rows to the writer’s file object.\n", "\n", "\n", "#### Further Reading\n", "- os Python library https://docs.python.org/3/library/os.html\n", "- os.listdir() tutorial https://www.tutorialspoint.com/python/os_listdir.htm\n", "- json Python library https://docs.python.org/3/library/json.html\n", "- csv Python library https://docs.python.org/3/library/csv.html" ] }, { "cell_type": "code", "execution_count": 2, "id": "81e0aaea", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# import relevant Python libraries\n", "\n", "try:\n", " import os\n", "except:\n", " %pip install os\n", " import os\n", "\n", "try:\n", " import json\n", "except:\n", " %pip install json\n", " import json \n", " \n", "import csv\n", "\n", "# list holding a\n", "artworkCSV = []\n", "\n", "# Linked Art JSON-LD file location\n", "artworkFileDir = \"./data/ruskin/output/json/\"\n", "artworkFileList =os.listdir(artworkFileDir)\n", "\n", "# iterate over Linked Art JSON-LD files\n", "for artworkFile in artworkFileList:\n", " # read file and append to \n", " with open( artworkFileDir + artworkFile) as artworkFileContents: \n", " \n", " # create json object `artwork` from file\n", " artworkObjJSON = json.load(artworkFileContents)\n", " \n", " # check for \"_label\" property \n", " if \"_label\" not in artworkObjJSON:\n", " continue\n", " \n", " # append artwork properties to artwork JSON object\n", " artworkCSV.append( { \n", " \"id\": artworkObjJSON[\"id\"], \n", " \"place\" : artworkObjJSON[\"_label\"], \n", " \"place_modified\": \" \", \n", " \"coords\": \" \"\n", " })\n", "\n", "# end loop\n", " \n", " \n", "# create CSV file\n", "artworkCsvFile = \"./data/ruskin/ruskin-places.csv\" # file location\n", "\n", "with open(artworkCsvFile, 'w') as f: \n", " # write column headings\n", " w = csv.DictWriter(f, [\"id\",\"place\",\"place_modified\",\"coords\"])\n", " w.writeheader()\n", " # write rows with artwork properties\n", " w.writerows(artworkCSV)" ] }, { "cell_type": "markdown", "id": "5825b0e1", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Result - CSV File with Place Names\n", "\n", "The contents of the resulting CSV file are shown below for illustration.\n", "\n", "The CSV file is read into a `pandas` dataframe. \n", "
`Pandas` is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.\n", " \n", " \n", "A `pandas dataframe` is a pandas data structure containing \n", "
two-dimensional, size-mutable, potentially heterogeneous tabular data.\n", "\n", "The pandas dataframe allows easy manipulation of two-dimensional tabular data. \n", "\n", "The `IPython` library is also used to display the contents of the CSV file \n", " \n", "#### Further Reading \n", "- pandas https://pandas.pydata.org/\n", "- pandas dataframe https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html\n", "- IPython use in Jupyter notebooks https://coderzcolumn.com/tutorials/python/how-to-display-contents-of-different-types-in-jupyter-notebook-lab" ] }, { "cell_type": "code", "execution_count": 3, "id": "85365962", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" id \\\n",
"0 https://collections.ashmolean.org/collection/1... \n",
"1 https://collections.ashmolean.org/collection/1... \n",
"2 https://www.harvardartmuseums.org/collections/... \n",
"3 https://collections.ashmolean.org/collection/1... \n",
"4 https://collections.ashmolean.org/collection/1... \n",
".. ... \n",
"274 https://collections.ashmolean.org/collection/1... \n",
"275 https://www.harvardartmuseums.org/collections/... \n",
"276 http://www.rijksmuseum.nl/nl/collectie/nl-RP-T... \n",
"277 https://collections.ashmolean.org/collection/2... \n",
"278 https://collections.ashmolean.org/collection/1... \n",
"\n",
" place place_modified coords \n",
"0 Engraving of Ruskin's Drawing of the Petal Vau... \n",
"1 Enlarged Study of a Prawn's Rostrum \n",
"2 Study of a Venetian Capital \n",
"3 Autumnal Cloud filling the Valley of Geneva, t... \n",
"4 Axmouth Landslip from Dolands Farm \n",
".. ... ... ... \n",
"274 The Head of a Kite, from Life \n",
"275 Part of a Sketch of the Northwest Porch of St.... \n",
"276 Gezicht op S. Anastasia te Verona, over de Adige \n",
"277 Architectural detail: stone bracket \n",
"278 Study of the Marble Inlaying on the Front of t... \n",
"\n",
"[279 rows x 4 columns]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"try:\n",
" import pandas as pd\n",
"except:\n",
" %pip install pandas\n",
" import pandas as pd\n",
"\n",
"try:\n",
" import IPython\n",
"except:\n",
" %pip install IPython\n",
" import IPython \n",
" \n",
"from IPython.display import display, HTML, Javascript\n",
"\n",
" # CSV file location\n",
"artworkCsvFile = \"./data/ruskin/ruskin-places.csv\"\n",
"\n",
"# read CSV file into pandas dataframe \n",
"dataFrame = pd.read_csv(artworkCsvFile,low_memory=False)\n",
"\n",
"# define how many columns and rows to display == all\n",
"pd.options.display.max_columns = len(dataFrame.columns)\n",
"#pd.options.display.max_rows = len(dataFrame.index)\n",
"\n",
"# use IPython display to display the contents of CSV file\n",
"display(dataFrame)"
]
},
{
"cell_type": "markdown",
"id": "931ce294",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 2. Identify Place Names in Title\n",
"\n",
"Next, get place name from artwork title\n",
"- extract possible place names from the artwork title field, to help with the reconciliation process. \n",
"- a list of possible place names is used to help identify place names in the field. \n",
"- add extracted place names to `place_modified` column \n",
"- update CSV file\n",
"\n",
"A list of place names `placeNames` is created to help with extracting place names from the artwork title. This was produced further to a review of the values in the place column.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "bdcc6a8f",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"artworkCsvFile = \"./data/ruskin/ruskin-places.csv\" # file location\n",
"\n",
"\n",
"# read CSV file into pandas dataframe\n",
"dataFrame = pd.read_csv(artworkCsvFile,low_memory=False)\n",
"\n",
"\n",
"# A list of place names `placeNames` is created to help with extracting place names from the artwork title.\n",
"placeNames = [\n",
"\"Florence\",\"Bologna\",\"Lucca\",\"Alps\",\"Oxford\",\"Rome\", \"Venice\",\"Fribourg\",\"Neuchâtel\",\"Sestri\",\"Visp\",\"Chamonix\",\n",
"\"Abbeville\",\"Schaffhausen\",\"Verona\",\"Vorarlberg\",\"Baden\",\"Schaffhausen\",\"Faido\",\"Normandy\",\"Genève\",\"Geneva\",\n",
"\"Gloucester\",\"Basel\",\"Luzern\",\"Padua\",\"Habsburg\",\"Rhine\",\"Zug\",\"Aix-la-Chapelle\",\"Siena\",\"Mont Blanc\",\"Lago di Como\",\n",
"\"Bellinzona\",\"Lake of Lecco\"\n",
"]\n",
"\n",
"places = {\"Venezia\":[\"Venice\",\"Venetian\",\"St Mark\",\"St. Mark\"]}\n",
"\n",
"\n",
"# iterate over dataframe\n",
"for index,row in dataFrame.iterrows():\n",
" \n",
" # iterate over place names\n",
" # check if any place name in placesNames is present in row\n",
" for place in placeNames:\n",
" # if place name found, add to place_modified column\n",
" if place in row[\"place\"]:\n",
" dataFrame.at[index,\"place_modified\"] = place\n",
" \n",
" # iterate over place names for Venice\n",
" for place in places[\"Venezia\"]:\n",
" # if place found add `Venezia` to place_modified column\n",
" if place in row[\"place\"]:\n",
" dataFrame.at[index,\"place_modified\"] = \"Venezia\"\n",
"\n",
" \n",
"# remove records where place_modified is blank\n",
"dataFrame = dataFrame[dataFrame.place_modified != \" \"]\n",
"dataFrame.to_csv(artworkCsvFile, index=False) \n"
]
},
{
"cell_type": "markdown",
"id": "d0814adc",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Result - CSV File containing Place Name for use in OpenRefine\n",
"\n",
"The result of this process is a CSV file with the column `place_modified` containing a place name string that will be used for reconciliation in the OpenRefine tool.\n",
"\n",
"Records where a place name has not be identified have been removed from the CSV file."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "5dfbd5dd",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | id | \n", "place | \n", "place_modified | \n", "coords | \n", "
|---|---|---|---|---|
| 0 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Engraving of Ruskin's Drawing of the Petal Vau... | \n", "\n", " | \n", " |
| 1 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Enlarged Study of a Prawn's Rostrum | \n", "\n", " | \n", " |
| 2 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Study of a Venetian Capital | \n", "\n", " | \n", " |
| 3 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Autumnal Cloud filling the Valley of Geneva, t... | \n", "\n", " | \n", " |
| 4 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Axmouth Landslip from Dolands Farm | \n", "\n", " | \n", " |
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 274 | \n", "https://collections.ashmolean.org/collection/1... | \n", "The Head of a Kite, from Life | \n", "\n", " | \n", " |
| 275 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Part of a Sketch of the Northwest Porch of St.... | \n", "\n", " | \n", " |
| 276 | \n", "http://www.rijksmuseum.nl/nl/collectie/nl-RP-T... | \n", "Gezicht op S. Anastasia te Verona, over de Adige | \n", "\n", " | \n", " |
| 277 | \n", "https://collections.ashmolean.org/collection/2... | \n", "Architectural detail: stone bracket | \n", "\n", " | \n", " |
| 278 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Study of the Marble Inlaying on the Front of t... | \n", "\n", " | \n", " |
279 rows × 4 columns
\n", "\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" id \\\n",
"0 https://www.harvardartmuseums.org/collections/... \n",
"1 https://collections.ashmolean.org/collection/1... \n",
"2 https://www.harvardartmuseums.org/collections/... \n",
"3 https://www.harvardartmuseums.org/collections/... \n",
"4 https://www.tate.org.uk/art/artworks/13033 \n",
".. ... \n",
"102 https://collections.ashmolean.org/collection/1... \n",
"103 https://www.nga.gov/collection/72870 \n",
"104 https://www.harvardartmuseums.org/collections/... \n",
"105 http://www.rijksmuseum.nl/nl/collectie/nl-RP-T... \n",
"106 https://collections.ashmolean.org/collection/1... \n",
"\n",
" place place_modified coords \n",
"0 Study of a Venetian Capital Venezia \n",
"1 Autumnal Cloud filling the Valley of Geneva, t... Geneva \n",
"2 Tom Tower, Christ Church, Oxford Oxford \n",
"3 Study of a Venetian Capital Venezia \n",
"4 View of Bologna Bologna \n",
".. ... ... ... \n",
"102 Sketch of the Oak Spray in Mantegna's Fresco o... Padua \n",
"103 The Garden of San Miniato near Florence Florence \n",
"104 Part of a Sketch of the Northwest Porch of St.... Venezia \n",
"105 Gezicht op S. Anastasia te Verona, over de Adige Verona \n",
"106 Study of the Marble Inlaying on the Front of t... Venezia \n",
"\n",
"[107 rows x 4 columns]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"artworkCsvFile = \"./data/ruskin/ruskin-places.csv\" # file location\n",
"\n",
"dataFrame = pd.read_csv(artworkCsvFile,low_memory=False)\n",
"\n",
"# display table for illustration\n",
"display(dataFrame)"
]
},
{
"cell_type": "markdown",
"id": "a10d8ccc",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 3. Use OpenRefine to Reconcile Place Names\n",
"\n",
"Next\n",
"- use OpenRefine to match the values in the place_modified field, with place names in the name authority, The Getty Thesaurus of Geographic Names® Online (TGN).\n",
"\n",
"`OpenRefine` is a tool for working with messy data, and includes support for reconciliation with external data such as name authorities.\n",
"\n",
"`The Getty Thesaurus of Geographic Names® Online (TGN)` is one of several Getty Vocabularies that provide a structured resource that can be used to improve access to information about art, architecture, and material culture. From the website:\n",
"\n",
"| \n", " | id | \n", "place | \n", "place_modified | \n", "coords | \n", "
|---|---|---|---|---|
| 0 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Study of a Venetian Capital | \n", "Venezia | \n", "\n", " |
| 1 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Autumnal Cloud filling the Valley of Geneva, t... | \n", "Geneva | \n", "\n", " |
| 2 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Tom Tower, Christ Church, Oxford | \n", "Oxford | \n", "\n", " |
| 3 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Study of a Venetian Capital | \n", "Venezia | \n", "\n", " |
| 4 | \n", "https://www.tate.org.uk/art/artworks/13033 | \n", "View of Bologna | \n", "Bologna | \n", "\n", " |
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 102 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Sketch of the Oak Spray in Mantegna's Fresco o... | \n", "Padua | \n", "\n", " |
| 103 | \n", "https://www.nga.gov/collection/72870 | \n", "The Garden of San Miniato near Florence | \n", "Florence | \n", "\n", " |
| 104 | \n", "https://www.harvardartmuseums.org/collections/... | \n", "Part of a Sketch of the Northwest Porch of St.... | \n", "Venezia | \n", "\n", " |
| 105 | \n", "http://www.rijksmuseum.nl/nl/collectie/nl-RP-T... | \n", "Gezicht op S. Anastasia te Verona, over de Adige | \n", "Verona | \n", "\n", " |
| 106 | \n", "https://collections.ashmolean.org/collection/1... | \n", "Study of the Marble Inlaying on the Front of t... | \n", "Venezia | \n", "\n", " |
107 rows × 4 columns
\n", "Through rich metadata and links, the Getty Vocabularies provide powerful conduits for knowledge creation, research, and discovery for digital art history and related disciplines.\n", "\n", "TGN is a thesaurus. TGN is not a geographic information system (GIS), although it may be linked to existing major, general-purpose, geographic databases and maps. While most records in TGN include coordinates, these coordinates are approximate and are intended for reference (\"finding purposes\") only (as is true of coordinates in most atlases and other resources, including NGA (formerly NIMA) databases).\n", "\n", "\n", "\n", "\n", "#### Further Reading\n", "\n", "- OpenRefine https://openrefine.org/\n", "- The Getty Thesaurus of Geographic Names® Online (TGN) http://www.getty.edu/research/tools/vocabularies/tgn" ] }, { "cell_type": "code", "execution_count": 6, "id": "fce5be0d", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
Open Refine Website
" ], "text/plain": [ "Open Refine Website
\"))\n", "\n", "display(IFrame('https://openrefine.org/documentation.html', '100%', '600px'))" ] }, { "cell_type": "markdown", "id": "1ad62f9a", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Method\n", " \n", "The method used to reconcile the place names:\n", "- Download and install OpenRefine https://openrefine.org/download.html\n", "- Open OpenRefine and create a project\n", "- Upload the places CSV file\n", "- Reconcile place names in `place_modified`\n", "- Choose the TGN service to reconcile data with\n", "- Review Reconciliation Search Results\n", "- Add a Column Containing Entity TGN Identifiers further to Reconciliation Process\n", "- Manual Reconciliation\n", "\n", "#### Further Reading\n", "- OpenRefine https://openrefine.org\n" ] }, { "cell_type": "markdown", "id": "0b6973cb", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Download and Install OpenRefine\n", "\n", "- Download OpenRefine at https://openrefine.org/download.html\n", "- Installation instructions at https://docs.openrefine.org/manual/installing" ] }, { "cell_type": "markdown", "id": "f8f3537a", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Open OpenRefine and Create a Project\n", "\n", "The following video illustrates how to create a project in OpenRefine using a CSV file on the local drive." ] }, { "cell_type": "code", "execution_count": 7, "id": "0b70b3c9", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "Video - OpenRefine - Create Project
" ], "text/plain": [ "Video - OpenRefine - Create Project
\"))\n", "\n", "\n", "# video of project creation in OpenRefine\n", "\n", "YouTubeVideo('h1aLc5uvdck', width=1024, height=576)" ] }, { "cell_type": "markdown", "id": "955da136", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Reconcile Place Names in `place_modified`\n", "\n", "- Right-click on `place_modified` column header\n", "- Select `Start reconciling`\n", "\n", "![](\"docs/media/img/openrefine2.png\"/)
"
]
},
{
"cell_type": "markdown",
"id": "4a068754",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Choose the TGN Service to Reconcile Data With\n",
"\n",
"- Choose the Getty Voculabary Reconciliation Service that includes the TGN \n",
"\n",
"The video shows the following process:\n",
"- select a column to reconcile \n",
"- select a service to reconcile with\n",
"- review options\n",
"- start reconciliation\n",
"\n",
"#### Further Reading \n",
"- Reconciliation services known to Wikidata - https://reconciliation-api.github.io/testbench/\n",
"- TGN https://www.getty.edu/research/tools/vocabularies/tgn/"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "d7e1102b",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
"Video - OpenRefine - Start Reconciliation
" ], "text/plain": [ "Video - OpenRefine - Start Reconciliation
\"))\n", "\n", "\n", "YouTubeVideo('Zm0woMobjpI', width=1024, height=576)\n", "\n" ] }, { "cell_type": "markdown", "id": "8cc9e363", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Review Reconciliation Search Results\n", "\n", "Once the reconciliation process has completed it is necessary to review the results. Using the TGN it is noticeable that there are many places in the United States of America that have the same names as locations in Italy. \n", "\n", "A review of each match is necessary. Once a correct match has been identified, this match can be applied to all cells with the same place name. " ] }, { "cell_type": "code", "execution_count": 9, "id": "ab8d959a", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "Video - OpenRefine - Review Reconciliation Results
" ], "text/plain": [ "Video - OpenRefine - Review Reconciliation Results
\"))\n", "\n", "YouTubeVideo('pT0b0vsPRJ0', width=1024, height=576)\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "bb324586", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Add a Column Containing Entity TGN Identifiers further to Reconciliation Process\n", "\n", "- Create new column to hold the TGN identifiers\n", "\n", "![](\"docs/media/img/tgncol.png\"/)
"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "81b01078",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
"Video - OpenRefine - Add Entity Identifier Column
" ], "text/plain": [ "Video - OpenRefine - Add Entity Identifier Column
\"))\n", "\n", "YouTubeVideo('PNPhs_7MQ6o', width=1024, height=576)\n" ] }, { "cell_type": "markdown", "id": "2bc1b729", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Manual Reconciliation\n", "\n", "Some additional manual reconciliation was required using the TGN search form.\n", " \n", "![](\"docs/media/img/tgn.png\"/)
\n",
"\n",
"#### Further Reading\n",
"- TGN search form http://www.getty.edu/research/tools/vocabularies/tgn\n",
" "
]
},
{
"cell_type": "markdown",
"id": "cf10d529",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Result - CSV file with TGN Identifiers\n",
"\n",
"The result of the reconciliation process is a new column with TGN name authority identifiers for place names identified in the artwork title. \n",
"\n",
"A CSV file is created with the following steps:\n",
"- export CSV file from OpenRefine\n",
"- save as [data/ruskin/ruskin-places-rec.csv](data/ruskin/ruskin-places-rec.csv) `"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b8e984f7",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"from IPython.display import YouTubeVideo, HTML\n",
"\n",
"display(HTML(\"Video - OpenRefine - Export Results as CSV
\"))\n", "\n", "YouTubeVideo('0tBjqr5AEmA', width=1024, height=576)" ] }, { "cell_type": "markdown", "id": "bbd90fbb", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "#### Remove rows without TGN identifier\n", "\n", "- A final step removes rows that do not have a TGN identifier.\n", "- The resulting dataset is shown in tabular format below." ] }, { "cell_type": "code", "execution_count": null, "id": "419bea9c", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "reconciledRuskinPlaces = \"data/ruskin/ruskin-places-rec.csv\" \n", "\n", "# read file into pandas dataframe\n", "df = pd.read_csv(reconciledRuskinPlaces,low_memory=False)\n", "\n", "# remove rows that have an empty tgn field value\n", "df = df[df.tgn != \"\"]\n", "\n", "# write dataframe to file \n", "df.to_csv(reconciledRuskinPlaces, index=False) \n", "\n", "# for illustration display dataframe\n", "\n", "display(df)" ] }, { "cell_type": "markdown", "id": "25a0c234", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Add Geographical Coordinates to CSV file\n", "The next step is to associate geographical coordinates with the Linked Art artwork representations, using the TGN identifiers to query the TGN web service and return geographical coordinates.\n", "\n", "Steps:\n", "- request JSON file from http://vocab.getty.edu/tgn/ using TGN identifier\n", "- extract geocoordinates from response\n", "- add geocoordinates to CSV file" ] }, { "cell_type": "code", "execution_count": null, "id": "cce77b47", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "import requests\n", "\n", "latprop = \"http://www.w3.org/2003/01/geo/wgs84_pos#lat\"\n", "lngprop = \"http://www.w3.org/2003/01/geo/wgs84_pos#long\"\n", "\n", "\n", "display(HTML(\"Geographical coordinates retrieved from TGN web service
\")) \n", "\n", "reconciledRuskinPlaces = \"./data/ruskin/ruskin-places-rec.csv\" \n", "reconciledRuskinPlacesCoords = \"./data/ruskin/ruskin-places-rec-coords.csv\" \n", "\n", "# create dataframe from CSV file containing reconciled data including TGN identifiers\n", "dataFrameRuskinPlaces = pd.read_csv(reconciledRuskinPlaces,low_memory=False)\n", "\n", "# set type for column 'coords' as string in dataframe\n", "dataFrameRuskinPlaces['coords'] = dataFrameRuskinPlaces['coords'].astype(str)\n", "\n", "\n", "display(HTML(\"Retrieving geocoordinates from vocab.getty.edu TGN API. Please wait for task to complete.
\"))\n", "\n", "# create dataframe to hold geographical coordinates with columns tng and latlng\n", "dataFrameGeo = pd.DataFrame({}, columns=['tgn', 'latlng'])\n", "\n", "# iterate through reconciled data containing place names and TGN identifiers\n", "for identifier_tgn in dataFrameRuskinPlaces['tgn'].unique():\n", " \n", " # print . to indicate progress\n", " print(\".\", end='')\n", " \n", " #create query string for web service - get tgn id using .split()\n", " query = \"http://vocab.getty.edu/tgn/\" + identifier_tgn.split(\"tgn/\",1)[1] +\"-place.json\"\n", " \n", " # use requests.get() to query TGN web service using TGN identifier to return geo coordinates \n", " resultsJSON = requests.get(query).json()\n", " \n", " # get lat lng from web service query results\n", " for record in resultsJSON:\n", " lat = resultsJSON[record][latprop][0][\"value\"]\n", " lng = resultsJSON[record][lngprop][0][\"value\"]\n", " \n", " # create string for lat lng\n", " latlng = str(lat) + \",\" + str(lng)\n", " \n", " \n", " # append TGN identifier and lat lng to the dataFrameGeo \n", " dataFrameGeo = dataFrameGeo.append(\n", " {\n", " 'tgn': identifier_tgn, \n", " 'latlng': latlng\n", " }, \n", " ignore_index=True)\n", " \n", "\n", "# for illustration display dataFrameGeo with addition of geo coords\n", "display(dataFrameGeo)" ] }, { "cell_type": "markdown", "id": "15c96dc7", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Update CSV File with Geographical Coordinates \n", "\n", "The following code \n", "- merges `dataFrameRuskinPlaces` with `dataFrameGeo` containing the geocoordinates.\n", "- removes `coords` from `dataFrameRuskinPlaces` \n", "- renames `latlng` to `coords` in `dataFrameRuskinPlaces` \n", "- writes `dataFrameRuskinPlaces` to a CSV file" ] }, { "cell_type": "code", "execution_count": null, "id": "107a351d", "metadata": {}, "outputs": [], "source": [ "reconciledRuskinPlacesCoords = \"./data/ruskin/ruskin-places-rec-coords.csv\" \n", "\n", "\n", "# merge dataframe with coords with dataframe from csv\n", "dataFrameRuskinPlaces = dataFrameRuskinPlaces.merge(dataFrameGeo, on='tgn') \n", "\n", "# drop column coords\n", "dataFrameRuskinPlaces = dataFrameRuskinPlaces.drop('coords', 1) # drop column coords\n", "\n", "# rename column latlng to coords\n", "dataFrameRuskinPlaces.rename(columns={'latlng': 'coords'}, inplace=True) # rename column tgn to coords\n", "\n", "# drop rows that have na value in coords column\n", "dataFrameRuskinPlaces.dropna(subset=['coords']) \n", "\n", "# write to CSV file\n", "dataFrameRuskinPlaces.to_csv(reconciledRuskinPlacesCoords, index=False)\n", "\n", "display(HTML(\"CSV file with Geographical Coordinates
\"))\n", "# display dataframe\n", "display(dataFrameRuskinPlaces)\n" ] }, { "cell_type": "markdown", "id": "1ce006eb", "metadata": {}, "source": [ "## 4. Define Geolocation Representation in Linked Art\n", "\n", "The next step is to define a representation in Linked Art for geographical coordinates of place depicted in artwork. The relevant parts of the Linked Art model are: \n", " - Depiction\n", " - Geospatial approximation\n", " - Depiction of place with approximate location\n", "\n", "\n", "### Linked Art Data Model - Depiction\n", "\n", "From https://linked.art/model/object/aboutness/#depiction\n", "\n", "Many sorts of artwork depict things that can be pointed out in the artwork. These could be identifiable entities, such as a known Person or Object with a name or identifier, or unidentifiable (perhaps fictional) instances of a class of entity, such as a depiction of a battle but not any particular battle. For example a portrait depicts the person sitting for it, or a sketch of a generic landscape depicts a place even if it's not a particular, known location. The depiction pattern describes what is in the artwork's image.\n", "\n", "
This is modeled using the `represents` property on the VisualItem, which refers to the entity that is being depicted.\n", "\n", "\n", "The following representation will be used for place depicted in Ruskin's artworks:\n", "\n", "\n", "`{\n", " \"@context\": \"https://linked.art/ns/v1/linked-art.json\",\n", " \"id\": \"https://linked.art/example/object/34\",\n", " \"type\": \"HumanMadeObject\",\n", " \"_label\": \"artwork title including place name\",\n", " \"shows\": [\n", " {\n", " \"type\": \"VisualItem\",\n", " \"represents\": [\n", " {\n", " \"type\": \"Place\",\n", " \"_label\": \"place name\"\n", " }\n", " ]\n", " }\n", " ]}`\n" ] }, { "cell_type": "markdown", "id": "1b781b3c", "metadata": {}, "source": [ "### Linked Art Data Model - Geospatial Approximation\n", "\n", "The Linked Art data model describes how to represent geospatial approximation.\n", "\n", "From https://linked.art/model/place/#geospatial-approximation\n", "\n", "
All recorded locations are approximate to some degree. It may be desirable to capture this approximation separately from the actual place, especially when that approximation is very uncertain. Especially if the place is the exact location of several events, and perhaps an address or other information is known, but not the exact geospatial coordinates.\n", " \n", "\n", "
Secondly, as a place is defined by exactly one definition, but there might be multiple approximations such as a polygon as well as the central point, the real place that an activity occured at can be related to multiple approximate places to capture these different approximations.\n", "\n", "\n", "Example Linked Art representation of geospatial approximation:\n", "\n", "`{\n", " \"@context\": \"https://linked.art/ns/v1/linked-art.json\",\n", " \"id\": \"https://linked.art/example/place/4\",\n", " \"type\": \"Place\",\n", " \"_label\": \"True Auction House Location\",\n", " \"approximated_by\": [\n", " {\n", " \"type\": \"Place\",\n", " \"_label\": \"Auction House Location Approximation\",\n", " \"defined_by\": \"POINT(-0.0032937526703165 51.515107154846)\"\n", " }\n", " ]\n", "}`\n", "\n", "\n", "### Linked Art Data Model - Depiction of Place with Approximate Location\n", "\n", "Relating the Linked Art model for geospatial approximation to the depiction of places in Ruskin's works, the following representation has been created:\n", "\n", "\n", "`{\n", " \"@context\": \"https://linked.art/ns/v1/linked-art.json\",\n", " \"id\": \"https://linked.art/example/object/34\",\n", " \"type\": \"HumanMadeObject\",\n", " \"_label\": \"artwork title including place name\",\n", " \"shows\": [\n", " {\n", " \"type\": \"VisualItem\",\n", " \"represents\": [\n", " {\n", " \"type\": \"Place\",\n", " \"_label\": \"Lucca\",\n", " \"approximated_by\": [\n", " {\n", " \"type\": \"Place\",\n", " \"_label\": \"Lucca - Location Approximation\",\n", " \"defined_by\": \"POINT(-0.0032937526703165 51.515107154846)\"\n", " }\n", " ]\n", " }\n", " ]\n", " }\n", " ]}`\n", "\n", "\n", "\n", "\n", "#### Further reading\n", "\n", "- Depiction https://linked.art/model/object/aboutness/#depiction\n", "\n", "- Geospatial approximation https://linked.art/model/place/#geospatial-approximation" ] }, { "cell_type": "markdown", "id": "c292ee0c", "metadata": {}, "source": [ "
Visualisation - Geographical Coordinates of Place Depicted in Artwork
\n", "Below is a visualisation of the Linked Art JSON-LD representation of geographical coordinates of a place depicted in an artwork. \n", " \n", "Further information\n", "- explore the representation by clicking on nodes\n", "- SVG representation \n", "- uses \n", " - D3.js\n", " - is a modified version of code available in the JSON-LD Playground codebase\n", "\n", "\n", "#### Further Reading\n", "\n", "- d3.js https://d3js.org/\n", "- jsonld-vis https://github.com/science-periodicals/jsonld-vis\n", "- jsonld playground https://json-ld.org/playground and https://json-ld.org/playground/jsonld-vis.js \n" ] }, { "cell_type": "markdown", "id": "687ef899", "metadata": {}, "source": [ "
" ] }, { "cell_type": "code", "execution_count": null, "id": "d1ccb958", "metadata": {}, "outputs": [], "source": [ "from IPython.core.display import Javascript\n", "\n", "code2 = \"var file = './data/examples/geolocation.json';\"\\\n", " \"var selector = '#vis';\" \\\n", " \"visjsonld(file, selector); \" \n", "\n", "with open('./src/js/visld.js', 'r') as _jscript:\n", " code = _jscript.read() + code2\n", "\n", "Javascript(code)" ] }, { "cell_type": "markdown", "id": "1dfa884a", "metadata": {}, "source": [ "## 5. Add Place Name and Coordinates into Linked Art JSON-LD Files\n", "\n", "The final step is to add place names and geocoordinates to the original Linked Art files. \n", "\n", "The updated Linked Art files, including the geocoordinates, will later be used in a storymap visualisation of the artworks of John Ruskin, mapping the artworks to the locations that they depict, using the geocoordinates.\n", "\n", "The `cromulent` Python library is used to create the JSON-LD representation." ] }, { "cell_type": "code", "execution_count": null, "id": "d37dd072", "metadata": {}, "outputs": [], "source": [ "try:\n", " import cromulent \n", "except:\n", " %pip install cromulent\n", " import cromulent\n", " \n", "from cromulent.model import factory\n", "\n", "from cromulent.model import factory, Actor, Production, BeginningOfExistence, EndOfExistence, TimeSpan, Place\n", "from cromulent.model import InformationObject, Phase, VisualItem \n", "from cromulent.vocab import Painting, Drawing,Miniature,add_art_setter, PrimaryName, Name, CollectionSet, instances, Sculpture \n", "from cromulent.vocab import aat_culture_mapping, AccessionNumber, Height, Width, SupportPart, Gallery, MuseumPlace \n", "from cromulent.vocab import BottomPart, Description, RightsStatement, MuseumOrg, Purchase\n", "from cromulent.vocab import Furniture, Mosaic, Photograph, Coin, Vessel, Graphic, Enamel, Embroidery, PhotographPrint\n", "from cromulent.vocab import PhotographAlbum, PhotographBook, PhotographColor, PhotographBW, Negative, Map, Clothing, Furniture\n", "from cromulent.vocab import Sample, Architecture, Armor, Book, DecArts, Implement, Jewelry, Manuscript, SiteInstallation, Text, Print\n", "from cromulent.vocab import TimeBasedMedia, Page, Folio, Folder, Box, Envelope, Binder, Case, FlatfileCabinet\n", "from cromulent.vocab import HumanMadeObject,Tapestry,LocalNumber\n", "from cromulent.vocab import Type,Set\n", "from cromulent.vocab import TimeSpan, Actor, Group, Acquisition, Place\n", "from cromulent.vocab import Production, TimeSpan, Actor\n", "from cromulent.vocab import LinguisticObject,DigitalObject, DigitalService\n", "from cromulent import reader\n", "\n", "try:\n", " import pandas as pd\n", "except:\n", " %pip install pandas\n", " import pandas as pd\n", " \n", "try:\n", " import os\n", "except:\n", " %pip install os\n", " import os\n", " \n", "try:\n", " import json\n", "except:\n", " %pip install json\n", " import json \n", " \n", "artwork = {}\n", "cnt=1\n", "\n", "# directory that will contain updated Ruskin artwork representations including geo coords\n", "storyvisdir = \"data/ruskin/storyvis/json\"\n", "\n", "# file containing reconciled data with coordinates\n", "filecoord = \"./data/ruskin/ruskin-places-rec-coords.csv\" \n", "# open file containing reconciled data with geo coordinates\n", "dataframeGeo = pd.read_csv(filecoord,low_memory=False)\n", "\n", "\n", "# directory containing Rusking artworks represented in Linked Art JSON-LD\n", "ruskindir = \"data/ruskin/output/json\"\n", "file_list=os.listdir(ruskindir)\n", "\n", "# for each linked art json file\n", "for file in file_list:\n", " # open file\n", " with open( ruskindir + \"/\" + file) as json_file:\n", " \n", " # get json object from file object with json.load() https://www.geeksforgeeks.org/json-load-in-python/\n", " artwork = json.load(json_file)\n", " \n", " # if id field is in the id field of data file containing geographical coordinates, add update the file\n", " if artwork[\"id\"] in dataframeGeo[\"id\"].tolist():\n", " \n", " display(HTML(\"\" + artwork[\"_label\"] + \"
\"))\n", " # get rows in dataframeGeo where id == artwork id from JSON-LD file\n", " # Access a group of rows and columns by label(s) or a boolean array\n", " # https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html\n", " rows = dataframeGeo.loc[dataframeGeo['id'] == artwork[\"id\"]]\n", " \n", " print(\"Matching row in geographical coordinates file for artwork\")\n", " display(rows)\n", " \n", " # get first row https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html\n", " row=rows.iloc[0]\n", " \n", " \n", " \n", " # get place name and coords from geocoords file \n", " placeName = row[\"place\"]\n", " coords = row[\"coords\"]\n", " # replace comma in coords with space\n", " coords = coords.replace(\",\", \" \")\n", " \n", " # increment counter\n", " cnt = cnt+1\n", " \n", " # use cromulent to create Linked Art representation of place depicted\n", " # https://github.com/thegetty/crom\n", " \n", " approx_place = Place()\n", " approx_place._label = placeName\n", " approx_place.defined_by = \"POINT(\" + coords + \")\"\n", " \n", " place = Place()\n", " place._label = placeName\n", " place.approximated_by = approx_place\n", " \n", " visualItem = VisualItem()\n", " visualItem.represents = place\n", " \n", " # append new representation to artwork json object\n", " artwork[\"shows\"] = factory.toJSON(visualItem)\n", " \n", " print(\"Geographical coordinates representation to be added:\")\n", " print(json.dumps(factory.toJSON(visualItem), indent=2))\n", " \n", " # open output file \n", " text_file = open(storyvisdir + \"/\" + str(cnt) + \".json\", \"wt\")\n", " \n", " # write to file and close\n", " n = text_file.write(json.dumps(artwork,indent=2))\n", " text_file.close()\n", " print(\"File updated\" )\n", "\n", "HTML(\"Files updated
\")" ] }, { "cell_type": "markdown", "id": "3b0c6139", "metadata": {}, "source": [ "## Example Linked Art JSON-LD including Geographical Identifier and Coordinates \n", "\n", "The following is an example JSON-LD Linked Art representation, updated to include geographical coordinates.\n", " \n", "### Image Title: Study of the Marble Inlaying on the Front of the Casa Loredan, Venice\n", "\n", "![](\"https://upload.wikimedia.org/wikipedia/commons/thumb/5/5f/Study_of_the_Marble_Inlaying_on_the_Front_of_the_Casa_Loredan.jpg/508px-Study_of_the_Marble_Inlaying_on_the_Front_of_the_Casa_Loredan.jpg\"/)
\n",
"\n",
"\n",
"\n",
"Ashmolean Museum artwork page\n",
"\n",
"\n",
"### JSON-LD Representation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "09c99833",
"metadata": {},
"outputs": [],
"source": [
"print(json.dumps(artwork,indent=2))"
]
},
{
"cell_type": "markdown",
"id": "9b5abb3e",
"metadata": {},
"source": [
"