Evolving the Web Through Open Data

Harry Halpin, <hhalpin -(a)- ibiblio.org>

RDF

Microsoft Research, June 2007

The Web 2.0

The Web as a Totalizing System

Increasingly all data is on the Web: a universal information space

TimBL

The "Web 2.0" is the use of the Web as a platform for applications, a universal computation space

What is the break point for the full realization of the Web as a platform for applications?

Parallel Webs?

Semantic Web research has till recently built a parallel Web instead of aligning the current hypertext Web with the Semantic Web.

Never underestimate the amount of information in HTML

or in databases behind firewalls...

Liberate the data and put it in a common universal format for accessing by our Web applications?

Microformats and the Semantic Web?

How to Get the Data!

Opposing Methodologies

  1. Authors make their data explicit
  2. Applications scrape the implicit data

Peter Norvig: We deal with millions of Web masters who can't configure a server, can't write HTML. It's hard for them to go to the next step.

Or is it?

Microformat Logo

Microformats: The lower-case semantic web

Shockingly, a large number of web sites are using microformats:

  1. Upcoming.org
  2. LinkedIn
  3. Yedda
  4. Yahoo! Local
  5. Yahoo! Tech Reviews

There's even a Dreamweaver plug-in to help authors!

Examples

hCal

<span class="vevent">
<p><abbr class="dtstart" title="2006-12-05">December 5-</abbr> 
<abbr class="dtend" title="2006-12-07">7th</abbr>
</b> At <b><span class="summary"&rt;XML 2006</span></b> 
(<span class="location">Boston, MA USA</span>) 
for a presentation on "Social Semantic Mashups".</span>

iCal Extraction

XFN

<li><a href="http://www.w3.org/People/Connolly/" rel="colleague met">Dan Connolly</a></li>
<li> <a href="http://seanmcgrath.blogspot.com/" rel="colleague met">Sean McGrath</a></li>
<li><a href="http://www.jclark.com/" rel="colleague">James Clark</a></li>

Microformats Unleashed

Microformats are centralized data formats for different types of data, often (nearly) isomorphic to already widely adopted non-Web standards:

The Straw that broke the Camel's Back

breaking straw

Too many services replicate the same sort of data....what if you have a Friendster, a Myspace, and a Twtter account?

Photo by Jon Hicks

The Limits of Microformats

The main problem with microformats is that they put your data into HTML, but you have no standard way to get the data out.

Except really hairy XSLT style-sheets such as Suda's x2v...

A further problem is that they cannot be validated easily. You can mix hCard and hCal and there's no way to guarantee you will interpret it correctly.

Domain-specific: You can not make a microformat for just anything!

Demo of Microformat Browsing using Tails

The Semantic Web

The Semantic Web is a solution in search of a problem: Data Integration

URIs can be used as a universal (globally-unique) identifier, like a foreign key in a database

RDF is a natural format for combining up any sort of data with any other sort of data!, since the URI preserves the identity of the data.

Due to the open world assumption and being based on links, it is an extremely flexible data format that should be able to capture data from relational databases, microformats, and more...

Micromodels for Microformats

For each popular microformat vocabulary we are in the midst of manufacturing an isomorphic RDF model.

Format

Domain

Model

Mapping

Status

DC

Document/Content Metadata

Dublin Core

dc-extract.xsl

usable (Sep 2005 testing status)

XFN

Social Networks

FOAF *

grokXFN.xsl

usable

hCard

People

FOAF

?

?

hCalendar

Calendars and Events

RDF Calendar

glean-hcal.xsl

usable

hReview

Opinions, Ratings and Reviews

RDF Review

hreview2rdfxml.xsl

in progress

relLicense

Licenses

Creative Commons

http://www.w3.org/2003/12/rdf-in-xhtml-xslts/grokCC.xsl

?

relTag

Tags, Keywords, Categories

Tag Ontology

?

Putting the Pedal to the Metal with GRDDL

GRDDL (Gleaning Resource Descriptions from Dialects of Languages) is a simple and low-cost way to boostrap RDF out of XML and in particular XHTML data by explicitly labelling transformations from RDF to XML.

GRDDL consists of three components:
  1. GRDDL-aware agent: a GRDDL-aware agent is a software agent able to identify the GRDDL transformations and run them to extract RDF.
  2. Source Document: an XML document which references at least one GRDDL transformation for a GRDDL-aware agent to use to extract RDF from it.
  3. GRDDL Transformation: a GRDDL transformation is an algorithm which, when applied to a compliant source document, allows a GRDDL-aware agent to extract RDF from this document.

Look at Dan Connolly's Homepage with Piggybank

Adding GRDDL to HTML


<html xmlns="http://www.w3.org/1999/xhtml">
<head profile="http://www.w3.org/2003/g/data-view">
<link rel="stylesheet" type="text/css" href="homepage.css"/>
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/"/>
<link rel="transformation" href="http://www.w3.org/2002/12/cal/glean-hcal.xsl"/>
<meta name="ICBM" content="055.94270, -003.18599"/>
<meta name="DC.title" content="Harry Halpin's Homepage"/>
<meta name="DC.creator" content="Harry Halpin"/>
<meta name="DC.language" content="en"/>

<title>Harry Halpin's Homepage</title>
</head>
...
<body>
</body>
</html>

GRDDL in Namespace Documents

No Transformation Links - just go to the namespace document!

In RDF, OWL, RDF Schema:

<rdf:RDF
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:dataview="http://www.w3.org/2003/g/data-view#">
 <rdf:Description rdf:about="http://www.w3.org/2004/01/rdxh/p3q-ns-example">
   <dataview:namespaceTransformation
       rdf:resource="http://www.w3.org/2004/01/rdxh/grokP3Q.xsl"/>
 </rdf:Description>
</rdf:RDF>

In XML Schema:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns="http:.../Order-1.0"
            targetNamespace="http:.../Order-1.0"
            version="1.0"
            ...
            xmlns:data-view="http://www.w3.org/2003/g/data-view#"
            data-view:transformation="http://www.w3.org/2003/g/embeddedRDF.xsl" >
    <xsd:element name="Order" type="OrderType">
    <xsd:annotation 
      <xsd:documentation>This element is the root element.</xsd:documentation>
    </xsd:annotation>
                 ...
  <xsd:annotation>
    <xsd:appinfo>
    <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
	<rdf:Description rdf:about="http://www.w3.org/2003/g/po-ex">
	  <data-view:namespaceTransformation
	      rdf:resource="grokPO.xsl" />
	</rdf:Description>
      </rdf:RDF>
    </xsd:appinfo>
  </xsd:annotation>
...

GRDDL-powered Excel Mashups

The vast majority of the organizational information is in Excel Spreadsheets

  • Data cells are the attendance indicators
  • Header cells are the cells containing names and dates
  • Attendance Excel

    See Spreadsheet example in GRDDL Primer

    Excel XML

    <?mso-application progid="Excel.Sheet"?>
    <Workbook
      xmlns="urn:schemas-microsoft-com:office:spreadsheet"
      xmlns:o="urn:schemas-microsoft-com:office:office"
      xmlns:x="urn:schemas-microsoft-com:office:excel"
      xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882"
      xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet"
      xmlns:html="http://www.w3.org/TR/REC-html40"
      xmlns:grddl="http://www.w3.org/2003/g/data-view#"
      grddl:transformation="xcel-mf2rdf.xsl">
     <!-- ... -->
     <CustomDocumentProperties xmlns="urn:schemas-microsoft-com:office:office">
      <profile dt:dt="string">http://www.mnot.net/2005/08/xcel-mf</profile>
      <namespace dt:dt="string">ahttp://example.org/attendance/</namespace>
     </CustomDocumentProperties>
     <!-- ... -->
     <Worksheet ss:Name="Sheet1">
      <Table>
       <!-- ... -->
       <Row ss:Index="3">
        <!-- ... -->
        <Cell ss:StyleID="s26"><Data ss:Type="String">2006-04</Data><NamedCell
          ss:Name="Header"/><NamedCell ss:Name="date"/></Cell>
        <!-- ... -->
       </Row>
       <!-- ... -->
       <Row>
        <Cell ss:StyleID="s25"><Data ss:Type="String">Robin</Data><NamedCell
          ss:Name="name"/><NamedCell ss:Name="Header"/></Cell>
        <!-- ... -->
        <Cell><Data ss:Type="String">n</Data><NamedCell ss:Name="present"/><NamedCell
          ss:Name="Data"/></Cell>
        <!-- ... -->
       </Row>
       <!-- ... -->
      </Table>
      <!-- ... -->
     </Worksheet>
    </Workbook>
    

    Spreadsheets in RDF

    <?xml version="1.0" encoding="UTF-8"?>
    <rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns="ahttp://example.org/attendance/">
     <!-- ... -->
     <rdf:Description>
      <name>Robin</name>
      <date>2006-04</date>
      <present>n</present>
     </rdf:Description>
     <!-- ... -->
    
    </rdf:RDF>
    

    Now merge with other spreadsheets!

    <?xml version="1.0" encoding="UTF-8"?>
    <rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns="ahttp://example.org/events/">
     <rdf:Description>
      <date>2006-03</date>
      <label>choose new meeting location</label>
     </rdf:Description>
    
    </rdf:RDF>
    

    Querying Merged Spreadsheet

    PREFIX att: <ahttp://example.org/attendance/>
    PREFIX ev: <ahttp://example.org/events/>
    
    SELECT ?name WHERE
    {
      ?event ev:label "choose new meeting location" .
      ?event ev:date ?date .
      ?attendance att:date ?date .
      ?attendance att:name ?name .
      ?attendance att:present "y" .
    }
    

    Which would give the following results:

    name
    Jane
    David

    Open Data

    The only way to make progress against the centralization of the web in services like Google is to make data open and decentralized.

    Will companies put their sensitive data in Google applications?

    But they want social and collaborative aspects of Google Web applications?

    The Semantic Web can deliver social and collaborative results in a decentralized manner

    Microsoft can deliver to customers by providing GRDDLs for their XML formats, preventing migration to centralized servers.

    Research: Social Semantic Search

    Can we build off of social search engines?

    How about microformats, as in GRDDL and Technorati's Microformat Search?

    Semantic Wikipedia

    Can we provide trusted search, through Semantic Rules and policy?

    Can we use RDF as a transport layer amongst decentralized search engines?

    In concert with Wikia search? Semantic Wikipedia?

    Why shouldn't the Semantic Web tackle the world's biggest problem: Search on the Web, in a decentralized manner?

    Wikia Search

    Wikia Search

    Research: Beyond Javascript

    How can we exploit natural occurring "semantic" HTML markup with a minimum of NLP?

    There is no reason why Javascript has to be the dominant language of the Web 2.0 - procedural with attendant security and modelling problems

    XSLT is functional, but cannot actually invert from RDF back to XSLT, control the entire XML Processing Model (DTD default attributes, XInclude), lacks interoperation with Web Services and native use of Semantic Web logic.

    The Curry-Howard Isomorphism which states that for every logical form there is a corresponding programming language

    One can convert between functional programming languages and logic, such that we can conceive of a program as steps in a logical proof. - could this be useful to the SemWeb?