Report Bug

1. Create a new blank tool

Create New Dataset

In this guide, we’re going to make a simple tool that finds images on Wikipedia, based on a given search term.

After registering and logging in, click the “Create a new dataset” button on your homepage.

Importer Chooser

You’ll be shown all the tools you can use to populate your new dataset.

We’re going to “Code a dataset” from scratch. Click it.

Rename your dataset

Before we start, let’s give this dataset a more memorable name so we can find it later. Use the dropdown dataset menu to rename it to something like “Search Wikipedia”.

2. SSH into the box

Each QuickCode dataset (and each of the tools attached to it, like the table view, and the spreadsheet downloader) are stored in what we call separate “boxes”.

You can find out more about boxes, tools and datasets in our developer documentation.

A box is a shell account on the web. You can access it via SSH or SFTP, and you can use special API endpoints to communicate with the rest of QuickCode.

SSH into the tool

Click the dropdown menu next to the dark grey “Code your own tool” icon.

Select “SSH in” from the menu, and follow the instructions to set up your SSH keys.

If you’re on Windows, you’ll want to install and run Git Bash.

Top tip! This guide assumes you’re happy editing files on a remote server using a command-line editor like Vim, Emacs, Nano or Joe. If you’d prefer to edit locally, using your normal text editor, check out swbox.

3. Make a user interface

Default dataset index.html

Your tool’s user interface lives in the ~/tool/http/ directory.

Use an editor like Nano or Vim to edit the index.html file.

$ nano ~/tool/http/index.html # good for beginners
$ vim ~/tool/http/index.html # more powerful for coders

Edit the index.html file so that it looks like this:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Wikipedia Image Search</title>
    <meta http-equiv="cleartype" content="on">
    <link rel="stylesheet" href="//scraperwiki.com/vendor/style/bootstrap.min.css">
    <link rel="stylesheet" href="//scraperwiki.com/style/scraperwiki.css">
    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.8/jquery.min.js"></script>
    <script src="//scraperwiki.com/vendor/js/bootstrap.min.js"></script>
    <script src="//scraperwiki.com/js/scraperwiki.js"></script>
    <script>
      $(function() {
          var execSuccess = function(execOutput) {
            var datasetUrl = "/dataset/" + scraperwiki.box
            scraperwiki.tool.redirect(datasetUrl)
          }
          $('#submit').on('click', function() {
            $(this).addClass('loading').html('Searching…')
            term = $('#q').val()
            scraperwiki.exec("tool/search.py '" + term + "'", execSuccess)
          })
      })
    </script>
  </head>
  <body style="text-align: center">
    <h1>Search Wikipedia Images</h1>
    <div class="input-append">
      <input type="text" name="q" size="80" id="q" placeholder="Search term, eg: cats">
      <button class="btn btn-primary" type="submit" id="submit">Search</button>
    </div>
  </body>
</html>

This displays a form with a box for the user to type a search term, and then has some JavaScript that passes the search term to a server-side script which does all the heavy lifting.

Extra marks if you can spot the intentional command injection vulnerability!

Short on time? Just run this to download the finished index.html file from our Github repo:

curl -o ~/tool/http/index.html "https://raw.github.com/scraperwiki/qs-wikipedia-images-tool/master/http/index.html"

4. Write your server-side code

The JavaScript, above, sends the user’s search term as a command line argument to a script at tool/search.py. We need to make that file.

Create a new file called search.py in the ~/tool/ directory, and fill it with this code:

#!/usr/bin/env python
import sys
import requests
import scraperwiki

URL = "http://en.wikipedia.org/w/api.php?action=query&list=allimages&aiprop=url|user|timestamp|size|mime&ailimit=100&format=json&aifrom=%s"

data = requests.get(URL % sys.argv[1]).json()
images = data['query']['allimages']
scraperwiki.sqlite.save(['name'], images, 'images')

Short on time? Just run this to download the finished search.py file from our Github repo:

curl -o ~/tool/search.py "https://raw.github.com/scraperwiki/qs-wikipedia-images-tool/master/search.py"

Remember to make your search.py file executable so that the JavaScript frontend can run it.

chmod +x ~/tool/search.py

5. Test it out

Completed Wikipedia tool

Reload your web browser to see your tool’s new look!

Type in a search term and see whether it works.

Once the search.py script has finished running, the JavaScript execSuccess callback function automatically redirects you to a table view of your data. Pretty!

Or try visualising your data using the Summarise Automatically tool. You’ll find it under “More tools…” in the Tools menu in the top right.

6. Submit it to the Tool Shop

If you've got this far, and you've made an interesting tool, please contact us to add it so everyone else can use it!