Now that we know the basics of Nutch, we can dive into our use case. We write scripts that do two things:
- Ingestion of the various configurations
- Execute and control crawls
This post will tackle ingesting the configs. I will specifically be using Python for the examples in this post, but the principles should apply to any language.
In our project, we had 50+ sites we wanted to crawl, all with different configuration needs. We organized these configurations into a nice JSON api that we ingest. In our examples, we will be using Python’s Requests API to get the JSON. We’ll also need a way to create a unique UUID for each configuration, so we’ll use Python’s UUID module. You can use the package installer pip to get them:
$ pip install requests
$ pip install uuidWe’re going to use a class to handle all of the processing for injection. We’ll create a file for this, call it configInjector.py. The beginning of the file should look something like this:
import os
import uuid
import requests
from shutil import copy2
class ConfigInjector(object):
def __init__(self):
passWe’re importing os and copy2 so we can create, edit, and copy files that we need. Next, we’re going to want to get the config itself, as well as an ID from the configuration node itself. We’ll make a new file for this, call it inject.py. This will be the script we actually run from cron for injection. It begins something like this:
import urllib2
import json
import argparse
import configInjector
parser = argparse.ArgumentParser(description="Ingests configs.")
parser.add_argument("confugUrl", help="URL of the JSON config endpoint.")
args = parser.parse_args()For our imports, we’ll use requests and UUID like earlier as well as urllib2 to download our remote JSON and argparse to give our script an argument for where to download JSON. We’re also importing our own configInjector class file.
The argparse module allows us to pass command line arguments to the Python script. In the code above, we instantiate the argument parser, add our argument (configUrl), and set the results of the argument to args. This allows us to pass in a url for the location of our JSON endpoint.
Now that we have the foundation set up let’s get the data. We’ll use urllib2 to grab the JSON and json.load() add it to a variable:
response = urllib2.urlopen(args.confugUrl)
configs = json.load(response)We’ll then loop through it and call our class for each config in the JSON:
for configId in configs:
configInjector.ConfigInjector(configId, configs[configId])Now that we are getting the configs, let’s fill out our class and process them. We’ll use the __init__ constructor to do the majority of our data transformations. The two major things we want to do is process and inject Nutch config settings and create regex-urlfilters.txt for each config.
First, we’ll do our transformations. We want to get our config options in order to plug into Nutch, so we’ll just set them as variables in the class:
class ConfigInjector(object, configId, config):
def __init__(self):self.config = config
self.configId = configId
# Config transformations
self.configTitle = self.config["configTitle"]
self.allowExternalDomains = self.config["allowExternalDomains"]
self.uuid = str(uuid.uuid3(uuid.NAMESPACE_DNS, str(self.configId)))We’re setting three things in this example: a config title and UUID for reference and a configuration state for the Nutch config db.ignore.external.links. We’re using the static configId to generate the UUID so that the same UUID is always used by each individual configuration.
Next, we’ll need to create some files for our seed urls and match patterns. We’re going to create two files, seed-XXXXXX.txt and regex-urlfilters-XXXXXX.txt, where XXXXXX is the configId. For the seed files, we’ll create our own directory (called seeds), but for the regex files, we must store them in $NUTCH_HOME/runtime/local/conf in order for Nutch to find them (this is due to Nutch’s configuration of the Java CLASSPATH). First, we’ll set the filenames based upon configId (this goes in the __init__ function):
self.regexFileName = 'regex-urlfilter-' + self.nodeId + '.txt'
self.seedFileName = 'seed-' + self.nodeId + '.txt'We also want to call the functions we are about to write here, so that when we call the class, we immediately run all the necessary functions to inject the config (again, in the __init__ function):
# Run processes
self._makeConfigDirectories()
self._configureSeedUrlFile()
self._copyRegexUrlfilter()
self._configureRegexUrlfilter()
self._prepInjection()Next, we’ll setup the directories (the underscore at the beginning of the function name just tells python not to load this function when being imported because it will only be used internally):
def _makeConfigDirectories(self):
if not os.path.exists('/path/to/nutch/runtime/local/conf/'):
os.makedirs('/path/to/nutch/runtime/local/conf/')
if not os.path.exists('/path/to/nutch/seeds/'):
os.makedirs('/path/to/nutch/seeds/')This simply checks to make sure the directories are there and makes them if they aren’t. Next, we’ll create the seed files:
def _configureSeedUrlFile(self):
furl = open('/path/to/nutch/seeds/' + self.seedFileName, "w")
for url in self.config["seedUrls"]:
furl.write(url + "\n")Basically, we are opening a file (or creating one if it doesn’t exist--this is how “w” functions) and writing each url from the JSON config to each line. We must end each url with a newline (\n) for Nutch to understand the file.
Now we’ll make the regex file. We’ll do it in two steps so that we can take advantage of what Nutch has pre-built. We’re going to copy Nutch’s built-in regex-urlfilters.txt so that we can use all of its defaults and add any defaults we would like to all configs. Before we do that, we have an important edit to make to regex-urlfilters.txt: remove the .+ from the end of the file in both /path/to/nutch/conf and /path/to/nutch/runtime/local/conf. We’ll add it back in the file ourselves, but if we leave it there, the filters won’t work at all because Nutch uses the first match when determining whether to fetch a url, and .+ means “match any”. For our use, we’re going to add this back on the end of the file after we write our regex to it.
We’ll copy regex-urlfilters.txt in this function:
def _copyRegexUrlfilter(self):
frurl = '/path/to/nutch/conf/regex-urlfilter.txt'
fwurl = '/path/to/nutch/runtime/local/conf/' + self.regexFileName
copy2(frurl, fwurl)Then, we write our filters from the config to it:
def _configureRegexUrlfilter(self):
notMatchPatterns = self.config["notMatchPatterns"]
matchPatterns = self.config["matchPatterns"]
regexUrlfilter = open('/path/to/nutch/runtime/local/conf/' + self.regexFileName, "a")
if notMatchPatterns:
for url in notMatchPatterns:
regexUrlfilter.write("-^" + url + "\n")
if matchPatterns:
for url in matchPatterns:
regexUrlfilter.write("+^" + url + "\n")regexUrlfilter.write("+.\n")
regexUrlfilter.close()A few things are going on here: we are opening and appending to the file we just copied (that’s how “a” works) and then, for each “do not match” pattern we have, we are adding it to the file, followed by the match patterns. This is because, as we said before, Nutch will use the first regex match it gets, so exclusion needs to go first to avoid conflicts. We then write .+ so that Nutch accepts anything else--you can leave it off if you would prefer Nutch exclude anything not matched, which is its default behavior.
As a quick side note, it is important to mention that designing it this way means that each time we inject our configuration into Nutch, we will be wiping out and recreating these files. This is the easiest pathway we found for implementation, and it affords no disadvantages except that you cannot manually manipulate these files in any permanent way. Just be aware.
Now that we have our files in place, the last thing we have to do is inject the configuration into Nutch itself. This will be our first use of the Nutchserver API. If you have not already, open a console on the server that hosts Nutch and run:
$ nutch nutchserverOptionally, you can add a --port argument to specify the port, but we’ll use the default: 8081. Then we’ll prep the data for injection into the API:
def _prepInjection(self):
config = {}
# Custom config values
config["meta.config.configId"] = self.configId
config["meta.config.configTitle"] = self.configTitle
config["meta.config.seedFile"] = '/path/to/nutch/seeds/' + self.seedFileName
# Crawl metadata
config["nutch.conf.uuid"] = self.uuid
# Crawl Config
config["urlfilter.regex.file"] = self.regexFileName
config["db.ignore.external.links"] = self.allowExternalDomains
self._injectConfig(config)Note that we are creating both our own custom variables for later use (we named them “meta.config.X”) and setting actual Nutch configuration settings. Another note: urlfilter.regex.file takes a string with the filename only. You CANNOT specify a path for this setting, which is why we store the regex files in /path/to/nutch/runtime/local/conf, where the CLASSPATH already points.
Lastly, we’ll do the actual injection. The self._injectConfig(config) at the end of the _prepInjection function starts injection:
def _injectConfig(self, config):
job = {"configId": self.uuid,"force": "true","params": config}
r = requests.post('http://localhost:8081/config/' + self.uuid, json = job)
return rAll we do here is set up the JSON to push to the API and then inject. Every configuration we send to the API must have a UUID as it’s configId (which we will reference later when creating crawl jobs). We set force to true so that configurations will get overwritten when they change upstream and then we pass in our configuration parameters.
We then use the requests python module to make the actual injection. This is significantly easier than using something like CURL. We post to a url containing the uuid and have the JSON as the body (requests has a handy json argument that converts Python dictionaries to json before adding it to the body). Lastly, we return the post response for later use if needed.
And that’s it! We have successfully posted our dynamic custom configuration to nutchserver and created the relevant files. In the next post, we’ll show you how to crawl a site using these configurations.
After the experience map and user personas were created, Mobomo began to develop screen sketches, the data architecture, and overall flow for the mobile application. By building these out early in the process we were able to identify possible data fails early on and come up with ways to mitigate these potential issues for the future. 
We specialize in interface design and we take pride in our process but there is a lot of work done behind the scenes before the design is complete. When you work as a digital interface designer, you spend your day interacting with many tools and files. Sometimes you find yourself doing repetitive tasks which can become annoying after awhile but being methodical and organized can definitely help. We have talked about design etiquette and how to keep your files and folders organized which can help improve your workflow - but what about shortcuts on tools…Let’s talk ways you can customize Photoshop to save you time and hopefully improve your workflow.
There are many ways you can take advantage of this tool, for example, instead of having to manually copy and paste the style of a layer - you can have it attached to a keyboard shortcut in an action. There are some tools that don't have an option for a keyboard shortcut, having actions allows the user to give a keyboard shortcut to a specific tool to use it later. 
Not all the tools in photoshop have keyboard shortcuts. Actions can be used manually in ‘Batch’ or you can utilize Droplet which are small applications that automatically process all files that are dragged onto their icon. 
Do you have suggestions or is there a particular tool that you would be interested in learning more about to improve your workflow? 
