Saffron Technology Organizes Data into Memories

Summary: Saffron Technology provides an analytical database that explores relationships among entities and their attributes. It can explore networks, find similarities and guide decisions. Saffron is considerably more flexible than standard semantic engines.

As I was preparing my June 3 review of link analysis vendor Centrifuge Systems, Centrifuge introduced me to their business partners at Saffron Technology. This is an interesting product in its own right.

Saffron describes itself as an “associative memory base product,” a phrase that definitely takes some explaining. In simplest terms, Saffron organizes information into sets of three or four related items.

More specifically, it stores pairs of items in the context of an entity. For example, “Jack-climb-hill”, “Jack-plant-beanstalk”, “Jack-jump-candlestick” and “Jack-build-house” are all part of what Saffron calls Jack’s “memory”.

Related sets can be grouped into "matrices" that share another item. Thus, one matrix could contain “Jack-Jill-up-hill”, “Jack-Jill-fetch-water”, “Jack-Jill-fall-down” and “Jack-Jill-break-crown”, while “Jack-beanstalk-plant-seed”, “Jack-beanstalk-climb-up”, “Jack-beanstalk-steal-harp” and “Jack-beanstalk-kill-giant” form a separate matrix.

Saffron physically prejoins the data in Jack’s memory so it can be accessed easily and shared elements can be stored only once. Jill’s memory is stored separately from Jack’s even though some sets contain the same data. Jill’s memory may also contain sets without Jack (but let’s not tell him). Depending on how the system is configured, the hill could have its own memory as well.

Saffon’s approach lets it handle the subject-verb-object triples used in semantic analysis. But unlike standard semantic triplestores, Saffron is not limited to this structure. Sets could contain all nouns (Jack-Jill-hill), which can be useful even if the precise relationship among the items isn’t known. Or one of the items could be a time dimension. The system also counts of how often each item pair occurs in the source data, supporting statistical as well as semantic analysis.

These features make Saffron substantially more flexible than a semantic system. They also let it work with many more data sources, since reliable subject-verb-object relationships are often unavailable.

This may sound pretty dry, but Saffron's actual applications have been cloak-and-dagger stuff like looking for terrorists and finding roadside bombs. Remember that I was introduced to Saffron by Centrifuge, whose link analysis system is used primarily for law enforcement and security investigations. Saffron’s approach works particularly well with the Centrifuge front-end.

The fundamental advantage of Saffron is that associations among different items are directly accessible for analysis. This lets the system support different types of queries including:

- connections (identifying relationships among items)
- networks (showing how entities connect with each other)
- analogies (finding entities with similar connections)
- classifications (placing similar entities into groups)
- trends (reporting how connections change over time)
- episodes (finding patterns that repeat over time)

Real-world applications extend beyond link analysis to classifications and decisions. For example, the system can select medical treatments for a particular patient or assign suspended loans to different collection processes. These are complex processes. The system must identify entities with similar characteristics, identify the treatments and outcomes for those entities and estimate the likely outcomes of applying different treatments to the current entity.

Saffron's advantage with such applications is that the characteristics, treatments and outcomes are all just items in the data store. There's no need to load them differently or build a causal model of how they interact. In other words, the system needs no inherent assumptions about how the world works. This lets it effortlessly incorporate new data, uncover hidden relationships and react to new situations.

Well, “effortlessly” is a bit of an exaggeration. Saffron does some pretty complicated calculations to decide which entities are most similar and which treatments have the highest expected value (i.e., outcome value x outcome probability). Users have to review results and make judgments to tune the system. But this is still much less work than conventional statistical modeling or rule-based systems.

Saffron’s technology lets users define the types of data to will store, load data into the system, and execute API calls for the different types of analysis (connections, networks, analogies, etc.) Saffron generally relies on external systems to identify entities within source data, classify them into the specified categories, and report their associations to Saffron. Saffron can load structured data as well.

Saffron runs on a 64 bit “soft appliance” that distributes its data over clusters of server drives. The company claims world-record performance at ingesting, storing and accessing triplestore data, as well as compression to about 20 bytes per triple compared with 50 to 150 bytes per triple in other triplestore systems.

But let’s not get too carried away: Saffron works with large but not gigantic databases. Customer systems have been in the one-terabyte range.

Pricing for Saffron is starts at $125,000 per server, where a standard server is an eight core machine with 16 gigabytes of RAM per core. A trial version of the system is also available on the Amazon EC2 cloud as SaffronSierra.

Centrifuge Systems Offers Powerful, Flexible Link Analysis

Summary: Centrifuge Systems offers powerful, server-based link analysis and data visualization. It lets non-technical users load their own data, allowing them to work with minimal external support.

Centrifuge Systems offers data visualization software with a specialty in link analysis (that is, finding relationships among entities such as members of a social network). It isn’t the only vendor in the field – a quick search brought up this list of link analysis systems, which itself is not complete. Centrifuge tells me they are unique in offering link analysis that doesn’t require client software (only a browser with Adobe Flash) and works without a predefined data model. I can't personally confirm this, but was intrigued enough by Centrifuge that its uniqueness is not a prime concern.

Like other link analysis systems, Centrifuge has been used primarily for criminal and intelligence investigations. However, it is currently looking for additional applications such as marketing analysis to understand relationships between customers, locations and products. Since it does conventional data visualization in addition to the link analysis, Centrifugre is at least a potential replacement for visualization tools like Tableau and TIBCO Spotfire.

I had a briefing from Centrifuge a few months ago and recently downloaded their free trial system to play with it a bit. Not surprisingly, it was harder to use by myself than when a salesman was showing it to me. But the basic interface made sense and I can see that with a bit of practice, this would be a pretty effective system for a business analyst even if they lacked deep technical skills.

Setting up a project involves uploading data to the server, connecting to it, and then dragging data elements into position as dimensions and measures. Links between data elements are also defined by dragging fields into place. Users can refine their views by creating filters, derived values and bundles to combine selected items. Results can then be displayed from multiple perspectives including tables, link maps, charts (bar, line, pie, etc.), geographic and bubble maps, timeline, geospatial and drill-down charts.

The dragging itself wasn't as smooth as a typical desktop application, but it was perfectly serviceable and pretty impressive for working within a browser. Charts rendered almost instantly using the small sample data set. A larger volume might slow things down, but the heavy lifting is done on the server, so the system should scale well if the server is adequate. Centrifuge says its largest installations involve thousands of users and many millions of database rows.

The most important feature of Centrifuge is probably its ability to upload and link pretty much any type of data. External connections use JDBC drivers, which support sources including spreadsheets, XML and live feeds as well as conventional databases. The latest release lets analysts add new data sources by themselves, letting them work quickly with minimal technical support.

The system also lets users extract a subset of data and analyze it independently, reducing the load on the server. They can share their work by publishing it to a server as a PDF or live asset available to others. Newly published assets can be listed in an RSS feed.

Centrifuge was founded in 2007 and has multiple government clients, plus a few in private industry. Pricing starts at $4,000 for a single user perpetual license plus 18% annual maintenance.