Private Intelligence - Why it matters
The founders at Jigsaw Security all originated out of the intelligence community. Upon leaving the IC, many of us missed the tools the capabilities that we have been accustomed to when making intelligence decisions. In the private sector many of these tools were outside of our price range for a startup so we decided to go on the hunt and find solutions that could scale to millions upon millions of documents. Some of our engineers had worked on major intelligence systems that collected millions of documents and witnessed first had just how hard it was for the DOD to maintain these very large and complex systems. We decided it was time to create our own solution and that brings us to 2019. In mid 2019, Jigsaw Security began deploying the components to create the first fusion capability in the private sector.
Ingest is always the hardest part of any intelligence platform
As with most systems you have to understand what data you need that can help you make the proper decisions and the flow of information into the intelligence platform must be continuous. With the advent of many large big data platforms such as Hadoop and Elasticsearch, there were options in the stack but they were suited to really structured and formatted data. From the start we knew we want to be able to ingest ANY type of document and extract (with optical character recognition) the text and then index and score that text so we could easily recall documents during research or intelligence operations. There are very few companies that actually house private intelligence (only a handful) and the ones we know about are focused only on cyber security or physical security operations. We needed a system that could take in large amounts of documents regardless of format and then tag the documents in such a way as we could see the connections between these documents and the system had to be fast, reliable with high availability.
In addition to getting data into the system, we needed to be able to store entity and action (we call it EA). By knowing who is doing what, we can then determine what actions or response to implement. Throughout this blog post we will explain how we do this and why it's important.
In the screen capture from our system above you can clearly see that we are capturing the entities. In addition to entities we also capture attributes (see below).
Being able to pull out types of documents, countries, names, email addresses, iban numbers, phone numbers, etc. is critical to connect the dots. These references make the data more valuable to the analyst.
Entity extraction is a very valuable resource to analyst when trying to find links between threat actors and documents or actions carried out by threat actors.
Highlighting and Keywords
Searching for keywords (in this case above we searched for "Cyber Warfare". As you can see the search highlighted terms within a single document but the system also returns results across many document collections.
The system provides granular control over what data is available to what end user based on clearance level, access restrictions, document collections or even down to individual documents. Analyst can have their own document collections which they share or can share a single document with other analyst.
This granular control ensures that teams can protect their information until it is ready to be published to the larger community or the public.
In short there was a need to have intelligence capabilities in the private sector. The solution is inexpensive and cost effective but highly capable of scaling to Petabytes of storage. It no longer makes sense to spend millions of dollars on Palantir when this solution exist and is able to do just as much as Palantir or Keylines. Open source has caught up and surpassed the previous leader in this space.
To obtain a demonstration of the Jigsaw Intelligence platform please contact Jigsaw Security. The reason this matters is because companies spend millions of dollars on solutions that are no more effective than what is being put out in the open source community.