There are some things you should NOT try to do yourself. Building and maintaining Deep Packet Inspection (DPI) technology is an example of something most companies should not try to do themselves.
DPI and IP classification are key technologies residing inside modern networking equipment, used to optimize traffic, to measure quality of experience, or to protect from cyber threats. Equipment vendors in theory have a choice of developing DPI and IP classification software internally or to source this technology from a specialist. This blog post argues that this is best left to a specialist.
Why Should DPI be Left to a Specialist?
1. DPI and IP classification require continuous re-development of software
Here I don’t mean software evolution, but actually continuously replacing code with new code that can handle new versions and a growing number of protocols. Signatures for some basic protocols are easy to manage since they are relatively stable (IMAP, SMTP, POP, HTTP, etc.). But most modern protocols evolve at a rapid pace without warning (Webmails, P2P, social media, gaming, etc.). The way to manage this is through reverse engineering, since these layer 7 application protocols are proprietary. Each time a new protocol version is released, the corresponding software has to be redeveloped, typically without much reuse of previous development…
This requires particular skills and special methods: reverse engineering, custom-made tools, fast reaction to protocol changes. And you have to be able to support this process in the long term, for very large number of protocols and applications, with minimum latency, and maximum quality.
2. You cannot use traditional product development methods
The high-tech industry typically uses a structured approach for product development and management.
Most key activities are aligned around go/no go decision points and defined time lines, based on known specs. These processes are built to ensure that new products are delivered on time, according to customer specifications and with the appropriate quality.
BUT: these methods do not apply to protocol plugins and signatures! Most layer 7 application protocols have no known specs, and change continuously without notice, which means that development roadmaps cannot be easily controlled. Development teams must be quick to react to new protocol evolutions and use home-made reverse engineering techniques to update their DPI software. This way of working is counter-cultural for typical high-tech companies, who often find it difficult to recruit and retain this type of talent.
3. You must master highly specialized and complex technology
Networking and security vendors who have already embedded DPI into their solutions need to go further in terms of layer 7 visibility, with metadata extraction. This is a very specific expertise proposed by only a few specialist companies. Specialist companies in DPI develop their own tools for metadata extraction including a specific meta-programming language which generates an HTTP plugin automatically for any Website or Web application. These companies also boast proprietary techniques and tools for quality assurance and to make reverse engineering more efficient. This is very different from the business of someone who sells complete solutions. In most cases, a separate R&D organization must be created, with specific processes and skills. Last but not least, DPI and IP classification software must cope with increasing throughputs and be optimized to take advantage of new, sophisticated hardware capabilities.
Sometimes I hear “It is too expensive working with specialists, I will just outsource my DPI needs to a low cost country”. The harsh reality is that there is no miracle and you may get disappointed…
So my advice is: don’t try this at home, leave it to a specialist.
How should you choose the right DPI and IP classification software specialist?
Favor a pure-play technology vendor (avoids conflict with solution business)
Check that your supplier offers extensive protocol coverage and metadata extraction
Make sure the DPI software supports all leading processor architectures (Intel, Broadcom, Cavium, EZChip), so that you can change hardware while keeping the same DPI software.
