The last two blog posts (Part 1 and Part 2) demonstrated how to add a JPEG file analysis plugin. This part will show you how to take our working JPEG source code and make it a Zeek package. A Zeek package can be installed dynamically instead of requiring compilation directly into the Zeek source code tree like we did in the last post. At this point, it might be a good idea to read the following blog post if you are unfamiliar with packages:
You will find that one of the benefits of using Zeek packages is that your development time will be shortened considerably when compared to the method we used in the past two posts. Basically, the Zeek source code will only need to be compiled and installed once while our package can be repeatedly compiled in a minute or two. For that benefit alone, it is worth understanding how you can quickly create new file analyzers for Zeek through packages.
I wish I could say that there are easy methods for determining the changes to the source code from our last blog post to make it load as a package, but there are no shortcuts here. Most of the changes discussed in this blog post were determined based upon several days of trial and error to match the logic we wrote previously to the requirements for a package produced by the “init-plugin” script. I will try to describe the reasons for the changes the best that I can, but as convoluted as it might sound, remember that you have all of the source code through the links above. This means you can experiment as I did to understand the concepts I try to present here if my explanations are unclear.
What Are Zeek Plugins and Packages?
Zeek plugins are extensions to the Zeek C++ source code. Plugins can be distributed in the Zeek core repository or they can be distributed separately as packages. Compiled plugins eventually end up as dynamically linked libraries that can be inserted into the main Zeek source code at run time, along with any required Zeek scripts. This allows for a plugable model for execution such that you do not have to provide the whole Zeek source code tree in addition to your plugin logic to your end users. If you were to try to distribute your plugin source code with the rest of Zeek, say because it didn’t fit in the core Zeek source code, this distribution mechanism would quickly become cumbersome for you and your users. Packages containing plugins make distribution of your custom logic much easier.
Another benefit to developing packages versus on the core Zeek source code is that your development time will be shortened dramatically. The Zeek source code will only need to be compiled and installed once. Our package can be repeatedly compiled in a minute or two for each little change we might like to try and inserted in our working copy of Zeek.
For development velocity alone, it is worth understanding how you can quickly create new file analyzers for Zeek through packages. However, there are other benefits. With packages, you are able to keep portions of your custom logic private and only share it with users that might want it without worrying about them having to compile and install a brand new version of Zeek. Therefore, this post will assume that you want to translate the JPEG code we developed in the last two posts into a package we can install into Zeek.
At this point, compile and install the stock version of Zeek with debugging enabled (this was discussed in part 1 (https://blog.zeek.org/2019/12/how-to-add-jpeg-file-analyzer-to-zeek.html). We want to overwrite custom version we compiled from our last blog post because it may conflict with the code in this post. We don’t need our custom version of Zeek anymore when we have a plugin!
For your future reference, the basic Zeek package documentation is available at the following links:
The Zeek source code comes with a shell script that will create the basic structure of a Zeek loadable plugin called “init-plugin” at https://github.com/zeek/zeek-aux/blob/master/plugin-support/init-plugin. You can create the plugin skeleton in the directory “~/Source/ZeekFileAnalyzers” with the Zeek namespace “FileAnalyzers” named “JPEG” with the following command:
I recommend you navigate through all of the files in this skeleton and become familiar with the information in the comments. You will find this information helpful as I walk you through the portions of the skeleton that must be modified to support our JPEG file analyzer. The subsections below will walk you through the remaining steps and the reasoning behind the changes to the skeleton source code.
The changes to the output of the “init-plugin” command to add our JPEG logic are explained in a thirteen step process below. Each short section for each step below will explain the minimal changes required to get our logic working and compiling as a package.
Step 1 - Copy Over Our JPEG Logic
First, we know we are going to need the following custom files we wrote previously, so copy them to the “src” directory in the skeleton. We know we need these files because there are no equivalent files in the skeleton for us to modify, so just copy them as they are from our last blog post:
The next several steps will fix up some of these and other files in the repository we are creating so that they are able to be compiled and installed as a Zeek package.
Step 2 - Fix Up JPEG.h
Most of JPEG.h is correct, but we need to include two new files that will exist in our package. In the include section, add the following lines:
This change makes the “file_jpeg_marker event accept the new type “FileAnalyzers::JPEGMarker” we will be defining next.
Step 4 - Add types.bif
We need to add the type we just used in Step 3. We will be defining this type in a new file called “types.bif”. You will remember this type from our last blog post. Your new types.bif file should look like this:
This file contains the metadata for the package and was created by “init-plugin”. zkg.meta tells the various Zeek tools how to install this package. This file should have been created mostly correct, but the configure command for our package needs the Zeek installation directory. Change your “build_command” to match the one in this file:
Note that the modification is the record type “BifType::Record::FileAnalyzers::JPEGMarker” instead of the original value of “BifType::Record::JPEG::JPEGMarker” since we have a new namespace with this package.
Step 9 - Fix Up CMakeLists.txt
We must add the new binpac, cc, and bif files we placed in this project to the makefiles. This is done with lines 9-17 in the following file:
Specifically, we are changing the argument from “m: JPEG::JPEGMarker” to “m: FileAnalyzers::JPEGMarker”.
Step 12 - Add types.zeek
Instead of having our types ready through the “init-bare.zeek” script, as we showed in the last post, we are going to add it to a file named “types.zeek” that identifies this record. Create the content in the following file:
You can execute Zeek with our JPEG analyzer on the PCAP we used from the last blog post, like so:
$ zeek -r pcaps/http_with_jpegs.cap jpeg.zeek
The output is too lengthy to reproduce here, but if you are following along you will see output for every JPEG marker encountered, just as we saw in the last blog post. Note that because we load “main.zeek” from “__load__.zeek” script, we only have to load the module with “@load FileAnalyzers/JPEG” to get things started from our custom scripts. The “file_jpeg_marker” event is fired because we see the output from our script.
At this point, we have replicated our JPEG functionality as an installable package. The various “zkg” commands will work on this directory, or if you point it at the GitHub URL, such as:
This blog post walked you through the steps required to make your Zeek JPEG file analyzer into a dynamically loadable package. A considerable portion of the logic could be copied “as-is”, but there were several modifications needed so that it could be compiled and installed as a package. This blog post discussed those modifications before building, installing, and using the JPEG analyzer. Lastly, this blog post created the baseline required for “btest”, the testing system Zeek uses. The next blog post will discuss how tests work in Zeek’s world, and we will add some additional tests to our JPEG analyzer.
About Keith J. Jones, Ph.D Dr. Jones is an internationally industry-recognized expert with over two decades of experience in cyber security, incident response, and computer forensics. His expertise includes software development, innovative prototyping, information security consulting, application security, malware analysis & reverse engineering, software analysis/design and image/video/audio analysis.
Dr. Jones holds an Electrical Engineering and Computer Engineering undergraduate degrees from Michigan State University. He also earned a Master of Science degree in Electrical Engineering from MSU. Dr. Jones recently completed his Ph.D. in Cyber Operations from Dakota State University in 2019.