Thursday, April 18, 2019

Save the Date - ZeekWeek 2019

Save the Date 

October 8th - 10th

ZeekWeek 2019 
(formerly BroCon)

King Street Ballroom & Perch, Hilton Embassy Suites

255 South King Street, Seattle WA 98104

This year ZeekWeek (formerly BroCon) will be held 8-10 October 2019 in the King Street Ballroom & Perch at the newly renovated Hilton Embassy Suites in Seattle Washington.

Attendees will be able to “Zeek-out” on workshops, training, community presentations and visit with each of the vendors, sponsors and more.

Haven’t been to a Zeek event? Check out the lineup from last year.

Registration and Call for Participation will open soon, so check back often.

Helpful Links and information:

Getting Involved: If you would like to be part of the Open Source Zeek Community and contribute to the success of the project please sign up for our mailing lists, join our IRC Channel, come to our events, follow the blog and or Twitter feed. If you’re writing scripts or plugins for Zeek we would love to hear from you! Can’t figure out what your next step should be just reach out. Together we can find a place for you to actively contribute and be a part of this growing community.

About Zeek (formerly Bro): Zeek is a powerful network analysis framework that is much different from the typical IDS you may know.

Wednesday, April 17, 2019

People of Zeek - Interview Series - Robin Sommer

Today we kick off our weekly interview series with the Open Source Zeek Leadership Team and community contributors with Robin Sommer.

Before we get started, I’d like to thank Robin for taking the time to answer my interview questions and kick off this series.

Amber Graner (AG): As one of the core developers of Zeek (formerly Bro) I am sure you are no stranger to the community. However, can you introduce yourself to readers who may not know you?

Robin Sommer (RS): I became involved with Zeek in the early 2000s when I was still a graduate student in Germany looking for a research subject in network security. After starting to contribute small patches, Zeek quickly became my primary research platform; that work led to a number of new capabilities that then made back into the open source distribution. During that time I spent a couple of summers in Berkeley working with Vern at the International Computer Science Institute, and I later joined his group as staff researcher. When the National Science Foundation decided to invest into building out Zeek as an operational tool for the US research and education community, I began leading the team behind that effort. Today, I am the CTO at Corelight, a startup that Vern, Seth, and myself co-founded to bring Zeek’s capabilities to enterprise environments.

AG: When you got interested in then Bro, now Zeek, what was the driving factor that made you want to join forces with Vern to get involved and drive this project?

RS: Zeek has always offered both a powerful platform for experimentation and a strong deployment base in real-world networks—a combination that’s gold for academic research. On the platform side, Zeek’s clean architecture separates low-level packet processing from high-level analysis and detection. As result, Zeek can not only accommodate very different environments, but also support new technical approaches that would be challenging to incorporate into other systems. On the deployment side, some of the most demanding sites started using Zeek pretty much from the beginning—which provided invaluable operational feedback for evaluation & refinement.

AG: At the recent Open Source Zeek European Workshop 2019 held at CERN in Geneva, Switzerland, you gave a talk, Looking Forward: On Supervisors, Packages, and Sandboxes. In this talk you discussed 2 topics, 1) Replacing BroControl and 2) Rethinking Zeek Packages. For those who weren’t able to be at this event and hear your talk can you give a summary here?

RS: This talk was a bit of an outlook into new things that we have on our roadmap. The more concrete of the two efforts is developing a replacement for BroControl, Zeek’s current management framework. BroControl was originally developed for a setting that’s quite different to what most Zeek users are using today: a physically distributed cluster of individual PCs all running Zeek. BroControl came out of our own desire to avoid some of the maintenance tasks coming with such a setup: installing Zeek across many identical systems, pushing out new configurations, health monitoring, etc. Today, however, even large environments are often running Zeek just on a single system, leveraging all the cores available these days instead of scaling across systems. On a single system, BroControl can feel pretty quirky unfortunately. To address that, we are planing to turn Zeek itself into a persistently running service that—while still using the traditional cluster architecture internally—will have a dedicated supervisor process that spawns and manages the necessary processes as its children behind the scenes. Doing so will fit much better into standard Unix service models.

The second part of the talk was really only some early brainstorming exploring better support for running externally provided, untrusted scripts in your Zeek installation. Zeek’s package manager has been a huge success, with more than 80 packages now out there contributed by the community. However, when installing one of these packages, there’s currently no isolation between what your local scripts and that external code. That can quickly turn out problematic: It’s pretty common for Zeek scripts to exhibit widely different performance characteristics depending on the network they are running in. If an external package ends up using up all your CPU or memory, that will shut down all your monitoring. To address that, I’d like to explore isolating scripts by running them inside separate processes. This idea is quite speculative at the moment; it’s unclear if it’s even technical feasible. But it’s an example of how we have often approached new Zeek functionality in the past: Starting with a proposal for discussion and feedback, we’d then try implementing a first prototype to gauge if it makes sense to go in further.

AG: In addition to Replacing BroControl and Rethinking the Zeek Packages, what else would you like the Zeek community to get involved with. Scripts? Plugins?

RS: There are really many ways to contribute to Zeek these days, you don’t need to write C++ code to bring value to the community (although you can!). The most direct way is publishing Zeek packages: If you have written a custom Zeek script, I encourage you to make that available; it's not hard. Beyond scripts, documentation is probably (still) our main bottleneck. While we want to improve, there's only so much that we can do on the project's side. The real value of Zeek comes from the many different use cases that it supports, and only the community as a whole can collect & disseminate that expertise. So I would like to encourage people to share what they have learned: Write blog postings, give presentations, participate on the mailing lists, join meet-ups.

AG: Besides Zeek, are there any other projects that you’re involved with and would like to share?

RS: Most of my open source work focuses on the broader Zeek ecosystem—there’s always plenty to do there. One of the related projects that I’m most excited about is Spicy: A new protocol parser generator to replace Zeek’s existing BinPAC system. As many in the community may have heard, there’s already a Spicy prototype available that demonstrates the new capabilities, but it isn’t ready for production usage yet. At Corelight, we are working on a new version of Spicy that will make Spicy real. There’s no release date yet, but the code will become open source under BSD license.

AG: Is there anything that you’d like to share with readers about Zeek that I haven’t asked you about?

RS: I would just like to thank the Zeek community for all the amazing things everybody is doing. There’s a reason that many of us on the core team have been involved with the project for such a long time: It’s extremely rewarding to work with such an excited community that always finds new ways to push the system’s boundaries into directions nobody had ever anticipated.

Thanks again, Robin!

Helpful Links and information:

Events: Stay tuned we’ll be announcing ZeekWeek dates and location this week.

Getting Involved: If you would like to be part of the Open Source Zeek Community and contribute to the success of the project please sign up for our mailing lists, join our IRC Channel, come to our events, follow the blog and or Twitter feed. If you’re writing scripts or plugins for Zeek we would love to hear from you! Can’t figure out what your next step should be just reach out. Together we can find a place for you to actively contribute and be a part of this growing community.

About Zeek (formerly Bro): Zeek is a powerful network analysis framework that is much different from the typical IDS you may know.

About Corelight:
Corelight makes powerful network security monitoring solutions that transform network traffic into rich logs, extracted files, and security insights for incident responders and threat hunters. Corelight Sensors run on open-source Zeek (formerly called “Bro”) and simplify Zeek deployment and management while expanding its performance and capabilities.

Tuesday, April 9, 2019

“Mission First, People Always.”

I’d like to take a moment and introduce myself.  I'm Amber Granerand I’m excited to join Corelight, Inc as the Director of Community for the open source Zeek project.  
When I volunteered to join the U.S. Army in 1989, the saying “Mission first, people always” was something that was often demonstrated by my senior leadership.  We worked hard. We played hard. I served, protected and belonged to something bigger than myself. Being in the Army let me be a part of history as I worked together with those in my unit to protect against enemy threats. The espirit de corps I felt when I was part of the Army, was something I always looked for in every job after the military.
You may be asking yourself, “Ok, Amber, what does joining the Army have to do with becoming the Director of Community at Corelight for the Zeek (formerly Bro) community?”  
I am so glad you asked!
When I was interviewed by the team at Corelight, I heard the words “Mission first,” and I saw “people always,” demonstrated by all of the employees that I met. I leaned in. I listened. I wanted to know more.  
Corelight is a cybersecurity company founded on open source software whose employees are demonstrating the principles and philosophies that I believe in, live by and are passionate about.  I was hooked.
Open Source –  check!
Mission First, People Always – check!
Freedoms – check!
Protect against threats – check!
Belonging to something bigger than myself – check!
Opportunity to serve and collaborate with a community – check!
There was something challenging, yet comfortably familiar about this opportunity. I wanted to be part of this organization. I wanted to get to know the people. I wanted to serve the community.
When Corelight offered me the position, I answered, “yes!”
It seemed like a perfect fit.
Before coming to Corelight I was the community manager and operations director for the Open Compute Project (OCP) Foundation, a 501(c) (6) organization which was founded in 2011 by Facebook, Intel, and Rackspace with a mission to apply the benefits of open source to hardware and rapidly increase the pace of innovation in, near and around the data center and beyond.
Prior to OCP I was the first community manager for Linaro, a collaborative engineering organization consolidating and optimizing open source software and tools for the Arm architecture.
Before diving headfirst into applying open source principles and philosophies to hardware development and innovation, I got involved in the Ubuntu community. I began my personal journey into open source in 2009, when I was given an Ubuntu 8.10 live CD, a laptop and instructions to “use the community” if I ran into any problems.
Once involved in the Ubuntu community, I was elected to the Ubuntu Community Council and became the editor-in-chief for the Ubuntu Weekly Newsletter (UWN) as well as an Ubuntu Women Project leader. Later I was asked to be a technical reviewer of edition 5 of the Official Ubuntu Book and later co-author for editions 6 and 7 of the same book.  I was also a technical reviewer for Jono Bacon’s Art of Community, Editions 1 and 2 and have written for the Linux New Media publication which featured my “You in Ubuntu” blog. My articles have also appeared in print in the Ubuntu User magazine.
However, my love for open source began way before 2009, when in 1994 I sat on the dropzone at Fort Bragg, North Carolina, watching members of the 82nd Airborne Division establish an uplink to an aircraft in flight using Red Hat Linux (Halloween Edition) and the early Acer laptops. This feat proved intelligence could be provided en route to a battlefield allowing our troops to have the best possible information before they had to exit the aircraft, and ensured the safety and protection of the troops and their area of operations.
Mission first, people always”
The opportunity to work for Corelight and collaborate with the open source Zeek  community completes the circle and brings me back to my roots.
I’m looking forward to getting to know the community. If you are passionate about open source, protecting your network and want to get involved in the open source Zeek community, ask me how.  
Look for me on the mailing lists, blog, twitter and in person at various Corelight and events. Without “U” there is no commUnity, so let’s work together to protect your network and beyond!

[Reposted with permission from the blog.]

Monday, April 1, 2019

New Zeek Release Schedule

Over the years we have released new Zeek (Bro) versions on a somewhat regular annual basis, often around the time of BroCon. We also often did smaller bug fix releases in between, typically without adding any new functionality. However, while this annual cycle gave Zeek users a stable version to run operationally, it also meant that sites seeking to leverage new features that hadn’t made it into a release yet, either had to wait a substantial amount of time before getting access, or were forced to switch to running an unstable development version from Zeek’s master git branch. To address this challenge, we are switching to a more frequent release cycle that combines the best of both worlds by (1) offering regular feature releases while also (2) providing dedicated longer-term stable versions for users who prioritize stability over new functionality. 


The next (and first :-) Zeek version will be 3.0.0, cut from git’s master branch at the time of release. It’ll be a stable version that we will support with critical bug fixes for one year, released as 3.0.x versions. During that year we will also be providing smaller feature releases 3.x.0 about every four months, again cutting them from master at that time. We’ll always support the most recent feature release with critical bug fixes (3.x.y).

About a year later, the process will repeat with stable version 4.0.0, followed by feature releases 4.x.0and so on. For stable releases, we’ll select what we’ll deem the most solid base at that point. That could mean current master, a patched version of the most recent feature release, or even just that feature release itself. For all releases with substantial changesfeature and stablewe’ll be making beta versions available early for users to validate.

In short:
  • x.0.0 are stables releases with one year of support
  • x.y.0 with y > 0 are feature releases with about 4 months of support
  • x.y.z with z > 0 are maintenance releases 

Backwards Compatibility

As a key part of the new release schedule, we will strive to maintain backwards compatibility between two subsequent stable releases, usually by first deprecating legacy functionality in stable release X, and then removing it in stable release X+1. That means that users of stable versions will have a year to adapt their installations before functionality becomes unavailable. The in-between feature releases will go ahead and take out functionality that’s cleared for removal with the next stable version. For example, if a feature becomes deprecated with 3.0.0, we can remove it in 3.1.0; and then also in 4.0.0.

Please note that we cannot promise 100% backwards compatibility between stable versions, as sometimes there’s just no good deprecation path without blocking development for an extended period. However, we will seek community input in cases where we need to choose.

Next Steps 

Zeek 3.0.0 will be the first version to follow the new schedule. We don’t have a release date for that yet. The main remaining work items for that version are finishing the renaming, and removing any functionality that’s currently deprecated in 2.6.

Monday, March 18, 2019

Beyond BroControl - A New Process Supervision Model for Zeek

Current State of Affairs

A near-term item on the Zeek Roadmap is to provide an alternative, and eventual successor, to BroControl.  For context on why that's the case, there's the following pain points:
  • Process supervision in an external tool/process like BroControl is flaky.
Other modern examples of process supervision tools recognize the benefit/control gained from being the direct parent of supervised-processes.  By moving all the process supervision logic into Zeek itself, we can have more confidence and control of the ongoing health/state of a deployment.
  • It's awkward to develop and test new scripts that are destined for production environments.
The common use-case is load-balanced network traffic across a cluster of worker processes.  We need to make it easy to test, from the command-line, using just PCAP files, a complete cluster deployment (scaled down) as it would work in production.  Having to use a separate/intermediate tool, like BroControl, during development is not conducive to a fluid programming workflow or feedback loop.
  • Atypical system/service/container management and administration.
A goal is to cater more towards modern sysadmin expectations and we've gotten feedback that the current approach of using BroControl over Zeek/Bro directly is not a typical way of operating.  We want to improve that by no longer requiring an install/run of an entirely separate software.  The main Zeek process will be all that's needed to both supervise a deployment and serve as the central point of integration in existing service/system management schemes.

BroControl evolved from a prior tool that was originally built to satisfy a particular research use-case, not necessarily modern deployments.  That's expected, coming from such an early point in time, however, with a large user-base now depending on Zeek for production use, it's wise to design a new tool that, from the start, takes into account the wider community needs.

The Plan

There's been a brief round of internal discussion already with the following design and implementation notes produced from that:

Zeek Supervisor Design Doc

To summarize the goal: we want to make the main Zeek/Bro process the point of entry for deployments and allow just running the Zeek/Bro process to create a cluster deployment comparable to what BroControl would currently configure.

We haven't started implementing any of this yet in order to capture and respond to community feedback, so please get in touch with any you may have.  The mailing list ( is a good place to discuss.

Thursday, October 11, 2018

Renaming the Bro Project

More than 20 years ago I chose the name "Bro" as "an Orwellian reminder that monitoring comes hand in hand with the potential for privacy violations", as the original Bro paper put it. Today that warning is needed more than ever ... but it's clear that now the name "Bro" is alas much more of a distraction than a reminder.

On the Leadership Team of the Bro Project, we heard clear concerns from the Bro community that the name "Bro" has taken on strongly negative connotations, such as "Bro culture". These send a sharp, anti-inclusive - and wholly unintended and undesirable - message to those who might use Bro. The problems were significant enough that during BroCon community sessions, several people have mentioned substantial difficulties in getting their upper management to even consider using open-source software with such a seemingly ill-chosen, off-putting name.

Accordingly, in 2017 the Leadership Team undertook to find a new name for the project. We solicited suggestions for new names from the Bro community, receiving several hundred that covered a wide range of sensibilities and rationales. The LT extensively discussed the candidates internally but was unable to come close to a consensus on a satisfactory choice. Names that some LT members quite liked, others found quite deficient. This process proved particularly hard because some well-liked names had trademark issues and such.

Given that impasse, the LT engaged with a professional naming/branding consultancy to identify other possible names. The process elicited reflection on just what we would like the name to convey, which included notions of insight/visibility, soundness, flexibility, and Bro's heritage.

As the process proceeded, a number of LT members identified their fondness for quirky, pithy names for open-source projects. One name in particular dates all the way back to the very beginning of Bro in 1995 at the Lawrence Berkeley National Laboratory: Zeek. At LBL, the production Bro monitoring ran as a pseudo-user named "zeek" - this included both running the Bro process itself, and also the batch jobs and parallel tcpdump captures used to ensure robust 24x7 operation - a usage that continued for decades.

Why Zeek? The name was inspired by Gary Larson's use of Zeek characters in various "The Far Side" cartoons. We were big Far Side fans at LBL!

As Bro's originator, I have to say that I find switching the system's name to Zeek not only timely, but - in its quirkiness and history - endearing. I am thrilled that the Leadership Team identified and subsequently strongly backed the choice. (It was great, too, to find that we could secure the domain

The name "Bro" offered numerous opportunities for modest-but-memorable wordplay, such as referring to the project's "Broadmap", beginning conferences with a "Broverview", and coining the term "brogrammer". We've only begun exploring the possibilities with Zeek. However, the speed with which we readily found our first slogan holds promise in this regard. In looking for a new name, we had particularly identified wanting to find one that underscores the system's ability to provide deep insight into network traffic. We put that goal aside for our final selection. But shortly after we settled on our choice, a project member offered:

Zeek, and ye shall find!

Thursday, July 19, 2018

Broker is Coming, Part 2: Replacing &synchronized

As a quick followup to Part1, I want to bring attention to the Reminder about Events and Module Namespaces.  This is something that I'd forgotten about (or never knew) that will save you a lot of headaches when converting scripts to use Broker.  While I'm at it, when testing data stores, it's important to keep in mind any data expiration times you have set.  More than once I got back to testing the next morning and wondered where my persistent data had gone only to eventually realize it had just been correctly expired. 

In Part 1, I discussed using Broker to interact with persistent data stores, so now I want to go over a couple options for synchronizing data across your Bro cluster. We'll also look at how to debug Broker communication to verify things are working as you expected. As a reminder, below I've included a basic example of the old method of using &sychronized, which is now depreciated.

Publish Data to the Manager

There are several common use cases for Broker communication and it comes down to where your data needs to be for your policy to work as expected. Do the workers need access to the full view of the data or can the proxies or manager make the decisions? The first case we'll examine is using Broker to publish the addresses to the manager, check out the &sychronized example near the bottom to see the script we started from.

Broker uses a pub/sub model in combination with events to communicate between nodes so we need to send a message to the recipient, but also define the event that will be processed as a result. If you aren't familiar with pub/sub and the concept of topics, here is an overview.

@load base/frameworks/cluster module EX; export { global known_clients: set[addr] &create_expire=1day &synchronized; global add_client: event(newaddr: addr); } event add_client(newaddr: addr){ if(newaddr !in known_clients){ add known_clients[newaddr]; } } event connection_established(c: connection){ if(c$id$resp_h =={ Broker::publish(Cluster::manager_topic,EX::add_client, c$id$orig_h); } }

The event for adding the address to the known_clients set will be handled by the manager, so we moved the check for existence in the set to just before adding the client data. The event definition is exported for global use. Then in the connection_established event, we replace the add to the set with our Broker::publish call. We need to specify the pub/sub topic, but Bro pre-defines Cluster::manager_topic for us already and the manager is automatically subscribed to it. The only other arguments are the event we want the manager to execute, along with any arguments. Don't forget to remove &synchronized! Re-deploying the new script on the cluster, let's see the result:

[bro@host ~]$ broctl deploy
[bro@host ~]$ wget -q
[bro@host ~]$ broctl print EX::known_clients
      logger   EX::known_clients = {
     manager   EX::known_clients = {
     proxy-1   EX::known_clients = {
     proxy-2   EX::known_clients = {
    worker-1   EX::known_clients = {
    worker-2   EX::known_clients = {
    worker-3   EX::known_clients = {
    worker-4   EX::known_clients = {

This method of offloading work to the manager has pretty clear tradeoffs. The workers aren't all storing copies of the set in memory and they also don't have to do any additional processing work. However, you're putting that load on the manager. There may be a better way, which we'll look at when we discuss distributing work among the proxies.

Making it Work in Standalone Mode

If you do any testing by running Bro in a standalone mode, or would like your policies to support people who do, you'll need to do a little more work to make your script do both.

event connection_established(c: connection){
      if(c$id$resp_h =={
            @if( Cluster::is_enabled())
event EX::add_client(c$id$orig_h);


Sometimes you'll see additional checks for which kind of cluster node is processing the event, but that's an added layer of complication we won't discuss right now. Example: @if ( ! Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER ) For the rest of these examples I will skip the standalone code to make things simpler.

Distribute Work Among Proxies

Another option when the workers don't need access to the data is to distribute the processing amongst the proxy nodes. We could publish the data as we did with the manager, but as there may be more than one proxy, we don't want them to multiply the effort. Conveniently, Bro provides a "highest random weight" hashing algorithm to spread the load evenly. All we have to do is change the Broker::publish line:

event connection_established(c: connection){ if(c$id$resp_h =={ Cluster::publish_hrw(Cluster::proxy_pool, c$id$orig_h, EX::add_client, c$id$orig_h); } }

The Cluster::publish_hrw function distributes events across a set of nodes, here we're using the proxy pool. The second argument is the value that will be hashed to determine the distribution. Finally, we give it the event to send along with any arguments.

[bro@host ~]$ broctl deploy [bro@host ~]$ wget -q [bro@host ~]$ broctl print EX::known_clients logger EX::known_clients = { } manager EX::known_clients = { } proxy-1 EX::known_clients = { } proxy-2 EX::known_clients = { } worker-1 EX::known_clients = { } worker-2 EX::known_clients = { } worker-3 EX::known_clients = { } worker-4 EX::known_clients = { }

Note that it only wrote to one of the proxies as expected.  If you test the policy from multiple clients you should end up with a fairly even distribution across proxies.  Obviously, all we're doing in the example is adding an address to a set, but in policies where additional logic is needed, this could be a huge relief off both the workers and the manager.  A good example of this usage is in the core policies known-hosts.bro, known-services.bro, and known-certs.bro.

Syncing Data Between Workers

Instead of the manager or proxies, let's imagine we want the workers to all have the same data so they can make decisions on it immediately.  We need to be cautious with this method because it means duplicating the memory footprint across all workers, but there are cases where being able to lookup data in a set or table will save more additional work.

Workers aren't connected directly to each other so in order to publish data to them we need to relay it through either the manager or the proxies.  To do this, we need to establish two things: a topic for the worker to publish to and we need to handle that event on the manager or proxies to forward the event to the other workers.  I'll be using a method that can be seen in the core Bro distribution as part of scripts/base/protocols/irc/dcc-send.bro.

First let's discuss the Broker topic.  We could relay the event through the master node, but ideally we should keep the load on the proxies as that's what they're for.  It would also be nice to distribute the load evenly across all available proxies which we can do using a provided round robin function, Cluster::rr_topic().  The result is that we can build a new function that determines which proxy or manager topic to use:

function example_relay_topic(): string{
   local rval = Cluster::rr_topic(Cluster::proxy_pool, "example_rr_key");

   if ( rval == "" )
       # No proxy is alive, so relay via manager instead.
       return Cluster::manager_topic;

   return rval;


This simple function tries to round robin through our proxies, but if for some reason we don't have any available, it will fall back to using the manager node.  The second parameter is just a key that the Cluster framework uses internally.  With the topic deciding function in place, we can modify the core Broker::publish line.

event connection_established(c: connection){
     if(c$id$resp_h =={
EX::add_client, c$id$orig_h);

Note that all we've really done at this point is send the EX::add_client event to one of the proxies, we still need to have them forward it to the workers.  That can be accomplished within that same event:

event add_client(newaddr: addr){
@if ( Cluster::local_node_type() == Cluster::PROXY ||
     Cluster::local_node_type() == Cluster::MANAGER )
     if(newaddr !in known_clients){
       add known_clients[newaddr];


And then we can test just like before:

[bro@host ~]$ wget -q
[bro@host ~]$ broctl print EX::known_clients
      logger   EX::known_clients = {
     manager   EX::known_clients = {
     proxy-1   EX::known_clients = {
     proxy-2   EX::known_clients = {
    worker-1   EX::known_clients = {
    worker-2   EX::known_clients = {
    worker-3   EX::known_clients = {
    worker-4   EX::known_clients = {

So there we have it, the known client address was distributed to all of the workers.

Debugging Broker Communication

Clearly there's a lot more to Broker than I've covered here and it can start to get confusing so let's look at a way to dig deeper into where these messages are going.  To start, you'll need to recompile Bro from source and include the --enable-debug flag.  

Now we need to pass the -B broker argument to bro which we can do by setting BroArgs in $BROPATH/etc/broctl.cfg.  (Other debugging options can be see with bro -B help.)

[bro@host ~]$ head -n 4 /usr/local/bro/etc/broctl.cfg
## Global BroControl configuration file.

BroArgs = -B broker

[bro@host ~]$ broctl deploy
[bro@host ~]$ wget -q

We need to look at the debug logs for the individual nodes which will be found in the $BROPATH/spool/[nodename]/ directories while the cluster is running.  Let's look at the proxies first:

[bro@host ~]$ grep add_client /usr/local/bro/spool/proxy*/debug.log
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460614 [broker] Process event: EX::add_client: []
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460633 [broker] Publishing event: EX::add_client([]) -> bro/cluster/worker

Here we can see where proxy-1 received the relay event and then published the event to the workers.  The second proxy never saw anything because of the round robin nature of the relay.  The logs also contain the event called and arguments which can be very useful.  Now the workers:

[bro@host ~]$ grep add_client /usr/local/bro/spool/worker*/debug.log
/usr/local/bro/spool/worker-1/debug.log:1530925958.531330/1530925958.531552 [broker] Process event: EX::add_client []
/usr/local/bro/spool/worker-2/debug.log:1530925958.532718/1530925958.533077 [broker] Process event: EX::add_client []
/usr/local/bro/spool/worker-3/debug.log:1530925958.527955/1530925958.528265 [broker] Process event: EX::add_client []
/usr/local/bro/spool/worker-4/debug.log:1530925958.531130/1530925958.531427 [broker] Process event: EX::add_client []

As you can see, all of the workers received the event as expected.


As mentioned at the start, I'm including this section simply as a reminder of what a simple policy would have looked like with the old &synchronized attribute.

@load base/frameworks/cluster
module EX; export { global known_clients: set[addr] &create_expire=1day &synchronized; } event connection_established(c: connection){ # clients if(c$id$resp_h == && c$id$orig_h !in known_clients){ add known_clients[c$id$orig_h]; } }
All we're doing is creating a set of addresses that we see as clients connecting to  To test this, I've setup a cluster with four workers and two proxies (realistically we only need one, but two will be important for a later example).  We can use broctl print on the command line to check values on a running cluster:

[bro@host ~]$ wget -q [bro@host ~]$ broctl print EX::known_clients logger EX::known_clients = { } manager EX::known_clients = { } proxy-1 EX::known_clients = { } proxy-2 EX::known_clients = { } worker-1 EX::known_clients = { } worker-2 EX::known_clients = { } worker-3 EX::known_clients = { } worker-4 EX::known_clients = { }

We made one connection and only one of the workers shows an address in the set proving that &synchronized is depreciated.


I hope this helps folks get started with Broker. There's so much more to learn with Broker, but I felt it was a good idea to help ease the transition with the most common aspects.