Tuesday, November 12, 2019

ZeekWeek 2019 - Summary and Slides

The global community of Zeek developers and users gathered together in Seattle last month, October 8-11, for the annual ZeekWeek (formerly BroCon) event. 


171 network security professionals representing 84 organizations travelled from all over the world to share ideas and knowledge of Zeek.

This year’s event consisted of 2 Zeek training sessions, 17 presentations, lightning talks, a community Q&A panel discussion, and more.

In case you missed this year’s event, here is a list of all the talks as well as all the slides that were made available to organizers. The full agenda and talk descriptions can be found on the ZeekWeek.com website. (Please note: There will be a message that states this event has already happened; just hit the escape key and it will go away. Also, videos coming soon!)


Sessions

8 October 2019 - Pre-conference Training

(Training slides available for attendees only)
  • Intro to Zeek, Keith Lehigh, Indiana University
  • Making Sense of Encrypted Traffic, Matt Bromely and Aaron Soto

9 October 2019 - ZeekWeek Day 1 - Sessions

  • Opening Remarks, Keith Lehigh, Indiana University (Slides)
  • Keynote: The Threats are Changing, So are We as Defenders, Freddy Dezure, Founder and former Head CERT-EU (Slides)
  • eZeeKonfigurator: Web Frontend for the Config Framework, Vlad Grigorescu, ESnet (Slides)
  • BZAR – Bro/Zeek ATT&CK-based Analytics and Reporting, Mark Fernandez, Lead Cybersecurity Engineer The MITRE Corporation (Slides)
  • Run, Zeek, Run!, Jim Mellander, Cybersecurity Engineer, ESnet (Slides)
  • DNSSEC Protocol Parser - A Case Study, Fatema Bannat Wala, Security Engineer, University of Delaware (Slides)
  • Profiling in Production, Justin Azoff, Corelight (Slides)
  • Identifying Small Heavy-Hitter Flows Using Zeek to Optimize Network Performance, Jordi Ros-Giralt, Managing Engineer, Reservoir Labs (Slides)

10 October 2019 - ZeekWeek Day 2 - Sessions


  • 7 Years with Zeek on Commodity Hardware, Michal Purzynski. Engineer, Mozilla Corporation (Slides)
  • Zeek 3.0.0 and beyond, Robin Sommer, Corelight, CTO and Co-Founder (Slides)
  • Baseline the Network with Zeek, Adam Pumphrey, Consultant, Nimbus LLC (Slides)
  • Without U There is No CommUnity, Amber Graner, Zeek Community Director, Corelight (Slides)
  • Zeek - Incident Response and Beyond, Aashish Sharma, Lawrence Berkeley National Lab
  • Encrypted Things: Network Detection and Response in an Encrypted World, TJ Biehle, Sr. Technical Account Manager, Insight (Slides)
  • Lightning Talks (Various presenters)
    • Zeek Based IPS (Slides)
    • Challenge: Zeek on a large amount of low power sensors, Alex Bortok (Slides)
    • Using BRO [Zeek] to tattle on other tools, Patrick Cain, The Cooper-Cain Group. Inc. (Slides)
    • Contributing to Zeek (How to do a Pull Request), Tim Wojtulewicz, Corelight (Slides)
    • Dynamite-NSM, Open-source project for network traffic analysis with Zeek, Suricata, Flow Data and ELK, Oleg Sinitsin, Dynamite.AI (Slides)
    • eZeeKonfigurator - notice config, Michael Dopheide, ESnet (Slides)
    • How I became a Zeeker & Why I Zeek, Jeff Atkinson (Slides)
  • Using Zeek for SSL Research, Johanna Amann, Senior Researcher, ICSI / Corelight / LBL (Slides)

11 October 2019 - ZeekWeek Day 3 - Sessions

  • New Implementation of Zeek Dictionary to use Less Memory, Jason Lu, Senior Staff Software Engineer, Gigamon (Slides)
  • Introduction to Zeek Script Writing, Seth Hall, Corelight, Chief Evangelist and Co-Founder (No slides were used for this talk; live scripting)
  • Visualizing, Analyzing and Filtering Zeek Events using a Graphical Frontend and OpenGL, Nick Skelsey, Security Engineer, Secure Network (Slides) (Demo Vids)

Thoughts on the event

"ZeekWeek 2019 was another great opportunity to catch up with colleagues across both R&E and industry. It's always inspiring to see what people have been doing with Zeek over the last year." ~ Michael Dopheide, ESnet
“Great experience sharing knowledge and collaborating with the community in this year's ZeekWeek, so much useful content and great place to “zeek out” with fellow Zeekers!” ~ Fatema Bannat Wala, Security Engineer, University of Delaware
“ZeekWeek2019 provided a great opportunity to share knowledge in pursuit of defending networks. Without the people, Zeek is just a tool.” ~Keith Lehigh, Indiana University and Chair, Zeek Leadership Team
“We use Zeek, you should too!” ~ Aashish Sharma, Lawrence Berkeley National Lab and Zeek Leadership Team Member
"It's amazing to see everything this community is doing with Zeek." ~ Robin Sommer, Corelight, CTO and Co-Founder; Zeek Leadership Team Member


Many Thanks and Much Appreciation


Zeek events, such as this year’s ZeekWeek, are only possible through the generous support of the Zeek community, its sponsors and hosts. A huge shoutout and “THANK YOU” to all out sponsors and speakers!!

Helpful Links and information:


Getting Involved: If you would like to be part of the Open Source Zeek Community and contribute to the success of the project please sign up for our mailing lists, join our IRC Channel, come to our events, follow the blog and/or Twitter feed. If you’re writing scripts or plugins for Zeek we would love to hear from you! Can’t figure out what your next step should be, just reach out. Together we can find a place for you to actively contribute and be a part of this growing community.

About Zeek (formerly Bro): Zeek is the world’s leading platform for network security monitoring. Flexible, open source, and powered by defenders. https://www.zeek.org/

Tuesday, October 8, 2019

ZeekWeek Q&A with the Community: Bricata

by Amber Graner, Zeek Director of Community



As ZeekWeek gets underway, we wanted to find out what’s new among members of the Zeek Community. Accordingly, we had a chance to catch up with the Bricata team.

Bricata is a contributor to the Zeek community, and supporter of ZeekWeek as the exclusive sponsor of the Welcome Reception for the 2019 event.

1. For those who are new to the network security monitoring (NSM) space can you tell people about Bricata?

Bricata: Bricata is laser-focused on empowering security analysts to hunt effectively. The platform provides analysts with the tools they need to adequately respond to network threats and provide comprehensive network protection. Bricata gives security teams the capabilities to do things like:

  • Obtain network visibility quickly to thoroughly understand what’s taking place in their environment
  • Respond to alerts and understand their context. Alerts are triggered by our multiple threat detection engines, including Zeek; Suricata; IOC matching, and AI-based binary conviction
  • Hunt for zero-day threats using Zeek-generated metadata and PCAPs and develop countermeasures against future attacks

From a workflow perspective, Bricata is especially well-suited to threat investigation and hunting. That means the platform provides a streamlined approach to foraging through network data and developing insight. It’s the metadata produced by Zeek that provides the context for investigating alerts and taking action with the platform.

Flexibility is an important principle here. Bricata gives security organizations the flexibility to customize and enrich the network metadata so that it’s meaningful within the context of their specific environments and use cases. In addition, our dashboard and visualization tools can be easily tailored to an individual analyst’s preferences.

2. Why is ZeekWeek and the Zeek Project important to Bricata?
Bricata: ZeekWeek is a time for everyone in the community to get together. We’ve found it to be a very devoted group of people sharing their experiences working with Zeek and sharing how they’ve worked out solutions to difficult, but common challenges.

In the past, we’ve used this opportunity to share successes we’ve had with the Zeek Project in the context of our solution and our customers’ use of Zeek. For example, we previously released a labeling module to the community, which provides a way for analysts to share their knowledge about the environment. Those labels are matched with network data that Zeek is generating, which in turn enables more sophisticated threat detection and network analysis.

We expect to see a lot of focus on machine learning this year with Zeek-produced datasets and particularly how people are optimizing their use and management of it. That’s important because network speeds keep getting faster and unconstrained, Zeek is known to produce a high volume of data.

3. What can attendees expect to learn if they visit your booth at ZeekWeek?

Bricata: Visitors will see just how easy we’ve made it to deploy and use Zeek in their environment. They can stand it up and get usable network visibility very quickly. This allows them to easily incorporate it into their IT infrastructure and security operations.

Secondly, people that haven’t seen the solution in a while will find some of the most recent enhancements we’ve made for our customers interesting. For example, as members of the community know, Zeek can generate a wealth of metadata. While that’s useful, it can also be overwhelming, so we’ve incorporated fine-grain filters that permit security teams to precisely control the Zeek logs they require. This ability prevents the costly processing and storage of unnecessary metadata.

Finally, and this one of the benefits of the community, we’ve adopted the 5-tuple Community ID hash. We’re using it to help consolidate similar alerts under a single grouping as a means to reduce the alert fatigue the SOC can sometimes experience. Bricata is bullish on the Community ID because we see it as an up-and-coming standard that will enable seamless interoperability with other security solutions.

4. What else would you like attendees to know about that I haven't asked you about?
Bricata: Fly-Away kits are one of the initiatives we have that extends beyond the traditional use cases for NSM. Zeek is an integral part and here are a couple examples:

  • We’ve partnered with a solution provider that makes network taps to develop a portable flyaway kit for incident response. This brings visibility to environments that are not properly instrumented, or where the response team is unfamiliar with the environment.
  • We’re continuing to build traction among service providers who provide digital forensics and incident response (DFIR). Their teams are using our platform when deployed to dynamic situations like data breaches, insider threats, or any sort of suspected malicious network activity. It helps incident responders quickly understand what is happening on a network, detect threats and facilitate the incident response process.

* * *
ZeekWeek 2019 attendees interested in learning more about Bricata should look for their display on the exhibition floor. In addition, you can check out their website, and stay in touch on LinkedIn or Twitter.


Friday, October 4, 2019

Zeek, Corelight and Humio help make observability accessible

Guest post by Humio


We’re proud to have Humio on board as the exclusive training sponsor for ZeekWeek 2019. As a thought leader in the observability space, Humio has a deep understanding of making observability accessible, comprehensive, and affordable.

Humio can help you efficiently visualize and get answers from the Zeek log volumes that Corelight sensors generate. By pairing Corelight’s deep network monitoring and logging with Humio’s fast and affordable log management technology, you’ll get accurate answers to critical security and IT questions more quickly and more easily than you thought possible.

Humio shares their thoughts about how the need for comprehensive observability is driving a cultural shift.

Our industry is moving at lightning speed towards distributed service-driven architectures, and engineers are on a quest to improve how they observe their systems as a whole. Adoption of microservices and containerized architectures has elevated the need for developers and operations teams to use observability solutions to correlate events, identify threats, and troubleshoot problems. From a business value point of view, managers want observability solutions that allow them to keep calm when their software infrastructure and services are hit with incidents or failures.

Many organizations adopt a combination of log management, metrics, and tracing solutions for observability across their infrastructure. We have found that just having these tools isn’t enough to ensure that engineering teams are able to reap value from them. A cultural shift is required.

Excerpt from O’Reilly’s Distributed Systems and Observability Book by CindySridharan 
“As my friend Brian Knox, who manages the Observability team at DigitalOcean,
said, 
“The goal of an Observability team is not to collect logs, metrics, or traces. It is to
build a culture of engineering based on facts and feedback, and then spread that
culture within the broader organization. 
“The same can be said about observability itself, in that it’s not about logs, metrics,
or traces, but about being data-driven during debugging and using the feedback to
iterate on and improve the product.”

As Brian Knox and Cindy Sridharan mention in the excerpt above, observability is about having an engineering culture that values facts and feedback, “being data driven” during debugging, and using this mindset to iterate, improve, and solve problems.

At Humio, we meet many teams that have yet to access the full value they could get from their log data. This isn’t because they don’t have or want a “data driven” observability engineering culture, but rather that their current log solution restricts them from being able to.

Commonly, teams encounter three restrictions with their log solutions:

1. Volume: Modern organizations generate large amounts of unstructured log data — a lot of time is spent on limiting or deciding what data to send to the system. 
2. Speed: Slow queries and latency between index and search phases take too long. Ultimately, the data isn’t available fast enough. 
3. Simplicity: Logging solutions that are not easy to use, query, deploy, or manage result in limited use or frustration using them.

Data-driven Log Management


Our approach at Humio is to remove these restrictions, so data-driven observability teams can gain more value from their log data. We encourage engineers to send all relevant log data, and for all the data to be accessible. Limiting data based on what a logging solution can handle is restrictive, and often it is the logs that were left out that create frustrating debugging scenarios.

Humio is built to scale linearly, and efficiently store data so users aren’t wasting their compute resources. These days, speed matters, and by using real-time streaming capabilities for querying and dashboards, Humio superpowers live system visibility for engineers. Our CTO, Kresten Krab Thorup, wrote a blog post to explain how Humio scales and handles data.

For data-driven logging to succeed, engineering teams should use it for the value it provides. Humio’s query language and ease of use speeds adoption past just the Ops team to the developer organizations, making it a shared solution for everyone. For example,Lunar Way’s developer-driven ops uses Humio across both its development and operations team.

Observability Site License


Humio’s approach to logging is valuable for both small- and large-volume users. For teams with large logging volumes (multi TB/day), Humio software is available On-Premises at a fixed annual site license price. This enables companies to access large log volumes without volume-based licensing costs or extra manpower required in running complicated cluster logging environments. With this model, organizations can add instances and scale up as their data volumes grow or burst. For observability or infrastructure teams who want to deploy multi-tenant logging infrastructures across teams within an organization, Humio can provide simple pricing.

At Humio, we believe in the value of data-driven logging, and the benefits companies derive from this in their observability stack. With a unique product and simple pricing, Humio is on a mission to bring this value to engineering teams who’ve been struggling until now.

Thursday, October 3, 2019

ZeekWeek 2019 - Thank you to our sponsors



The Zeek Project Leadership Team (LT) would like to thank all of the ZeekWeek 2019 sponsors for their generous support. Without their ongoing support ZeekWeek would not be possible.

ZeekWeek is the most important community event for users, developers, incident responders, threat hunters and architects who rely on the open-source Zeek network security monitor as a critical element in their security stack.

If you want to meet with the Zeek Leadership team, core maintainers or our sponsors, registration is still open.

We look forward to seeing you all and our sponsors in Seattle on 8-11 October.

This year’s sponsors include:


Reception





Training





Lunch




40 GIG











10 GIG












Hosted by:




Monday, September 23, 2019

Zeek 3.0.0

(Note: This is a slightly updated version of a previous posting announcing the initial release candidate.)



We just published Zeek 3.0.0—our first major release since Bro 2.0 came out in 2012. This version is quite special as it undertakes The Big Zeekification™: It is executing on the technical side of the name change that we announced last year by now renaming the tool itself, including binaries, scripts, and even some events. “Bro” is now “Zeek.” 

This name change brings some disruption for existing users, which is unavoidable for a long-term codebase where the original name had more than 20 years to proliferate into pretty much every corner. Nevertheless, we have been working hard to maintain backwards compatibility from Zeek 3.0.0 to Bro 2.6 as much as possible to facilitate smooth upgrades. Wherever we reasonably could, we put aliases and redirects in place so that old names remain working in parallel to the new ones. When using the old names, you will in many cases see explicit deprecation warnings that point you to the places that need updating. These transition mechanisms will remain in place for the Zeek 3.0.x series. We’ll remove them with the next feature release 3.1.0 and likewise with the next long-term stable release 4.0.0, in accordance with our new release schedule.

Below is a more detailed summary of the main changes coming with the renaming. In addition, Zeek 3.0.0 comes with a number of new features as well, including:

  • New analyzers for NTP and MQTT, and extended analyzers for DNS (SPF/DNSSEC), RDP, SMB, and TLS. 
  • Support for decapsulating VXLAN tunnels.
  • Support for logging in UTF8.
  • Several extensions of the scripting language:
  • Closures for anonymous functions
    • Iteration over key/value pairs of a table through for ( key, value in t ) ...)
    • Python-style vector slicing (v[2:4])
    • A new data structure, paraglob, for efficiently matching strings against large list of globs.
  • See the NEWS file for more detailed release notes, and CHANGES for the complete list of changes

Upgrading to Zeek 3


The following summarizes the main naming-related changes that you will encounter after installing Zeek 3.0.0. Unless otherwise noted, the Bro 2.6 names and paths will continue to work with this release, but often trigger deprecation warnings.

  • The names of all executables that had “bro” in their name have changed: bro -> zeekbro-config -> zeek-configbroctl -> zeekctlbro-cut -> zeek-cut. Zeek 3.0.0 installs wrappers under the old names that will let them continue to work.
  • The default install prefix is now /usr/local/zeek instead of /usr/local/bro. If your existing installation used the previous default and you are using the new default when upgrading, we'll symlink /usr/local/zeek to /usr/local/bro. Certain subdirectories get similar treatment: share/broinclude/bro, and lib/bro.
  • Along with BroControl becoming ZeekControl, installation directories and files with broctl in their name have changed to use zeekctl instead. However, these changes remain backwards compatible with previous Bro installations by continuing to pull from existing locations where customizations might have been made. For example, if you have a broctl.cfg file from a previous installation, installing Zeek over it will retain that file and even symlink the new zeekctl.cfg to it.
  • The new extension for Zeek scripts is .zeek. This leads to two major changes:
    • All scripts ending in .bro have been renamed to .zeek. In particular, $prefix/share/bro/site/local.bro has been renamed to local.zeek. However, if you have an existing local.bro file from a previous Bro installation—possibly with customizations made to it—Zeek will install a symlink local.zeek file that points to that pre-existing local.bro. In that case, you may want to just copy local.bro into the new local.zeek location to avoid confusion, but things should generally also work properly without intervention.
    • The search logic for the @load script directive now prefers files ending in .zeek, but will still fallback to loading a .bro file if it exists. E.g. @load foo will first check for a foo.zeek file to load and then otherwise foo.bro. Note that @load foo.bro (with the explicit .bro file suffix) prefers the opposite order: it first checks for foo.bro and then falls back to a foo.zeek, if that exists.
  • Changes affecting scripts:
    • The events bro_init, bro_done, and bro_script_loaded are now deprecated; use zeek_init, zeek_done, and zeek_script_loaded instead. Any existing event handlers for the deprecated versions will automatically alias to the new events such that existing code will not break, but their usage will emit deprecation warnings.
    • The functions bro_is_terminating and bro_version function are deprecated and replaced by functions named zeek_is_terminating and zeek_version. The old names likewise continue to work with deprecation warnings.
  • The namespace used by all the builtin plugins that ship with Zeek have changed to use Zeek::.
  • Any Broker topic names used in scripts shipped with Zeek that previously were prefixed with bro/ are now prefixed with zeek/ instead. In the case where external applications were using a bro/ topic to send data into a Bro process, a Zeek process still subscribes to those topics in addition to the equivalently named zeek/ topic. In the case where external applications were using a bro/ topic to subscribe to remote messages or query data stores, there's no backwards compatibility and external applications must be changed to use the new zeek/ topic. The NEWS have a list of the most common topic names that one may need to change.
  • The Broxygen component, which is used to generate our Doxygen-like scripting API documentation, has been renamed to Zeekygen. This likely has no breaking or visible changes for most users, except in the case one used it to generate their own documentation via the --broxygen flag, which is now named --zeekygen. Besides that, various documentation in scripts has also been updated to replace Sphinx cross-referencing roles and directives like :bro:see: with :zeek:see:.


Upgrading to the Zeek Package Manager


The external package manager switched its name as well, from bro-pkg to zkg. On PyPI, both the old bro-pkg and new zkg packages share the same code-base, so you may continue using bro-pkg if you want, but it’s easy enough to switch for sake of consistency: run pip uninstall bro-pkg && pip install zkg.  Either way, a wrapper script is provided that forwards from bro-pkg to zkg


    Renaming External Packages  


    It's up to a package’s maintainer whether they want to rename a package that’s been using “bro” in its name—there’s nothing about such a package name that will be incompatible with Zeek 3.0.0. If you do want to rename your package, we recommend the following process, assuming it’s hosted on GitHub:
    1. Rename your GitHub repository from bro-foo to zeek-foo. GitHub will automatically provide a redirect from the old URL to the new URL, so people who had installed a package using the old URL will still be fine going forward.
    2. Add an alias to the package’s metadata: aliases = zeek-foo bro-foo. This tells zkg that old and new names are referring to the same package, and it will create corresponding symlinks so that explicit @load bro-foo directives will continue to work. See the documentation for more on aliases.
    3. Optionally, update the depends metadata field. The special dependencies zeek and zkg are replacing bro and bro-pkg, respectively, and zkg treats them as aliases. Note, however, that existing bro-pkg installations won’t recognize the new names yet, so you might want to leave them in there to support users who have not yet upgraded. See the documentation for more.
    4. Re-register the renamed package, zeek-foo with central package source. Follow the normal directions to update your index file: remove the old URL for bro-foo and add the URL for zeek-foo.


    Common Issues When Upgrading 


    • If you were running Bro as the bro user and intend to use a zeek user now, don't forget to remove/update any potential cron jobs you may have.
    • If you're installing Zeek on an old Bro host, remember to first shut down the old cluster using broctl.
    • Symptoms of overlapping Bro/Zeek installations:
      • Plugins may have failing symbol problems depending on if you run Zeek or Bro.
      • zkg packages may fail to install with an error that btest can't find init-bare.bro.  This may be caused by certain packages using an old version of the get-bro-env script or bro_dist metadata substitution in combination with having the bro-pkg/zkg configuration set to use a mismatched Bro/Zeek sourcetree. 
    • Not remembering to update zkg configuration (i.e. updating the paths in ~/.zkg/config or ~/.bro-pkg/config in case you’re now using a different source/installation path for Zeek 3.0.0)
    • Not updating PATH environment variable (to either remove an old /usr/local/bro path or to add the new /usr/local/zeek path)
    • Plugins will generally need to be recompiled for Zeek 3.0.0 (as is usually the case with new versions). Plugins that require --bro-dist have been seen to have build issues. The best solution is to switch the plugin to the new skeleton code. However, we will try to address any specific issues if you file a ticket with instructions on how to reproduce.
    • If you run the BHR scripts, you may need to change those to run as the zeek user as well as the permissions on the queue directory.
    • Not remembering to update both where an external processes (e.g. cron job) writes Intel files into the old installation tree and where the Intel configuration (e.g. Intel::read_files) expects to read such files in the case you choose to use the new default installation path. e.g. if Intel was previously written to /usr/local/bro and you now want to use /usr/local/zeek, remember to update both the Zeek configuration and whatever external process may be writing the Intel files.

    Contributors


    Thanks to Mike Dopheide, Jon Siwek, and Justin Azoff for contributing to this blog posting.

    Thursday, September 12, 2019

    Zeek Week to Gather Expert Users and Developers from Around the World to Showcase New Zeek Technology Innovations and Enhancements



    The leading event for open-source Zeek network security monitor comes to Seattle

    San Francisco, Calif. – Sept. 12, 2019 – Zeek Week 2019 (formerly BroCon), the most important community event for users, developers, incident responders, threat hunters and security architects who rely on the open-source Zeek network security monitor, today announced a full lineup of speakers with areas of expertise including DNSSEC protocol parsing, MITRE ATT&CK-based analytics, SSL/TLS encryption, Zeek performance optimization, and incident response.

    The week will kick off with a keynote from Freddy Dezeure, founder and former head of CERT-EU. A renowned expert in cybersecurity and cyber risk management, Dezeure has held a variety of management positions with the European Commission for more than 20 years. He set up the EU Computer Emergency Response Team (CERT-EU) in 2011, and in that time it has grown to one of the most respected CERTs in Europe.

    Dezeure’s keynote, “Threats are Changing, So are We as Defenders,” will present insights into the current attack trends used by adversaries, their motives and techniques, and the challenges these create for enterprises.

    “The changing threat landscape requires us to continuously adapt our defenses to mitigate the risk to our organizations - and society as a whole - to an acceptable level,” said Dezeure of his keynote topic. “Complacency is not an option.”

    Zeek Week, presented by the Zeek open-source community and hosted by Corelight, providers of the most powerful network visibility solution for cybersecurity, is an annual user conference featuring technical talks, demonstrations and discussions about the project, its many applications, and its future.

    “This past year we have seen a major rise in innovation across the Zeek user community and we are excited to highlight many of these new uses cases and developments at Zeek Week,” said Dr. Vern Paxson, Zeek creator and co-founder of Corelight. “In the more than two decades since Zeek was created, the technology has thrived, thanks in part to a dedicated and growing user community that has augmented the platform with powerful new functionality.

    “I look forward to this gathering every year because it provides the single greatest opportunity to learn how open source Zeek is transforming network traffic analysis for thousands of users and organizations around the globe,” added Paxson.

    Zeek Week content and sessions are focused on the ever-evolving cybersecurity landscape and how Zeek is helping organizations across the public and private sectors by providing better data and network traffic analytics. In addition, this year’s conference will include announcement of the winners of the Zeek Package Contest, which will award the creators of five of the most innovative and useful open source Zeek packages that extend Zeek’s threat hunting and detection capabilities.

    The full agenda is now live and scheduled speakers includes:

    • Vlad Grigorescu, ESnet
    • Mark Fernandez, The MITRE Corporation
    • Jim Mellander, Lawrence Berkeley National Laboratory
    • Robin Sommer, Corelight and Zeek Leadership Team
    • Fatema Bannat Wala, University of Delaware
    • Jordi Ros-Giralt, Reservoir Labs
    • Justin Azoff, Corelight
    • Michal Purzynski, Mozilla Corporation and Zeek Leadership Team
    • Adam Pumphrey, Nimbus LLC
    • Seth Hall, Corelight and Zeek Leadership Team
    • Aashish Sharma, Lawrence Berkeley National Laboratory and Zeek Leadership Team
    • Justin Kohler, Gigamon
    • Jason Lu, Gigamon
    • Johanna Amann, ICSI, Corelight and Zeek Leadership Team
    • Nick Skelsey, Secure Network
    • Keith Lehigh, Indiana University and Zeek Leadership Team
    • Amber Graner, Corelight


    Zeek Week 2019 will take place at the King St. Ballroom & Perch at Embassy Suites by Hilton in Seattle, Wash., October 8-11. For additional information, or to register, visit https://www.zeekweek.com.

    Zeek Week 2019 is generously sponsored by Bricata, Humio, AlphaSOC, Reservoir Labs, BluVector, Gigamon, and Brim Security.


    About Zeek

    Zeek (formerly known as Bro) is a powerful open-source network analysis framework that is much different from the typical IDS you may know. While focusing on network security monitoring, Zeek provides a comprehensive platform for more general network traffic analysis as well. For more information, visit https://www.zeek.org.

    Thursday, August 8, 2019

    Zeek 3.0.0 RC1 released

    (Note: We will update this blog posting for the final release.  Please provide feedback on anything that would be helpful to add.)



    We just published a release candidate for Zeek 3.0.0—our first major release since Bro 2.0 came out in 2012. This version is quite special as it undertakes The Big Zeekification™: It is executing on the technical side of the name change that we announced last year by now renaming the tool itself, including binaries, scripts, and even some events. “Bro” is now “Zeek.” 

    This name change brings some disruption for existing users, which is unavoidable for a long-term codebase where the original name had more than 20 years to proliferate into pretty much every corner. Nevertheless, we have been trying hard to maintain backwards compatibility from Zeek 3.0.0 to Bro 2.6 as much as possible to facilitate smooth upgrades. Wherever we reasonably could, we put aliases and redirects in place so that old names remain working in parallel to the new ones. When using the old names, you will in many cases see explicit deprecation warnings that point you to the places that need updating. These transition mechanisms will remain in place for the Zeek 3.0.x series. We’ll remove them with the next feature release 3.1.0 and likewise with the next long-term stable release 4.0.0, in accordance with our new release schedule.

    Below is a more detailed summary of the main changes coming with the renaming. In addition, Zeek 3.0.0 comes with a number of new features as well, including:

    • New analyzers for NTP and MQTT, and extended analyzers for DNS (SPF/DNSSEC), RDP, SMB, and TLS. 
    • Support for decapsulating VXLAN tunnels.
    • Support for logging in UTF8.
    • Several extensions of the scripting language:
    • Closures for anonymous functions
      • Iteration over key/value pairs of a table through for ( key, value in t ) ...)
      • Python-style vector slicing (v[2:4])
      • A new data structure, paraglob, for efficiently matching strings against large list of globs.
    • See the NEWS file for more detailed release notes, and CHANGES for the complete list of changes

    Upgrading to Zeek 3


    The following summarizes the main naming-related changes that you will encounter after installing Zeek 3.0.0. Unless otherwise noted, the Bro 2.6 names and paths will continue to work with this release, but often trigger deprecation warnings.

    • The names of all executables that had “bro” in their name have changed: bro -> zeek, bro-config -> zeek-config, broctl -> zeekctlbro-cut -> zeek-cut. Zeek 3.0.0 installs wrappers under the old names that will let them continue to work.
    • The default install prefix is now /usr/local/zeek instead of /usr/local/bro. If your existing installation used the previous default and you are using the new default when upgrading, we'll symlink /usr/local/zeek to /usr/local/bro. Certain subdirectories get similar treatment: share/bro, include/bro, and lib/bro.
    • Along with BroControl becoming ZeekControl, installation directories and files with broctl in their name have changed to use zeekctl instead. However, these changes remain backwards compatible with previous Bro installations by continuing to pull from existing locations where customizations might have been made. For example, if you have a broctl.cfg file from a previous installation, installing Zeek over it will retain that file and even symlink the new zeekctl.cfg to it.
    • The new extension for Zeek scripts is .zeek. This leads to two major changes:
      • All scripts ending in .bro have been renamed to .zeek. In particular, $prefix/share/bro/site/local.bro has been renamed to local.zeek. However, if you have an existing local.bro file from a previous Bro installation—possibly with customizations made to it—Zeek will install a symlink local.zeek file that points to that pre-existing local.bro. In that case, you may want to just copy local.bro into the new local.zeek location to avoid confusion, but things should generally also work properly without intervention.
      • The search logic for the @load script directive now prefers files ending in .zeek, but will still fallback to loading a .bro file if it exists. E.g. @load foo will first check for a foo.zeek file to load and then otherwise foo.bro. Note that @load foo.bro (with the explicit .bro file suffix) prefers the opposite order: it first checks for foo.bro and then falls back to a foo.zeek, if that exists.
    • Changes affecting scripts:
      • The events bro_init, bro_done, and bro_script_loaded are now deprecated; use zeek_init, zeek_done, and zeek_script_loaded instead. Any existing event handlers for the deprecated versions will automatically alias to the new events such that existing code will not break, but their usage will emit deprecation warnings.
      • The functions bro_is_terminating and bro_version function are deprecated and replaced by functions named zeek_is_terminating and zeek_version. The old names likewise continue to work with deprecation warnings.
    • The namespace used by all the builtin plugins that ship with Zeek have changed to use Zeek::.
    • Any Broker topic names used in scripts shipped with Zeek that previously were prefixed with bro/ are now prefixed with zeek/ instead. In the case where external applications were using a bro/ topic to send data into a Bro process, a Zeek process still subscribes to those topics in addition to the equivalently named zeek/ topic. In the case where external applications were using a bro/ topic to subscribe to remote messages or query data stores, there's no backwards compatibility and external applications must be changed to use the new zeek/ topic. The NEWS have a list of the most common topic names that one may need to change.
    • The Broxygen component, which is used to generate our Doxygen-like scripting API documentation, has been renamed to Zeekygen. This likely has no breaking or visible changes for most users, except in the case one used it to generate their own documentation via the --broxygen flag, which is now named --zeekygen. Besides that, various documentation in scripts has also been updated to replace Sphinx cross-referencing roles and directives like :bro:see: with :zeek:see:.


    Upgrading to the Zeek Package Manager


    The external package manager switched its name as well, from bro-pkg to zkg. On PyPI, both the old bro-pkg and new zkg packages share the same code-base, so you may continue using bro-pkg if you want, but it’s easy enough to switch for sake of consistency: run pip uninstall bro-pkg && pip install zkg.  Either way, a wrapper script is provided that forwards from bro-pkg to zkg


    Renaming External Packages  


    It's up to a package’s maintainer whether they want to rename a package that’s been using “bro” in its name—there’s nothing about such a package name that will be incompatible with Zeek 3.0.0. If you do want to rename your package, we recommend the following process, assuming it’s hosted on GitHub:
    1. Rename your GitHub repository from bro-foo to zeek-foo. GitHub will automatically provide a redirect from the old URL to the new URL, so people who had installed a package using the old URL will still be fine going forward.
    2. Add an alias to the package’s metadata: aliases = zeek-foo bro-foo. This tells zkg that old and new names are referring to the same package, and it will create corresponding symlinks so that explicit @load bro-foo directives will continue to work. See the documentation for more on aliases.
    3. Optionally, update the depends metadata field. The special dependencies zeek and zkg are replacing bro and bro-pkg, respectively, and zkg treats them as aliases. Note, however, that existing bro-pkg installations won’t recognize the new names yet, so you might want to leave them in there to support users who have not yet upgraded. See the documentation for more.
    4. Re-register the renamed package, zeek-foo with central package source. Follow the normal directions to update your index file: remove the old URL for bro-foo and add the URL for zeek-foo.


    Common Issues When Upgrading 


    • If you were running Bro as the bro user and intend to use a zeek user now, don't forget to remove/update any potential cron jobs you may have.
    • If you're installing Zeek on an old Bro host, remember to first shut down the old cluster using broctl.
    • Symptoms of overlapping Bro/Zeek installations:
      • Plugins may have failing symbol problems depending on if you run Zeek or Bro.
      • zkg packages may fail to install with an error that btest can't find init-bare.bro.  This may be caused by certain packages using an old version of the get-bro-env script or bro_dist metadata substitution in combination with having the bro-pkg/zkg configuration set to use a mismatched Bro/Zeek sourcetree. 
    • Not remembering to update zkg configuration (i.e. updating the paths in ~/.zkg/config or ~/.bro-pkg/config in case you’re now using a different source/installation path for Zeek 3.0.0)
    • Not updating PATH environment variable (to either remove an old /usr/local/bro path or to add the new /usr/local/zeek path)
    • Plugins will generally need to be recompiled for Zeek 3.0.0 (as is usually the case with new versions). Plugins that require --bro-dist have been seen to have build issues. The best solution is to switch the plugin to the new skeleton code. However, we will try to address any specific issues if you file a ticket with instructions on how to reproduce.
    • If you run the BHR scripts, you may need to change those to run as the zeek user as well as the permissions on the queue directory.
    • Not remembering to update both where an external processes (e.g. cron job) writes Intel files into the old installation tree and where the Intel configuration (e.g. Intel::read_files) expects to read such files in the case you choose to use the new default installation path. e.g. if Intel was previously written to /usr/local/bro and you now want to use /usr/local/zeek, remember to update both the Zeek configuration and whatever external process may be writing the Intel files.

    Feedback 


    We realize that we may have missed some places where the name change can impact existing setups. We need your help to close those gaps: if you’re running into any issues upgrading from Bro 2.6 to Zeek 3.0.0, please let us know. If it’s something that we can/should fix, please file a ticket on GitHub. If you have advice for others on how to adapt their setups, scripts, or packages, please leave a comment on this blog posting or email the Zeek mailing list. We’ll be updating this blog posting once the final 3.0.0 release comes out.


    Contributors


    Thanks to Mike Dopheide, Jon Siwek, and Justin Azoff for contributing to this blog posting.