Linux - BPF Tools -> high performance packet filtering

eBPF - Berkeley Packet Filter
 enables programmers to write code which gets executed in kernel space in a more secure and restricted environment. Yet this environment enables them to create tools which otherwise would require writing a new kernel module.

eBPF introduces a new syscall, bpf(2). This syscall is used for all eBPF operations like loading programs, attaching them to certain events, creating eBPF maps and access the map contents from tools.


echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main

# echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main

echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list



root@sam:~# apt-get update
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Get:2 http://dl.google.com/linux/chrome/deb stable Release [943 B]
Get:3 http://dl.google.com/linux/chrome/deb stable Release.gpg [819 B]
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
Err:3 http://dl.google.com/linux/chrome/deb stable Release.gpg
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 78BD65473CB3BD13
Hit:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Ign:6 https://repo.iovisor.org/apt/xenial xenial-nightly InRelease
Hit:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:8 https://repo.iovisor.org/apt/xenial xenial-nightly Release
Hit:9 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Ign:10 https://repo.iovisor.org/apt/xenial xenial-nightly Release.gpg
Fetched 819 B in 1s (1,022 B/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://dl.google.com/linux/chrome/deb stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 78BD65473CB3BD13
W: Failed to fetch http://dl.google.com/linux/chrome/deb/dists/stable/Release.gpg  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 78BD65473CB3BD13
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@sam:~# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 78BD65473CB3BD13
Executing: /tmp/apt-key-gpghome.0CeQj9iCln/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-key 78BD65473CB3BD13
gpg: key 7721F63BD38B4796: 8 duplicate signatures removed
gpg: key 7721F63BD38B4796: 18 signatures not checked due to missing keys
gpg: key 7721F63BD38B4796: "Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>" 17 new signatures
gpg: key 7721F63BD38B4796: "Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>" 1 new subkey
gpg: Total number processed: 1
gpg:            new subkeys: 1
gpg:         new signatures: 17
root@sam:~# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 78BD65473CB3BD13
Executing: /tmp/apt-key-gpghome.gI4uDLvk1F/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-key 78BD65473CB3BD13
gpg: key 7721F63BD38B4796: 8 duplicate signatures removed
gpg: key 7721F63BD38B4796: 18 signatures not checked due to missing keys
gpg: key 7721F63BD38B4796: "Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
root@sam:~# apt-get update
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Get:2 http://dl.google.com/linux/chrome/deb stable Release [943 B]
Get:3 http://dl.google.com/linux/chrome/deb stable Release.gpg [819 B]
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
Ign:5 https://repo.iovisor.org/apt/xenial xenial-nightly InRelease
Hit:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Get:7 http://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,109 B]
Hit:8 https://repo.iovisor.org/apt/xenial xenial-nightly Release
Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Ign:10 https://repo.iovisor.org/apt/xenial xenial-nightly Release.gpg
Hit:11 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Fetched 1,928 B in 1s (2,201 B/s)
Reading package lists... Done
root@sam:~#

Install BCC tools
root@sam:/usr/share# apt-get install binutils bcc bcc-tools libbcc-examples python-bcc

Add man path and command path
# export MANPATH=$MANPATH:/usr/share/bcc/man
# export PATH=$PATH:/usr/share/bcc/tools
Test with python command

root@sam:/usr/share# python /usr/share/bcc/examples/hello_world.py

# python /usr/share/bcc/examples/tracing/task_switch.py
.................
task_switch[ 4055->    0]=322
task_switch[    0->   34]=1
task_switch[ 3757->29449]=1
task_switch[17211->    0]=5
task_switch[ 4037->17325]=1
task_switch[ 4382-> 4385]=2
task_switch[    0->  487]=1
task_switch[    0-> 4079]=15
task_switch[ 4084->17326]=1
task_switch[ 2346->    0]=1
task_switch[ 3757-> 4385]=1
task_switch[32002-> 3757]=2
task_switch[17327-> 3757]=1


Some BCC tools in ubuntu
1. ardist command displays function marameter values as a histogram or frequency count

root@sam:/usr/share/bcc/tools# ./argdist -h

histogram of buffer sizes passed to the write() function across the system

root@sam:/usr/share/bcc/tools# ./argdist -c -H 'p:c:write(int fd, void *buf, size_t len):size_t:len'

2. biosnop - traces block device I/O (disk I/O), and prints a line of output per I/O
root@sam:/usr/share/bcc/tools# ./biosnoop

root@sam:/usr/share/bcc/tools# iostat -x 1
Command 'iostat' not found, but can be installed with:
apt install sysstat
root@sam:/usr/share/bcc/tools#

3. biolatency - traces block device I/O (disk I/O), and records the distribution of I/O latency
root@sam:/usr/share/bcc/tools# ./biolatency

4. ext4slower - shows I/O slower
root@sam:/usr/share/bcc/tools# ./ext4slower 1

5. bashreadline tool - prints bash commands from all running bash shells on the system
root@sam:/usr/share/bcc/tools# bashreadline
TIME      PID    COMMAND
03:23:13  8277   sh myfile.sh

6. biotop - block device I/O top
root@sam:/usr/share/bcc/tools# biotop


Got idea from
https://opensource.com/article/17/11/bccbpf-performance
http://www.brendangregg.com/bpf-performance-tools-book.html
https://www.amazon.com/gp/reader/0136554822
http://www.brendangregg.com/blog/2019-07-15/bpf-performance-tools-book.html
https://linoxide.com/linux-how-to/bcc-kernel-tracing-tools-linux-performance/
https://github.com/iovisor/bcc
https://www.redhat.com/en/blog/introduction-ebpf-red-hat-enterprise-linux-7