Fix all the bugs

American Fuzzy Lop and Address Sanitizer

Hanno Böck

Introduction

Hanno Böck, freelance journalist and hacker.

Writing for Golem.de and others.

Author of monthly Bulletproof TLS Newsletter.

Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative.

Bug example

(QT file src/plugins/platforminputcontexts/compose/qcomposeplatforminputcontext.cpp)

KDE / QT bugs

Use after free in qmake

Underflow / out of bounds read in QT / QCompose

Out of bounds read in QtGui

Out of bounds read in Kwin

(last 2 not sure if I should blame Xorg/xcb API)

For fairness: GNOME

#762417: Out of bounds read in glib / token parser

#762483: Out of bounds read in glib / unicode parser

#768441: Heap overflow in gnome-session parameter parsing

#770027: Out of bounds read in pango / test suite

Bug example

/* +2 for our new arguments,
	+1 for NULL */
new_argv = g_malloc (argc + 3 *
	sizeof (*argv));

(gnome-session 3.20.1, bug #768441)

Address Sanitizer (ASAN)

All this bugs can be trivially found with Address Sanitizer.

Just add -fsanitize=address to the compiler flags in GCC/CLANG.

Find bugs with ASAN

./configure CFLAGS="-fsanitize=address -g" \
	CXXFLAGS="-fsanitize=address -g" \
	LDFLAGS="-fsanitize=address"
make
make check

What is ASAN doing?

Shadow memory tracking which memory areas are valid.

Finds out of bounds access (read/write) and use after free bugs (and other less common issues).

Out of bounds read

#include <stdio.h>
int main() {
	int a[2] = {3,1};
	int i = 2;
	printf("%i\n", a[i]);
}

Use after free

#include <stdio.h>
#include <stdlib.h>
int main() {
	char *c = calloc(10,1);
	printf("%i\n", c[0]);
	free(c);
	printf("%i\n", c[1]);
}

Mission: Test everyting with ASAN

Every project using C/C++ code should test with ASAN.

Gentoo with ASAN

Why not build everything in a Linux system with ASAN?

Gentoo + ASAN: It runs!

Found bugs in Bash, Coreutils/Shred, man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail, ProFTPD ICU, TCL, Dovecot, Glib, GNOME, Qt, KDE, Libarchive, Squid, CMake, Gettext, SpamAssassin, ...

Fuzzing

Throw garbage at software.

Fuzzing

Example: Image parser

Take valid image, add random errors to it, see if parser crashes.

Darpa Cyber Grand challenge

Rarely told story

Most teams and all three winners of the Darpa Cyber Grand Challenge used American Fuzzy Lop (AFL) with some addons as a bug finding tool.

American Fuzzy Lop

Fuzzing strategies

Dumb fuzzing: Easy, but not very effective.

Template-based fuzzing: More effective, lots of work, doesn't scale.

Coverage-based fuzzing: Easy and effective.

American Fuzzy Lop

American Fuzzy Lop (AFL) made the idea of coverage-based fuzzing popular.

Step 1: Compile with afl-wrapper (afl-gcc or afl-clang-fast)

Step 2: Fuzz

AFL is easy

./configure CC=afl-clang-fast CXX=afl-clang-fast++ --disable-shared; make

[put sample file into directory in/]

afl-fuzz -i in -o out [path_to_parser_executable] @@

American Fuzzy Lop

AFL found bugs in ...

OpenSSL, OpenSSH, libjpeg-turbo, libpng, sqlite, GnuPG, Bash, Stagefright, BIND, NTPD, ...

There isn't any major piece of C parser code where AFL hasn't found bugs.

Fuzzing with superpowers

AFL finds bugs, ASAN finds more bugs.

Best to combine AFL and ASAN.

Set AFL_USE_ASAN=1 and add "-m none".

AFL/ASAN meet Heartbleed

Could Fuzzing find the Heartbleed bug?

Experiment: Implement wrapper that accepts handshake messages as file input.

Success after ~ 6 hours.

(Kostya Serebryany showed that LibFuzzer finds it in 5 minutes)

LibFuzzer

Also coverage based fuzzing.

Part of LLVM/CLANG.

AFL fuzzes executables, LibFuzzer fuzzes functions.

Faster, but more initial work (write code).

LibFuzzer example

#include <stdint.h>
#include <stddef.h>
#include <openssl/asn1.h>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
	ASN1_STRING *out = 0;
	ASN1_mbstring_copy(&out, Data, Size, MBSTRING_BMP, 0);
	if (out!=0) ASN1_STRING_free(out);
	return 0;
}

Differential testing

Typical fuzzing: look for crashes / memory corruption.

Differential testing: Feed two different implementations doing the same thing the same input, compare the output.

Differential testing on math

Crypto is important.

Crypto uses mathematics - but is the math correct?

BN_sqr() bug (CVE-2014-3570)

OpenSSL had a bug in the squaring function.

On very rare inputs (1 in 2^128) it produced wrong results.

Surprising: AFL found this bug (first tested by Ralph-Philipp Weinmann).

AFL is good at this

OpenSSL / BN_mod_exp (CVE-2015-3193)

Nettle / ECC (CVE-2015-8803, CVE-2015-8804)

NSS / mp_div() / exptmod() (CVE-2016-1938)

OpenSSL / Poly1305

MatrixSSL / pstm_exptmod (CVE-2016-6885, CVE-2016-6886, CVE-2016-6887)

What is a vulnerabilit?

The vast majority of bugs found with AFL+ASAN are heap out of bounds reads.

Are these vulnerabilities? Sometimes (Heartbleed!)

Be prepared for pointless discussions whether these should be called vulnerabilities.

IMHO: Just fix them and skip that discussion.

More Tools

Other Sanitizers

Undefined Behavior Sanitizer (UBSAN) - easy to use, but finds many bugs, mostly not very interesting.

Memory Sanitizer (MSAN) - finds uninitialized memory, tricky to use.

Thread Sanitizer (TSAN) - mostly interesting for larger C++ projects.

Undefined behavior sanitizer (UBSAN)

#include <limits.h>
int main() {
	int i = 10;
	int j = -1;
	i <<= j;
	i = INT_MAX;
	i++;
}

Memory Sanitizer (MSAN)

int main(int argc, char **argv) {
	int x[10];
	x[1] = 1;
	if (x[argc]) return 1;
}

KASAN, KUBSAN, KTSAN, Syzcaller

Sanitizers and coverage-based fuzzing have been adapted for the Linux Kernel.

Network-fuzzing

Tricky - no really good solution yet.

Preeny - uses LD_PRELOADing.

Patch from Doug Birdwell for AFL, fragile.

Wrappers to parser functions.

AFL + symbolic execution

Some work on this (e. g. in Darpa Challenge), but nothing easily usable yet.

Will have to proove it's useful.

It's free

All presented tools (AFL, LibFuzzer, ASAN, other Sanitizers, Preeny, KASAN, Syzcaller) are published as Free and Open Source Software.

The C/C++ problem

Most fuzzing/ASAN-related bugs are typical C/C++ problems.

Maybe we should just rewrite everything in Rust.

Comparing vendor reactions

DPKG

DPKG

2015-11-18: Reported 2 bugs in .deb parsing

2015-11-26: Debian and Ubuntu publish updates and security advisories (USN-2820-1, DSA-3407-1)

RPM

RPM

2015-11-20: Reported 3 bugs in .rpm parsing to Red Hat

Answer: We already got 30 crash reports, may take some time.

RPM...

RPM is an independent project since 2007, used by Red Hat, Suse and others.

Or not? rpm.org belongs to Red Hat Inc.

Red Hat Security: "However, we don't own rpm.org domain, it's upstream project, so there's not much we can do about it."

rpm.org

Trac installation.

Trying to register account: Certificate error.

To create a bug you should ask for permission in IRC or on the mailing list.

rpm repository

RPM development happens on Github these days.

The rpm.org webpage does not mention that.

What's the latest version of RPM?

According to rpm.org/releases: 4.12.0.1

According to Github repository: 4.12.0

According to Fedora: 4.13.0

Status RPM

One Stack Buffer Overflow still unfixed in the latest Git Code.

No release (that can be found) since 2014.

There are more bugs, including ones that happen pre-signature-check.

Advertisement block: BerlinSec Meetup

Tomorrow (5th Sept) at Mozilla Berlin Community space

https://berlinsec.github.io/

Thanks for listening

Test with Address Sanitizer

Fuzz your software

Questions?

https://fuzzing-project.org/