09: FAT and NTFS

Drives on computers

Depending upon your OS, you might have graphical tools and/or command-line tools built into the system to view drives on your computer. For example, on OS X I can use “Disk Utility” to examine the drives that are mountable on a system (demo w/ laptop and USB drive). But it doesn’t show me everything; “System Information” shows a more comprehensive view. (Note “BSD Name:”, more on this in a bit.)

On a Linux system, there are various ways to see what drives are available. mount shows which drives (partitions, actually) are mounted. (OS X supports this command too.) During boot, the kernel shows a list of hardware as it’s detected. You can use dmesg to view this log after-the-fact. (Demo on virtual machine and/or elnux.)

PCs generally often have some sort of “boot menu” that lets you see what drives are attached, too, using the BIOS (or these days, UEFI running on built-in firmware that runs on the computer’s motherboard) but I can’t demo that on my Mac.

And of course, we can turn to the sleuthkit to interrogate a drive, like my Mac’s HD, and a USB:

sudo mmls /dev/rdisk0
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Safety Table
001:  -------   0000000000   0000000039   0000000040   Unallocated
002:  Meta      0000000001   0000000001   0000000001   GPT Header
003:  Meta      0000000002   0000000033   0000000032   Partition Table
004:  000       0000000040   0000409639   0000409600   EFI system partition
005:  001       0000409640   0585210495   0584800856   Iron
006:  002       0585210496   0586480031   0001269536   Recovery HD
007:  -------   0586480032   0586481663   0000001632   Unallocated
008:  003       0586481664   0976842879   0390361216   Apple_HFS_Untitled_2
009:  -------   0976842880   0977105059   0000262180   Unallocated

sudo mmls /dev/rdisk2
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000000031   0000000032   Unallocated
002:  000:000   0000000032   0003913727   0003913696   DOS FAT16 (0x06)

(More on /dev/rdisk[n] in a moment)

You can think of hard disk drives (old-style spinning metal or newer SSDs) as a giant array of bytes. This obviously isn’t the most useful view all the time (where did I put my program? At byte 3D7A149F?) but it is the most fundamental way to view the drive.

Unix-like OSes (including OS X) expose disk drives as a file, stored in a special location. Most devices that can be exposed as files are in the /dev directory, and disks are no exception.

For example, on OS X, the raw bytes of my disk drives are accessible /dev/rdisk* (and the block-buffered bytes at /dev/disk* – these are the “BSD Names”). Linux has similar devices at /dev/sda*. You can directly access these if you like. Here’s the first 512 bytes of my Mac’s hard disk:

sudo hexdump -Cv -n 512 /dev/rdisk0

00000000  33 c0 8e d0 bc 00 7c 8e  c0 8e d8 be 00 7c bf 00  |3.....|......|..|
00000010  06 b9 00 02 fc f3 a4 50  68 1c 06 cb fb b9 04 00  |.......Ph.......|
00000020  bd be 07 80 7e 00 00 7c  0b 0f 85 0e 01 83 c5 10  |....~..|........|
00000030  e2 f1 cd 18 88 56 00 55  c6 46 11 05 c6 46 10 00  |.....V.U.F...F..|
00000040  b4 41 bb aa 55 cd 13 5d  72 0f 81 fb 55 aa 75 09  |.A..U..]r...U.u.|
00000050  f7 c1 01 00 74 03 fe 46  10 66 60 80 7e 10 00 74  |....t..F.f`.~..t|
00000060  26 66 68 00 00 00 00 66  ff 76 08 68 00 00 68 00  |&fh....f.v.h..h.|
00000070  7c 68 01 00 68 10 00 b4  42 8a 56 00 8b f4 cd 13  ||h..h...B.V.....|
00000080  9f 83 c4 10 9e eb 14 b8  01 02 bb 00 7c 8a 56 00  |............|.V.|
00000090  8a 76 01 8a 4e 02 8a 6e  03 cd 13 66 61 73 1c fe  |.v..N..n...fas..|
000000a0  4e 11 75 0c 80 7e 00 80  0f 84 8a 00 b2 80 eb 84  |N.u..~..........|
000000b0  55 32 e4 8a 56 00 cd 13  5d eb 9e 81 3e fe 7d 55  |U2..V...]...>.}U|
000000c0  aa 75 6e ff 76 00 e8 8d  00 75 17 fa b0 d1 e6 64  |.un.v....u.....d|
000000d0  e8 83 00 b0 df e6 60 e8  7c 00 b0 ff e6 64 e8 75  |......`.|....d.u|
000000e0  00 fb b8 00 bb cd 1a 66  23 c0 75 3b 66 81 fb 54  |.......f#.u;f..T|
000000f0  43 50 41 75 32 81 f9 02  01 72 2c 66 68 07 bb 00  |CPAu2....r,fh...|
00000100  00 66 68 00 02 00 00 66  68 08 00 00 00 66 53 66  |.fh....fh....fSf|
00000110  53 66 55 66 68 00 00 00  00 66 68 00 7c 00 00 66  |SfUfh....fh.|..f|
00000120  61 68 00 00 07 cd 1a 5a  32 f6 ea 00 7c 00 00 cd  |ah.....Z2...|...|
00000130  18 a0 b7 07 eb 08 a0 b6  07 eb 03 a0 b5 07 32 e4  |..............2.|
00000140  05 00 07 8b f0 ac 3c 00  74 09 bb 07 00 b4 0e cd  |......<.t.......|
00000150  10 eb f2 f4 eb fd 2b c9  e4 64 eb 00 24 02 e0 f8  |......+..d..$...|
00000160  24 02 c3 49 6e 76 61 6c  69 64 20 70 61 72 74 69  |$..Invalid parti|
00000170  74 69 6f 6e 20 74 61 62  6c 65 00 45 72 72 6f 72  |tion table.Error|
00000180  20 6c 6f 61 64 69 6e 67  20 6f 70 65 72 61 74 69  | loading operati|
00000190  6e 67 20 73 79 73 74 65  6d 00 4d 69 73 73 69 6e  |ng system.Missin|
000001a0  67 20 6f 70 65 72 61 74  69 6e 67 20 73 79 73 74  |g operating syst|
000001b0  65 6d 00 00 00 63 7b 9a  7d 6e 00 00 00 00 00 fe  |em...c{.}n......|
000001c0  ff ff ee fe ff ff 01 00  00 00 a3 70 3d 3a 00 00  |...........p=:..|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

and of my virtual machine’s hard disk:

00000000  eb 63 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.c..............|
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.........|...t..|
00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.....|.........|
00000050  00 00 00 00 00 00 00 00  00 00 00 80 01 00 00 00  |................|
00000060  00 00 00 00 ff fa 90 90  f6 c2 80 74 05 f6 c2 70  |...........t...p|
00000070  74 02 b2 80 ea 79 7c 00  00 31 c0 8e d8 8e d0 bc  |t....y|..1......|
00000080  00 20 fb a0 64 7c 3c ff  74 02 88 c2 52 bb 17 04  |. ..d|<.t...R...|
00000090  f6 07 03 74 06 be 88 7d  e8 17 01 be 05 7c b4 41  |...t...}.....|.A|
000000a0  bb aa 55 cd 13 5a 52 72  3d 81 fb 55 aa 75 37 83  |..U..ZRr=..U.u7.|
000000b0  e1 01 74 32 31 c0 89 44  04 40 88 44 ff 89 44 02  |..t21..D.@.D..D.|
000000c0  c7 04 10 00 66 8b 1e 5c  7c 66 89 5c 08 66 8b 1e  |....f..\|f.\.f..|
000000d0  60 7c 66 89 5c 0c c7 44  06 00 70 b4 42 cd 13 72  |`|f.\..D..p.B..r|
000000e0  05 bb 00 70 eb 76 b4 08  cd 13 73 0d 5a 84 d2 0f  |...p.v....s.Z...|
000000f0  83 d0 00 be 93 7d e9 82  00 66 0f b6 c6 88 64 ff  |.....}...f....d.|
00000100  40 66 89 44 04 0f b6 d1  c1 e2 02 88 e8 88 f4 40  |@f.D...........@|
00000110  89 44 08 0f b6 c2 c0 e8  02 66 89 04 66 a1 60 7c  |.D.......f..f.`||
00000120  66 09 c0 75 4e 66 a1 5c  7c 66 31 d2 66 f7 34 88  |f..uNf.\|f1.f.4.|
00000130  d1 31 d2 66 f7 74 04 3b  44 08 7d 37 fe c1 88 c5  |.1.f.t.;D.}7....|
00000140  30 c0 c1 e8 02 08 c1 88  d0 5a 88 c6 bb 00 70 8e  |0........Z....p.|
00000150  c3 31 db b8 01 02 cd 13  72 1e 8c c3 60 1e b9 00  |.1......r...`...|
00000160  01 8e db 31 f6 bf 00 80  8e c6 fc f3 a5 1f 61 ff  |...1..........a.|
00000170  26 5a 7c be 8e 7d eb 03  be 9d 7d e8 34 00 be a2  |&Z|..}....}.4...|
00000180  7d e8 2e 00 cd 18 eb fe  47 52 55 42 20 00 47 65  |}.......GRUB .Ge|
00000190  6f 6d 00 48 61 72 64 20  44 69 73 6b 00 52 65 61  |om.Hard Disk.Rea|
000001a0  64 00 20 45 72 72 6f 72  0d 0a 00 bb 01 00 b4 0e  |d. Error........|
000001b0  cd 10 ac 3c 00 75 f4 c3  de 9c 09 00 00 00 80 20  |...<.u......... |
000001c0  21 00 83 a9 23 65 00 08  00 00 df f7 ff 04 00 00  |!...#e..........|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

and of a USB key:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  60 9d b9 ec 00 00 00 00  |........`.......|
000001c0  21 00 06 2a ea ca 20 00  00 00 e0 b7 3b 00 00 00  |!..*.. .....;...|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

There are some similarities: you can see some sort of error message (presumably displayed if this disk cannot boot) on the bootable drives and you’ll notice the end of the 512 bytes are zero-padded, but end with the values 55 aa. Not a coincidence!

Interpreting the data: Disk volumes

“Volumes” are organized by “volume systems”, assembled from one or more disks, usually but not always 1:1. A volume is a collection of addressable “sectors” (sometimes also: “blocks”) which are the smallest unit of addressing that most disk controllers support (older disks: 512 bytes; newer disks can be larger (4Kn or Advanced Format) but have backward-compatibility translation layers(!!)) that appear continuous to the OS.

Volumes are usually divided into one or more “partitions”. A partition names a particular collection of consecutive sectors in a volume. For example, my Mac’s HD has two user-visible partitions (show in Disk Utility), as well as two that are not visible through the GUI (show in System Report):

EFI:
  Capacity: 209.7 MB (209,715,200 bytes)
  BSD Name: disk0s1
  Content:  EFI
  Volume UUID:  BDC1974F-6B8C-3DAE-9DB2-3AA3C17BF506
disk0s2:
  Capacity: 299.42 GB (299,418,038,272 bytes)
  BSD Name: disk0s2
  Content:  Apple_CoreStorage
Recovery HD:
  Capacity: 650 MB (650,002,432 bytes)
  BSD Name: disk0s3
  Content:  Apple_Boot
  Volume UUID:  42320CA6-833A-3699-AAC3-64080DD45FF8
Cobalt:
  Capacity: 199.86 GB (199,864,942,592 bytes)
  Available:    198.94 GB (198,938,521,600 bytes)
  Writable: Yes
  File System:  Journaled HFS+
  BSD Name: disk0s4
  Mount Point:  /Volumes/Cobalt
  Content:  Apple_HFS
  Volume UUID:  42E09BD5-9547-36E4-9879-4BB9C01484EB

And here’s a partition on the USB key:

MARC'S USB:
  Capacity: 2 GB (2,003,812,352 bytes)
  Available:    2 GB (2,001,010,688 bytes)
  Writable: Yes
  File System:  MS-DOS FAT16
  BSD Name: disk2s1
  Mount Point:  /Volumes/MARC'S USB
  Content:  DOS_FAT_16
  Volume UUID:  8117D4F3-4CD8-38D5-8B94-75D720C736A5

Different operating systems and devices use different methods of partitioning drives and labeling them, though most modern consumer PC hardware and OSes (Windows, Mac, Linux) these days are converging on two standards: the newer Unified Extensible Firmware Interface’s GUID Partition Table, and legacy support for Master Boot Records (notably, on basically every USB thumb drive and SD storage card in existence, as well as older PCs). There are other formats, but understanding these two will give you the basis to understand others if you need to.

If you want to carve out individual partitions, you can use mmls to find their offsets, then dd to read/write them (note you can pass a skip or iseek parameter to dd). But usually you would just operate on the entire image; most toolsets (TSK included) have other, volume-aware tools to deal with individual partitions, as we showed in our demo in Lecture 2.

The Master Boot Record

The older “master boot record” (MBR) style of partitioning (sometimes called “DOS” style, as it’s the OS that popularized this format) has no formal standard, though it’s well documented in many places since it’s so ubiquitious.

In its simplest form, a MBR contains three things:

boot code (literally: it’s executable code that the BIOS will load then execute)
a partition table
a signature

Note that modern MBRs on HDDs and SSDs don’t contain bootcode – the firmware handles this now, and the bootcode area is used for other information instead. We’ll talk more about this when we get to UEFI. (See https://en.wikipedia.org/wiki/Master_boot_record#Sector_layout.) And USBs don’t usually contain bootcode either.

The signature value we’ve seen already: The last two bytes of a sector containing a MBR are 55 AA; this is another “magic number” not unlike the SOI/EOI tags in a JPEG file. Seeing it is not a guarantee of a valid MBR, but not seeing it means it’s definitely not an MBR.

A disk with three partitions has an MBR then three partitions, and the MBR has entries for each of the three partitions: (on board). These entries are stored in the partition table.

Each of the (at most four) entries contains the following (https://en.wikipedia.org/wiki/Master_boot_record#PTE):

status/flags (1 byte)
starting CHS address (3 bytes)
type (1 byte)
ending CHS address (3 bytes)
starting LBA sector (4 bytes)
number of sectors (4 bytes)

CHS is a legacy system (when disk locations where referred to by Cylinder / Head / Sector, see https://en.wikipedia.org/wiki/Cylinder-head-sector if you’re curious) and basically irrelevant these days. These values are often hardcoded to various marker values that indicate that LBA address should be used instead, e.g., (1023, 254, 63) or (1023, 255, 63).

The type is a value you look up in a table (e.g., https://en.wikipedia.org/wiki/Partition_type) though different OSes and tools might name different types slightly differently due to the lack of a standard here.

The LBA sector is an offset from the start of the disk, in terms of 512-byte sectors, to the start of the partition, and the number of sectors tells you how bit that partition is. Easy-peasy.

A note about partitions and slack

There’s nothing in the MBR that implies all sectors will be part of a partition. Space that’s not allocated is not accessible under OSes. One could imagine deleting a partition entry in an MBR to “hide” it from examination, and putting it back in when you wanted to access the data.

GPTs

Modern PCs and Macs use the GUID Partition Table (GPT) standard on fixed hard disks. The GPT is a more sophisticated system that parallels the MBR in many ways but fixes many of its problems.

A GPT-formatted disk is laid out as follows:

GUID partition table

Note that the GPT is duplicated at the start and end of disk, and supposed to be kept in sync. This is a backup to mitigate data corruption.

Also note that there’s a “protective MBR”. What’s that about? If you put a GPT-formatted disk into a MBR-only computer, it contains dummy values (a single maximum size partition of type 0xEE) so that the disk shows up as formatted. Hey, what were those bytes on my Mac’s “MBR”? Remember, the first partition starts at 0x1BE:

hexdump -Cv -n 512 disk-image.dd 
00000000  33 c0 8e d0 bc 00 7c 8e  c0 8e d8 be 00 7c bf 00  |3.....|......|..|
00000010  06 b9 00 02 fc f3 a4 50  68 1c 06 cb fb b9 04 00  |.......Ph.......|
00000020  bd be 07 80 7e 00 00 7c  0b 0f 85 0e 01 83 c5 10  |....~..|........|
00000030  e2 f1 cd 18 88 56 00 55  c6 46 11 05 c6 46 10 00  |.....V.U.F...F..|
00000040  b4 41 bb aa 55 cd 13 5d  72 0f 81 fb 55 aa 75 09  |.A..U..]r...U.u.|
00000050  f7 c1 01 00 74 03 fe 46  10 66 60 80 7e 10 00 74  |....t..F.f`.~..t|
00000060  26 66 68 00 00 00 00 66  ff 76 08 68 00 00 68 00  |&fh....f.v.h..h.|
00000070  7c 68 01 00 68 10 00 b4  42 8a 56 00 8b f4 cd 13  ||h..h...B.V.....|
00000080  9f 83 c4 10 9e eb 14 b8  01 02 bb 00 7c 8a 56 00  |............|.V.|
00000090  8a 76 01 8a 4e 02 8a 6e  03 cd 13 66 61 73 1c fe  |.v..N..n...fas..|
000000a0  4e 11 75 0c 80 7e 00 80  0f 84 8a 00 b2 80 eb 84  |N.u..~..........|
000000b0  55 32 e4 8a 56 00 cd 13  5d eb 9e 81 3e fe 7d 55  |U2..V...]...>.}U|
000000c0  aa 75 6e ff 76 00 e8 8d  00 75 17 fa b0 d1 e6 64  |.un.v....u.....d|
000000d0  e8 83 00 b0 df e6 60 e8  7c 00 b0 ff e6 64 e8 75  |......`.|....d.u|
000000e0  00 fb b8 00 bb cd 1a 66  23 c0 75 3b 66 81 fb 54  |.......f#.u;f..T|
000000f0  43 50 41 75 32 81 f9 02  01 72 2c 66 68 07 bb 00  |CPAu2....r,fh...|
00000100  00 66 68 00 02 00 00 66  68 08 00 00 00 66 53 66  |.fh....fh....fSf|
00000110  53 66 55 66 68 00 00 00  00 66 68 00 7c 00 00 66  |SfUfh....fh.|..f|
00000120  61 68 00 00 07 cd 1a 5a  32 f6 ea 00 7c 00 00 cd  |ah.....Z2...|...|
00000130  18 a0 b7 07 eb 08 a0 b6  07 eb 03 a0 b5 07 32 e4  |..............2.|
00000140  05 00 07 8b f0 ac 3c 00  74 09 bb 07 00 b4 0e cd  |......<.t.......|
00000150  10 eb f2 f4 eb fd 2b c9  e4 64 eb 00 24 02 e0 f8  |......+..d..$...|
00000160  24 02 c3 49 6e 76 61 6c  69 64 20 70 61 72 74 69  |$..Invalid parti|
00000170  74 69 6f 6e 20 74 61 62  6c 65 00 45 72 72 6f 72  |tion table.Error|
00000180  20 6c 6f 61 64 69 6e 67  20 6f 70 65 72 61 74 69  | loading operati|
00000190  6e 67 20 73 79 73 74 65  6d 00 4d 69 73 73 69 6e  |ng system.Missin|
000001a0  67 20 6f 70 65 72 61 74  69 6e 67 20 73 79 73 74  |g operating syst|
000001b0  65 6d 00 00 00 63 7b 9a  7d 6e 00 00 00 00 00 fe  |em...c{.}n......|
000001c0  ff ff ee fe ff ff 01 00  00 00 a3 70 3d 3a 00 00  |...........p=:..|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|

00 fe ff ff *ee* fe ff ff 01 00 00 00 a3 70 3d 3a

ee! Well, there you go then.

Where’s the actual GPT data? It starts in a header in the next sector.

https://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_table_header_.28LBA_1.29

dd if=disk-image.dd bs=512 count=1 skip=1 | hexdump -Cv
1+0 records in
1+0 records out
512 bytes transferred in 0.000016 secs (31580642 bytes/sec)
00000000  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000010  7f 95 63 62 00 00 00 00  01 00 00 00 00 00 00 00  |..cb............|
00000020  a3 70 3d 3a 00 00 00 00  22 00 00 00 00 00 00 00  |.p=:....".......|
00000030  82 70 3d 3a 00 00 00 00  a8 4d 00 00 93 38 00 00  |.p=:.....M...8..|
00000040  3d 11 00 00 f3 62 00 00  02 00 00 00 00 00 00 00  |=....b..........|
00000050  80 00 00 00 80 00 00 00  81 d5 7a 4e 00 00 00 00  |..........zN....|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200

Note, for example, we have the right header (EFI PART), and that the partition entries (partition table) offset is located at byte 0x48 and is equal to 2 (that is, it’s at the next sector on the disk). There’s also a 4-byte value for the number of entries and the size of each entry at 0x50 and 0x54 (very much like the IFD). In this image, there are 128 entries (little endian: 08 00 00 00) each 128 bytes long (which is typical).

Let’s pull out the next sector to see the first few entries:

dd if=disk-image.dd bs=512 count=1 skip=2 | hexdump -Cv
1+0 records in
1+0 records out
512 bytes transferred in 0.000478 secs (1071599 bytes/sec)
00000000  28 73 2a c1 1f f8 d2 11  ba 4b 00 a0 c9 3e c9 3b  |(s*......K...>.;|
00000010  e3 22 00 00 4b 2b 00 00  1e 43 00 00 72 0b 00 00  |."..K+...C..r...|
00000020  28 00 00 00 00 00 00 00  27 40 06 00 00 00 00 00  |(.......'@......|
00000030  00 00 00 00 00 00 00 00  45 00 46 00 49 00 20 00  |........E.F.I. .|
00000040  73 00 79 00 73 00 74 00  65 00 6d 00 20 00 70 00  |s.y.s.t.e.m. .p.|
00000050  61 00 72 00 74 00 69 00  74 00 69 00 6f 00 6e 00  |a.r.t.i.t.i.o.n.|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  72 6f 74 53 67 61 aa 11  aa 11 00 30 65 43 ec ac  |rotSga.....0eC..|
00000090  41 5a 00 00 e1 30 00 00  8a 74 00 00 df 60 00 00  |AZ...0...t...`..|
000000a0  28 40 06 00 00 00 00 00  7f 9a e1 22 00 00 00 00  |(@........."....|
000000b0  00 00 00 00 00 00 00 00  49 00 72 00 6f 00 6e 00  |........I.r.o.n.|
000000c0  00 00 6d 00 65 00 72 00  00 00 00 00 00 00 00 00  |..m.e.r.........|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  74 6f 6f 42 00 00 aa 11  aa 11 00 30 65 43 ec ac  |tooB.......0eC..|
00000110  40 0b 46 a0 6e 7e 24 42  91 02 75 57 32 ab c9 dd  |@.F.n~$B..uW2...|
00000120  80 9a e1 22 00 00 00 00  9f f9 f4 22 00 00 00 00  |..."......."....|
00000130  00 00 00 00 00 00 02 00  52 00 65 00 63 00 6f 00  |........R.e.c.o.|
00000140  76 00 65 00 72 00 79 00  20 00 48 00 44 00 00 00  |v.e.r.y. .H.D...|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 53 46 48 00 00 aa 11  aa 11 00 30 65 43 ec ac  |.SFH.......0eC..|
00000190  c0 34 9b 30 09 0b fb 41  95 02 8b 95 10 1f 4e 41  |.4.0...A......NA|
000001a0  00 00 f5 22 00 00 00 00  7f 70 39 3a 00 00 00 00  |...".....p9:....|
000001b0  00 00 00 00 00 00 00 00  41 00 70 00 70 00 6c 00  |........A.p.p.l.|
000001c0  65 00 5f 00 48 00 46 00  53 00 5f 00 55 00 6e 00  |e._.H.F.S._.U.n.|
000001d0  74 00 69 00 74 00 6c 00  65 00 64 00 5f 00 32 00  |t.i.t.l.e.d._.2.|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200

Entries look like: https://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_entries

Let’s look at the first few:

00000000  28 73 2a c1 1f f8 d2 11  ba 4b 00 a0 c9 3e c9 3b  |(s*......K...>.;|
00000010  e3 22 00 00 4b 2b 00 00  1e 43 00 00 72 0b 00 00  |."..K+...C..r...|
00000020  28 00 00 00 00 00 00 00  27 40 06 00 00 00 00 00  |(.......'@......|
00000030  00 00 00 00 00 00 00 00  45 00 46 00 49 00 20 00  |........E.F.I. .|
00000040  73 00 79 00 73 00 74 00  65 00 6d 00 20 00 70 00  |s.y.s.t.e.m. .p.|
00000050  61 00 72 00 74 00 69 00  74 00 69 00 6f 00 6e 00  |a.r.t.i.t.i.o.n.|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Each entry starts with a 16-byte partition type GUID. This one’s is: 28 73 2a c1 1f f8 d2 11 ba 4b 00 a0 c9 3e c9 3b. Hmm, it doesn’t seem to be in the list at https://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_type_GUIDs. What might it be? Turns out it’s: an EFI System Partition (spoiler: the name is stored as UTF-16LE text!). https://en.wikipedia.org/wiki/EFI_system_partition. Also, the entry really is there – look at it and consider endianness.

Bootable drives in UEFI systems put the bootloader and associated utilities on this partition, which is FAT-like (though actually standardized as part of UEFI, so not exactly FAT). Different OSes put different stuff here; it’s a standardized, extensible way to get more than the 446 bytes the MBR provided for a full-featured boot loader (and whatever else your OS provider wants to give you).

At 0x10, the next 16 bytes are a unique-to-this-partition-on-this-computer GUID.

At 0x20, there are 8 bytes (LE) telling you the first LBA of this partition, then 8 bytes telling you the last (inclusive).

struct.unpack('<QQ', bytes.fromhex('28 00 00 00 00 00 00 00  27 40 06 00 00 00 00 00'))
# => (40, 409639)

Again, does that match what mmls told us?

mmls /dev/rdisk0
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Safety Table
001:  -------   0000000000   0000000039   0000000040   Unallocated
002:  Meta      0000000001   0000000001   0000000001   GPT Header
003:  Meta      0000000002   0000000033   0000000032   Partition Table
004:  000       0000000040   0000409639   0000409600   EFI system partition
005:  001       0000409640   0585210495   0584800856   Iron
006:  002       0585210496   0586480031   0001269536   Recovery HD
007:  -------   0586480032   0586481663   0000001632   Unallocated
008:  003       0586481664   0976842879   0390361216   Apple_HFS_Untitled_2
009:  -------   0976842880   0977105059   0000262180   Unallocated

Yup, exactly right. Then 8 bytes of “attribute flags,” with only the first three bits being universally defined so far, and the last 16 defined by each partition type. Finally at 0x38 the partition name, in 72 bytes of UTF-16LE encoded text.

The next one looks like:

00000080  72 6f 74 53 67 61 aa 11  aa 11 00 30 65 43 ec ac  |rotSga.....0eC..|
00000090  41 5a 00 00 e1 30 00 00  8a 74 00 00 df 60 00 00  |AZ...0...t...`..|
000000a0  28 40 06 00 00 00 00 00  7f 9a e1 22 00 00 00 00  |(@........."....|
000000b0  00 00 00 00 00 00 00 00  49 00 72 00 6f 00 6e 00  |........I.r.o.n.|
000000c0  00 00 6d 00 65 00 72 00  00 00 00 00 00 00 00 00  |..m.e.r.........|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

This is an Apple partition, and you can see it’s named “Iron”, which is my Mac’s primary partition’s name.

If we kept going, you’d see that unused entries are zeroed out (that is, they consist entirely of bytes set to the value 00).

So you get the idea here, anyway. GPTs are a fairly straightforward binary data structure to store hard disk partition information. They’re slightly more complex than MBRs, but they don’t require the contortions of MBRs (which is a legacy format originally designed for computers and disks over 30 years ago!).

So that’s it for lecture content on volumes and partitions, though you will probably want to Read the Book, supplemented with more recent web resources, for the fine details of MBRs and GPTs.

Filesystems

Filesystems are provided by operating systems to programs (and ultimately, to users). They provide a mechanism to store data in a structured way, usually organized around “files” and “directories”.

The filesystem doesn’t worry about the internal structure of files (like the JPEGs we’ve dealt with); it treats them as blobs of bytes to be stored and retrieved when requested.

The filesystem instead manages the files that are stored and the space in which they are stored. Minimally, a file system must be able to store, retrieve, and remove data from a partition corresponding to a particular file. To do so, it must keep track of:

free and allocated space within the partition that it is responsible for;
the location of each file within the partition
the user-visible name of each file (the filename)
the user-visible location of each file (the path to the file)

Carrier refers to these items as essential and therefore “more trustworthy” in some sense than nonessential items. The reasoning here is that the above must be correct, or the filesystem isn’t fulfilling its basic purpose.

Filesystems might also track many other items of metadata including:

creation, modification, or last-accessed time
ownership information
permissions (readable, writeable, executable, by who?)

Carrier provides a broad overview of the analysis techniques you might use at an abstract level in Chapter 8; we’re going to jump right into the details of the FAT filesystem and see them as we go along.

FAT

The File Allocation Table (FAT) filesystem is simple (maybe too simple, in some ways) and extraordinarily widespread. While to doesn’t see much use on new computers, it was the domininant filesystem on PCs from the early 80s through the late 90s / early 00s.

Being simple and nearly ubiquitious, virtually every OS (not just Microsoft’s) had and have full support for it.

It lives on as the filesystem on virtually every USB thumb drive and SD-card (either FAT16, FAT32, or “exFAT”), and as a variant in the EFI System partition on every bootable GPT volume.

There are several variants you might encounter: FAT16, FAT32, and exFAT, with the latter two more common now (as FAT16 cannot support partitions larger than 2GB, and even “small” USB thumbdrives are generally larger than this nowadays).

FAT allocates storage in units called “clusters”; a cluster is the minimum allocable unit on a FAT filesystem. Clusters are a number of sectors; the number must be a power of two.

FAT layout

FAT manages files, directories, and free space using two data structures:

The directory entry, which contains the file (or directory) name, size, starting address, and other metadata; and
The file allocation table, which has two functions. First, it tracks the allocation status of clusters (that is, is part of a file stored at a given cluster?). Second, if a file is split across more than one cluster, it lets you find the clusters after the first (the first’s address being stored in the directory entry).

At a high level, FAT filesystems have three parts:

the reserved area, which stores essential data the describes the remainder of the filesystem,
the FAT area, where one or more (typically two) copies of the FAT structures are stored, and
the data area, where directory entries and file data are stored.

Let’s look at each of these in turn.

Reserved area

Let’s do these on the reserved area of my USB key. Remember, we can use mmls to view the partition map, or parse the MBR (or GPT) ourselves.

mmls /dev/rdisk4
Password:
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000000001   0000000002   Unallocated
002:  000:000   0000000002   0003911679   0003911678   DOS FAT16 (0x06)```

So the FAT partition begins on sector 2 of the drive. Let’s extract it and view it:

dd if=/dev/rdisk4 bs=512 skip=2 count=1 | hexdump -Cv

1+0 records in
1+0 records out
512 bytes transferred in 0.000573 secs (893668 bytes/sec)
00000000  eb 3c 90 6d 6b 66 73 2e  66 61 74 00 02 40 01 00  |.<.mkfs.fat..@..|
00000010  02 00 04 00 00 f8 00 01  3e 00 3e 00 02 00 00 00  |........>.>.....|
00000020  fe af 3b 00 80 00 29 08  47 d5 4b 4d 41 52 43 27  |..;...).G.KMARC'|
00000030  73 20 55 53 42 20 46 41  54 31 36 20 20 20 0e 1f  |s USB FAT16   ..|
00000040  be 5b 7c ac 22 c0 74 0b  56 b4 0e bb 07 00 cd 10  |.[|.".t.V.......|
00000050  5e eb f0 32 e4 cd 16 cd  19 eb fe 54 68 69 73 20  |^..2.......This |
00000060  69 73 20 6e 6f 74 20 61  20 62 6f 6f 74 61 62 6c  |is not a bootabl|
00000070  65 20 64 69 73 6b 2e 20  20 50 6c 65 61 73 65 20  |e disk.  Please |
00000080  69 6e 73 65 72 74 20 61  20 62 6f 6f 74 61 62 6c  |insert a bootabl|
00000090  65 20 66 6c 6f 70 70 79  20 61 6e 64 0d 0a 70 72  |e floppy and..pr|
000000a0  65 73 73 20 61 6e 79 20  6b 65 79 20 74 6f 20 74  |ess any key to t|
000000b0  72 79 20 61 67 61 69 6e  20 2e 2e 2e 20 0d 0a 00  |ry again ... ...|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

Note that like in the MBR, multibyte values in the FAT boot sector are little-endian.

0x0, 3 bytes: The reserved area starts with machine code that (in older bootable FAT partitions) tells the CPU how far ahead to JMP to the next part of the boot loader.

0x3, 8 bytes: Then 8 bytes of ASCII-coded OEM. “mkfs.fat”

0xB, 2 bytes: bytes per sector “00 02” -> 512

0xD, 1 byte: sectors per cluster “40” -> 64 (so a cluster is 64 * 512 = 32KB on this USB key)

0xE, 2 bytes: size of the reserved area, in sectors “01 00” -> 1

0x10, 1 byte: # FATs “02”

0x11 2 bytes: (0 in FAT32) (in FAT16: max # files in root directory) “00 04” -> 1024

0x13 2 bytes: # sectors; if doesn’t fit in 2 bytes, should be zero “00 00 “

0x15 1 byte: media type (0xf8 fixed disk, 0xf0 removable) “f8”

0x16 2 bytes: (in FAT32: 0) (in FAT16: size in sectors of the FAT) “00 01” -> 256

0x18 2 bytes: sectors per track “3e 00”

0x1A 2 bytes: # heads “3e 00”

0x1C 4 bytes: # sectors before start of partition (from beginning of disk, or extended partition) “02 00 00 00” -> 2

0x20 4 bytes: # sectors in file system (will be zero if bytes at 0x13 are non-zero, and vice versa) “FE AF 3B 00” -> 3911678

0x24 1 byte: BIOS INT13h drive number (see https://en.wikipedia.org/wiki/INT_13H; not used these days)

0x25 1 byte: not used

0x26 1 byte: signature for validity of next entries; 0x29 if valid.

0x27 4 bytes: volume (partition) serial number

0x2b 11 bytes: volume (partition) label, ASCII

0x36 8 bytes: FS type label in ASCII – usually “FAT16”

0x3e–0x1FE: unused by FAT; sometimes contains boot code (it might be what’s jumped to by the JMP instructions)

0x1FE (2 bytes): signature value (55 AA)

This boot sector is the only thing in its reserved area; the reserved area in FAT32 might contain a few more things (including a FSINFO structure) but we’ll skip those details here.

We can use fsstat to view this information via Sleuthkit. We have to tell it where the filesystem if we access the volume directly, as it doesn’t do partition table parsing itself:

fsstat -o 2 /dev/rdisk4
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT16

OEM Name: mkfs.fat
Volume ID: 0x4bd54708
Volume Label (Boot Sector): MARC's USB 
Volume Label (Root Directory): MARC's USB 
File System Type Label: FAT16   

Sectors before file system: 2

File System Layout (in sectors)
Total Range: 0 - 3911677
* Reserved: 0 - 0
** Boot Sector: 0
* FAT 0: 1 - 256
* FAT 1: 257 - 512
* Data Area: 513 - 3911677
** Root Directory: 513 - 576
** Cluster Area: 577 - 3911616
** Non-clustered: 3911617 - 3911677

METADATA INFORMATION
--------------------------------------------
Range: 2 - 62578646
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 32768
Total Cluster Range: 2 - 61111

FAT CONTENTS (in sectors)
--------------------------------------------
577-640 (64) -> EOF
641-704 (64) -> EOF
705-768 (64) -> EOF
...more output...

Immediately following the reserved area (whose size is described above) comes the FATs themselves, with no header or footer.

File Allocation Tables

The FATs come next. In our example, there are 2 of them, each 256 sectors long. In FAT16, they are a sequence of 16-bit entries, one per cluster in the data area.

Entries corresponding to unallocated clusters store the value zero.

Entries corresponding to file/directories that are the last (including only) cluster allocated to the file contain any value greater than or equal to 0xFFF8 (again, little endian).

Entries containing 0xFFF7 indicate a bad sector (modern hard drives usually handle this, though it might matter if someone was attempting to hide data).

Other values tell you the next cluster of a file (or directory), given the current cluster.

Weirdly, the address of the first cluster in the data area is 2. And so the first two entries of the FAT don’t correspond to clusters; they are usually used store a copy of the media type, and the “dirty status” — whether the disk was ejected properly and/or the OS was shut down before poweroff.

Let’s look at the start of our FAT:

dd if=/dev/rdisk4 bs=512 skip=3 count=1 |hexdump -Cv

1+0 records in
1+0 records out
512 bytes transferred in 0.000652 secs (785473 bytes/sec)
00000000  f8 ff ff 7f ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000010  ff ff ff ff 00 00 ff ff  00 00 ff ff ff ff ff ff  |................|
00000020  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000030  00 00 1a 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000040  ff ff ff ff ff ff 24 00  ff ff 26 00 ff ff 28 00  |......$...&...(.|
00000050  29 00 2a 00 ff ff 2c 00  2d 00 2e 00 ff ff 30 00  |).*...,.-.....0.|
00000060  ff ff 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000070  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000080  ff ff ff ff ff ff 00 00  ff ff ff ff ff ff ff ff  |................|
00000090  ff ff ff ff ff ff ff ff  00 00 4e 00 ff ff 00 00  |..........N.....|
000000a0  00 00 ff ff ff ff 00 00  00 00 ff ff 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200

fsstat shows us some of this too, but note all TSK tools use sectors instead of clusters, so the offsets differ.

fsstat -o 2 /dev/rdisk4
...other stuff comes first...

FAT CONTENTS (in sectors)
--------------------------------------------
577-640 (64) -> EOF
641-704 (64) -> EOF
705-768 (64) -> EOF
769-832 (64) -> EOF
833-896 (64) -> EOF
897-960 (64) -> EOF
961-1024 (64) -> EOF
1025-1088 (64) -> EOF
1153-1216 (64) -> EOF
1281-1344 (64) -> EOF
1345-1408 (64) -> EOF
1409-1472 (64) -> EOF
1473-1536 (64) -> EOF
1537-1600 (64) -> EOF
1601-1664 (64) -> EOF
1665-1728 (64) -> EOF
1729-1792 (64) -> EOF
1793-1856 (64) -> EOF
1857-1920 (64) -> EOF
1921-1984 (64) -> EOF
2049-2176 (128) -> EOF
2177-2240 (64) -> EOF
2241-2304 (64) -> EOF
2305-2368 (64) -> EOF
2369-2432 (64) -> EOF
2433-2496 (64) -> EOF
2497-2560 (64) -> EOF
2561-2624 (64) -> EOF
2625-2688 (64) -> EOF
2689-2816 (128) -> EOF
2817-2944 (128) -> EOF
2945-3200 (256) -> EOF
3201-3456 (256) -> EOF
3457-3584 (128) -> EOF
3649-3712 (64) -> EOF
3713-3776 (64) -> EOF
3777-3840 (64) -> EOF
3841-3904 (64) -> EOF
3905-3968 (64) -> EOF
3969-4032 (64) -> EOF
4033-4096 (64) -> EOF
4097-4160 (64) -> EOF
4161-4224 (64) -> EOF
4225-4288 (64) -> EOF
4289-4352 (64) -> EOF
4353-4416 (64) -> EOF
4417-4480 (64) -> EOF
4481-4544 (64) -> EOF
4545-4608 (64) -> EOF
4609-4672 (64) -> EOF
4673-4736 (64) -> EOF
4801-4864 (64) -> EOF
4865-4928 (64) -> EOF
4929-4992 (64) -> EOF
4993-5056 (64) -> EOF
5057-5120 (64) -> EOF
5121-5184 (64) -> EOF
5185-5248 (64) -> EOF
5249-5312 (64) -> EOF
5377-5504 (128) -> EOF
5633-5696 (64) -> EOF
5697-5760 (64) -> EOF
5889-5952 (64) -> EOF

More on FAT

Today we’re going to finish up talking about the FAT structure then move on to the last piece of FAT, the directory entries.

Let’s look at the adams.dd disk image way back from lecture 2. First, note that it’s not an image of an entire disk:

> mmls adams.dd
Cannot determine partition type

Though my local version of file gives a hint:

> file adams.dd
adams.dd: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD  4.4", sectors/cluster 2, root entries 512, sectors 10239 (volumes <=32 MB) , sectors/FAT 20, sectors/track 32, heads 16, serial number 0x36c013ef, label: "ADAMS      ", FAT (16 bit)

Looks like someone extracted only the FAT16 partition. So let’s use fsstat to take a look:

FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT16

OEM Name: BSD  4.4
Volume ID: 0x36c013ef
Volume Label (Boot Sector): ADAMS      
Volume Label (Root Directory):
File System Type Label: FAT16   

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 10238
* Reserved: 0 - 0
** Boot Sector: 0
* FAT 0: 1 - 20
* FAT 1: 21 - 40
* Data Area: 41 - 10238
** Root Directory: 41 - 72
** Cluster Area: 73 - 10238

METADATA INFORMATION
--------------------------------------------
Range: 2 - 163174
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 1024
Total Cluster Range: 2 - 5084

FAT CONTENTS (in sectors)
--------------------------------------------
75-76 (2) -> EOF
3743-8792 (5050) -> EOF

Most of this information comes from the boot sector as we described last lecture. There are a few things that don’t, though, namely the Volume Label (Root Directory), the METADATA INFORMATION (well, not exactly), and the FAT CONTENTS.

Let’s take a look at FAT1 (the first FAT in this FAT16 filesystem), which starts at sector 1 and runs through sector 20. I’m going to use dd and pipe into hexdump so that the offsets are from the start of the FAT. Also note I’m going to omit the -v argument, so that identical lines in the hexdump are condensed:

dd if=adams.dd bs=512 skip=1 count=20| hexdump -C
20+0 records in
20+0 records out
10240 bytes transferred in 0.000112 secs (91382283 bytes/sec)
00000000  f0 ff ff ff 00 00 ff ff  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000e50  00 00 00 00 00 00 00 00  00 00 2e 07 2f 07 30 07  |............/.0.|
00000e60  31 07 32 07 33 07 34 07  35 07 36 07 37 07 38 07  |1.2.3.4.5.6.7.8.|

The * means that every line following the previous had the same value (in this case, all 00s).

Let’s pick apart these first few lines, then examine fsstat‘s output to see how it represents them.

First, remember that entries are sequential in the FAT, 16 bits per cluster. So the first two entries are f0 ff and ff ff. But they don’t refer to a cluster, since the clusters start at 2! (Thanks, Microsoft!). They mean different things depending upon the OS. Generally, the first is a copy of the media type field (aka the FAT ID usually F0, followed by an all-ones byte FF) , and the second indicates “dirty status” of the filesystem (in other words, was the disk ejected cleanly?).

The entry for cluster 2 starts at offset 0x4 (and this is generally true in FAT16: you can find the entry in the FAT for cluster N at offset N x 2 from the start of the FAT. Its value here is 00 00, which means the cluster is unallocated: there is no file data stored there.

The next cluster, 3, is represented by the entry at 0x6, and its value is ff ff, indicating it’s the last cluster in whatever it represents. Nothing in the FAT tells you if a cluster is the first (or only) cluster of a file: that information is in the directory entries, which we’ll get to in a bit. The FAT entries are kinda just like entries in a linked list, but the “head” of that list is in the directory entry.

Let’s pause and see how fsstat has represented this allocation in its FAT CONTENTS:

75-76 (2) -> EOF

What’s going on here? Sleuth Kit mostly uses sectors, not clusters, when referring to locations on disk. So it’s telling us that there’s a something that runs from sector 75 through sector 76 (2 sectors), and then ends. Does that match the FAT?

Well, according to fsstat, the Cluster Area starts a sector 73. And clusters in this FAT are 1024 bytes (two sectors). So cluster 2 is sectors 73 and 74, and cluster 3 is on sectors 75 and 76, just as fsstat shows.

Let’s go the other way. fsstat lists the next allocated sectors as:

3743-8792 (5050) -> EOF

Which cluster(s) does this correspond to? Well, the first sector is 3743. That’s the 3670th sector of the cluster area. Since on this filesystem the clusters are two sectors long, it’s the 1835th cluster. But clusters start numbering at 2, so it’s actually cluster 1837. Cluster 1837 will be located at offset 1837 x 2 = 3674 from the start of the FAT, which is 0xe5a. What’s in the hexdump at that offset?

00000e50  00 00 00 00 00 00 00 00  00 00 2e 07 2f 07 30 07  |............/.0.|
00000e60  31 07 32 07 33 07 34 07  35 07 36 07 37 07 38 07  |1.2.3.4.5.6.7.8.|

2e 07. If we interpret that as a little-endian value, what is it? 1838 – the next cluster! What’s there? 2f 07 (1839). And so on. This run of clusters is actually 2525 clusters long, so we’re not going to do it all here, but you get the idea, I hope.

Directory entries

Time for the last piece of the puzzle. How do you store file and directory metadata in a FAT filesystem, and how do you map files to the clusters where their contents are stored? FAT uses a 32-byte structure called the directory entry to handle these tasks. Directory entries are stored in the cluster area just like files.

The contents (files and directories) of a directory (like C:\MARC\) is represented by set of directory entries stored in a cluster (or several clusters, just like a multi-cluster file). Which cluster? The enclosing directory’s directory entry will tell you. Where’s the root directory? In FAT16 its location is hardcoded as right after the FAT, at the start of the data area (but before the first cluster). Its size can be calculated by checking the max number of entries (listed in the FAT boot sector, usually 512) and multiplying by 32 (bytes per directory entry). Cluster 2 follows immediately on disk. To be clear: in FAT16, the root directory are is not part of the cluster area! (In FAT32 it is, but we’re going to ignore mostly ignore FAT32 as it introduces additional complications.)

Directory entries are laid out as follows:

0x0 (11 B): 8.3 filename in ASCII; first byte will be 0x00 or 0xe5 if unallocated; 0xe5 means deleted and 0x00 (usually) means that no later entry in this sequence of directory entries is allocated

0xB (1 B): attributes, ORed together

0x01: read only
0x02: hidden
0x04: system
0x08: volume label
0x10: directory
0x20: archive

(This is a special “long file name” entry, not a directory entry, if its attributes are equal to 0x0f. More on that later.)

0xC (1 B): reserved

0xD (5 B): creation date/time in wacky FAT format

0x12 (2B): accessed date

0x14 (2B): high-order bytes of first cluster address (always 0 in FAT 16, only used in FAT32)

0x16 (4B): written date/time

0x1A (2B): low-order bytes of the first cluster address

0x1C (4B): file size (0 for directories)

I’m going to leave the fine details of parsing the FAT date/time format to Carrier: read the text or relevant Wikipedia entry if and when you need them.

Let’s look at an example. The root directory entries for adams.dd start at sector 41 of the image, so let’s take a look:

dd if=adams.dd bs=512 skip=41 count=32| hexdump -C
32+0 records in
32+0 records out
16384 bytes transferred in 0.000110 secs (149066110 bytes/sec)
00000000  41 44 41 4d 53 20 20 20  20 20 20 28 00 00 00 00  |ADAMS      (....|
00000010  00 00 00 00 00 00 e1 62  1e 39 00 00 00 00 00 00  |.......b.9......|
00000020  41 69 00 6d 00 61 00 67  00 65 00 0f 00 71 73 00  |Ai.m.a.g.e...qs.|
00000030  00 00 ff ff ff ff ff ff  ff ff 00 00 ff ff ff ff  |................|
00000040  49 4d 41 47 45 53 20 20  20 20 20 10 00 00 c4 79  |IMAGES     ....y|
00000050  e1 38 1c 39 00 00 4f 84  1c 39 03 00 00 00 00 00  |.8.9..O..9......|
00000060  41 44 00 65 00 73 00 69  00 67 00 0f 00 d4 6e 00  |AD.e.s.i.g....n.|
00000070  73 00 2e 00 64 00 6f 00  63 00 00 00 00 00 ff ff  |s...d.o.c.......|
00000080  44 45 53 49 47 4e 53 20  44 4f 43 20 00 00 4e 81  |DESIGNS DOC ..N.|
00000090  1c 39 1c 39 00 00 4e 81  1c 39 2d 07 00 72 27 00  |.9.9..N..9-..r'.|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004000

Each entry is 32 bytes (two lines of this hexdump). What type is the first entry? Looking at offset 0x0b, we see the value 28, which means this directory entry is a “Volume label” – “ADAMS” in particular.

Let’s skip to the next one. 0f: a “long file name” (LFN). We’ll come back to this (I promise!) soon.

Next, 10: a directory! The two bytes at offset 0x1a (from the start of this entry) tell us which cluster the directory is stored in. 03 00 means that the IMAGES directory’s entries are stored in cluster 3 of this filesystem. Ring a bell from earlier?

What’s at that location, anyway? Cluster 3 is ((3 - 2) * 2) + 73 sectors past the start of the filesystem, at sector 75:

dd if=adams.dd bs=512 skip=75 count=2| hexdump -C
2+0 records in
2+0 records out
1024 bytes transferred in 0.000023 secs (44278013 bytes/sec)
00000000  2e 20 20 20 20 20 20 20  20 20 20 10 00 00 4e 5c  |.          ...N\|
00000010  a1 38 a1 38 00 00 4e 5c  a1 38 03 00 00 00 00 00  |.8.8..N\.8......|
00000020  2e 2e 20 20 20 20 20 20  20 20 20 10 00 00 4e 5c  |..         ...N\|
00000030  a1 38 a1 38 00 00 4e 5c  a1 38 00 00 00 00 00 00  |.8.8..N\.8......|
00000040  e5 4d 47 5f 33 30 32 37  4a 50 47 20 00 00 c4 79  |.MG_3027JPG ...y|
00000050  e1 38 e1 38 00 00 c4 79  e1 38 04 00 8c a0 1c 00  |.8.8...y.8......|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

The first item is a directory (again, at offset 0xb into the entry, value 10), named “.”, which is shorthand for “the current directory.” It’s at cluster 03 00 – right here.

Next comes an entry for a directory called “..”, which command line nerds know means “the enclosing directory,” in other words, a back reference. Since the enclosing directory is the root directory, which isn’t at a particular cluster in FAT16, this gets the marker value 00 00.

Finally, there’s an entry for a deleted file, which we can identify by the leading e5 value in its name. It used to be located at cluster 04 00 – cluster 4, but it’s gone now. Or is it? Ponder.

Anyway, back to the root directory entries:

00000060  41 44 00 65 00 73 00 69  00 67 00 0f 00 d4 6e 00  |AD.e.s.i.g....n.|
00000070  73 00 2e 00 64 00 6f 00  63 00 00 00 00 00 ff ff  |s...d.o.c.......|
00000080  44 45 53 49 47 4e 53 20  44 4f 43 20 00 00 4e 81  |DESIGNS DOC ..N.|
00000090  1c 39 1c 39 00 00 4e 81  1c 39 2d 07 00 72 27 00  |.9.9..N..9-..r'.|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Next, another LFN.

Finally(!!), a file named DESIGNS.DOC, stored at cluster 2d 07, that is, cluster 1837. Ring a bell from earlier? We know (from the directory entry) that it starts at cluster 1837, and we know from the FAT that it runs contiguously for 2525 clusters. Is it exactly 2525 * 1024 (the cluster size) = 2,585,600 bytes? No, its actual size is stored in the last four bytes of the entry: 00 72 27 00 -> 2,585,088, which fits into the cluster with 512 bytes to spare (these are “slack space”).

NTFS: An overview

NTFS is the default filesystem on all modern versions of Windows; thus, like FAT, it’s in wide use. Unlike FAT, it’s in use as the filesystem that most home and business computers run on.

In some ways NTFS is simpler than FAT: it was designed from the ground up to be extensible, and so its design is more principled (and free of the legacy cruft that encumbers, say, the FAT boot sector and its FAT12/16/32 nonsense).

That’s the good news. The bad news is that the extensibility is not theoretical. Generic on-disk data structures wrap more specific data structures, so that the internals can be updated over time. To understand NTFS, we’ll need to cover both the generic and specific data structures (though not all of them), and that means there’s a lot of details to keep straight in your head. I’ll do my best in lecture, but you are almost certainly going to need to read and re-read Carrier as well to understand this material.

One core concept in NTFS is simple: Everything is (or is stored in) a file. Regular files, directories, the structures that control the filesystem’s layout on disk (like the FAT from FAT16) – all are either files or stored in files. There’s no separate plane of existence for filesystem metadata (like inodes in a UNIX-y filesystem or the FAT + directory entries in FAT16). Certain special files have special names (like what we think of as the “boot sector” from FAT is called $Boot in NTFS), but they’re all just considered files by the file system.

The next core concept in NTFS is that the above is a bit of a cheat. Most (but not all) of what we think of as filesystem metadata is stored in one particular data structure: the Master File Table (MFT), which is stored as a file (named $MFT of course) but contents-wise is analogous to the FATs and dirents, as we’ll see. We’re going to spend a lot of time today and next week talking about the MFT and how it relates to the files stored on disk.

The final high-level thing you need to know about NTFS is that it breaks a disk up into allocatable units called clusters, just like FAT. Just like FAT, clusters are sized as a power-of-two-multiple of the underlying disk sector size. Unlike FAT, though, cluster 0 starts at the beginning of the partition, so there’s none of the “first cluster is cluster number 2” nonsense to contend with.

Finding the MFT

The Master File Table (MFT) contains information about all files and directories in its NTFS. Each file/directory has an MFT entry; the table is just a linear array of MFT entries, numbered with a file number, starting from 0.

How do we find the MFT? Just like in FAT, the first sector of the volume contains a boot sector (which again, in NTFS is just a file, named $Boot). That boot sector encodes the minimal information necessary to understand and parse the volume, including the bytes per sector, sectors per cluster, cluster address of the MFT, and the MFT entry size. Once you can find the MFT, you need to go there to learn the rest of what you need to know about the volume.

Let’s look at an example. Download simple.ntfs and follow along with Table 13.18 on page 380 in Carrier.

00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....|
00000010  00 00 00 00 00 f8 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 80 00 80 00  ff 4f 00 00 00 00 00 00  |.........O......|
00000030  04 00 00 00 00 00 00 00  ff 04 00 00 00 00 00 00  |................|
00000040  f6 00 00 00 01 00 00 00  a4 a4 a1 72 46 d9 dc 42  |...........rF..B|
00000050  00 00 00 00 fa 33 c0 8e  d0 bc 00 7c fb 68 c0 07  |.....3.....|.h..|

The bytes per sector are stored in bytes 11–12. Here it’s 512.

The sectors per cluster are stored in byte 13. Here it’s 8, so clusters are 8 * 512 B = 4 KB clusters.

The cluster address of the MFT is stored in bytes 48–55. Here, it’s 4.

The size of the file record is at byte 64. It (and the size of the index record, at byte 68) is stored in a special format. If, when interpreted as a signed byte it’s positive, then it’s the number of clusters used for that record. If it’s negative, than 2^(abs(value)) bytes are used. Here it’s -10, which means that file records are 1KB each (this is the default value).

Compare this with index records, the size of which are stored in byte 68. Here, it’s 1, which means index records are 4KB (one cluster) long.

So now we can go and find the start of the MFT in the volume. It’s at cluster 4. Cluster 4 is 4 * 4,096 bytes into the file, at offset 0x4000.

Let’s double check against the output of fsstat to see that we’re doing this correctly.

fsstat simple.ntfs
# ... output follows ...

What’s in the MFT?

The MFT is just a sequence of entries. The first 16 are reserved by MS for filesystem metadata information, but in practice it’s the first 24 that are reserved. Table 11.1 shows the contents of the reserved entries.

Entry 0 is an entry for the MFT itself. We need this, because although the boot sector tells us where the MFT starts, it (the MFT) might run across multiple clusters. This entry tells us where to find the rest of the MFT!

Entry 3 is the $Volume information; entry 6 is for the $Bitmap (similar to the FAT, but it only tracks allocation, not runs); entry 7 is for the $Boot sector, and so on.

We’re going to look at one shortly. But before we do, let’s talk a little about the general structure of an MFT entry.

It starts with an MFT entry header, described in detail in Table 13.1. Then there’s a sequence of attribute (header, content) pairs, with (usually) some unused space at the end of the entry.

The attribute header identifies the attribute type, size, and name, among other things.

The attribute contents can have any format and any size: one perhaps obvious use is to store the contents of a file corresponding to the entry. Small attribute contents can fit in the MFT entry (one systems consequence is that small enough – roughly, under 700 B – files don’t automatically waste tons of space, as they do in FAT, since they don’t live in a cluster). These are called resident attributes generally, whether they store files or just other attribute content.

Larger attributes (again, might be files, might be other things) might not fit in the entry; these are called non-resident. Non-resident attributes are stored in clusters. The clusters are identified by runlists. Runlists are just lists of runs of contiguous clusters that hold the file. See Figure 11.6.

We know that the MFT is at 16K into this volume; let’s use some UNIXy tools to pull out the first entry so that we can see offsets from zero in this entry:

dd if=simple.ntfs of=zeroth-mft-entry bs=1024 count=1 skip=16

…and take a look at it.

00000000  46 49 4c 45 30 00 03 00  00 00 00 00 00 00 00 00  |FILE0...........|
00000010  01 00 01 00 38 00 01 00  98 01 00 00 00 04 00 00  |....8...........|
00000020  00 00 00 00 00 00 00 00  04 00 00 00 00 00 00 00  |................|
00000030  03 00 00 00 00 00 00 00  10 00 00 00 60 00 00 00  |............`...|
00000040  00 00 18 00 00 00 00 00  48 00 00 00 18 00 00 00  |........H.......|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000070  06 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  30 00 00 00 68 00 00 00  |........0...h...|
000000a0  00 00 18 00 00 00 02 00  4a 00 00 00 18 00 01 00  |........J.......|
000000b0  05 00 00 00 00 00 05 00  80 64 e1 e7 8b a1 d2 01  |.........d......|
000000c0  80 64 e1 e7 8b a1 d2 01  80 64 e1 e7 8b a1 d2 01  |.d.......d......|
000000d0  80 64 e1 e7 8b a1 d2 01  00 70 00 00 00 00 00 00  |.d.......p......|
000000e0  00 6c 00 00 00 00 00 00  06 00 00 00 00 00 00 00  |.l..............|
000000f0  04 03 24 00 4d 00 46 00  54 00 00 00 00 00 00 00  |..$.M.F.T.......|
00000100  80 00 00 00 48 00 00 00  01 00 40 00 00 00 01 00  |....H.....@.....|
00000110  00 00 00 00 00 00 00 00  12 00 00 00 00 00 00 00  |................|
00000120  40 00 00 00 00 00 00 00  00 30 01 00 00 00 00 00  |@........0......|
00000130  00 04 01 00 00 00 00 00  00 04 01 00 00 00 00 00  |................|
00000140  11 13 04 00 00 00 00 00  b0 00 00 00 48 00 00 00  |............H...|
00000150  01 00 40 00 00 00 03 00  00 00 00 00 00 00 00 00  |..@.............|
00000160  00 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000170  00 10 00 00 00 00 00 00  10 00 00 00 00 00 00 00  |................|
00000180  10 00 00 00 00 00 00 00  11 01 02 00 00 00 00 00  |................|
00000190  ff ff ff ff 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 03 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 03 00  |................|
00000400

See Table 13.1 on page 353.

It starts with the four byte sequence corresponding to ASCII "FILE" (or "BAAD" if there’s an error on the disk in this entry). Let’s find the first attribute – remember, all entries in the MFT consist of an MFT header, followed by attributes (which themselves consist of headers and contents). This first attribute’s offset (from the start of the entry) is stored in bytes 20–21. Here, its value is 56 (0x38).

Attributes: headers and contents

Let’s skip to the attribute, which starts with a header in a standard format. See Tables 13.2–13.4. The first 16 bytes are the same in resident and non-resident attribute headers; after that they diverge.

The header starts with a four-byte type tag. Here, it’s 16, which is a $STANDARD_INFORMATION header. This and $FILE_NAME (48) are two attributes that nearly every entry will have.

The next four bytes (4–7) tell us the length. Here it’s 96; this means the next attribute starts at offset 56 + 96 = 152 (96 from the start of the current attribute’s start).

Byte 8 (offset from 56, remember, so byte 64 in the dump) tells us if the attribute’s content is non-resident. Here it’s zero, so this attribute;s content is resident – that is, it’s embedded in the MFT entry.

(Discussion names for standard attributes? E.g., as ADS?)

Let’s jump ahead to bytes 16–19 (again: offset from 56, so go to 72 0x48) to get the size (72) and bytes 20–21 (offset 76: 0x4c) to get the offset (24) of this attribute’s content. (Sanity check: 72 + 24 = 96, which is the size of the attribute in total and also the offset to the next attribute)

So you get the idea; Tables 13.5 and 13.6 tell you how to parse the $STANDARD_INFORMATION attribute’s contents.

More next class.

NTFS: An re-overview

Boot sector super fast review

The boot sector (a file named $Boot) is always in the zeroth cluster. We can parse it to find the byte offset to the start of the $MFT: (from $BOOT bytes_per_sector, and sectors_per_cluster, MFT_cluster_start). Also we can parse the entry_size, which is almost always 1,024 bytes.

What’s in the MFT?

Entry 3 is the $Volume information; entry 6 is for the $Bitmap (similar to the FAT, but it only tracks allocation, not runs); entry 7 is for the $Boot sector, and so on.

We’re going to look at one shortly. But before we do, let’s talk a little about the general structure of an MFT entry.

It starts with an MFT entry header (42 bytes), described in detail in Table 13.1. Then there’s a sequence of attribute (header, content) pairs, with (usually) some unused space at the end of the entry.

The attribute header (16 byte) identifies the attribute type, size, and name, among other things.

We know that the MFT is at 16K into this volume; let’s use some UNIXy tools to pull out the first entry so that we can see offsets from zero in this entry:

dd if=simple.ntfs of=zeroth-mft-entry bs=1024 count=1 skip=16

…and take a look at it.

00000000  46 49 4c 45 30 00 03 00  00 00 00 00 00 00 00 00  |FILE0...........|
00000010  01 00 01 00 38 00 01 00  98 01 00 00 00 04 00 00  |....8...........|
00000020  00 00 00 00 00 00 00 00  04 00 00 00 00 00 00 00  |................|
00000030  03 00 00 00 00 00 00 00  10 00 00 00 60 00 00 00  |............`...|
00000040  00 00 18 00 00 00 00 00  48 00 00 00 18 00 00 00  |........H.......|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000070  06 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  30 00 00 00 68 00 00 00  |........0...h...|
000000a0  00 00 18 00 00 00 02 00  4a 00 00 00 18 00 01 00  |........J.......|
000000b0  05 00 00 00 00 00 05 00  80 64 e1 e7 8b a1 d2 01  |.........d......|
000000c0  80 64 e1 e7 8b a1 d2 01  80 64 e1 e7 8b a1 d2 01  |.d.......d......|
000000d0  80 64 e1 e7 8b a1 d2 01  00 70 00 00 00 00 00 00  |.d.......p......|
000000e0  00 6c 00 00 00 00 00 00  06 00 00 00 00 00 00 00  |.l..............|
000000f0  04 03 24 00 4d 00 46 00  54 00 00 00 00 00 00 00  |..$.M.F.T.......|
00000100  80 00 00 00 48 00 00 00  01 00 40 00 00 00 01 00  |....H.....@.....|
00000110  00 00 00 00 00 00 00 00  12 00 00 00 00 00 00 00  |................|
00000120  40 00 00 00 00 00 00 00  00 30 01 00 00 00 00 00  |@........0......|
00000130  00 04 01 00 00 00 00 00  00 04 01 00 00 00 00 00  |................|
00000140  11 13 04 00 00 00 00 00  b0 00 00 00 48 00 00 00  |............H...|
00000150  01 00 40 00 00 00 03 00  00 00 00 00 00 00 00 00  |..@.............|
00000160  00 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000170  00 10 00 00 00 00 00 00  10 00 00 00 00 00 00 00  |................|
00000180  10 00 00 00 00 00 00 00  11 01 02 00 00 00 00 00  |................|
00000190  ff ff ff ff 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 03 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 03 00  |................|
00000400

See Table 13.1 on page 353.

Attributes: headers and contents

The header starts with a four-byte type tag. Here, it’s 16, which is a $STANDARD_INFORMATION header. This and $FILE_NAME (48) are two attributes that nearly every entry will have.

The next four bytes (4–7) tell us the length. Here it’s 96; this means the next attribute starts at offset 56 + 96 = 152 (96 from the start of the current attribute’s start).

(Discussion names for standard attributes? E.g., as ADS?)

So you get the idea; Tables 13.5 and 13.6 tell you how to parse the $STANDARD_INFORMATION attribute’s contents.

Let’s look at the next attribute for this entry (remember, we’re still looking at the zeroth entry in the MFT). We find it by skipping past the previous one. The previous one started at offset 56 and had a length of 96, so we need to skip ahead to byte 56 + 96 = 152 (0x98) of the entry.

This one has type 48 ($FILE_NAME). Let’s dig into this attribute. We see it’s 104 (0x68) bytes long, so let’s pull it out using dd so we can see offsets from 0 (or just do it in hexfiend):

dd if=zeroth-mft-entry of=file_name_attribute bs=1 skip=152 count=104

00000000  30 00 00 00 68 00 00 00  00 00 18 00 00 00 02 00  |0...h...........|
00000010  4a 00 00 00 18 00 01 00  05 00 00 00 00 00 05 00  |J...............|
00000020  80 64 e1 e7 8b a1 d2 01  80 64 e1 e7 8b a1 d2 01  |.d.......d......|
*
00000040  00 70 00 00 00 00 00 00  00 6c 00 00 00 00 00 00  |.p.......l......|
00000050  06 00 00 00 00 00 00 00  04 03 24 00 4d 00 46 00  |..........$.M.F.|
00000060  54 00 00 00 00 00 00 00                           |T.......|

The first four bytes tell us the type (48); the next four are the length (104 bytes). This attribute is also resident. Let’s skip ahead to bytes 20–21, which are the offset (from the start of the attribute) to the content. Here, it’s 24, so let’s go there.

The $FILE_NAME attribute is described in Table 13.7 on page 362. I’m going to use Hex Fiend to do the extraction here, but it’s the same as using dd previously, or as slicing a sequence of bytes or seek()ing in Python.

dd if=file_name_attribute of=file_name_attribute_content bs=1 skip=24
00000000  05 00 00 00 00 00 05 00  80 64 e1 e7 8b a1 d2 01  |.........d......|
00000010  80 64 e1 e7 8b a1 d2 01  80 64 e1 e7 8b a1 d2 01  |.d.......d......|
00000020  80 64 e1 e7 8b a1 d2 01  00 70 00 00 00 00 00 00  |.d.......p......|
00000030  00 6c 00 00 00 00 00 00  06 00 00 00 00 00 00 00  |.l..............|
00000040  04 03 24 00 4d 00 46 00  54 00 00 00 00 00 00 00  |..$.M.F.T.......|

The first eight bytes are the file reference of the parent directory. File references are composed of two parts: the file number and the sequence number. The file number we’ve already seen: it’s the index into the MFT to get to this entry, starting from zero. The sequence number is incremented each time an MFT entry is allocated to use. The two numbers are concatenated, with the 16-bit sequence number in the higher-order bytes (little endian), and the 48-bit file number in the lower-order bytes (little endian), to form a 64-bit file reference number. Note that like all values, it’s stored little endian, so the final format is:

FF FF FF FF FF FF SS SS

where FN are file number bytes and SS are sequence number bytes.

So in this $FILE_NAME, the file number is 5. 5 is one of the reserved slots in the MFT, which if we look up in Table 11.1, we see is the root directory.

The next sequence of 8 bytes is the file creation time. NTFS stores file-related times as the number of blocks of 100 ns since January 1st, 1601 UTC.

Here, the value is 8064E1E7 8BA1D201 so we can get the number in Python using:

import struct
timestamp = struct.unpack('<Q', bytes.fromhex('8064E1E7 8BA1D201'))[0]

To convert that to a time, we convert to a UNIX-style epoch (the number of seconds since January 1, 1970). This time is 116444736000000000 100 ns blocks since January 1st, 1601 UTC. So to convert, we can write:

import datetime
def as_datetime(windows_timestamp):
    return datetime.datetime.fromtimestamp((windows_timestamp - 116444736000000000) / 10000000)
print(str(as_datetime(timestamp)))

Bytes 16–23 are the modification time; 24–31 are the MFT modification time; 32–39 are the last access time.

Bytes 40–47 are the allocated file size and bytes 48–55 are the actual size, but these are not required to be accurate unless this attribute is used in a directory index.

Bytes 56–59 are flags, just like in FAT (see Table 13.6).

Byte 64 is the length of the filename (4). Byte 65 is the namespace (See Table 13.8.; here it’s the Windows/DOS namespace). Bytes 66 onward are the name, in this case in UTF-16 (LE): $MFT. This is the $MFT entry, just like we expected for entry 0.

Note we can check all this using istat on the relevant entry:

istat simple.ntfs 0-128-1

One last attribute to look at here, the $DATA attribute, which is next. Going back to the zeroth-mft-entry, it starts at offset 256 (from the start of the entry, not the start of the entire MFT). How did we get this? It’s not a magic value: we computed it. Remember, the previous attribute ($FILE_NAME) started at offset 152 and was 54 bytes long. 54+152 = 256.

dd if=zeroth-mft-entry of=data_attribute bs=1 skip=256 count=72
00000000  80 00 00 00 48 00 00 00  01 00 40 00 00 00 01 00  |....H.....@.....|
00000010  00 00 00 00 00 00 00 00  12 00 00 00 00 00 00 00  |................|
00000020  40 00 00 00 00 00 00 00  00 30 01 00 00 00 00 00  |@........0......|
00000030  00 04 01 00 00 00 00 00  00 04 01 00 00 00 00 00  |................|
00000040  11 13 04 00 00 00 00 00                           |........|

The first four bytes (value: 128) tell us it’s the $DATA attribute, and the next four tell us its length (72). The next byte tells us that this is a non-resident attribute, so its contents are stored somewhere in cluster(s) on the disk.

Let’s jump ahead to figure out where (that is, which clusters) the data is stored in.

Looking at Table 13.4, bytes 16–23 and 24–31 are used to tell us the starting and ending VCN of the runlist. The VCN is just a sequence of numbers 0..n-1 referring to the n clusters in a file in order. This is in contrast to the LCN, which is a list of the actual cluster numbers (on disk) that correspond to the VCN clusters. (on board)

Why does the non-resident header have this marker? Because for very long, fragmented files, you might not be able to fit the runlist into a single MFT entry. NTFS then needs to split the attributed across several MFT entries; this is how you figure out “where you are” in the entry.

Then there’s an offset to the runlist (from the start of this attribute) at bytes 32-33. The runlist is in the following format:

First there’s a single byte that describes the length and offset of the next run; then there’s a variable number of bytes describing the length of the run, and a variable number of bytes describing the offset to the run.

In more detail: The first byte is split into two nibbles (4-bit values). The low-order bits tell you the number of bytes in the run length; the high-order bits tell you the number of bytes in the offset to the run.

These values stored in the length and offset are in units of clusters, not bytes or sectors, and the offset bytes are signed.

And, the while the length is what you’d expect, the offset is relative to the previous offset in the runlist (the first offset is relative to the start of the filesystem, that is, cluster 0). Let’s look at some data:

11 13 04

So, byte 1 contains 11, which in binary is

0001 0001

This run is described by a single byte offset and length. The length comes first: 13, so it’s 19 clusters long. The offset comes next: 04, so it starts 4 clusters past the start of the file system. Which is what we expect, again as shown by istat.

istat simple.ntfs 0-128-1

...
Type: $DATA (128-1)   Name: N/A   Non-Resident   size: 66560  init_size: 66560
4 5 6 7 8 9 10 11 
12 13 14 15 16 17 18 19 
20 0 0 
...

Notice there are 19 values there. They start at 4 and then there are 19 of them. The last two are zero. Why? The allocated size of this attribute’s content (bytes 40–47: as an int: 77824) is exactly 19 4kB clusters. But the actual size (bytes 48–55: 66560) fits in 16.25 clusters. So the last two clusters are allocated but not used. istat represents this by showing their numbers as zeroes.

If you continue parsing this entry, you’ll see that there’s one more attribute of type 0xb0, which is a $BITMAP attribute. This is used to track index records in the MFT, but we’ll not concern ourselves with it.

How do you know you are done parsing attributes? The MFT entry is of fixed size, and doesn’t include a “total length” in its header. Instead, you look for the hex value ffff ffff where you’d expect an attribute to begin; that indicates you have finished. In this MFT entry they’re at offset 0x190 from the start of the entry.

Directories in NTFS

Directories are represented just like everything else in NTFS, as a file. Well, as a combination entry-in-MFT and potentially as a file. First thing to know: Directories are logically represented as sorted lists of files – but the list is actually stored as a tree.

Not a binary tree, but a B-Tree whose exact paramters depend upon things like the size of the directory. A B-tree is like a binary tree but with a variable number of entries per node rather than just one.

Each entry (ultimately) refers to another entry in the MFT, representing a file, or another directory, or a non-resident attribute. The entries themselves are stored in the MFT in an $INDEX_ROOT attribute with space for a few entries; if more than that are needed, then a child node, stored in a non-resident attribute called an $INDEX_ALLOCATION, stores them. One $INDEX_ALLOCATION can store multiple index entries, each in a list called an “index record”; whether any particular record is active or not is stored in another attribute called a $BITMAP.

Recovering deleted file metadata in NTFS can thus be a hassle: if the bitmap is just deallocated, then the data might still be there. But if the tree needs rebalancing, metadata can be overwritten and lost.