BigFAT
Backward compatible FAT extension for unlimited file size
BigFAT is a free and open specification allowing storage of files larger than 4 GB on FAT-formatted drives, maintaining full FAT compatibility.
No patents, no fees.
Overview
FAT (short for File Allocation Table) is probably the most widely used file system. Introduced in 1977 by Microsoft for floppy disks, it has been enhanced multiple times, moving from FAT12 to FAT16, then FAT32. It has also been extended in other respects, such as adding support for long file names, while generally maintaining backward compatibility.
FAT is probably the most popular choice of format for removable media, such as SD cards, USB memory sticks as it is supported by all operating systems, and embedded devices. Unfortunately, FAT has one significant limitation: A storage medium formatted as FAT can only hold files smaller than 4 GB, which can be a limitation for some types of data, such as movies, databases or virtual HD containers.
SEGGER’s BigFAT is a specification defining how to store large files well beyond 1 TB on any FAT-formatted drive. Regular FAT systems see big files as multiple files, which they can read, write, and copy.
Key features
- Maintains full compatibility with FAT
- Supports all storage types, such as NAND, NOR, SD/MMC/CompactFlash cards
- Free utility to read and write big files from a PC
- Specifications fully public and open
- Not encumbered by any patents
- emFile component available to handle BigFAT on embedded systems
BigFAT versus exFAT
FAT is a great file system for removable storage, such as USB memory sticks. Its only limitation is the file size limit of 4 GB.
exFAT is Microsoft’s answer to the FAT 4GB file size limit. It handles file sizes that are orders of magnitude greater than the regular FAT system essentially eliminating the problem.
BigFAT is SEGGER’s answer to the FAT 4GB file size limit. It addresses this by defining how to split a file of 4 GB or more into chunks and how to name the chunks. Files smaller than 4GB are stored the same way so only big files are stored as multiples files. BigFAT therefore handles unlimited file sizes.
The use of BigFAT versus exFAT
exFAT was introduced by Microsoft in 2006. The reason for the new software was that it is "optimized for flash memory such as USB flash drives and SD cards" [1]. There was probably also a commercial reason. All patents on FAT had expired meaning Microsoft could no longer license or control the technology. Microsoft does, however, hold several patents related to exFAT and its use requires a license (available through SEGGER - see below).
The SD Association has adopted exFAT as the default file system for SDXC cards larger than 32 GB.
The BigFAT specification is fully open the to public. Anyone is welcome to add support for BigFAT to their software. No license fees or permissions of any kind are needed. Whoever wants to implement it or make something compatible is free to do so. SEGGER's BigFAT converter is a free command line program, available on Windows, macOS or Linux to anyone. SEGGER encourages BigFAT’s widespread adoption as a new, open standard!
SD or SDXC cards can simply be reformatted with the good old, proven and free FAT file system and with BigFAT the 4 GB per file restriction is lifted.
Licensing exFAT through SEGGER
For those who want or need exFAT, SEGGER has partnered with Microsoft to make a direct Microsoft license available through SEGGER. In addition, SEGGER can also offer a highly-tuned implementation and provide a top quality end-customer experience, including warranty and support.
__________
[1] exFAT (2022-07-22) in Wikipedia, [online] https://en.wikipedia.org/wiki/ExFAT.
Using BigFAT
BigFAT remains an alternative to exFAT. It is license-free, backward compatible, open to the public and unencumbered by patents.
Transfer of large files such as movies to TVs
Movie files are in many cases larger than 4GB. Transferring these to a TV can be quite tricky or require splitting up the movie into different parts or reducing quality. With BigFAT, these transfers can be easily performed.
Creation of large files on embedded devices
Nowadays, more and more data is collected, e.g. in production environments where sensors in machines record data on temperature, pressure, fill levels, etc. over long periods of time. Other examples are battery management systems (BMS), which collect data on charge, temperature, current, voltage for each cell over very long periods of time. This data must be stored, ideally as a single file, which can now be done even on standard SD card using BigFAT and the standard underlying FAT format.
Large databases on embedded systems
BigFAT allows random read and write access to any file, even if larger than 4GB, as required by databases. With BigFAT, the only limit to the size of the database on an Embedded System is the size of the storage available.
Specification
BigFAT is built on top of the FAT. It splits the data of a file of 4 GB or larger into smaller files (chunks) that can be handled by the underlying FAT file system. BigFAT directs file operations to the appropriate chunks, depending on the file position that is accessed.
Files smaller than 4 GB - 128 Kbytes are treated like normal files and the file access works like in a standard file system implementation. As soon as the file grows above the limit, the file gets extended. An extended file consists of one base file and one or more continuation files. The continuation files are stored in the same directory as the base file, with the name derived from the name of the base file plus an additional file extension. The name of the base file remains unchanged. The file extension of a continuation file has the format .<Index>.BigFAT where the Index is a three digit decimal number. The index of the first continuation file is set to one and increases by one with each additional continuation file. BigFAT requires support for long file names on the underlying file system layer in order to be able to apply the additional file extension of continuation files.
When the base file grows larger than 4 GB - 128 Kbytes, the first continuation file is created. This procedure is required in order to guarantee that the first byte in the continuation file is aligned to a logical sector boundary. This helps increase the performance of the file system accessing the file data. The same procedure is applied when the first continuation file exceeds the 4 GB - 128 Kbytes limit, with a second continuation file being created and so on.
BigFAT Converter
SEGGER's BigFAT Converter is a simple command line program, available for Windows, Linux and macOS. It can convert large files (≥ 4 GB) into chunks which are then recognized as a single big file by any BigFAT compatible system and vice versa, convert the small chunks into a single large file.
The command line utility is able to merge the fragment files generated on a target device into a single file larger than 4 GB. In addition, the command line utility is also able to split a file larger than 4 GB into fragment files that can later be copied to a target device in order to be accessed by a target application.
emFile BigFAT component
The emFile BigFAT component is an optional add-on of emFile that allows a target application to access files that are compliant with the SEGGER BigFAT specification. The access to the files is done via a dedicated API that follows the same design as the regular emFile API. The target application can use the BigFAT API to open a file, perform read and write operations on it and then to close it. In addition, API functions are available for performing operations on large files such as copying, moving and deleting.
The integration of the emFile BigFAT component in a target application is straightforward. It only requires the addition of a single source file and the activation of the support for long file names. No other special configuration is required.
FAQ
Q: What is the advantage of BigFAT over exFAT?
A: BigFAT is an open specification while exFAT is a closed specification protected by patents owned by Microsoft. exFAT cannot be legally used without permission from Microsoft.
Q: Can I implement BigFAT myself?
A: Absolutely. BigFAT is a specification made available by SEGGER. Anybody is free to write a piece of software implementing it. No fees, no royalties, no headaches. You do not even have to let SEGGER or anybody else know.
Q: What is in it for SEGGER? Why do you make it available free of charge to anyone?
A: We see emFile customers asking for solutions for bigger files. Implementing exFAT is not an option for us, as it is patent encumbered. SEGGER would need Microsoft's permission to implement and offer it, and our customers need to deal with Microsoft again to be able to use it in their products. This can be time-consuming and also expensive. We feel there should be a free alternative. The more popular BigFAT becomes, the better.