yaffs文件系统

1. 概述
yaffs文件系统专为Nandflash设计的日志文件系统，占用page中oob区域。目前有两个版本的yaffs文件系统。nandflash不可靠，存在坏块，存在数据错误，需要软件弥补纠正此错误。
在yaffs中分配单元为CHUNK（chunk），一般情况，典型情况下其值与一页（page）大小相同（同时可灵活配置）。yaffs中文件数据的存储以chunk为单位。
YAFFS是专为为NandFlash设计的文件系统，基于MTD，为单线程程序。官网：
http://www.yaffs.net/
yaffs架构：

读文档 HowYaffsWorks 了解YAFFS演进过程及实现原理、特性。
2. YDI为移植和应用提供方便
The Yaffs Direct Interface (YDI) which allows Yaffs to be simply integrated with embedded systems, with or without an RTOS.
YDI包装Yaffs以易于集成的方式。
Yaffs Direct Interface (YDI) wraps Yaffs in a way that is simple to integrate. You need to provide a few functions for Yaffs to use to talk to your hardware and OS. Yaffs provides a set of POSIX-compliant functions for applications to use to talk to it.
In addition to the Yaffs core file system, the YDI has three parts which are each explained in more detail in further sections:
>>POSIX Application Interface: This is the interface that the application code uses to access the Yaffs file system. (open, close, read, write, etc)
>>RTOS Integration Interface: These are the functions that must be provided for Yaffs to access the RTOSsystem resources. (initialise, lock, unlock, get time, set error)
>>Flash Configuration and Access Interface: These are the functions that must be provided for Yaffs to access the NAND flash. (initialise, read chunk, write chunk, erase block, etc). These functions might be supplied by a chipset vendor or might need to be written by the integrator.
3. Yaffs1 VS Yaffs2
Yaffs1为小块设计（512bytes），布局为工厂模式（Smart-Media-like）。应用deletion makers标记坏块（打乱顺序写规则）。
Yaffs2兼容大小块（2Kbytes），可灵活配置布局。舍弃deletion marker，0回写（不允许重复写），一块内顺序写。
4. YAFFS tags
YAFFS tags in YAFFS1 mode:
18-bit Object ID (2^18 files, i.e. > 260,000 files). File id 0- is not valid and indicates a deleted page. File od 0x3ffff is also not valid. Synonymous with inode.
2-bit serial number
20-bit Chunk ID within file. Limit of 2^20 chunks/pages per file (i.e. > 500MB max file size). Chunk ID 0 is the file header for the file.
10-bit counter of the number of bytes used in the page.
12 bit ECC on tags

YAFFS tags in YAFFS2 mode:
4 bytes 32-bit chunk ID
4 bytes 32-bit object ID
2 bytes Number of data bytes in this chunk
4 bytes Sequence number for this block
3 bytes ECC on tags
12 bytes ECC on data (3 bytes per 256 bytes of data)
注：现在yaffs2对tags支持有较大变化，不再提供ECC on data支持，其余字段全为4字节。
5. 源码
yaffs_allocator.c Allocates Yaffs object and tnode structures.
yaffs_checkpointrw.c Streamer for writing checkpoint data
yaffs_ecc.c ECC code,软件ECC算法
yaffs_guts.c The major Yaffs algorithms.
yaffs_nand.c Flash interfacing abstraction.
yaffs_packedtags1.c
yaffs_packedtags2.c Tags packing code
yaffs_qsort.c Qsort used during Yaffs2 scanning，现无
yaffs_tagscompat.c Tags compatibility code to support Yaffs1 mode.
yaffs_tagsvalidity.c Tags validity checking. 现无
最新版yaffs2还增加如下文件：
yaffs_atribs.c 设置YAFFS对象（object）属性
yaffs_bitmap.c Yaffs位图操作（页bitmap）
yaffs_endian.c Yaffs对大小端支持
yaffs_mtdif.c 与mtd对接端口
yaffs_nameval.c 对name-value存储的支持
yaffs_summary.c 对一块中每页tags摘要支持
yaffs_tagsmarshall.c 对internal（内存）tags和stored tags之间转换的支持
yaffs_verify.c.c Yaffs各种校验支持
yaffs_vfs.c 与vfs接口（对linux vfs支持）
Yaffs_yaffs1.c Yaffs1扫描实现
Yaffs_yaffs2.c Yaffs2中checkpoint及两次扫描支持

从中可以看出yaffs对上层及下层接口文件：
>>对下层（硬件）采用mtd设备，对接文件yaffs_mtdif.c，
void yaffs_mtd_drv_install(struct yaffs_dev *dev)
{
struct yaffs_driver *drv = &dev->drv;

drv->drv_write_chunk_fn = yaffs_mtd_write;
drv->drv_read_chunk_fn = yaffs_mtd_read;
drv->drv_erase_fn = yaffs_mtd_erase;
drv->drv_mark_bad_fn = yaffs_mtd_mark_bad;
drv->drv_check_bad_fn = yaffs_mtd_check_bad;
drv->drv_initialise_fn = yaffs_mtd_initialise;
drv->drv_deinitialise_fn = yaffs_mtd_deinitialise;
}
Yaffs_mtd_write() ==> mtd_write_oob();
>>对上层（FS）采用vfs接口，对接文件yaffs_vfs.c对文件系统的各种支持inode，file_operations，inode_operations，mount，register等支持。
6. 设计考量
>>Corruption of file allocation tables and similar structures is a common failure mechanism for embedded file systems. The lack of such structures makes Yaffs more robust. Or, to put it another way, you can't corrupt what you don't store.
7. Objects
Yaffs以对象方式管理所有数据。
In Yaffs-speak, an Object is anything that is stored in the file system. These are:
● Regular data files
● Directories
● Hard-links
● Symbolic links
● Special objects (pipes, devices etc).
All objects are identified by a unique integer object Id.
In the standard POSIX model, as used by Linux, there are inodes and directory entries (dentries).
An inode may be a regular file, directory, or special file. Directory entries provide the naming mechanism that is used to locate inodes. Under POSIX, each inode may have zero, one or many dentries:
● One dentry per inode is commonly understood.
● Multiple dentries per inode allow the same inode to be accessed via multiple names. This is achieved through hard links.
● Zero dentries per inode is less often understood. This happens when an inode is unlinked while there is still a handle open to the inode. The unlinking deletes the dentry but the inode still exists. As soon as the handle is closed the inode is deleted too.
8. Yaffs1 Nand model
Yaffs中分配的单元是chunk，典型值为页大小，同时可配置为多页大小（灵活配置）。
Yaffs1每个chunk包含两种类型数据：
● Data chunk: A chunk holding regular data file contents.
● Object Header: A descriptor for an object (directory, regular data file, hard link, soft link, special descriptor,...). This holds details such as the identifier for the parent directory, object name, etc. Each chunk has tags associated with it. The tags comprise the following important fields (others being ignored for now):
● ObjectId: Identifies which object the chunk belongs to.
● ChunkId: Identifies where in the file this chunk belongs. A chunkId of zero signifies that this chunk contains an objectHeader. ChunkId==1 signifies the first chunk in the file (ie. At file offset 0), chunkId==2 is the next chunk and so on.
● Deletion Marker: (Yaffs1 only) Shows that this chunk is no longer in use.
● Byte Count: Number of bytes of data if this is a data chunk.
● Serial Number: (Yaffs1 only) Serial number used to differentiate chunks with the same objectId and chunkId.
9. Yaffs2 Nand model
Yaffs2 is an extension of Yaffs designed to fulfill a new set of objectives to work with newer NAND types including:
● Zero overwrites. Yaffs1 only ever overwrites a single byte in the spare area of the chunks to set the deletion marker. More modern NAND devices are less tolerant of overwrites and Yaffs2 performs no overwrites at all.
● Sequential chunk writing within a block. More modern NAND reliability specifications tend to assume sequential writing. Since Yaffs2 does not write deletion markers, the writing of chunks within a block is strictly sequential. Since Yaffs2 performs less writes, there is potential for Yaffs2 to have improved write performance.
Yaffs2 cannot use a deletion marker because that would violate the zero overwrites mandate and alternate mechanisms are provided to determine which chunks are current and which are deleted.

These mechanisms are:
● Sequence Number: As each block is allocated, the file system's sequence number is incremented and each chunk in the block is marked with that sequence number. The sequence number thus provides a way of organising the log in chronological order.
● Shrink Header Marker: The shrink header marker is used to mark object headers that are written to shrink the size of a data file. This is explained more in the accompanying text.
Note: The Yaffs2 sequence number is not the same as the Yaffs1 serial number!
Note: We still refer to chunk deletion in Yaffs2. The chunks are marked as deleted in the RAM data structures used by garbage collection etc, but there is no deletion marker written to flash.
10. Bad block handling NAND error handling
Any flash file system without an effective bad block handling policy is unsuitable for NAND flash.
Yaffs1 uses the Smart-Media style of bad block marking. This checks the sixth byte (byte 5) of the spare area. In a good block this should be 0xFF.
Yaffs2 mode is designed to support a wider range of devices and spare-area layouts. Thus, Yaffs2 does not decide which bytes to mark but instead calls driver functions to determine whether the block is bad, or to mark it bad.
So, when does Yaffs mark a block bad? Yaffs will mark a block bad if a read or write operation fails or if three ECC errors are detected. A block that is marked bad is retired from use, thus improving the robustness of the file system. This policy is suitable for SLC flash. MLC policies are under development.
Further, NAND flash cells can sometimes be disturbed (corrupted) by NAND activity and charge loss. These errors are corrected by error correction codes (ECC) which may be implemented in hardware, software drivers, or within Yaffs itself. Again, any flash file system lacking effective ECC handling is unsuitable for NAND flash.
Yaffs1 mode can use either the built in ECC or use ECC provided by the driver or hardware.
Since Yaffs2 mode is designed for a wider range of devices, it does not provide ECC internally but requires that the driver provide the ECC.
The ECC code supplied with Yaffs (yaffs_ecc.c) is the fastest C code implementation of a Smart Media compatible ECC algorithm that we are aware of. This code will correct any single bit error within a 256-byte data block and detect 2 errors per 256-byte data block. This is sufficient error correction to provide very high reliability on SLC-type NAND flash.

11. RAM结构
Yaffs文件系统提供了很多RAM structure来提高性能。
While it is theoretically possible to use a log structured file system with very few RAM structures, this would provide reduce performance. Thus, significant RAM data structures are required to store enough information to provide sufficiently high performance.
The main structures are defined in yaffs_guts.h and their main purposes, are :
● Device/partition: This is named yaffs_dev. This is required to hold information relating to a Yaffs “partition” or “mount point”. Using these, rather than globals, allows Yaffs to simultaneously support multiple partitions and partitions of different types. In fact, almost other data structures mentioned here are part of, or accessed via, the structure.设备或分区
● NAND Block Information: This is named yaffs_block_info and holds the current state of the NAND blocks. Each yaffs_Device has an array of these.
● NAND Chunk Information: This is a bitfield attached to the yaffs_Device that holds the current in-use state of each chunk in the system. There is one bit per chunk in the partition.
● Object: This is named yaffs_obj and holds the state of an object (see section 3).There is one of these for each object in the file system, where an object is one of a regular file, directory, hard link, symbolic link or special link. There are different variants of yaffs_Objects to reflect the different data required for these different object types. Each object is uniquely identified by an objectId.
● File structure: For each file object, Yaffs holds a tree which provides the mechanism to find the data chunks in a file. The tree consists of nodes called yaffs_tnode (tree nodes).
● Directory structure: The directory structure allows objects to be found by name. The directory structure is built from a doubly-linked list that is rooted in the directory and binds together the sibling objects within a directory.允许按name找对象
● Object number hash table: The object number hash table provides a mechanism to find an object from its objectId. The objectId is hashed to select a hash bucket. Each hash bucket has a doubly-linked list of objects that belong in this hash bucket, as well as a count of the objects in the bucket.允许按objectId找对象。
● Cache: Yaffs provides a read/write cache that significantly improves performance for short operations. The size of the cache can be set at run-time.
Yaffs大量运用双向链表，可快速插入、删除、遍历（traversal）。
>>The yaffs_obj's main function is to store most of the object metadata and type-specific information.
The metadata includes, amongst others:
● objectId: The number used to identify the object.
● parent: A pointer to the parent directory. Not all objects have a parent. The root director and special directories for unlinked and deleted files do not have parents so this is NULL.
● Short name: If the name is short enough to fit in a small fixed-sized array (default 16 characters) then it is stored here otherwise it must be fetched from flash each time it is requested.
● type: Type of object.
● Permissions, time, ownership and other attributes
Depending on the object type, yaffs_obj also stores:
● A tnode tree, file extents etc for data files
● A directory chain for directories
● An equivalent object for hard links
● A string for symbolic links
>>Each yaffs_dev uses a hash table for this purpose. The hash table has 256 “buckets” (tunable), each holding a doubly-linked list of the yaffs_objs in that bucket.
>>The purpose of the directory structure is to rapidly access an object by name.
>>Each Yaffs partition has a root directory.
>>Each name within a directory must be unique.
>>Name resolution is sped up by two mechanisms:
● Short names are stored directly in the yaffs_obj so that they don't have to be loaded from flash.
● Each object has a “name sum”. This is a simple hashing of the name which speeds up the matching process because it is cheaper to compare the hash than compare the whole name string.
12. Tnode
The file objects have type YAFFS_OBJECT_TYPE_FILE, and an associated file variant. This stores the following main values:
● fileSize: file size
● topLevel: depth of the Tnode tree
● top: pointer to top of Tnode tree
The top and topLevel work together to control the Tnode trees.
Each file has a tnode tree to provide the mapping from file position to actual NAND chunk address. top is a pointer to the top of the tnode tree

The tnode tree is made up of tnodes (tree nodes) arranged in levels, each of which holds either:
● At levels greater than 0, a tnode has 8 pointers to other tnodes in the next level down.
● At level 0, a tnode has 16 NAND chunk Ids which identify the chunk's location in RAM. These numbers are powers of 2 which makes for simple look-ups by just applying bitmasks.
13. Yaffs机制—block and chunk管理
Thus Yaffs now always starts allocation on a new block after a mount. This slightly increases the amount of garbage (typically by an insignificant amount) but improves robustness.
>>Wear leveling
Wear leveling is far less of a concern in a log structured file system such as Yaffs since we always write to the end of the log and thus don't keep on overwriting the same blocks. There are basically two ways to achieve wear leveling:
● Use a specifically written set of functions that actively perform wear leveling.
● Allow a degree of wear leveling to happen as a side effect of other activity.
Yaffs takes the second approach.
>>Management of yaffs_tnode and yaffs_obj structures
To prevent these problems, Yaffs manages these objects internally. Yaffs creates many structures (100 or so) at a time and then allocates them singly from a free list. When structures are released they are placed on the free list for reuse.
14. Yaffs机制—Scanning
Scanning involves reading the tags from all the active chunks in the system so can take quite a while, but probably less time than you'd expect.
The scanning process is different for Yaffs1 vs Yaffs2 because of the differences mentioned before.
>>Yaffs1 scanning is rather straight forward because Yaffs1 uses deletion markers. Once the scanning is complete we still need to fix up a few things: Hard links must be connected up properly (it is more efficient to defer this until after the scanning has completed). Also, objects which were marked as unlinked must be deleted since there are no longer handles open to these objects.
>>Yaffs2 has no deletion markers which makes the scanning significantly more complex.
共两次扫描，首先扫描block顺序号以排序，之后反向扫描。
The first thing that needs to be done is to pre-scan the blocks to determine the sequence number for the blocks. The list of blocks and sequence numbers is then sorted to make a chronologically ordered list. Yaffs then scans backwards through these blocks (ie in reverse chronological order). Thus, the first encountered objectId:chunkId pair is the current one and subsequent chunks with those tags are out of date.
We read the tags for each chunk in each block. If it is a data chunk ( ie. chunkId > 0) then:
● If a chunk with the same objectId:chunkId has already been found then this is not the current chunk. Delete this one.
● Else if the object is marked as deleted, or corrupted, then delete it.
● Else if the object does not yet exist then this chunk was written since the last object header write for this object so this chunk must be the last written chunk in the file and must have happened just before an unclean shutdown with the file still open. We can create the file and use the chunkId and nbytes to set the file extents.
● Else if the object does exist and the chunk lies beyond the object's file scan size (extents) then it is a truncated chunk and can be deleted.
● Else it is placed into the file's tnode tree.

If it is an object header (ie. chunkId == 0) then:
● If the object is in the deleted directory then mark the object as deleted and set the object's shrink size to zero.
● Else if an object header has not yet been found for this object, then this is the most current object header and holds the most current name etc. If no data chunks have yet been found for this chunk then the size in the object header is also the file extents. The file size is used to set the file scan size since at the time it was written the file did not extend past that point and we can safely ignore further data chunks for that file that extend past that point.
● Else just use the file size as the scan size if it is smaller than the current file scan size.
● Else this an obsolete object header. It is deleted
A bit more explanation is probably needed to describe the file scan size. The whole purpose of this is to determine data chunks that should be ignored due to the file being truncated.
More
15. Yaffs机制—checkpoint
Mount scanning takes quite a lot of time and slows mounting, though perhaps by less than you'd think. Checkpointing is a mechanism to speed the mounting by taking a snapshot of the Yaffs runtime state at unmount or sync() and then reconstituting the runtime state on remounting. The speed up is dramatic. The actual checkpoint mechanism is quite simple. A “stream” of data is written to a set of blocks which are marked as holding checkpoint data and the important runtime state is written to the stream.
16. Yaffs机制—扩展tags
Extended tags is an abstraction of a way to store more information in the tags, allowing for faster scan times. Packed Tags is a physical implementation of the storage of extended tags.
Object headers always have a chunkId of zero and nbytes has no meaning because it is not a data chunk. By using a single bit to identify object headers we suddenly open up a whole lot of space that we can stuff full of useful data. By being clever as to what data we use, we can pack in the important data in most circumstances.
17. Yaffs机制—Block summaries
Block summaries write all the tags data for chunks in the block into the last few (typically only one) chunk in a block. This allows all the tags for chunks in that block to be read in one hit. 减少mount时间。
18. Yaffs机制—带内tags
But what happens if we don't have enough space in the spare area to store the tags? Inband tags to the rescue! With inband tags we store the tags in the flash data area alongside the data. This gives us far more flexibility in the types of storage that Yaffs can support, but breaks the power-of-2 alignment.
Thus, it is better, performance wise, to only use inband tags when regular tags will not work.
19. Yaffs机制—soft deletion
Soft deletion is a mechanism that was added to speed up file deletion in Yaffs1 mode.
Yaffs2 mode does not write deletion markers and thus does not use soft deletion.

参考：
1. How Yaffs Works
2. http://www.yaffs.net/