leveldb的基本使用和功能介绍
references: https://github.com/google/leveldb/blob/main/doc/index.md
Base Operations
打开数据库，并进行数据的写入、读取、删除操作
int main(int argc, char **argv) {
// open a database and create if necessary
leveldb::DB* db;
leveldb::Options options;
// set options if database file is not exist to create database
options.create_if_missing = true;
// if database file is exist to raise exception
// options.error_if_exists = true;
leveldb::Status status = leveldb::DB::Open(options, "./testdb", &db);
// check that the status
if (!status.ok()) {
std::cerr << status.ToString() << std::endl;
} else {
std::cout << "success!" << std::endl;
}
/*
➜ build git:(main) ✗ ./LevelDBTest
Invalid argument: ./testdb: exists (error_if_exists is true)
*/
// assert(status.ok());
// std::cout << "create success" << std::endl;
/*
➜ build git:(main) ✗ ls
CMakeCache.txt CMakeFiles cmake_install.cmake LevelDBTest Makefile testdb
➜ build git:(main) ✗ ls testdb
000003.log CURRENT LOCK LOG MANIFEST-000002
➜ build git:(main) ✗ ./LevelDBTest
LevelDBTest: /root/pkg/leveldb/examples/_1_open_options/main.cc:14: int main(int, char**): Assertion `status.ok()' failed.
[1] 93525 abort (core dumped) ./LevelDBTest
*/
// reads and writes
std::string key1 = "key1";
std::string val1 = "val1";
leveldb::Status s = db->Put(leveldb::WriteOptions(), key1, val1);
if (s.ok()) {
s = db->Get(leveldb::ReadOptions(), key1, &val1);
} else {
std::cout << "get failed" << std::endl;
}
if (s.ok()) {
s = db->Delete(leveldb::WriteOptions(), key1);
} else {
std::cout << "delete failed" << std::endl;
}
s = db->Put(leveldb::WriteOptions(), key1, val1);
delete db;
}
支持将多个操作作为一个atomic update
int main(int argc, char** argv) {
leveldb::DB *db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status s = leveldb::DB::Open(options, "./testdb", &db);
if (!s.ok()) {
std::cout << "open failed" << std::endl;
} else {
std::cout << "opened successfully" << std::endl;
}
// atomic updates
// Note that if the process dies after the Put of key2 but before the delete of key1, the
// same value may be left stored under multiple keys. Such problems can be avoided by using
// the WriteBatch class to atomically apply a set of updates
std::string key1 = "key1", value1 = "value1";
std::string key2 = "key2";
s = db->Get(leveldb::ReadOptions(), key1, &value1);
if (!s.ok()) {
std::cout << "get failed" << std::endl;
} else {
leveldb::WriteBatch batch;
// the execution in a batch is ordered by the definition
// delete before put, so that we don't get a error if key1 is equal to key2
batch.Delete(key1);
batch.Put(key2, value1);
s = db->Write(leveldb::WriteOptions(), &batch);
if (!s.ok()) {
std::cout << "write batch failed" << std::endl;
}
}
}
设置同步开关
leveldb数据写入时默认是一个异步操作，并不会直接写到磁盘，而是写到操作系统内存后写操作就会直接返回，设置此选项后会是的写动作写入到磁盘完成才返回
int main(int argc, char** argv) {
leveldb::DB *db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status s = leveldb::DB::Open(options, "./testdb", &db);
if (!s.ok()) {
std::cout << "open failed" << std::endl;
} else {
std::cout << "opened successfully" << std::endl;
}
// open the flag for synchronous write
// By default, each write to leveldb is asynchronous: it returns after pushing the write from
// the process into the operating system. The transfer from operating system memory to the
// underlying persistent storage happens asynchronously. The sync flag can be turned on for
// a particular write to make the write operation not return until the data being written has
// been pushed all the way to persistent storage.
leveldb::WriteOptions write_options;
write_options.sync = true;
std::string key1 = "key1", value1 = "value1";
s = db->Put(write_options, key1, value1);
if (!s.ok()) {
std::cout << "write error: " << s.ToString() << std::endl;
}
}
遍历数据库内元素的方式
int main(int argc, char** argv) {
leveldb::DB *db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status s = leveldb::DB::Open(options, "./testdb", &db);
if (!s.ok()) {
std::cout << "open failed" << std::endl;
} else {
std::cout << "opened successfully" << std::endl;
}
// Iteration
// print key value
leveldb::Iterator *it = db->NewIterator(leveldb::ReadOptions());
std::cout << "print by order" << std::endl;
for (it->SeekToFirst(); it->Valid(); it->Next()) {
std::cout << it->key().ToString() << ":" << it->value().ToString() << std::endl;
}
std::cout << "print reverse order" << std::endl;
for (it->SeekToLast(); it->Valid(); it->Prev()) {
std::cout << it->key().ToString() << ":" << it->value().ToString() << std::endl;
}
/*
➜ build git:(main) ✗ ./LevelDBTest
opened successfully
print by order
key1:value1
key2:value2
key3:value3
print reverse order
key3:value3
key2:value2
key1:value
*/
// print value just the key in range [start, limit)
// std::cout << "range:" << std::endl;
// auto start, limit; // Slice
// for (it->Seek(start); it->Valid() && it->key().ToString() < limit; it->Next()) {
// std::cout << it->key().ToString() << ":" << it->value().ToString() << std::endl;
// }
assert(it->status().ok());
delete it;
}
Snapshot
int main(int argc, char** argv) {
leveldb::DB *db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status s = leveldb::DB::Open(options, "./testdb", &db);
if (!s.ok()) {
std::cout << "open failed" << std::endl;
} else {
std::cout << "opened successfully" << std::endl;
}
// Snapshot
// Snapshots provide consistent read-only views over the entire state of the key-value store.
// ReadOptions::snapshot may be non-NULL to indicate that a read should operate on a particular
// version of the DB state. If ReadOptions::snapshot is NULL, the read will operate on an
// implicit snapshot of the current state.
leveldb::ReadOptions r_options;
r_options.snapshot = db->GetSnapshot();
std::string key5 = "key5", value5 = "value5";
s = db->Put(leveldb::WriteOptions(), key5, value5);
if (!s.ok()) {
std::cout << "Error inserting" << std::endl;
}
leveldb::Iterator *iter = db->NewIterator(r_options);
std::cout << "Transaction snapshot" << std::endl;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
std::cout << iter->key().ToString() << ":" << iter->value().ToString() << std::endl;
}
leveldb::Iterator *iter2 = db->NewIterator(leveldb::ReadOptions());
std::cout << "Transaction database" << std::endl;
for (iter2->SeekToFirst(); iter2->Valid(); iter2->Next()) {
std::cout << iter2->key().ToString() << ":" << iter2->value().ToString() << std::endl;
}
db->ReleaseSnapshot(r_options.snapshot);
/*
➜ build git:(main) ✗ ./LevelDBTest
opened successfully
Transaction snapshot
key1:value1
key2:value2
key3:value3
key4:value4
Transaction database
key1:value1
key2:value2
key3:value3
key4:value4
key5:value5
*/
}
Slice
int main(int argc, char** argv) {
// Slice
// Slice is a simple structure that contains a length and a pointer to an external byte array.
// Returning a Slice is a cheaper alternative to returning a std::string since we do not need
// to copy potentially large keys and values. In addition, leveldb methods do not return
// null-terminated C-style strings since leveldb keys and values are allowed to contain '\0' bytes.
leveldb::Slice s1 = "test";
// string to Slice
std::string str("test");
leveldb::Slice s2(str);
// Slice to string
std::string str2 = s1.ToString();
assert(str == str2);
// be careful when using Slices since it is up to the caller to ensure that the external byte
// array into which the Slice points remains live while the Slice is in use. For example, the following is buggy
leveldb::Slice s3;
if (true) {
// When the if statement goes out of scope, str will be destroyed and the backing storage for slice will disappear.
std::string str3("asdad");
s3 = str3;
}
// user s3
// std::cout << s3.ToString() << std::endl;
}
Comparator
define a proper subclass of leveldb::Comparator that expresses your rule.
class TwoPartComparator : public leveldb::Comparator {
public:
int Compare(const leveldb::Slice& a, const leveldb::Slice& b) const {
int a1, a2, b1, b2;
ParseKey(a, &a1, &a2);
ParseKey(b, &b1, &b2);
if (a1 < b1) return -1;
if (a1 > b1) return 1;
if (a2 > b2) return -1;
if (a2 < b2) return 1;
return 0;
}
void ParseKey(const leveldb::Slice& a, int* a1, int* a2) const {
}
const char* Name() const { return "TwoPartComparator"; }
void FindShortestSeparator(std::string*, leveldb::Slice&) const {}
void FindShortSuccessor(std::string* key) const {}
/**
* use
*
* TwoPartComparator cmp;
* leveldb::DB* db;
* leveldb::Options options;
* options.comparator = cmp;
* ...
*
*/
};
Performance
可以通过修改leveldb的某些选项值达到改变性能的目的，这些选项在include/options.h
Block size
leveldb groups adjacent keys together into the same block and such a block is the unit of transfer to and from persistent storage. The default block size is approximately 4096 uncompressed bytes. Applications that mostly do bulk scans over the contents of the database may wish to increase this size. Applications that do a lot of point reads of small values may wish to switch to a smaller block size if performance measurements indicate an improvement. There isn't much benefit in using blocks smaller than one kilobyte, or larger than a few megabytes. Also note that compression will be more effective with larger block sizes.
在必要时设定kNoCompression
Each block is individually compressed before being written to persistent storage. Compression is on by default since the default compression method is very fast, and is automatically disabled for uncompressible data. In rare cases, applications may want to disable compression entirely, but should only do so if benchmarks show a performance improvement:
leveldb::Options options;
options.compression = leveldb::kNoCompression;
... leveldb::DB::Open(options, name, ...) ....
Cache
通过使用缓存将常用数据的未压缩形式放在缓存中以此提高性能
The contents of the database are stored in a set of files in the filesystem and each file stores a sequence of compressed blocks. If options.block_cache is non-NULL, it is used to cache frequently used uncompressed block contents.
这里的缓存指的是leveldb client提供的，而非操作系统自身的缓存
#include "leveldb/cache.h"
leveldb::Options options;
options.block_cache = leveldb::NewLRUCache(100 * 1048576); // 100MB cache
leveldb::DB* db;
leveldb::DB::Open(options, name, &db);
... use the db ...
delete db
delete options.block_cache;
在进行数据库的数据遍历读操作时，可能希望此次操作是针对整体的数据库数据，而非缓存，因此在读的时候需要关闭缓存的选项
leveldb::ReadOptions options;
options.fill_cache = false;
leveldb::Iterator* it = db->NewIterator(options);
for (it->SeekToFirst(); it->Valid(); it->Next()) {
...
}
delete it;
Key Layout
根据不通类型的数据的访问频率，将不同类型的数据放在缓存和磁盘不通的位置，从而是的访问更加迅速
Note that the unit of disk transfer and caching is a block. Adjacent keys (according to the database sort order) will usually be placed in the same block. Therefore the application can improve its performance by placing keys that are accessed together near each other and placing infrequently used keys in a separate region of the key space.
For example, suppose we are implementing a simple file system on top of leveldb. The types of entries we might wish to store are:
filename -> permission-bits, length, list of file_block_ids
file_block_id -> data
We might want to prefix filename keys with one letter (say '/') and the file_block_id keys with a different letter (say '0') so that scans over just the metadata do not force us to fetch and cache bulky file contents.
filter
通过filter减少在执行Get操作时不必要的磁盘读
Because of the way leveldb data is organized on disk, a single Get() call may involve multiple reads from disk. The optional FilterPolicy mechanism can be used to reduce the number of disk reads substantially.
leveldb::Options options;
options.filter_policy = NewBloomFilterPolicy(10);
leveldb::DB* db;
leveldb::DB::Open(options, "/tmp/testdb", &db);
... use the database ...
delete db;
delete options.filter_policy;
使用布隆过滤器从而减少不必要的快访问，但同时会增加内存使用
The preceding code associates a Bloom filter based filtering policy with the database. Bloom filter based filtering relies on keeping some number of bits of data in memory per key (in this case 10 bits per key since that is the argument we passed to NewBloomFilterPolicy). This filter will reduce the number of unnecessary disk reads needed for Get() calls by a factor of approximately a 100. Increasing the bits per key will lead to a larger reduction at the cost of more memory usage.** We recommend that applications whose working set does not fit in memory and that do a lot of random reads set a filter policy.**
If you are using a custom comparator, you should ensure that the filter policy you are using is compatible with your comparator. For example, consider a comparator that ignores trailing spaces when comparing keys. NewBloomFilterPolicy must not be used with such a comparator. Instead, the application should provide a custom filter policy that also ignores trailing spaces. For example:
class CustomFilterPolicy : public leveldb::FilterPolicy {
private:
leveldb::FilterPolicy* builtin_policy_;
public:
CustomFilterPolicy() : builtin_policy_(leveldb::NewBloomFilterPolicy(10)) {}
~CustomFilterPolicy() { delete builtin_policy_; }
const char* Name() const { return "IgnoreTrailingSpacesFilter"; }
void CreateFilter(const leveldb::Slice* keys, int n, std::string* dst) const {
// Use builtin bloom filter code after removing trailing spaces
std::vector<leveldb::Slice> trimmed(n);
for (int i = 0; i < n; i++) {
trimmed[i] = RemoveTrailingSpaces(keys[i]);
}
builtin_policy_->CreateFilter(trimmed.data(), n, dst);
}
};
filter的更多信息可以在这里看leveldb/filter_policy.h
Ckecksums
leveldb对所有存储在文件系统的数据提供checksum，通过设置不同的option，可能控制在每次读之前进行checksum或者打开数据库的时候进行checksum
ReadOptions::verify_checksums may be set to true to force checksum verification of all data that is read from the file system on behalf of a particular read. By default, no such verification is done.
Options::paranoid_checks may be set to true before opening a database to make the database implementation raise an error as soon as it detects an internal corruption. Depending on which portion of the database has been corrupted, the error may be raised when the database is opened, or later by another database operation. By default, paranoid checking is off so that the database can be used even if parts of its persistent storage have been corrupted.
If a database is corrupted (perhaps it cannot be opened when paranoid checking is turned on), the leveldb::RepairDB function may be used to recover as much of the data as possible
Approximate Size
获取指定key范围内的数据在文件系统中占据的空间大小，单位时byte
The GetApproximateSizes method can used to get the approximate number of bytes of file system space used by one or more key ranges.
leveldb::Range ranges[2];
ranges[0] = leveldb::Range("a", "c");
ranges[1] = leveldb::Range("x", "z");
uint64_t sizes[2];// 分别存储上述两个指定范围的大小估计值
db->GetApproximateSizes(ranges, 2, sizes);
Environment
leveldb 实现发出的所有文件操作（和其他操作系统调用）都通过 leveldb::Env 对象进行路由。 成熟的客户可能希望提供自己的 Env 实现以获得更好的控制。 例如，应用程序可能会在文件 IO 路径中引入人为延迟，以限制 leveldb 对系统中其他活动的影响。
All file operations (and other operating system calls) issued by the leveldb implementation are routed through a leveldb::Env object. Sophisticated clients may wish to provide their own Env implementation to get better control. For example, an application may introduce artificial delays in the file IO paths to limit the impact of leveldb on other activities in the system.
class SlowEnv : public leveldb::Env {
... implementation of the Env interface ...
};
SlowEnv env;
leveldb::Options options;
options.env = &env;
Status s = leveldb::DB::Open(options, ...);
Port
平台移植相关
leveldb may be ported to a new platform by providing platform specific implementations of the types/methods/functions exported by leveldb/port/port.h. See leveldb/port/port_example.h for more details.
In addition, the new platform may need a new default leveldb::Env implementation. See leveldb/util/env_posix.h for an example.
