chromium 缓存 electron缓存的读取 Disk Cache chrome 多线程的很好使用示例

chrome 缓存查看

chrome://version/ 可以看到缓存目录

chrome://cache/ == chrome://view-http-cache/ 可以看到已经编码的缓存文件。需要在chrome 65以前版本。后来的删除了。

electron 缓存目录

unix dir: /home/user_a/.config/hello_electron/Cache

在创建browserwindow时加入：

console.log("========== cache:"+app.getPath('userData') )

重置缓存目录：

app.setPath ('userData', "path/to/new/directory");

electron的实现代码：

// Return the path constant from string.
int GetPathConstant(const std::string& name) {
  if (name == "appData")
    return DIR_APP_DATA;
  else if (name == "userData")
    return DIR_USER_DATA;
  else if (name == "cache")
    return DIR_CACHE;
  else if (name == "userCache")
    return DIR_USER_CACHE;
  else if (name == "logs")
    return DIR_APP_LOGS;
  else if (name == "home")
    return base::DIR_HOME;
  else if (name == "temp")
    return base::DIR_TEMP;
  else if (name == "userDesktop" || name == "desktop")
    return base::DIR_USER_DESKTOP;
  else if (name == "exe")
    return base::FILE_EXE;
  else if (name == "module")
    return base::FILE_MODULE;
  else if (name == "documents")
    return chrome::DIR_USER_DOCUMENTS;
  else if (name == "downloads")
    return chrome::DIR_DEFAULT_DOWNLOADS;
  else if (name == "music")
    return chrome::DIR_USER_MUSIC;
  else if (name == "pictures")
    return chrome::DIR_USER_PICTURES;
  else if (name == "videos")
    return chrome::DIR_USER_VIDEOS;
  else if (name == "pepperFlashSystemPlugin")
    return chrome::FILE_PEPPER_FLASH_SYSTEM_PLUGIN;
  else
    return -1;
}

he Electron stores it's cache in these folders:

Windows:
C:\Users\<user>\AppData\Roaming\<yourAppName>\Cache

Linux:
/home/<user>/.config/<yourAppName>/Cache

OS X:
/Users/<user>/Library/Application Support/<yourAppName>/Cache

代码实现

命令行示例：D:\dev\electron7\src\net\tools\cachetool\cachetool.cc

编译文件：D:\dev\electron7\src\net\BUILD.gn 参数上没有windows。

if (is_linux || is_mac) {
  executable("cachetool") {
    testonly = true
    sources = [
      "tools/cachetool/cachetool.cc",
    ]
    deps = [
      ":net",
      ":test_support",
      "//base",
    ]
  }

编译生成：ninja -C out/Testing cachetool

调用：

out/Release/cachetool 
cachetool <cache_path> <cache_backend_type> <subcommand> 

Available cache backend types: simple, blockfile
Available subcommands:
  batch: Starts cachetool to process serialized commands passed down by the standard input and return commands output in the stdout until the stop command is received.
  delete_key <key>: Delete key from cache.
  delete_stream <key> <index>: Delete a particular stream of a given key.
  get_size: Calculate the total size of the cache in bytes.
  get_stream <key> <index>: Print a particular stream for a given key.
  list_keys: List all keys in the cache.
  list_dups: List all resources with duplicate bodies in the cache.
  update_raw_headers <key>: Update stdin as the key's raw response headers.
  stop: Verify that the cache can be opened and return, confirming the cache exists and is of the right type.
Expected values of <index> are:
  0 (HTTP response headers)
  1 (transport encoded content)
  2 (compiled content)

示例：

out/Release/cachetool /home/a/.config/hello/Cache simple get_size
35073536

out/Release/cachetool /home/a/.config/hello/Cache simple list_keys

源码分析：

很好的例子用了 chromium 多线程。

示例一、

1，回调函数

void SetSuccessCodeOnCompletion(base::RunLoop* run_loop,
                                bool* succeeded,
                                int net_error) {
  if (net_error == net::OK) {
    *succeeded = true;
  } else {
    *succeeded = false;
  }
  run_loop->Quit();
}

2，创建一个backend实例：

std::unique_ptr<Backend> CreateAndInitBackend(const CacheSpec& spec) {
  //定义了两个，是为了最后swap，交换了传回一个，避免出来作用域后被释放。
  std::unique_ptr<Backend> result;
  std::unique_ptr<Backend> backend;

  //下面这两个参数会传入回调
  bool succeeded = false;
  base::RunLoop run_loop;

  //绑定，生成一个回调实例。用绑定就是为了初始化参数，等于前两个参数固定了。可变的只有第三个参数了。
  //第三个参数才是int, 它才是net::CompletionOnceCallback这个函数原型的参数 void(int)。
  net::CompletionOnceCallback callback =
      base::BindOnce(&SetSuccessCodeOnCompletion, &run_loop, &succeeded);
  
//调用生成，传入回调
const int net_error =
      CreateCacheBackend(spec.cache_type, spec.backend_type, spec.path, 0,
                         false, nullptr, &backend, std::move(callback));
  if (net_error == net::OK)
    SetSuccessCodeOnCompletion(&run_loop, &succeeded, net::OK);//ok就不走回调了。直接再去调用一遍。
  else
    run_loop.Run();//不ok，启动消息循环阻塞在这里，直到回调里面调用run_loop.Quit()才往下执行。

  //到这里肯定是调过回调了，里面会设置succeeded值。这里开始判断。
  if (!succeeded) {
    LOG(ERROR) << "Could not initialize backend in "
               << spec.path.LossyDisplayName();
    return result;
  }

  //打完收工，通过swap，完成unique pointer变身
  result.swap(backend);
  return result;
}

但是这种用 base::RunLoop run_loop方式有缺陷，因为run_loop.run方法只能运行一次，如果在循环里面，无法再次赋值给回调。

//RunLoop::Run can only be called once
// per RunLoop lifetime. Create a RunLoop on the stack and call Run/Quit to run
// a nested RunLoop but please do not use nested loops in production code!

示例二、打开缓存，查看列表文件

这次回调函数原型是void(EntryResult result)

void OnEntryResultComplete(base::RunLoop* run_loop,
                           bool* succeeded, int *sum,
                           EntryResult result) {
  auto rv = result.net_error();

  if (rv == net::OK) {
    *succeeded = true;

  Entry* entry = result.ReleaseEntry();
  std::string url = entry->GetKey();
  std::cout << url << std::endl;
  entry->Close();


  } else {
    *succeeded = false;
  }

  run_loop->Quit();

  if (rv == net::ERR_FAILED) {
    std::cout << "=== read failed." << std::endl;
  }
}

列出主函数

 {
      bool succeeded0;
      std::unique_ptr<base::RunLoop> run_loop0;
      int sum = 0;

      std::unique_ptr<disk_cache::Backend::Iterator> iter_ =
          backend->CreateIterator();
      while (true) {
        //初始化和重新绑定回调，因为loop只能run一次。
        succeeded0 = false;
        run_loop0 = std::make_unique<base::RunLoop>(
            base::RunLoop::Type::kNestableTasksAllowed);
        disk_cache::Backend::EntryResultCallback entryCallback = base::BindOnce(
            &OnEntryResultComplete, run_loop0.get(), &succeeded0, &sum);

        EntryResult result = iter_->OpenNextEntry(std::move(entryCallback));

        auto rv = result.net_error();
        if (rv != net::OK) {
          // 运行loop.run阻塞等待。。。
          run_loop0->Run();
          run_loop0.reset();         
        } else {
          std::cout << "=== direct get result. net::OK." << std::endl;
          OnEntryResultComplete(run_loop0.get(), &succeeded0, &sum,
                                {});  // std::move(result) EntryResult构造被禁用了。
        }

        if (!succeeded0) {
          std::cout << "Could not get backend in "
                    << it->path.LossyDisplayName();
          break;
        }
      }
    }

示例三、chromium还提供了高大上的通用模板

主要是基类1

class TestCompletionCallbackBaseInternal {
 public:
  bool have_result() const { return have_result_; }

 protected:
  TestCompletionCallbackBaseInternal();
  virtual ~TestCompletionCallbackBaseInternal();

  void DidSetResult();
  void WaitForResult();

 private:
  // RunLoop.  Only non-NULL during the call to WaitForResult, so the class is
  // reusable.
  std::unique_ptr<base::RunLoop> run_loop_;
  bool have_result_;

  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallbackBaseInternal);
};

它的实现

void TestCompletionCallbackBaseInternal::DidSetResult() {
  have_result_ = true;
  if (run_loop_)
    run_loop_->Quit();
}

void TestCompletionCallbackBaseInternal::WaitForResult() {
  DCHECK(!run_loop_);
  if (!have_result_) {
    run_loop_ = std::make_unique<base::RunLoop>(
        base::RunLoop::Type::kNestableTasksAllowed);
    run_loop_->Run();
    run_loop_.reset();
    DCHECK(have_result_);
  }
  have_result_ = false;  // Auto-reset for next callback.
}

继承基类的template：

template <typename R, typename IsPendingHelper = NetErrorIsPendingHelper<R>>
class TestCompletionCallbackTemplate
    : public TestCompletionCallbackBaseInternal {
 public:
  virtual ~TestCompletionCallbackTemplate() override {}

  R WaitForResult() {
    TestCompletionCallbackBaseInternal::WaitForResult(); //调了父类的等待
    return std::move(result_);//取回值
  }

  R GetResult(R result) {
    IsPendingHelper check_pending;
    if (!check_pending(result))
      return std::move(result);
    return WaitForResult();
  }

 protected:
  TestCompletionCallbackTemplate() : result_(R()) {}

  // Override this method to gain control as the callback is running.
  virtual void SetResult(R result) {
    result_ = std::move(result);
    DidSetResult();
  }

 private:
  R result_;

  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallbackTemplate);
};

定义TestCompletionCallbackBase：

typedef internal::TestCompletionCallbackTemplate<int>
    TestCompletionCallbackBase;

回调进来的SetResult，会结束调loop通过run_loop->quit().并且用返回result的值。

// Base class overridden by custom implementations of TestCompletionCallback.
typedef internal::TestCompletionCallbackTemplate<int>
    TestCompletionCallbackBase;

typedef internal::TestCompletionCallbackTemplate<int64_t>
    TestInt64CompletionCallbackBase;

class TestCompletionCallback : public TestCompletionCallbackBase {
 public:
  TestCompletionCallback() {}
  ~TestCompletionCallback() override;

  CompletionOnceCallback callback() {
    return base::BindOnce(&TestCompletionCallback::SetResult,
                          base::Unretained(this));
  }

 private:
  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallback);
};

新加的返回entry的callback：

// -----------------------------------------------------------------------
// TestEntryResult callback define
// Like net::TestCompletionCallback, but for EntryResultCallback.
struct EntryResultIsPendingHelper {
  bool operator()(const disk_cache::EntryResult& result) const {
    return result.net_error() == net::ERR_IO_PENDING;
  }
};
using TestEntryResultCompletionCallbackBase =
    net::internal::TestCompletionCallbackTemplate<disk_cache::EntryResult,
                                                  EntryResultIsPendingHelper>;

class TestEntryResultCompletionCallback
    : public TestEntryResultCompletionCallbackBase {
 public:
  TestEntryResultCompletionCallback();
  ~TestEntryResultCompletionCallback() override;

  disk_cache::Backend::EntryResultCallback callback();

 private:
  DISALLOW_COPY_AND_ASSIGN(TestEntryResultCompletionCallback);
};

TestEntryResultCompletionCallback::TestEntryResultCompletionCallback() =
    default;

TestEntryResultCompletionCallback::~TestEntryResultCompletionCallback() =
    default;

disk_cache::Backend::EntryResultCallback
TestEntryResultCompletionCallback::callback() {
  return base::BindOnce(&TestEntryResultCompletionCallback::SetResult,
                        base::Unretained(this));
}

参考文件：

cachetool.cc 这就是用上面实现的。

修改的disk_cache_memory_test.c,可以在win下编译使用：

// Copyright 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

/*
ninja -C out/Testing disk_cache_memory_test

D:\dev\electron7\src>out\Testing\disk_cache_memory_test.exe
--spec-1=block_file:disk_cache:xxx
*/

#include <cstdlib>
#include <fstream>
#include <iostream>
#include <memory>
#include <string>
#include <vector>

#include "base/at_exit.h"
#include "base/bind.h"
#include "base/callback.h"
#include "base/command_line.h"
#include "base/files/file_path.h"
#include "base/logging.h"
#include "base/message_loop/message_pump_type.h"
#include "base/run_loop.h"
#include "base/strings/string_number_conversions.h"
#include "base/strings/string_piece.h"
#include "base/strings/string_split.h"
#include "base/strings/string_util.h"
#include "base/strings/stringprintf.h"
#include "base/task/single_thread_task_executor.h"
#include "base/task/thread_pool/thread_pool_instance.h"
#include "base/threading/thread_task_runner_handle.h"
#include "net/base/cache_type.h"
#include "net/base/net_errors.h"
#include "net/disk_cache/disk_cache.h"
#include "net/disk_cache/simple/simple_backend_impl.h"
#include "net/disk_cache/simple/simple_index.h"


#include "net/disk_cache/blockfile/backend_impl.h"
#include "net/disk_cache/blockfile/file.h"
#include "net/disk_cache/cache_util.h"



#include "base/bind_helpers.h"
#include "base/compiler_specific.h"
#include "net/base/io_buffer.h"

#include "base/files/file_path.h"
#include "base/macros.h"
#include "base/timer/timer.h"
#include "build/build_config.h"

//-----------------------------------------------------------------------------
#include "net/http/http_cache.h"
#include "net/http/http_response_headers.h"
#include "net/http/http_util.h"

#include "net/disk_cache/cache_tool_ti.h"
///
/// 
//-----------------------------------------------------------------------------
// completion callback helper

// A helper class for completion callbacks, designed to make it easy to run
// tests involving asynchronous operations.  Just call WaitForResult to wait
// for the asynchronous operation to complete.  Uses a RunLoop to spin the
// current MessageLoop while waiting.  The callback must be invoked on the same
// thread WaitForResult is called on.
//
// NOTE: Since this runs a message loop to wait for the completion callback,
// there could be other side-effects resulting from WaitForResult.  For this
// reason, this class is probably not ideal for a general application.
//
namespace base {
class RunLoop;
}

namespace net {

class IOBuffer;

namespace internal {

class TestCompletionCallbackBaseInternal {
 public:
  bool have_result() const { return have_result_; }

 protected:
  TestCompletionCallbackBaseInternal();
  virtual ~TestCompletionCallbackBaseInternal();

  void DidSetResult();
  void WaitForResult();

 private:
  // RunLoop.  Only non-NULL during the call to WaitForResult, so the class is
  // reusable.
  std::unique_ptr<base::RunLoop> run_loop_;
  bool have_result_;

  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallbackBaseInternal);
};

template <typename R>
struct NetErrorIsPendingHelper {
  bool operator()(R status) const { return status == ERR_IO_PENDING; }
};

template <typename R, typename IsPendingHelper = NetErrorIsPendingHelper<R>>
class TestCompletionCallbackTemplate
    : public TestCompletionCallbackBaseInternal {
 public:
  virtual ~TestCompletionCallbackTemplate() override {}

  R WaitForResult() {
    TestCompletionCallbackBaseInternal::WaitForResult();
    return std::move(result_);
  }

  R GetResult(R result) {
    IsPendingHelper check_pending;
    if (!check_pending(result))
      return std::move(result);
    return WaitForResult();
  }

 protected:
  TestCompletionCallbackTemplate() : result_(R()) {}

  // Override this method to gain control as the callback is running.
  virtual void SetResult(R result) {
    result_ = std::move(result);
    DidSetResult();
  }

 private:
  R result_;

  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallbackTemplate);
};


void TestCompletionCallbackBaseInternal::DidSetResult() {
  have_result_ = true;
  if (run_loop_)
    run_loop_->Quit();
}

void TestCompletionCallbackBaseInternal::WaitForResult() {
  DCHECK(!run_loop_);
  if (!have_result_) {
    run_loop_ = std::make_unique<base::RunLoop>(
        base::RunLoop::Type::kNestableTasksAllowed);
    run_loop_->Run();
    run_loop_.reset();
    DCHECK(have_result_);
  }
  have_result_ = false;  // Auto-reset for next callback.
}

TestCompletionCallbackBaseInternal::TestCompletionCallbackBaseInternal()
    : have_result_(false) {}

TestCompletionCallbackBaseInternal::~TestCompletionCallbackBaseInternal() =
    default;


}  // namespace internal


// Base class overridden by custom implementations of TestCompletionCallback.
typedef internal::TestCompletionCallbackTemplate<int>
    TestCompletionCallbackBase;

typedef internal::TestCompletionCallbackTemplate<int64_t>
    TestInt64CompletionCallbackBase;

class TestCompletionCallback : public TestCompletionCallbackBase {
 public:
  TestCompletionCallback() {}
  ~TestCompletionCallback() override;

  CompletionOnceCallback callback() {
    return base::BindOnce(&TestCompletionCallback::SetResult,
                          base::Unretained(this));
  }

 private:
  DISALLOW_COPY_AND_ASSIGN(TestCompletionCallback);
};

class TestInt64CompletionCallback : public TestInt64CompletionCallbackBase {
 public:
  TestInt64CompletionCallback() {}
  ~TestInt64CompletionCallback() override;

  Int64CompletionOnceCallback callback() {
    return base::BindOnce(&TestInt64CompletionCallback::SetResult,
                          base::Unretained(this));
  }

 private:
  DISALLOW_COPY_AND_ASSIGN(TestInt64CompletionCallback);
};

// Makes sure that the buffer is not referenced when the callback runs.
class ReleaseBufferCompletionCallback : public TestCompletionCallback {
 public:
  explicit ReleaseBufferCompletionCallback(IOBuffer* buffer);
  ~ReleaseBufferCompletionCallback() override;

 private:
  void SetResult(int result) override;

  IOBuffer* buffer_;
  DISALLOW_COPY_AND_ASSIGN(ReleaseBufferCompletionCallback);
};


TestCompletionCallback::~TestCompletionCallback() = default;

TestInt64CompletionCallback::~TestInt64CompletionCallback() = default;

ReleaseBufferCompletionCallback::ReleaseBufferCompletionCallback(
    IOBuffer* buffer)
    : buffer_(buffer) {}

ReleaseBufferCompletionCallback::~ReleaseBufferCompletionCallback() = default;

void ReleaseBufferCompletionCallback::SetResult(int result) {
  if (!buffer_->HasOneRef())
    result = ERR_FAILED;
  TestCompletionCallback::SetResult(result);
}
}  // namespace net



// -----------------------------------------------------------------------
//this file

using base::Time;
using base::TimeDelta;



// -----------------------------------------------------------------------

namespace disk_cache {
namespace {

    
// -----------------------------------------------------------------------
// TestEntryResult callback define
// Like net::TestCompletionCallback, but for EntryResultCallback.
struct EntryResultIsPendingHelper {
  bool operator()(const disk_cache::EntryResult& result) const {
    return result.net_error() == net::ERR_IO_PENDING;
  }
};
using TestEntryResultCompletionCallbackBase =
    net::internal::TestCompletionCallbackTemplate<disk_cache::EntryResult,
                                                  EntryResultIsPendingHelper>;

class TestEntryResultCompletionCallback
    : public TestEntryResultCompletionCallbackBase {
 public:
  TestEntryResultCompletionCallback();
  ~TestEntryResultCompletionCallback() override;

  disk_cache::Backend::EntryResultCallback callback();

 private:
  DISALLOW_COPY_AND_ASSIGN(TestEntryResultCompletionCallback);
};

TestEntryResultCompletionCallback::TestEntryResultCompletionCallback() =
    default;

TestEntryResultCompletionCallback::~TestEntryResultCompletionCallback() =
    default;

disk_cache::Backend::EntryResultCallback
TestEntryResultCompletionCallback::callback() {
  return base::BindOnce(&TestEntryResultCompletionCallback::SetResult,
                        base::Unretained(this));
}


const char kBlockFileBackendType[] = "block_file";
const char kSimpleBackendType[] = "simple";

const char kDiskCacheType[] = "disk_cache";
const char kAppCacheType[] = "app_cache";

const char kPrivateDirty[] = "Private_Dirty:";
const char kReadWrite[] = "rw-";
const char kHeap[] = "[heap]";
const char kKb[] = "kB";

struct CacheSpec {
 public:
  static std::unique_ptr<CacheSpec> Parse(const std::string& spec_string) {
    std::vector<std::string> tokens = base::SplitString(
        spec_string, ":", base::TRIM_WHITESPACE, base::SPLIT_WANT_ALL);
    if (tokens.size() != 3)
      return std::unique_ptr<CacheSpec>();
    if (tokens[0] != kBlockFileBackendType && tokens[0] != kSimpleBackendType)
      return std::unique_ptr<CacheSpec>();
    if (tokens[1] != kDiskCacheType && tokens[1] != kAppCacheType)
      return std::unique_ptr<CacheSpec>();

    #if 1
    const base::FilePath::CharType cacheFileName[] =
        L"C:/Users/zhibin/AppData/Roaming/eventdemo/Cache";
    //FilePath log_file_path(kLogFileName);

    return std::unique_ptr<CacheSpec>(new CacheSpec(
        tokens[0] == kBlockFileBackendType ? net::CACHE_BACKEND_BLOCKFILE
                                           : net::CACHE_BACKEND_SIMPLE,
        tokens[1] == kDiskCacheType ? net::DISK_CACHE : net::APP_CACHE,
        base::FilePath(cacheFileName )));
    #else
    return std::unique_ptr<CacheSpec>(new CacheSpec(
        tokens[0] == kBlockFileBackendType ? net::CACHE_BACKEND_BLOCKFILE
                                           : net::CACHE_BACKEND_SIMPLE,
        tokens[1] == kDiskCacheType ? net::DISK_CACHE : net::APP_CACHE,
        base::FilePath(tokens[2])));
#endif
  }

  const net::BackendType backend_type;
  const net::CacheType cache_type;
  const base::FilePath path;

 private:
  CacheSpec(net::BackendType backend_type,
            net::CacheType cache_type,
            const base::FilePath& path)
      : backend_type(backend_type),
        cache_type(cache_type),
        path(path) {
  }
};

void SetSuccessCodeOnCompletion(base::RunLoop* run_loop,
                                bool* succeeded,
                                int net_error) {
  if (net_error == net::OK) {
    *succeeded = true;
  } else {
    *succeeded = false;
  }
  run_loop->Quit();
}



std::unique_ptr<Backend> CreateAndInitBackend(const CacheSpec& spec) {
  std::unique_ptr<Backend> result;
  std::unique_ptr<Backend> backend;
  bool succeeded = false;
  base::RunLoop run_loop;
  net::CompletionOnceCallback callback =
      base::BindOnce(&SetSuccessCodeOnCompletion, &run_loop, &succeeded);
  const int net_error =
      CreateCacheBackend(spec.cache_type, spec.backend_type, spec.path, 0,
                         false, nullptr, &backend, std::move(callback));
  if (net_error == net::OK)
    SetSuccessCodeOnCompletion(&run_loop, &succeeded, net::OK);
  else
    run_loop.Run();
  if (!succeeded) {
    LOG(ERROR) << "Could not initialize backend in "
               << spec.path.LossyDisplayName();
    return result;
  }
  // For the simple cache, the index may not be initialized yet.
  if (spec.backend_type == net::CACHE_BACKEND_SIMPLE) {
    base::RunLoop index_run_loop;
    net::CompletionOnceCallback index_callback = base::BindOnce(
        &SetSuccessCodeOnCompletion, &index_run_loop, &succeeded);
    SimpleBackendImpl* simple_backend =
        static_cast<SimpleBackendImpl*>(backend.get());
    simple_backend->index()->ExecuteWhenReady(std::move(index_callback));
    index_run_loop.Run();
    if (!succeeded) {
      LOG(ERROR) << "Could not initialize Simple Cache in "
                 << spec.path.LossyDisplayName();
      return result;
    }
  }
  DCHECK(backend);
  result.swap(backend);
  return result;
}













#if 1//zhibin:test
int getpid() {
  return 0;
}
#endif

// Parses range lines from /proc/<PID>/smaps, e.g. (anonymous read write):
// 7f819d88b000-7f819d890000 rw-p 00000000 00:00 0
bool ParseRangeLine(const std::string& line,
                    std::vector<std::string>* tokens,
                    bool* is_anonymous_read_write) {
  *tokens = base::SplitString(line, base::kWhitespaceASCII,
                              base::KEEP_WHITESPACE, base::SPLIT_WANT_NONEMPTY);
  if (tokens->size() == 5) {
    const std::string& mode = (*tokens)[1];
    *is_anonymous_read_write = !mode.compare(0, 3, kReadWrite);
    return true;
  }
  // On Android, most of the memory is allocated in the heap, instead of being
  // mapped.
  if (tokens->size() == 6) {
    const std::string& type = (*tokens)[5];
    *is_anonymous_read_write = (type == kHeap);
    return true;
  }
  return false;
}

// Parses range property lines from /proc/<PID>/smaps, e.g.:
// Private_Dirty:        16 kB
//
// Returns |false| iff it recognizes a new range line. Outputs non-zero |size|
// only if parsing succeeded.
bool ParseRangeProperty(const std::string& line,
                        std::vector<std::string>* tokens,
                        uint64_t* size,
                        bool* is_private_dirty) {
  *tokens = base::SplitString(line, base::kWhitespaceASCII,
                              base::KEEP_WHITESPACE, base::SPLIT_WANT_NONEMPTY);

  // If the line is long, attempt to parse new range outside of this scope.
  if (tokens->size() > 3)
    return false;

  // Skip the line on other parsing error occasions.
  if (tokens->size() < 3)
    return true;
  const std::string& type = (*tokens)[0];
  if (type != kPrivateDirty)
    return true;
  const std::string& unit = (*tokens)[2];
  if (unit != kKb) {
    LOG(WARNING) << "Discarding value not in kB: " << line;
    return true;
  }
  const std::string& size_str = (*tokens)[1];
  uint64_t map_size = 0;
  if (!base::StringToUint64(size_str, &map_size))
    return true;
  *is_private_dirty = true;
  *size = map_size;
  return true;
}

uint64_t GetMemoryConsumption() {
#if 1//zhibin:test
  if (true)
  return 0;
  else {

    #endif
  std::ifstream maps_file(
      base::StringPrintf("/proc/%d/smaps", getpid()).c_str());
  if (!maps_file.good()) {
    LOG(ERROR) << "Could not open smaps file.";
    return false;
  }
  std::string line;
  std::vector<std::string> tokens;
  uint64_t total_size = 0;
  if (!std::getline(maps_file, line) || line.empty())
    return total_size;
  while (true) {
    bool is_anonymous_read_write = false;
    if (!ParseRangeLine(line, &tokens, &is_anonymous_read_write)) {
      LOG(WARNING) << "Parsing smaps - did not expect line: " << line;
    }
    if (!std::getline(maps_file, line) || line.empty())
      return total_size;
    bool is_private_dirty = false;
    uint64_t size = 0;
    while (ParseRangeProperty(line, &tokens, &size, &is_private_dirty)) {
      if (is_anonymous_read_write && is_private_dirty) {
        total_size += size;
        is_private_dirty = false;
      }
      if (!std::getline(maps_file, line) || line.empty())
        return total_size;
    }
  }
  return total_size;
  #if 1//zhibin:test
  }
  #endif
}


// Gets a key's stream to a buffer.
scoped_refptr<net::GrowableIOBuffer> GetStreamForKeyBuffer(
    Backend* backend,
    const std::string& key,
    int index) {

  TestEntryResultCompletionCallback cb_open;
  EntryResult result = backend->OpenEntry(
      key, net::HIGHEST, cb_open.callback());
  result = cb_open.GetResult(std::move(result));
  if (result.net_error() != net::OK) {
    std::cout << "Couldn't find key's entry." << std::endl;
    return nullptr;
  }
  Entry* cache_entry = result.ReleaseEntry();

  const int kInitBufferSize = 8192;
  scoped_refptr<net::GrowableIOBuffer> buffer =
      base::MakeRefCounted<net::GrowableIOBuffer>();
  buffer->SetCapacity(kInitBufferSize);
  net::TestCompletionCallback cb;
  while (true) {
    int rv = cache_entry->ReadData(index, buffer->offset(), buffer.get(),
                                   buffer->capacity() - buffer->offset(),
                                   cb.callback());
    rv = cb.GetResult(rv);
    if (rv < 0) {
      cache_entry->Close();
      std::cout << "Stream read error.." << std::endl;
      return nullptr;
    }
    buffer->set_offset(buffer->offset() + rv);
    if (rv == 0)
      break;
    buffer->SetCapacity(buffer->offset() * 2);
  }
  cache_entry->Close();
  return buffer;
}
using disk_cache::Backend;
using disk_cache::Entry;
using disk_cache::EntryResult;
constexpr int kResponseInfoIndex = 0;
constexpr int kResponseContentIndex = 1;
void GetStreamForKey(Backend* backend, std::string url, int index) {
  std::string key = "https://home.baidu.com/Public/img/play.png?v=12";
  //"https://home.baidu.com/Public/js/fontbase.js?v=12";
 // "http://192.168.50.206:8080/jquery.js";
 // "https://home.baidu.com/Public/img/play.png?v=12";
  // index = 0;//0 header 1,2 
   std::cout << "=== index ="<<index << std::endl;
  scoped_refptr<net::GrowableIOBuffer> buffer(
      GetStreamForKeyBuffer(backend, key, index));
  
  if (index == kResponseInfoIndex) {
    net::HttpResponseInfo response_info;
    bool truncated_response_info = false;
    if (!net::HttpCache::ParseResponseInfo(buffer->StartOfBuffer(),
                                           buffer->offset(), &response_info,
                                           &truncated_response_info)) {
      // This can happen when reading data stored by content::CacheStorage.
      std::cerr << "WARNING: Returning empty response info for key: " << key
                << std::endl;
      //command_marshal->ReturnSuccess();
      //return command_marshal->ReturnString("");
    }
    if (truncated_response_info)
      std::cerr << "WARNING: Truncated HTTP response." << std::endl;
    //command_marshal->ReturnSuccess();
    std::cout<<
        net::HttpUtil::ConvertHeadersBackToHTTPResponse(
            response_info.headers->raw_headers())<<std::endl;
  } else if (index == kResponseContentIndex) {
    //command_marshal->ReturnSuccess();
    std::cout.write(buffer->StartOfBuffer(), buffer->offset());
  }
}


void OnEntryResultComplete(base::RunLoop* run_loop,
                           bool* succeeded,int* sum,
                           EntryResult result) {
  auto rv = result.net_error();

  if (rv == net::OK) {
    *succeeded = true;

    Entry* entry = result.ReleaseEntry();
    std::string url = entry->GetKey();
    (*sum)++;
    std::cout << *sum << "\t:" << url << std::endl;
    entry->Close();

  } else {
    *succeeded = false;
  }

  run_loop->Quit();

  if (rv == net::ERR_FAILED) {
    std::cout << "=== read failed." << std::endl;
  }
}

bool CacheMemTest(const std::vector<std::unique_ptr<CacheSpec>>& specs) {
  std::vector<std::unique_ptr<Backend>> backends;
  for (const auto& it : specs) {
    std::unique_ptr<Backend> backend = CreateAndInitBackend(*it);
    if (!backend) {
      std::cout << "Get backend error.";
      return false;
    }

    std::cout << "Number of entries in " << it->path.LossyDisplayName() << " : "
              << backend->GetEntryCount() << std::endl;
    if (true) {
        //complecated for list
        //http://tb2.bdstatic.com/tb/static-common/img/search_logo_big_v1_8d039f9.png
        //
    if (false) {
      //list
      std::unique_ptr<Backend::Iterator> entry_iterator =
          backend->CreateIterator();
      TestEntryResultCompletionCallback cb;
      EntryResult result = entry_iterator->OpenNextEntry(cb.callback());
      //command_marshal->ReturnSuccess();
      while ((result = cb.GetResult(std::move(result))).net_error() ==
             net::OK) {
        Entry* entry = result.ReleaseEntry();
        std::string url = entry->GetKey();
      
        std ::cout << url << std ::endl;
        entry->Close();
        result = entry_iterator->OpenNextEntry(cb.callback());
      }
        }//command_marshal->ReturnString("");
    else {
      GetStreamForKey(backend.get(), "", 0);
          GetStreamForKey(backend.get(), "", 1);
      GetStreamForKey(backend.get(), "", 2);
    }
    }else {
      bool succeeded0;
      std::unique_ptr<base::RunLoop> run_loop0;
      int sum = 0;

      std::unique_ptr<disk_cache::Backend::Iterator> iter_ =
          backend->CreateIterator();
      while (true) {
        //初始化和重新绑定回调，因为loop只能run一次。
        succeeded0 = false;
        run_loop0 = std::make_unique<base::RunLoop>(
            base::RunLoop::Type::kNestableTasksAllowed);
        disk_cache::Backend::EntryResultCallback entryCallback = base::BindOnce(
            &OnEntryResultComplete, run_loop0.get(), &succeeded0, &sum);

        EntryResult result = iter_->OpenNextEntry(std::move(entryCallback));

        auto rv = result.net_error();
        if (rv != net::OK) {
          // 运行loop.run阻塞等待。。。
          run_loop0->Run();
          run_loop0.reset();         
        } else {
          std::cout << "=== direct get result. net::OK." << std::endl;
          OnEntryResultComplete(run_loop0.get(), &succeeded0, &sum,
                                {});  // std::move(result) EntryResult构造被禁用了。
        }

        if (!succeeded0) {
          std::cout << "Could not get backend in "
                    << it->path.LossyDisplayName();
          break;
        }
      }
    }

    backends.push_back(std::move(backend));
  }
  const uint64_t memory_consumption = GetMemoryConsumption();
  std::cout << "Private dirty memory: " << memory_consumption << " kB"
            << std::endl;
  return true;
}

void PrintUsage(std::ostream* stream) {
  *stream << "Usage: disk_cache_mem_test "
          << "--spec-1=<spec> "
          << "[--spec-2=<spec>]"
          << std::endl
          << "  with <cache_spec>=<backend_type>:<cache_type>:<cache_path>"
          << std::endl
          << "       <backend_type>='block_file'|'simple'" << std::endl
          << "       <cache_type>='disk_cache'|'app_cache'" << std::endl
          << "       <cache_path>=file system path" << std::endl;
}

bool ParseAndStoreSpec(const std::string& spec_str,
                       std::vector<std::unique_ptr<CacheSpec>>* specs) {
  std::unique_ptr<CacheSpec> spec = CacheSpec::Parse(spec_str);
  if (!spec) {
    PrintUsage(&std::cerr);
    return false;
  }
  specs->push_back(std::move(spec));
  return true;
}

bool Main(int argc, char** argv) {
  base::AtExitManager at_exit_manager;

  if (true) {
    auto* util = new disk_cache::util::CacheUtil();  //::GetContent("");
    auto buffer = util->GetContent("");
    std::cout.write(buffer->StartOfBuffer(), buffer->offset());
    return true;
  }

  base::SingleThreadTaskExecutor executor(base::MessagePumpType::IO);
  base::ThreadPoolInstance::CreateAndStartWithDefaultParams(
      "disk_cache_memory_test");
  base::CommandLine::Init(argc, argv);
  const base::CommandLine& command_line =
      *base::CommandLine::ForCurrentProcess();
  if (command_line.HasSwitch("help")) {
    PrintUsage(&std::cout);
    return true;
  }
  if ((command_line.GetSwitches().size() != 1 &&
       command_line.GetSwitches().size() != 2) ||
      !command_line.HasSwitch("spec-1") ||
      (command_line.GetSwitches().size() == 2 &&
       !command_line.HasSwitch("spec-2"))) {
    PrintUsage(&std::cerr);
    return false;
  }
  std::vector<std::unique_ptr<CacheSpec>> specs;
  const std::string spec_str_1 = command_line.GetSwitchValueASCII("spec-1");
  if (!ParseAndStoreSpec(spec_str_1, &specs))
    return false;
  if (command_line.HasSwitch("spec-2")) {
    const std::string spec_str_2 = command_line.GetSwitchValueASCII("spec-2");
    if (!ParseAndStoreSpec(spec_str_2, &specs))
      return false;
  }
  return CacheMemTest(specs);
}

}  // namespace
}  // namespace disk_cache

int main(int argc, char** argv) {
//  int i;
//std::cin >> i;

  return !disk_cache::Main(argc, argv);
}

View Code

上面是独立的exe程序使用。不能在chrome中用。

js 获取图片url的Blob值并预览
1）使用 XMLHttpRequest 对象获取图片url的Blob值

复制代码
//获取图片的Blob值
function getImageBlob(url, cb) {
    var xhr          = new XMLHttpRequest();
    xhr.open("get", url, true);
    xhr.responseType = "blob";
    xhr.onload       = function() {
        if (this.status == 200) {
            if(cb) cb(this.response);
        }
    };
    xhr.send();
}
复制代码
注意这里的XMLHttpRequest必须使用异步模式，同步模式不能设置 responseType = "blob"

 

2）使用 FileReader 对象获取图片 Blob 对象的 data 数据

复制代码
function preView(url){
    let reader    = new FileReader();

    getImageBlob( url , function(blob){
        reader.readAsDataURL(blob);
    });

    reader.onload = function(e) {
        var img = document.createElement("img");
        img.src = e.target.result;
        document.body.appendChild(img);
    }
}
复制代码

终极版：

js这样发起调用：

var myFirstPromise =win.webContents.session.getCacheSize();
myFirstPromise.then(function(successMessage){
    //successMessage的值是上面调用resolve(...)方法传入的值.
    console.log("========== getCacheSize:"+successMessage
)

c++对应实现：

atom_api_session.cc electron的js binding函数实现
v8::Local<v8::Promise> Session::GetCacheSize() {
  auto* isolate = v8::Isolate::GetCurrent();
  auto promise = util::Promise(isolate);
  auto handle = promise.GetHandle();

  content::BrowserContext::GetDefaultStoragePartition(browser_context_.get())
      ->GetNetworkContext()
      ->ComputeHttpCacheSize(base::Time(), base::Time::Max(),
                             base::BindOnce(
                                 [](util::Promise promise, bool is_upper_bound,
                                    int64_t size_or_error) {
                                   if (size_or_error < 0) {
                                     promise.RejectWithErrorMessage(
                                         net::ErrorToString(size_or_error));
                                   } else {
                                     promise.Resolve(size_or_error);
                                   }
                                 },
                                 std::move(promise)));

  return handle;
}

D:\dev\electron7\src\services\network\network_context.cc  mojom 实现
void NetworkContext::ComputeHttpCacheSize(
    base::Time start_time,
    base::Time end_time,
    ComputeHttpCacheSizeCallback callback) {
  // It's safe to use Unretained below as the HttpCacheDataCounter is owned by
  // |this| and guarantees it won't call its callback if deleted.
  http_cache_data_counters_.push_back(HttpCacheDataCounter::CreateAndStart(
      url_request_context_, start_time, end_time,
      base::BindOnce(&NetworkContext::OnHttpCacheSizeComputed,
                     base::Unretained(this), std::move(callback))));
}

//
void NetworkContext::OnHttpCacheSizeComputed(
    ComputeHttpCacheSizeCallback callback,
    HttpCacheDataCounter* counter,
    bool is_upper_limit,
    int64_t result_or_error) {
  EraseIf(http_cache_data_counters_, base::MatchesUniquePtr(counter));
  std::move(callback).Run(is_upper_limit, result_or_error);//这里设置了最终运算的结果，然后回调到 atom_api_session.cc的红色部分回调函数实现，通过run。

中间通过mojom远程调用连接类似RMI：

接口文件：D:\dev\electron7\src\services\network\public\mojom\network_context.mojom

实现：

D:\dev\electron7\src\services\network\network_context.cc

D:\dev\electron7\src\services\network\network_context.h

  // Invoked when the computation for ComputeHttpCacheSize() has been completed,
  // to report result to user via |callback| and clean things up.
  void OnHttpCacheSizeComputed(ComputeHttpCacheSizeCallback callback,
                               HttpCacheDataCounter* counter,
                               bool is_upper_limit,
                               int64_t result_or_error);

真正实现读取cache size的类

D:\dev\electron7\src\services\network\http_cache_data_counter.h

HttpCacheDataCounter::CreateAndStart创建在 D:\dev\electron7\src\services\network\http_cache_data_counter.cc

std::unique_ptr<HttpCacheDataCounter> HttpCacheDataCounter::CreateAndStart(
    net::URLRequestContext* url_request_context,
    base::Time start_time,
    base::Time end_time,
    HttpCacheDataCounterCallback callback) {
  HttpCacheDataCounter* instance =
      new HttpCacheDataCounter(start_time, end_time, std::move(callback));
  net::HttpCache* http_cache =
      url_request_context->http_transaction_factory()->GetCache();
  if (!http_cache) {
    // No cache, no space used. Posts a task, so it will run after the return.
    instance->PostResult(false, 0);
  } else {
    // Tricky here: if |this| gets deleted before |http_cache| gets deleted,
    // GetBackend may still write things out (even though the callback will
    // abort due to weak pointer), so the destination for the pointer can't be
    // owned by |this|.
    //
    // While it can be transferred to the callback to GetBackend, that callback
    // also needs to be kept alive for the duration of this method in order to
    // get at the backend pointer in the synchronous result case.
    auto backend = std::make_unique<disk_cache::Backend*>();
    disk_cache::Backend** backend_ptr = backend.get();

    auto get_backend_callback =
        base::BindRepeating(&HttpCacheDataCounter::GotBackend,
                            instance->GetWeakPtr(), base::Passed(&backend));
    int rv = http_cache->GetBackend(backend_ptr, get_backend_callback);
    if (rv != net::ERR_IO_PENDING) {
      instance->GotBackend(std::make_unique<disk_cache::Backend*>(*backend_ptr),
                           rv);
    }
  }

  return base::WrapUnique(instance);
}

HttpCacheDataCounter::HttpCacheDataCounter(
    base::Time start_time,
    base::Time end_time,
    HttpCacheDataCounterCallback callback)
    : start_time_(start_time),
      end_time_(end_time),
      callback_(std::move(callback)) {}

HttpCacheDataCounter::~HttpCacheDataCounter() {}

void HttpCacheDataCounter::GotBackend(
    std::unique_ptr<disk_cache::Backend*> backend,
    int error_code) {
  DCHECK_LE(error_code, 0);
  bool is_upper_limit = false;
  if (error_code != net::OK) {
    PostResult(is_upper_limit, error_code);
    return;
  }

  if (!*backend) {
    PostResult(is_upper_limit, 0);
    return;
  }

  int64_t rv;
  disk_cache::Backend* cache = *backend;

  // Handle this here since some backends would DCHECK on this.
  if (start_time_ > end_time_) {
    PostResult(is_upper_limit, 0);
    return;
  }

  if (start_time_.is_null() && end_time_.is_max()) {
    rv = cache->CalculateSizeOfAllEntries(base::BindOnce(//计算缓存大小的代码在这里！！！不计代价的算。
        &HttpCacheDataCounter::PostResult, GetWeakPtr(), is_upper_limit));
  } else {
    //在给定的时间限制内算
    rv = cache->CalculateSizeOfEntriesBetween(
        start_time_, end_time_,
        base::BindOnce(&HttpCacheDataCounter::PostResult, GetWeakPtr(),
                       is_upper_limit));
    if (rv == net::ERR_NOT_IMPLEMENTED) {
      is_upper_limit = true;
      rv = cache->CalculateSizeOfAllEntries(base::BindOnce(
          &HttpCacheDataCounter::PostResult, GetWeakPtr(), is_upper_limit));
    }
  }
  if (rv != net::ERR_IO_PENDING)
    PostResult(is_upper_limit, rv);
}

void HttpCacheDataCounter::PostResult(bool is_upper_limit,
                                      int64_t result_or_error) {
  base::SequencedTaskRunnerHandle::Get()->PostTask(
      FROM_HERE, base::BindOnce(std::move(callback_), this, is_upper_limit,
                                result_or_error));
}

最终取得的缓存大小值：

void HttpCacheDataCounter::PostResult(bool is_upper_limit,
                                      int64_t result_or_error) {
  base::SequencedTaskRunnerHandle::Get()->PostTask(
      FROM_HERE, base::BindOnce(std::move(callback_), this, is_upper_limit,
                                result_or_error));
}

GetBlobData 原理：

v8::Local<v8::Promise> Session::GetBlobData(v8::Isolate* isolate,const std::string& url) {
                                
  gin::Handle<DataPipeHolder> holder = DataPipeHolder::From(isolate, url);
  if (holder.IsEmpty()) {
    util::Promise promise(isolate);
    promise.RejectWithErrorMessage("Could not get blob data handle");
    return promise.GetHandle();
  }

  return holder->ReadAll(isolate);  
}

DataPipeHolder： data_pipe_ 是远程stub，在哪里实现？？？？？？

v8::Local<v8::Promise> DataPipeHolder::ReadAll(v8::Isolate* isolate) {
  util::Promise promise(isolate);
  v8::Local<v8::Promise> handle = promise.GetHandle();
  if (!data_pipe_) {
    promise.RejectWithErrorMessage("Could not get blob data");
    return handle;
  }

  new DataPipeReader(std::move(promise), std::move(data_pipe_));
  return handle;
}

new datapipe reader:

// Utility class to read from data pipe.
class DataPipeReader {
 public:
  DataPipeReader(util::Promise promise,
                 network::mojom::DataPipeGetterPtr data_pipe_getter)
      : promise_(std::move(promise)),
        data_pipe_getter_(std::move(data_pipe_getter)),
        handle_watcher_(FROM_HERE,
                        mojo::SimpleWatcher::ArmingPolicy::MANUAL,
                        base::SequencedTaskRunnerHandle::Get()) {
    // Get a new data pipe and start.
    mojo::DataPipe data_pipe;
    data_pipe_getter_->Read(std::move(data_pipe.producer_handle),
                            base::BindOnce(&DataPipeReader::ReadCallback, //远程回调
                                           weak_factory_.GetWeakPtr()));
    data_pipe_ = std::move(data_pipe.consumer_handle);
    handle_watcher_.Watch(data_pipe_.get(), MOJO_HANDLE_SIGNAL_READABLE,
                          base::BindRepeating(&DataPipeReader::OnHandleReadable, //远程回调
                                              weak_factory_.GetWeakPtr()));
  }

 network::mojom::DataPipeGetterPtr这个调往远程接口。

所有的回调：

  // Callback invoked by DataPipeGetter::Read.
  void ReadCallback(int32_t status, uint64_t size) {
    if (status != net::OK) {
      OnFailure();
      return;
    }
    buffer_.resize(size);
    head_ = &buffer_.front();
    remaining_size_ = size;
    handle_watcher_.ArmOrNotify();
  }

  // Called by |handle_watcher_| when data is available or the pipe was closed,
  // and there's a pending Read() call.
  void OnHandleReadable(MojoResult result) {
    if (result != MOJO_RESULT_OK) {
      OnFailure();
      return;
    }

    // Read.
    uint32_t length = remaining_size_;
    result = data_pipe_->ReadData(head_, &length, MOJO_READ_DATA_FLAG_NONE);
    if (result == MOJO_RESULT_OK) {  // success
      remaining_size_ -= length;
      head_ += length;
      if (remaining_size_ == 0)
        OnSuccess();
    } else if (result == MOJO_RESULT_SHOULD_WAIT) {  // IO pending
      handle_watcher_.ArmOrNotify();
    } else {  // error
      OnFailure();
    }
  }

  void OnFailure() {
    promise_.RejectWithErrorMessage("Could not get blob data");
    delete this;
  }

  void OnSuccess() {
    // Pass the buffer to JS.
    //
    // Note that the lifetime of the native buffer belongs to us, and we will
    // free memory when JS buffer gets garbage collected.
    v8::Locker locker(promise_.isolate());
    v8::HandleScope handle_scope(promise_.isolate());
    v8::Local<v8::Value> buffer =
        node::Buffer::New(promise_.isolate(), &buffer_.front(), buffer_.size(),
                          &DataPipeReader::FreeBuffer, this)
            .ToLocalChecked();
    promise_.Resolve(buffer);

    // Destroy data pipe.
    handle_watcher_.Cancel();
    data_pipe_.reset();
    data_pipe_getter_ = nullptr;
  }

  static void FreeBuffer(char* data, void* self) {
    delete static_cast<DataPipeReader*>(self);
  }

资源参考：

For Developers‎ > ‎Design Documents‎ > ‎Network Stack‎ > ‎

Disk Cache

Overview

The disk cache stores resources fetched from the web so that they can be accessed quickly at a latter time if needed. The main characteristics of Chromium disk cache are:

The cache should not grow unbounded so there must be an algorithm for deciding when to remove old entries.
While it is not critical to lose some data from the cache, having to discard the whole cache should be minimized. The current design should be able to gracefully handle application crashes, no matter what is going on at that time, only discarding the resources that were open at that time. However, if the whole computer crashes while we are updating the cache, everything on the cache probably will be discarded.
Access to previously stored data should be reasonably efficient, and it should be possible to use synchronous or asynchronous operations.
We should be able to avoid conflicts that prevent us from storing two given resources simultaneously. In other words, the design should avoid cache trashing.
It should be possible to remove a given entry from the cache, and keep working with a given entry while at the same time making it inaccessible to other requests (as if it was never stored).
The cache should not be using explicit multithread synchronization because it will always be called from the same thread. However, callbacks should avoid reentrancy problems so they must be issued through the thread's message loop.

A new version of the cache is under development, and some sections of this documents are going to be stale soon. In particular, the description of a cache entry, and the big picture diagram (sections 3.4 and 3.5) are only valid for files saved with version 2.x

Note that on Android we don't use this implementation; we use the simple cache instead.

External Interface

Any implementation of Chromium's cache exposes two interfaces: disk_cache::Backend and disk_cache::Entry. (see src/net/disk_cache/disk_cache.h). The Backend provides methods to enumerate the resources stored on the cache (a.k.a Entries), open old entries or create new ones etc. Operations specific to a given resource are handled with the Entry interface.

An entry is identified by its key, which is just the name of the resource (for example http://www.google.com/favicon.ico ). Once an entry is created, the data for that particular resource is stored in separate chunks or data streams: one for the HTTP headers and another one for the actual resource data, so the index for the required stream is an argument to the Entry::ReadData and Entry::WriteData methods.

Disk Structure

All the files that store Chromium’s disk cache live in a single folder (you guessed it, it is called cache), and every file inside that folder is considered to be part of the cache (so it may be deleted by Chromium at some point!).

Chromium uses at least five files: one index file and four data files. If any of those files is missing or corrupt, the whole set of files is recreated. The index file contains the main hash table used to locate entries on the cache, and the data files contain all sorts of interesting data, from bookkeeping information to the actual HTTP headers and data of a given request. These data files are also known as block-files, because their file format is optimized to store information on fixed-size “blocks”. For instance, a given block-file may store blocks of 256 bytes and it will be used to store data that can span from one to four such blocks, in other words, data with a total size of 1 KB or less.

When the size of a piece of data is bigger than disk_cache::kMaxBlockSize (16 KB), it will no longer be stored inside one of our standard block-files. In this case, it will be stored in a “separate file”, which is a file that has no special headers and contains only the data we want to save. The name of a separate file follows the form f_xx, where xx is just the hexadecimal number that identifies the file.

Cache Address

Every piece of data stored by the disk cache has a given “cache address”. The cache address is simply a 32-bit number that describes exactly where the data is actually located.

A cache entry will have an address; the HTTP headers will have another address, the actual request data will have a different address, the entry name (key) may have another address and auxiliary information for the entry (such as the rankings info for the eviction algorithm) will have another address. This allows us to reuse the same infrastructure to efficiently store different types of data while at the same time keeping frequently modified data together, so that we can leverage the underlying operating system to reduce access latency.

The structure of a cache address is defined on disk_cache/addr.h, and basically tells if the required data is stored inside a block-file or as a separate file and the number of the file (block file or otherwise). If the data is part of a block-file, the cache address also has the number of the first block with the data, the number of blocks used and the type of block file.

These are few examples of valid addresses:

0x00000000: not initialized

0x8000002A: external file f_00002A

0xA0010003: block-file number 1 (data_1), initial block number 3, 1 block of length.

Index File Structure

The index file structure is specified on disk_cache/disk_format.h. Basically, it is just an disk_cache::IndexHeader structure followed by the actual hash table. The number of entries in the table is at least disk_cache::kIndexTablesize (65536), but the actual size is controlled by the table_len member of the header.

The whole file is memory mapped to allow fast translation between the hash of the name of a resource (the key), and the cache address that stores the resource. The low order bits of the hash are used to index the table, and the content of the table is the address of the first stored resource with the same low order bits on the hash.

One of the things that must be verified when dealing with the disk cache files (the index and every block-file) is that the magic number on the header matches the expected value, and that the version is correct. The version has a mayor and a minor part, and the expected behavior is that any change on the mayor number means that the format is now incompatible with older formats.

Block File Structure

The block-file structure is specified on disk_cache/disk_format.h. Basically, it is just a file header (disk_cache::BlockFileHeader) followed by a variable number of fixed-size data blocks. Block files are named data_n, where n is the decimal file number.

The header of the file (8 KB) is memory mapped to allow efficient creation and deletion of elements of the file. The bulk of the header is actually a bitmap that identifies used blocks on the file. The maximum number of blocks that can be stored on a single file is thus a little less than 64K.

Whenever there are not enough free blocks on a file to store more data, the file is grown by 1024 blocks until the maximum number of blocks is reached. At that moment, a new block-file of the same type is created, and the two files are linked together using the next_file member of the header. The type of the block-file is simply the size of the blocks that the file stores, so all files that store blocks of the same size are linked together. Keep in mind that even if there are multiple block-files chained together, the cache address points directly to the file that stores a given record. The chain is only used when looking for space to allocate a new record.

To simplify allocation of disk space, it is only possible to store records that use from one to four actual blocks. If the total size of the record is bigger than that, another type of block-file must be used. For example, to store a string of 2420 bytes, three blocks of 1024 bytes are needed, so that particular string will go to the block-file that has blocks of 1KB.

Another simplification of the allocation algorithm is that a piece of data is not going to cross the four block alignment boundary. In other words, if the bitmap says that block 0 is used, and everything else is free (state A), and we want to allocate space for four blocks, the new record will use blocks 4 through 7 (state B), leaving three unused blocks in the middle. However, if after that we need to allocate just two blocks instead of four, the new record will use blocks 1 and 2 (state C).

There are a couple of fields on the header to help the process of allocating space for a new record. The empty field stores counters of available space per block type and hints stores the last scanned location per block type. In this context, a block type is the number of blocks requested for the allocation. When a file is empty, it can store up to X records of four blocks each (X being close to 64K / 4). After a record of one block is allocated, it is able be able to store X-1 records of four blocks, and one record of three blocks. If after that, a record of two blocks is allocated, the new capacity is X-1 records of four blocks and one record of one block, because the space that was available to store the record of three blocks was used to store the new record (two blocks), leaving one empty block.

It is important to realize that once a record has been allocated, its size cannot be increased. The only way to grow a record that was already saved is to read it, then delete it from the file and allocate a new record of the required size.

From the reliability point of view, having the header memory mapped allows us to detect scenarios when the application crashes while we are in the middle of modifying the allocation bitmap. The updating field of the header provides a way to signal that we are updating something on the headers, so that if the field is set when the file is open, the header must be checked for consistency.

Cache Entry

An entry is basically a complete entity stored by the cache. It is divided in two main parts: the disk_cache::EntryStore stores the part that fully identifies the entry and doesn’t change very often, and the disk_cache::RankingsNode stores the part that changes often and is used to implement the eviction algorithm.

The RankingsNode is always the same size (36 bytes), and it is stored on a dedicated type of block files (with blocks of 36 bytes). On the other hand, the EntryStore can use from one to four blocks of 256 bytes each, depending on the actual size of the key (name of the resource). In case the key is too long to be stored directly as part of the EntryStore structure, the appropriate storage will be allocated and the address of the key will be saved on the long_key field, instead of the full key.

The other things stored within EntryStore are addresses of the actual data streams associated with this entry, the key’s hash and a pointer to the next entry that has the same low-order hash bits (and thus shares the same position on the index table).

Whenever an entry is in use, its RankingsNode is marked as in-use so that when a new entry is read from disk we can tell if it was properly closed or not.

The Big Picture

This diagram shows a disk cache with 7 files on disk: the index file, 5 block-files and one separate file. data_1 and data_4 are chained together so they store blocks of the same size (256 bytes), while data_2 stores blocks of 1KB and data_3 stores blocks of 4 KB. The depicted entry has its key stored outside the EntryStore structure, and given that it uses two blocks, it must be between one and two kilobytes. This entry also has two data streams, one for the HTTP headers (less than 256 bytes) and another one for the actual payload (more than 16 KB so it lives on a dedicated file). All blue arrows indicate that a cache address is used to locate another piece of data.

Implementation Notes

Chromium has two different implementations of the cache interfaces: while the main one is used to store info on a given disk, there is also a very simple implementation that doesn’t use a hard drive at all, and stores everything in memory. The in-memory implementation is used for the Incognito mode so that even if the application crashes it will be quite difficult to extract any information that was accessed while browsing in that mode.

There are a few different types of caches (see net/base/cache_type.h), mainly defined by their intended use: there is a media specific cache, the general purpose disk cache, and another one that serves as the back end storage for AppCache, in addition to the in-memory type already mentioned. All types of caches behave in a similar way, with the exception that the eviction algorithm used by the general purpose cache is not the same LRU used by the others.

The regular cache implementation is located on disk_cache/backend_impl.cc and disk_cache/entry_impl.cc. Most of the files on that folder are actually related to the main implementation, except for a few that implement the in-memory cache: disk_cache/mem_backend_impl.cc and disk_cache/entry_impl.cc.

Lower Interface

The lower interface of the disk cache (the one that deals with the OS) is handled mostly by two files: disk_cache/file.h and disk_cache/mapped_file.h, with separate implementations per operating system. The most notable requirement is support for partially memory-mapped files, but asynchronous interfaces and a decent file system level cache go a long way towards performance (we don’t want to replicate the work of the OS).

To deal with all the details about block-file access, the disk cache keeps a single object that deals with all of them: a disk_cache::BlockFiles object. This object enables allocation and deletion of disk space, and provides disk_cache::File object pointers to other people so that they can access the information that they need.

A StorageBlock is a simple template that represents information stored on a block-file, and it provides methods to load and store the required data from disk (based on the record’s cache address). We have two instantiations of the template, one for dealing with the EntryStore structure and another one for dealing with the RankingsNode structure. With this template, it is common to find code like entry->rankings()->Store().

Eviction

Support for the eviction algorithm of the cache is implemented on disk_cache/rankings (and mem_rankings for the in-memory one), and the eviction itself is implemented on disk_cache/eviction. Right now we have a simple Least Recently Used algorithm that just starts deleting old entries once a certain limit is exceeded, and a second algorithm that takes reuse and age into account before evicting an entry. We also have the concept of transaction when one of the the lists is modified so that if the application crashes in the middle of inserting or removing an entry, next time we will roll the change back or forward so that the list is always consistent.

In order to support reuse as a factor for evictions, we keep multiple lists of entries depending on their type: not reused, with low reuse and highly reused. We also have a list of recently evicted entries so that if we see them again we can adjust their eviction next time we need the space. There is a time-target for each list and we try to avoid eviction of entries without having the chance to see them again. If the cache uses only LRU, all lists except the not-reused are empty.

Buffering

When we start writing data for a new entry we allocate a buffer of 16 KB where we keep the first part of the data. If the total length is less than the buffer size, we only write the information to disk when the entry is closed; however, if we receive more than 16 KB, then we start growing that buffer until we reach a limit for this stream (1 MB), or for the total size of all the buffers that we have. This scheme gives us immediate response when receiving small entries (we just copy the data), and works well with the fact that the total record size is required in order to create a new cache address for it. It also minimizes the number of writes to disk so it improves performance and reduces disk fragmentation.

Deleting Entries

To delete entries from the cache, one of the Doom*() methods can be used. All that they do is to mark a given entry to be deleted once all users have closed the entry. Of course, this means that it is possible to open a given entry multiple times (and read and write to it simultaneously). When an entry is doomed (marked for deletion), it is removed from the index table so that any attempt to open it again will fail (and creating the entry will succeed), even when an already created Entry object can still be used to read and write the old entry.

When two objects are open at the same time, both users will see what the other is doing with the entry (there is only one “real” entry, and they see a consistent state of it). That’s even true if the entry is doomed after it was open twice. However, once the entry is created after it was doomed, we end up with basically two separate entries, one for the old, doomed entry, and another one for the newly created one.

Enumerations

A good example of enumerating the entries stored by the cache is located at src/net/url_request/url_request_view_cache_job.cc . It should be noted that this interface is not making any statements about the order in which the entries are enumerated, so it is not a good idea to make assumptions about it. Also, it could take a long time to go through all the info stored on disk.

Sparse Data

An entry can be used to store sparse data instead of a single, continuous stream. In this case, only two streams can be stored by the entry, a regular one (the first one), and a sparse one (the second one). Internally, the cache will distribute sparse chunks among a set of dedicated entries (child entries) that are linked together from the main entry (the parent entry). Each child entry will store a particular range of the sparse data, and inside that range we could have "holes" that have not been written yet. This design allows the user to store files of any length (even bigger than the total size of the cache), while the cache is in fact simultaneously evicting parts of that file, according to the regular eviction policy. Most of this logic is implemented on disk_cache/sparse_control (and disk_cache/mem_entry_impl for the in-memory case).

Dedicated Thread

We have a dedicated thread to perform most of the work, while the public API is called on the regular network thread (the browser's IO thread).

The reason for this dedicated thread is to be able to remove any potentially blocking call from the IO thread, because that thread serves the IPC messages with all the renderer and plugin processes: even if the browser's UI remains responsive when the IO thread is blocked, there is no way to talk to any renderer so tabs look unresponsive. On the other hand, if the computer's IO subsystem is under heavy load, any disk access can block for a long time.

Note that it may be possible to extend the use of asynchronous IO and just keep using the same thread. However, we are not really using asynchronous IO for Posix (due to compatibility issues), and even in Windows, not every operation can be performed asynchronously; for instance, opening and closing a file are always synchronous operations, so they are subject to significant delays under the proper set of circumstances.

Another thing to keep in mind is that we tend to perform a large number of IO operations, knowing that most of the time they just end up being completed by the system's cache. It would have been possible to use asynchronous operations all the time, but the code would have been much harder to understand because that means a lot of very fragmented state machines. And of course that doesn't solve the problem with Open/Close.

As a result, we have a mechanism to post tasks from the main thread (IO thread), to a background thread (Cache thread), and back, and we forward most of the API to the actual implementation that runs on the background thread. See disk_cache/in_flight_io and disk_cache/in_flight_backend_io. There are a few methods that are not forwarded to the dedicated thread, mostly because they don't interact with the files, and only provide state information. There is no locking to access the cache, so these methods are generally racing with actual modifications, but that non-racy guarantee is not made by the API. For example, getting the size of a data stream (entry::GetDataSize()) is racing with any pending WriteData operation, so it may return the value before of after the write completes.

Note that we have multiple instances of disk-caches, and they all have the same background thread.

Data Integrity

There is a balance to achieve between performance and crash resilience. At one extreme, every unexpected failure will lead to unrecoverable corrupt information and at the other extreme every action has to be flushed to disk before moving on to be able to guarantee the correct ordering of operations. We didn’t want to add the complexity of a journaling system given that the data stored by the cache is not critical by definition, and doing that implies some performance degradation.

The current system relies heavily on the presence of an OS-wide file system cache that provides adequate performance characteristics at the price of losing some deterministic guarantees about when the data actually reaches the disk (we just know that at some point, some part of the OS will actually decide that it is time to write the information to disk, but not if page X will get to disk before page Y).

Some critical parts of the system are directly memory mapped so that, besides providing optimum performance, even if the application crashes the latest state will be flushed to disk by the system. Of course, if the computer crashes we’ll end up on a pretty bad state because we don’t know if some part of the information reached disk or not (each memory page can be in a different state).

The most common problem if the system crashes is that the lists used by the eviction algorithm will be corrupt because some pages will have reached disk while others will effectively be on a “previous” state, still linking to entries that were removed etc. In this case, the corruption will not be detected at start up (although individual dirty entries will be detected and handled correctly), but at some point we’ll find out and proceed to discard the whole cache. It could be possible to start “saving” individual good entries from the cache, but the benefit is probably not worth the increased complexity.

参考：

https://source.chromium.org/chromium/chromium/src/+/master:net/disk_cache/disk_cache.h

disk_cache API

The disk_cache API provides for caches that can store multiple random-access byte streams of data associated with a key on disk (or in memory).

There are two kinds of entries that can be stored: regular and sparse.

Regular entries contain up to 3 separate data streams. Usually stream 0 would be used for some kind of primary small metadata (e.g. HTTP headers); stream 1 would contain the main payload (e.g. HTTP body); and stream 2 would optionally contain some auxiliary metadata that's needed only some of the time (e.g. V8 compilation cache). There is no requirement that these streams be used in this way, but implementations may expect similar size and usage characteristics.

Sparse entries have a stream 0 and a separate sparse stream that's accessed with special methods that have Sparse in their names. It's an API misuse to try to access streams 1 or 2 of sparse entries or to call WriteSparseData on entries that have contents in those streams. Calling SparseReadData or GetAvailableRange to check whether entries are sparse is, however, permitted. An added entry becomes a regular entry once index 1 or 2 is written to, or it becomes a sparse entry once the sparse stream is written to. Once that is done, it cannot change type and the access/modification restrictions relevant to the type apply. Type of an entry can always be determined using SparseReadData or GetAvailableRange.

The sparse streams are named as such because they are permitted to have holes in the byte ranges of contents they represent (and implementations may also drop some pieces independently). For example, in the case of a regular entry, starting with an empty entry, and performing WriteData on some stream at offset = 1024, length = 1024, then another WriteData at offset = 3072, length = 1024, results in the stream having length = 4096, and the areas not written to filled in with zeroes.

In contrast, after the same sequence of WriteSparseData operations, the entry will actually keep track that [1024, 2048) and [3072, 4096) are valid, and will permit queries with GetAvailableRange, and only allow reads of the defined ranges.

net/disk_cache/disk_cache.h is the only include you need if you just want to use this API. disk_cache::CreateCacheBackend() is the first method you'll need to call.

[TOC]

The implementations

disk_cache/blockfile directory

This implementation backs the HTTP cache on Windows and OS X. It tries to pack many small entries caches typically have into "block" files, which can help performance but introduces a lot of complexity and makes recovery from corruption very tricky.

disk_cache/memory directory

This contains the in-memory-only implementation. It's used for incognito mode.

disk_cache/simple directory

This implementation backs the HTTP cache on Android, ChromeOS, and Linux, and is used to implement some features like CacheStorage on all platforms. The design is centered around roughly having a single file per cache entry (more precisely for streams 0 and 1), with a compact and simple in-memory index for membership tests, which makes it very robust against failures, but also highly sensitive to OS file system performance.

posted @ 2020-08-20 17:00 Bigben 阅读(3212) 评论(0) 编辑收藏举报

刷新页面返回顶部