Rails下cloud datastore的使用
Rails下cloud datastore的使用
背景
部门有一个项目要用Ruby做 WebAPI，DB使用关系型数据库Cloud Sql和非关系型数据库Cloud Datastore 。
还不了解Ruby On Rails和CloudDatastore的请参考下面的链接。
https://thinkit.co.jp/story/2015/02/05/5594
1、 Windows下开发环境构建
1.1、构建Ruby On Rails 开发环境
① Ruby安装
http://www.runoob.com/ruby/ruby-installation-windows.html
② Rail安装
从CMD提示窗口输入指令：gem install rails 开始安装rails。
注意：在中国直接安装会提示错误，因为gem默认的安装源https://rubygems.org在国内不能访问，还好先辈们早已为我们新搭建了一个gem安装源http://gems.ruby-china.org。下面是切换安装源的具体步骤，gem安装源切换之后，Rails安装就可以顺利进行了。
第一步：删除默认安装源
CMD提示窗中输入”gem sources -remove https://rubygems.org/”，回车换行。
第二步：添加新的安装源
CMD提示窗输入“gem sources -add http://gems.ruby-china.org/”，回车换行。
③ ruby的IDE Ruby Mine安装
http://qiita.com/HALU5071/items/6d6a39e44865d8d04de8
1.2、搭建开发用Cloud Datastore 数据库
① 安装Google Cloud SDK
http://www.apps-gcp.com/google-cloud-sdk-install
② 安装cloud-datastore-emulator
CMD提示窗中输入“gcloud components install cloud-datastore-emulator”，回车换行。
③ 创建一个开发用的DB:dev_datastore
CMD提示窗中输入“cloud_datastore_emulator create db/dev_datastore”,回车换行。
前提条件：CMD当前运行目录下要先创建一个db的文件夹。
创建好的DB文件结构图如下：
2、 WEB API工程创建
在CMD提示窗中输入“rails new my-first-api --api”，回车换行。
创建好的API工程结构见下图，rails执行结果见附录1。
|
フォルダ構造 |
||
|
|
|
説明 |
|
Gemfile |
|
gemの依存関係を指定できるファイル |
|
README.rdoc |
|
説明書 |
|
Rakefile |
ターミナルから実行可能なタスク |
|
|
config.ru |
|
Rackの設定 |
|
app/ |
|
アプリケーションを格納するディレクトリ |
|
主要なプログラムはこの配下に格納 |
||
|
|
controllers/ |
コントローラを格納するディレクトリ |
|
|
controllers/application_controller.rb |
アプリケーションで共通のコントローラ |
|
|
models/ |
モデルを格納するディレクトリ |
|
config/ |
|
プロジェクトの設定ファイルを格納するディレクトリ |
|
config/environments/ |
|
環境単位の設定ファイルを格納するディレクトリ |
|
config/initializers/ |
|
初期化ファイルを格納するディレクトリ |
|
config/locales/ |
|
辞書ファイルを格納するディレクトリ |
|
db/ |
|
データベースの設定ファイルを格納するディレクトリ |
|
doc/ |
|
ドキュメントを格納するディレクトリ |
|
lib/ |
|
複数のアプリケーション間で共有するライブラリを格納するディレクトリ |
|
|
tasks/ |
自分で生成したRakefileを格納するディレクトリ |
|
log/ |
|
ログファイルが格納されるディレクトリ。ログファイルはアプリケーションと環境ごとに作成される |
|
public/ |
|
Web上に公開するファイルを格納するディレクトリ |
|
tmp/ |
|
キャッシュなど、一時的なファイルを格納されるディレクトリ |
|
test/ |
|
アプリケーションのテストに使うファイルを格納するディレクトリ |
|
vendor/ |
|
ライブラリや他のアプリケーションで共有するような外部ライブラリを格納するディレクトリ |
在附录1的最后我们看到，bundle install 并没有成功。原因是项目的gem安装源出了问题。打开Gemfile把第一行的source 'https://rubygems.org'替换成source 'http://gems.ruby-china.org'。然后在CMD里重新执行bundle install命令，这样项目需要的gem就能成功安装。
3、 新建cloud Datastore 的dataset对象
① Gem安装
在Gemfile 中追加
gem 'google-cloud-datastore'
cmd里重新执行bundle install命令。
② 配置文件
在config/initializers/目录下新建cloud_datastore.rb文件，文件内容如下图所示。
从图中我们可以看到cloud datastore 配置了三种环境下创建daset的参数，这些参数我们都放在了config/database.yml里。下面是database.yml里开发环境的配置信息，datastore 安装的主机和datastore的数据库名，这样我们就可以使用datastore了。
development:
host: 'localhost:8180'
project: ' dev_datastore '
cloud_datastore.rb
4、 非关系型数据库 中ActiveModel 的使用
如果我们想要cloud datastore和关系型数据库一样，可以方便快捷使用model（ActiveModel），怎么办？下面就是你想要的答案。
① 追加文件
在config/initializers/目录下新建active_model_cloud_datastore.rb文件。
文件内容参照附录2。
② model Class写法
require_relative "../../config/initializers/active_model_cloud_datastore"
class Customer
include ActiveModelCloudDatastore
attr_accessor :customer_code, :customer_name
def attributes
%w(code customer_name)# 注意多个字段之间是半角空格来区分的
end
end
这样我们就可以使用activeModel很多功能，
#检索所有数据
@customers = Customer.all
#保存一条新的数据
@customer = Customer.new
@customer.save
这里就不做详细说明，具体参照附录2里的各种方法。
5、 spec测试
- 1.
- 2.
- 3.
- 4.
- 5.
① 测试环境安装
在Gemfile中追加下面的gem。
group :development, :test do
# Test
gem 'rspec-rails', '~> 3.0'
gem 'rails-controller-testing'
end
然后在cmd工程目录my-first-api下执行bundle install，安装追加的gem。
② 测试环境初期化
在cmd工程目录my-first-api下执行
bundle exec rails generate rspec:install
会生成下面的文件
create .rspec
create spec
create spec/spec_helper.rb
create spec/rails_helper.rb
③ controller测试
参考下面链接
http://qiita.com/shizuma/items/84e07e558abd6593df15
http://blog.naichilab.com/entry/2016/01/19/011514
④ model测试
参考下面的测试思路。
http://qiita.com/shizuma/items/c7b8d7b91e8f325f8ad9
⑤ Mock使用
参考下面链接
http://qiita.com/jnchito/items/640f17e124ab263a54dd
6、 测试覆盖率报告
① 环境搭建
在Gemfile中追加下面的gem。
group :development, :test do
gem 'simplecov'
end
然后在cmd工程目录my-first-api下执行bundle install，安装追加的gem。
② spec_helper.rb修改
在spec_helper.rb头部插入下面的语句，覆盖率测试中要过滤掉spec下的测试代码。
require 'simplecov'
SimpleCov.start do
add_filter "/spec/ "
end
③ 查看报告
在cmd工程目录my-first-api下，再次执行rspec spec命令，覆盖率报告就会自动生成再demo/coverage下，用google chrome浏览器打开index.html，就可以看到详细的信息。下面是一个覆盖率报告的截图。
7、 结束语
上面我们讲述的是Ruby下怎么使用cloud datastore的开发和测试，在Google Cloud Platform上怎么部署产品还有待下一步探索。期间遇到的各种技术问题难题，为了解决这些问题，调查的网站以日语和英语为主，总结的时候也使用了很多日语网站，由于时间有限，没能一一翻译过来，给不懂日语的朋友带来不少困难表示歉意。
附录1
API工程创建文件list
create
create README.md
create Rakefile
create config.ru
create .gitignore
create Gemfile
create app
create app/assets/config/manifest.js
create app/assets/javascripts/application.js
create app/assets/javascripts/cable.js
create app/assets/stylesheets/application.css
create app/channels/application_cable/channel.rb
create app/channels/application_cable/connection.rb
create app/controllers/application_controller.rb
create app/helpers/application_helper.rb
create app/jobs/application_job.rb
create app/mailers/application_mailer.rb
create app/models/application_record.rb
create app/views/layouts/application.html.erb
create app/views/layouts/mailer.html.erb
create app/views/layouts/mailer.text.erb
create app/assets/images/.keep
create app/assets/javascripts/channels
create app/assets/javascripts/channels/.keep
create app/controllers/concerns/.keep
create app/models/concerns/.keep
create bin
create bin/bundle
create bin/rails
create bin/rake
create bin/setup
create bin/update
create config
create config/routes.rb
create config/application.rb
create config/environment.rb
create config/secrets.yml
create config/cable.yml
create config/puma.rb
create config/environments
create config/environments/development.rb
create config/environments/production.rb
create config/environments/test.rb
create config/initializers
create config/initializers/application_controller_renderer.rb
create config/initializers/assets.rb
create config/initializers/backtrace_silencers.rb
create config/initializers/cookies_serializer.rb
create config/initializers/cors.rb
create config/initializers/filter_parameter_logging.rb
create config/initializers/inflections.rb
create config/initializers/mime_types.rb
create config/initializers/new_framework_defaults.rb
create config/initializers/session_store.rb
create config/initializers/wrap_parameters.rb
create config/locales
create config/locales/en.yml
create config/boot.rb
create config/database.yml
create db
create db/seeds.rb
create lib
create lib/tasks
create lib/tasks/.keep
create lib/assets
create lib/assets/.keep
create log
create log/.keep
create public
create public/404.html
create public/422.html
create public/500.html
create public/apple-touch-icon-precomposed.png
create public/apple-touch-icon.png
create public/favicon.ico
create public/robots.txt
create test/fixtures
create test/fixtures/.keep
create test/fixtures/files
create test/fixtures/files/.keep
create test/controllers
create test/controllers/.keep
create test/mailers
create test/mailers/.keep
create test/models
create test/models/.keep
create test/helpers
create test/helpers/.keep
create test/integration
create test/integration/.keep
create test/test_helper.rb
create tmp
create tmp/.keep
create tmp/cache
create tmp/cache/assets
create vendor/assets/stylesheets
create vendor/assets/stylesheets/.keep
remove app/assets
remove lib/assets
remove tmp/cache/assets
remove vendor/assets
remove app/helpers
remove test/helpers
remove app/views/layouts/application.html.erb
remove public/404.html
remove public/422.html
remove public/500.html
remove public/apple-touch-icon-precomposed.png
remove public/apple-touch-icon.png
remove public/favicon.ico
remove app/assets/javascripts
remove config/initializers/assets.rb
remove config/initializers/session_store.rb
remove config/initializers/cookies_serializer.rb
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies...
Using rake 12.0.0
Using i18n 0.7.0
Using minitest 5.10.1
Using thread_safe 0.3.5
Using builder 3.2.2
Using erubis 2.7.0
Using mini_portile2 2.1.0
Using rack 2.0.1
Using nio4r 1.2.1
Using websocket-extensions 0.1.2
Using mime-types-data 3.2016.0521
Using arel 7.1.4
Using bundler 1.13.6
Using method_source 0.8.2
Using puma 3.6.2
Using thor 0.19.4
Using sqlite3 1.3.12
Gem::RemoteFetcher::FetchError: SSL_connect returned=1 errno=0 state=SSLv3 read
server certificate B: certificate verify failed
(https://rubygems.org/gems/concurrent-ruby-1.0.4.gem)
An error occurred while installing concurrent-ruby (1.0.4), and Bundler cannot
continue.
Make sure that `gem install concurrent-ruby -v '1.0.4'` succeeds before
bundling.
附录2
active_model_cloud_datastore.rb文件内容：
# frozen_string_literal: true
require_relative 'cloud_datastore'
require 'active_model'
require 'active_support'
# Integrates ActiveModel with the Google::Cloud::Datastore
module ActiveModelCloudDatastore
extend ActiveSupport::Concern
include ActiveModel::Model
include ActiveModel::Dirty
include ActiveModel::Validations
include ActiveModel::Validations::Callbacks
included do
private_class_method :query_options, :query_sort, :query_property_filter
define_model_callbacks :save, :update, :destroy
attr_accessor :id
end
def attributes
[]
end
# Used by ActiveModel for determining polymorphic routing.
def persisted?
id.present?
end
# Resets the ActiveModel::Dirty tracked changes
def reload!
clear_changes_information
end
# Updates attribute values on the ActiveModel::Model object with the provided params.
# Example, such as submitted form params.
#
# @param [Hash] params
def update_model_attributes(params)
params.each do |name, value|
send "#{name}=", value if respond_to? "#{name}="
end
end
# Builds the Cloud Datastore entity with attributes from the Model object.
#
# @return [Entity] the updated Google::Cloud::Datastore::Entity
def build_entity(parent = nil)
entity = CloudDatastore.dataset.entity(self.class.name, id)
entity.key.parent = parent if parent
attributes.each do |attr|
entity[attr] = instance_variable_get("@#{attr}")
end
entity
end
def save(parent = nil)
run_callbacks :save do
if valid?
entity = build_entity(parent)
success = self.class.retry_on_exception? { CloudDatastore.dataset.save(entity) }
if success
self.id = entity.key.id
return true
end
end
false
end
end
def update(params)
run_callbacks :update do
update_model_attributes(params)
if valid?
entity = build_entity
self.class.retry_on_exception? { CloudDatastore.dataset.save(entity) }
else
false
end
end
end
def destroy
run_callbacks :destroy do
key = CloudDatastore.dataset.key(self.class.name, id)
self.class.retry_on_exception? { CloudDatastore.dataset.delete(key) }
end
end
# Methods defined here will be class methods whenever we 'include DatastoreUtils'.
module ClassMethods
# Queries all objects from Cloud Datastore by named kind and using the provided options.
#
# @param [Hash] options the options to construct the query with.
#
# @option options [Google::Cloud::Datastore::Key] :ancestor filter for inherited results
# @option options [Hash] :where filter, Array in the format [name, operator, value]
#
# @return [Array<Model>] an array of ActiveModel results.
def all(options = {})
query = CloudDatastore.dataset.query(name)
query.ancestor(options[:ancestor]) if options[:ancestor]
query_property_filter(query, options)
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
from_entities(entities.flatten)
end
# Queries objects from Cloud Datastore in batches by named kind and using the provided options.
# When a limit option is provided queries up to the limit and returns results with a cursor.
#
# @param [Hash] options the options to construct the query with. See build_query for options.
#
# @return [Array<Model>, String] an array of ActiveModel results and a cursor that can be used
# to query for additional results.
def find_in_batches(options = {})
next_cursor = nil
query = build_query(options)
if options[:limit]
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
next_cursor = entities.cursor if entities.size == options[:limit]
else
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
end
model_entities = from_entities(entities.flatten)
return model_entities, next_cursor
end
# Retrieves an entity by key and by an optional parent.
#
# @param [Integer or String] id_or_name id or name value of the entity Key.
# @param [Google::Cloud::Datastore::Key] parent the parent Key of the entity.
#
# @return [Entity, nil] a Google::Cloud::Datastore::Entity object or nil.
def find_entity(id_or_name, parent = nil)
key = CloudDatastore.dataset.key(name, id_or_name)
key.parent = parent if parent
retry_on_exception { CloudDatastore.dataset.find(key) }
end
# Find object by ID.
#
# @return [Model, nil] an ActiveModel object or nil.
def find(id)
entity = find_entity(id.to_i)
from_entity(entity)
end
# Find object by parent and ID.
#
# @return [Model, nil] an ActiveModel object or nil.
def find_by_parent(id, parent)
entity = find_entity(id.to_i, parent)
from_entity(entity)
end
def from_entities(entities)
entities.map { |entity| from_entity(entity) }
end
# Translates between Google::Cloud::Datastore::Entity objects and ActiveModel::Model objects.
#
# @param [Entity] entity from Cloud Datastore
# @return [Model] the translated ActiveModel object.
def from_entity(entity)
return if entity.nil?
model_entity = new
model_entity.id = entity.key.id unless entity.key.id.nil?
model_entity.id = entity.key.name unless entity.key.name.nil?
entity.properties.to_hash.each do |name, value|
model_entity.send "#{name}=", value
end
model_entity.reload!
model_entity
end
def exclude_from_index(entity, boolean)
entity.properties.to_h.keys.each do |value|
entity.exclude_from_indexes! value, boolean
end
end
# Constructs a Google::Cloud::Datastore::Query.
#
# @param [Hash] options the options to construct the query with.
#
# @option options [Google::Cloud::Datastore::Key] :ancestor filter for inherited results
# @option options [String] :cursor sets the cursor to start the results at
# @option options [Integer] :limit sets a limit to the number of results to be returned
# @option options [String] :order sort the results by property name
# @option options [String] :desc_order sort the results by descending property name
# @option options [Array] :select retrieve only select properties from the matched entities
# @option options [Hash] :where filter, Array in the format [name, operator, value]
#
# @return [Query] a datastore query.
def build_query(options = {})
query = CloudDatastore.dataset.query(name)
query_options(query, options)
end
def retry_on_exception?
retry_count = 0
sleep_time = 0.5 # 0.5, 1, 2, 4 second between retries
begin
yield
rescue => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
puts 'Rescued exception, retrying...'
sleep sleep_time
sleep_time *= 2
retry_count += 1
return false if retry_count > 3
retry
end
true
end
def retry_on_exception
retry_count = 0
sleep_time = 0.5 # 0.5, 1, 2, 4 second between retries
begin
yield
rescue => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
puts 'Rescued exception, retrying...'
sleep sleep_time
sleep_time *= 2
retry_count += 1
raise e if retry_count > 3
retry
end
end
def log_google_cloud_error
yield
rescue Google::Cloud::Error => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
raise e
end
# private
def query_options(query, options)
query.ancestor(options[:ancestor]) if options[:ancestor]
query.cursor(options[:cursor]) if options[:cursor]
query.limit(options[:limit]) if options[:limit]
query_sort(query, options)
query.select(options[:select]) if options[:select]
query_property_filter(query, options)
end
# Adds sorting to the results by a property name if included in the options.
def query_sort(query, options)
query.order(options[:order]) if options[:order]
query.order(options[:desc_order], :desc) if options[:desc_order]
query
end
# Adds property filters to the query if included in the options.
# Accepts individual or nested Arrays:
# [['superseded', '=', false], ['email', '=', 'something']]
def query_property_filter(query, options)
if options[:where]
opts = options[:where]
if opts[0].is_a?(Array)
opts.each do |opt|
query.where(opt[0], opt[1], opt[2]) unless opt.nil?
end
else
query.where(opts[0], opts[1], opts[2])
end
end
query
end
end
end
