Apache HBase HBCK2 Tool

HBCK2是hbase-1.x修复工具（也称作hbck1）的继承者。使用它代替hbck1进行hbase-2.x安装修复。

hbck1

不应该对hbase-2.x群集运行hbase-1.x附带的hbck工具（ hbck1）。它可能会造成损坏。虽然hbck1仍然捆绑在hbase-2.x--中以最大限度地减少意外（它在其输出的帮助有一个指向HBCK2的提示） - 它的写入工具（-fix）已被删除。它可以报告hbase-2.x群集的状态，但其评估可能不准确，因为它不了解hbase-2.x的内部工作原理。

HBCK2比hbck1做的少得多，因为hbck1解决的许多问题都不再是hbase-2.x中的问题，或者我们已经（或将会制作）专用工具来执行hbck1所使用的内容。 HBCK2的工作方式与hbck1的操作方式不同，要求HBase Master响应请求，而不是在hbck1工具中复制Master以外的功能。

Building HBCK2

Run:

mvn install

The built HBCK2 fat jar will be in the target sub-directory.

Running HBCK2

org.apache.hbase.HBCK2 是HBCK2的类名. 运行如下命令可以查看 HBCK2 用法:

 $ HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2

usage: HBCK2 [OPTIONS] COMMAND <ARGS>

Options:-d,--debug                                 run with debug output-h,--help                                  output this help message-p,--hbase.zookeeper.property.clientPort   port of target hbase ensemble-q,--hbase.zookeeper.quorum <arg>          ensemble of target hbase-v,--version                               this hbck2 version-z,--zookeeper.znode.parent                parent znode of target hbase

Commands:assigns [OPTIONS] <ENCODED_REGIONNAME>...Options:-o,--override  override ownership by another procedureA 'raw' assign that can be used even during Master initialization.Skirts Coprocessors. Pass one or more encoded RegionNames.1588230740 is the hard-coded name for the hbase:meta region andde00010733901a05f5a2a3a382e27dd4 is an example of what a user-spaceencoded Region name looks like. For example:$ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
bypass [OPTIONS] <PID>...Options:-o,--override   override if procedure is running/stuck-r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!-w,--lockWait   milliseconds to wait on lock before giving up;
default=1Pass one (or more) procedure 'pid's to skip to procedure finish.Parent of bypassed procedure will also be skipped to the finish.Entities will be left in an inconsistent state and will requiremanual fixup. May need Master restart to clear locks still held.Bypass fails if procedure has children. Add 'recursive' if allyou have is a parent pid to finish parent and children. Thisis SLOW, and dangerous so use selectively. Does not always work.
unassigns <ENCODED_REGIONNAME>...Options:-o,--override  override ownership by another procedureA 'raw' unassign that can be used even during Master initialization.Skirts Coprocessors. Pass one or more encoded RegionNames:1588230740 is the hard-coded name for the hbase:meta region andde00010733901a05f5a2a3a382e27dd4 is an example of what a user-spaceencoded Region name looks like. For example:$ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4Returns the pid(s) of the created UnassignProcedure(s) or -1 if none.
setTableState <TABLENAME> <STATE>Possible table states: ENABLED, DISABLED, DISABLING, ENABLINGTo read current table state, in the hbase shell run:hbase> get 'hbase:meta', '<TABLENAME>', 'table:state'A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc.An example making table name 'user' ENABLED:$ HBCK2 setTableState users ENABLEDReturns whatever the previous table state was.

HBCK2 Overview

HBCK2 is currently a simple tool that does one thing at a time only.

HBCK2 does not do diagnosis, leaving that function to other tooling, described below.

HBCK2目前是一个简单的工具，一次只做一件事。

HBCK2不做诊断，将该功能留给其他工具，如下面所描述的。

In hbase-2.x, the Master is the final arbiter of all state, so a general principal of HBCK2 is that it asks the Master to effect all repair. This means a Master must be up before you can run an HBCK2 command.

在hbase-2.x中，Master是所有状态的最终仲裁者，因此HBCK2的一般原则是它要求Master执行所有修复。这意味着在运行HBCK2命令之前必须启动Master。

HBCK2 works by making use of an intentionally obscured HbckService hosted on the Master. The Service publishes a few methods for the HBCK2 tool to pull on. The first thing HBCK2 does is poke the cluster to ensure the service is available. It will fail if it is not or if the HbckService is lacking a wanted facility. HBCK2 versions should be able to work across multiple hbase-2 releases. It will fail with a complaint if it is unable to run. There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions.

HBCK2通过使用托管在Master上的HbckService来工作。该服务发布了一些HBCK2工具的方法。 HBCK2做的第一件事就是确保集群的服务可用。如果不可用或者HbckService缺少想要的设施，它将会失败.HBCK2版本应该能够在多个hbase-2版本中工作。如果无法运行，它将提示失败。在2.0.3和2.1.1之前的hbase版本中没有HbckService。 HBCK2不适用于这些版本。

发现问题

hbck1分析并报告你集群情况是好是坏，hbck2就没那么自以为是了。在hbase-2.x中，操作人确定需要修复的内容，然后使用包括hbck2在内的工具进行修复。

要发现分配中的问题，请使用以下工具

诊断工具

Master Logs

The Master runs all assignments, server crash handling, cluster start and stop, etc. In hbase-2.x, all that the Master does has been cast as Procedures run on a state machine engine. See Procedure Framework and Assignment Manager

for detail on how this new infrastructure works.

Master执行所有分配，服务器崩溃处理，集群启动和停止等。在hbase-2.x中，Master执行的所有操作显示为过程，运行在状态机引擎上。

Each Procedure has a unique Procedure id, its pid, that it lists on each logging. Following the pid, you can trace the lifecycle of a Procedure in the Master logs as Procedures transition from start, through each of the Procedure's various stages to finish. Some Procedures spawn sub-procedures, wait on their Children, and then themselves finish. Each child logs its pid but also its ppid; its parent's pid.

每个过程都有一个唯一的过程ID，它的pid，它在每次日志记录中列出。你可以通过pid跟踪Master日志中过程的生命周期，从开始、各个阶段、直至完成。一些过程产生子过程，等待他们的子进程完成，然后自身才完成。每个孩子都记录了自己的pid，还记录了ppid: 它的父母的pid。

Generally all runs problem free but if some unforeseen circumstance arises, the assignment framework may sustain damage requiring operator intervention. Below we will discuss some such scenarios but they can manifest in the Master log as a Region being STUCK or a Procedure transitioning an entity -- a Region or a Table -- may be blocked because another Procedure holds the exclusive lock and is not letting go.

通常所有运行都没有问题，但是如果出现一些不可预见的情况，分配体系可能会发生需要操作员干预的损坏。下面我们将讨论一些这样的场景，在Master日志中显示为一个region是STUCK或一个过程转换region或表状态被阻止，因为另一个过程持有独占锁并且不放手。

STUCK Procedures look like this:

被阻塞的procedure如下：

2018-09-12 15:29:06,558 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=va1001.example.org,22101,1536173230599, table=IntegrationTestBigLinkedList_20180626110336, region=dbdb56242f17610c46ea044f7a42895b

/master-status#tables

This section about midway down in Master UI home-page shows a list of tables with columns for whether the table is ENABLED, ENABLING, DISABLING, or DISABLED among other attributes. Also listed are columns with counts of Regions in their various transition states: OPEN, CLOSED, etc. A read of this table is good for figuring if the Regions of this table have a proper disposition. For example if a table is ENABLED and there are Regions that are not in the OPEN state and the Master Log is silent about any ongoing assigns, then something is amiss.

Master UI主页中间显示了一个table列表，其中包含表是否为ENABLED，ENABLING，DISABLING或DISABLED等列。还列出了具有各种过渡状态（如OPEN，CLOSED）的region个数。读取此表有助于确定此表的region是否处于适当的状态。例如，如果表是ENABLED并且存在未处于OPEN状态的region，而Master Log 对任何正在进行的分配毫无反应，那么肯定存在问题。

Procedures & Locks

This page off the Master UI home page under the Procedures & Locks menu item in the page heading lists all ongoing Procedures and Locks as well as the current set of Master Procedure WALs (named pv2-0000000000000000###.log under the MasterProcWALs directory in your hbase install). On startup, on a large cluster when furious assigning is afoot, this page is filled with lists of Procedures and Locks. The count of MasterProcWALs will bloat too. If after the cluster settles, there is a stuck Lock or Procedure or the count of WALs doesn't ever come down but only grows, then operator intervention is needed to alieve the blockage.

在Master UI主页的“Procedures & Locks ”菜单下，列出了所有正在进行的过程和锁以及当前的主过程WAL集合（在MasterProcWALs目录下命名为pv2-0000000000000000 ##log）。在启动时，在大型集群上进行激烈分配时，此页面将填充过程和锁定列表，MasterProcWALs数量也会膨胀。如果在群集稳定后，存在卡住的锁或程序，或者WAL的计数不会降低但只会增长（WAL用完会移到oldWALs），则需要操作员干预来解决堵塞问题。

Lists of locks and procedures can also be obtained via the hbase shell:

$ echo "list_locks"| hbase shell &> /tmp/locks.txt
$ echo "list_procedures"| hbase shell &> /tmp/procedures.txt

The HBase Canary Tool

The Canary tool is useful verifying the state of assign. It can be run with a table focus or against the whole cluster.

The Canary tool 用来探查region的assign情况。

For example, to check cluster assigns:

$ hbase canary -f false -t 6000000 &>/tmp/canary.log

The -f false tells the Canary to keep going across failed Region fetches and the -t 6000000 tells the Canary run for ~two hours maximum. When done, check out /tmp/canary.log. Grep for ERROR lines to find problematic Region assigns.

You can do a probe like the Canary's in the hbase shell. For example, given a Region that has a start row of d1dddd0c belonging to the table testtable, do as follows:

实际上的原理是scan一个表中每个region的startrow起的十行。

hbase> scan 'testtable', {STARTROW => 'd1dddd0c', LIMIT => 10}

For an overview on parsing a Region name into its constituent parts, see RegionInfo API.

Other Tools

To figure the list of Regions that are not OPEN on an ENABLED or ENABLING table, read the hbase:meta table info:state column. For example, to find the state of all Regions in the table IntegrationTestBigLinkedList_20180626064758, do the following:

为了了解在一个enable的table里那些region没打开，可以执行如下命令：

$ echo " scan 'hbase:meta', {ROWPREFIXFILTER => 'IntegrationTestBigLinkedList_20180626064758,', COLUMN => 'info:state'}"| hbase shell > /tmp/t.txt

...then grep for OPENING or CLOSING Regions.

去获取OPENING或者CLOSING的region。

To move an OPENING issue to OPEN so it agrees with a table's ENABLED state, use the assign command in the hbase shell to queue a new Assign Procedure (watch the Master logs to see the Assign run). If many Regions to assign, use the HBCK2 tool. It can do bulk assigning.

为了让OPENING变为OPEN，可以在hbaseshell中使用assign命令，或者使用HBCK2批量分配。

修复

General principals include a Region can not be assigned if it is in CLOSING state (or the inverse, unassigned if in OPENING state) without first transitioning via CLOSED: Regions must always move from CLOSED, to OPENING, to OPEN, and then to CLOSING, CLOSED.

一般原则包括如果region处于CLOSING状态而尚未通过CLOSED，则不能assign。或者不能被uassign如果处于OPENING状态。region必须始终按照CLOSED，OPENING，OPEN，然后再到CLOSING， CLOSED的顺序变换状态。

When making repair, do fixup a table at a time.

Also, if a table is DISABLED, you cannot assign a Region. In the Master logs, you will see that the Master will report that the assign has been skipped because the table is DISABLED. You may want to assign a Region because it is currently in the OPENING state and you want it in the CLOSED state so it agrees with the table's DISABLED state. In this situation, you may have to temporarily set the table status to ENABLED, just so you can do the assign, and then set it back again after the unassign. HBCK2 has facility to allow you do this. See the HBCK2 usage output.

此外，如果表为DISABLED，则无法分配Region。在Master日志中，您将看到Master将报告已跳过assign，因为该表为DISABLED。您可能希望分配一个Region，因为它当前处于OPENING状态，并且您希望它处于CLOSED状态，因此它与表的DISABLED状态一致。在这种情况下，您可能必须暂时将表状态设置为ENABLED，以便您可以执行分配，然后在取消分配后再次将其设置.HBCK2具有允许您执行此操作的功能。请参阅HBCK2使用输出。

Start-over

At an extreme, if the Master is distraught and all attempts at fixup only turn up undoable locks or Procedures that won't finish, and/or the set of MasterProcWALs is growing without bound, it is possible to wipe the Master state clean. Just move aside the /hbase/MasterProcWALs/ directory under your hbase install and restart the Master process. It will come back as a tabula rasa without memory of the bad times past.

在极端情况下，如果master一片混乱并且所有修复尝试锁或过程都无效，或/MasterProcWAL集合无限制地增长，则可以清理master状态。只需将/ hbase / MasterProcWALs /目录移开，然后重新启动主进程。它将作为一个白板而回归，而不会记住过去的糟糕时期。

If at the time of the erasure, all Regions were happily assigned or offlined, then on Master restart, the Master should pick up and continue as though nothing happened. But if there were Regions-In-Transition at the time, then the operator may have to intervene to bring outstanding assigns/unassigns to their terminal point. Read the hbase:meta info:state columns as described above to figure what needs assigning/unassigning. Having erased all history moving aside the MasterProcWALs, none of the entities should be locked so you are free to bulk assign/unassign.

如果在清除时，所有region都被愉快地分配或离线，然后master重新启动，master应该接收并继续，好像什么也没发生。但是如果当时存在regions-in-transition，那么操作员可能不得不进行干预以将未完成的分配/取消分配。阅读如上所述的hbase：meta info：state列，以确定需要分配/取消分配的内容。如果已MasterProcWAL的所有历史记录，不应有锁定的任何实体，以便您可以自由批量分配/取消分配。

Assigning/Unassigning

Generally, on assign, the Master will persist until successful. An assign takes an exclusive lock on the Region. This precludes a concurrent assign or unassign from running. An assign against a locked Region will wait until the lock is released before making progress. See the [Procedures & Locks] section above for current list of outstanding Locks.

通常，在分配时，主人将持续到成功为止。 assign对区域进行独占锁定。这排除了并发分配或取消分配运行。针对锁定区域的分配将等到锁定被释放后再进行。有关未完成锁定的当前列表，请参阅上面的[过程和锁定]部分。

Master startup cannot progress, in holding-pattern until region onlined

meta表没上线

This should never happen. If it does, here is what it looks like:

2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.example.org,22101,1538449648131}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.

The Master is unable to continue startup because there is no Procedure to assign hbase:meta (or hbase:namespace). To inject one, use the HBCK2 tool:

HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 assigns 1588230740

...where 1588230740 is the encoded name of the hbase:meta Region.

The same may happen to the hbase:namespace system table. Look for the encoded Region name of the hbase:namespace Region and do similar to what we did for hbase:meta.