linux-toradex.git/block/blk-mq-sysfs.c, branch v4.9.6

blk-mq: register device instead of disk

2016-09-21T13:56:16+00:00

Enable devices without a gendisk instance to register itself with blk-mq
and expose the associated multi-queue sysfs entries.

Signed-off-by: Matias Bjørling 
Signed-off-by: Jens Axboe

blk-mq: account higher order dispatch

2016-09-16T20:03:04+00:00

We currently account a '0' dispatch, and anything above that still falls
below the range set by BLK_MQ_MAX_DISPATCH_ORDER. If we dispatch more,
we don't account it.

Change the last bucket to be inclusive of anything above the range we
track, and have the sysfs file reflect that by including a '+' in the
output:

$ cat /sys/block/nvme0n1/mq/0/dispatched
        0	1006
        1	20229
        2	1
        4	0
        8	0
       16	0
       32+	0

Signed-off-by: Jens Axboe 
Reviewed-by: Omar Sandoval

block: enable zeroing of io_poll statistics

2016-09-14T14:41:23+00:00

Allow the io_poll statistics to be zeroed to make for easier logging
of polling event.

Signed-off-by: Stephen Bates 
Acked-by: Christoph Hellwig 
Signed-off-by: Jens Axboe

block: add poll_considered statistic

2016-09-14T14:41:21+00:00

In order to help determine the effectiveness of polling in a running
system it is usful to determine the ratio of how often the poll
function is called vs how often the completion is checked. For this
reason we add a poll_considered variable and add it to the sysfs entry
for io_poll.

Signed-off-by: Stephen Bates 
Acked-by: Christoph Hellwig 
Signed-off-by: Jens Axboe

blk-mq: fix deadlock in blk_mq_register_disk() error path

2016-08-04T20:19:16+00:00

If we fail registering any of the hardware queues, we call
into blk_mq_unregister_disk() with the hotplug mutex already
held. Since blk_mq_unregister_disk() attempts to acquire the
same mutex, we end up in a less than happy place.

Reported-by: Jinpu Wang 
Signed-off-by: Jens Axboe

blk-mq: Use proper cpumask iterator

2016-03-20T15:34:02+00:00

queue_for_each_ctx() iterates over per_cpu variables under the assumption that
the possible cpu mask cannot have holes. That's wrong as all cpumasks can have
holes. In case there are holes the iteration ends up accessing uninitialized
memory and crashing as a result.

Replace the macro by a proper for_each_possible_cpu() loop and drop the unused
macro blk_ctx_sum() which references queue_for_each_ctx().

Reported-by: Xiong Zhou 
Signed-off-by: Thomas Gleixner 
Signed-off-by: Jens Axboe

blk-mq: dynamic h/w context count

2016-02-09T19:42:17+00:00

The hardware's provided queue count may change at runtime with resource
provisioning. This patch allows a block driver to alter the number of
h/w queues available when its resource count changes.

The main part is a new blk-mq API to request a new number of h/w queues
for a given live tag set. The new API freezes all queues using that set,
then adjusts the allocated count prior to remapping these to CPUs.

The bulk of the rest just shifts where h/w contexts and all their
artifacts are allocated and freed.

The number of max h/w contexts is capped to the number of possible cpus
since there is no use for more than that. As such, all pre-allocated
memory for pointers need to account for the max possible rather than
the initial number of queues.

A side effect of this is that the blk-mq will proceed successfully as
long as it can allocate at least one h/w context. Previously it would
fail request queue initialization if less than the requested number
was allocated.

Signed-off-by: Keith Busch 
Reviewed-by: Christoph Hellwig 
Tested-by: Jon Derrick 
Signed-off-by: Jens Axboe

block: add block polling support

2015-11-07T17:40:47+00:00

Add basic support for polling for specific IO to complete. This uses
the cookie that blk-mq passes back, which enables the block layer
to pass this cookie to the driver to spin for a specific request.

This will be combined with request latency tracking, so we can make
qualified decisions about when to poll and when not to. For now, for
benchmark purposes, we add a sysfs file that controls whether polling
is enabled or not.

Signed-off-by: Jens Axboe 
Acked-by: Christoph Hellwig 
Acked-by: Keith Busch

block: generic request_queue reference counting

2015-10-21T20:43:41+00:00

Allow pmem, and other synchronous/bio-based block drivers, to fallback
on a per-cpu reference count managed by the core for tracking queue
live/dead state.

The existing per-cpu reference count for the blk_mq case is promoted to
be used in all block i/o scenarios.  This involves initializing it by
default, waiting for it to drop to zero at exit, and holding a live
reference over the invocation of q->make_request_fn() in
generic_make_request().  The blk_mq code continues to take its own
reference per blk_mq request and retains the ability to freeze the
queue, but the check that the queue is frozen is moved to
generic_make_request().

This fixes crash signatures like the following:

 BUG: unable to handle kernel paging request at ffff880140000000
 [..]
 Call Trace:
  [] ? copy_user_handle_tail+0x5f/0x70
  [] pmem_do_bvec.isra.11+0x70/0xf0 [nd_pmem]
  [] pmem_make_request+0xd1/0x200 [nd_pmem]
  [] ? mempool_alloc+0x72/0x1a0
  [] generic_make_request+0xd6/0x110
  [] submit_bio+0x76/0x170
  [] submit_bh_wbc+0x12f/0x160
  [] submit_bh+0x12/0x20
  [] jbd2_write_superblock+0x8d/0x170
  [] jbd2_mark_journal_empty+0x5d/0x90
  [] jbd2_journal_destroy+0x24b/0x270
  [] ? put_pwq_unlocked+0x2a/0x30
  [] ? destroy_workqueue+0x225/0x250
  [] ext4_put_super+0x64/0x360
  [] generic_shutdown_super+0x6a/0xf0

Cc: Jens Axboe 
Cc: Keith Busch 
Cc: Ross Zwisler 
Suggested-by: Christoph Hellwig 
Reviewed-by: Christoph Hellwig 
Tested-by: Ross Zwisler 
Signed-off-by: Dan Williams 
Signed-off-by: Jens Axboe

blk-mq: fix deadlock when reading cpu_list

2015-09-29T17:32:51+00:00

CPU hotplug handling for blk-mq (blk_mq_queue_reinit) acquires
all_q_mutex in blk_mq_queue_reinit_notify() and then removes sysfs
entries by blk_mq_sysfs_unregister().  Removing sysfs entry needs to
be blocked until the active reference of the kernfs_node to be zero.

On the other hand, reading blk_mq_hw_sysfs_cpu sysfs entry (e.g.
/sys/block/nullb0/mq/0/cpu_list) acquires all_q_mutex in
blk_mq_hw_sysfs_cpus_show().

If these happen at the same time, a deadlock can happen.  Because one
can wait for the active reference to be zero with holding all_q_mutex,
and the other tries to acquire all_q_mutex with holding the active
reference.

The reason that all_q_mutex is acquired in blk_mq_hw_sysfs_cpus_show()
is to avoid reading an imcomplete hctx->cpumask.  Since reading sysfs
entry for blk-mq needs to acquire q->sysfs_lock, we can avoid deadlock
and reading an imcomplete hctx->cpumask by protecting q->sysfs_lock
while hctx->cpumask is being updated.

Signed-off-by: Akinobu Mita 
Reviewed-by: Ming Lei 
Cc: Ming Lei 
Cc: Wanpeng Li 
Reviewed-by: Christoph Hellwig 
Signed-off-by: Jens Axboe