Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Tags: Platforms default
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

Platforms default
Tino Reichardt
ZTS: use openssl for md5digest and sha256digest

On larger files this should improve the speed.

Sample values of my system:

[mcmilk@xz]$ time dd if=/dev/zero bs=128k count=1k | sha256sum
254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917  -
real    0m1,050s
user    0m0,985s
sys    0m0,153s

[mcmilk@xz]$ time dd if=/dev/zero bs=128k count=1k | openssl sha256 -r
254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917 *stdin
real    0m0,254s
user    0m0,206s
sys    0m0,160s

I think cli_root/zdb/zdb_backup.ksh runs also an FreeBSD and I needed to
include the sysutils/coreutils package for the FreeBSD tests within the
QEMU patchset.

This could be reverted, when this pull request gets upstream

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16543 part 1/1
Rob Norris
zfs_log: add flex array fields to log record structs

ZIL log record structs (lr_XX_t) are frequently allocated with extra
space after the struct to carry variable-sized "payload" items.

Linux 6.10+ compiled with CONFIG_FORTIFY_SOURCE has been doing runtime
bounds checking on memcpy() calls. Because these types had no indicator
that they might use more space than their simple definition,
__fortify_memcpy_chk will frequently complain about overruns eg:

    memcpy: detected field-spanning write (size 7) of single field
        "lr + 1" at zfs_log.c:425 (size 0)
    memcpy: detected field-spanning write (size 9) of single field
        "(char *)(lr + 1)" at zfs_log.c:593 (size 0)
    memcpy: detected field-spanning write (size 4) of single field
        "(char *)(lr + 1) + snamesize" at zfs_log.c:594 (size 0)
    memcpy: detected field-spanning write (size 7) of single field
        "lr + 1" at zfs_log.c:425 (size 0)
    memcpy: detected field-spanning write (size 9) of single field
        "(char *)(lr + 1)" at zfs_log.c:593 (size 0)
    memcpy: detected field-spanning write (size 4) of single field
        "(char *)(lr + 1) + snamesize" at zfs_log.c:594 (size 0)
    memcpy: detected field-spanning write (size 7) of single field
        "lr + 1" at zfs_log.c:425 (size 0)
    memcpy: detected field-spanning write (size 9) of single field
        "(char *)(lr + 1)" at zfs_log.c:593 (size 0)
    memcpy: detected field-spanning write (size 4) of single field
        "(char *)(lr + 1) + snamesize" at zfs_log.c:594 (size 0)

To fix this, this commit adds flex array fields to all lr_XX_t structs
that require them, and then uses those fields to access that
end-of-struct area rather than more complicated casts and pointer
addition.

Signed-off-by: Rob Norris <robn@despairlabs.com>
Sponsored-by: https://despairlabs.com/sponsor/

Pull-request: #16539 part 1/1
Tino Reichardt
ZTS: Use QEMU for tests on Linux and FreeBSD

This commit adds functional tests for these systems:
- AlmaLinux 8, AlmaLinux 9, ArchLinux
- CentOS Stream 9, Fedora 39, Fedora 40
- Debian 11, Debian 12
- FreeBSD 13, FreeBSD 14, FreeBSD 15
- Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

- enabled by default:
- AlmaLinux 8, AlmaLinux 9
- Debian 11, Debian 12
- Fedora 39, Fedora 40
- FreeBSD 13, FreeBSD 14

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

Real-world benefits from this PR:

1. The github runner scripts are in the zfs repo itself. That means
  you can just open a PR against zfs, like "Add Fedora 41 tester", and
  see the results directly in the PR. ZFS admins no longer need
  manually to login to the buildbot server to update the buildbot config
  with new version of Fedora/Almalinux.

2. Github runners allow you to run the entire test suite against your
  private branch before submitting a formal PR to openzfs. Just open a
  PR against your private zfs repo, and the exact same
  Fedora/Alma/FreeBSD runners will fire up and run ZTS. This can be
  useful if you want to iterate on a ZTS change before submitting a
  formal PR.

3. buildbot is incredibly cumbersome. Our buildbot config files alone
  are ~1500 lines (not including any build/setup scripts)!
  It's a huge pain to setup.

4. We're running the super ancient buildbot 0.8.12. It's so ancient
  it requires python2. We actually have to build python2 from source
  for almalinux9 just to get it to run. Ugrading to a more modern
  buildbot is a huge undertaking, and the UI on the newer versions is
  worse.

5. Buildbot uses EC2 instances. EC2 is a pain because:
  * It costs money
  * They throttle IOPS and CPU usage, leading to mysterious,
  * hard-to-diagnose, failures and timeouts in ZTS.
  * EC2 is high maintenance. We have to setup security groups, SSH
  * keys, networking, users, etc, in AWS and it's a pain. We also
  * have to periodically go in an kill zombie EC2 instances that
  * buildbot is unable to kill off.

6. Buildbot doesn't always handle failures well. One of the things we
  saw in the past was the FreeBSD builders would often die, and each
  builder death would take up a "slot" in buildbot. So we would
  periodically have to restart buildbot via a cron job to get the slots
  back.

7. This PR divides up the ZTS test list into two parts, launches two
  VMs, and on each VM runs half the test suite. The test results are
  then merged and shown in the sumary page. So we're basically
  parallelizing ZTS on the same github runner. This leads to lower
  overall ZTS runtimes (2.5-3 hours vs 4+ hours on buildbot), and one
  unified set of results per runner, which is nice.

8. Since the tests are running on a VM, we have much more control over
  what happens. We can capture the serial console output even if the
  test completely brings down the VM. In the future, we could also
  restart the test on the VM where it left off, so that if a single test
  panics the VM, we can just restart it and run the remaining ZTS tests
  (this functionaly is not yet implemented though, just an idea).

9. Using the runners, users can manually kill or restart a test run
  via the github IU. That really isn't possible with buildbot unless
  you're an admin.

10. Anecdotally, the tests seem to be more stable and constant under
    the QEMU runners.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 4/4
Tino Reichardt
ZTS: increase timeout of mmap_sync_001_pos

On load the test needs sometimes a bit more time then just one second.
Doubling the time will help on the QEMU based testings.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 3/4
Tino Reichardt
ZTS: fix raidz_expand_001_pos and raidz_expand_002_pos

Sometimes the pool may start an auto scrub.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 2/4
Tino Reichardt
ZTS: fix zpool_status_008_pos test on qemu vm's

The test needs some adjusting within the timings.

Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Co-authored-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 1/4
Tino Reichardt
ZTS: Use QEMU for tests on Linux and FreeBSD

This commit adds functional tests for these systems:
- AlmaLinux 8, AlmaLinux 9, ArchLinux
- CentOS Stream 9, Fedora 39, Fedora 40
- Debian 11, Debian 12
- FreeBSD 13, FreeBSD 14, FreeBSD 15
- Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

- enabled by default:
  - AlmaLinux 8, AlmaLinux 9
  - Debian 11, Debian 12
  - Fedora 39, Fedora 40
  - FreeBSD 13, FreeBSD 14

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 4/4
Tino Reichardt
ZTS: increase timeout of mmap_sync_001_pos

On load the test needs sometimes a bit more time then just one second.
Doubling the time will help on the QEMU based testings.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 3/4
Tino Reichardt
ZTS: fix raidz_expand_001_pos and raidz_expand_002_pos

Sometimes the pool may start an auto scrub.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 2/4
Tino Reichardt
ZTS: fix zpool_status_008_pos test on qemu vm's

The test needs some adjusting within the timings.

Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Co-authored-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 1/4
Tino Reichardt
ZTS: Use QEMU for tests on Linux and FreeBSD

This commit adds functional tests for these systems:
- AlmaLinux 8, AlmaLinux 9, ArchLinux
- CentOS Stream 9, Fedora 39, Fedora 40
- Debian 11, Debian 12
- FreeBSD 13, FreeBSD 14, FreeBSD 15
- Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

- enabled by default:
  - AlmaLinux 8, AlmaLinux 9
  - Debian 11, Debian 12
  - Fedora 39, Fedora 40
  - FreeBSD 13, FreeBSD 14

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 4/4
Tino Reichardt
ZTS: increase timeout of mmap_sync_001_pos

On load the test needs sometimes a bit more time then just one second.
Doubling the time will help on the QEMU based testings.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 3/4
Tino Reichardt
ZTS: fix raidz_expand_001_pos and raidz_expand_002_pos

Sometimes the pool may start an auto scrub.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 2/4
Tino Reichardt
ZTS: fix zpool_status_008_pos test on qemu vm's

The test needs some adjusting within the timings.

Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Co-authored-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 1/4
Tino Reichardt
ZTS: Use QEMU for tests on Linux and FreeBSD

This commit adds functional tests for these systems:
- AlmaLinux 8, AlmaLinux 9, ArchLinux
- CentOS Stream 9, Fedora 39, Fedora 40
- Debian 11, Debian 12
- FreeBSD 13, FreeBSD 14, FreeBSD 15
- Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

- enabled by default:
  - AlmaLinux 8, AlmaLinux 9
  - Debian 11, Debian 12
  - Fedora 39, Fedora 40
  - FreeBSD 13, FreeBSD 14

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 3/3
Tino Reichardt
ZTS: fix raidz_expand_001_pos and raidz_expand_002_pos

Sometimes the pool may start an auto scrub.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>

Pull-request: #16537 part 2/3
Tino Reichardt
ZTS: fix zpool_status_008_pos test on qemu vm's

The test needs some adjusting within the timings.

Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Co-authored-by: Tino Reichardt <milky-zfs@mcmilk.de>

Pull-request: #16537 part 1/3
Alexander Motin
Remove extra newline from spa_set_allocator().

zfs_dbgmsg() does not need newline at the end of the message.

While there, slightly update/sync FreeBSD __dprintf().

Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored by: iXsystems, Inc.

Pull-request: #16536 part 1/1
Don Brady
Avoid fault diagnosis if multiple vdevs have errors

When multiple drives are throwing errors, it is likely not
a drive failing but rather a failure above the drives, like
a controller.  The active cases context of the drive's peers
is now considered when making a diagnosis.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

Signed-off-by: Don Brady <don.brady@klarasystems.com>

Pull-request: #16531 part 1/1
Theera K.
arcstat: add structural, types, states breakdown

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

Delete cmd/arc_summary.py

incorrect file extension

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat.1 : add mfusz, mrusz, l2wbytes

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add mfusz, mrusz, l2wbytes

mfusz: MFU size
mrusz: MRU size
l2wbytes: Bytes write per second to the L2ARC

Signed-off-by: Theera K. <tkittich@hotmail.com>

remove extra spaces

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

prettynum: add special case when interval is 1

when interval is 1 only show decimal when value is < 1

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix scale == 100 & change prettynum()

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix prettynum rounding

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 1/1
GitHub
fix prettynum rounding

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 2/2
Theera K.
arcstat: add structural, types, states breakdown

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

Delete cmd/arc_summary.py

incorrect file extension

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat.1 : add mfusz, mrusz, l2wbytes

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add mfusz, mrusz, l2wbytes

mfusz: MFU size
mrusz: MRU size
l2wbytes: Bytes write per second to the L2ARC

Signed-off-by: Theera K. <tkittich@hotmail.com>

remove extra spaces

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

prettynum: add special case when interval is 1

when interval is 1 only show decimal when value is < 1

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix scale == 100 & change prettynum()

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 1/1
GitHub
fix scale == 100 & change prettynum()

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 2/2
Theera K.
arcstat: add structural, types, states breakdown

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

Delete cmd/arc_summary.py

incorrect file extension

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat.1 : add mfusz, mrusz, l2wbytes

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add mfusz, mrusz, l2wbytes

mfusz: MFU size
mrusz: MRU size
l2wbytes: Bytes write per second to the L2ARC

Signed-off-by: Theera K. <tkittich@hotmail.com>

remove extra spaces

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

prettynum: add special case when interval is 1

when interval is 1 only show decimal when value is < 1

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 1/1
GitHub
prettynum: add special case when interval is 1

when interval is 1 only show decimal when value is < 1

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 2/2
Theera K.
arcstat: add structural, types, states breakdown

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

rename column names to fit 5 chars

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

add data target, metadata target

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix abd typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

make column name a bit shorter

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

Delete cmd/arc_summary.py

incorrect file extension

Signed-off-by: Theera K. <tkittich@hotmail.com>

fix structural typo

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add structural, types, states breakdown

add ARC structural breakdown, ARC types breakdown, ARC states breakdown similar to arc_summary

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add Anonymous, MFU, MRU, Uncached

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat.1 : add mfusz, mrusz, l2wbytes

Signed-off-by: Theera K. <tkittich@hotmail.com>

Update arcstat: add mfusz, mrusz, l2wbytes

mfusz: MFU size
mrusz: MRU size
l2wbytes: Bytes write per second to the L2ARC

Signed-off-by: Theera K. <tkittich@hotmail.com>

remove extra spaces

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

arcstat: add target size of data, meta, MFU, MRU

arcstat: add target size of ARC data, ARC metadata, MFU, MRU

Signed-off-by: Theera K. <tkittich@hotmail.com>

shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 1/1
GitHub
shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 3/3
GitHub
shorten new column names; show decimal when < 10

Signed-off-by: Theera K. <tkittich@hotmail.com>

Pull-request: #16509 part 2/3
Rob Norris
zfs_file: implement zfs_file_deallocate for FreeBSD 14

FreeBSD 14 gained a `VOP_DEALLOCATE` VFS operation and a `fspacectl`
syscall to use it. At minimum, these zero the given region, and if the
underlying filesystem supports it, can make the region sparse. We can
use this to get TRIM-like behaviour for file vdevs.

Sponsored-by: https://despairlabs.com/sponsor/
Signed-off-by: Rob Norris <robn@despairlabs.com>

Pull-request: #16496 part 2/2
Rob Norris
zfs_file: rename zfs_file_fallocate to zfs_file_deallocate

We only use it on a specific way: to punch a hole in (make sparse) a
region of a file, in order to implement TRIM-like behaviour.

So, call the op "deallocate", and move the Linux-style mode flags down
into the Linux implementation, since they're an implementation detail.

FreeBSD gets a no-op stub (for the moment).

Sponsored-by: https://despairlabs.com/sponsor/
Signed-off-by: Rob Norris <robn@despairlabs.com>

Pull-request: #16496 part 1/2
Rob Norris
zfs_debug: specific variant for userspace

Just nice and simple, with room to grow.

Signed-off-by: Rob Norris <robn@despairlabs.com>
Sponsored-by: https://despairlabs.com/sponsor/

Pull-request: #16492 part 6/6
Rob Norris
config: fix page_mapping test

It always failed from "unused variable" warnings-errors. The resulting
`#define page_mapping(...)` happend to work because it always overrode
the kernel's function prototype, but that's brittle.

Sponsored-by: https://despairlabs.com/sponsor/
Signed-off-by: Rob Norris <robn@despairlabs.com>

Pull-request: #16479 part 104/104
Rob Norris
linux: log a scary warning when used with an experimental kernel

Since the person using the kernel may not be the person who built it,
show a warning at module load too, in case they aren't aware that it
might be weird.

Signed-off-by: Rob Norris <robn@despairlabs.com>
Sponsored-by: https://despairlabs.com/sponsor/

Pull-request: #15986 part 2/2
Rob Norris
config/kernel: introduce "maximum experimental" kernel version

META lists the maximum kernel version we consider to be fully supported.
However, we don't enforce this.

Sometimes we ship experimental patches for a newer kernel than we're
ready to support or, less often, we compile just fine against a newer
kernel. Invariably, something doesn't quite work properly, and it's
difficult for users to understan that they're actually running against a
kernel that we're not yet ready to support.

This commit tries to improve this situation. First, it simply enforces
Linux-Maximum, by having configure bail out if you try to compile
against a newer version that.

Then, it adds the --enable-linux-experimental switch to configure. When
supplied, this disables enforcing the maximum version, allowing the user
to attempt to build against a kernel with version higher than
Linux-Maximum.

Finally, if the switch is supplied _and_ configure is run against a
higher kernel version, it shows a big warning message when configure
finishes, and defines HAVE_LINUX_EXPERIMENTAL for the build. This allows
us to add code to modify runtime behaviour as well.  also define HAVE

Signed-off-by: Rob Norris <robn@despairlabs.com>
Sponsored-by: https://despairlabs.com/sponsor/

Pull-request: #15986 part 1/2
Brian Atkinson
Adding Direct IO Support

Adding O_DIRECT support to ZFS to bypass the ARC for writes/reads.

O_DIRECT support in ZFS will always ensure there is coherency between
buffered and O_DIRECT IO requests. This ensures that all IO requests,
whether buffered or direct, will see the same file contents at all
times. Just as in other FS's , O_DIRECT does not imply O_SYNC. While
data is written directly to VDEV disks, metadata will not be synced
until the associated  TXG is synced.
For both O_DIRECT read and write request the offset and requeset sizes,
at a minimum, must be PAGE_SIZE aligned. In the event they are not,
then EINVAL is returned unless the direct property is set to always (see
below).

For O_DIRECT writes:
The request also must be block aligned (recordsize) or the write
request will take the normal (buffered) write path. In the event that
request is block aligned and a cached copy of the buffer in the ARC,
then it will be discarded from the ARC forcing all further reads to
retrieve the data from disk.

For O_DIRECT reads:
The only alignment restrictions are PAGE_SIZE alignment. In the event
that the requested data is in buffered (in the ARC) it will just be
copied from the ARC into the user buffer.

For both O_DIRECT writes and reads the O_DIRECT flag will be ignored in
the event that file contents are mmap'ed. In this case, all requests
that are at least PAGE_SIZE aligned will just fall back to the buffered
paths. If the request however is not PAGE_SIZE aligned, EINVAL will
be returned as always regardless if the file's contents are mmap'ed.

Since O_DIRECT writes go through the normal ZIO pipeline, the
following operations are supported just as with normal buffered writes:
Checksum
Compression
Encryption
Erasure Coding
There is one caveat for the data integrity of O_DIRECT writes that is
distinct for each of the OS's supported by ZFS.
FreeBSD - FreeBSD is able to place user pages under write protection so
          any data in the user buffers and written directly down to the
  VDEV disks is guaranteed to not change. There is no concern
  with data integrity and O_DIRECT writes.
Linux - Linux is not able to place anonymous user pages under write
        protection. Because of this, if the user decides to manipulate
the page contents while the write operation is occurring, data
integrity can not be guaranteed. However, there is a module
parameter `zfs_vdev_direct_write_verify` that contols the
if a O_DIRECT writes that can occur to a top-level VDEV before
a checksum verify is run before the contents of the I/O buffer
        are committed to disk. In the event of a checksum verification
failure the write will return EIO. The number of O_DIRECT write
checksum verification errors can be observed by doing
`zpool status -d`, which will list all verification errors that
have occurred on a top-level VDEV. Along with `zpool status`, a
ZED event will be issues as `dio_verify` when a checksum
verification error occurs.

ZVOLs and dedup is not currently supported with Direct I/O.

A new dataset property `direct` has been added with the following 3
allowable values:
disabled - Accepts O_DIRECT flag, but silently ignores it and treats
  the request as a buffered IO request.
standard - Follows the alignment restrictions  outlined above for
  write/read IO requests when the O_DIRECT flag is used.
always  - Treats every write/read IO request as though it passed
          O_DIRECT and will do O_DIRECT if the alignment restrictions
  are met otherwise will redirect through the ARC. This
  property will not allow a request to fail.

There is also a module paramter zfs_dio_enabled that can be used to
force all reads and writes through the ARC. By setting this module
paramter to 0, it mimics as if the  direct dataset property is set to
disabled.

Signed-off-by: Brian Atkinson <batkinson@lanl.gov>
Co-authored-by: Mark Maybee <mark.maybee@delphix.com>
Co-authored-by: Matt Macy <mmacy@FreeBSD.org>
Co-authored-by: Brian Behlendorf <behlendorf@llnl.gov>

Pull-request: #10018 part 1/1
Brian Atkinson
Updating based on PR Feedback(6)

1. Updated typo in man page zfs.4.
2. Fix fat fingered typing errors in zpl_aio_write().
3. Fixed spelling O_DIRECT typo in zpl_direct_IO_impl()
4. Updated dmu_write_uio_dnode() to issue a write_size based multiple
  dn->dn_datablksz chunks at once.
5. Removed empty lines in zfs_write().
6. Returned code back to same indentation in zfs_get_data().
7. Removed duplicate ASSERT statements in dmu_buf_will_clone_or_dio().
8. Fixed spelling typo of cause in comment in
  dmu_buf_will_clone_or_dio().
9. Return 0 in FreeBSD zfs_uio_get_pages() when count != nr_pages.
10. Updated FreeBSD zfs_uio_get_dio_pages_alloc() to unhold pages in
    the event of an error.
11. Linux changed zfs_uio_iov_step() to use SET_ERROR() so it matches
    the FreeBSD implementation.
12. Upated zfs_read() to add back dio_remaining_resid to n in the event
    of an error.
13. Added an ASSERT in zio_ddt_write() making sure no Direct I/O writes
    are issued with deduplication. Also, added a comment with ASSERT to
    state why Direct I/O writes can not use deduplication.
14. Removed _KERNEL include guard around zfs_dio_page_aligned(). The
    proper uio_impl.h or uio.h is included through zfs_context.h.
15. Updated the zfsprop man page to state that Direct I/O is not
  currenntly available for zvols. Also that Direct I/O writes are
  incompatiable with dedup, so dedup will not take place with Direct
  I/O writes.
16. Fixed typo in comment in manipulate_user_buffer.c.

Signed-off-by: Brian Atkinson <batkinson@lanl.gov>

Pull-request: #10018 part 7/7
Brian Atkinson
Updating based on PR Feedback(6)

1. Updated typo in man page zfs.4.
2. Fix fat fingered typing errors in zpl_aio_write().
3. Fixed spelling O_DIRECT typo in zpl_direct_IO_impl()
4. Updated dmu_write_uio_dnode() to issue a write_size based multiple
  dn->dn_datablksz chunks at once.
5. Removed empty lines in zfs_write().
6. Returned code back to same indentation in zfs_get_data().
7. Removed duplicate ASSERT statements in dmu_buf_will_clone_or_dio().
8. Fixed spelling typo of cause in comment in
  dmu_buf_will_clone_or_dio().
9. Return 0 in FreeBSD zfs_uio_get_pages() when count != nr_pages.
10. Updated FreeBSD zfs_uio_get_dio_pages_alloc() to unhold pages in
    the event of an error.
11. Linux changed zfs_uio_iov_step() to use SET_ERROR() so it matches
    the FreeBSD implementation.
12. Upated zfs_read() to add back dio_remaining_resid to n in the event
    of an error.
13. Added an ASSERT in zio_ddt_write() making sure no Direct I/O writes
    are issued with deduplication. Also, added a comment with ASSERT to
    state why Direct I/O writes can not use deduplication.
14. Removed _KERNEL include guard around zfs_dio_page_aligned(). The
    proper uio_impl.h or uio.h is included through zfs_context.h.

Signed-off-by: Brian Atkinson <batkinson@lanl.gov>

Pull-request: #10018 part 7/7