Merge branch 'dev'

Refactored CFROUND

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Required data**
+ - Miner log as text or screenshot
+ - Config file or command line (without wallets)
+ - OS: [e.g. Windows]
+ - For GPU related issues: information about GPUs and driver version.
+
+**Additional context**
+Add any other context about the problem here.

CHANGELOG.md

@@ -1,111 +1,159 @@
-# v4.6.1-beta
+# v5.11.1
+- [#1652](https://github.com/xmrig/xmrig/pull/1652) Up to 1% RandomX perfomance improvement on recent AMD CPUs.
+- [#1306](https://github.com/xmrig/xmrig/issues/1306) Fixed possible double connection to a pool.
+- [#1654](https://github.com/xmrig/xmrig/issues/1654) Fixed build with LibreSSL.
+
+# v5.11.0
+- **[#1632](https://github.com/xmrig/xmrig/pull/1632) Added AstroBWT CUDA support ([CUDA plugin](https://github.com/xmrig/xmrig-cuda) v3.0.0 or newer required).**
+- [#1605](https://github.com/xmrig/xmrig/pull/1605) Fixed AstroBWT OpenCL for NVIDIA GPUs.
+- [#1635](https://github.com/xmrig/xmrig/pull/1635) Added pooled memory allocation of RandomX VMs (+0.5% speedup on Zen2).
+- [#1641](https://github.com/xmrig/xmrig/pull/1641) RandomX JIT refactoring, smaller memory footprint and a bit faster overall.
+- [#1643](https://github.com/xmrig/xmrig/issues/1643) Fixed build on CentOS 7.
+
+# v5.10.0
+- [#1602](https://github.com/xmrig/xmrig/pull/1602) Added AMD GPUs support for AstroBWT algorithm.
+- [#1590](https://github.com/xmrig/xmrig/pull/1590) MSR mod automatically deactivated after switching from RandomX algorithms.
+- [#1592](https://github.com/xmrig/xmrig/pull/1592) Added AVX2 optimized code for AstroBWT algorithm.
+  - Added new config option `astrobwt-avx2` in `cpu` object and command line option `--astrobwt-avx2`.
+- [#1596](https://github.com/xmrig/xmrig/issues/1596) Major TLS (Transport Layer Security) subsystem update.
+  - Added new TLS options, please check [xmrig-proxy documentation](https://xmrig.com/docs/proxy/tls) for details.
+- `cn/gpu` algorithm now disabled by default and will be removed in next major (v6.x.x) release, no ETA for it right now.
+- Added command line option `--data-dir`.
+
+# v5.9.0
+- [#1578](https://github.com/xmrig/xmrig/pull/1578) Added new RandomKEVA algorithm for upcoming Kevacoin fork, as `"algo": "rx/keva"` or `"coin": "keva"`.
+- [#1584](https://github.com/xmrig/xmrig/pull/1584) Fixed invalid AstroBWT hashes after algorithm switching.
+- [#1585](https://github.com/xmrig/xmrig/issues/1585) Fixed build without HTTP support.
+- Added command line option `--astrobwt-max-size`.
+
+# v5.8.2
+- [#1580](https://github.com/xmrig/xmrig/pull/1580) AstroBWT algorithm 20-50% speedup.
+  - Added new option `astrobwt-max-size`.
+- [#1581](https://github.com/xmrig/xmrig/issues/1581) Fixed macOS build.
+
+# v5.8.1
+- [#1575](https://github.com/xmrig/xmrig/pull/1575) Fixed new block detection for DERO solo mining.
+
+# v5.8.0
+- [#1573](https://github.com/xmrig/xmrig/pull/1573) Added new AstroBWT algorithm for upcoming DERO fork, as `"algo": "astrobwt"` or `"coin": "dero"`.
+
+# v5.7.0
+- **Added SOCKS5 proxies support for Tor https://xmrig.com/docs/miner/tor.**
+- [#377](https://github.com/xmrig/xmrig-proxy/issues/377) Fixed duplicate jobs in daemon (solo) mining client.
+- [#1560](https://github.com/xmrig/xmrig/pull/1560) RandomX 0.3-0.4% speedup depending on CPU.
+- Fixed possible crashes in HTTP client.
+
+# v5.6.0
+- [#1536](https://github.com/xmrig/xmrig/pull/1536) Added workaround for new AMD GPU drivers.
+- [#1546](https://github.com/xmrig/xmrig/pull/1546) Fixed generic OpenCL code for AMD Navi GPUs.
+- [#1551](https://github.com/xmrig/xmrig/pull/1551) Added RandomX JIT for AMD Navi  GPUs.
+- Added health information for AMD GPUs (clocks/power/fan/temperature) via ADL (Windows) and sysfs (Linux).
+- Fixed possible nicehash nonce overflow in some conditions.
+- Fixed wrong OpenCL platform on macOS, option `platform` now ignored on this OS.
+
+# v5.5.3
+- [#1529](https://github.com/xmrig/xmrig/pull/1529) Fixed crash on Bulldozer CPUs.
+
+# v5.5.2
+- [#1500](https://github.com/xmrig/xmrig/pull/1500) Removed unnecessary code from RandomX JIT compiler.
+- [#1502](https://github.com/xmrig/xmrig/pull/1502) Optimizations for AMD Bulldozer.
+- [#1508](https://github.com/xmrig/xmrig/pull/1508) Added support for BMI2 instructions.
+- [#1510](https://github.com/xmrig/xmrig/pull/1510) Optimized `CFROUND` instruction for RandomX.
+- [#1520](https://github.com/xmrig/xmrig/pull/1520) Fixed thread affinity.
+
+# v5.5.1
+- [#1469](https://github.com/xmrig/xmrig/issues/1469) Fixed build with gcc 4.8.
+- [#1473](https://github.com/xmrig/xmrig/pull/1473) Added RandomX auto-config for mobile Ryzen APUs.
+- [#1477](https://github.com/xmrig/xmrig/pull/1477) Fixed build with Clang.
+- [#1489](https://github.com/xmrig/xmrig/pull/1489) RandomX JIT compiler tweaks.
+- [#1493](https://github.com/xmrig/xmrig/pull/1493) Default value for Intel MSR preset changed to `15`.
+- Fixed unwanted resume after RandomX dataset change.
+
+# v5.5.0
+- [#179](https://github.com/xmrig/xmrig/issues/179) Added support for [environment variables](https://xmrig.com/docs/miner/environment-variables) in config file.
+- [#1445](https://github.com/xmrig/xmrig/pull/1445) Removed `rx/v` algorithm.
+- [#1453](https://github.com/xmrig/xmrig/issues/1453) Fixed crash on 32bit systems.
+- [#1459](https://github.com/xmrig/xmrig/issues/1459) Fixed crash on very low memory systems.
+- [#1465](https://github.com/xmrig/xmrig/pull/1465) Added fix for 1st-gen Ryzen crashes.
+- [#1466](https://github.com/xmrig/xmrig/pull/1466) Added `cn-pico/tlo` algorithm.
+- Added `--randomx-no-rdmsr` command line option.
+- Added console title for Windows with miner name and version.
+- On Windows `priority` option now also change base priority.
+
+# v5.4.0
+- [#1434](https://github.com/xmrig/xmrig/pull/1434) Added RandomSFX (`rx/sfx`) algorithm for Safex Cash.
+- [#1445](https://github.com/xmrig/xmrig/pull/1445) Added RandomV (`rx/v`) algorithm for *new* MoneroV.
+- [#1419](https://github.com/xmrig/xmrig/issues/1419) Added reverting MSR changes on miner exit, use `"rdmsr": false,` in `"randomx"` object to disable this feature.
+- [#1423](https://github.com/xmrig/xmrig/issues/1423) Fixed conflicts with exists WinRing0 driver service.
+- [#1425](https://github.com/xmrig/xmrig/issues/1425) Fixed crash on first generation Zen CPUs (MSR mod accidentally enable Opcache), additionally now you can disable Opcache and enable MSR mod via config `"wrmsr": ["0xc0011020:0x0", "0xc0011021:0x60", "0xc0011022:0x510000", "0xc001102b:0x1808cc16"],`.
+- Added advanced usage for `wrmsr` option, for example: `"wrmsr": ["0x1a4:0x6"],` (Intel) and `"wrmsr": ["0xc0011020:0x0", "0xc0011021:0x40:0xffffffffffffffdf", "0xc0011022:0x510000", "0xc001102b:0x1808cc16"],` (Ryzen).
+- Added new config option `"verbose"` and command line option `--verbose`.
+
+# v5.3.0
+- [#1414](https://github.com/xmrig/xmrig/pull/1414) Added native MSR support for Windows, by using signed **WinRing0 driver** (© 2007-2009 OpenLibSys.org).
+- Added new [MSR documentation](https://xmrig.com/docs/miner/randomx-optimization-guide/msr).
+- [#1418](https://github.com/xmrig/xmrig/pull/1418) Increased stratum send buffer size.
+
+# v5.2.1
+- [#1408](https://github.com/xmrig/xmrig/pull/1408) Added RandomX boost script for Linux (if you don't like run miner with root privileges).
+- Added support for [AMD Ryzen MSR registers](https://www.reddit.com/r/MoneroMining/comments/e962fu/9526_hs_on_ryzen_7_3700x_xmrig_520_1gb_pages_msr/) (Linux only).
+- Fixed command line option `--randomx-wrmsr` option without parameters.
+
+# v5.2.0
+- **[#1388](https://github.com/xmrig/xmrig/pull/1388) Added [1GB huge pages support](https://xmrig.com/docs/miner/hugepages#onegb-huge-pages) for Linux.**
+  - Added new option `1gb-pages` in `randomx` object with command line equivalent `--randomx-1gb-pages`.
+  - Added automatic huge pages configuration on Linux if use the miner with root privileges.
+- **Added [automatic Intel prefetchers configuration](https://xmrig.com/docs/miner/randomx-optimization-guide#intel-specific-optimizations) on Linux.**
+   - Added new option `wrmsr` in `randomx` object with command line equivalent `--randomx-wrmsr=6`.
+- [#1396](https://github.com/xmrig/xmrig/pull/1396) [#1401](https://github.com/xmrig/xmrig/pull/1401) New performance optimizations for Ryzen CPUs. 
+- [#1385](https://github.com/xmrig/xmrig/issues/1385) Added `max-threads-hint` option support for RandomX dataset initialization threads.  
+- [#1386](https://github.com/xmrig/xmrig/issues/1386) Added `priority` option support for RandomX dataset initialization threads. 
+- For official builds all dependencies (libuv, hwloc, openssl) updated to recent versions.
+- Windows `msvc` builds now use Visual Studio 2019 instead of 2017.
+
+# v5.1.1
+- [#1365](https://github.com/xmrig/xmrig/issues/1365) Fixed various system response/stability issues.
+  - Added new CPU option `yield` and command line equivalent `--cpu-no-yield`.
+- [#1363](https://github.com/xmrig/xmrig/issues/1363) Fixed wrong priority of main miner thread.
+
+# v5.1.0
+- [#1351](https://github.com/xmrig/xmrig/pull/1351) RandomX optimizations and fixes.
+  - Improved RandomX performance (up to +6-7% on Intel CPUs, +2-3% on Ryzen CPUs)
+  - Added workaround for Intel JCC erratum bug see https://www.phoronix.com/scan.php?page=article&item=intel-jcc-microcode&num=1 for details.
+  - Note! Always disable "Hardware prefetcher" and "Adjacent cacheline prefetch" in BIOS for Intel CPUs to get the optimal RandomX performance.
+- [#1307](https://github.com/xmrig/xmrig/issues/1307) Fixed mining resume after donation round for pools with `self-select` feature.
+- [#1318](https://github.com/xmrig/xmrig/issues/1318#issuecomment-559676080) Added option `"mode"` (or `--randomx-mode`) for RandomX.
+  - Added memory information on miner startup.
+  - Added `resources` field to summary API with memory information and load average.
+
+# v5.0.1
+- [#1234](https://github.com/xmrig/xmrig/issues/1234) Fixed compatibility with some AMD GPUs.
+- [#1284](https://github.com/xmrig/xmrig/issues/1284) Fixed build without RandomX.
+- [#1285](https://github.com/xmrig/xmrig/issues/1285) Added command line options `--cuda-bfactor-hint` and `--cuda-bsleep-hint`.
+- [#1290](https://github.com/xmrig/xmrig/pull/1290) Fixed 32-bit ARM compilation.
+
+# v5.0.0
+This version is first stable unified 3 in 1 GPU+CPU release, OpenCL support built in in miner and not require additional external dependencies on compile time, NVIDIA CUDA available as external [CUDA plugin](https://github.com/xmrig/xmrig-cuda), for convenient, 3 in 1 downloads with recent CUDA version also provided.
+
+This release based on 4.x.x series and include all features from v4.6.2-beta, changelog below include only the most important changes, [full changelog](doc/CHANGELOG_OLD.md) available separately.
+
 - [#1272](https://github.com/xmrig/xmrig/pull/1272) Optimized hashrate calculation.
-- [#1273](https://github.com/xmrig/xmrig/issues/1273) Fixed crash when use `GET /2/backends` API endpoint with disabled CUDA.
-
-# v4.6.0-beta
 - [#1263](https://github.com/xmrig/xmrig/pull/1263) Added new option `dataset_host` for NVIDIA GPUs with less than 4 GB memory (RandomX only).
-
-# v4.5.0-beta
-- Added NVIDIA CUDA support via external [CUDA plugun](https://github.com/xmrig/xmrig-cuda). XMRig now is unified 3 in 1 miner.
-
-# v4.4.0-beta
 - [#1068](https://github.com/xmrig/xmrig/pull/1068) Added support for `self-select` stratum protocol extension.
-- [#1240](https://github.com/xmrig/xmrig/pull/1240) Sync with the latest RandomX code.
-- [#1241](https://github.com/xmrig/xmrig/issues/1241) Fixed regression with colors on old Windows systems.
-- [#1243](https://github.com/xmrig/xmrig/pull/1243) Fixed incorrect OpenCL memory size detection in some cases.
-- [#1247](https://github.com/xmrig/xmrig/pull/1247) Fixed ARM64 RandomX code alignment.
-- [#1248](https://github.com/xmrig/xmrig/pull/1248) Fixed RandomX code cache cleanup on iOS/Darwin.
-
-# v4.3.1-beta
-- Fixed regression in v4.3.0, miner didn't create `cn` mining profile with default config example.
-
-# v4.3.0-beta
 - [#1227](https://github.com/xmrig/xmrig/pull/1227) Added new algorithm `rx/arq`, RandomX variant for upcoming ArQmA fork.
 - [#808](https://github.com/xmrig/xmrig/issues/808#issuecomment-539297156) Added experimental support for persistent memory for CPU mining threads.
 - [#1221](https://github.com/xmrig/xmrig/issues/1221) Improved RandomX dataset memory usage and initialization speed for NUMA machines.
-
-# v4.2.1-beta
-- [#1150](https://github.com/xmrig/xmrig/issues/1150) Fixed build on FreeBSD.
 - [#1175](https://github.com/xmrig/xmrig/issues/1175) Fixed support for systems where total count of NUMA nodes not equal usable nodes count.
-- [#1199](https://github.com/xmrig/xmrig/issues/1199) Fixed excessive memory allocation for OpenCL threads with low intensity.
-- [#1212](https://github.com/xmrig/xmrig/issues/1212) Fixed low RandomX performance after fast algorithm switching.
-
-# v4.2.0-beta
-- [#1202](https://github.com/xmrig/xmrig/issues/1202) Fixed algorithm verification in donate strategy.
-- Added per pool option `coin` with single possible value `monero` for pools without algorithm negotiation, for upcoming Monero fork.
 - Added config option `cpu/max-threads-hint` and command line option `--cpu-max-threads-hint`.
-
-# v4.1.0-beta
-- **OpenCL backend disabled by default.**.
-- [#1183](https://github.com/xmrig/xmrig/issues/1183) Fixed compatibility with systemd.
 - [#1185](https://github.com/xmrig/xmrig/pull/1185) Added JIT compiler for RandomX on ARMv8.
 - Improved API endpoint `GET /2/backends` and added support for this endpoint to [workers.xmrig.info](http://workers.xmrig.info).
 - Added command line option `--no-cpu` to disable CPU backend.
 - Added OpenCL specific command line options: `--opencl`, `--opencl-devices`, `--opencl-platform`, `--opencl-loader` and `--opencl-no-cache`.
+- Added CUDA specific command line options: `--cuda`, `--cuda-loader` and `--no-nvml`.
 - Removed command line option `--http-enabled`, HTTP API enabled automatically if any other `--http-*` option provided.
-
-# v4.0.1-beta
-- [#1177](https://github.com/xmrig/xmrig/issues/1177) Fixed compatibility with old AMD drivers.
-- [#1180](https://github.com/xmrig/xmrig/issues/1180) Fixed possible duplicated shares after algorithm switching.
-- Added support for case if not all backend threads successfully started.
-- Fixed wrong config file permissions after write (only gcc builds on recent Windows 10 affected).
-
-# v4.0.0-beta
 - [#1172](https://github.com/xmrig/xmrig/issues/1172) **Added OpenCL mining backend.**
   - [#268](https://github.com/xmrig/xmrig-amd/pull/268) [#270](https://github.com/xmrig/xmrig-amd/pull/270) [#271](https://github.com/xmrig/xmrig-amd/pull/271) [#273](https://github.com/xmrig/xmrig-amd/pull/273) [#274](https://github.com/xmrig/xmrig-amd/pull/274) [#1171](https://github.com/xmrig/xmrig/pull/1171) Added RandomX support for OpenCL, thanks [@SChernykh](https://github.com/SChernykh).
 - Algorithm `cn/wow` removed, as no longer alive. 
 
-# v3.2.0
-- Added per pool option `coin` with single possible value `monero` for pools without algorithm negotiation, for upcoming Monero fork.
-- [#1183](https://github.com/xmrig/xmrig/issues/1183) Fixed compatibility with systemd.
-
-# v3.1.3
-- [#1180](https://github.com/xmrig/xmrig/issues/1180) Fixed possible duplicated shares after algorithm switching.
-- Fixed wrong config file permissions after write (only gcc builds on recent Windows 10 affected).
-
-# v3.1.2
-- Many RandomX optimizations and fixes.
-  - [#1132](https://github.com/xmrig/xmrig/issues/1132) Fixed build on CentOS 7.
-  - [#1163](https://github.com/xmrig/xmrig/pull/1163) Optimized soft AES code, up to +30% hashrate on CPU without AES support and other optimizations.
-  - [#1166](https://github.com/xmrig/xmrig/pull/1166) Fixed crash when initialize dataset with big threads count (eg 272).
-  - [#1168](https://github.com/xmrig/xmrig/pull/1168) Optimized loading from scratchpad.
-- [#1128](https://github.com/xmrig/xmrig/issues/1128) Fixed CMake 2.8 compatibility.
-
-# v3.1.1
-- [#1133](https://github.com/xmrig/xmrig/issues/1133) Fixed syslog regression.
-- [#1138](https://github.com/xmrig/xmrig/issues/1138) Fixed multiple network bugs.
-- [#1141](https://github.com/xmrig/xmrig/issues/1141) Fixed log in background mode.
-- [#1142](https://github.com/xmrig/xmrig/pull/1142) RandomX hashrate improved by 0.5-1.5% depending on variant and CPU.
-- [#1146](https://github.com/xmrig/xmrig/pull/1146) Fixed race condition in RandomX thread init.
-- [#1148](https://github.com/xmrig/xmrig/pull/1148) Fixed, on Linux linker marking entire executable as having an executable stack.
-- Fixed, for Argon2 algorithms command line options like `--threads` was ignored.
-- Fixed command line options for single pool, free order allowed again.
-
-# v3.1.0
-- [#1107](https://github.com/xmrig/xmrig/issues/1107#issuecomment-522235892) Added Argon2 algorithm family: `argon2/chukwa` and `argon2/wrkz`.
-
-# v3.0.0
-- **[#1111](https://github.com/xmrig/xmrig/pull/1111) Added RandomX (`rx/test`) algorithm for testing and benchmarking.**
-- **[#1036](https://github.com/xmrig/xmrig/pull/1036) Added RandomWOW (`rx/wow`) algorithm for [Wownero](http://wownero.org/).**
-- **[#1050](https://github.com/xmrig/xmrig/pull/1050) Added RandomXL (`rx/loki`) algorithm for [Loki](https://loki.network/).**
-- **[#1077](https://github.com/xmrig/xmrig/issues/1077) Added NUMA support via hwloc**.
-- **Added flexible [multi algorithm](doc/CPU.md) configuration.**
-- **Added unlimited switching between incompatible algorithms, all mining options can be changed in runtime.**
-- [#257](https://github.com/xmrig/xmrig-nvidia/pull/257) New logging subsystem, file and syslog now always without colors.
-- [#314](https://github.com/xmrig/xmrig-proxy/issues/314) Added donate over proxy feature.
-- [#1007](https://github.com/xmrig/xmrig/issues/1007) Old HTTP API backend based on libmicrohttpd, replaced to custom HTTP server (libuv + http_parser).
-- [#1010](https://github.com/xmrig/xmrig/pull/1010#issuecomment-482632107) Added daemon support (solo mining).
-- [#1066](https://github.com/xmrig/xmrig/issues/1066#issuecomment-518080529) Added error message if pool not ready for RandomX.
-- [#1105](https://github.com/xmrig/xmrig/issues/1105) Improved auto configuration for `cn-pico` algorithm.
-- Added commands `pause` and `resume` via JSON RPC 2.0 API (`POST /json_rpc`).
-- Added command line option `--export-topology` for export hwloc topology to a XML file.
-- Breaked backward compatibility with previous configs and command line, `variant` option replaced to `algo`, global option `algo` removed, all CPU related settings moved to `cpu` object.
-- Options `av`, `safe` and `max-cpu-usage` removed.
-- Algorithm `cn/msr` renamed to `cn/fast`.
-- Algorithm `cn/xtl` removed.
-- API endpoint `GET /1/threads` replaced to `GET /2/backends`.
-- Added global uptime and extended connection information in API.
-- API now return current algorithm.
-
 # Previous versions
 [doc/CHANGELOG_OLD.md](doc/CHANGELOG_OLD.md)

CMakeLists.txt

@@ -6,17 +6,21 @@ option(WITH_HWLOC           "Enable hwloc support" ON)
 option(WITH_CN_LITE         "Enable CryptoNight-Lite algorithms family" ON)
 option(WITH_CN_HEAVY        "Enable CryptoNight-Heavy algorithms family" ON)
 option(WITH_CN_PICO         "Enable CryptoNight-Pico algorithm" ON)
-option(WITH_CN_GPU          "Enable CryptoNight-GPU algorithm" ON)
+option(WITH_CN_GPU          "Enable CryptoNight-GPU algorithm" OFF)
 option(WITH_RANDOMX         "Enable RandomX algorithms family" ON)
 option(WITH_ARGON2          "Enable Argon2 algorithms family" ON)
+option(WITH_ASTROBWT        "Enable AstroBWT algorithms family" ON)
 option(WITH_HTTP            "Enable HTTP protocol support (client/server)" ON)
 option(WITH_DEBUG_LOG       "Enable debug log output" OFF)
 option(WITH_TLS             "Enable OpenSSL support" ON)
 option(WITH_ASM             "Enable ASM PoW implementations" ON)
+option(WITH_MSR             "Enable MSR mod & 1st-gen Ryzen fix" ON)
+option(WITH_ENV_VARS        "Enable environment variables support in config file" ON)
 option(WITH_EMBEDDED_CONFIG "Enable internal embedded JSON config" OFF)
 option(WITH_OPENCL          "Enable OpenCL backend" ON)
 option(WITH_CUDA            "Enable CUDA backend" ON)
 option(WITH_NVML            "Enable NVML (NVIDIA Management Library) support (only if CUDA backend enabled)" ON)
+option(WITH_ADL             "Enable ADL (AMD Display Library) or sysfs support (only if OpenCL backend enabled)" ON)
 option(WITH_STRICT_CACHE    "Enable strict checks for OpenCL cache" ON)
 option(WITH_INTERLEAVE_DEBUG_LOG "Enable debug log for threads interleave" OFF)
 
@@ -30,6 +34,7 @@ set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake")
 
 include (CheckIncludeFile)
 include (cmake/cpu.cmake)
+include (cmake/os.cmake)
 include (src/base/base.cmake)
 include (src/backend/backend.cmake)
 
@@ -50,7 +55,6 @@ set(HEADERS
     src/net/JobResult.h
     src/net/JobResults.h
     src/net/Network.h
-    src/net/NetworkState.h
     src/net/strategies/DonateStrategy.h
     src/Summary.h
     src/version.h
@@ -73,9 +77,7 @@ set(HEADERS_CRYPTO
     src/crypto/cn/hash.h
     src/crypto/cn/skein_port.h
     src/crypto/cn/soft_aes.h
-    src/crypto/common/Algorithm.h
+    src/crypto/common/HugePagesInfo.h
-    src/crypto/common/Coin.h
-    src/crypto/common/keccak.h
     src/crypto/common/MemoryPool.h
     src/crypto/common/Nonce.h
     src/crypto/common/portable/mm_malloc.h
@@ -99,7 +101,6 @@ set(SOURCES
     src/core/Miner.cpp
     src/net/JobResults.cpp
     src/net/Network.cpp
-    src/net/NetworkState.cpp
     src/net/strategies/DonateStrategy.cpp
     src/Summary.cpp
     src/xmrig.cpp
@@ -112,9 +113,7 @@ set(SOURCES_CRYPTO
     src/crypto/cn/c_skein.c
     src/crypto/cn/CnCtx.cpp
     src/crypto/cn/CnHash.cpp
-    src/crypto/common/Algorithm.cpp
+    src/crypto/common/HugePagesInfo.cpp
-    src/crypto/common/Coin.cpp
-    src/crypto/common/keccak.cpp
     src/crypto/common/MemoryPool.cpp
     src/crypto/common/Nonce.cpp
     src/crypto/common/VirtualMemory.cpp
@@ -131,40 +130,36 @@ if (WITH_HWLOC)
         )
 endif()
 
-if (WIN32)
+if (XMRIG_OS_WIN)
-    set(SOURCES_OS
+    list(APPEND SOURCES_OS
-        "${SOURCES_OS}"
         res/app.rc
         src/App_win.cpp
         src/crypto/common/VirtualMemory_win.cpp
         )
 
-    add_definitions(/DWIN32)
     set(EXTRA_LIBS ws2_32 psapi iphlpapi userenv)
-elseif (APPLE)
+elseif (XMRIG_OS_APPLE)
-    set(SOURCES_OS
+    list(APPEND SOURCES_OS
-        "${SOURCES_OS}"
         src/App_unix.cpp
         src/crypto/common/VirtualMemory_unix.cpp
         )
 else()
-    set(SOURCES_OS
+    list(APPEND SOURCES_OS
-        "${SOURCES_OS}"
         src/App_unix.cpp
         src/crypto/common/VirtualMemory_unix.cpp
         )
 
-    if (CMAKE_SYSTEM_NAME STREQUAL FreeBSD)
+    if (XMRIG_OS_ANDROID)
-        set(EXTRA_LIBS kvm pthread)
+        set(EXTRA_LIBS pthread rt dl log)
-    else()
+    elseif (XMRIG_OS_LINUX)
-        set(EXTRA_LIBS pthread rt dl)
+        list(APPEND SOURCES_OS
-    endif()
+            src/crypto/common/LinuxMemory.h
-endif()
+            src/crypto/common/LinuxMemory.cpp
+            )
 
-if (CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "Android")
+        set(EXTRA_LIBS pthread rt dl)
-    EXECUTE_PROCESS(COMMAND uname -o COMMAND tr -d '\n' OUTPUT_VARIABLE OPERATING_SYSTEM)
+    elseif (XMRIG_OS_FREEBSD)
-    if (OPERATING_SYSTEM MATCHES "Android")
+        set(EXTRA_LIBS kvm pthread)
-        set(EXTRA_LIBS ${EXTRA_LIBS} log)
     endif()
 endif()
 
@@ -176,6 +171,7 @@ find_package(UV REQUIRED)
 include(cmake/flags.cmake)
 include(cmake/randomx.cmake)
 include(cmake/argon2.cmake)
+include(cmake/astrobwt.cmake)
 include(cmake/OpenSSL.cmake)
 include(cmake/asm.cmake)
 include(cmake/cn-gpu.cmake)
@@ -210,3 +206,8 @@ endif()
 
 add_executable(${CMAKE_PROJECT_NAME} ${HEADERS} ${SOURCES} ${SOURCES_OS} ${SOURCES_CPUID} ${HEADERS_CRYPTO} ${SOURCES_CRYPTO} ${SOURCES_SYSLOG} ${TLS_SOURCES} ${XMRIG_ASM_SOURCES} ${CN_GPU_SOURCES})
 target_link_libraries(${CMAKE_PROJECT_NAME} ${XMRIG_ASM_LIBRARY} ${OPENSSL_LIBRARIES} ${UV_LIBRARIES} ${EXTRA_LIBS} ${CPUID_LIB} ${ARGON2_LIBRARY})
+
+if (WIN32)
+    add_custom_command(TARGET ${CMAKE_PROJECT_NAME} POST_BUILD
+        COMMAND ${CMAKE_COMMAND} -E copy_if_different "${CMAKE_SOURCE_DIR}/bin/WinRing0/WinRing0x64.sys" $<TARGET_FILE_DIR:${CMAKE_PROJECT_NAME}>)
+endif()

README.md

@@ -1,7 +1,5 @@
 # XMRig
 
-**:warning: [Monero will change PoW algorithm to RandomX on November 30.](https://github.com/xmrig/xmrig/issues/1204)**
-
 [![Github All Releases](https://img.shields.io/github/downloads/xmrig/xmrig/total.svg)](https://github.com/xmrig/xmrig/releases)
 [![GitHub release](https://img.shields.io/github/release/xmrig/xmrig/all.svg)](https://github.com/xmrig/xmrig/releases)
 [![GitHub Release Date](https://img.shields.io/github/release-date-pre/xmrig/xmrig.svg)](https://github.com/xmrig/xmrig/releases)
@@ -9,14 +7,14 @@
 [![GitHub stars](https://img.shields.io/github/stars/xmrig/xmrig.svg)](https://github.com/xmrig/xmrig/stargazers)
 [![GitHub forks](https://img.shields.io/github/forks/xmrig/xmrig.svg)](https://github.com/xmrig/xmrig/network)
 
-XMRig High performance, open source, cross platform RandomX, CryptoNight and Argon2 CPU/GPU miner, with official support for Windows.
+XMRig High performance, open source, cross platform RandomX, CryptoNight, AstroBWT and Argon2 CPU/GPU miner, with official support for Windows.
 
 ## Mining backends
 - **CPU** (x64/x86/ARM)
 - **OpenCL** for AMD GPUs.
 - **CUDA** for NVIDIA GPUs via external [CUDA plugin](https://github.com/xmrig/xmrig-cuda).
 
-<img src="doc/screenshot.png" width="808" >
+<img src="doc/screenshot_v5_2_0.png" width="833" >
 
 ## Download
 * Binary releases: https://github.com/xmrig/xmrig/releases
@@ -38,7 +36,8 @@ Network:
   -u, --user=USERNAME           username for mining server
   -p, --pass=PASSWORD           password for mining server
   -O, --userpass=U:P            username:password pair for mining server
-  -k, --keepalive               send keepalived packet for prevent timeout (needs pool support)
+  -x, --proxy=HOST:PORT         connect through a SOCKS5 proxy
+  -k, --keepalive               send keepalive packet for prevent timeout (needs pool support)
       --nicehash                enable nicehash.com support
       --rig-id=ID               rig identifier for pool-side statistics (needs pool support)
       --tls                     enable SSL/TLS support (needs pool support)
@@ -59,10 +58,17 @@ CPU backend:
       --cpu-priority            set process priority (0 idle, 2 normal to 5 highest)
       --cpu-max-threads-hint=N  maximum CPU threads count (in percentage) hint for autoconfig
       --cpu-memory-pool=N       number of 2 MB pages for persistent memory pool, -1 (auto), 0 (disable)
+      --cpu-no-yield            prefer maximum hashrate rather than system response/stability
       --no-huge-pages           disable huge pages support
       --asm=ASM                 ASM optimizations, possible values: auto, none, intel, ryzen, bulldozer
-      --randomx-init=N          threads count to initialize RandomX dataset
+      --randomx-init=N          thread count to initialize RandomX dataset
       --randomx-no-numa         disable NUMA support for RandomX
+      --randomx-mode=MODE       RandomX mode: auto, fast, light
+      --randomx-1gb-pages       use 1GB hugepages for dataset (Linux only)
+      --randomx-wrmsr=N         write custom value (0-15) to Intel MSR register 0x1a4 or disable MSR mod (-1)
+      --randomx-no-rdmsr        disable reverting initial MSR values on exit
+      --astrobwt-max-size=N     skip hashes with large stage 2 size, default: 550, min: 400, max: 1200
+      --astrobwt-avx2           enable AVX2 optimizations for AstroBWT algorithm
 
 API:
       --api-worker-id=ID        custom worker-id for API
@@ -74,7 +80,7 @@ API:
 
 OpenCL backend:
       --opencl                  enable OpenCL mining backend
-      --opencl-devices=N        list of OpenCL devices to use
+      --opencl-devices=N        comma separated list of OpenCL devices to use
       --opencl-platform=N       OpenCL platform index or name
       --opencl-loader=PATH      path to OpenCL-ICD-Loader (OpenCL.dll or libOpenCL.so)
       --opencl-no-cache         disable OpenCL cache
@@ -83,14 +89,27 @@ OpenCL backend:
 CUDA backend:
       --cuda                    enable CUDA mining backend
       --cuda-loader=PATH        path to CUDA plugin (xmrig-cuda.dll or libxmrig-cuda.so)
+      --cuda-devices=N          comma separated list of CUDA devices to use
+      --cuda-bfactor-hint=N     bfactor hint for autoconfig (0-12)
+      --cuda-bsleep-hint=N      bsleep hint for autoconfig
       --no-nvml                 disable NVML (NVIDIA Management Library) support
 
+TLS:
+      --tls-gen=HOSTNAME        generate TLS certificate for specific hostname
+      --tls-cert=FILE           load TLS certificate chain from a file in the PEM format
+      --tls-cert-key=FILE       load TLS certificate private key from a file in the PEM format
+      --tls-dhparam=FILE        load DH parameters for DHE ciphers from a file in the PEM format
+      --tls-protocols=N         enable specified TLS protocols, example: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
+      --tls-ciphers=S           set list of available ciphers (TLSv1.2 and below)
+      --tls-ciphersuites=S      set list of available TLSv1.3 ciphersuites
+
 Logging:
   -S, --syslog                  use system log for output messages
   -l, --log-file=FILE           log all output to a file
       --print-time=N            print hashrate report every N seconds
       --health-print-time=N     print health report every N seconds
       --no-color                disable colored output
+      --verbose                 verbose output
 
 Misc:
   -c, --config=FILE             load a JSON-format configuration file

bin/WinRing0/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2007-2009 OpenLibSys.org. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

cmake/OpenSSL.cmake

@@ -6,16 +6,37 @@ if (WITH_TLS)
         set(OPENSSL_MSVC_STATIC_RT TRUE)
 
         set(EXTRA_LIBS ${EXTRA_LIBS} Crypt32)
+    elseif (APPLE)
+        set(OPENSSL_USE_STATIC_LIBS TRUE)
     endif()
 
     find_package(OpenSSL)
 
     if (OPENSSL_FOUND)
-        set(TLS_SOURCES src/base/net/stratum/Tls.h src/base/net/stratum/Tls.cpp)
+        set(TLS_SOURCES
+            src/base/net/stratum/Tls.cpp
+            src/base/net/stratum/Tls.h
+            src/base/net/tls/ServerTls.cpp
+            src/base/net/tls/ServerTls.h
+            src/base/net/tls/TlsConfig.cpp
+            src/base/net/tls/TlsConfig.h
+            src/base/net/tls/TlsContext.cpp
+            src/base/net/tls/TlsContext.h
+            src/base/net/tls/TlsGen.cpp
+            src/base/net/tls/TlsGen.h
+            )
+
         include_directories(${OPENSSL_INCLUDE_DIR})
 
         if (WITH_HTTP)
-            set(TLS_SOURCES ${TLS_SOURCES} src/base/net/http/HttpsClient.h src/base/net/http/HttpsClient.cpp)
+            set(TLS_SOURCES ${TLS_SOURCES}
+                src/base/net/https/HttpsClient.cpp
+                src/base/net/https/HttpsClient.h
+                src/base/net/https/HttpsContext.cpp
+                src/base/net/https/HttpsContext.h
+                src/base/net/https/HttpsServer.cpp
+                src/base/net/https/HttpsServer.h
+                )
         endif()
     else()
         message(FATAL_ERROR "OpenSSL NOT found: use `-DWITH_TLS=OFF` to build without TLS support")
@@ -27,5 +48,12 @@ else()
     set(OPENSSL_LIBRARIES "")
     remove_definitions(/DXMRIG_FEATURE_TLS)
 
+    if (WITH_HTTP)
+        set(TLS_SOURCES ${TLS_SOURCES}
+            src/base/net/http/HttpServer.cpp
+            src/base/net/http/HttpServer.h
+            )
+    endif()
+
     set(CMAKE_PROJECT_NAME "${CMAKE_PROJECT_NAME}-notls")
 endif()

cmake/astrobwt.cmake

@@ -0,0 +1,47 @@
+if (WITH_ASTROBWT)
+    add_definitions(/DXMRIG_ALGO_ASTROBWT)
+
+    list(APPEND HEADERS_CRYPTO
+        src/crypto/astrobwt/AstroBWT.h
+        src/crypto/astrobwt/sha3.h
+    )
+
+    list(APPEND SOURCES_CRYPTO
+        src/crypto/astrobwt/AstroBWT.cpp
+        src/crypto/astrobwt/sha3.cpp
+    )
+
+    if (XMRIG_ARM)
+        list(APPEND HEADERS_CRYPTO
+            src/crypto/astrobwt/salsa20_ref/ecrypt-config.h
+            src/crypto/astrobwt/salsa20_ref/ecrypt-machine.h
+            src/crypto/astrobwt/salsa20_ref/ecrypt-portable.h
+            src/crypto/astrobwt/salsa20_ref/ecrypt-sync.h
+        )
+
+        list(APPEND SOURCES_CRYPTO
+            src/crypto/astrobwt/salsa20_ref/salsa20.c
+        )
+    else()
+        if (CMAKE_SIZEOF_VOID_P EQUAL 8)
+            add_definitions(/DASTROBWT_AVX2)
+            if (CMAKE_C_COMPILER_ID MATCHES MSVC)
+                enable_language(ASM_MASM)
+                list(APPEND SOURCES_CRYPTO src/crypto/astrobwt/sha3_256_avx2.asm)
+            else()
+                enable_language(ASM)
+                list(APPEND SOURCES_CRYPTO src/crypto/astrobwt/sha3_256_avx2.S)
+            endif()
+        endif()
+
+        list(APPEND HEADERS_CRYPTO
+            src/crypto/astrobwt/Salsa20.hpp
+        )
+
+        list(APPEND SOURCES_CRYPTO
+            src/crypto/astrobwt/Salsa20.cpp
+        )
+    endif()
+else()
+    remove_definitions(/DXMRIG_ALGO_ASTROBWT)
+endif()

cmake/flags.cmake

@@ -57,9 +57,9 @@ if (CMAKE_CXX_COMPILER_ID MATCHES GNU)
     add_definitions(/DHAVE_BUILTIN_CLEAR_CACHE)
 
 elseif (CMAKE_CXX_COMPILER_ID MATCHES MSVC)
+    set(CMAKE_C_FLAGS_RELEASE "/MT /O2 /Oi /DNDEBUG /GL")
+    set(CMAKE_CXX_FLAGS_RELEASE "/MT /O2 /Oi /DNDEBUG /GL")
 
-    set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} /Ox /Ot /Oi /MT /GL")
-    set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /Ox /Ot /Oi /MT /GL")
     add_definitions(/D_CRT_SECURE_NO_WARNINGS)
     add_definitions(/D_CRT_NONSTDC_NO_WARNINGS)
     add_definitions(/DNOMINMAX)

cmake/os.cmake

@@ -0,0 +1,45 @@
+if (WIN32)
+    set(XMRIG_OS_WIN ON)
+elseif (APPLE)
+    set(XMRIG_OS_APPLE ON)
+
+    if (IOS OR CMAKE_SYSTEM_NAME STREQUAL iOS)
+        set(XMRIG_OS_IOS ON)
+    else()
+        set(XMRIG_OS_MACOS ON)
+    endif()
+else()
+    set(XMRIG_OS_UNIX ON)
+
+    if (ANDROID OR CMAKE_SYSTEM_NAME MATCHES "Android")
+        set(XMRIG_OS_ANDROID ON)
+    elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
+        set(XMRIG_OS_LINUX ON)
+    elseif(CMAKE_SYSTEM_NAME STREQUAL FreeBSD)
+        set(XMRIG_OS_FREEBSD ON)
+    endif()
+endif()
+
+
+if (XMRIG_OS_WIN)
+    add_definitions(/DWIN32)
+    add_definitions(/DXMRIG_OS_WIN)
+elseif(XMRIG_OS_APPLE)
+    add_definitions(/DXMRIG_OS_APPLE)
+
+    if (XMRIG_OS_IOS)
+        add_definitions(/DXMRIG_OS_IOS)
+    else()
+        add_definitions(/DXMRIG_OS_MACOS)
+    endif()
+elseif(XMRIG_OS_UNIX)
+    add_definitions(/DXMRIG_OS_UNIX)
+
+    if (XMRIG_OS_ANDROID)
+        add_definitions(/DXMRIG_OS_ANDROID)
+    elseif (XMRIG_OS_LINUX)
+        add_definitions(/DXMRIG_OS_LINUX)
+    elseif (XMRIG_OS_FREEBSD)
+        add_definitions(/DXMRIG_OS_FREEBSD)
+    endif()
+endif()

cmake/randomx.cmake

@@ -75,13 +75,27 @@ if (WITH_RANDOMX)
             )
 
         list(APPEND SOURCES_CRYPTO
-             src/crypto/rx/RxConfig_hwloc.cpp
              src/crypto/rx/RxNUMAStorage.cpp
             )
+    endif()
+
+    if (WITH_MSR AND NOT XMRIG_ARM AND CMAKE_SIZEOF_VOID_P EQUAL 8 AND (XMRIG_OS_WIN OR XMRIG_OS_LINUX))
+        add_definitions(/DXMRIG_FEATURE_MSR)
+        add_definitions(/DXMRIG_FIX_RYZEN)
+        message("-- WITH_MSR=ON")
+
+        if (XMRIG_OS_WIN)
+            list(APPEND SOURCES_CRYPTO src/crypto/rx/Rx_win.cpp)
+        elseif (XMRIG_OS_LINUX)
+            list(APPEND SOURCES_CRYPTO src/crypto/rx/Rx_linux.cpp)
+        endif()
+
+        list(APPEND HEADERS_CRYPTO src/crypto/rx/msr/MsrItem.h)
+        list(APPEND SOURCES_CRYPTO src/crypto/rx/msr/MsrItem.cpp)
     else()
-        list(APPEND SOURCES_CRYPTO
+        remove_definitions(/DXMRIG_FEATURE_MSR)
-             src/crypto/rx/RxConfig_basic.cpp
+        remove_definitions(/DXMRIG_FIX_RYZEN)
-            )
+        message("-- WITH_MSR=OFF")
     endif()
 else()
     remove_definitions(/DXMRIG_ALGO_RANDOMX)

doc/ALGORITHMS.md

@@ -12,6 +12,8 @@ Option `coin` useful for pools without algorithm negotiation support or daemon t
 
 | Name | Memory | Version | Notes |
 |------|--------|---------|-------|
+| `rx/sfx` | 2 MB | 5.4.0+ | RandomSFX (RandomX variant for Safex). |
+| `rx/v` | 2 MB | 5.4.0+ | RandomV (RandomX variant for new MoneroV). |
 | `rx/arq` | 256 KB | 4.3.0+ | RandomARQ (RandomX variant for ArQmA). |
 | `rx/0` | 2 MB | 3.2.0+ | RandomX (Monero). |
 | `argon2/chukwa` | 512 KB | 3.1.0+ | Argon2id (Chukwa). |
@@ -23,7 +25,6 @@ Option `coin` useful for pools without algorithm negotiation support or daemon t
 | `cn/zls` | 2 MB | 2.14.0+ | CryptoNight variant 2 with 3/4 iterations. |
 | `cn/double` | 2 MB | 2.14.0+ | CryptoNight variant 2 with double iterations. |
 | `cn/r` | 2 MB | 2.13.0+ | CryptoNightR (Monero's variant 4). |
-| `cn/wow` | 2 MB | 2.12.0+ | CryptoNightR (Wownero). |
 | `cn/gpu` | 2 MB | 2.11.0+ | CryptoNight-GPU. |
 | `cn-pico` | 256 KB | 2.10.0+ | CryptoNight-Pico. |
 | `cn/half` | 2 MB | 2.9.0+ | CryptoNight variant 2 with half iterations. |

doc/CHANGELOG_OLD.md

@@ -1,3 +1,116 @@
+# v4.6.2-beta
+- [#1274](https://github.com/xmrig/xmrig/issues/1274) Added `--cuda-devices` command line option.
+- [#1277](https://github.com/xmrig/xmrig/pull/1277) Fixed function names for clang on Apple.
+
+# v4.6.1-beta
+- [#1272](https://github.com/xmrig/xmrig/pull/1272) Optimized hashrate calculation.
+- [#1273](https://github.com/xmrig/xmrig/issues/1273) Fixed crash when use `GET /2/backends` API endpoint with disabled CUDA.
+
+# v4.6.0-beta
+- [#1263](https://github.com/xmrig/xmrig/pull/1263) Added new option `dataset_host` for NVIDIA GPUs with less than 4 GB memory (RandomX only).
+
+# v4.5.0-beta
+- Added NVIDIA CUDA support via external [CUDA plugun](https://github.com/xmrig/xmrig-cuda). XMRig now is unified 3 in 1 miner.
+
+# v4.4.0-beta
+- [#1068](https://github.com/xmrig/xmrig/pull/1068) Added support for `self-select` stratum protocol extension.
+- [#1240](https://github.com/xmrig/xmrig/pull/1240) Sync with the latest RandomX code.
+- [#1241](https://github.com/xmrig/xmrig/issues/1241) Fixed regression with colors on old Windows systems.
+- [#1243](https://github.com/xmrig/xmrig/pull/1243) Fixed incorrect OpenCL memory size detection in some cases.
+- [#1247](https://github.com/xmrig/xmrig/pull/1247) Fixed ARM64 RandomX code alignment.
+- [#1248](https://github.com/xmrig/xmrig/pull/1248) Fixed RandomX code cache cleanup on iOS/Darwin.
+
+# v4.3.1-beta
+- Fixed regression in v4.3.0, miner didn't create `cn` mining profile with default config example.
+
+# v4.3.0-beta
+- [#1227](https://github.com/xmrig/xmrig/pull/1227) Added new algorithm `rx/arq`, RandomX variant for upcoming ArQmA fork.
+- [#808](https://github.com/xmrig/xmrig/issues/808#issuecomment-539297156) Added experimental support for persistent memory for CPU mining threads.
+- [#1221](https://github.com/xmrig/xmrig/issues/1221) Improved RandomX dataset memory usage and initialization speed for NUMA machines.
+
+# v4.2.1-beta
+- [#1150](https://github.com/xmrig/xmrig/issues/1150) Fixed build on FreeBSD.
+- [#1175](https://github.com/xmrig/xmrig/issues/1175) Fixed support for systems where total count of NUMA nodes not equal usable nodes count.
+- [#1199](https://github.com/xmrig/xmrig/issues/1199) Fixed excessive memory allocation for OpenCL threads with low intensity.
+- [#1212](https://github.com/xmrig/xmrig/issues/1212) Fixed low RandomX performance after fast algorithm switching.
+
+# v4.2.0-beta
+- [#1202](https://github.com/xmrig/xmrig/issues/1202) Fixed algorithm verification in donate strategy.
+- Added per pool option `coin` with single possible value `monero` for pools without algorithm negotiation, for upcoming Monero fork.
+- Added config option `cpu/max-threads-hint` and command line option `--cpu-max-threads-hint`.
+
+# v4.1.0-beta
+- **OpenCL backend disabled by default.**.
+- [#1183](https://github.com/xmrig/xmrig/issues/1183) Fixed compatibility with systemd.
+- [#1185](https://github.com/xmrig/xmrig/pull/1185) Added JIT compiler for RandomX on ARMv8.
+- Improved API endpoint `GET /2/backends` and added support for this endpoint to [workers.xmrig.info](http://workers.xmrig.info).
+- Added command line option `--no-cpu` to disable CPU backend.
+- Added OpenCL specific command line options: `--opencl`, `--opencl-devices`, `--opencl-platform`, `--opencl-loader` and `--opencl-no-cache`.
+- Removed command line option `--http-enabled`, HTTP API enabled automatically if any other `--http-*` option provided.
+
+# v4.0.1-beta
+- [#1177](https://github.com/xmrig/xmrig/issues/1177) Fixed compatibility with old AMD drivers.
+- [#1180](https://github.com/xmrig/xmrig/issues/1180) Fixed possible duplicated shares after algorithm switching.
+- Added support for case if not all backend threads successfully started.
+- Fixed wrong config file permissions after write (only gcc builds on recent Windows 10 affected).
+
+# v4.0.0-beta
+- [#1172](https://github.com/xmrig/xmrig/issues/1172) **Added OpenCL mining backend.**
+  - [#268](https://github.com/xmrig/xmrig-amd/pull/268) [#270](https://github.com/xmrig/xmrig-amd/pull/270) [#271](https://github.com/xmrig/xmrig-amd/pull/271) [#273](https://github.com/xmrig/xmrig-amd/pull/273) [#274](https://github.com/xmrig/xmrig-amd/pull/274) [#1171](https://github.com/xmrig/xmrig/pull/1171) Added RandomX support for OpenCL, thanks [@SChernykh](https://github.com/SChernykh).
+- Algorithm `cn/wow` removed, as no longer alive. 
+
+# v3.2.0
+- Added per pool option `coin` with single possible value `monero` for pools without algorithm negotiation, for upcoming Monero fork.
+- [#1183](https://github.com/xmrig/xmrig/issues/1183) Fixed compatibility with systemd.
+
+# v3.1.3
+- [#1180](https://github.com/xmrig/xmrig/issues/1180) Fixed possible duplicated shares after algorithm switching.
+- Fixed wrong config file permissions after write (only gcc builds on recent Windows 10 affected).
+
+# v3.1.2
+- Many RandomX optimizations and fixes.
+  - [#1132](https://github.com/xmrig/xmrig/issues/1132) Fixed build on CentOS 7.
+  - [#1163](https://github.com/xmrig/xmrig/pull/1163) Optimized soft AES code, up to +30% hashrate on CPU without AES support and other optimizations.
+  - [#1166](https://github.com/xmrig/xmrig/pull/1166) Fixed crash when initialize dataset with big threads count (eg 272).
+  - [#1168](https://github.com/xmrig/xmrig/pull/1168) Optimized loading from scratchpad.
+- [#1128](https://github.com/xmrig/xmrig/issues/1128) Fixed CMake 2.8 compatibility.
+
+# v3.1.1
+- [#1133](https://github.com/xmrig/xmrig/issues/1133) Fixed syslog regression.
+- [#1138](https://github.com/xmrig/xmrig/issues/1138) Fixed multiple network bugs.
+- [#1141](https://github.com/xmrig/xmrig/issues/1141) Fixed log in background mode.
+- [#1142](https://github.com/xmrig/xmrig/pull/1142) RandomX hashrate improved by 0.5-1.5% depending on variant and CPU.
+- [#1146](https://github.com/xmrig/xmrig/pull/1146) Fixed race condition in RandomX thread init.
+- [#1148](https://github.com/xmrig/xmrig/pull/1148) Fixed, on Linux linker marking entire executable as having an executable stack.
+- Fixed, for Argon2 algorithms command line options like `--threads` was ignored.
+- Fixed command line options for single pool, free order allowed again.
+
+# v3.1.0
+- [#1107](https://github.com/xmrig/xmrig/issues/1107#issuecomment-522235892) Added Argon2 algorithm family: `argon2/chukwa` and `argon2/wrkz`.
+
+# v3.0.0
+- **[#1111](https://github.com/xmrig/xmrig/pull/1111) Added RandomX (`rx/test`) algorithm for testing and benchmarking.**
+- **[#1036](https://github.com/xmrig/xmrig/pull/1036) Added RandomWOW (`rx/wow`) algorithm for [Wownero](http://wownero.org/).**
+- **[#1050](https://github.com/xmrig/xmrig/pull/1050) Added RandomXL (`rx/loki`) algorithm for [Loki](https://loki.network/).**
+- **[#1077](https://github.com/xmrig/xmrig/issues/1077) Added NUMA support via hwloc**.
+- **Added flexible [multi algorithm](doc/CPU.md) configuration.**
+- **Added unlimited switching between incompatible algorithms, all mining options can be changed in runtime.**
+- [#257](https://github.com/xmrig/xmrig-nvidia/pull/257) New logging subsystem, file and syslog now always without colors.
+- [#314](https://github.com/xmrig/xmrig-proxy/issues/314) Added donate over proxy feature.
+- [#1007](https://github.com/xmrig/xmrig/issues/1007) Old HTTP API backend based on libmicrohttpd, replaced to custom HTTP server (libuv + http_parser).
+- [#1010](https://github.com/xmrig/xmrig/pull/1010#issuecomment-482632107) Added daemon support (solo mining).
+- [#1066](https://github.com/xmrig/xmrig/issues/1066#issuecomment-518080529) Added error message if pool not ready for RandomX.
+- [#1105](https://github.com/xmrig/xmrig/issues/1105) Improved auto configuration for `cn-pico` algorithm.
+- Added commands `pause` and `resume` via JSON RPC 2.0 API (`POST /json_rpc`).
+- Added command line option `--export-topology` for export hwloc topology to a XML file.
+- Breaked backward compatibility with previous configs and command line, `variant` option replaced to `algo`, global option `algo` removed, all CPU related settings moved to `cpu` object.
+- Options `av`, `safe` and `max-cpu-usage` removed.
+- Algorithm `cn/msr` renamed to `cn/fast`.
+- Algorithm `cn/xtl` removed.
+- API endpoint `GET /1/threads` replaced to `GET /2/backends`.
+- Added global uptime and extended connection information in API.
+- API now return current algorithm.
+
 # v2.99.6-beta
 - Added commands `pause` and `resume` via JSON RPC 2.0 API (`POST /json_rpc`).
 - Fixed autoconfig regression (since 2.99.5), mostly `rx/wow` was affected by this bug.

doc/CPU.md

@@ -99,4 +99,7 @@ Allow override automatically detected Argon2 implementation, this option added m
 Maximum CPU threads count (in percentage) hint for autoconfig. [CPU_MAX_USAGE.md](CPU_MAX_USAGE.md)
 
 #### `memory-pool` (since v4.3.0)
-Use continuous, persistent memory block for mining threads, useful for preserve huge pages allocation while algorithm swithing. Default value `false` (feature disabled) or `true` or specific count of 2 MB huge pages.
+Use continuous, persistent memory block for mining threads, useful for preserve huge pages allocation while algorithm swithing. Possible values `false` (feature disabled, by default) or `true` or specific count of 2 MB huge pages.
+
+#### `yield` (since v5.1.1)
+Prefer system better system response/stability `true` (default value) or maximum hashrate `false`.

doc/build/CMAKE_OPTIONS.md

@@ -1,5 +1,5 @@
 # CMake options
-This document contains list of useful cmake options.
+**Recent version of this document: https://xmrig.com/docs/miner/cmake-options**
 
 ## Algorithms
 
@@ -32,7 +32,7 @@ This feature add external dependency to libhwloc (1.10.0+) (except MSVC builds).
 
 ## Special build options
 
-* **`-DXMRIG_DEPS=<path>`** path to precompiled dependensices https://github.com/xmrig/xmrig-deps
+* **`-DXMRIG_DEPS=<path>`** path to precompiled dependencies https://github.com/xmrig/xmrig-deps
 * **`-DARM_TARGET=<number>`** override ARM target, possible values `7` (ARMv7) and `8` (ARMv8).
 * **`-DUV_INCLUDE_DIR=<path>`** custom path to libuv headers.
 * **`-DUV_LIBRARY=<path>`** custom path to libuv library.

doc/gpg_keys/xmrig.asc

@@ -0,0 +1,30 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQENBF3VSRIBCADfFjDUbq0WLGulFeSou0A+jTvweNllPyLNOn3SNCC0XLEYyEcu
+JiEBK80DlvR06TVr8Aw1rT5S2iH0i5Tl8DqShH2mmcN1rBp1M0Y95D89KVj3BIhE
+nxmgmD4N3Wgm+5FmEH4W/RpG1xdYWJx3eJhtWPdFJqpg083E2D5P30wIQem+EnTR
+5YrtTZPh5cPj2KRY+UmsDE3ahmxCgP7LYgnnpZQlWBBiMV932s7MvYBPJQc1wecS
+0wi1zxyS81xHc3839EkA7wueCeNo+5jha+KH66tMKsfrI2WvfPHTCPjK9v7WJc/O
+/eRp9d+wacn09D1L6CoRO0ers5p10GO84VhTABEBAAG0GVhNUmlnIDxzdXBwb3J0
+QHhtcmlnLmNvbT6JAU4EEwEIADgWIQSaxM6o5m41pcfN3BtEalNji+lECQUCXdVJ
+EgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBEalNji+lECbkQB/9nRou0
+tOlBwYn8xVgBu7IiDWNVETRWfrjrtdTvSahgbbo6lWgjA/vBLkjN9fISdBQ/n/Mt
+hNDJbEtxHHt2baJhvT8du1eWcIHHXCV/rmv+iY/hTXa1gKqHiHDJrtYSVBG3BMme
+1rdsUHTiKf3t5yRHOXAfY2C+XNblKAV7mhlxQBiKxdFDIkFEQKNrHNUvnzkOqoCT
+2kTZZ2tPUMQdOn1eek6zG/+C7SwcBpJnakJ8jce4yA/xZbOVKetNWO3Ufu3TE34k
+OdA+H4PU9+fV77XfOY8DtXeS3boUI97ei+4s/mwX/NFC0i8CPXyefxl3WRUBGDOI
+w//kPNQVh4HobOCeuQENBF3VSRIBCADl29WorEi+vRA/3kg9VUXtxSU6caibFS3N
+VXANiFRjrOmICdfrIgOSGNrYCQFsXu0Xe0udDYVX8yX6WJk+CT02Pdg0gkXiKoze
+KrnK15mo3xXbb2tr1o9ROPgwY/o2AwQHj0o1JhdS2cybfuRiUQRoGgBX7a9X0cTY
+r4ZJvOjzgAajl3ciwB3yWUmDiRlzZpO7YWESXbOhGVzyCnP5MlMEJ/fPRw9h38vK
+HNKLhzcRfsLpXk34ghY3SxIv4NWUfuZXFWqpSdC9JgNc5zA72lJEQcF4DHJCKl7B
+ddmrfsr9mdiIpo+/ZZFPPngdeZ2kvkJ2YKaZNVu2XooJARPQ8B8tABEBAAGJATYE
+GAEIACAWIQSaxM6o5m41pcfN3BtEalNji+lECQUCXdVJEgIbDAAKCRBEalNji+lE
+CdPUB/4nH1IdhHGmfko2kxdaHqQgCGLqh3pcrQXD9mBv/LYVnoHZpVRHsIDgg2Z4
+lQYrIRRqe69FjVxo7sA2eMIlV0GRDlUrw+HeURFpEhKPEdwFy6i/cti2MY0YxOrB
+TvQoRutUoMnyjM4TBJWaaqccbTsavMdLmG3JHdAkiHtUis/fUwVctmEQwN+d/J2b
+wJAtliqw3nXchUfdIfwHF/7hg8seUuYUaifzkazBZhVWvRkTVLVanzZ51HRfuzwD
+ntaa7kfYGdE+4TKOylAPh+8E6WnR19RRTpsaW0dVBgOiBTE0uc7rUv2HWS/u6RUR
+t7ldSBzkuDTlM2V59Iq2hXoSC6dT
+=cIG9
+-----END PGP PUBLIC KEY BLOCK-----

doc/releases/5_0_1/SHA256SUMS

@@ -0,0 +1,5 @@
+6bb1a2e3a0fbca5195be6022f2a9fbff8a353c37c7542e7ab89420cb45b64505  xmrig-5.0.1-gcc-win32.zip
+24dba9ec281acfb2ea2c401ebd0e4e2d1f1ee5fd557da5ff3c7049020c1f78b6  xmrig-5.0.1-gcc-win64.zip
+86d65c6693ec9e35cd7547329580638b85c9eb0cf8383892a1c15199de5b556f  xmrig-5.0.1-msvc-cuda10_1-win64.zip
+0fbfe518b1c4b6993b0f66ff01302626375b15620ccf8f64d6fb97845068ffca  xmrig-5.0.1-msvc-win64.zip
+aa34890738a3494de2fa0e44db346937fea7339852f5f10b5d4655f95e2d8f1f  xmrig-5.0.1-xenial-x64.tar.gz

doc/releases/5_0_1/SHA256SUMS.sig

@@ -0,0 +1,11 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQEzBAABCgAdFiEEmsTOqOZuNaXHzdwbRGpTY4vpRAkFAl3VcsoACgkQRGpTY4vp
+RAm9vQgA1MyTUU2jley2TCYLUzQy2Fffc8fbXYv64r44jbWOjC/6qo2iIlRgPhIc
+oVyPKr5TYS3QjDzCEm8IvozS0YudS6soESbPzqDonboK8pd0K4bsML9TQY2feV7A
+NL5vln0rfVHp1wxLLrQpfBqAgvJUXEyaHece6gFQN79JOGhEo2bHL2NyrOl+FViS
+b2BaMtXq410Fh+XT6ShnOaG/2EuO8ZqSGdCO6A/2LHQw1UY+mZiCvue6P6B06HmB
+WD/urOv38V389v+V+Sp4UlEW6VpBOOjvtChoVWtLt+tKzydrnt2EmoWWWg475pka
+4G6whHuMWS8CTt5/PDhJpvVXNQTIOw==
+=C764
+-----END PGP SIGNATURE-----

scripts/build_deps.sh

@@ -0,0 +1,43 @@
+#!/bin/bash -e
+
+UV_VERSION="1.34.0"
+OPENSSL_VERSION="1.1.1d"
+HWLOC_VERSION="2.1.0"
+
+mkdir deps
+mkdir deps/include
+mkdir deps/lib
+
+mkdir build && cd build
+
+wget https://github.com/libuv/libuv/archive/v${UV_VERSION}.tar.gz
+tar -xzf v${UV_VERSION}.tar.gz
+
+wget https://download.open-mpi.org/release/hwloc/v2.1/hwloc-${HWLOC_VERSION}.tar.bz2
+tar -xjf hwloc-${HWLOC_VERSION}.tar.bz2
+
+wget https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz
+tar -xzf openssl-${OPENSSL_VERSION}.tar.gz
+
+cd libuv-${UV_VERSION}
+sh autogen.sh
+./configure --disable-shared
+make -j$(nproc)
+cp -fr include/ ../../deps
+cp .libs/libuv.a ../../deps/lib
+cd ..
+
+cd hwloc-${HWLOC_VERSION}
+./configure --disable-shared --enable-static --disable-io --disable-libudev --disable-libxml2
+make -j$(nproc)
+cp -fr include/ ../../deps
+cp hwloc/.libs/libhwloc.a ../../deps/lib
+cd ..
+
+cd openssl-${OPENSSL_VERSION}
+./config -no-shared -no-asm -no-zlib -no-comp -no-dgram -no-filenames -no-cms
+make -j$(nproc)
+cp -fr include/ ../../deps
+cp libcrypto.a ../../deps/lib
+cp libssl.a ../../deps/lib
+cd ../..

scripts/enable_1gb_pages.sh

@@ -0,0 +1,12 @@
+#!/bin/bash -e
+
+# https://xmrig.com/docs/miner/hugepages#onegb-huge-pages
+
+sysctl -w vm.nr_hugepages=$(nproc)
+
+for i in $(find /sys/devices/system/node/node* -maxdepth 0 -type d);
+do
+    echo 3 > "$i/hugepages/hugepages-1048576kB/nr_hugepages";
+done
+
+echo "1GB pages successfully enabled"

scripts/generate_cl.js

@@ -60,6 +60,7 @@ function rx()
         'randomx_constants_wow.h',
         'randomx_constants_loki.h',
         'randomx_constants_arqma.h',
+        'randomx_constants_keva.h',
         'aes.cl',
         'blake2b.cl',
         'randomx_vm.cl',
@@ -75,6 +76,15 @@ function rx()
 }
 
 
+function astrobwt()
+{
+    const astrobwt = opencl_minify(addIncludes('astrobwt.cl', [ 'BWT.cl', 'salsa20.cl', 'sha3.cl' ]));
+
+    // fs.writeFileSync('astrobwt_gen.cl', astrobwt);
+    fs.writeFileSync('astrobwt_cl.h', text2h(astrobwt, 'xmrig', 'astrobwt_cl'));
+}
+
+
 process.chdir(path.resolve('src/backend/opencl/cl/cn'));
 
 cn();
@@ -84,4 +94,9 @@ cn_gpu();
 process.chdir(cwd);
 process.chdir(path.resolve('src/backend/opencl/cl/rx'));
 
-rx();
+rx();
+
+process.chdir(cwd);
+process.chdir(path.resolve('src/backend/opencl/cl/astrobwt'));
+
+astrobwt();

scripts/js/opencl_minify.js

@@ -2,7 +2,8 @@
 
 function opencl_minify(input)
 {
-    let out = input.replace(/\/\*[\s\S]*?\*\/|\/\/.*$/gm, '');  // comments
+    let out = input.replace(/\r/g, '');
+    out = out.replace(/\/\*[\s\S]*?\*\/|\/\/.*$/gm, ''); // comments
     out = out.replace(/^#\s+/gm, '#');        // macros with spaces
     out = out.replace(/\n{2,}/g, '\n');       // empty lines
     out = out.replace(/^\s+/gm, '');          // leading whitespace
@@ -10,7 +11,7 @@ function opencl_minify(input)
 
     let array = out.split('\n').map(line => {
         if (line[0] === '#') {
-            return line
+            return line;
         }
 
         line = line.replace(/, /g, ',');

scripts/randomx_boost.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+modprobe msr
+
+if cat /proc/cpuinfo | grep "AMD Ryzen" > /dev/null;
+	then
+		echo "Detected Ryzen"
+		wrmsr -a 0xc0011022 0x510000
+		wrmsr -a 0xc001102b 0x1808cc16
+		wrmsr -a 0xc0011020 0
+		wrmsr -a 0xc0011021 0x40
+		echo "MSR register values for Ryzen applied"
+elif cat /proc/cpuinfo | grep "Intel" > /dev/null;
+	then
+		echo "Detected Intel"
+		wrmsr -a 0x1a4 0xf
+		echo "MSR register values for Intel applied"
+else
+	echo "No supported CPU detected"
+fi

src/3rdparty/adl/adl_sdk.h

@@ -0,0 +1,44 @@
+//
+// Copyright (c) 2016 Advanced Micro Devices, Inc. All rights reserved.
+//
+// MIT LICENSE:
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+/// \file adl_sdk.h
+/// \brief Contains the definition of the Memory Allocation Callback.\n <b>Included in ADL SDK</b>
+///
+/// \n\n
+/// This file contains the definition of the Memory Allocation Callback.\n
+/// It also includes definitions of the respective structures and constants.\n
+/// <b> This is the only header file to be included in a C/C++ project using ADL </b>
+
+#ifndef ADL_SDK_H_
+#define ADL_SDK_H_
+
+#include "adl_structures.h"
+
+#if defined (LINUX)
+#define __stdcall
+#endif /* (LINUX) */
+
+/// Memory Allocation Call back 
+typedef void* ( __stdcall *ADL_MAIN_MALLOC_CALLBACK )( int );
+
+
+#endif /* ADL_SDK_H_ */

src/3rdparty/http-parser/http_parser.c

@@ -381,7 +381,10 @@ enum header_states
   , h_transfer_encoding
   , h_upgrade
 
+  , h_matching_transfer_encoding_token_start
   , h_matching_transfer_encoding_chunked
+  , h_matching_transfer_encoding_token
+
   , h_matching_connection_token_start
   , h_matching_connection_keep_alive
   , h_matching_connection_close
@@ -1257,9 +1260,9 @@ reexecute:
 
           switch (parser->header_state) {
             case h_general: {
-              size_t limit = data + len - p;
+              size_t left = data + len - p;
-              limit = MIN(limit, max_header_size);
+              const char* pe = p + MIN(left, max_header_size);
-              while (p+1 < data + limit && TOKEN(p[1])) {
+              while (p+1 < pe && TOKEN(p[1])) {
                 p++;
               }
               break;
@@ -1335,6 +1338,7 @@ reexecute:
                 parser->header_state = h_general;
               } else if (parser->index == sizeof(TRANSFER_ENCODING)-2) {
                 parser->header_state = h_transfer_encoding;
+                parser->flags |= F_TRANSFER_ENCODING;
               }
               break;
 
@@ -1416,10 +1420,14 @@ reexecute:
             if ('c' == c) {
               parser->header_state = h_matching_transfer_encoding_chunked;
             } else {
-              parser->header_state = h_general;
+              parser->header_state = h_matching_transfer_encoding_token;
             }
             break;
 
+          /* Multi-value `Transfer-Encoding` header */
+          case h_matching_transfer_encoding_token_start:
+            break;
+
           case h_content_length:
             if (UNLIKELY(!IS_NUM(ch))) {
               SET_ERRNO(HPE_INVALID_CONTENT_LENGTH);
@@ -1496,28 +1504,25 @@ reexecute:
 
           switch (h_state) {
             case h_general:
-            {
+              {
-              const char* p_cr;
+                size_t left = data + len - p;
-              const char* p_lf;
+                const char* pe = p + MIN(left, max_header_size);
-              size_t limit = data + len - p;
 
-              limit = MIN(limit, max_header_size);
+                for (; p != pe; p++) {
-
+                  ch = *p;
-              p_cr = (const char*) memchr(p, CR, limit);
+                  if (ch == CR || ch == LF) {
-              p_lf = (const char*) memchr(p, LF, limit);
+                    --p;
-              if (p_cr != NULL) {
+                    break;
-                if (p_lf != NULL && p_cr >= p_lf)
+                  }
-                  p = p_lf;
+                  if (!lenient && !IS_HEADER_CHAR(ch)) {
-                else
+                    SET_ERRNO(HPE_INVALID_HEADER_TOKEN);
-                  p = p_cr;
+                    goto error;
-              } else if (UNLIKELY(p_lf != NULL)) {
+                  }
-                p = p_lf;
+                }
-              } else {
+                if (p == data + len)
-                p = data + len;
+                  --p;
+                break;
               }
-              --p;
-              break;
-            }
 
             case h_connection:
             case h_transfer_encoding:
@@ -1566,16 +1571,41 @@ reexecute:
               goto error;
 
             /* Transfer-Encoding: chunked */
+            case h_matching_transfer_encoding_token_start:
+              /* looking for 'Transfer-Encoding: chunked' */
+              if ('c' == c) {
+                h_state = h_matching_transfer_encoding_chunked;
+              } else if (STRICT_TOKEN(c)) {
+                /* TODO(indutny): similar code below does this, but why?
+                 * At the very least it seems to be inconsistent given that
+                 * h_matching_transfer_encoding_token does not check for
+                 * `STRICT_TOKEN`
+                 */
+                h_state = h_matching_transfer_encoding_token;
+              } else if (c == ' ' || c == '\t') {
+                /* Skip lws */
+              } else {
+                h_state = h_general;
+              }
+              break;
+
             case h_matching_transfer_encoding_chunked:
               parser->index++;
               if (parser->index > sizeof(CHUNKED)-1
                   || c != CHUNKED[parser->index]) {
-                h_state = h_general;
+                h_state = h_matching_transfer_encoding_token;
               } else if (parser->index == sizeof(CHUNKED)-2) {
                 h_state = h_transfer_encoding_chunked;
               }
               break;
 
+            case h_matching_transfer_encoding_token:
+              if (ch == ',') {
+                h_state = h_matching_transfer_encoding_token_start;
+                parser->index = 0;
+              }
+              break;
+
             case h_matching_connection_token_start:
               /* looking for 'Connection: keep-alive' */
               if (c == 'k') {
@@ -1634,7 +1664,7 @@ reexecute:
               break;
 
             case h_transfer_encoding_chunked:
-              if (ch != ' ') h_state = h_general;
+              if (ch != ' ') h_state = h_matching_transfer_encoding_token;
               break;
 
             case h_connection_keep_alive:
@@ -1768,12 +1798,17 @@ reexecute:
           REEXECUTE();
         }
 
-        /* Cannot use chunked encoding and a content-length header together
+        /* Cannot us transfer-encoding and a content-length header together
-           per the HTTP specification. */
+           per the HTTP specification. (RFC 7230 Section 3.3.3) */
-        if ((parser->flags & F_CHUNKED) &&
+        if ((parser->flags & F_TRANSFER_ENCODING) &&
             (parser->flags & F_CONTENTLENGTH)) {
-          SET_ERRNO(HPE_UNEXPECTED_CONTENT_LENGTH);
+          /* Allow it for lenient parsing as long as `Transfer-Encoding` is
-          goto error;
+           * not `chunked`
+           */
+          if (!lenient || (parser->flags & F_CHUNKED)) {
+            SET_ERRNO(HPE_UNEXPECTED_CONTENT_LENGTH);
+            goto error;
+          }
         }
 
         UPDATE_STATE(s_headers_done);
@@ -1848,8 +1883,31 @@ reexecute:
           UPDATE_STATE(NEW_MESSAGE());
           CALLBACK_NOTIFY(message_complete);
         } else if (parser->flags & F_CHUNKED) {
-          /* chunked encoding - ignore Content-Length header */
+          /* chunked encoding - ignore Content-Length header,
+           * prepare for a chunk */
           UPDATE_STATE(s_chunk_size_start);
+        } else if (parser->flags & F_TRANSFER_ENCODING) {
+          if (parser->type == HTTP_REQUEST && !lenient) {
+            /* RFC 7230 3.3.3 */
+
+            /* If a Transfer-Encoding header field
+             * is present in a request and the chunked transfer coding is not
+             * the final encoding, the message body length cannot be determined
+             * reliably; the server MUST respond with the 400 (Bad Request)
+             * status code and then close the connection.
+             */
+            SET_ERRNO(HPE_INVALID_TRANSFER_ENCODING);
+            RETURN(p - data); /* Error */
+          } else {
+            /* RFC 7230 3.3.3 */
+
+            /* If a Transfer-Encoding header field is present in a response and
+             * the chunked transfer coding is not the final encoding, the
+             * message body length is determined by reading the connection until
+             * it is closed by the server.
+             */
+            UPDATE_STATE(s_body_identity_eof);
+          }
         } else {
           if (parser->content_length == 0) {
             /* Content-Length header given but zero: Content-Length: 0\r\n */
@@ -2103,6 +2161,12 @@ http_message_needs_eof (const http_parser *parser)
     return 0;
   }
 
+  /* RFC 7230 3.3.3, see `s_headers_almost_done` */
+  if ((parser->flags & F_TRANSFER_ENCODING) &&
+      (parser->flags & F_CHUNKED) == 0) {
+    return 1;
+  }
+
   if ((parser->flags & F_CHUNKED) || parser->content_length != ULLONG_MAX) {
     return 0;
   }

src/3rdparty/http-parser/http_parser.h

@@ -27,7 +27,7 @@ extern "C" {
 /* Also update SONAME in the Makefile whenever you change these. */
 #define HTTP_PARSER_VERSION_MAJOR 2
 #define HTTP_PARSER_VERSION_MINOR 9
-#define HTTP_PARSER_VERSION_PATCH 0
+#define HTTP_PARSER_VERSION_PATCH 3
 
 #include <stddef.h>
 #if defined(_WIN32) && !defined(__MINGW32__) && \
@@ -225,6 +225,7 @@ enum flags
   , F_UPGRADE               = 1 << 5
   , F_SKIPBODY              = 1 << 6
   , F_CONTENTLENGTH         = 1 << 7
+  , F_TRANSFER_ENCODING     = 1 << 8
   };
 
 
@@ -271,6 +272,8 @@ enum flags
      "unexpected content-length header")                             \
   XX(INVALID_CHUNK_SIZE,                                             \
      "invalid character in chunk size header")                       \
+  XX(INVALID_TRANSFER_ENCODING,                                      \
+     "request has invalid transfer-encoding")                        \
   XX(INVALID_CONSTANT, "invalid constant string")                    \
   XX(INVALID_INTERNAL_STATE, "encountered unexpected internal state")\
   XX(STRICT, "strict mode assertion failed")                         \
@@ -293,11 +296,11 @@ enum http_errno {
 struct http_parser {
   /** PRIVATE **/
   unsigned int type : 2;         /* enum http_parser_type */
-  unsigned int flags : 8;        /* F_* values from 'flags' enum; semi-public */
   unsigned int state : 7;        /* enum state from http_parser.c */
   unsigned int header_state : 7; /* enum header_state from http_parser.c */
   unsigned int index : 7;        /* index into current matcher */
   unsigned int lenient_http_headers : 1;
+  unsigned int flags : 16;       /* F_* values from 'flags' enum; semi-public */
 
   uint32_t nread;          /* # bytes read in various scenarios */
   uint64_t content_length; /* # bytes in body (0 if no Content-Length header) */

src/3rdparty/hwloc/AUTHORS

@@ -21,6 +21,7 @@ Nathalie Furmento            CNRS
 Bryon Gloden
 Brice Goglin                 Inria
 Gilles Gouaillardet          RIST
+Valentin Hoyet               Inria
 Joshua Hursey                UWL
 Alexey Kardashevskiy         IBM
 Rob Latham                   ANL

src/3rdparty/hwloc/CMakeLists.txt

@@ -5,7 +5,7 @@ include_directories(include)
 include_directories(src)
 
 add_definitions(/D_CRT_SECURE_NO_WARNINGS)
-set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} /MT")
+set(CMAKE_C_FLAGS_RELEASE "/MT /O2 /Ob2 /DNDEBUG")
 
 set(HEADERS
     include/hwloc.h

src/3rdparty/hwloc/NEWS

@@ -13,8 +13,96 @@ $HEADER$
 
 This file contains the main features as well as overviews of specific
 bug fixes (and other actions) for each version of hwloc since version
-0.9 (as initially released as "libtopology", then re-branded to "hwloc"
+0.9.
-in v0.9.1).
+
+
+Version 2.1.0
+-------------
+* API
+  + Add a new "Die" object (HWLOC_OBJ_DIE) for upcoming x86 processors
+    with multiple dies per package, in the x86 and Linux backends.
+  + Add the new HWLOC_OBJ_MEMCACHE object type for memory-side caches.
+    - They are filtered-out by default, except in command-line tools.
+    - They are only available on very recent platforms running Linux 5.2+
+      and uptodate ACPI tables.
+    - The KNL MCDRAM in cache mode is still exposed as a L3 unless
+      HWLOC_KNL_MSCACHE_L3=0 in the environment.
+  + Add HWLOC_RESTRICT_FLAG_BYNODESET and _REMOVE_MEMLESS for restricting
+    topologies based on some memory nodes.
+  + Add hwloc_topology_set_components() for blacklisting some components
+    from being enabled in a topology.
+  + Add hwloc_bitmap_nr_ulongs() and hwloc_bitmap_from/to_ulongs(),
+    thanks to Junchao Zhang for the suggestion.
+  + Improve the API for dealing with disallowed resources
+    - HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM is replaced with FLAG_INCLUDE_DISALLOWED
+      and --whole-system command-line options with --disallowed.
+      . Former names are still accepted for backward compatibility.
+    - Add hwloc_topology_allow() for changing allowed sets after load().
+    - Add the HWLOC_ALLOW=all environment variable to totally ignore
+      administrative restrictions such as Linux Cgroups.
+    - Add disallowed_pu and disallowed_numa bits to the discovery support
+      structure.
+  + Group objects have a new "dont_merge" attribute to prevent them from
+    being automatically merged with identical parent or children.
+  + Add more distances-related features:
+    - Add hwloc_distances_get_name() to retrieve a string describing
+      what a distances structure contain.
+    - Add hwloc_distances_get_by_name() to retrieve distances structures
+      based on their name.
+    - Add hwloc_distances_release_remove()
+    - Distances may now cover objects of different types with new kind
+      HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES.
+* Backends
+  + Add support for Linux 5.3 new sysfs cpu topology files with Die information.
+  + Add support for Intel v2 Extended Topology Enumeration in the x86 backend.
+  + Improve memory locality on Linux by using HMAT initiators (exposed
+    since Linux 5.2+), and NUMA distances for CPU-less NUMA nodes.
+  + The x86 backend now properly handles offline CPUs.
+  + Detect the locality of NVIDIA GPU OpenCL devices.
+  + Ignore NUMA nodes that correspond to NVIDIA GPU by default.
+    - They may be unignored if HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=1 in the environment.
+    - Fix their CPU locality and add info attributes to identify them.
+    Thanks to Max Katz and Edgar Leon for the help.
+  + Add support for IBM S/390 drawers.
+  + Rework the heuristics for discovering KNL Cluster and Memory modes
+    to stop assuming all CPUs are online (required for mOS support).
+    Thanks to Sharath K Bhat for testing patches.
+  + Ignore NUMA node information from AMD topoext in the x86 backend,
+    unless HWLOC_X86_TOPOEXT_NUMANODES=1 is set in the environment.
+  + Expose Linux DAX devices as hwloc Block OS devices.
+  + Remove support for /proc/cpuinfo-only topology discovery in Linux
+    kernel prior to 2.6.16.
+  + Disable POWER device-tree-based topology on Linux by default.
+    - It may be reenabled by setting HWLOC_USE_DT=1 in the environment.
+  + Discovery components are now divided in phases that may be individually
+    blacklisted.
+    - The linuxio component has been merged back into the linux component.
+* Tools
+  + lstopo
+    - lstopo factorizes objects by default in the graphical output when
+      there are more than 4 identical children.
+      . New options --no-factorize and --factorize may be used to configure this.
+      . Hit the 'f' key to disable factorizing in interactive outputs.
+    - Both logical and OS/physical indexes are now displayed by default
+      for PU and NUMA nodes.
+    - The X11 and Windows interactive outputs support many keyboard
+      shortcuts to dynamically customize the attributes, legend, etc.
+    - Add --linespacing and change default margins and linespacing.
+    - Add --allow for changing allowed sets.
+    - Add a native SVG backend. Its graphical output may be slightly less
+      pretty than Cairo (still used by default if available) but the SVG
+      code provides attributes to manipulate objects from HTML/JS.
+      See dynamic_SVG_example.html for an example.
+  + Add --nodeset options to hwloc-calc for converting between cpusets and
+    nodesets.
+  + Add --no-smt to lstopo, hwloc-bind and hwloc-calc to ignore multiple
+    PU in SMT cores.
+  + hwloc-annotate may annotate multiple locations at once.
+  + Add a HTML/JS version of hwloc-ps. See contrib/hwloc-ps.www/README.
+  + Add bash completions.
+* Misc
+  + Add several FAQ entries in "Compatibility between hwloc versions"
+    about API version, ABI, XML, Synthetic strings, and shmem topologies.
 
 
 Version 2.0.4 (also included in 1.11.13 when appropriate)
@@ -214,6 +302,54 @@ Version 2.0.0
   + hwloc now requires a C99 compliant compiler.
 
 
+Version 1.11.13 (also included in 2.0.4)
+---------------
+* Add support for Linux 5.3 new sysfs cpu topology files with Die information.
+* Add support for Intel v2 Extended Topology Enumeration in the x86 backend.
+* Tiles, Modules and Dies are exposed as Groups for now.
+  + HWLOC_DONT_MERGE_DIE_GROUPS=1 may be set in the environment to prevent
+    Die groups from being automatically merged with identical parent or children.
+* Ignore NUMA node information from AMD topoext in the x86 backend,
+  unless HWLOC_X86_TOPOEXT_NUMANODES=1 is set in the environment.
+* Group objects have a new "dont_merge" attribute to prevent them from
+  being automatically merged with identical parent or children.
+
+
+Version 1.11.12 (also included in 2.0.3)
+---------------
+* Fix a corner case of hwloc_topology_restrict() where children would
+  become out-of-order.
+* Fix the return length of export_xmlbuffer() functions to always
+  include the ending \0.
+
+
+Version 1.11.11 (also included in 2.0.2)
+---------------
+* Add support for Hygon Dhyana processors in the x86 backend,
+  thanks to Pu Wen for the patch.
+* Fix symbol renaming to also rename internal components,
+  thanks to Evan Ramos for the patch.
+* Fix build on HP-UX, thanks to Richard Lloyd for reporting the issues.
+* Detect PCI link speed without being root on Linux >= 4.13.
+
+
+Version 1.11.10 (also included in 2.0.1)
+---------------
+* Fix detection of cores and hyperthreads on Mac OS X.
+* Serialize pciaccess discovery to fix concurrent topology loads in
+  multiple threads.
+* Fix first touch area memory binding on Linux when thread memory
+  binding is different.
+* Some minor fixes to memory binding.
+* Fix hwloc-dump-hwdata to only process SMBIOS information that correspond
+  to the KNL and KNM configuration.
+* Add a heuristic for guessing KNL/KNM memory and cluster modes when
+  hwloc-dump-hwdata could not run as root earlier.
+* Fix discovery of NVMe OS devices on Linux >= 4.0.
+* Add get_area_memlocation() on Windows.
+* Add CPUVendor, Model, ... attributes on Mac OS X.
+
+
 Version 1.11.9
 --------------
 * Add support for Zhaoxin ZX-C and ZX-D processors in the x86 backend,
@@ -941,7 +1077,7 @@ Version 1.6.0
   + Add a section about Synthetic topologies in the documentation.
 
 
-Version 1.5.2 (some of these changes are in v1.6.2 but not in v1.6)
+Version 1.5.2 (some of these changes are in 1.6.2 but not in 1.6)
 -------------
 * Use libpciaccess instead of pciutils/libpci by default for I/O discovery.
   pciutils/libpci is only used if --enable-libpci is given to configure
@@ -1076,9 +1212,8 @@ Version 1.4.2
   for most of them.
 
 
-Version 1.4.1
+Version 1.4.1 (contains all 1.3.2 changes)
 -------------
-* This release contains all changes from v1.3.2.
 * Fix hwloc_alloc_membind, thanks Karl Napf for reporting the issue.
 * Fix memory leaks in some get_membind() functions.
 * Fix helpers converting from Linux libnuma to hwloc (hwloc/linux-libnuma.h)
@@ -1091,7 +1226,7 @@ Version 1.4.1
   issues.
 
 
-Version 1.4.0 (does not contain all v1.3.2 changes)
+Version 1.4.0 (does not contain all 1.3.2 changes)
 -------------
 * Major features
   + Add "custom" interface and "assembler" tools to build multi-node
@@ -1536,7 +1671,7 @@ Version 1.0.0
 
 
 Version 0.9.4 (unreleased)
---------------------------
+-------------
 
 * Fix reseting colors to normal in lstopo -.txt output.
 * Fix Linux pthread_t binding error report.
@@ -1593,7 +1728,7 @@ Version 0.9.1
   the physical location of IB devices.
 
 
-Version 0.9 (libtopology)
+Version 0.9 (formerly named "libtopology")
--------------------------
+-----------
 
 * First release.

src/3rdparty/hwloc/VERSION

@@ -8,8 +8,8 @@
 # Please update HWLOC_VERSION* in contrib/windows/hwloc_config.h too.
 
 major=2
-minor=0
+minor=1
-release=4
+release=0
 
 # greek is used for alpha or beta release tags.  If it is non-empty,
 # it will be appended to the version number.  It does not have to be
@@ -22,7 +22,7 @@ greek=
 
 # The date when this release was created
 
-date="Jun 03, 2019"
+date="Sep 30, 2019"
 
 # If snapshot=1, then use the value from snapshot_version as the
 # entire hwloc version (i.e., ignore major, minor, release, and
@@ -41,7 +41,7 @@ snapshot_version=${major}.${minor}.${release}${greek}-git
 # 2. Version numbers are described in the Libtool current:revision:age
 # format.
 
-libhwloc_so_version=15:3:0
+libhwloc_so_version=16:0:1
 libnetloc_so_version=0:0:0
 
 # Please also update the <TargetName> lines in contrib/windows/libhwloc.vcxproj

src/3rdparty/hwloc/include/hwloc.h

@@ -53,7 +53,8 @@
 #ifndef HWLOC_H
 #define HWLOC_H
 
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
+
 #include <sys/types.h>
 #include <stdio.h>
 #include <string.h>
@@ -62,13 +63,13 @@
 /*
  * Symbol transforms
  */
-#include <hwloc/rename.h>
+#include "hwloc/rename.h"
 
 /*
  * Bitmap definitions
  */
 
-#include <hwloc/bitmap.h>
+#include "hwloc/bitmap.h"
 
 
 #ifdef __cplusplus
@@ -86,13 +87,13 @@ extern "C" {
  * actually modifies the API.
  *
  * Users may check for available features at build time using this number
- * (see \ref faq_upgrade).
+ * (see \ref faq_version_api).
  *
  * \note This should not be confused with HWLOC_VERSION, the library version.
  * Two stable releases of the same series usually have the same ::HWLOC_API_VERSION
  * even if their HWLOC_VERSION are different.
  */
-#define HWLOC_API_VERSION 0x00020000
+#define HWLOC_API_VERSION 0x00020100
 
 /** \brief Indicate at runtime which hwloc API version was used at build time.
  *
@@ -101,7 +102,7 @@ extern "C" {
 HWLOC_DECLSPEC unsigned hwloc_get_api_version(void);
 
 /** \brief Current component and plugin ABI version (see hwloc/plugins.h) */
-#define HWLOC_COMPONENT_ABI 5
+#define HWLOC_COMPONENT_ABI 6
 
 /** @} */
 
@@ -186,7 +187,8 @@ typedef enum {
   HWLOC_OBJ_PACKAGE,	/**< \brief Physical package.
 			  * The physical package that usually gets inserted
 			  * into a socket on the motherboard.
-			  * A processor package usually contains multiple cores.
+			  * A processor package usually contains multiple cores,
+			  * and possibly some dies.
 			  */
   HWLOC_OBJ_CORE,	/**< \brief Core.
 			  * A computation unit (may be shared by several
@@ -233,6 +235,10 @@ typedef enum {
 			  * It is usually close to some cores (the corresponding objects
 			  * are descendants of the NUMA node object in the hwloc tree).
 			  *
+			  * This is the smallest object representing Memory resources,
+			  * it cannot have any child except Misc objects.
+			  * However it may have Memory-side cache parents.
+			  *
 			  * There is always at least one such object in the topology
 			  * even if the machine is not NUMA.
 			  *
@@ -279,6 +285,24 @@ typedef enum {
 			  * Misc objects have NULL CPU and node sets.
 			  */
 
+  HWLOC_OBJ_MEMCACHE,	/**< \brief Memory-side cache (filtered out by default).
+			  * A cache in front of a specific NUMA node.
+			  *
+			  * This object always has at least one NUMA node as a memory child.
+			  *
+			  * Memory objects are not listed in the main children list,
+			  * but rather in the dedicated Memory children list.
+			  *
+			  * Memory-side cache have a special depth ::HWLOC_TYPE_DEPTH_MEMCACHE
+			  * instead of a normal depth just like other objects in the
+			  * main tree.
+			  */
+
+  HWLOC_OBJ_DIE,	/**< \brief Die within a physical package.
+			 * A subpart of the physical package, that contains multiple cores.
+			 * \hideinitializer
+			 */
+
   HWLOC_OBJ_TYPE_MAX    /**< \private Sentinel value */
 } hwloc_obj_type_t;
 
@@ -297,8 +321,8 @@ typedef enum hwloc_obj_bridge_type_e {
 
 /** \brief Type of a OS device. */
 typedef enum hwloc_obj_osdev_type_e {
-  HWLOC_OBJ_OSDEV_BLOCK,	/**< \brief Operating system block device.
+  HWLOC_OBJ_OSDEV_BLOCK,	/**< \brief Operating system block device, or non-volatile memory device.
-				  * For instance "sda" on Linux. */
+				  * For instance "sda" or "dax2.0" on Linux. */
   HWLOC_OBJ_OSDEV_GPU,		/**< \brief Operating system GPU device.
 				  * For instance ":0.0" for a GL display,
 				  * "card0" for a Linux DRM device. */
@@ -434,9 +458,15 @@ struct hwloc_obj {
 					 * These children are listed in \p memory_first_child.
 					 */
   struct hwloc_obj *memory_first_child;	/**< \brief First Memory child.
-					 * NUMA nodes are listed here (\p memory_arity and \p memory_first_child)
+					 * NUMA nodes and Memory-side caches are listed here
+					 * (\p memory_arity and \p memory_first_child)
 					 * instead of in the normal children list.
 					 * See also hwloc_obj_type_is_memory().
+					 *
+					 * A memory hierarchy starts from a normal CPU-side object
+					 * (e.g. Package) and ends with NUMA nodes as leaves.
+					 * There might exist some memory-side caches between them
+					 * in the middle of the memory subtree.
 					 */
   /**@}*/
 
@@ -471,7 +501,7 @@ struct hwloc_obj {
                                           * object and known how (the children path between this object and the PU
                                           * objects).
                                           *
-                                          * If the ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM configuration flag is set,
+                                          * If the ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED configuration flag is set,
                                           * some of these CPUs may not be allowed for binding,
                                           * see hwloc_topology_get_allowed_cpuset().
                                           *
@@ -483,7 +513,7 @@ struct hwloc_obj {
                                           *
                                           * This may include not only the same as the cpuset field, but also some CPUs for
                                           * which topology information is unknown or incomplete, some offlines CPUs, and
-                                          * the CPUs that are ignored when the ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM flag
+                                          * the CPUs that are ignored when the ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED flag
                                           * is not set.
                                           * Thus no corresponding PU object may be found in the topology, because the
                                           * precise position is undefined. It is however known that it would be somewhere
@@ -501,7 +531,7 @@ struct hwloc_obj {
                                           *
                                           * In the end, these nodes are those that are close to the current object.
                                           *
-                                          * If the ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM configuration flag is set,
+                                          * If the ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED configuration flag is set,
                                           * some of these nodes may not be allowed for allocation,
                                           * see hwloc_topology_get_allowed_nodeset().
                                           *
@@ -516,7 +546,7 @@ struct hwloc_obj {
                                           *
                                           * This may include not only the same as the nodeset field, but also some NUMA
                                           * nodes for which topology information is unknown or incomplete, some offlines
-                                          * nodes, and the nodes that are ignored when the ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM
+                                          * nodes, and the nodes that are ignored when the ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED
                                           * flag is not set.
                                           * Thus no corresponding NUMA node object may be found in the topology, because the
                                           * precise position is undefined. It is however known that it would be
@@ -770,7 +800,8 @@ enum hwloc_get_type_depth_e {
     HWLOC_TYPE_DEPTH_BRIDGE = -4,     /**< \brief Virtual depth for bridge object level. \hideinitializer */
     HWLOC_TYPE_DEPTH_PCI_DEVICE = -5, /**< \brief Virtual depth for PCI device object level. \hideinitializer */
     HWLOC_TYPE_DEPTH_OS_DEVICE = -6,  /**< \brief Virtual depth for software device object level. \hideinitializer */
-    HWLOC_TYPE_DEPTH_MISC = -7        /**< \brief Virtual depth for Misc object. \hideinitializer */
+    HWLOC_TYPE_DEPTH_MISC = -7,       /**< \brief Virtual depth for Misc object. \hideinitializer */
+    HWLOC_TYPE_DEPTH_MEMCACHE = -8    /**< \brief Virtual depth for MemCache object. \hideinitializer */
 };
 
 /** \brief Return the depth of parents where memory objects are attached.
@@ -1781,6 +1812,31 @@ HWLOC_DECLSPEC int hwloc_topology_set_xml(hwloc_topology_t __hwloc_restrict topo
  */
 HWLOC_DECLSPEC int hwloc_topology_set_xmlbuffer(hwloc_topology_t __hwloc_restrict topology, const char * __hwloc_restrict buffer, int size);
 
+/** \brief Flags to be passed to hwloc_topology_set_components()
+ */
+enum hwloc_topology_components_flag_e {
+  /** \brief Blacklist the target component from being used.
+   * \hideinitializer
+   */
+  HWLOC_TOPOLOGY_COMPONENTS_FLAG_BLACKLIST = (1UL<<0)
+};
+
+/** \brief Prevent a discovery component from being used for a topology.
+ *
+ * \p name is the name of the discovery component that should not be used
+ * when loading topology \p topology. The name is a string such as "cuda".
+ *
+ * For components with multiple phases, it may also be suffixed with the name
+ * of a phase, for instance "linux:io".
+ *
+ * \p flags should be ::HWLOC_TOPOLOGY_COMPONENTS_FLAG_BLACKLIST.
+ *
+ * This may be used to avoid expensive parts of the discovery process.
+ * For instance, CUDA-specific discovery may be expensive and unneeded
+ * while generic I/O discovery could still be useful.
+ */
+HWLOC_DECLSPEC int hwloc_topology_set_components(hwloc_topology_t __hwloc_restrict topology, unsigned long flags, const char * __hwloc_restrict name);
+
 /** @} */
 
 
@@ -1800,28 +1856,27 @@ HWLOC_DECLSPEC int hwloc_topology_set_xmlbuffer(hwloc_topology_t __hwloc_restric
  * They may also be returned by hwloc_topology_get_flags().
  */
 enum hwloc_topology_flags_e {
- /** \brief Detect the whole system, ignore reservations.
+ /** \brief Detect the whole system, ignore reservations, include disallowed objects.
    *
    * Gather all resources, even if some were disabled by the administrator.
    * For instance, ignore Linux Cgroup/Cpusets and gather all processors and memory nodes.
    *
    * When this flag is not set, PUs and NUMA nodes that are disallowed are not added to the topology.
    * Parent objects (package, core, cache, etc.) are added only if some of their children are allowed.
+   * All existing PUs and NUMA nodes in the topology are allowed.
+   * hwloc_topology_get_allowed_cpuset() and hwloc_topology_get_allowed_nodeset()
+   * are equal to the root object cpuset and nodeset.
    *
    * When this flag is set, the actual sets of allowed PUs and NUMA nodes are given
    * by hwloc_topology_get_allowed_cpuset() and hwloc_topology_get_allowed_nodeset().
    * They may be smaller than the root object cpuset and nodeset.
    *
-   * When this flag is not set, all existing PUs and NUMA nodes in the topology
-   * are allowed. hwloc_topology_get_allowed_cpuset() and hwloc_topology_get_allowed_nodeset()
-   * are equal to the root object cpuset and nodeset.
-   *
    * If the current topology is exported to XML and reimported later, this flag
    * should be set again in the reimported topology so that disallowed resources
    * are reimported as well.
    * \hideinitializer
    */
-  HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM = (1UL<<0),
+  HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED = (1UL<<0),
 
  /** \brief Assume that the selected backend provides the topology for the
    * system on which we are running.
@@ -1901,6 +1956,10 @@ struct hwloc_topology_discovery_support {
   unsigned char numa;
   /** \brief Detecting the amount of memory in NUMA nodes is supported. */
   unsigned char numa_memory;
+  /** \brief Detecting and identifying PU objects that are not available to the current process is supported. */
+  unsigned char disallowed_pu;
+  /** \brief Detecting and identifying NUMA nodes that are not available to the current process is supported. */
+  unsigned char disallowed_numa;
 };
 
 /** \brief Flags describing actual PU binding support for this topology.
@@ -1998,7 +2057,7 @@ HWLOC_DECLSPEC const struct hwloc_topology_support *hwloc_topology_get_support(h
  *
  * By default, most objects are kept (::HWLOC_TYPE_FILTER_KEEP_ALL).
  * Instruction caches, I/O and Misc objects are ignored by default (::HWLOC_TYPE_FILTER_KEEP_NONE).
- * Group levels are ignored unless they bring structure (::HWLOC_TYPE_FILTER_KEEP_STRUCTURE).
+ * Die and Group levels are ignored unless they bring structure (::HWLOC_TYPE_FILTER_KEEP_STRUCTURE).
  *
  * Note that group objects are also ignored individually (without the entire level)
  * when they do not bring structure.
@@ -2063,11 +2122,15 @@ HWLOC_DECLSPEC int hwloc_topology_get_type_filter(hwloc_topology_t topology, hwl
  */
 HWLOC_DECLSPEC int hwloc_topology_set_all_types_filter(hwloc_topology_t topology, enum hwloc_type_filter_e filter);
 
-/** \brief Set the filtering for all cache object types.
+/** \brief Set the filtering for all CPU cache object types.
+ *
+ * Memory-side caches are not involved since they are not CPU caches.
  */
 HWLOC_DECLSPEC int hwloc_topology_set_cache_types_filter(hwloc_topology_t topology, enum hwloc_type_filter_e filter);
 
-/** \brief Set the filtering for all instruction cache object types.
+/** \brief Set the filtering for all CPU instruction cache object types.
+ *
+ * Memory-side caches are not involved since they are not CPU caches.
  */
 HWLOC_DECLSPEC int hwloc_topology_set_icache_types_filter(hwloc_topology_t topology, enum hwloc_type_filter_e filter);
 
@@ -2110,6 +2173,19 @@ enum hwloc_restrict_flags_e {
    */
   HWLOC_RESTRICT_FLAG_REMOVE_CPULESS = (1UL<<0),
 
+  /** \brief Restrict by nodeset instead of CPU set.
+   * Only keep objects whose nodeset is included or partially included in the given set.
+   * This flag may not be used with ::HWLOC_RESTRICT_FLAG_BYNODESET.
+   */
+  HWLOC_RESTRICT_FLAG_BYNODESET =  (1UL<<3),
+
+  /** \brief Remove all objects that became Memory-less.
+   * By default, only objects that contain no PU and no memory are removed.
+   * This flag may only be used with ::HWLOC_RESTRICT_FLAG_BYNODESET.
+   * \hideinitializer
+   */
+  HWLOC_RESTRICT_FLAG_REMOVE_MEMLESS = (1UL<<4),
+
   /** \brief Move Misc objects to ancestors if their parents are removed during restriction.
    * If this flag is not set, Misc objects are removed when their parents are removed.
    * \hideinitializer
@@ -2123,28 +2199,70 @@ enum hwloc_restrict_flags_e {
   HWLOC_RESTRICT_FLAG_ADAPT_IO = (1UL<<2)
 };
 
-/** \brief Restrict the topology to the given CPU set.
+/** \brief Restrict the topology to the given CPU set or nodeset.
  *
  * Topology \p topology is modified so as to remove all objects that
- * are not included (or partially included) in the CPU set \p cpuset.
+ * are not included (or partially included) in the CPU set \p set.
  * All objects CPU and node sets are restricted accordingly.
  *
+ * If ::HWLOC_RESTRICT_FLAG_BYNODESET is passed in \p flags,
+ * \p set is considered a nodeset instead of a CPU set.
+ *
  * \p flags is a OR'ed set of ::hwloc_restrict_flags_e.
  *
  * \note This call may not be reverted by restricting back to a larger
- * cpuset. Once dropped during restriction, objects may not be brought
+ * set. Once dropped during restriction, objects may not be brought
  * back, except by loading another topology with hwloc_topology_load().
  *
  * \return 0 on success.
  *
- * \return -1 with errno set to EINVAL if the input cpuset is invalid.
+ * \return -1 with errno set to EINVAL if the input set is invalid.
  * The topology is not modified in this case.
  *
  * \return -1 with errno set to ENOMEM on failure to allocate internal data.
  * The topology is reinitialized in this case. It should be either
  * destroyed with hwloc_topology_destroy() or configured and loaded again.
  */
-HWLOC_DECLSPEC int hwloc_topology_restrict(hwloc_topology_t __hwloc_restrict topology, hwloc_const_cpuset_t cpuset, unsigned long flags);
+HWLOC_DECLSPEC int hwloc_topology_restrict(hwloc_topology_t __hwloc_restrict topology, hwloc_const_bitmap_t set, unsigned long flags);
+
+/** \brief Flags to be given to hwloc_topology_allow(). */
+enum hwloc_allow_flags_e {
+  /** \brief Mark all objects as allowed in the topology.
+   *
+   * \p cpuset and \p nođeset given to hwloc_topology_allow() must be \c NULL.
+   * \hideinitializer */
+  HWLOC_ALLOW_FLAG_ALL = (1UL<<0),
+
+  /** \brief Only allow objects that are available to the current process.
+   *
+   * The topology must have ::HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM so that the set
+   * of available resources can actually be retrieved from the operating system.
+   *
+   * \p cpuset and \p nođeset given to hwloc_topology_allow() must be \c NULL.
+   * \hideinitializer */
+  HWLOC_ALLOW_FLAG_LOCAL_RESTRICTIONS = (1UL<<1),
+
+  /** \brief Allow a custom set of objects, given to hwloc_topology_allow() as \p cpuset and/or \p nodeset parameters.
+   * \hideinitializer */
+  HWLOC_ALLOW_FLAG_CUSTOM = (1UL<<2)
+};
+
+/** \brief Change the sets of allowed PUs and NUMA nodes in the topology.
+ *
+ * This function only works if the ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED
+ * was set on the topology. It does not modify any object, it only changes
+ * the sets returned by hwloc_topology_get_allowed_cpuset() and
+ * hwloc_topology_get_allowed_nodeset().
+ *
+ * It is notably useful when importing a topology from another process
+ * running in a different Linux Cgroup.
+ *
+ * \p flags must be set to one flag among ::hwloc_allow_flags_e.
+ *
+ * \note Removing objects from a topology should rather be performed with
+ * hwloc_topology_restrict().
+ */
+HWLOC_DECLSPEC int hwloc_topology_allow(hwloc_topology_t __hwloc_restrict topology, hwloc_const_cpuset_t cpuset, hwloc_const_nodeset_t nodeset, unsigned long flags);
 
 /** \brief Add a MISC object as a leaf of the topology
  *
@@ -2250,21 +2368,21 @@ HWLOC_DECLSPEC int hwloc_obj_add_other_obj_sets(hwloc_obj_t dst, hwloc_obj_t src
 
 
 /* high-level helpers */
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
 
 /* inline code of some functions above */
-#include <hwloc/inlines.h>
+#include "hwloc/inlines.h"
 
 /* exporting to XML or synthetic */
-#include <hwloc/export.h>
+#include "hwloc/export.h"
 
 /* distances */
-#include <hwloc/distances.h>
+#include "hwloc/distances.h"
 
 /* topology diffs */
-#include <hwloc/diff.h>
+#include "hwloc/diff.h"
 
 /* deprecated headers */
-#include <hwloc/deprecated.h>
+#include "hwloc/deprecated.h"
 
 #endif /* HWLOC_H */

src/3rdparty/hwloc/include/hwloc/autogen/config.h

@@ -11,10 +11,10 @@
 #ifndef HWLOC_CONFIG_H
 #define HWLOC_CONFIG_H
 
-#define HWLOC_VERSION "2.0.4"
+#define HWLOC_VERSION "2.1.0"
 #define HWLOC_VERSION_MAJOR 2
-#define HWLOC_VERSION_MINOR 0
+#define HWLOC_VERSION_MINOR 1
-#define HWLOC_VERSION_RELEASE 4
+#define HWLOC_VERSION_RELEASE 0
 #define HWLOC_VERSION_GREEK ""
 
 #define __hwloc_restrict

src/3rdparty/hwloc/include/hwloc/bitmap.h

@@ -13,7 +13,8 @@
 #ifndef HWLOC_BITMAP_H
 #define HWLOC_BITMAP_H
 
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
+
 #include <assert.h>
 
 
@@ -198,6 +199,9 @@ HWLOC_DECLSPEC int hwloc_bitmap_from_ulong(hwloc_bitmap_t bitmap, unsigned long
 /** \brief Setup bitmap \p bitmap from unsigned long \p mask used as \p i -th subset */
 HWLOC_DECLSPEC int hwloc_bitmap_from_ith_ulong(hwloc_bitmap_t bitmap, unsigned i, unsigned long mask);
 
+/** \brief Setup bitmap \p bitmap from unsigned longs \p masks used as first \p nr subsets */
+HWLOC_DECLSPEC int hwloc_bitmap_from_ulongs(hwloc_bitmap_t bitmap, unsigned nr, const unsigned long *masks);
+
 
 /*
  * Modifying bitmaps.
@@ -256,6 +260,29 @@ HWLOC_DECLSPEC unsigned long hwloc_bitmap_to_ulong(hwloc_const_bitmap_t bitmap)
 /** \brief Convert the \p i -th subset of bitmap \p bitmap into unsigned long mask */
 HWLOC_DECLSPEC unsigned long hwloc_bitmap_to_ith_ulong(hwloc_const_bitmap_t bitmap, unsigned i) __hwloc_attribute_pure;
 
+/** \brief Convert the first \p nr subsets of bitmap \p bitmap into the array of \p nr unsigned long \p masks
+ *
+ * \p nr may be determined earlier with hwloc_bitmap_nr_ulongs().
+ *
+ * \return 0
+ */
+HWLOC_DECLSPEC int hwloc_bitmap_to_ulongs(hwloc_const_bitmap_t bitmap, unsigned nr, unsigned long *masks);
+
+/** \brief Return the number of unsigned longs required for storing bitmap \p bitmap entirely
+ *
+ * This is the number of contiguous unsigned longs from the very first bit of the bitmap
+ * (even if unset) up to the last set bit.
+ * This is useful for knowing the \p nr parameter to pass to hwloc_bitmap_to_ulongs()
+ * (or which calls to hwloc_bitmap_to_ith_ulong() are needed)
+ * to entirely convert a bitmap into multiple unsigned longs.
+ *
+ * When called on the output of hwloc_topology_get_topology_cpuset(),
+ * the returned number is large enough for all cpusets of the topology.
+ *
+ * \return -1 if \p bitmap is infinite.
+ */
+HWLOC_DECLSPEC int hwloc_bitmap_nr_ulongs(hwloc_const_bitmap_t bitmap) __hwloc_attribute_pure;
+
 /** \brief Test whether index \p id is part of bitmap \p bitmap.
  *
  * \return 1 if the bit at index \p id is set in bitmap \p bitmap, 0 otherwise.

src/3rdparty/hwloc/include/hwloc/cuda.h

@@ -16,11 +16,11 @@
 #ifndef HWLOC_CUDA_H
 #define HWLOC_CUDA_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
 #endif
 
 #include <cuda.h>

src/3rdparty/hwloc/include/hwloc/cudart.h

@@ -16,11 +16,11 @@
 #ifndef HWLOC_CUDART_H
 #define HWLOC_CUDART_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
 #endif
 
 #include <cuda.h> /* for CUDA_VERSION */

src/3rdparty/hwloc/include/hwloc/deprecated.h

@@ -1,6 +1,6 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2017 Inria.  All rights reserved.
+ * Copyright © 2009-2018 Inria.  All rights reserved.
  * Copyright © 2009-2012 Université Bordeaux
  * Copyright © 2009-2010 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
@@ -21,6 +21,8 @@
 extern "C" {
 #endif
 
+/* backward compat with v2.0 before WHOLE_SYSTEM renaming */
+#define HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED
 /* backward compat with v1.11 before System removal */
 #define HWLOC_OBJ_SYSTEM HWLOC_OBJ_MACHINE
 /* backward compat with v1.10 before Socket->Package renaming */

src/3rdparty/hwloc/include/hwloc/distances.h

@@ -87,7 +87,12 @@ enum hwloc_distances_kind_e {
    * Such values are currently ignored for distance-based grouping.
    * \hideinitializer
    */
-  HWLOC_DISTANCES_KIND_MEANS_BANDWIDTH = (1UL<<3)
+  HWLOC_DISTANCES_KIND_MEANS_BANDWIDTH = (1UL<<3),
+
+  /** \brief This distances structure covers objects of different types.
+   * \hideinitializer
+   */
+  HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES = (1UL<<4)
 };
 
 /** \brief Retrieve distance matrices.
@@ -131,20 +136,32 @@ hwloc_distances_get_by_depth(hwloc_topology_t topology, int depth,
  *
  * Identical to hwloc_distances_get() with the additional \p type filter.
  */
-static __hwloc_inline int
+HWLOC_DECLSPEC int
 hwloc_distances_get_by_type(hwloc_topology_t topology, hwloc_obj_type_t type,
 			    unsigned *nr, struct hwloc_distances_s **distances,
-			    unsigned long kind, unsigned long flags)
+			    unsigned long kind, unsigned long flags);
-{
-  int depth = hwloc_get_type_depth(topology, type);
-  if (depth == HWLOC_TYPE_DEPTH_UNKNOWN || depth == HWLOC_TYPE_DEPTH_MULTIPLE) {
-    *nr = 0;
-    return 0;
-  }
-  return hwloc_distances_get_by_depth(topology, depth, nr, distances, kind, flags);
-}
 
-/** \brief Release a distance matrix structure previously returned by hwloc_distances_get(). */
+/** \brief Retrieve a distance matrix with the given name.
+ *
+ * Usually only one distances structure may match a given name.
+ */
+HWLOC_DECLSPEC int
+hwloc_distances_get_by_name(hwloc_topology_t topology, const char *name,
+			    unsigned *nr, struct hwloc_distances_s **distances,
+			    unsigned long flags);
+
+/** \brief Get a description of what a distances structure contains.
+ *
+ * For instance "NUMALatency" for hardware-provided NUMA distances (ACPI SLIT),
+ * or NULL if unknown.
+ */
+HWLOC_DECLSPEC const char *
+hwloc_distances_get_name(hwloc_topology_t topology, struct hwloc_distances_s *distances);
+
+/** \brief Release a distance matrix structure previously returned by hwloc_distances_get().
+ *
+ * \note This function is not required if the structure is removed with hwloc_distances_release_remove().
+ */
 HWLOC_DECLSPEC void
 hwloc_distances_release(hwloc_topology_t topology, struct hwloc_distances_s *distances);
 
@@ -221,11 +238,11 @@ enum hwloc_distances_add_flag_e {
  * The distance from object i to object j is in slot i*nbobjs+j.
  *
  * \p kind specifies the kind of distance as a OR'ed set of ::hwloc_distances_kind_e.
+ * Kind ::HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES will be automatically added
+ * if objects of different types are given.
  *
  * \p flags configures the behavior of the function using an optional OR'ed set of
  * ::hwloc_distances_add_flag_e.
- *
- * Objects must be of the same type. They cannot be of type Group.
  */
 HWLOC_DECLSPEC int hwloc_distances_add(hwloc_topology_t topology,
 				       unsigned nbobjs, hwloc_obj_t *objs, hwloc_uint64_t *values,
@@ -237,7 +254,7 @@ HWLOC_DECLSPEC int hwloc_distances_add(hwloc_topology_t topology,
  * gathered through the OS.
  *
  * If these distances were used to group objects, these additional
- *Group objects are not removed from the topology.
+ * Group objects are not removed from the topology.
  */
 HWLOC_DECLSPEC int hwloc_distances_remove(hwloc_topology_t topology);
 
@@ -260,6 +277,12 @@ hwloc_distances_remove_by_type(hwloc_topology_t topology, hwloc_obj_type_t type)
   return hwloc_distances_remove_by_depth(topology, depth);
 }
 
+/** \brief Release and remove the given distance matrice from the topology.
+ *
+ * This function includes a call to hwloc_distances_release().
+ */
+HWLOC_DECLSPEC int hwloc_distances_release_remove(hwloc_topology_t topology, struct hwloc_distances_s *distances);
+
 /** @} */
 
 

src/3rdparty/hwloc/include/hwloc/gl.h

@@ -14,7 +14,7 @@
 #ifndef HWLOC_GL_H
 #define HWLOC_GL_H
 
-#include <hwloc.h>
+#include "hwloc.h"
 
 #include <stdio.h>
 #include <string.h>

src/3rdparty/hwloc/include/hwloc/glibc-sched.h

@@ -17,8 +17,9 @@
 #ifndef HWLOC_GLIBC_SCHED_H
 #define HWLOC_GLIBC_SCHED_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
+
 #include <assert.h>
 
 #if !defined _GNU_SOURCE || !defined _SCHED_H || (!defined CPU_SETSIZE && !defined sched_priority)

src/3rdparty/hwloc/include/hwloc/helper.h

@@ -527,30 +527,36 @@ hwloc_obj_type_is_io(hwloc_obj_type_t type);
  *
  * Memory objects are objects attached to their parents
  * in the Memory children list.
- * This current only includes NUMA nodes.
+ * This current includes NUMA nodes and Memory-side caches.
  *
  * \return 1 if an object of type \p type is a Memory object, 0 otherwise.
  */
 HWLOC_DECLSPEC int
 hwloc_obj_type_is_memory(hwloc_obj_type_t type);
 
-/** \brief Check whether an object type is a Cache (Data, Unified or Instruction).
+/** \brief Check whether an object type is a CPU Cache (Data, Unified or Instruction).
+ *
+ * Memory-side caches are not CPU caches.
  *
  * \return 1 if an object of type \p type is a Cache, 0 otherwise.
  */
 HWLOC_DECLSPEC int
 hwloc_obj_type_is_cache(hwloc_obj_type_t type);
 
-/** \brief Check whether an object type is a Data or Unified Cache.
+/** \brief Check whether an object type is a CPU Data or Unified Cache.
  *
- * \return 1 if an object of type \p type is a Data or Unified Cache, 0 otherwise.
+ * Memory-side caches are not CPU caches.
+ *
+ * \return 1 if an object of type \p type is a CPU Data or Unified Cache, 0 otherwise.
  */
 HWLOC_DECLSPEC int
 hwloc_obj_type_is_dcache(hwloc_obj_type_t type);
 
-/** \brief Check whether an object type is a Instruction Cache,
+/** \brief Check whether an object type is a CPU Instruction Cache,
  *
- * \return 1 if an object of type \p type is a Instruction Cache, 0 otherwise.
+ * Memory-side caches are not CPU caches.
+ *
+ * \return 1 if an object of type \p type is a CPU Instruction Cache, 0 otherwise.
  */
 HWLOC_DECLSPEC int
 hwloc_obj_type_is_icache(hwloc_obj_type_t type);
@@ -914,7 +920,7 @@ hwloc_topology_get_complete_cpuset(hwloc_topology_t topology) __hwloc_attribute_
  * \note The returned cpuset is not newly allocated and should thus not be
  * changed or freed; hwloc_bitmap_dup() must be used to obtain a local copy.
  *
- * \note This is equivalent to retrieving the root object complete CPU-set.
+ * \note This is equivalent to retrieving the root object CPU-set.
  */
 HWLOC_DECLSPEC hwloc_const_cpuset_t
 hwloc_topology_get_topology_cpuset(hwloc_topology_t topology) __hwloc_attribute_pure;
@@ -923,11 +929,11 @@ hwloc_topology_get_topology_cpuset(hwloc_topology_t topology) __hwloc_attribute_
  *
  * \return the CPU set of allowed logical processors of the system.
  *
- * \note If the topology flag ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM was not set,
+ * \note If the topology flag ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED was not set,
  * this is identical to hwloc_topology_get_topology_cpuset(), which means
  * all PUs are allowed.
  *
- * \note If ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM was set, applying
+ * \note If ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED was set, applying
  * hwloc_bitmap_intersects() on the result of this function and on an object
  * cpuset checks whether there are allowed PUs inside that object.
  * Applying hwloc_bitmap_and() returns the list of these allowed PUs.
@@ -945,7 +951,7 @@ hwloc_topology_get_allowed_cpuset(hwloc_topology_t topology) __hwloc_attribute_p
  * \note The returned nodeset is not newly allocated and should thus not be
  * changed or freed; hwloc_bitmap_dup() must be used to obtain a local copy.
  *
- * \note This is equivalent to retrieving the root object complete CPU-set.
+ * \note This is equivalent to retrieving the root object complete nodeset.
  */
 HWLOC_DECLSPEC hwloc_const_nodeset_t
 hwloc_topology_get_complete_nodeset(hwloc_topology_t topology) __hwloc_attribute_pure;
@@ -959,7 +965,7 @@ hwloc_topology_get_complete_nodeset(hwloc_topology_t topology) __hwloc_attribute
  * \note The returned nodeset is not newly allocated and should thus not be
  * changed or freed; hwloc_bitmap_dup() must be used to obtain a local copy.
  *
- * \note This is equivalent to retrieving the root object complete CPU-set.
+ * \note This is equivalent to retrieving the root object nodeset.
  */
 HWLOC_DECLSPEC hwloc_const_nodeset_t
 hwloc_topology_get_topology_nodeset(hwloc_topology_t topology) __hwloc_attribute_pure;
@@ -968,11 +974,11 @@ hwloc_topology_get_topology_nodeset(hwloc_topology_t topology) __hwloc_attribute
  *
  * \return the node set of allowed memory of the system.
  *
- * \note If the topology flag ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM was not set,
+ * \note If the topology flag ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED was not set,
  * this is identical to hwloc_topology_get_topology_nodeset(), which means
  * all NUMA nodes are allowed.
  *
- * \note If ::HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM was set, applying
+ * \note If ::HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED was set, applying
  * hwloc_bitmap_intersects() on the result of this function and on an object
  * nodeset checks whether there are allowed NUMA nodes inside that object.
  * Applying hwloc_bitmap_and() returns the list of these allowed NUMA nodes.

src/3rdparty/hwloc/include/hwloc/intel-mic.h

@@ -13,11 +13,13 @@
 #ifndef HWLOC_INTEL_MIC_H
 #define HWLOC_INTEL_MIC_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
+
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
+
 #include <dirent.h>
 #include <string.h>
 #endif

src/3rdparty/hwloc/include/hwloc/linux-libnuma.h

@@ -15,7 +15,8 @@
 #ifndef HWLOC_LINUX_LIBNUMA_H
 #define HWLOC_LINUX_LIBNUMA_H
 
-#include <hwloc.h>
+#include "hwloc.h"
+
 #include <numa.h>
 
 

src/3rdparty/hwloc/include/hwloc/linux.h

@@ -15,7 +15,8 @@
 #ifndef HWLOC_LINUX_H
 #define HWLOC_LINUX_H
 
-#include <hwloc.h>
+#include "hwloc.h"
+
 #include <stdio.h>
 
 

src/3rdparty/hwloc/include/hwloc/nvml.h

@@ -13,11 +13,11 @@
 #ifndef HWLOC_NVML_H
 #define HWLOC_NVML_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
 #endif
 
 #include <nvml.h>

src/3rdparty/hwloc/include/hwloc/opencl.h

@@ -14,19 +14,17 @@
 #ifndef HWLOC_OPENCL_H
 #define HWLOC_OPENCL_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
 #endif
 
 #ifdef __APPLE__
 #include <OpenCL/cl.h>
-#include <OpenCL/cl_ext.h>
 #else
 #include <CL/cl.h>
-#include <CL/cl_ext.h>
 #endif
 
 #include <stdio.h>
@@ -37,17 +35,75 @@ extern "C" {
 #endif
 
 
+/* OpenCL extensions aren't always shipped with default headers, and
+ * they don't always reflect what the installed implementations support.
+ * Try everything and let the implementation return errors when non supported.
+ */
+/* Copyright (c) 2008-2018 The Khronos Group Inc. */
+
+/* needs "cl_amd_device_attribute_query" device extension, but not strictly required for clGetDeviceInfo() */
+#define HWLOC_CL_DEVICE_TOPOLOGY_AMD 0x4037
+typedef union {
+    struct { cl_uint type; cl_uint data[5]; } raw;
+    struct { cl_uint type; cl_char unused[17]; cl_char bus; cl_char device; cl_char function; } pcie;
+} hwloc_cl_device_topology_amd;
+#define HWLOC_CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD 1
+
+/* needs "cl_nv_device_attribute_query" device extension, but not strictly required for clGetDeviceInfo() */
+#define HWLOC_CL_DEVICE_PCI_BUS_ID_NV 0x4008
+#define HWLOC_CL_DEVICE_PCI_SLOT_ID_NV 0x4009
+
+
 /** \defgroup hwlocality_opencl Interoperability with OpenCL
  *
  * This interface offers ways to retrieve topology information about
  * OpenCL devices.
  *
- * Only the AMD OpenCL interface currently offers useful locality information
+ * Only AMD and NVIDIA OpenCL implementations currently offer useful locality
- * about its devices.
+ * information about their devices.
  *
  * @{
  */
 
+/** \brief Return the domain, bus and device IDs of the OpenCL device \p device.
+ *
+ * Device \p device must match the local machine.
+ */
+static __hwloc_inline int
+hwloc_opencl_get_device_pci_busid(cl_device_id device,
+                               unsigned *domain, unsigned *bus, unsigned *dev, unsigned *func)
+{
+	hwloc_cl_device_topology_amd amdtopo;
+	cl_uint nvbus, nvslot;
+	cl_int clret;
+
+	clret = clGetDeviceInfo(device, HWLOC_CL_DEVICE_TOPOLOGY_AMD, sizeof(amdtopo), &amdtopo, NULL);
+	if (CL_SUCCESS == clret
+	    && HWLOC_CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD == amdtopo.raw.type) {
+		*domain = 0; /* can't do anything better */
+		*bus = (unsigned) amdtopo.pcie.bus;
+		*dev = (unsigned) amdtopo.pcie.device;
+		*func = (unsigned) amdtopo.pcie.function;
+		return 0;
+	}
+
+	clret = clGetDeviceInfo(device, HWLOC_CL_DEVICE_PCI_BUS_ID_NV, sizeof(nvbus), &nvbus, NULL);
+	if (CL_SUCCESS == clret) {
+		clret = clGetDeviceInfo(device, HWLOC_CL_DEVICE_PCI_SLOT_ID_NV, sizeof(nvslot), &nvslot, NULL);
+		if (CL_SUCCESS == clret) {
+			/* FIXME: PCI bus only uses 8bit, assume nvidia hardcodes the domain in higher bits */
+			*domain = nvbus >> 8;
+			*bus = nvbus & 0xff;
+			/* non-documented but used in many other projects */
+			*dev = nvslot >> 3;
+			*func = nvslot & 0x7;
+			return 0;
+		}
+	}
+
+	return -1;
+}
+
 /** \brief Get the CPU set of logical processors that are physically
  * close to OpenCL device \p device.
  *
@@ -62,7 +118,7 @@ extern "C" {
  * and hwloc_opencl_get_device_osdev_by_index().
  *
  * This function is currently only implemented in a meaningful way for
- * Linux with the AMD OpenCL implementation; other systems will simply
+ * Linux with the AMD or NVIDIA OpenCL implementation; other systems will simply
  * get a full cpuset.
  */
 static __hwloc_inline int
@@ -70,35 +126,28 @@ hwloc_opencl_get_device_cpuset(hwloc_topology_t topology __hwloc_attribute_unuse
 			       cl_device_id device __hwloc_attribute_unused,
 			       hwloc_cpuset_t set)
 {
-#if (defined HWLOC_LINUX_SYS) && (defined CL_DEVICE_TOPOLOGY_AMD)
+#if (defined HWLOC_LINUX_SYS)
-	/* If we're on Linux + AMD OpenCL, use the AMD extension + the sysfs mechanism to get the local cpus */
+	/* If we're on Linux, try AMD/NVIDIA extensions + the sysfs mechanism to get the local cpus */
 #define HWLOC_OPENCL_DEVICE_SYSFS_PATH_MAX 128
 	char path[HWLOC_OPENCL_DEVICE_SYSFS_PATH_MAX];
-	cl_device_topology_amd amdtopo;
+	unsigned pcidomain, pcibus, pcidev, pcifunc;
-	cl_int clret;
 
 	if (!hwloc_topology_is_thissystem(topology)) {
 		errno = EINVAL;
 		return -1;
 	}
 
-	clret = clGetDeviceInfo(device, CL_DEVICE_TOPOLOGY_AMD, sizeof(amdtopo), &amdtopo, NULL);
+	if (hwloc_opencl_get_device_pci_busid(device, &pcidomain, &pcibus, &pcidev, &pcifunc) < 0) {
-	if (CL_SUCCESS != clret) {
-		hwloc_bitmap_copy(set, hwloc_topology_get_complete_cpuset(topology));
-		return 0;
-	}
-	if (CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD != amdtopo.raw.type) {
 		hwloc_bitmap_copy(set, hwloc_topology_get_complete_cpuset(topology));
 		return 0;
 	}
 
-	sprintf(path, "/sys/bus/pci/devices/0000:%02x:%02x.%01x/local_cpus",
+	sprintf(path, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/local_cpus", pcidomain, pcibus, pcidev, pcifunc);
-		(unsigned) amdtopo.pcie.bus, (unsigned) amdtopo.pcie.device, (unsigned) amdtopo.pcie.function);
 	if (hwloc_linux_read_path_as_cpumask(path, set) < 0
 	    || hwloc_bitmap_iszero(set))
 		hwloc_bitmap_copy(set, hwloc_topology_get_complete_cpuset(topology));
 #else
-	/* Non-Linux + AMD OpenCL systems simply get a full cpuset */
+	/* Non-Linux systems simply get a full cpuset */
 	hwloc_bitmap_copy(set, hwloc_topology_get_complete_cpuset(topology));
 #endif
   return 0;
@@ -140,8 +189,8 @@ hwloc_opencl_get_device_osdev_by_index(hwloc_topology_t topology,
  * Use OpenCL device attributes to find the corresponding hwloc OS device object.
  * Return NULL if there is none or if useful attributes are not available.
  *
- * This function currently only works on AMD OpenCL devices that support
+ * This function currently only works on AMD and NVIDIA OpenCL devices that support
- * the CL_DEVICE_TOPOLOGY_AMD extension. hwloc_opencl_get_device_osdev_by_index()
+ * relevant OpenCL extensions. hwloc_opencl_get_device_osdev_by_index()
  * should be preferred whenever possible, i.e. when platform and device index
  * are known.
  *
@@ -159,17 +208,10 @@ static __hwloc_inline hwloc_obj_t
 hwloc_opencl_get_device_osdev(hwloc_topology_t topology __hwloc_attribute_unused,
 			      cl_device_id device __hwloc_attribute_unused)
 {
-#ifdef CL_DEVICE_TOPOLOGY_AMD
 	hwloc_obj_t osdev;
-	cl_device_topology_amd amdtopo;
+	unsigned pcidomain, pcibus, pcidevice, pcifunc;
-	cl_int clret;
 
-	clret = clGetDeviceInfo(device, CL_DEVICE_TOPOLOGY_AMD, sizeof(amdtopo), &amdtopo, NULL);
+	if (hwloc_opencl_get_device_pci_busid(device, &pcidomain, &pcibus, &pcidevice, &pcifunc) < 0) {
-	if (CL_SUCCESS != clret) {
-		errno = EINVAL;
-		return NULL;
-	}
-	if (CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD != amdtopo.raw.type) {
 		errno = EINVAL;
 		return NULL;
 	}
@@ -181,18 +223,15 @@ hwloc_opencl_get_device_osdev(hwloc_topology_t topology __hwloc_attribute_unused
 			continue;
 		if (pcidev
 		    && pcidev->type == HWLOC_OBJ_PCI_DEVICE
-		    && pcidev->attr->pcidev.domain == 0
+		    && pcidev->attr->pcidev.domain == pcidomain
-		    && pcidev->attr->pcidev.bus == amdtopo.pcie.bus
+		    && pcidev->attr->pcidev.bus == pcibus
-		    && pcidev->attr->pcidev.dev == amdtopo.pcie.device
+		    && pcidev->attr->pcidev.dev == pcidevice
-		    && pcidev->attr->pcidev.func == amdtopo.pcie.function)
+		    && pcidev->attr->pcidev.func == pcifunc)
 			return osdev;
 		/* if PCI are filtered out, we need a info attr to match on */
 	}
 
 	return NULL;
-#else
-	return NULL;
-#endif
 }
 
 /** @} */

src/3rdparty/hwloc/include/hwloc/openfabrics-verbs.h

@@ -19,10 +19,10 @@
 #ifndef HWLOC_OPENFABRICS_VERBS_H
 #define HWLOC_OPENFABRICS_VERBS_H
 
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
 #ifdef HWLOC_LINUX_SYS
-#include <hwloc/linux.h>
+#include "hwloc/linux.h"
 #endif
 
 #include <infiniband/verbs.h>

src/3rdparty/hwloc/include/hwloc/plugins.h

@@ -1,5 +1,5 @@
 /*
- * Copyright © 2013-2017 Inria.  All rights reserved.
+ * Copyright © 2013-2019 Inria.  All rights reserved.
  * Copyright © 2016 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
@@ -13,7 +13,8 @@
 
 struct hwloc_backend;
 
-#include <hwloc.h>
+#include "hwloc.h"
+
 #ifdef HWLOC_INSIDE_PLUGIN
 /* needed for hwloc_plugin_check_namespace() */
 #include <ltdl.h>
@@ -25,52 +26,36 @@ struct hwloc_backend;
  * @{
  */
 
-/** \brief Discovery component type */
-typedef enum hwloc_disc_component_type_e {
-  /** \brief CPU-only discovery through the OS, or generic no-OS support.
-   * \hideinitializer */
-  HWLOC_DISC_COMPONENT_TYPE_CPU = (1<<0),
-
-  /** \brief xml or synthetic,
-   * platform-specific components such as bgq.
-   * Anything the discovers CPU and everything else.
-   * No misc backend is expected to complement a global component.
-   * \hideinitializer */
-  HWLOC_DISC_COMPONENT_TYPE_GLOBAL = (1<<1),
-
-  /** \brief OpenCL, Cuda, etc.
-   * \hideinitializer */
-  HWLOC_DISC_COMPONENT_TYPE_MISC = (1<<2)
-} hwloc_disc_component_type_t;
-
 /** \brief Discovery component structure
  *
  * This is the major kind of components, taking care of the discovery.
  * They are registered by generic components, either statically-built or as plugins.
  */
 struct hwloc_disc_component {
-  /** \brief Discovery component type */
-  hwloc_disc_component_type_t type;
-
   /** \brief Name.
    * If this component is built as a plugin, this name does not have to match the plugin filename.
    */
   const char *name;
 
-  /** \brief Component types to exclude, as an OR'ed set of ::hwloc_disc_component_type_e.
+  /** \brief Discovery phases performed by this component.
+   * OR'ed set of ::hwloc_disc_phase_t
+   */
+  unsigned phases;
+
+  /** \brief Component phases to exclude, as an OR'ed set of ::hwloc_disc_phase_t.
    *
-   * For a GLOBAL component, this usually includes all other types (~0).
+   * For a GLOBAL component, this usually includes all other phases (\c ~UL).
    *
    * Other components only exclude types that may bring conflicting
    * topology information. MISC components should likely not be excluded
    * since they usually bring non-primary additional information.
    */
-  unsigned excludes;
+  unsigned excluded_phases;
 
   /** \brief Instantiate callback to create a backend from the component.
    * Parameters data1, data2, data3 are NULL except for components
    * that have special enabling routines such as hwloc_topology_set_xml(). */
-  struct hwloc_backend * (*instantiate)(struct hwloc_disc_component *component, const void *data1, const void *data2, const void *data3);
+  struct hwloc_backend * (*instantiate)(struct hwloc_topology *topology, struct hwloc_disc_component *component, unsigned excluded_phases, const void *data1, const void *data2, const void *data3);
 
   /** \brief Component priority.
    * Used to sort topology->components, higher priority first.
@@ -107,6 +92,72 @@ struct hwloc_disc_component {
  * @{
  */
 
+/** \brief Discovery phase */
+typedef enum hwloc_disc_phase_e {
+  /** \brief xml or synthetic, platform-specific components such as bgq.
+   * Discovers everything including CPU, memory, I/O and everything else.
+   * A component with a Global phase usually excludes all other phases.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_GLOBAL = (1U<<0),
+
+  /** \brief CPU discovery.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_CPU = (1U<<1),
+
+  /** \brief Attach memory to existing CPU objects.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_MEMORY = (1U<<2),
+
+  /** \brief Attach PCI devices and bridges to existing CPU objects.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_PCI = (1U<<3),
+
+  /** \brief I/O discovery that requires PCI devices (OS devices such as OpenCL, CUDA, etc.).
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_IO = (1U<<4),
+
+  /** \brief Misc objects that gets added below anything else.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_MISC = (1U<<5),
+
+  /** \brief Annotating existing objects, adding distances, etc.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_ANNOTATE = (1U<<6),
+
+  /** \brief Final tweaks to a ready-to-use topology.
+   * This phase runs once the topology is loaded, before it is returned to the topology.
+   * Hence it may only use the main hwloc API for modifying the topology,
+   * for instance by restricting it, adding info attributes, etc.
+   * \hideinitializer */
+  HWLOC_DISC_PHASE_TWEAK = (1U<<7)
+} hwloc_disc_phase_t;
+
+/** \brief Discovery status flags */
+enum hwloc_disc_status_flag_e {
+  /** \brief The sets of allowed resources were already retrieved \hideinitializer */
+  HWLOC_DISC_STATUS_FLAG_GOT_ALLOWED_RESOURCES = (1UL<<1)
+};
+
+/** \brief Discovery status structure
+ *
+ * Used by the core and backends to inform about what has been/is being done
+ * during the discovery process.
+ */
+struct hwloc_disc_status {
+  /** \brief The current discovery phase that is performed.
+   * Must match one of the phases in the component phases field.
+   */
+  hwloc_disc_phase_t phase;
+
+  /** \brief Dynamically excluded phases.
+   * If a component decides during discovery that some phases are no longer needed.
+   */
+  unsigned excluded_phases;
+
+  /** \brief OR'ed set of hwloc_disc_status_flag_e */
+  unsigned long flags;
+};
+
 /** \brief Discovery backend structure
  *
  * A backend is the instantiation of a discovery component.
@@ -116,6 +167,14 @@ struct hwloc_disc_component {
  * hwloc_backend_alloc() initializes all fields to default values
  * that the component may change (except "component" and "next")
  * before enabling the backend with hwloc_backend_enable().
+ *
+ * Most backends assume that the topology is_thissystem flag is
+ * set because they talk to the underlying operating system.
+ * However they may still be used in topologies without the
+ * is_thissystem flag for debugging reasons.
+ * In practice, they are usually auto-disabled in such cases
+ * (excluded by xml or synthetic backends, or by environment
+ *  variables when changing the Linux fsroot or the x86 cpuid path).
  */
 struct hwloc_backend {
   /** \private Reserved for the core, set by hwloc_backend_alloc() */
@@ -127,12 +186,20 @@ struct hwloc_backend {
   /** \private Reserved for the core. Used internally to list backends topology->backends. */
   struct hwloc_backend * next;
 
+  /** \brief Discovery phases performed by this component, possibly without some of them if excluded by other components.
+   * OR'ed set of ::hwloc_disc_phase_t
+   */
+  unsigned phases;
+
   /** \brief Backend flags, currently always 0. */
   unsigned long flags;
 
   /** \brief Backend-specific 'is_thissystem' property.
-   * Set to 0 or 1 if the backend should enforce the thissystem flag when it gets enabled.
+   * Set to 0 if the backend disables the thissystem flag for this topology
-   * Set to -1 if the backend doesn't care (default). */
+   * (e.g. loading from xml or synthetic string,
+   *  or using a different fsroot on Linux, or a x86 CPUID dump).
+   * Set to -1 if the backend doesn't care (default).
+   */
   int is_thissystem;
 
   /** \brief Backend private data, or NULL if none. */
@@ -147,20 +214,22 @@ struct hwloc_backend {
    * or because of an actual discovery/gathering failure.
    * May be NULL.
    */
-  int (*discover)(struct hwloc_backend *backend);
+  int (*discover)(struct hwloc_backend *backend, struct hwloc_disc_status *status);
 
-  /** \brief Callback used by the PCI backend to retrieve the locality of a PCI object from the OS/cpu backend.
+  /** \brief Callback to retrieve the locality of a PCI object.
-   * May be NULL. */
+   * Called by the PCI core when attaching PCI hierarchy to CPU objects.
+   * May be NULL.
+   */
   int (*get_pci_busid_cpuset)(struct hwloc_backend *backend, struct hwloc_pcidev_attr_s *busid, hwloc_bitmap_t cpuset);
 };
 
 /** \brief Allocate a backend structure, set good default values, initialize backend->component and topology, etc.
  * The caller will then modify whatever needed, and call hwloc_backend_enable().
  */
-HWLOC_DECLSPEC struct hwloc_backend * hwloc_backend_alloc(struct hwloc_disc_component *component);
+HWLOC_DECLSPEC struct hwloc_backend * hwloc_backend_alloc(struct hwloc_topology *topology, struct hwloc_disc_component *component);
 
 /** \brief Enable a previously allocated and setup backend. */
-HWLOC_DECLSPEC int hwloc_backend_enable(struct hwloc_topology *topology, struct hwloc_backend *backend);
+HWLOC_DECLSPEC int hwloc_backend_enable(struct hwloc_backend *backend);
 
 /** @} */
 
@@ -480,7 +549,9 @@ HWLOC_DECLSPEC hwloc_obj_type_t hwloc_pcidisc_check_bridge_type(unsigned device_
  *
  * Returns -1 and destroys /p obj if bridge fields are invalid.
  */
-HWLOC_DECLSPEC int hwloc_pcidisc_setup_bridge_attr(hwloc_obj_t obj, const unsigned char *config);
+HWLOC_DECLSPEC int hwloc_pcidisc_find_bridge_buses(unsigned domain, unsigned bus, unsigned dev, unsigned func,
+						   unsigned *secondary_busp, unsigned *subordinate_busp,
+						   const unsigned char *config);
 
 /** \brief Insert a PCI object in the given PCI tree by looking at PCI bus IDs.
  *
@@ -490,10 +561,7 @@ HWLOC_DECLSPEC void hwloc_pcidisc_tree_insert_by_busid(struct hwloc_obj **treep,
 
 /** \brief Add some hostbridges on top of the given tree of PCI objects and attach them to the topology.
  *
- * For now, they will be attached to the root object. The core will move them to their actual PCI
+ * Other backends may lookup PCI objects or localities (for instance to attach OS devices)
- * locality using hwloc_pci_belowroot_apply_locality() at the end of the discovery.
- *
- * In the meantime, other backends lookup PCI objects or localities (for instance to attach OS devices)
  * by using hwloc_pcidisc_find_by_busid() or hwloc_pcidisc_find_busid_parent().
  */
 HWLOC_DECLSPEC int hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, struct hwloc_obj *tree);
@@ -507,32 +575,14 @@ HWLOC_DECLSPEC int hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, st
  * @{
  */
 
-/** \brief Find the PCI object that matches the bus ID.
- *
- * To be used after a PCI backend added PCI devices with hwloc_pcidisc_tree_attach()
- * and before the core moves them to their actual location with hwloc_pci_belowroot_apply_locality().
- *
- * If no exactly matching object is found, return the container bridge if any, or NULL.
- *
- * On failure, it may be possible to find the PCI locality (instead of the PCI device)
- * by calling hwloc_pcidisc_find_busid_parent().
- *
- * \note This is semantically identical to hwloc_get_pcidev_by_busid() which only works
- * after the topology is fully loaded.
- */
-HWLOC_DECLSPEC struct hwloc_obj * hwloc_pcidisc_find_by_busid(struct hwloc_topology *topology, unsigned domain, unsigned bus, unsigned dev, unsigned func);
-
 /** \brief Find the normal parent of a PCI bus ID.
  *
  * Look at PCI affinity to find out where the given PCI bus ID should be attached.
  *
- * This function should be used to attach an I/O device directly under a normal
+ * This function should be used to attach an I/O device under the corresponding
- * (non-I/O) object, instead of below a PCI object.
+ * PCI object (if any), or under a normal (non-I/O) object with same locality.
- * It is usually used by backends when hwloc_pcidisc_find_by_busid() failed
- * to find the hwloc object corresponding to this bus ID, for instance because
- * PCI discovery is not supported on this platform.
  */
-HWLOC_DECLSPEC struct hwloc_obj * hwloc_pcidisc_find_busid_parent(struct hwloc_topology *topology, unsigned domain, unsigned bus, unsigned dev, unsigned func);
+HWLOC_DECLSPEC struct hwloc_obj * hwloc_pci_find_parent_by_busid(struct hwloc_topology *topology, unsigned domain, unsigned bus, unsigned dev, unsigned func);
 
 /** @} */
 

src/3rdparty/hwloc/include/hwloc/rename.h

@@ -1,13 +1,13 @@
 /*
  * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
- * Copyright © 2010-2018 Inria.  All rights reserved.
+ * Copyright © 2010-2019 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
 #ifndef HWLOC_RENAME_H
 #define HWLOC_RENAME_H
 
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
 
 
 #ifdef __cplusplus
@@ -49,7 +49,9 @@ extern "C" {
 
 #define HWLOC_OBJ_MACHINE HWLOC_NAME_CAPS(OBJ_MACHINE)
 #define HWLOC_OBJ_NUMANODE HWLOC_NAME_CAPS(OBJ_NUMANODE)
+#define HWLOC_OBJ_MEMCACHE HWLOC_NAME_CAPS(OBJ_MEMCACHE)
 #define HWLOC_OBJ_PACKAGE HWLOC_NAME_CAPS(OBJ_PACKAGE)
+#define HWLOC_OBJ_DIE HWLOC_NAME_CAPS(OBJ_DIE)
 #define HWLOC_OBJ_CORE HWLOC_NAME_CAPS(OBJ_CORE)
 #define HWLOC_OBJ_PU HWLOC_NAME_CAPS(OBJ_PU)
 #define HWLOC_OBJ_L1CACHE HWLOC_NAME_CAPS(OBJ_L1CACHE)
@@ -116,7 +118,7 @@ extern "C" {
 
 #define hwloc_topology_flags_e HWLOC_NAME(topology_flags_e)
 
-#define HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM HWLOC_NAME_CAPS(TOPOLOGY_FLAG_WHOLE_SYSTEM)
+#define HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED HWLOC_NAME_CAPS(TOPOLOGY_FLAG_WITH_DISALLOWED)
 #define HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM HWLOC_NAME_CAPS(TOPOLOGY_FLAG_IS_THISSYSTEM)
 #define HWLOC_TOPOLOGY_FLAG_THISSYSTEM_ALLOWED_RESOURCES HWLOC_NAME_CAPS(TOPOLOGY_FLAG_THISSYSTEM_ALLOWED_RESOURCES)
 
@@ -124,6 +126,9 @@ extern "C" {
 #define hwloc_topology_set_synthetic HWLOC_NAME(topology_set_synthetic)
 #define hwloc_topology_set_xml HWLOC_NAME(topology_set_xml)
 #define hwloc_topology_set_xmlbuffer HWLOC_NAME(topology_set_xmlbuffer)
+#define hwloc_topology_components_flag_e HWLOC_NAME(hwloc_topology_components_flag_e)
+#define HWLOC_TOPOLOGY_COMPONENTS_FLAG_BLACKLIST HWLOC_NAME_CAPS(TOPOLOGY_COMPONENTS_FLAG_BLACKLIST)
+#define hwloc_topology_set_components HWLOC_NAME(topology_set_components)
 
 #define hwloc_topology_set_flags HWLOC_NAME(topology_set_flags)
 #define hwloc_topology_is_thissystem HWLOC_NAME(topology_is_thissystem)
@@ -151,10 +156,18 @@ extern "C" {
 
 #define hwloc_restrict_flags_e HWLOC_NAME(restrict_flags_e)
 #define HWLOC_RESTRICT_FLAG_REMOVE_CPULESS HWLOC_NAME_CAPS(RESTRICT_FLAG_REMOVE_CPULESS)
+#define HWLOC_RESTRICT_FLAG_BYNODESET HWLOC_NAME_CAPS(RESTRICT_FLAG_BYNODESET)
+#define HWLOC_RESTRICT_FLAG_REMOVE_MEMLESS HWLOC_NAME_CAPS(RESTRICT_FLAG_REMOVE_MEMLESS)
 #define HWLOC_RESTRICT_FLAG_ADAPT_MISC HWLOC_NAME_CAPS(RESTRICT_FLAG_ADAPT_MISC)
 #define HWLOC_RESTRICT_FLAG_ADAPT_IO HWLOC_NAME_CAPS(RESTRICT_FLAG_ADAPT_IO)
 #define hwloc_topology_restrict HWLOC_NAME(topology_restrict)
 
+#define hwloc_allow_flags_e HWLOC_NAME(allow_flags_e)
+#define HWLOC_ALLOW_FLAG_ALL HWLOC_NAME_CAPS(ALLOW_FLAG_ALL)
+#define HWLOC_ALLOW_FLAG_LOCAL_RESTRICTIONS HWLOC_NAME_CAPS(ALLOW_FLAG_LOCAL_RESTRICTIONS)
+#define HWLOC_ALLOW_FLAG_CUSTOM HWLOC_NAME_CAPS(ALLOW_FLAG_CUSTOM)
+#define hwloc_topology_allow HWLOC_NAME(topology_allow)
+
 #define hwloc_topology_insert_misc_object HWLOC_NAME(topology_insert_misc_object)
 #define hwloc_topology_alloc_group_object HWLOC_NAME(topology_alloc_group_object)
 #define hwloc_topology_insert_group_object HWLOC_NAME(topology_insert_group_object)
@@ -172,6 +185,7 @@ extern "C" {
 #define HWLOC_TYPE_DEPTH_OS_DEVICE HWLOC_NAME_CAPS(TYPE_DEPTH_OS_DEVICE)
 #define HWLOC_TYPE_DEPTH_MISC HWLOC_NAME_CAPS(TYPE_DEPTH_MISC)
 #define HWLOC_TYPE_DEPTH_NUMANODE HWLOC_NAME_CAPS(TYPE_DEPTH_NUMANODE)
+#define HWLOC_TYPE_DEPTH_MEMCACHE HWLOC_NAME_CAPS(TYPE_DEPTH_MEMCACHE)
 
 #define hwloc_get_depth_type HWLOC_NAME(get_depth_type)
 #define hwloc_get_nbobjs_by_depth HWLOC_NAME(get_nbobjs_by_depth)
@@ -266,10 +280,12 @@ extern "C" {
 #define hwloc_bitmap_zero HWLOC_NAME(bitmap_zero)
 #define hwloc_bitmap_fill HWLOC_NAME(bitmap_fill)
 #define hwloc_bitmap_from_ulong HWLOC_NAME(bitmap_from_ulong)
-
+#define hwloc_bitmap_from_ulongs HWLOC_NAME(bitmap_from_ulongs)
 #define hwloc_bitmap_from_ith_ulong HWLOC_NAME(bitmap_from_ith_ulong)
 #define hwloc_bitmap_to_ulong HWLOC_NAME(bitmap_to_ulong)
 #define hwloc_bitmap_to_ith_ulong HWLOC_NAME(bitmap_to_ith_ulong)
+#define hwloc_bitmap_to_ulongs HWLOC_NAME(bitmap_to_ulongs)
+#define hwloc_bitmap_nr_ulongs HWLOC_NAME(bitmap_nr_ulongs)
 #define hwloc_bitmap_only HWLOC_NAME(bitmap_only)
 #define hwloc_bitmap_allbut HWLOC_NAME(bitmap_allbut)
 #define hwloc_bitmap_set HWLOC_NAME(bitmap_set)
@@ -380,10 +396,13 @@ extern "C" {
 #define HWLOC_DISTANCES_KIND_FROM_USER HWLOC_NAME_CAPS(DISTANCES_KIND_FROM_USER)
 #define HWLOC_DISTANCES_KIND_MEANS_LATENCY HWLOC_NAME_CAPS(DISTANCES_KIND_MEANS_LATENCY)
 #define HWLOC_DISTANCES_KIND_MEANS_BANDWIDTH HWLOC_NAME_CAPS(DISTANCES_KIND_MEANS_BANDWIDTH)
+#define HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES HWLOC_NAME_CAPS(DISTANCES_KIND_HETEROGENEOUS_TYPES)
 
 #define hwloc_distances_get HWLOC_NAME(distances_get)
 #define hwloc_distances_get_by_depth HWLOC_NAME(distances_get_by_depth)
 #define hwloc_distances_get_by_type HWLOC_NAME(distances_get_by_type)
+#define hwloc_distances_get_by_name HWLOC_NAME(distances_get_by_name)
+#define hwloc_distances_get_name HWLOC_NAME(distances_get_name)
 #define hwloc_distances_release HWLOC_NAME(distances_release)
 #define hwloc_distances_obj_index HWLOC_NAME(distances_obj_index)
 #define hwloc_distances_obj_pair_values HWLOC_NAME(distances_pair_values)
@@ -396,6 +415,7 @@ extern "C" {
 #define hwloc_distances_remove HWLOC_NAME(distances_remove)
 #define hwloc_distances_remove_by_depth HWLOC_NAME(distances_remove_by_depth)
 #define hwloc_distances_remove_by_type HWLOC_NAME(distances_remove_by_type)
+#define hwloc_distances_release_remove HWLOC_NAME(distances_release_remove)
 
 /* diff.h */
 
@@ -469,6 +489,8 @@ extern "C" {
 
 /* opencl.h */
 
+#define hwloc_cl_device_topology_amd HWLOC_NAME(cl_device_topology_amd)
+#define hwloc_opencl_get_device_pci_busid HWLOC_NAME(opencl_get_device_pci_ids)
 #define hwloc_opencl_get_device_cpuset HWLOC_NAME(opencl_get_device_cpuset)
 #define hwloc_opencl_get_device_osdev HWLOC_NAME(opencl_get_device_osdev)
 #define hwloc_opencl_get_device_osdev_by_index HWLOC_NAME(opencl_get_device_osdev_by_index)
@@ -502,13 +524,22 @@ extern "C" {
 
 /* hwloc/plugins.h */
 
-#define hwloc_disc_component_type_e HWLOC_NAME(disc_component_type_e)
+#define hwloc_disc_phase_e HWLOC_NAME(disc_phase_e)
-#define HWLOC_DISC_COMPONENT_TYPE_CPU HWLOC_NAME_CAPS(DISC_COMPONENT_TYPE_CPU)
+#define HWLOC_DISC_PHASE_GLOBAL HWLOC_NAME_CAPS(DISC_PHASE_GLOBAL)
-#define HWLOC_DISC_COMPONENT_TYPE_GLOBAL HWLOC_NAME_CAPS(DISC_COMPONENT_TYPE_GLOBAL)
+#define HWLOC_DISC_PHASE_CPU HWLOC_NAME_CAPS(DISC_PHASE_CPU)
-#define HWLOC_DISC_COMPONENT_TYPE_MISC HWLOC_NAME_CAPS(DISC_COMPONENT_TYPE_MISC)
+#define HWLOC_DISC_PHASE_MEMORY HWLOC_NAME_CAPS(DISC_PHASE_MEMORY)
-#define hwloc_disc_component_type_t HWLOC_NAME(disc_component_type_t)
+#define HWLOC_DISC_PHASE_PCI HWLOC_NAME_CAPS(DISC_PHASE_PCI)
+#define HWLOC_DISC_PHASE_IO HWLOC_NAME_CAPS(DISC_PHASE_IO)
+#define HWLOC_DISC_PHASE_MISC HWLOC_NAME_CAPS(DISC_PHASE_MISC)
+#define HWLOC_DISC_PHASE_ANNOTATE HWLOC_NAME_CAPS(DISC_PHASE_ANNOTATE)
+#define HWLOC_DISC_PHASE_TWEAK HWLOC_NAME_CAPS(DISC_PHASE_TWEAK)
+#define hwloc_disc_phase_t HWLOC_NAME(disc_phase_t)
 #define hwloc_disc_component HWLOC_NAME(disc_component)
 
+#define hwloc_disc_status_flag_e HWLOC_NAME(disc_status_flag_e)
+#define HWLOC_DISC_STATUS_FLAG_GOT_ALLOWED_RESOURCES HWLOC_NAME_CAPS(DISC_STATUS_FLAG_GOT_ALLOWED_RESOURCES)
+#define hwloc_disc_status HWLOC_NAME(disc_status)
+
 #define hwloc_backend HWLOC_NAME(backend)
 
 #define hwloc_backend_alloc HWLOC_NAME(backend_alloc)
@@ -540,12 +571,11 @@ extern "C" {
 #define hwloc_pcidisc_find_cap HWLOC_NAME(pcidisc_find_cap)
 #define hwloc_pcidisc_find_linkspeed HWLOC_NAME(pcidisc_find_linkspeed)
 #define hwloc_pcidisc_check_bridge_type HWLOC_NAME(pcidisc_check_bridge_type)
-#define hwloc_pcidisc_setup_bridge_attr HWLOC_NAME(pcidisc_setup_bridge_attr)
+#define hwloc_pcidisc_find_bridge_buses HWLOC_NAME(pcidisc_find_bridge_buses)
 #define hwloc_pcidisc_tree_insert_by_busid HWLOC_NAME(pcidisc_tree_insert_by_busid)
 #define hwloc_pcidisc_tree_attach HWLOC_NAME(pcidisc_tree_attach)
 
-#define hwloc_pcidisc_find_by_busid HWLOC_NAME(pcidisc_find_by_busid)
+#define hwloc_pci_find_parent_by_busid HWLOC_NAME(pcidisc_find_busid_parent)
-#define hwloc_pcidisc_find_busid_parent HWLOC_NAME(pcidisc_find_busid_parent)
 
 /* hwloc/deprecated.h */
 
@@ -571,8 +601,9 @@ extern "C" {
 
 /* private/misc.h */
 
+#ifndef HWLOC_HAVE_CORRECT_SNPRINTF
 #define hwloc_snprintf HWLOC_NAME(snprintf)
-#define hwloc_namecoloncmp HWLOC_NAME(namecoloncmp)
+#endif
 #define hwloc_ffsl_manual HWLOC_NAME(ffsl_manual)
 #define hwloc_ffs32 HWLOC_NAME(ffs32)
 #define hwloc_ffsl_from_ffs32 HWLOC_NAME(ffsl_from_ffs32)
@@ -631,8 +662,9 @@ extern "C" {
 #define hwloc_backends_is_thissystem HWLOC_NAME(backends_is_thissystem)
 #define hwloc_backends_find_callbacks HWLOC_NAME(backends_find_callbacks)
 
-#define hwloc_backends_init HWLOC_NAME(backends_init)
+#define hwloc_topology_components_init HWLOC_NAME(topology_components_init)
 #define hwloc_backends_disable_all HWLOC_NAME(backends_disable_all)
+#define hwloc_topology_components_fini HWLOC_NAME(topology_components_fini)
 
 #define hwloc_components_init HWLOC_NAME(components_init)
 #define hwloc_components_fini HWLOC_NAME(components_fini)
@@ -656,7 +688,6 @@ extern "C" {
 
 #define hwloc_cuda_component HWLOC_NAME(cuda_component)
 #define hwloc_gl_component HWLOC_NAME(gl_component)
-#define hwloc_linuxio_component HWLOC_NAME(linuxio_component)
 #define hwloc_nvml_component HWLOC_NAME(nvml_component)
 #define hwloc_opencl_component HWLOC_NAME(opencl_component)
 #define hwloc_pci_component HWLOC_NAME(pci_component)
@@ -669,6 +700,9 @@ extern "C" {
 #define hwloc_special_level_s HWLOC_NAME(special_level_s)
 
 #define hwloc_pci_forced_locality_s HWLOC_NAME(pci_forced_locality_s)
+#define hwloc_pci_locality_s HWLOC_NAME(pci_locality_s)
+
+#define hwloc_topology_forced_component_s HWLOC_NAME(topology_forced_component)
 
 #define hwloc_alloc_root_sets HWLOC_NAME(alloc_root_sets)
 #define hwloc_setup_pu_level HWLOC_NAME(setup_pu_level)
@@ -687,8 +721,8 @@ extern "C" {
 #define hwloc_pci_discovery_init HWLOC_NAME(pci_discovery_init)
 #define hwloc_pci_discovery_prepare HWLOC_NAME(pci_discovery_prepare)
 #define hwloc_pci_discovery_exit HWLOC_NAME(pci_discovery_exit)
+#define hwloc_pci_find_by_busid HWLOC_NAME(pcidisc_find_by_busid)
 #define hwloc_find_insert_io_parent_by_complete_cpuset HWLOC_NAME(hwloc_find_insert_io_parent_by_complete_cpuset)
-#define hwloc_pci_belowroot_apply_locality HWLOC_NAME(pci_belowroot_apply_locality)
 
 #define hwloc__add_info HWLOC_NAME(_add_info)
 #define hwloc__add_info_nodup HWLOC_NAME(_add_info_nodup)

src/3rdparty/hwloc/include/hwloc/shmem.h

@@ -10,7 +10,7 @@
 #ifndef HWLOC_SHMEM_H
 #define HWLOC_SHMEM_H
 
-#include <hwloc.h>
+#include "hwloc.h"
 
 #ifdef __cplusplus
 extern "C" {

src/3rdparty/hwloc/include/private/components.h

@@ -1,5 +1,5 @@
 /*
- * Copyright © 2012-2015 Inria.  All rights reserved.
+ * Copyright © 2012-2019 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
@@ -16,13 +16,13 @@
 #ifndef PRIVATE_COMPONENTS_H
 #define PRIVATE_COMPONENTS_H 1
 
-#include <hwloc/plugins.h>
+#include "hwloc/plugins.h"
 
 struct hwloc_topology;
 
 extern int hwloc_disc_component_force_enable(struct hwloc_topology *topology,
 					     int envvar_forced, /* 1 if forced through envvar, 0 if forced through API */
-					     int type, const char *name,
+					     const char *name,
 					     const void *data1, const void *data2, const void *data3);
 extern void hwloc_disc_components_enable_others(struct hwloc_topology *topology);
 
@@ -30,10 +30,12 @@ extern void hwloc_disc_components_enable_others(struct hwloc_topology *topology)
 extern void hwloc_backends_is_thissystem(struct hwloc_topology *topology);
 extern void hwloc_backends_find_callbacks(struct hwloc_topology *topology);
 
-/* Initialize the list of backends used by a topology */
+/* Initialize the lists of components and backends used by a topology */
-extern void hwloc_backends_init(struct hwloc_topology *topology);
+extern void hwloc_topology_components_init(struct hwloc_topology *topology);
 /* Disable and destroy all backends used by a topology */
 extern void hwloc_backends_disable_all(struct hwloc_topology *topology);
+/* Cleanup the lists of components used by a topology */
+extern void hwloc_topology_components_fini(struct hwloc_topology *topology);
 
 /* Used by the core to setup/destroy the list of components */
 extern void hwloc_components_init(void); /* increases components refcount, should be called exactly once per topology (during init) */

src/3rdparty/hwloc/include/private/debug.h

@@ -11,8 +11,8 @@
 #ifndef HWLOC_DEBUG_H
 #define HWLOC_DEBUG_H
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 #ifdef HWLOC_DEBUG
 #include <stdarg.h>

src/3rdparty/hwloc/include/private/internal-components.h

@@ -1,5 +1,5 @@
 /*
- * Copyright © 2018 Inria.  All rights reserved.
+ * Copyright © 2018-2019 Inria.  All rights reserved.
  *
  * See COPYING in top-level directory.
  */
@@ -29,7 +29,6 @@ HWLOC_DECLSPEC extern const struct hwloc_component hwloc_x86_component;
 /* I/O discovery */
 HWLOC_DECLSPEC extern const struct hwloc_component hwloc_cuda_component;
 HWLOC_DECLSPEC extern const struct hwloc_component hwloc_gl_component;
-HWLOC_DECLSPEC extern const struct hwloc_component hwloc_linuxio_component;
 HWLOC_DECLSPEC extern const struct hwloc_component hwloc_nvml_component;
 HWLOC_DECLSPEC extern const struct hwloc_component hwloc_opencl_component;
 HWLOC_DECLSPEC extern const struct hwloc_component hwloc_pci_component;

src/3rdparty/hwloc/include/private/misc.h

@@ -1,6 +1,6 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2012 Université Bordeaux
  * Copyright © 2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
@@ -11,9 +11,9 @@
 #ifndef HWLOC_PRIVATE_MISC_H
 #define HWLOC_PRIVATE_MISC_H
 
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
 
 #ifdef HWLOC_HAVE_DECL_STRNCASECMP
 #ifdef HAVE_STRINGS_H
@@ -439,14 +439,14 @@ hwloc_linux_pci_link_speed_from_string(const char *string)
 static __hwloc_inline int hwloc__obj_type_is_normal (hwloc_obj_type_t type)
 {
   /* type contiguity is asserted in topology_check() */
-  return type <= HWLOC_OBJ_GROUP;
+  return type <= HWLOC_OBJ_GROUP || type == HWLOC_OBJ_DIE;
 }
 
-/* Any object attached to memory children, currently only NUMA nodes */
+/* Any object attached to memory children, currently NUMA nodes or Memory-side caches */
 static __hwloc_inline int hwloc__obj_type_is_memory (hwloc_obj_type_t type)
 {
   /* type contiguity is asserted in topology_check() */
-  return type == HWLOC_OBJ_NUMANODE;
+  return type == HWLOC_OBJ_NUMANODE || type == HWLOC_OBJ_MEMCACHE;
 }
 
 /* I/O or Misc object, without cpusets or nodesets. */
@@ -463,6 +463,7 @@ static __hwloc_inline int hwloc__obj_type_is_io (hwloc_obj_type_t type)
   return type >= HWLOC_OBJ_BRIDGE && type <= HWLOC_OBJ_OS_DEVICE;
 }
 
+/* Any CPU caches (not Memory-side caches) */
 static __hwloc_inline int
 hwloc__obj_type_is_cache(hwloc_obj_type_t type)
 {
@@ -572,12 +573,4 @@ typedef SSIZE_T ssize_t;
 #  endif
 #endif
 
-#if defined HWLOC_WIN_SYS && !defined __MINGW32__ && !defined(__CYGWIN__)
-/* MSVC doesn't support C99 variable-length array */
-#include <malloc.h>
-#define HWLOC_VLA(_type, _name, _nb) _type *_name = (_type*) _alloca((_nb)*sizeof(_type))
-#else
-#define HWLOC_VLA(_type, _name, _nb) _type _name[_nb]
-#endif
-
 #endif /* HWLOC_PRIVATE_MISC_H */

src/3rdparty/hwloc/include/private/private.h

@@ -22,11 +22,12 @@
 #ifndef HWLOC_PRIVATE_H
 #define HWLOC_PRIVATE_H
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/bitmap.h>
+#include "hwloc/bitmap.h"
-#include <private/components.h>
+#include "private/components.h"
-#include <private/misc.h>
+#include "private/misc.h"
+
 #include <sys/types.h>
 #ifdef HAVE_UNISTD_H
 #include <unistd.h>
@@ -39,7 +40,7 @@
 #endif
 #include <string.h>
 
-#define HWLOC_TOPOLOGY_ABI 0x20000 /* version of the layout of struct topology */
+#define HWLOC_TOPOLOGY_ABI 0x20100 /* version of the layout of struct topology */
 
 /*****************************************************
  * WARNING:
@@ -67,12 +68,13 @@ struct hwloc_topology {
   void *adopted_shmem_addr;
   size_t adopted_shmem_length;
 
-#define HWLOC_NR_SLEVELS 5
+#define HWLOC_NR_SLEVELS 6
 #define HWLOC_SLEVEL_NUMANODE 0
 #define HWLOC_SLEVEL_BRIDGE 1
 #define HWLOC_SLEVEL_PCIDEV 2
 #define HWLOC_SLEVEL_OSDEV 3
 #define HWLOC_SLEVEL_MISC 4
+#define HWLOC_SLEVEL_MEMCACHE 5
   /* order must match negative depth, it's asserted in setup_defaults() */
 #define HWLOC_SLEVEL_FROM_DEPTH(x) (HWLOC_TYPE_DEPTH_NUMANODE-(x))
 #define HWLOC_SLEVEL_TO_DEPTH(x) (HWLOC_TYPE_DEPTH_NUMANODE-(x))
@@ -86,6 +88,7 @@ struct hwloc_topology {
   hwloc_bitmap_t allowed_nodeset;
 
   struct hwloc_binding_hooks {
+    /* These are actually rather OS hooks since some of them are not about binding */
     int (*set_thisproc_cpubind)(hwloc_topology_t topology, hwloc_const_cpuset_t set, int flags);
     int (*get_thisproc_cpubind)(hwloc_topology_t topology, hwloc_cpuset_t set, int flags);
     int (*set_thisthread_cpubind)(hwloc_topology_t topology, hwloc_const_cpuset_t set, int flags);
@@ -127,20 +130,35 @@ struct hwloc_topology {
   int userdata_not_decoded;
 
   struct hwloc_internal_distances_s {
-    hwloc_obj_type_t type;
+    char *name; /* FIXME: needs an API to set it from user */
+
+    unsigned id; /* to match the container id field of public distances structure
+		  * not exported to XML, regenerated during _add()
+		  */
+
+    /* if all objects have the same type, different_types is NULL and unique_type is valid.
+     * otherwise unique_type is HWLOC_OBJ_TYPE_NONE and different_types contains individual objects types.
+     */
+    hwloc_obj_type_t unique_type;
+    hwloc_obj_type_t *different_types;
+
     /* add union hwloc_obj_attr_u if we ever support groups */
     unsigned nbobjs;
-    uint64_t *indexes; /* array of OS or GP indexes before we can convert them into objs. */
+    uint64_t *indexes; /* array of OS or GP indexes before we can convert them into objs.
+			* OS indexes for distances covering only PUs or only NUMAnodes.
+			*/
+#define HWLOC_DIST_TYPE_USE_OS_INDEX(_type) ((_type) == HWLOC_OBJ_PU || (_type == HWLOC_OBJ_NUMANODE))
     uint64_t *values; /* distance matrices, ordered according to the above indexes/objs array.
 		       * distance from i to j is stored in slot i*nbnodes+j.
 		       */
     unsigned long kind;
 
+#define HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID (1U<<0) /* if the objs array is valid below */
+    unsigned iflags;
+
     /* objects are currently stored in physical_index order */
     hwloc_obj_t *objs; /* array of objects */
-    int objs_are_valid; /* set to 1 if the array objs is still valid, 0 if needs refresh */
 
-    unsigned id; /* to match the container id field of public distances structure */
     struct hwloc_internal_distances_s *prev, *next;
   } *first_dist, *last_dist;
   unsigned next_dist_id;
@@ -153,8 +171,9 @@ struct hwloc_topology {
 
   /* list of enabled backends. */
   struct hwloc_backend * backends;
-  struct hwloc_backend * get_pci_busid_cpuset_backend;
+  struct hwloc_backend * get_pci_busid_cpuset_backend; /* first backend that provides get_pci_busid_cpuset() callback */
-  unsigned backend_excludes;
+  unsigned backend_phases;
+  unsigned backend_excluded_phases;
 
   /* memory allocator for topology objects */
   struct hwloc_tma * tma;
@@ -176,7 +195,6 @@ struct hwloc_topology {
   struct hwloc_numanode_attr_s machine_memory;
 
   /* pci stuff */
-  int need_pci_belowroot_apply_locality;
   int pci_has_forced_locality;
   unsigned pci_forced_locality_nr;
   struct hwloc_pci_forced_locality_s {
@@ -185,13 +203,32 @@ struct hwloc_topology {
     hwloc_bitmap_t cpuset;
   } * pci_forced_locality;
 
+  /* component blacklisting */
+  unsigned nr_blacklisted_components;
+  struct hwloc_topology_forced_component_s {
+    struct hwloc_disc_component *component;
+    unsigned phases;
+  } *blacklisted_components;
+
+  /* FIXME: keep until topo destroy and reuse for finding specific buses */
+  struct hwloc_pci_locality_s {
+    unsigned domain;
+    unsigned bus_min;
+    unsigned bus_max;
+    hwloc_bitmap_t cpuset;
+    hwloc_obj_t parent;
+    struct hwloc_pci_locality_s *prev, *next;
+  } *first_pci_locality, *last_pci_locality;
 };
 
 extern void hwloc_alloc_root_sets(hwloc_obj_t root);
 extern void hwloc_setup_pu_level(struct hwloc_topology *topology, unsigned nb_pus);
 extern int hwloc_get_sysctlbyname(const char *name, int64_t *n);
 extern int hwloc_get_sysctl(int name[], unsigned namelen, int *n);
-extern int hwloc_fallback_nbprocessors(struct hwloc_topology *topology);
+
+/* returns the number of CPU from the OS (only valid if thissystem) */
+#define HWLOC_FALLBACK_NBPROCESSORS_INCLUDE_OFFLINE 1 /* by default we try to get only the online CPUs */
+extern int hwloc_fallback_nbprocessors(unsigned flags);
 
 extern int hwloc__object_cpusets_compare_first(hwloc_obj_t obj1, hwloc_obj_t obj2);
 extern void hwloc__reorder_children(hwloc_obj_t parent);
@@ -208,19 +245,17 @@ extern void hwloc_pci_discovery_init(struct hwloc_topology *topology);
 extern void hwloc_pci_discovery_prepare(struct hwloc_topology *topology);
 extern void hwloc_pci_discovery_exit(struct hwloc_topology *topology);
 
+/* Look for an object matching the given domain/bus/func,
+ * either exactly or return the smallest container bridge
+ */
+extern struct hwloc_obj * hwloc_pci_find_by_busid(struct hwloc_topology *topology, unsigned domain, unsigned bus, unsigned dev, unsigned func);
+
 /* Look for an object matching complete cpuset exactly, or insert one.
  * Return NULL on failure.
  * Return a good fallback (object above) on failure to insert.
  */
 extern hwloc_obj_t hwloc_find_insert_io_parent_by_complete_cpuset(struct hwloc_topology *topology, hwloc_cpuset_t cpuset);
 
-/* Move PCI objects currently attached to the root object ot their actual location.
- * Called by the core at the end of hwloc_topology_load().
- * Prior to this call, all PCI objects may be found below the root object.
- * After this call and a reconnect of levels, all PCI objects are available through levels.
- */
-extern int hwloc_pci_belowroot_apply_locality(struct hwloc_topology *topology);
-
 extern int hwloc__add_info(struct hwloc_info_s **infosp, unsigned *countp, const char *name, const char *value);
 extern int hwloc__add_info_nodup(struct hwloc_info_s **infosp, unsigned *countp, const char *name, const char *value, int replace);
 extern int hwloc__move_infos(struct hwloc_info_s **dst_infosp, unsigned *dst_countp, struct hwloc_info_s **src_infosp, unsigned *src_countp);
@@ -313,8 +348,8 @@ extern void hwloc_internal_distances_prepare(hwloc_topology_t topology);
 extern void hwloc_internal_distances_destroy(hwloc_topology_t topology);
 extern int hwloc_internal_distances_dup(hwloc_topology_t new, hwloc_topology_t old);
 extern void hwloc_internal_distances_refresh(hwloc_topology_t topology);
-extern int hwloc_internal_distances_add(hwloc_topology_t topology, unsigned nbobjs, hwloc_obj_t *objs, uint64_t *values, unsigned long kind, unsigned long flags);
+extern int hwloc_internal_distances_add(hwloc_topology_t topology, const char *name, unsigned nbobjs, hwloc_obj_t *objs, uint64_t *values, unsigned long kind, unsigned long flags);
-extern int hwloc_internal_distances_add_by_index(hwloc_topology_t topology, hwloc_obj_type_t type, unsigned nbobjs, uint64_t *indexes, uint64_t *values, unsigned long kind, unsigned long flags);
+extern int hwloc_internal_distances_add_by_index(hwloc_topology_t topology, const char *name, hwloc_obj_type_t unique_type, hwloc_obj_type_t *different_types, unsigned nbobjs, uint64_t *indexes, uint64_t *values, unsigned long kind, unsigned long flags);
 extern void hwloc_internal_distances_invalidate_cached_objs(hwloc_topology_t topology);
 
 /* encode src buffer into target buffer.
@@ -330,13 +365,19 @@ extern int hwloc_encode_to_base64(const char *src, size_t srclength, char *targe
  */
 extern int hwloc_decode_from_base64(char const *src, char *target, size_t targsize);
 
-/* Check whether needle matches the beginning of haystack, at least n, and up
- * to a colon or \0 */
-extern int hwloc_namecoloncmp(const char *haystack, const char *needle, size_t n);
-
 /* On some systems, snprintf returns the size of written data, not the actually
- * required size.  hwloc_snprintf always report the actually required size. */
+ * required size. Sometimes it returns -1 on truncation too.
+ * And sometimes it doesn't like NULL output buffers.
+ * http://www.gnu.org/software/gnulib/manual/html_node/snprintf.html
+ *
+ * hwloc_snprintf behaves properly, but it's a bit overkill on the vast majority
+ * of platforms, so don't enable it unless really needed.
+ */
+#ifdef HWLOC_HAVE_CORRECT_SNPRINTF
+#define hwloc_snprintf snprintf
+#else
 extern int hwloc_snprintf(char *str, size_t size, const char *format, ...) __hwloc_attribute_format(printf, 3, 4);
+#endif
 
 /* Return the name of the currently running program, if supported.
  * If not NULL, must be freed by the caller.
@@ -356,7 +397,7 @@ extern char * hwloc_progname(struct hwloc_topology *topology);
 #define HWLOC_GROUP_KIND_INTEL_MODULE			102	/* no subkind */
 #define HWLOC_GROUP_KIND_INTEL_TILE			103	/* no subkind */
 #define HWLOC_GROUP_KIND_INTEL_DIE			104	/* no subkind */
-#define HWLOC_GROUP_KIND_S390_BOOK			110	/* no subkind */
+#define HWLOC_GROUP_KIND_S390_BOOK			110	/* subkind 0 is book, subkind 1 is drawer (group of books) */
 #define HWLOC_GROUP_KIND_AMD_COMPUTE_UNIT		120	/* no subkind */
 /* then, OS-specific groups */
 #define HWLOC_GROUP_KIND_SOLARIS_PG_HW_PERF		200	/* subkind is group width */

src/3rdparty/hwloc/include/private/xml.h

@@ -1,12 +1,12 @@
 /*
- * Copyright © 2009-2019 Inria.  All rights reserved.
+ * Copyright © 2009-2017 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
 #ifndef PRIVATE_XML_H
 #define PRIVATE_XML_H 1
 
-#include <hwloc.h>
+#include "hwloc.h"
 
 #include <sys/types.h>
 
@@ -54,7 +54,6 @@ struct hwloc_xml_backend_data_s {
   unsigned nbnumanodes;
   hwloc_obj_t first_numanode, last_numanode; /* temporary cousin-list for handling v1distances */
   struct hwloc__xml_imported_v1distances_s *first_v1dist, *last_v1dist;
-  int dont_merge_die_groups;
 };
 
 /**************

src/3rdparty/hwloc/src/base64.c

@@ -11,7 +11,7 @@
 /* include hwloc's config before anything else
  * so that extensions and features are properly enabled
  */
-#include <private/private.h>
+#include "private/private.h"
 
 /*	$OpenBSD: base64.c,v 1.5 2006/10/21 09:55:03 otto Exp $	*/
 

src/3rdparty/hwloc/src/bind.c

@@ -1,15 +1,16 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2010, 2012 Université Bordeaux
  * Copyright © 2011-2015 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <hwloc/helper.h>
+#include "hwloc/helper.h"
+
 #ifdef HAVE_SYS_MMAN_H
 #  include <sys/mman.h>
 #endif
@@ -885,6 +886,8 @@ hwloc_set_binding_hooks(struct hwloc_topology *topology)
   } else {
     /* not this system, use dummy binding hooks that do nothing (but don't return ENOSYS) */
     hwloc_set_dummy_hooks(&topology->binding_hooks, &topology->support);
+
+    /* Linux has some hooks that also work in this case, but they are not strictly needed yet. */
   }
 
   /* if not is_thissystem, set_cpubind is fake

src/3rdparty/hwloc/src/bitmap.c

@@ -1,18 +1,18 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2017 Inria.  All rights reserved.
+ * Copyright © 2009-2018 Inria.  All rights reserved.
  * Copyright © 2009-2011 Université Bordeaux
  * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc/autogen/config.h>
+#include "hwloc/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/misc.h>
+#include "private/misc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/debug.h>
+#include "private/debug.h"
-#include <hwloc/bitmap.h>
+#include "hwloc/bitmap.h"
 
 #include <stdarg.h>
 #include <stdio.h>
@@ -766,6 +766,21 @@ int hwloc_bitmap_from_ith_ulong(struct hwloc_bitmap_s *set, unsigned i, unsigned
 	return 0;
 }
 
+int hwloc_bitmap_from_ulongs(struct hwloc_bitmap_s *set, unsigned nr, const unsigned long *masks)
+{
+	unsigned j;
+
+	HWLOC__BITMAP_CHECK(set);
+
+	if (hwloc_bitmap_reset_by_ulongs(set, nr) < 0)
+		return -1;
+
+	for(j=0; j<nr; j++)
+		set->ulongs[j] = masks[j];
+	set->infinite = 0;
+	return 0;
+}
+
 unsigned long hwloc_bitmap_to_ulong(const struct hwloc_bitmap_s *set)
 {
 	HWLOC__BITMAP_CHECK(set);
@@ -780,6 +795,30 @@ unsigned long hwloc_bitmap_to_ith_ulong(const struct hwloc_bitmap_s *set, unsign
 	return HWLOC_SUBBITMAP_READULONG(set, i);
 }
 
+int hwloc_bitmap_to_ulongs(const struct hwloc_bitmap_s *set, unsigned nr, unsigned long *masks)
+{
+	unsigned j;
+
+	HWLOC__BITMAP_CHECK(set);
+
+	for(j=0; j<nr; j++)
+		masks[j] = HWLOC_SUBBITMAP_READULONG(set, j);
+	return 0;
+}
+
+int hwloc_bitmap_nr_ulongs(const struct hwloc_bitmap_s *set)
+{
+	unsigned last;
+
+	HWLOC__BITMAP_CHECK(set);
+
+	if (set->infinite)
+		return -1;
+
+	last = hwloc_bitmap_last(set);
+	return (last + HWLOC_BITS_PER_LONG-1)/HWLOC_BITS_PER_LONG;
+}
+
 int hwloc_bitmap_only(struct hwloc_bitmap_s * set, unsigned cpu)
 {
 	unsigned index_ = HWLOC_SUBBITMAP_INDEX(cpu);

src/3rdparty/hwloc/src/components.c

@@ -1,18 +1,19 @@
 /*
- * Copyright © 2009-2017 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2012 Université Bordeaux
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/xml.h>
+#include "private/xml.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 #define HWLOC_COMPONENT_STOP_NAME "stop"
 #define HWLOC_COMPONENT_EXCLUDE_CHAR '-'
 #define HWLOC_COMPONENT_SEPS ","
+#define HWLOC_COMPONENT_PHASESEP_CHAR ':'
 
 /* list of all registered discovery components, sorted by priority, higher priority first.
  * noos is last because its priority is 0.
@@ -232,17 +233,6 @@ hwloc_plugins_init(void)
 
 #endif /* HWLOC_HAVE_PLUGINS */
 
-static const char *
-hwloc_disc_component_type_string(hwloc_disc_component_type_t type)
-{
-  switch (type) {
-  case HWLOC_DISC_COMPONENT_TYPE_CPU: return "cpu";
-  case HWLOC_DISC_COMPONENT_TYPE_GLOBAL: return "global";
-  case HWLOC_DISC_COMPONENT_TYPE_MISC: return "misc";
-  default: return "**unknown**";
-  }
-}
-
 static int
 hwloc_disc_component_register(struct hwloc_disc_component *component,
 			      const char *filename)
@@ -256,21 +246,26 @@ hwloc_disc_component_register(struct hwloc_disc_component *component,
     return -1;
   }
   if (strchr(component->name, HWLOC_COMPONENT_EXCLUDE_CHAR)
+      || strchr(component->name, HWLOC_COMPONENT_PHASESEP_CHAR)
       || strcspn(component->name, HWLOC_COMPONENT_SEPS) != strlen(component->name)) {
     if (hwloc_components_verbose)
       fprintf(stderr, "Cannot register discovery component with name `%s' containing reserved characters `%c" HWLOC_COMPONENT_SEPS "'\n",
 	      component->name, HWLOC_COMPONENT_EXCLUDE_CHAR);
     return -1;
   }
-  /* check that the component type is valid */
+
-  switch ((unsigned) component->type) {
+  /* check that the component phases are valid */
-  case HWLOC_DISC_COMPONENT_TYPE_CPU:
+  if (!component->phases
-  case HWLOC_DISC_COMPONENT_TYPE_GLOBAL:
+      || (component->phases != HWLOC_DISC_PHASE_GLOBAL
-  case HWLOC_DISC_COMPONENT_TYPE_MISC:
+	  && component->phases & ~(HWLOC_DISC_PHASE_CPU
-    break;
+				   |HWLOC_DISC_PHASE_MEMORY
-  default:
+				   |HWLOC_DISC_PHASE_PCI
-    fprintf(stderr, "Cannot register discovery component `%s' with unknown type %u\n",
+				   |HWLOC_DISC_PHASE_IO
-	    component->name, (unsigned) component->type);
+				   |HWLOC_DISC_PHASE_MISC
+				   |HWLOC_DISC_PHASE_ANNOTATE
+				   |HWLOC_DISC_PHASE_TWEAK))) {
+    fprintf(stderr, "Cannot register discovery component `%s' with invalid phases 0x%x\n",
+	    component->name, component->phases);
     return -1;
   }
 
@@ -295,8 +290,8 @@ hwloc_disc_component_register(struct hwloc_disc_component *component,
     prev = &((*prev)->next);
   }
   if (hwloc_components_verbose)
-    fprintf(stderr, "Registered %s discovery component `%s' with priority %u (%s%s)\n",
+    fprintf(stderr, "Registered discovery component `%s' phases 0x%x with priority %u (%s%s)\n",
-	    hwloc_disc_component_type_string(component->type), component->name, component->priority,
+	    component->name, component->phases, component->priority,
 	    filename ? "from plugin " : "statically build", filename ? filename : "");
 
   prev = &hwloc_disc_components;
@@ -310,7 +305,7 @@ hwloc_disc_component_register(struct hwloc_disc_component *component,
   return 0;
 }
 
-#include <static-components.h>
+#include "static-components.h"
 
 static void (**hwloc_component_finalize_cbs)(unsigned long);
 static unsigned hwloc_component_finalize_cb_count;
@@ -415,31 +410,152 @@ hwloc_components_init(void)
 }
 
 void
-hwloc_backends_init(struct hwloc_topology *topology)
+hwloc_topology_components_init(struct hwloc_topology *topology)
 {
+  topology->nr_blacklisted_components = 0;
+  topology->blacklisted_components = NULL;
+
   topology->backends = NULL;
-  topology->backend_excludes = 0;
+  topology->backend_phases = 0;
+  topology->backend_excluded_phases = 0;
 }
 
+/* look for name among components, ignoring things after `:' */
 static struct hwloc_disc_component *
-hwloc_disc_component_find(int type /* hwloc_disc_component_type_t or -1 if any */,
+hwloc_disc_component_find(const char *name, const char **endp)
-			       const char *name /* name of NULL if any */)
 {
-  struct hwloc_disc_component *comp = hwloc_disc_components;
+  struct hwloc_disc_component *comp;
+  size_t length;
+  const char *end = strchr(name, HWLOC_COMPONENT_PHASESEP_CHAR);
+  if (end) {
+    length = end-name;
+    if (endp)
+      *endp = end+1;
+  } else {
+    length = strlen(name);
+    if (endp)
+      *endp = NULL;
+  }
+
+  comp = hwloc_disc_components;
   while (NULL != comp) {
-    if ((-1 == type || type == (int) comp->type)
+    if (!strncmp(name, comp->name, length))
-       && (NULL == name || !strcmp(name, comp->name)))
       return comp;
     comp = comp->next;
   }
   return NULL;
 }
 
+static unsigned
+hwloc_phases_from_string(const char *s)
+{
+  if (!s)
+    return ~0U;
+  if (s[0]<'0' || s[0]>'9') {
+    if (!strcasecmp(s, "global"))
+      return HWLOC_DISC_PHASE_GLOBAL;
+    else if (!strcasecmp(s, "cpu"))
+      return HWLOC_DISC_PHASE_CPU;
+    if (!strcasecmp(s, "memory"))
+      return HWLOC_DISC_PHASE_MEMORY;
+    if (!strcasecmp(s, "pci"))
+      return HWLOC_DISC_PHASE_PCI;
+    if (!strcasecmp(s, "io"))
+      return HWLOC_DISC_PHASE_IO;
+    if (!strcasecmp(s, "misc"))
+      return HWLOC_DISC_PHASE_MISC;
+    if (!strcasecmp(s, "annotate"))
+      return HWLOC_DISC_PHASE_ANNOTATE;
+    if (!strcasecmp(s, "tweak"))
+      return HWLOC_DISC_PHASE_TWEAK;
+    return 0;
+  }
+  return (unsigned) strtoul(s, NULL, 0);
+}
+
+static int
+hwloc_disc_component_blacklist_one(struct hwloc_topology *topology,
+				   const char *name)
+{
+  struct hwloc_topology_forced_component_s *blacklisted;
+  struct hwloc_disc_component *comp;
+  unsigned phases;
+  unsigned i;
+
+  if (!strcmp(name, "linuxpci") || !strcmp(name, "linuxio")) {
+    /* replace linuxpci and linuxio with linux (with IO phases)
+     * for backward compatibility with pre-v2.0 and v2.0 respectively */
+    if (hwloc_components_verbose)
+      fprintf(stderr, "Replacing deprecated component `%s' with `linux' IO phases in blacklisting\n", name);
+    comp = hwloc_disc_component_find("linux", NULL);
+    phases = HWLOC_DISC_PHASE_PCI | HWLOC_DISC_PHASE_IO | HWLOC_DISC_PHASE_MISC | HWLOC_DISC_PHASE_ANNOTATE;
+
+  } else {
+    /* normal lookup */
+    const char *end;
+    comp = hwloc_disc_component_find(name, &end);
+    phases = hwloc_phases_from_string(end);
+  }
+  if (!comp) {
+    errno = EINVAL;
+    return -1;
+  }
+
+  if (hwloc_components_verbose)
+    fprintf(stderr, "Blacklisting component `%s` phases 0x%x\n", comp->name, phases);
+
+  for(i=0; i<topology->nr_blacklisted_components; i++) {
+    if (topology->blacklisted_components[i].component == comp) {
+      topology->blacklisted_components[i].phases |= phases;
+      return 0;
+    }
+  }
+
+  blacklisted = realloc(topology->blacklisted_components, (topology->nr_blacklisted_components+1)*sizeof(*blacklisted));
+  if (!blacklisted)
+    return -1;
+
+  blacklisted[topology->nr_blacklisted_components].component = comp;
+  blacklisted[topology->nr_blacklisted_components].phases = phases;
+  topology->blacklisted_components = blacklisted;
+  topology->nr_blacklisted_components++;
+  return 0;
+}
+
+int
+hwloc_topology_set_components(struct hwloc_topology *topology,
+			      unsigned long flags,
+			      const char *name)
+{
+  if (topology->is_loaded) {
+    errno = EBUSY;
+    return -1;
+  }
+
+  if (flags & ~HWLOC_TOPOLOGY_COMPONENTS_FLAG_BLACKLIST) {
+    errno = EINVAL;
+    return -1;
+  }
+
+  /* this flag is strictly required for now */
+  if (flags != HWLOC_TOPOLOGY_COMPONENTS_FLAG_BLACKLIST) {
+    errno = EINVAL;
+    return -1;
+  }
+
+  if (!strncmp(name, "all", 3) && name[3] == HWLOC_COMPONENT_PHASESEP_CHAR) {
+    topology->backend_excluded_phases = hwloc_phases_from_string(name+4);
+    return 0;
+  }
+
+  return hwloc_disc_component_blacklist_one(topology, name);
+}
+
 /* used by set_xml(), set_synthetic(), ... environment variables, ... to force the first backend */
 int
 hwloc_disc_component_force_enable(struct hwloc_topology *topology,
 				  int envvar_forced,
-				  int type, const char *name,
+				  const char *name,
 				  const void *data1, const void *data2, const void *data3)
 {
   struct hwloc_disc_component *comp;
@@ -450,18 +566,28 @@ hwloc_disc_component_force_enable(struct hwloc_topology *topology,
     return -1;
   }
 
-  comp = hwloc_disc_component_find(type, name);
+  comp = hwloc_disc_component_find(name, NULL);
   if (!comp) {
     errno = ENOSYS;
     return -1;
   }
 
-  backend = comp->instantiate(comp, data1, data2, data3);
+  backend = comp->instantiate(topology, comp, 0U /* force-enabled don't get any phase blacklisting */,
+			      data1, data2, data3);
   if (backend) {
+    int err;
     backend->envvar_forced = envvar_forced;
     if (topology->backends)
       hwloc_backends_disable_all(topology);
-    return hwloc_backend_enable(topology, backend);
+    err = hwloc_backend_enable(backend);
+
+    if (comp->phases == HWLOC_DISC_PHASE_GLOBAL) {
+      char *env = getenv("HWLOC_ANNOTATE_GLOBAL_COMPONENTS");
+      if (env && atoi(env))
+	topology->backend_excluded_phases &= ~HWLOC_DISC_PHASE_ANNOTATE;
+    }
+
+    return err;
   } else
     return -1;
 }
@@ -469,29 +595,32 @@ hwloc_disc_component_force_enable(struct hwloc_topology *topology,
 static int
 hwloc_disc_component_try_enable(struct hwloc_topology *topology,
 				struct hwloc_disc_component *comp,
-				const char *comparg,
+				int envvar_forced,
-				int envvar_forced)
+				unsigned blacklisted_phases)
 {
   struct hwloc_backend *backend;
 
-  if (topology->backend_excludes & comp->type) {
+  if (!(comp->phases & ~(topology->backend_excluded_phases | blacklisted_phases))) {
+    /* all this backend phases are already excluded, exclude the backend entirely */
     if (hwloc_components_verbose)
       /* do not warn if envvar_forced since system-wide HWLOC_COMPONENTS must be silently ignored after set_xml() etc.
        */
-      fprintf(stderr, "Excluding %s discovery component `%s', conflicts with excludes 0x%x\n",
+      fprintf(stderr, "Excluding discovery component `%s' phases 0x%x, conflicts with excludes 0x%x\n",
-	      hwloc_disc_component_type_string(comp->type), comp->name, topology->backend_excludes);
+	      comp->name, comp->phases, topology->backend_excluded_phases);
     return -1;
   }
 
-  backend = comp->instantiate(comp, comparg, NULL, NULL);
+  backend = comp->instantiate(topology, comp, topology->backend_excluded_phases | blacklisted_phases,
+			      NULL, NULL, NULL);
   if (!backend) {
     if (hwloc_components_verbose || envvar_forced)
       fprintf(stderr, "Failed to instantiate discovery component `%s'\n", comp->name);
     return -1;
   }
 
+  backend->phases &= ~blacklisted_phases;
   backend->envvar_forced = envvar_forced;
-  return hwloc_backend_enable(topology, backend);
+  return hwloc_backend_enable(backend);
 }
 
 void
@@ -502,11 +631,12 @@ hwloc_disc_components_enable_others(struct hwloc_topology *topology)
   int tryall = 1;
   const char *_env;
   char *env; /* we'll to modify the env value, so duplicate it */
+  unsigned i;
 
   _env = getenv("HWLOC_COMPONENTS");
   env = _env ? strdup(_env) : NULL;
 
-  /* enable explicitly listed components */
+  /* blacklist disabled components */
   if (env) {
     char *curenv = env;
     size_t s;
@@ -516,21 +646,41 @@ hwloc_disc_components_enable_others(struct hwloc_topology *topology)
       if (s) {
 	char c;
 
-	/* replace linuxpci with linuxio for backward compatibility with pre-v2.0 */
+	if (curenv[0] != HWLOC_COMPONENT_EXCLUDE_CHAR)
-	if (!strncmp(curenv, "linuxpci", 8) && s == 8) {
-	  curenv[5] = 'i';
-	  curenv[6] = 'o';
-	  curenv[7] = *HWLOC_COMPONENT_SEPS;
-	} else if (curenv[0] == HWLOC_COMPONENT_EXCLUDE_CHAR && !strncmp(curenv+1, "linuxpci", 8) && s == 9) {
-	  curenv[6] = 'i';
-	  curenv[7] = 'o';
-	  curenv[8] = *HWLOC_COMPONENT_SEPS;
-	  /* skip this name, it's a negated one */
 	  goto nextname;
-	}
 
-	if (curenv[0] == HWLOC_COMPONENT_EXCLUDE_CHAR)
+	/* save the last char and replace with \0 */
-	  goto nextname;
+	c = curenv[s];
+	curenv[s] = '\0';
+
+	/* blacklist it, and just ignore failures to allocate */
+	hwloc_disc_component_blacklist_one(topology, curenv+1);
+
+	/* remove that blacklisted name from the string */
+	for(i=0; i<s; i++)
+	  curenv[i] = *HWLOC_COMPONENT_SEPS;
+
+	/* restore chars (the second loop below needs env to be unmodified) */
+	curenv[s] = c;
+      }
+
+    nextname:
+      curenv += s;
+      if (*curenv)
+	/* Skip comma */
+	curenv++;
+    }
+  }
+
+  /* enable explicitly listed components */
+  if (env) {
+    char *curenv = env;
+    size_t s;
+
+    while (*curenv) {
+      s = strcspn(curenv, HWLOC_COMPONENT_SEPS);
+      if (s) {
+	char c, *name;
 
 	if (!strncmp(curenv, HWLOC_COMPONENT_STOP_NAME, s)) {
 	  tryall = 0;
@@ -541,18 +691,31 @@ hwloc_disc_components_enable_others(struct hwloc_topology *topology)
 	c = curenv[s];
 	curenv[s] = '\0';
 
-	comp = hwloc_disc_component_find(-1, curenv);
+	name = curenv;
+	if (!strcmp(name, "linuxpci") || !strcmp(name, "linuxio")) {
+	  if (hwloc_components_verbose)
+	    fprintf(stderr, "Replacing deprecated component `%s' with `linux' in envvar forcing\n", name);
+	  name = "linux";
+	}
+
+	comp = hwloc_disc_component_find(name, NULL /* we enable the entire component, phases must be blacklisted separately */);
 	if (comp) {
-	  hwloc_disc_component_try_enable(topology, comp, NULL, 1 /* envvar forced */);
+	  unsigned blacklisted_phases = 0U;
+	  for(i=0; i<topology->nr_blacklisted_components; i++)
+	    if (comp == topology->blacklisted_components[i].component) {
+	      blacklisted_phases = topology->blacklisted_components[i].phases;
+	      break;
+	    }
+	  if (comp->phases & ~blacklisted_phases)
+	    hwloc_disc_component_try_enable(topology, comp, 1 /* envvar forced */, blacklisted_phases);
 	} else {
-	  fprintf(stderr, "Cannot find discovery component `%s'\n", curenv);
+	  fprintf(stderr, "Cannot find discovery component `%s'\n", name);
 	}
 
 	/* restore chars (the second loop below needs env to be unmodified) */
 	curenv[s] = c;
       }
 
-nextname:
       curenv += s;
       if (*curenv)
 	/* Skip comma */
@@ -566,26 +729,24 @@ nextname:
   if (tryall) {
     comp = hwloc_disc_components;
     while (NULL != comp) {
+      unsigned blacklisted_phases = 0U;
       if (!comp->enabled_by_default)
 	goto nextcomp;
-      /* check if this component was explicitly excluded in env */
+      /* check if this component was blacklisted by the application */
-      if (env) {
+      for(i=0; i<topology->nr_blacklisted_components; i++)
-	char *curenv = env;
+	if (comp == topology->blacklisted_components[i].component) {
-	while (*curenv) {
+	  blacklisted_phases = topology->blacklisted_components[i].phases;
-	  size_t s = strcspn(curenv, HWLOC_COMPONENT_SEPS);
+	  break;
-	  if (curenv[0] == HWLOC_COMPONENT_EXCLUDE_CHAR && !strncmp(curenv+1, comp->name, s-1) && strlen(comp->name) == s-1) {
-	    if (hwloc_components_verbose)
-	      fprintf(stderr, "Excluding %s discovery component `%s' because of HWLOC_COMPONENTS environment variable\n",
-	    hwloc_disc_component_type_string(comp->type), comp->name);
-	    goto nextcomp;
-	  }
-	  curenv += s;
-	  if (*curenv)
-	    /* Skip comma */
-	    curenv++;
 	}
+
+      if (!(comp->phases & ~blacklisted_phases)) {
+	if (hwloc_components_verbose)
+	  fprintf(stderr, "Excluding blacklisted discovery component `%s' phases 0x%x\n",
+		  comp->name, comp->phases);
+	goto nextcomp;
       }
-      hwloc_disc_component_try_enable(topology, comp, NULL, 0 /* defaults, not envvar forced */);
+
+      hwloc_disc_component_try_enable(topology, comp, 0 /* defaults, not envvar forced */, blacklisted_phases);
 nextcomp:
       comp = comp->next;
     }
@@ -597,7 +758,7 @@ nextcomp:
     backend = topology->backends;
     fprintf(stderr, "Final list of enabled discovery components: ");
     while (backend != NULL) {
-      fprintf(stderr, "%s%s", first ? "" : ",", backend->component->name);
+      fprintf(stderr, "%s%s(0x%x)", first ? "" : ",", backend->component->name, backend->phases);
       backend = backend->next;
       first = 0;
     }
@@ -638,7 +799,8 @@ hwloc_components_fini(void)
 }
 
 struct hwloc_backend *
-hwloc_backend_alloc(struct hwloc_disc_component *component)
+hwloc_backend_alloc(struct hwloc_topology *topology,
+		    struct hwloc_disc_component *component)
 {
   struct hwloc_backend * backend = malloc(sizeof(*backend));
   if (!backend) {
@@ -646,6 +808,12 @@ hwloc_backend_alloc(struct hwloc_disc_component *component)
     return NULL;
   }
   backend->component = component;
+  backend->topology = topology;
+  /* filter-out component phases that are excluded */
+  backend->phases = component->phases & ~topology->backend_excluded_phases;
+  if (backend->phases != component->phases && hwloc_components_verbose)
+    fprintf(stderr, "Trying discovery component `%s' with phases 0x%x instead of 0x%x\n",
+	    component->name, backend->phases, component->phases);
   backend->flags = 0;
   backend->discover = NULL;
   backend->get_pci_busid_cpuset = NULL;
@@ -665,14 +833,15 @@ hwloc_backend_disable(struct hwloc_backend *backend)
 }
 
 int
-hwloc_backend_enable(struct hwloc_topology *topology, struct hwloc_backend *backend)
+hwloc_backend_enable(struct hwloc_backend *backend)
 {
+  struct hwloc_topology *topology = backend->topology;
   struct hwloc_backend **pprev;
 
   /* check backend flags */
   if (backend->flags) {
-    fprintf(stderr, "Cannot enable %s discovery component `%s' with unknown flags %lx\n",
+    fprintf(stderr, "Cannot enable discovery component `%s' phases 0x%x with unknown flags %lx\n",
-	    hwloc_disc_component_type_string(backend->component->type), backend->component->name, backend->flags);
+	    backend->component->name, backend->component->phases, backend->flags);
     return -1;
   }
 
@@ -681,8 +850,8 @@ hwloc_backend_enable(struct hwloc_topology *topology, struct hwloc_backend *back
   while (NULL != *pprev) {
     if ((*pprev)->component == backend->component) {
       if (hwloc_components_verbose)
-	fprintf(stderr, "Cannot enable %s discovery component `%s' twice\n",
+	fprintf(stderr, "Cannot enable  discovery component `%s' phases 0x%x twice\n",
-		hwloc_disc_component_type_string(backend->component->type), backend->component->name);
+		backend->component->name, backend->component->phases);
       hwloc_backend_disable(backend);
       errno = EBUSY;
       return -1;
@@ -691,8 +860,8 @@ hwloc_backend_enable(struct hwloc_topology *topology, struct hwloc_backend *back
   }
 
   if (hwloc_components_verbose)
-    fprintf(stderr, "Enabling %s discovery component `%s'\n",
+    fprintf(stderr, "Enabling discovery component `%s' with phases 0x%x (among 0x%x)\n",
-	    hwloc_disc_component_type_string(backend->component->type), backend->component->name);
+	    backend->component->name, backend->phases, backend->component->phases);
 
   /* enqueue at the end */
   pprev = &topology->backends;
@@ -701,8 +870,8 @@ hwloc_backend_enable(struct hwloc_topology *topology, struct hwloc_backend *back
   backend->next = *pprev;
   *pprev = backend;
 
-  backend->topology = topology;
+  topology->backend_phases |= backend->component->phases;
-  topology->backend_excludes |= backend->component->excludes;
+  topology->backend_excluded_phases |= backend->component->excluded_phases;
   return 0;
 }
 
@@ -712,7 +881,7 @@ hwloc_backends_is_thissystem(struct hwloc_topology *topology)
   struct hwloc_backend *backend;
   const char *local_env;
 
-  /* Apply is_thissystem topology flag before we enforce envvar backends.
+  /*
    * If the application changed the backend with set_foo(),
    * it may use set_flags() update the is_thissystem flag here.
    * If it changes the backend with environment variables below,
@@ -775,11 +944,20 @@ hwloc_backends_disable_all(struct hwloc_topology *topology)
   while (NULL != (backend = topology->backends)) {
     struct hwloc_backend *next = backend->next;
     if (hwloc_components_verbose)
-      fprintf(stderr, "Disabling %s discovery component `%s'\n",
+      fprintf(stderr, "Disabling discovery component `%s'\n",
-	      hwloc_disc_component_type_string(backend->component->type), backend->component->name);
+	      backend->component->name);
     hwloc_backend_disable(backend);
     topology->backends = next;
   }
   topology->backends = NULL;
-  topology->backend_excludes = 0;
+  topology->backend_excluded_phases = 0;
+}
+
+void
+hwloc_topology_components_fini(struct hwloc_topology *topology)
+{
+  /* hwloc_backends_disable_all() must have been called earlier */
+  assert(!topology->backends);
+
+  free(topology->blacklisted_components);
 }

src/3rdparty/hwloc/src/diff.c

@@ -1,11 +1,11 @@
 /*
- * Copyright © 2013-2018 Inria.  All rights reserved.
+ * Copyright © 2013-2019 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 int hwloc_topology_diff_destroy(hwloc_topology_diff_t diff)
 {
@@ -351,7 +351,8 @@ int hwloc_topology_diff_build(hwloc_topology_t topo1,
 				err = 1;
 				break;
 			}
-			if (dist1->type != dist2->type
+			if (dist1->unique_type != dist2->unique_type
+			    || dist1->different_types || dist2->different_types /* too lazy to support this case */
 			    || dist1->nbobjs != dist2->nbobjs
 			    || dist1->kind != dist2->kind
 			    || memcmp(dist1->values, dist2->values, dist1->nbobjs * dist1->nbobjs * sizeof(*dist1->values))) {
@@ -463,6 +464,10 @@ int hwloc_topology_diff_apply(hwloc_topology_t topology,
 	  errno = EINVAL;
 	  return -1;
 	}
+	if (topology->adopted_shmem_addr) {
+	  errno = EPERM;
+	  return -1;
+	}
 
 	if (flags & ~HWLOC_TOPOLOGY_DIFF_APPLY_REVERSE) {
 		errno = EINVAL;

src/3rdparty/hwloc/src/distances.c

@@ -1,19 +1,22 @@
 /*
- * Copyright © 2010-2018 Inria.  All rights reserved.
+ * Copyright © 2010-2019 Inria.  All rights reserved.
  * Copyright © 2011-2012 Université Bordeaux
  * Copyright © 2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/debug.h>
+#include "private/debug.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 #include <float.h>
 #include <math.h>
 
+static struct hwloc_internal_distances_s *
+hwloc__internal_distances_from_public(hwloc_topology_t topology, struct hwloc_distances_s *distances);
+
 /******************************************************
  * Global init, prepare, destroy, dup
  */
@@ -70,6 +73,8 @@ void hwloc_internal_distances_prepare(struct hwloc_topology *topology)
 
 static void hwloc_internal_distances_free(struct hwloc_internal_distances_s *dist)
 {
+  free(dist->name);
+  free(dist->different_types);
   free(dist->indexes);
   free(dist->objs);
   free(dist->values);
@@ -96,15 +101,35 @@ static int hwloc_internal_distances_dup_one(struct hwloc_topology *new, struct h
   newdist = hwloc_tma_malloc(tma, sizeof(*newdist));
   if (!newdist)
     return -1;
+  if (olddist->name) {
+    newdist->name = hwloc_tma_strdup(tma, olddist->name);
+    if (!newdist->name) {
+      assert(!tma || !tma->dontfree); /* this tma cannot fail to allocate */
+      hwloc_internal_distances_free(newdist);
+      return -1;
+    }
+  } else {
+    newdist->name = NULL;
+  }
 
-  newdist->type = olddist->type;
+  if (olddist->different_types) {
+    newdist->different_types = hwloc_tma_malloc(tma, nbobjs * sizeof(*newdist->different_types));
+    if (!newdist->different_types) {
+      assert(!tma || !tma->dontfree); /* this tma cannot fail to allocate */
+      hwloc_internal_distances_free(newdist);
+      return -1;
+    }
+    memcpy(newdist->different_types, olddist->different_types, nbobjs * sizeof(*newdist->different_types));
+  } else
+    newdist->different_types = NULL;
+  newdist->unique_type = olddist->unique_type;
   newdist->nbobjs = nbobjs;
   newdist->kind = olddist->kind;
   newdist->id = olddist->id;
 
   newdist->indexes = hwloc_tma_malloc(tma, nbobjs * sizeof(*newdist->indexes));
   newdist->objs = hwloc_tma_calloc(tma, nbobjs * sizeof(*newdist->objs));
-  newdist->objs_are_valid = 0;
+  newdist->iflags = olddist->iflags & ~HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID; /* must be revalidated after dup() */
   newdist->values = hwloc_tma_malloc(tma, nbobjs*nbobjs * sizeof(*newdist->values));
   if (!newdist->indexes || !newdist->objs || !newdist->values) {
     assert(!tma || !tma->dontfree); /* this tma cannot fail to allocate */
@@ -150,6 +175,10 @@ int hwloc_distances_remove(hwloc_topology_t topology)
     errno = EINVAL;
     return -1;
   }
+  if (topology->adopted_shmem_addr) {
+    errno = EPERM;
+    return -1;
+  }
   hwloc_internal_distances_destroy(topology);
   return 0;
 }
@@ -163,6 +192,10 @@ int hwloc_distances_remove_by_depth(hwloc_topology_t topology, int depth)
     errno = EINVAL;
     return -1;
   }
+  if (topology->adopted_shmem_addr) {
+    errno = EPERM;
+    return -1;
+  }
 
   /* switch back to types since we don't support groups for now */
   type = hwloc_get_depth_type(topology, depth);
@@ -174,7 +207,7 @@ int hwloc_distances_remove_by_depth(hwloc_topology_t topology, int depth)
   next = topology->first_dist;
   while ((dist = next) != NULL) {
     next = dist->next;
-    if (dist->type == type) {
+    if (dist->unique_type == type) {
       if (next)
 	next->prev = dist->prev;
       else
@@ -190,6 +223,27 @@ int hwloc_distances_remove_by_depth(hwloc_topology_t topology, int depth)
   return 0;
 }
 
+int hwloc_distances_release_remove(hwloc_topology_t topology,
+				   struct hwloc_distances_s *distances)
+{
+  struct hwloc_internal_distances_s *dist = hwloc__internal_distances_from_public(topology, distances);
+  if (!dist) {
+    errno = EINVAL;
+    return -1;
+  }
+  if (dist->prev)
+    dist->prev->next = dist->next;
+  else
+    topology->first_dist = dist->next;
+  if (dist->next)
+    dist->next->prev = dist->prev;
+  else
+    topology->last_dist = dist->prev;
+  hwloc_internal_distances_free(dist);
+  hwloc_distances_release(topology, distances);
+  return 0;
+}
+
 /******************************************************
  * Add distances to the topology
  */
@@ -201,17 +255,34 @@ hwloc__groups_by_distances(struct hwloc_topology *topology, unsigned nbobjs, str
  * the caller gives us the distances and objs pointers, we'll free them later.
  */
 static int
-hwloc_internal_distances__add(hwloc_topology_t topology,
+hwloc_internal_distances__add(hwloc_topology_t topology, const char *name,
-			      hwloc_obj_type_t type, unsigned nbobjs, hwloc_obj_t *objs, uint64_t *indexes, uint64_t *values,
+			      hwloc_obj_type_t unique_type, hwloc_obj_type_t *different_types,
-			      unsigned long kind)
+			      unsigned nbobjs, hwloc_obj_t *objs, uint64_t *indexes, uint64_t *values,
+			      unsigned long kind, unsigned iflags)
 {
-  struct hwloc_internal_distances_s *dist = calloc(1, sizeof(*dist));
+  struct hwloc_internal_distances_s *dist;
+
+  if (different_types) {
+    kind |= HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES; /* the user isn't forced to give it */
+  } else if (kind & HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES) {
+    errno = EINVAL;
+    goto err;
+  }
+
+  dist = calloc(1, sizeof(*dist));
   if (!dist)
     goto err;
 
-  dist->type = type;
+  if (name)
+    dist->name = strdup(name); /* ignore failure */
+
+  dist->unique_type = unique_type;
+  dist->different_types = different_types;
   dist->nbobjs = nbobjs;
   dist->kind = kind;
+  dist->iflags = iflags;
+
+  assert(!!(iflags & HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID) == !!objs);
 
   if (!objs) {
     assert(indexes);
@@ -220,18 +291,16 @@ hwloc_internal_distances__add(hwloc_topology_t topology,
     dist->objs = calloc(nbobjs, sizeof(hwloc_obj_t));
     if (!dist->objs)
       goto err_with_dist;
-    dist->objs_are_valid = 0;
 
   } else {
     unsigned i;
     assert(!indexes);
     /* we only have objs, generate the indexes arrays so that we can refresh objs later */
     dist->objs = objs;
-    dist->objs_are_valid = 1;
     dist->indexes = malloc(nbobjs * sizeof(*dist->indexes));
     if (!dist->indexes)
       goto err_with_dist;
-    if (dist->type == HWLOC_OBJ_PU || dist->type == HWLOC_OBJ_NUMANODE) {
+    if (HWLOC_DIST_TYPE_USE_OS_INDEX(dist->unique_type)) {
       for(i=0; i<nbobjs; i++)
 	dist->indexes[i] = objs[i]->os_index;
     } else {
@@ -256,16 +325,19 @@ hwloc_internal_distances__add(hwloc_topology_t topology,
  err_with_dist:
   free(dist);
  err:
+  free(different_types);
   free(objs);
   free(indexes);
   free(values);
   return -1;
 }
 
-int hwloc_internal_distances_add_by_index(hwloc_topology_t topology,
+int hwloc_internal_distances_add_by_index(hwloc_topology_t topology, const char *name,
-					  hwloc_obj_type_t type, unsigned nbobjs, uint64_t *indexes, uint64_t *values,
+					  hwloc_obj_type_t unique_type, hwloc_obj_type_t *different_types, unsigned nbobjs, uint64_t *indexes, uint64_t *values,
 					  unsigned long kind, unsigned long flags)
 {
+  unsigned iflags = 0; /* objs not valid */
+
   if (nbobjs < 2) {
     errno = EINVAL;
     goto err;
@@ -279,24 +351,71 @@ int hwloc_internal_distances_add_by_index(hwloc_topology_t topology,
     goto err;
   }
 
-  return hwloc_internal_distances__add(topology, type, nbobjs, NULL, indexes, values, kind);
+  return hwloc_internal_distances__add(topology, name, unique_type, different_types, nbobjs, NULL, indexes, values, kind, iflags);
 
  err:
   free(indexes);
   free(values);
+  free(different_types);
   return -1;
 }
 
-int hwloc_internal_distances_add(hwloc_topology_t topology,
+static void
+hwloc_internal_distances_restrict(hwloc_obj_t *objs,
+				  uint64_t *indexes,
+				  uint64_t *values,
+				  unsigned nbobjs, unsigned disappeared);
+
+int hwloc_internal_distances_add(hwloc_topology_t topology, const char *name,
 				 unsigned nbobjs, hwloc_obj_t *objs, uint64_t *values,
 				 unsigned long kind, unsigned long flags)
 {
+  hwloc_obj_type_t unique_type, *different_types;
+  unsigned i, disappeared = 0;
+  unsigned iflags = HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID;
+
   if (nbobjs < 2) {
     errno = EINVAL;
     goto err;
   }
 
-  if (topology->grouping && (flags & HWLOC_DISTANCES_ADD_FLAG_GROUP)) {
+  /* is there any NULL object? (useful in case of problem during insert in backends) */
+  for(i=0; i<nbobjs; i++)
+    if (!objs[i])
+      disappeared++;
+  if (disappeared) {
+    /* some objects are NULL */
+    if (disappeared == nbobjs) {
+      /* nothing left, drop the matrix */
+      free(objs);
+      free(values);
+      return 0;
+    }
+    /* restrict the matrix */
+    hwloc_internal_distances_restrict(objs, NULL, values, nbobjs, disappeared);
+    nbobjs -= disappeared;
+  }
+
+  unique_type = objs[0]->type;
+  for(i=1; i<nbobjs; i++)
+    if (objs[i]->type != unique_type) {
+      unique_type = HWLOC_OBJ_TYPE_NONE;
+      break;
+    }
+  if (unique_type == HWLOC_OBJ_TYPE_NONE) {
+    /* heterogeneous types */
+    different_types = malloc(nbobjs * sizeof(*different_types));
+    if (!different_types)
+      goto err;
+    for(i=0; i<nbobjs; i++)
+      different_types[i] = objs[i]->type;
+
+  } else {
+    /* homogeneous types */
+    different_types = NULL;
+  }
+
+  if (topology->grouping && (flags & HWLOC_DISTANCES_ADD_FLAG_GROUP) && !different_types) {
     float full_accuracy = 0.f;
     float *accuracies;
     unsigned nbaccuracies;
@@ -310,8 +429,8 @@ int hwloc_internal_distances_add(hwloc_topology_t topology,
     }
 
     if (topology->grouping_verbose) {
-      unsigned i, j;
+      unsigned j;
-      int gp = (objs[0]->type != HWLOC_OBJ_NUMANODE && objs[0]->type != HWLOC_OBJ_PU);
+      int gp = !HWLOC_DIST_TYPE_USE_OS_INDEX(unique_type);
       fprintf(stderr, "Trying to group objects using distance matrix:\n");
       fprintf(stderr, "%s", gp ? "gp_index" : "os_index");
       for(j=0; j<nbobjs; j++)
@@ -329,7 +448,7 @@ int hwloc_internal_distances_add(hwloc_topology_t topology,
 			       kind, nbaccuracies, accuracies, 1 /* check the first matrice */);
   }
 
-  return hwloc_internal_distances__add(topology, objs[0]->type, nbobjs, objs, NULL, values, kind);
+  return hwloc_internal_distances__add(topology, name, unique_type, different_types, nbobjs, objs, NULL, values, kind, iflags);
 
  err:
   free(objs);
@@ -348,7 +467,6 @@ int hwloc_distances_add(hwloc_topology_t topology,
 			unsigned nbobjs, hwloc_obj_t *objs, hwloc_uint64_t *values,
 			unsigned long kind, unsigned long flags)
 {
-  hwloc_obj_type_t type;
   unsigned i;
   uint64_t *_values;
   hwloc_obj_t *_objs;
@@ -358,6 +476,10 @@ int hwloc_distances_add(hwloc_topology_t topology,
     errno = EINVAL;
     return -1;
   }
+  if (topology->adopted_shmem_addr) {
+    errno = EPERM;
+    return -1;
+  }
   if ((kind & ~HWLOC_DISTANCES_KIND_ALL)
       || hwloc_weight_long(kind & HWLOC_DISTANCES_KIND_FROM_ALL) != 1
       || hwloc_weight_long(kind & HWLOC_DISTANCES_KIND_MEANS_ALL) != 1
@@ -368,15 +490,8 @@ int hwloc_distances_add(hwloc_topology_t topology,
 
   /* no strict need to check for duplicates, things shouldn't break */
 
-  type = objs[0]->type;
-  if (type == HWLOC_OBJ_GROUP) {
-    /* not supported yet, would require we save the subkind together with the type. */
-    errno = EINVAL;
-    return -1;
-  }
-
   for(i=1; i<nbobjs; i++)
-    if (!objs[i] || objs[i]->type != type) {
+    if (!objs[i]) {
       errno = EINVAL;
       return -1;
     }
@@ -389,7 +504,7 @@ int hwloc_distances_add(hwloc_topology_t topology,
 
   memcpy(_objs, objs, nbobjs*sizeof(hwloc_obj_t));
   memcpy(_values, values, nbobjs*nbobjs*sizeof(*_values));
-  err = hwloc_internal_distances_add(topology, nbobjs, _objs, _values, kind, flags);
+  err = hwloc_internal_distances_add(topology, NULL, nbobjs, _objs, _values, kind, flags);
   if (err < 0)
     goto out; /* _objs and _values freed in hwloc_internal_distances_add() */
 
@@ -409,9 +524,9 @@ int hwloc_distances_add(hwloc_topology_t topology,
  * Refresh objects in distances
  */
 
-static hwloc_obj_t hwloc_find_obj_by_type_and_gp_index(hwloc_topology_t topology, hwloc_obj_type_t type, uint64_t gp_index)
+static hwloc_obj_t hwloc_find_obj_by_depth_and_gp_index(hwloc_topology_t topology, unsigned depth, uint64_t gp_index)
 {
-  hwloc_obj_t obj = hwloc_get_obj_by_type(topology, type, 0);
+  hwloc_obj_t obj = hwloc_get_obj_by_depth(topology, depth, 0);
   while (obj) {
     if (obj->gp_index == gp_index)
       return obj;
@@ -420,12 +535,31 @@ static hwloc_obj_t hwloc_find_obj_by_type_and_gp_index(hwloc_topology_t topology
   return NULL;
 }
 
-static void
+static hwloc_obj_t hwloc_find_obj_by_type_and_gp_index(hwloc_topology_t topology, hwloc_obj_type_t type, uint64_t gp_index)
-hwloc_internal_distances_restrict(struct hwloc_internal_distances_s *dist,
+{
-				  hwloc_obj_t *objs,
+  int depth = hwloc_get_type_depth(topology, type);
-				  unsigned disappeared)
+  if (depth == HWLOC_TYPE_DEPTH_UNKNOWN)
+    return NULL;
+  if (depth == HWLOC_TYPE_DEPTH_MULTIPLE) {
+    int topodepth = hwloc_topology_get_depth(topology);
+    for(depth=0; depth<topodepth; depth++) {
+      if (hwloc_get_depth_type(topology, depth) == type) {
+	hwloc_obj_t obj = hwloc_find_obj_by_depth_and_gp_index(topology, depth, gp_index);
+	if (obj)
+	  return obj;
+      }
+    }
+    return NULL;
+  }
+  return hwloc_find_obj_by_depth_and_gp_index(topology, depth, gp_index);
+}
+
+static void
+hwloc_internal_distances_restrict(hwloc_obj_t *objs,
+				  uint64_t *indexes,
+				  uint64_t *values,
+				  unsigned nbobjs, unsigned disappeared)
 {
-  unsigned nbobjs = dist->nbobjs;
   unsigned i, newi;
   unsigned j, newj;
 
@@ -433,7 +567,7 @@ hwloc_internal_distances_restrict(struct hwloc_internal_distances_s *dist,
     if (objs[i]) {
       for(j=0, newj=0; j<nbobjs; j++)
 	if (objs[j]) {
-	  dist->values[newi*(nbobjs-disappeared)+newj] = dist->values[i*nbobjs+j];
+	  values[newi*(nbobjs-disappeared)+newj] = values[i*nbobjs+j];
 	  newj++;
 	}
       newi++;
@@ -442,25 +576,25 @@ hwloc_internal_distances_restrict(struct hwloc_internal_distances_s *dist,
   for(i=0, newi=0; i<nbobjs; i++)
     if (objs[i]) {
       objs[newi] = objs[i];
-      dist->indexes[newi] = dist->indexes[i];
+      if (indexes)
+	indexes[newi] = indexes[i];
       newi++;
     }
-
-  dist->nbobjs -= disappeared;
 }
 
 static int
 hwloc_internal_distances_refresh_one(hwloc_topology_t topology,
 				     struct hwloc_internal_distances_s *dist)
 {
-  hwloc_obj_type_t type = dist->type;
+  hwloc_obj_type_t unique_type = dist->unique_type;
+  hwloc_obj_type_t *different_types = dist->different_types;
   unsigned nbobjs = dist->nbobjs;
   hwloc_obj_t *objs = dist->objs;
   uint64_t *indexes = dist->indexes;
   unsigned disappeared = 0;
   unsigned i;
 
-  if (dist->objs_are_valid)
+  if (dist->iflags & HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID)
     return 0;
 
   for(i=0; i<nbobjs; i++) {
@@ -468,12 +602,16 @@ hwloc_internal_distances_refresh_one(hwloc_topology_t topology,
     /* TODO use cpuset/nodeset to find pus/numas from the root?
      * faster than traversing the entire level?
      */
-    if (type == HWLOC_OBJ_PU)
+    if (HWLOC_DIST_TYPE_USE_OS_INDEX(unique_type)) {
-      obj = hwloc_get_pu_obj_by_os_index(topology, (unsigned) indexes[i]);
+      if (unique_type == HWLOC_OBJ_PU)
-    else if (type == HWLOC_OBJ_NUMANODE)
+	obj = hwloc_get_pu_obj_by_os_index(topology, (unsigned) indexes[i]);
-      obj = hwloc_get_numanode_obj_by_os_index(topology, (unsigned) indexes[i]);
+      else if (unique_type == HWLOC_OBJ_NUMANODE)
-    else
+	obj = hwloc_get_numanode_obj_by_os_index(topology, (unsigned) indexes[i]);
-      obj = hwloc_find_obj_by_type_and_gp_index(topology, type, indexes[i]);
+      else
+	abort();
+    } else {
+      obj = hwloc_find_obj_by_type_and_gp_index(topology, different_types ? different_types[i] : unique_type, indexes[i]);
+    }
     objs[i] = obj;
     if (!obj)
       disappeared++;
@@ -483,10 +621,12 @@ hwloc_internal_distances_refresh_one(hwloc_topology_t topology,
     /* became useless, drop */
     return -1;
 
-  if (disappeared)
+  if (disappeared) {
-    hwloc_internal_distances_restrict(dist, objs, disappeared);
+    hwloc_internal_distances_restrict(objs, dist->indexes, dist->values, nbobjs, disappeared);
+    dist->nbobjs -= disappeared;
+  }
 
-  dist->objs_are_valid = 1;
+  dist->iflags |= HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID;
   return 0;
 }
 
@@ -520,32 +660,64 @@ hwloc_internal_distances_invalidate_cached_objs(hwloc_topology_t topology)
 {
   struct hwloc_internal_distances_s *dist;
   for(dist = topology->first_dist; dist; dist = dist->next)
-    dist->objs_are_valid = 0;
+    dist->iflags &= ~HWLOC_INTERNAL_DIST_FLAG_OBJS_VALID;
 }
 
 /******************************************************
  * User API for getting distances
  */
 
+/* what we actually allocate for user queries, even if we only
+ * return the distances part of it.
+ */
+struct hwloc_distances_container_s {
+  unsigned id;
+  struct hwloc_distances_s distances;
+};
+
+#define HWLOC_DISTANCES_CONTAINER_OFFSET ((char*)&((struct hwloc_distances_container_s*)NULL)->distances - (char*)NULL)
+#define HWLOC_DISTANCES_CONTAINER(_d) (struct hwloc_distances_container_s *) ( ((char*)_d) - HWLOC_DISTANCES_CONTAINER_OFFSET )
+
+static struct hwloc_internal_distances_s *
+hwloc__internal_distances_from_public(hwloc_topology_t topology, struct hwloc_distances_s *distances)
+{
+  struct hwloc_distances_container_s *cont = HWLOC_DISTANCES_CONTAINER(distances);
+  struct hwloc_internal_distances_s *dist;
+  for(dist = topology->first_dist; dist; dist = dist->next)
+    if (dist->id == cont->id)
+      return dist;
+  return NULL;
+}
+
 void
 hwloc_distances_release(hwloc_topology_t topology __hwloc_attribute_unused,
 			struct hwloc_distances_s *distances)
 {
+  struct hwloc_distances_container_s *cont = HWLOC_DISTANCES_CONTAINER(distances);
   free(distances->values);
   free(distances->objs);
-  free(distances);
+  free(cont);
+}
+
+const char *
+hwloc_distances_get_name(hwloc_topology_t topology, struct hwloc_distances_s *distances)
+{
+  struct hwloc_internal_distances_s *dist = hwloc__internal_distances_from_public(topology, distances);
+  return dist ? dist->name : NULL;
 }
 
 static struct hwloc_distances_s *
 hwloc_distances_get_one(hwloc_topology_t topology __hwloc_attribute_unused,
 			struct hwloc_internal_distances_s *dist)
 {
+  struct hwloc_distances_container_s *cont;
   struct hwloc_distances_s *distances;
   unsigned nbobjs;
 
-  distances = malloc(sizeof(*distances));
+  cont = malloc(sizeof(*cont));
-  if (!distances)
+  if (!cont)
     return NULL;
+  distances = &cont->distances;
 
   nbobjs = distances->nbobjs = dist->nbobjs;
 
@@ -560,18 +732,20 @@ hwloc_distances_get_one(hwloc_topology_t topology __hwloc_attribute_unused,
   memcpy(distances->values, dist->values, nbobjs*nbobjs*sizeof(*distances->values));
 
   distances->kind = dist->kind;
+
+  cont->id = dist->id;
   return distances;
 
  out_with_objs:
   free(distances->objs);
  out:
-  free(distances);
+  free(cont);
   return NULL;
 }
 
 static int
 hwloc__distances_get(hwloc_topology_t topology,
-		     hwloc_obj_type_t type,
+		     const char *name, hwloc_obj_type_t type,
 		     unsigned *nrp, struct hwloc_distances_s **distancesp,
 		     unsigned long kind, unsigned long flags __hwloc_attribute_unused)
 {
@@ -602,7 +776,10 @@ hwloc__distances_get(hwloc_topology_t topology,
     unsigned long kind_from = kind & HWLOC_DISTANCES_KIND_FROM_ALL;
     unsigned long kind_means = kind & HWLOC_DISTANCES_KIND_MEANS_ALL;
 
-    if (type != HWLOC_OBJ_TYPE_NONE && type != dist->type)
+    if (name && (!dist->name || strcmp(name, dist->name)))
+      continue;
+
+    if (type != HWLOC_OBJ_TYPE_NONE && type != dist->unique_type)
       continue;
 
     if (kind_from && !(kind_from & dist->kind))
@@ -640,7 +817,7 @@ hwloc_distances_get(hwloc_topology_t topology,
     return -1;
   }
 
-  return hwloc__distances_get(topology, HWLOC_OBJ_TYPE_NONE, nrp, distancesp, kind, flags);
+  return hwloc__distances_get(topology, NULL, HWLOC_OBJ_TYPE_NONE, nrp, distancesp, kind, flags);
 }
 
 int
@@ -655,14 +832,40 @@ hwloc_distances_get_by_depth(hwloc_topology_t topology, int depth,
     return -1;
   }
 
-  /* switch back to types since we don't support groups for now */
+  /* FIXME: passing the depth of a group level may return group distances at a different depth */
   type = hwloc_get_depth_type(topology, depth);
   if (type == (hwloc_obj_type_t)-1) {
     errno = EINVAL;
     return -1;
   }
 
-  return hwloc__distances_get(topology, type, nrp, distancesp, kind, flags);
+  return hwloc__distances_get(topology, NULL, type, nrp, distancesp, kind, flags);
+}
+
+int
+hwloc_distances_get_by_name(hwloc_topology_t topology, const char *name,
+			    unsigned *nrp, struct hwloc_distances_s **distancesp,
+			    unsigned long flags)
+{
+  if (flags || !topology->is_loaded) {
+    errno = EINVAL;
+    return -1;
+  }
+
+  return hwloc__distances_get(topology, name, HWLOC_OBJ_TYPE_NONE, nrp, distancesp, HWLOC_DISTANCES_KIND_ALL, flags);
+}
+
+int
+hwloc_distances_get_by_type(hwloc_topology_t topology, hwloc_obj_type_t type,
+			    unsigned *nrp, struct hwloc_distances_s **distancesp,
+			    unsigned long kind, unsigned long flags)
+{
+  if (flags || !topology->is_loaded) {
+    errno = EINVAL;
+    return -1;
+  }
+
+  return hwloc__distances_get(topology, NULL, type, nrp, distancesp, kind, flags);
 }
 
 /******************************************************
@@ -823,10 +1026,14 @@ hwloc__groups_by_distances(struct hwloc_topology *topology,
 			   float *accuracies,
 			   int needcheck)
 {
-  HWLOC_VLA(unsigned, groupids, nbobjs);
+  unsigned *groupids;
   unsigned nbgroups = 0;
   unsigned i,j;
   int verbose = topology->grouping_verbose;
+  hwloc_obj_t *groupobjs;
+  unsigned * groupsizes;
+  uint64_t *groupvalues;
+  unsigned failed = 0;
 
   if (nbobjs <= 2)
       return;
@@ -836,6 +1043,10 @@ hwloc__groups_by_distances(struct hwloc_topology *topology,
     /* TODO hwloc__find_groups_by_max_distance() for bandwidth */
     return;
 
+  groupids = malloc(nbobjs * sizeof(*groupids));
+  if (!groupids)
+    return;
+
   for(i=0; i<nbaccuracies; i++) {
     if (verbose)
       fprintf(stderr, "Trying to group %u %s objects according to physical distances with accuracy %f\n",
@@ -847,13 +1058,13 @@ hwloc__groups_by_distances(struct hwloc_topology *topology,
       break;
   }
   if (!nbgroups)
-    return;
+    goto out_with_groupids;
 
-  {
+  groupobjs = malloc(nbgroups * sizeof(*groupobjs));
-      HWLOC_VLA(hwloc_obj_t, groupobjs, nbgroups);
+  groupsizes = malloc(nbgroups * sizeof(*groupsizes));
-      HWLOC_VLA(unsigned, groupsizes, nbgroups);
+  groupvalues = malloc(nbgroups * nbgroups * sizeof(*groupvalues));
-      HWLOC_VLA(uint64_t, groupvalues, nbgroups*nbgroups);
+  if (!groupobjs || !groupsizes || !groupvalues)
-      unsigned failed = 0;
+    goto out_with_groups;
 
       /* create new Group objects and record their size */
       memset(&(groupsizes[0]), 0, sizeof(groupsizes[0]) * nbgroups);
@@ -884,7 +1095,7 @@ hwloc__groups_by_distances(struct hwloc_topology *topology,
 
       if (failed)
 	/* don't try to group above if we got a NULL group here, just keep this incomplete level */
-	return;
+	goto out_with_groups;
 
       /* factorize values */
       memset(&(groupvalues[0]), 0, sizeof(groupvalues[0]) * nbgroups * nbgroups);
@@ -916,5 +1127,11 @@ hwloc__groups_by_distances(struct hwloc_topology *topology,
 #endif
 
       hwloc__groups_by_distances(topology, nbgroups, groupobjs, groupvalues, kind, nbaccuracies, accuracies, 0 /* no need to check generated matrix */);
-  }
+
+ out_with_groups:
+  free(groupobjs);
+  free(groupsizes);
+  free(groupvalues);
+ out_with_groupids:
+  free(groupids);
 }

src/3rdparty/hwloc/src/misc.c

@@ -1,14 +1,14 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2015 Inria.  All rights reserved.
+ * Copyright © 2009-2018 Inria.  All rights reserved.
  * Copyright © 2009-2010 Université Bordeaux
  * Copyright © 2009-2018 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 #include <stdarg.h>
 #ifdef HAVE_SYS_UTSNAME_H
@@ -28,6 +28,7 @@ extern char *program_invocation_name;
 extern char *__progname;
 #endif
 
+#ifndef HWLOC_HAVE_CORRECT_SNPRINTF
 int hwloc_snprintf(char *str, size_t size, const char *format, ...)
 {
   int ret;
@@ -77,21 +78,7 @@ int hwloc_snprintf(char *str, size_t size, const char *format, ...)
 
   return ret;
 }
-
+#endif
-int hwloc_namecoloncmp(const char *haystack, const char *needle, size_t n)
-{
-  size_t i = 0;
-  while (*haystack && *haystack != ':') {
-    int ha = *haystack++;
-    int low_h = tolower(ha);
-    int ne = *needle++;
-    int low_n = tolower(ne);
-    if (low_h != low_n)
-      return 1;
-    i++;
-  }
-  return i < n;
-}
 
 void hwloc_add_uname_info(struct hwloc_topology *topology __hwloc_attribute_unused,
 			  void *cached_uname __hwloc_attribute_unused)

src/3rdparty/hwloc/src/pci-common.c

@@ -1,14 +1,14 @@
 /*
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/plugins.h>
+#include "hwloc/plugins.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/debug.h>
+#include "private/debug.h"
-#include <private/misc.h>
+#include "private/misc.h"
 
 #include <fcntl.h>
 #ifdef HAVE_UNISTD_H
@@ -23,6 +23,11 @@
 #define close _close
 #endif
 
+
+/**************************************
+ * Init/Exit and Forced PCI localities
+ */
+
 static void
 hwloc_pci_forced_locality_parse_one(struct hwloc_topology *topology,
 				    const char *string /* must contain a ' ' */,
@@ -109,11 +114,11 @@ hwloc_pci_forced_locality_parse(struct hwloc_topology *topology, const char *_en
 void
 hwloc_pci_discovery_init(struct hwloc_topology *topology)
 {
-  topology->need_pci_belowroot_apply_locality = 0;
-
   topology->pci_has_forced_locality = 0;
   topology->pci_forced_locality_nr = 0;
   topology->pci_forced_locality = NULL;
+
+  topology->first_pci_locality = topology->last_pci_locality = NULL;
 }
 
 void
@@ -135,7 +140,7 @@ hwloc_pci_discovery_prepare(struct hwloc_topology *topology)
       if (!err) {
 	if (st.st_size <= 64*1024) { /* random limit large enough to store multiple cpusets for thousands of PUs */
 	  buffer = malloc(st.st_size+1);
-	  if (read(fd, buffer, st.st_size) == st.st_size) {
+	  if (buffer && read(fd, buffer, st.st_size) == st.st_size) {
 	    buffer[st.st_size] = '\0';
 	    hwloc_pci_forced_locality_parse(topology, buffer);
 	  }
@@ -152,16 +157,31 @@ hwloc_pci_discovery_prepare(struct hwloc_topology *topology)
 }
 
 void
-hwloc_pci_discovery_exit(struct hwloc_topology *topology __hwloc_attribute_unused)
+hwloc_pci_discovery_exit(struct hwloc_topology *topology)
 {
+  struct hwloc_pci_locality_s *cur;
   unsigned i;
+
   for(i=0; i<topology->pci_forced_locality_nr; i++)
     hwloc_bitmap_free(topology->pci_forced_locality[i].cpuset);
   free(topology->pci_forced_locality);
 
+  cur = topology->first_pci_locality;
+  while (cur) {
+    struct hwloc_pci_locality_s *next = cur->next;
+    hwloc_bitmap_free(cur->cpuset);
+    free(cur);
+    cur = next;
+  }
+
   hwloc_pci_discovery_init(topology);
 }
 
+
+/******************************
+ * Inserting in Tree by Bus ID
+ */
+
 #ifdef HWLOC_DEBUG
 static void
 hwloc_pci_traverse_print_cb(void * cbdata __hwloc_attribute_unused,
@@ -324,32 +344,16 @@ hwloc_pcidisc_tree_insert_by_busid(struct hwloc_obj **treep,
   hwloc_pci_add_object(NULL /* no parent on top of tree */, treep, obj);
 }
 
-int
+
-hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, struct hwloc_obj *old_tree)
+/**********************
+ * Attaching PCI Trees
+ */
+
+static struct hwloc_obj *
+hwloc_pcidisc_add_hostbridges(struct hwloc_topology *topology,
+			      struct hwloc_obj *old_tree)
 {
-  struct hwloc_obj **next_hb_p;
+  struct hwloc_obj * new = NULL, **newp = &new;
-  enum hwloc_type_filter_e bfilter;
-
-  if (!old_tree)
-    /* found nothing, exit */
-    return 0;
-
-#ifdef HWLOC_DEBUG
-  hwloc_debug("%s", "\nPCI hierarchy:\n");
-  hwloc_pci_traverse(NULL, old_tree, hwloc_pci_traverse_print_cb);
-  hwloc_debug("%s", "\n");
-#endif
-
-  next_hb_p = &hwloc_get_root_obj(topology)->io_first_child;
-  while (*next_hb_p)
-    next_hb_p = &((*next_hb_p)->next_sibling);
-
-  bfilter = topology->type_filter[HWLOC_OBJ_BRIDGE];
-  if (bfilter == HWLOC_TYPE_FILTER_KEEP_NONE) {
-    *next_hb_p = old_tree;
-    topology->modified = 1;
-    goto done;
-  }
 
   /*
    * tree points to all objects connected to any upstream bus in the machine.
@@ -358,15 +362,29 @@ hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, struct hwloc_obj *old
    */
   while (old_tree) {
     /* start a new host bridge */
-    struct hwloc_obj *hostbridge = hwloc_alloc_setup_object(topology, HWLOC_OBJ_BRIDGE, HWLOC_UNKNOWN_INDEX);
+    struct hwloc_obj *hostbridge;
-    struct hwloc_obj **dstnextp = &hostbridge->io_first_child;
+    struct hwloc_obj **dstnextp;
-    struct hwloc_obj **srcnextp = &old_tree;
+    struct hwloc_obj **srcnextp;
-    struct hwloc_obj *child = *srcnextp;
+    struct hwloc_obj *child;
-    unsigned short current_domain = child->attr->pcidev.domain;
+    unsigned short current_domain;
-    unsigned char current_bus = child->attr->pcidev.bus;
+    unsigned char current_bus;
-    unsigned char current_subordinate = current_bus;
+    unsigned char current_subordinate;
 
-    hwloc_debug("Starting new PCI hostbridge %04x:%02x\n", current_domain, current_bus);
+    hostbridge = hwloc_alloc_setup_object(topology, HWLOC_OBJ_BRIDGE, HWLOC_UNKNOWN_INDEX);
+    if (!hostbridge) {
+      /* just queue remaining things without hostbridges and return */
+      *newp = old_tree;
+      return new;
+    }
+    dstnextp = &hostbridge->io_first_child;
+
+    srcnextp = &old_tree;
+    child = *srcnextp;
+    current_domain = child->attr->pcidev.domain;
+    current_bus = child->attr->pcidev.bus;
+    current_subordinate = current_bus;
+
+    hwloc_debug("Adding new PCI hostbridge %04x:%02x\n", current_domain, current_bus);
 
   next_child:
     /* remove next child from tree */
@@ -395,19 +413,14 @@ hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, struct hwloc_obj *old
     hostbridge->attr->bridge.downstream.pci.domain = current_domain;
     hostbridge->attr->bridge.downstream.pci.secondary_bus = current_bus;
     hostbridge->attr->bridge.downstream.pci.subordinate_bus = current_subordinate;
-    hwloc_debug("New PCI hostbridge %04x:[%02x-%02x]\n",
+    hwloc_debug("  new PCI hostbridge covers %04x:[%02x-%02x]\n",
 		current_domain, current_bus, current_subordinate);
 
-    *next_hb_p = hostbridge;
+    *newp = hostbridge;
-    next_hb_p = &hostbridge->next_sibling;
+    newp = &hostbridge->next_sibling;
-    topology->modified = 1; /* needed in case somebody reconnects levels before the core calls hwloc_pci_belowroot_apply_locality()
-			     * or if hwloc_pci_belowroot_apply_locality() keeps hostbridges below root.
-			     */
   }
 
- done:
+  return new;
-  topology->need_pci_belowroot_apply_locality = 1;
-  return 0;
 }
 
 static struct hwloc_obj *
@@ -458,6 +471,9 @@ hwloc__pci_find_busid_parent(struct hwloc_topology *topology, struct hwloc_pcide
   unsigned i;
   int err;
 
+  hwloc_debug("Looking for parent of PCI busid %04x:%02x:%02x.%01x\n",
+	      busid->domain, busid->bus, busid->dev, busid->func);
+
   /* try to match a forced locality */
   if (topology->pci_has_forced_locality) {
     for(i=0; i<topology->pci_forced_locality_nr; i++) {
@@ -489,7 +505,7 @@ hwloc__pci_find_busid_parent(struct hwloc_topology *topology, struct hwloc_pcide
       }
       if (*env) {
 	/* force the cpuset */
-	hwloc_debug("Overriding localcpus using %s in the environment\n", envname);
+	hwloc_debug("Overriding PCI locality using %s in the environment\n", envname);
 	hwloc_bitmap_sscanf(cpuset, env);
 	forced = 1;
       }
@@ -499,7 +515,7 @@ hwloc__pci_find_busid_parent(struct hwloc_topology *topology, struct hwloc_pcide
   }
 
   if (!forced) {
-    /* get the cpuset by asking the OS backend. */
+    /* get the cpuset by asking the backend that provides the relevant hook, if any. */
     struct hwloc_backend *backend = topology->get_pci_busid_cpuset_backend;
     if (backend)
       err = backend->get_pci_busid_cpuset(backend, busid, cpuset);
@@ -510,7 +526,7 @@ hwloc__pci_find_busid_parent(struct hwloc_topology *topology, struct hwloc_pcide
       hwloc_bitmap_copy(cpuset, hwloc_topology_get_topology_cpuset(topology));
   }
 
-  hwloc_debug_bitmap("Attaching PCI tree to cpuset %s\n", cpuset);
+  hwloc_debug_bitmap("  will attach PCI bus to cpuset %s\n", cpuset);
 
   parent = hwloc_find_insert_io_parent_by_complete_cpuset(topology, cpuset);
   if (parent) {
@@ -526,11 +542,129 @@ hwloc__pci_find_busid_parent(struct hwloc_topology *topology, struct hwloc_pcide
   return parent;
 }
 
+int
+hwloc_pcidisc_tree_attach(struct hwloc_topology *topology, struct hwloc_obj *tree)
+{
+  enum hwloc_type_filter_e bfilter;
+
+  if (!tree)
+    /* found nothing, exit */
+    return 0;
+
+#ifdef HWLOC_DEBUG
+  hwloc_debug("%s", "\nPCI hierarchy:\n");
+  hwloc_pci_traverse(NULL, tree, hwloc_pci_traverse_print_cb);
+  hwloc_debug("%s", "\n");
+#endif
+
+  bfilter = topology->type_filter[HWLOC_OBJ_BRIDGE];
+  if (bfilter != HWLOC_TYPE_FILTER_KEEP_NONE) {
+    tree = hwloc_pcidisc_add_hostbridges(topology, tree);
+  }
+
+  while (tree) {
+    struct hwloc_obj *obj, *pciobj;
+    struct hwloc_obj *parent;
+    struct hwloc_pci_locality_s *loc;
+    unsigned domain, bus_min, bus_max;
+
+    obj = tree;
+
+    /* hostbridges don't have a PCI busid for looking up locality, use their first child */
+    if (obj->type == HWLOC_OBJ_BRIDGE && obj->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_HOST)
+      pciobj = obj->io_first_child;
+    else
+      pciobj = obj;
+    /* now we have a pci device or a pci bridge */
+    assert(pciobj->type == HWLOC_OBJ_PCI_DEVICE
+	   || (pciobj->type == HWLOC_OBJ_BRIDGE && pciobj->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_PCI));
+
+    if (obj->type == HWLOC_OBJ_BRIDGE) {
+      domain = obj->attr->bridge.downstream.pci.domain;
+      bus_min = obj->attr->bridge.downstream.pci.secondary_bus;
+      bus_max = obj->attr->bridge.downstream.pci.subordinate_bus;
+    } else {
+      domain = pciobj->attr->pcidev.domain;
+      bus_min = pciobj->attr->pcidev.bus;
+      bus_max = pciobj->attr->pcidev.bus;
+    }
+
+    /* find where to attach that PCI bus */
+    parent = hwloc__pci_find_busid_parent(topology, &pciobj->attr->pcidev);
+
+    /* reuse the previous locality if possible */
+    if (topology->last_pci_locality
+	&& parent == topology->last_pci_locality->parent
+	&& domain == topology->last_pci_locality->domain
+	&& (bus_min == topology->last_pci_locality->bus_max
+	    || bus_min == topology->last_pci_locality->bus_max+1)) {
+      hwloc_debug("  Reusing PCI locality up to bus %04x:%02x\n",
+		  domain, bus_max);
+      topology->last_pci_locality->bus_max = bus_max;
+      goto done;
+    }
+
+    loc = malloc(sizeof(*loc));
+    if (!loc) {
+      /* fallback to attaching to root */
+      parent = hwloc_get_root_obj(topology);
+      goto done;
+    }
+
+    loc->domain = domain;
+    loc->bus_min = bus_min;
+    loc->bus_max = bus_max;
+    loc->parent = parent;
+    loc->cpuset = hwloc_bitmap_dup(parent->cpuset);
+    if (!loc->cpuset) {
+      /* fallback to attaching to root */
+      free(loc);
+      parent = hwloc_get_root_obj(topology);
+      goto done;
+    }
+
+    hwloc_debug("Adding PCI locality %s P#%u for bus %04x:[%02x:%02x]\n",
+		hwloc_obj_type_string(parent->type), parent->os_index, loc->domain, loc->bus_min, loc->bus_max);
+    if (topology->last_pci_locality) {
+      loc->prev = topology->last_pci_locality;
+      loc->next = NULL;
+      topology->last_pci_locality->next = loc;
+      topology->last_pci_locality = loc;
+    } else {
+      loc->prev = NULL;
+      loc->next = NULL;
+      topology->first_pci_locality = loc;
+      topology->last_pci_locality = loc;
+    }
+
+  done:
+    /* dequeue this object */
+    tree = obj->next_sibling;
+    obj->next_sibling = NULL;
+    hwloc_insert_object_by_parent(topology, parent, obj);
+  }
+
+  return 0;
+}
+
+
+/*********************************
+ * Finding PCI objects or parents
+ */
+
 struct hwloc_obj *
-hwloc_pcidisc_find_busid_parent(struct hwloc_topology *topology,
+hwloc_pci_find_parent_by_busid(struct hwloc_topology *topology,
-				unsigned domain, unsigned bus, unsigned dev, unsigned func)
+			       unsigned domain, unsigned bus, unsigned dev, unsigned func)
 {
   struct hwloc_pcidev_attr_s busid;
+  hwloc_obj_t parent;
+
+  /* try to find that exact busid */
+  parent = hwloc_pci_find_by_busid(topology, domain, bus, dev, func);
+  if (parent)
+    return parent;
+
+  /* try to find the locality of that bus instead */
   busid.domain = domain;
   busid.bus = bus;
   busid.dev = dev;
@@ -538,66 +672,10 @@ hwloc_pcidisc_find_busid_parent(struct hwloc_topology *topology,
   return hwloc__pci_find_busid_parent(topology, &busid);
 }
 
-int
+/* return the smallest object that contains the desired busid */
-hwloc_pci_belowroot_apply_locality(struct hwloc_topology *topology)
-{
-  struct hwloc_obj *root = hwloc_get_root_obj(topology);
-  struct hwloc_obj **listp, *obj;
-
-  if (!topology->need_pci_belowroot_apply_locality)
-    return 0;
-  topology->need_pci_belowroot_apply_locality = 0;
-
-  /* root->io_first_child contains some PCI hierarchies, any maybe some non-PCI things.
-   * insert the PCI trees according to their PCI-locality.
-   */
-  listp = &root->io_first_child;
-  while ((obj = *listp) != NULL) {
-    struct hwloc_pcidev_attr_s *busid;
-    struct hwloc_obj *parent;
-
-    /* skip non-PCI objects */
-    if (obj->type != HWLOC_OBJ_PCI_DEVICE
-	&& !(obj->type == HWLOC_OBJ_BRIDGE && obj->attr->bridge.downstream_type == HWLOC_OBJ_BRIDGE_PCI)
-	&& !(obj->type == HWLOC_OBJ_BRIDGE && obj->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_PCI)) {
-      listp = &obj->next_sibling;
-      continue;
-    }
-
-    if (obj->type == HWLOC_OBJ_PCI_DEVICE
-	|| (obj->type == HWLOC_OBJ_BRIDGE
-	    && obj->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_PCI))
-      busid = &obj->attr->pcidev;
-    else {
-      /* hostbridges don't have a PCI busid for looking up locality, use their first child if PCI */
-      hwloc_obj_t child = obj->io_first_child;
-      if (child && (child->type == HWLOC_OBJ_PCI_DEVICE
-		    || (child->type == HWLOC_OBJ_BRIDGE
-			&& child->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_PCI)))
-	busid = &obj->io_first_child->attr->pcidev;
-      else
-	continue;
-    }
-
-    /* attach the object (and children) where it belongs */
-    parent = hwloc__pci_find_busid_parent(topology, busid);
-    if (parent == root) {
-      /* keep this object here */
-      listp = &obj->next_sibling;
-    } else {
-      /* dequeue this object */
-      *listp = obj->next_sibling;
-      obj->next_sibling = NULL;
-      hwloc_insert_object_by_parent(topology, parent, obj);
-    }
-  }
-
-  return 0;
-}
-
 static struct hwloc_obj *
-hwloc__pci_belowroot_find_by_busid(hwloc_obj_t parent,
+hwloc__pci_find_by_busid(hwloc_obj_t parent,
-				   unsigned domain, unsigned bus, unsigned dev, unsigned func)
+			 unsigned domain, unsigned bus, unsigned dev, unsigned func)
 {
   hwloc_obj_t child;
 
@@ -622,7 +700,7 @@ hwloc__pci_belowroot_find_by_busid(hwloc_obj_t parent,
 	  && child->attr->bridge.downstream.pci.secondary_bus <= bus
 	  && child->attr->bridge.downstream.pci.subordinate_bus >= bus)
 	/* not the right bus id, but it's included in the bus below that bridge */
-	return hwloc__pci_belowroot_find_by_busid(child, domain, bus, dev, func);
+	return hwloc__pci_find_by_busid(child, domain, bus, dev, func);
 
     } else if (child->type == HWLOC_OBJ_BRIDGE
 	       && child->attr->bridge.upstream_type != HWLOC_OBJ_BRIDGE_PCI
@@ -632,7 +710,7 @@ hwloc__pci_belowroot_find_by_busid(hwloc_obj_t parent,
 	       && child->attr->bridge.downstream.pci.secondary_bus <= bus
 	       && child->attr->bridge.downstream.pci.subordinate_bus >= bus) {
       /* contains our bus, recurse */
-      return hwloc__pci_belowroot_find_by_busid(child, domain, bus, dev, func);
+      return hwloc__pci_find_by_busid(child, domain, bus, dev, func);
     }
   }
   /* didn't find anything, return parent */
@@ -640,17 +718,54 @@ hwloc__pci_belowroot_find_by_busid(hwloc_obj_t parent,
 }
 
 struct hwloc_obj *
-hwloc_pcidisc_find_by_busid(struct hwloc_topology *topology,
+hwloc_pci_find_by_busid(struct hwloc_topology *topology,
-			    unsigned domain, unsigned bus, unsigned dev, unsigned func)
+			unsigned domain, unsigned bus, unsigned dev, unsigned func)
 {
+  struct hwloc_pci_locality_s *loc;
   hwloc_obj_t root = hwloc_get_root_obj(topology);
-  hwloc_obj_t parent = hwloc__pci_belowroot_find_by_busid(root, domain, bus, dev, func);
+  hwloc_obj_t parent = NULL;
-  if (parent == root)
+
+  hwloc_debug("pcidisc looking for bus id %04x:%02x:%02x.%01x\n", domain, bus, dev, func);
+  loc = topology->first_pci_locality;
+  while (loc) {
+    if (loc->domain == domain && loc->bus_min <= bus && loc->bus_max >= bus) {
+      parent = loc->parent;
+      assert(parent);
+      hwloc_debug("  found pci locality for %04x:[%02x:%02x]\n",
+		  loc->domain, loc->bus_min, loc->bus_max);
+      break;
+    }
+    loc = loc->next;
+  }
+  /* if we failed to insert localities, look at root too */
+  if (!parent)
+    parent = root;
+
+  hwloc_debug("  looking for bus %04x:%02x:%02x.%01x below %s P#%u\n",
+	      domain, bus, dev, func,
+	      hwloc_obj_type_string(parent->type), parent->os_index);
+  parent = hwloc__pci_find_by_busid(parent, domain, bus, dev, func);
+  if (parent == root) {
+    hwloc_debug("  found nothing better than root object, ignoring\n");
     return NULL;
-  else
+  } else {
+    if (parent->type == HWLOC_OBJ_PCI_DEVICE
+	|| (parent->type == HWLOC_OBJ_BRIDGE && parent->attr->bridge.upstream_type == HWLOC_OBJ_BRIDGE_PCI))
+      hwloc_debug("  found busid %04x:%02x:%02x.%01x\n",
+		  parent->attr->pcidev.domain, parent->attr->pcidev.bus,
+		  parent->attr->pcidev.dev, parent->attr->pcidev.func);
+    else
+      hwloc_debug("  found parent %s P#%u\n",
+		  hwloc_obj_type_string(parent->type), parent->os_index);
     return parent;
+  }
 }
 
+
+/*******************************
+ * Parsing the PCI Config Space
+ */
+
 #define HWLOC_PCI_STATUS 0x06
 #define HWLOC_PCI_STATUS_CAP_LIST 0x10
 #define HWLOC_PCI_CAPABILITY_LIST 0x34
@@ -703,13 +818,14 @@ hwloc_pcidisc_find_linkspeed(const unsigned char *config,
    * PCIe Gen2 = 5  GT/s signal-rate per lane with 8/10 encoding    = 0.5 GB/s data-rate per lane
    * PCIe Gen3 = 8  GT/s signal-rate per lane with 128/130 encoding = 1   GB/s data-rate per lane
    * PCIe Gen4 = 16 GT/s signal-rate per lane with 128/130 encoding = 2   GB/s data-rate per lane
+   * PCIe Gen5 = 32 GT/s signal-rate per lane with 128/130 encoding = 4   GB/s data-rate per lane
    */
 
   /* lanespeed in Gbit/s */
   if (speed <= 2)
     lanespeed = 2.5f * speed * 0.8f;
   else
-    lanespeed = 8.0f * (1<<(speed-3)) * 128/130; /* assume Gen5 will be 32 GT/s and so on */
+    lanespeed = 8.0f * (1<<(speed-3)) * 128/130; /* assume Gen6 will be 64 GT/s and so on */
 
   /* linkspeed in GB/s */
   *linkspeed = lanespeed * width / 8;
@@ -738,30 +854,27 @@ hwloc_pcidisc_check_bridge_type(unsigned device_class, const unsigned char *conf
 #define HWLOC_PCI_SUBORDINATE_BUS 0x1a
 
 int
-hwloc_pcidisc_setup_bridge_attr(hwloc_obj_t obj,
+hwloc_pcidisc_find_bridge_buses(unsigned domain, unsigned bus, unsigned dev, unsigned func,
+				unsigned *secondary_busp, unsigned *subordinate_busp,
 				const unsigned char *config)
 {
-  struct hwloc_bridge_attr_s *battr = &obj->attr->bridge;
+  unsigned secondary_bus, subordinate_bus;
-  struct hwloc_pcidev_attr_s *pattr = &battr->upstream.pci;
 
-  if (config[HWLOC_PCI_PRIMARY_BUS] != pattr->bus) {
+  if (config[HWLOC_PCI_PRIMARY_BUS] != bus) {
     /* Sometimes the config space contains 00 instead of the actual primary bus number.
      * Always trust the bus ID because it was built by the system which has more information
      * to workaround such problems (e.g. ACPI information about PCI parent/children).
      */
     hwloc_debug("  %04x:%02x:%02x.%01x bridge with (ignored) invalid PCI_PRIMARY_BUS %02x\n",
-		pattr->domain, pattr->bus, pattr->dev, pattr->func, config[HWLOC_PCI_PRIMARY_BUS]);
+		domain, bus, dev, func, config[HWLOC_PCI_PRIMARY_BUS]);
   }
 
-  battr->upstream_type = HWLOC_OBJ_BRIDGE_PCI;
+  secondary_bus = config[HWLOC_PCI_SECONDARY_BUS];
-  battr->downstream_type = HWLOC_OBJ_BRIDGE_PCI;
+  subordinate_bus = config[HWLOC_PCI_SUBORDINATE_BUS];
-  battr->downstream.pci.domain = pattr->domain;
-  battr->downstream.pci.secondary_bus = config[HWLOC_PCI_SECONDARY_BUS];
-  battr->downstream.pci.subordinate_bus = config[HWLOC_PCI_SUBORDINATE_BUS];
 
-  if (battr->downstream.pci.secondary_bus <= pattr->bus
+  if (secondary_bus <= bus
-      || battr->downstream.pci.subordinate_bus <= pattr->bus
+      || subordinate_bus <= bus
-      || battr->downstream.pci.secondary_bus > battr->downstream.pci.subordinate_bus) {
+      || secondary_bus > subordinate_bus) {
     /* This should catch most cases of invalid bridge information
      * (e.g. 00 for secondary and subordinate).
      * Ideally we would also check that [secondary-subordinate] is included
@@ -769,15 +882,21 @@ hwloc_pcidisc_setup_bridge_attr(hwloc_obj_t obj,
      * because objects may be discovered out of order (especially in the fsroot case).
      */
     hwloc_debug("  %04x:%02x:%02x.%01x bridge has invalid secondary-subordinate buses [%02x-%02x]\n",
-		pattr->domain, pattr->bus, pattr->dev, pattr->func,
+		domain, bus, dev, func,
-		battr->downstream.pci.secondary_bus, battr->downstream.pci.subordinate_bus);
+		secondary_bus, subordinate_bus);
-    hwloc_free_unlinked_object(obj);
     return -1;
   }
 
+  *secondary_busp = secondary_bus;
+  *subordinate_busp = subordinate_bus;
   return 0;
 }
 
+
+/****************
+ * Class Strings
+ */
+
 const char *
 hwloc_pci_class_string(unsigned short class_id)
 {

src/3rdparty/hwloc/src/shmem.c

@@ -1,12 +1,12 @@
 /*
- * Copyright © 2017-2018 Inria.  All rights reserved.
+ * Copyright © 2017-2019 Inria.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/shmem.h>
+#include "hwloc/shmem.h"
-#include <private/private.h>
+#include "private/private.h"
 
 #ifndef HWLOC_WIN_SYS
 
@@ -214,6 +214,8 @@ hwloc_shmem_topology_adopt(hwloc_topology_t *topologyp,
   new->support.discovery = malloc(sizeof(*new->support.discovery));
   new->support.cpubind = malloc(sizeof(*new->support.cpubind));
   new->support.membind = malloc(sizeof(*new->support.membind));
+  if (!new->support.discovery || !new->support.cpubind || !new->support.membind)
+    goto out_with_support;
   memcpy(new->support.discovery, old->support.discovery, sizeof(*new->support.discovery));
   memcpy(new->support.cpubind, old->support.cpubind, sizeof(*new->support.cpubind));
   memcpy(new->support.membind, old->support.membind, sizeof(*new->support.membind));
@@ -230,6 +232,11 @@ hwloc_shmem_topology_adopt(hwloc_topology_t *topologyp,
   *topologyp = new;
   return 0;
 
+ out_with_support:
+  free(new->support.discovery);
+  free(new->support.cpubind);
+  free(new->support.membind);
+  free(new);
  out_with_components:
   hwloc_components_fini();
  out_with_mmap:

src/3rdparty/hwloc/src/topology-noos.c

@@ -1,26 +1,34 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2017 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2012 Université Bordeaux
  * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
 
 static int
-hwloc_look_noos(struct hwloc_backend *backend)
+hwloc_look_noos(struct hwloc_backend *backend, struct hwloc_disc_status *dstatus)
 {
+  /*
+   * This backend uses the underlying OS.
+   * However we don't enforce topology->is_thissystem so that
+   * we may still force use this backend when debugging with !thissystem.
+   */
+
   struct hwloc_topology *topology = backend->topology;
   int nbprocs;
 
+  assert(dstatus->phase == HWLOC_DISC_PHASE_CPU);
+
   if (topology->levels[0][0]->cpuset)
     /* somebody discovered things */
     return -1;
 
-  nbprocs = hwloc_fallback_nbprocessors(topology);
+  nbprocs = hwloc_fallback_nbprocessors(0);
   if (nbprocs >= 1)
     topology->support.discovery->pu = 1;
   else
@@ -33,13 +41,15 @@ hwloc_look_noos(struct hwloc_backend *backend)
 }
 
 static struct hwloc_backend *
-hwloc_noos_component_instantiate(struct hwloc_disc_component *component,
+hwloc_noos_component_instantiate(struct hwloc_topology *topology,
+				 struct hwloc_disc_component *component,
+				 unsigned excluded_phases __hwloc_attribute_unused,
 				 const void *_data1 __hwloc_attribute_unused,
 				 const void *_data2 __hwloc_attribute_unused,
 				 const void *_data3 __hwloc_attribute_unused)
 {
   struct hwloc_backend *backend;
-  backend = hwloc_backend_alloc(component);
+  backend = hwloc_backend_alloc(topology, component);
   if (!backend)
     return NULL;
   backend->discover = hwloc_look_noos;
@@ -47,9 +57,9 @@ hwloc_noos_component_instantiate(struct hwloc_disc_component *component,
 }
 
 static struct hwloc_disc_component hwloc_noos_disc_component = {
-  HWLOC_DISC_COMPONENT_TYPE_CPU,
   "no_os",
-  HWLOC_DISC_COMPONENT_TYPE_GLOBAL,
+  HWLOC_DISC_PHASE_CPU,
+  HWLOC_DISC_PHASE_GLOBAL,
   hwloc_noos_component_instantiate,
   40, /* lower than native OS component, higher than globals */
   1,

src/3rdparty/hwloc/src/topology-synthetic.c

@@ -6,11 +6,11 @@
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
-#include <private/debug.h>
+#include "private/debug.h"
 
 #include <limits.h>
 #include <assert.h>
@@ -122,6 +122,7 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
     unsigned long nbs = 1;
     unsigned j, mul;
     const char *tmp;
+    struct hwloc_synthetic_intlv_loop_s *loops;
 
     tmp = attr;
     while (tmp) {
@@ -132,9 +133,10 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
       tmp++;
     }
 
-   {
     /* nr_loops colon-separated fields, but we may need one more at the end */
-    HWLOC_VLA(struct hwloc_synthetic_intlv_loop_s, loops, nr_loops+1);
+    loops = malloc((nr_loops+1) * sizeof(*loops));
+    if (!loops)
+      goto out_with_array;
 
     if (*attr >= '0' && *attr <= '9') {
       /* interleaving as x*y:z*t:... */
@@ -148,11 +150,13 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
 	if (tmp2 == tmp || *tmp2 != '*') {
 	  if (verbose)
 	    fprintf(stderr, "Failed to read synthetic index interleaving loop '%s' without number before '*'\n", tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	if (!step) {
 	  if (verbose)
 	    fprintf(stderr, "Invalid interleaving loop with step 0 at '%s'\n", tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	tmp2++;
@@ -160,11 +164,13 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
 	if (tmp3 == tmp2 || (*tmp3 && *tmp3 != ':' && *tmp3 != ')' && *tmp3 != ' ')) {
 	  if (verbose)
 	    fprintf(stderr, "Failed to read synthetic index interleaving loop '%s' without number between '*' and ':'\n", tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	if (!nb) {
 	  if (verbose)
 	    fprintf(stderr, "Invalid interleaving loop with number 0 at '%s'\n", tmp2);
+	  free(loops);
 	  goto out_with_array;
 	}
 	loops[cur_loop].step = step;
@@ -192,11 +198,13 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
 	if (err < 0) {
 	  if (verbose)
 	    fprintf(stderr, "Failed to read synthetic index interleaving loop type '%s'\n", tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	if (type == HWLOC_OBJ_MISC || type == HWLOC_OBJ_BRIDGE || type == HWLOC_OBJ_PCI_DEVICE || type == HWLOC_OBJ_OS_DEVICE) {
 	  if (verbose)
 	    fprintf(stderr, "Misc object type disallowed in synthetic index interleaving loop type '%s'\n", tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	for(i=0; ; i++) {
@@ -217,6 +225,7 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
 	  if (verbose)
 	    fprintf(stderr, "Failed to find level for synthetic index interleaving loop type '%s'\n",
 		    tmp);
+	  free(loops);
 	  goto out_with_array;
 	}
 	tmp = strchr(tmp, ':');
@@ -235,6 +244,7 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
 	  if (loops[i].level_depth == mydepth && i != cur_loop) {
 	    if (verbose)
 	      fprintf(stderr, "Invalid duplicate interleaving loop type in synthetic index '%s'\n", attr);
+	    free(loops);
 	    goto out_with_array;
 	  }
 	  if (loops[i].level_depth < mydepth
@@ -264,6 +274,7 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
       } else {
 	if (verbose)
 	  fprintf(stderr, "Invalid index interleaving total width %lu instead of %lu\n", nbs, total);
+	free(loops);
 	goto out_with_array;
       }
     }
@@ -278,6 +289,8 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
       mul *= nb;
     }
 
+    free(loops);
+
     /* check that we have the right values (cannot pass total, cannot give duplicate 0) */
     for(j=0; j<total; j++) {
       if (array[j] >= total) {
@@ -293,7 +306,6 @@ hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
     }
 
     indexes->array = array;
-   }
   }
 
   return;
@@ -527,7 +539,8 @@ hwloc_backend_synthetic_init(struct hwloc_synthetic_backend_data_s *data,
 
     if (*pos < '0' || *pos > '9') {
       if (hwloc_type_sscanf(pos, &type, &attrs, sizeof(attrs)) < 0) {
-	if (!strncmp(pos, "Die", 3) || !strncmp(pos, "Tile", 4) || !strncmp(pos, "Module", 6)) {
+	if (!strncmp(pos, "Tile", 4) || !strncmp(pos, "Module", 6)) {
+	  /* possible future types */
 	  type = HWLOC_OBJ_GROUP;
 	} else {
 	  /* FIXME: allow generic "Cache" string? would require to deal with possibly duplicate cache levels */
@@ -645,6 +658,12 @@ hwloc_backend_synthetic_init(struct hwloc_synthetic_backend_data_s *data,
     errno = EINVAL;
     return -1;
   }
+  if (type_count[HWLOC_OBJ_DIE] > 1) {
+    if (verbose)
+      fprintf(stderr, "Synthetic string cannot have several die levels\n");
+    errno = EINVAL;
+    return -1;
+  }
   if (type_count[HWLOC_OBJ_NUMANODE] > 1) {
     if (verbose)
       fprintf(stderr, "Synthetic string cannot have several NUMA node levels\n");
@@ -829,6 +848,7 @@ hwloc_synthetic_set_attr(struct hwloc_synthetic_attr_s *sattr,
     obj->attr->numanode.page_types[0].count = sattr->memorysize / 4096;
     break;
   case HWLOC_OBJ_PACKAGE:
+  case HWLOC_OBJ_DIE:
     break;
   case HWLOC_OBJ_L1CACHE:
   case HWLOC_OBJ_L2CACHE:
@@ -953,13 +973,19 @@ hwloc__look_synthetic(struct hwloc_topology *topology,
 }
 
 static int
-hwloc_look_synthetic(struct hwloc_backend *backend)
+hwloc_look_synthetic(struct hwloc_backend *backend, struct hwloc_disc_status *dstatus)
 {
+  /*
+   * This backend enforces !topology->is_thissystem by default.
+   */
+
   struct hwloc_topology *topology = backend->topology;
   struct hwloc_synthetic_backend_data_s *data = backend->private_data;
   hwloc_bitmap_t cpuset = hwloc_bitmap_alloc();
   unsigned i;
 
+  assert(dstatus->phase == HWLOC_DISC_PHASE_GLOBAL);
+
   assert(!topology->levels[0][0]->cpuset);
 
   hwloc_alloc_root_sets(topology->levels[0][0]);
@@ -1001,7 +1027,9 @@ hwloc_synthetic_backend_disable(struct hwloc_backend *backend)
 }
 
 static struct hwloc_backend *
-hwloc_synthetic_component_instantiate(struct hwloc_disc_component *component,
+hwloc_synthetic_component_instantiate(struct hwloc_topology *topology,
+				      struct hwloc_disc_component *component,
+				      unsigned excluded_phases __hwloc_attribute_unused,
 				      const void *_data1,
 				      const void *_data2 __hwloc_attribute_unused,
 				      const void *_data3 __hwloc_attribute_unused)
@@ -1021,7 +1049,7 @@ hwloc_synthetic_component_instantiate(struct hwloc_disc_component *component,
     }
   }
 
-  backend = hwloc_backend_alloc(component);
+  backend = hwloc_backend_alloc(topology, component);
   if (!backend)
     goto out;
 
@@ -1051,8 +1079,8 @@ hwloc_synthetic_component_instantiate(struct hwloc_disc_component *component,
 }
 
 static struct hwloc_disc_component hwloc_synthetic_disc_component = {
-  HWLOC_DISC_COMPONENT_TYPE_GLOBAL,
   "synthetic",
+  HWLOC_DISC_PHASE_GLOBAL,
   ~0,
   hwloc_synthetic_component_instantiate,
   30,
@@ -1267,6 +1295,12 @@ hwloc__export_synthetic_obj(struct hwloc_topology * topology, unsigned long flag
     /* if exporting to v1 or without extended-types, use all-v1-compatible Socket name */
     res = hwloc_snprintf(tmp, tmplen, "Socket%s", aritys);
 
+  } else if (obj->type == HWLOC_OBJ_DIE
+	     && (flags & (HWLOC_TOPOLOGY_EXPORT_SYNTHETIC_FLAG_NO_EXTENDED_TYPES
+			  |HWLOC_TOPOLOGY_EXPORT_SYNTHETIC_FLAG_V1))) {
+    /* if exporting to v1 or without extended-types, use all-v1-compatible Group name */
+    res = hwloc_snprintf(tmp, tmplen, "Group%s", aritys);
+
   } else if (obj->type == HWLOC_OBJ_GROUP /* don't export group depth */
       || flags & HWLOC_TOPOLOGY_EXPORT_SYNTHETIC_FLAG_NO_EXTENDED_TYPES) {
     res = hwloc_snprintf(tmp, tmplen, "%s%s", hwloc_obj_type_string(obj->type), aritys);
@@ -1323,16 +1357,26 @@ hwloc__export_synthetic_memory_children(struct hwloc_topology * topology, unsign
   }
 
   while (mchild) {
-    /* v2: export all NUMA children */
+    /* FIXME: really recurse to export memcaches and numanode,
-
+     * but it requires clever parsing of [ memcache [numa] [numa] ] during import,
-    assert(mchild->type == HWLOC_OBJ_NUMANODE); /* only NUMA node memory children for now */
+     * better attaching of things to describe the hierarchy.
+     */
+    hwloc_obj_t numanode = mchild;
+    /* only export the first NUMA node leaf of each memory child
+     * FIXME: This assumes mscache aren't shared between nodes, that's true in current platforms
+     */
+    while (numanode && numanode->type != HWLOC_OBJ_NUMANODE) {
+      assert(numanode->arity == 1);
+      numanode = numanode->memory_first_child;
+    }
+    assert(numanode); /* there's always a numanode at the bottom of the memory tree */
 
     if (needprefix)
       hwloc__export_synthetic_add_char(&ret, &tmp, &tmplen, ' ');
 
     hwloc__export_synthetic_add_char(&ret, &tmp, &tmplen, '[');
 
-    res = hwloc__export_synthetic_obj(topology, flags, mchild, (unsigned)-1, tmp, tmplen);
+    res = hwloc__export_synthetic_obj(topology, flags, numanode, (unsigned)-1, tmp, tmplen);
     if (hwloc__export_synthetic_update_status(&ret, &tmp, &tmplen, res) < 0)
       return -1;
 
@@ -1366,9 +1410,8 @@ hwloc_check_memory_symmetric(struct hwloc_topology * topology)
     assert(node);
 
     first_parent = node->parent;
-    assert(hwloc__obj_type_is_normal(first_parent->type)); /* only depth-1 memory children for now */
 
-    /* check whether all object on parent's level have same number of NUMA children */
+    /* check whether all object on parent's level have same number of NUMA bits */
     for(i=0; i<hwloc_get_nbobjs_by_depth(topology, first_parent->depth); i++) {
       hwloc_obj_t parent, mchild;
 
@@ -1379,10 +1422,9 @@ hwloc_check_memory_symmetric(struct hwloc_topology * topology)
       if (parent->memory_arity != first_parent->memory_arity)
 	goto out_with_bitmap;
 
-      /* clear these NUMA children from remaining_nodes */
+      /* clear children NUMA bits from remaining_nodes */
       mchild = parent->memory_first_child;
       while (mchild) {
-	assert(mchild->type == HWLOC_OBJ_NUMANODE); /* only NUMA node memory children for now */
 	hwloc_bitmap_clr(remaining_nodes, mchild->os_index); /* cannot use parent->nodeset, some normal children may have other NUMA nodes */
 	mchild = mchild->next_sibling;
       }

src/3rdparty/hwloc/src/topology-windows.c

@@ -1,6 +1,6 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2012 Université Bordeaux
  * Copyright © 2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
@@ -9,10 +9,10 @@
 /* To try to get all declarations duplicated below.  */
 #define _WIN32_WINNT 0x0601
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/debug.h>
+#include "private/debug.h"
 
 #include <windows.h>
 
@@ -731,8 +731,14 @@ hwloc_win_get_area_memlocation(hwloc_topology_t topology __hwloc_attribute_unuse
  */
 
 static int
-hwloc_look_windows(struct hwloc_backend *backend)
+hwloc_look_windows(struct hwloc_backend *backend, struct hwloc_disc_status *dstatus)
 {
+  /*
+   * This backend uses the underlying OS.
+   * However we don't enforce topology->is_thissystem so that
+   * we may still force use this backend when debugging with !thissystem.
+   */
+
   struct hwloc_topology *topology = backend->topology;
   hwloc_bitmap_t groups_pu_set = NULL;
   SYSTEM_INFO SystemInfo;
@@ -740,6 +746,8 @@ hwloc_look_windows(struct hwloc_backend *backend)
   int gotnuma = 0;
   int gotnumamemory = 0;
 
+  assert(dstatus->phase == HWLOC_DISC_PHASE_CPU);
+
   if (topology->levels[0][0]->cpuset)
     /* somebody discovered things */
     return -1;
@@ -1136,13 +1144,15 @@ static void hwloc_windows_component_finalize(unsigned long flags __hwloc_attribu
 }
 
 static struct hwloc_backend *
-hwloc_windows_component_instantiate(struct hwloc_disc_component *component,
+hwloc_windows_component_instantiate(struct hwloc_topology *topology,
+				    struct hwloc_disc_component *component,
+				    unsigned excluded_phases __hwloc_attribute_unused,
 				    const void *_data1 __hwloc_attribute_unused,
 				    const void *_data2 __hwloc_attribute_unused,
 				    const void *_data3 __hwloc_attribute_unused)
 {
   struct hwloc_backend *backend;
-  backend = hwloc_backend_alloc(component);
+  backend = hwloc_backend_alloc(topology, component);
   if (!backend)
     return NULL;
   backend->discover = hwloc_look_windows;
@@ -1150,9 +1160,9 @@ hwloc_windows_component_instantiate(struct hwloc_disc_component *component,
 }
 
 static struct hwloc_disc_component hwloc_windows_disc_component = {
-  HWLOC_DISC_COMPONENT_TYPE_CPU,
   "windows",
-  HWLOC_DISC_COMPONENT_TYPE_GLOBAL,
+  HWLOC_DISC_PHASE_CPU,
+  HWLOC_DISC_PHASE_GLOBAL,
   hwloc_windows_component_instantiate,
   50,
   1,
@@ -1168,10 +1178,12 @@ const struct hwloc_component hwloc_windows_component = {
 };
 
 int
-hwloc_fallback_nbprocessors(struct hwloc_topology *topology __hwloc_attribute_unused) {
+hwloc_fallback_nbprocessors(unsigned flags __hwloc_attribute_unused) {
   int n;
   SYSTEM_INFO sysinfo;
 
+  /* TODO handle flags & HWLOC_FALLBACK_NBPROCESSORS_INCLUDE_OFFLINE */
+
   /* by default, ignore groups (return only the number in the current group) */
   GetSystemInfo(&sysinfo);
   n = sysinfo.dwNumberOfProcessors; /* FIXME could be non-contigous, rather return a mask from dwActiveProcessorMask? */

src/3rdparty/hwloc/src/topology-xml-nolibxml.c

@@ -1,18 +1,18 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2011 Université Bordeaux
  * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <hwloc/plugins.h>
+#include "hwloc/plugins.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
-#include <private/xml.h>
+#include "private/xml.h"
-#include <private/debug.h>
+#include "private/debug.h"
 
 #include <string.h>
 #include <assert.h>
@@ -27,9 +27,8 @@
  *******************/
 
 struct hwloc__nolibxml_backend_data_s {
-  size_t buflen; /* size of both buffer and copy buffers, set during backend_init() */
+  size_t buflen; /* size of both buffer, set during backend_init() */
   char *buffer; /* allocated and filled during backend_init() */
-  char *copy; /* allocated during backend_init(), used later during actual parsing */
 };
 
 typedef struct hwloc__nolibxml_import_state_data_s {
@@ -260,14 +259,11 @@ hwloc_nolibxml_look_init(struct hwloc_xml_backend_data_s *bdata,
   struct hwloc__nolibxml_backend_data_s *nbdata = bdata->data;
   unsigned major, minor;
   char *end;
-  char *buffer;
+  char *buffer = nbdata->buffer;
+  char *tagname;
 
   HWLOC_BUILD_ASSERT(sizeof(*nstate) <= sizeof(state->data));
 
-  /* use a copy in the temporary buffer, we may modify during parsing */
-  buffer = nbdata->copy;
-  memcpy(buffer, nbdata->buffer, nbdata->buflen);
-
   /* skip headers */
   while (!strncmp(buffer, "<?xml ", 6) || !strncmp(buffer, "<!DOCTYPE ", 10)) {
     buffer = strchr(buffer, '\n');
@@ -281,14 +277,17 @@ hwloc_nolibxml_look_init(struct hwloc_xml_backend_data_s *bdata,
     bdata->version_major = major;
     bdata->version_minor = minor;
     end = strchr(buffer, '>') + 1;
+    tagname = "topology";
   } else if (!strncmp(buffer, "<topology>", 10)) {
     bdata->version_major = 1;
     bdata->version_minor = 0;
     end = buffer + 10;
+    tagname = "topology";
   } else if (!strncmp(buffer, "<root>", 6)) {
     bdata->version_major = 0;
     bdata->version_minor = 9;
     end = buffer + 6;
+    tagname = "root";
   } else
     goto failed;
 
@@ -301,7 +300,7 @@ hwloc_nolibxml_look_init(struct hwloc_xml_backend_data_s *bdata,
   state->parent = NULL;
   nstate->closed = 0;
   nstate->tagbuffer = end;
-  nstate->tagname = (char *) "topology";
+  nstate->tagname = tagname;
   nstate->attrbuffer = NULL;
   return 0; /* success */
 
@@ -320,10 +319,6 @@ hwloc_nolibxml_free_buffers(struct hwloc_xml_backend_data_s *bdata)
     free(nbdata->buffer);
     nbdata->buffer = NULL;
   }
-  if (nbdata->copy) {
-    free(nbdata->copy);
-    nbdata->copy = NULL;
-  }
 }
 
 static void
@@ -429,19 +424,11 @@ hwloc_nolibxml_backend_init(struct hwloc_xml_backend_data_s *bdata,
       goto out_with_nbdata;
   }
 
-  /* allocate a temporary copy buffer that we may modify during parsing */
-  nbdata->copy = malloc(nbdata->buflen+1);
-  if (!nbdata->copy)
-    goto out_with_buffer;
-  nbdata->copy[nbdata->buflen] = '\0';
-
   bdata->look_init = hwloc_nolibxml_look_init;
   bdata->look_done = hwloc_nolibxml_look_done;
   bdata->backend_exit = hwloc_nolibxml_backend_exit;
   return 0;
 
-out_with_buffer:
-  free(nbdata->buffer);
 out_with_nbdata:
   free(nbdata);
 out:
@@ -666,7 +653,7 @@ hwloc__nolibxml_export_end_object(hwloc__xml_export_state_t state, const char *n
 }
 
 static void
-hwloc__nolibxml_export_add_content(hwloc__xml_export_state_t state, const char *buffer, size_t length)
+hwloc__nolibxml_export_add_content(hwloc__xml_export_state_t state, const char *buffer, size_t length __hwloc_attribute_unused)
 {
   hwloc__nolibxml_export_state_data_t ndata = (void *) state->data;
   int res;
@@ -678,7 +665,7 @@ hwloc__nolibxml_export_add_content(hwloc__xml_export_state_t state, const char *
   }
   ndata->has_content = 1;
 
-  res = hwloc_snprintf(ndata->buffer, ndata->remaining, buffer, length);
+  res = hwloc_snprintf(ndata->buffer, ndata->remaining, "%s", buffer);
   hwloc__nolibxml_export_update_buffer(ndata, res);
 }
 
@@ -799,6 +786,7 @@ hwloc___nolibxml_prepare_export_diff(hwloc_topology_diff_t diff, const char *ref
   state.new_prop = hwloc__nolibxml_export_new_prop;
   state.add_content = hwloc__nolibxml_export_add_content;
   state.end_object = hwloc__nolibxml_export_end_object;
+  state.global = NULL;
 
   ndata->indent = 0;
   ndata->written = 0;

src/3rdparty/hwloc/src/topology-xml.c

@@ -6,12 +6,12 @@
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/xml.h>
+#include "private/xml.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
-#include <private/debug.h>
+#include "private/debug.h"
 
 #include <math.h>
 
@@ -158,7 +158,7 @@ hwloc__xml_import_object_attr(struct hwloc_topology *topology,
 
   else if (!strcmp(name, "cache_size")) {
     unsigned long long lvalue = strtoull(value, NULL, 10);
-    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD)
+    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD || obj->type == HWLOC_OBJ_MEMCACHE)
       obj->attr->cache.size = lvalue;
     else if (hwloc__xml_verbose())
       fprintf(stderr, "%s: ignoring cache_size attribute for non-cache object type\n",
@@ -167,7 +167,7 @@ hwloc__xml_import_object_attr(struct hwloc_topology *topology,
 
   else if (!strcmp(name, "cache_linesize")) {
     unsigned long lvalue = strtoul(value, NULL, 10);
-    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD)
+    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD || obj->type == HWLOC_OBJ_MEMCACHE)
       obj->attr->cache.linesize = lvalue;
     else if (hwloc__xml_verbose())
       fprintf(stderr, "%s: ignoring cache_linesize attribute for non-cache object type\n",
@@ -176,7 +176,7 @@ hwloc__xml_import_object_attr(struct hwloc_topology *topology,
 
   else if (!strcmp(name, "cache_associativity")) {
     int lvalue = atoi(value);
-    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD)
+    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD || obj->type == HWLOC_OBJ_MEMCACHE)
       obj->attr->cache.associativity = lvalue;
     else if (hwloc__xml_verbose())
       fprintf(stderr, "%s: ignoring cache_associativity attribute for non-cache object type\n",
@@ -185,7 +185,7 @@ hwloc__xml_import_object_attr(struct hwloc_topology *topology,
 
   else if (!strcmp(name, "cache_type")) {
     unsigned long lvalue = strtoul(value, NULL, 10);
-    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD) {
+    if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD || obj->type == HWLOC_OBJ_MEMCACHE) {
       if (lvalue == HWLOC_OBJ_CACHE_UNIFIED
 	  || lvalue == HWLOC_OBJ_CACHE_DATA
 	  || lvalue == HWLOC_OBJ_CACHE_INSTRUCTION)
@@ -211,7 +211,7 @@ hwloc__xml_import_object_attr(struct hwloc_topology *topology,
 
   else if (!strcmp(name, "depth")) {
     unsigned long lvalue = strtoul(value, NULL, 10);
-     if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD) {
+     if (hwloc__obj_type_is_cache(obj->type) || obj->type == _HWLOC_OBJ_CACHE_OLD || obj->type == HWLOC_OBJ_MEMCACHE) {
 	obj->attr->cache.depth = lvalue;
      } else if (obj->type == HWLOC_OBJ_GROUP || obj->type == HWLOC_OBJ_BRIDGE) {
        /* will be overwritten by the core */
@@ -805,21 +805,13 @@ hwloc__xml_import_object(hwloc_topology_t topology,
 		      state->global->msgprefix);
 	    goto error_with_object;
 	  }
-	} else if (!strcasecmp(attrvalue, "Die")) {
-	  /* deal with possible future type */
-	  obj->type = HWLOC_OBJ_GROUP;
-	  obj->subtype = strdup("Die");
-	  obj->attr->group.kind = HWLOC_GROUP_KIND_INTEL_DIE;
-	  obj->attr->group.dont_merge = data->dont_merge_die_groups;
 	} else if (!strcasecmp(attrvalue, "Tile")) {
 	  /* deal with possible future type */
 	  obj->type = HWLOC_OBJ_GROUP;
-	  obj->subtype = strdup("Tile");
 	  obj->attr->group.kind = HWLOC_GROUP_KIND_INTEL_TILE;
 	} else if (!strcasecmp(attrvalue, "Module")) {
 	  /* deal with possible future type */
 	  obj->type = HWLOC_OBJ_GROUP;
-	  obj->subtype = strdup("Module");
 	  obj->attr->group.kind = HWLOC_GROUP_KIND_INTEL_MODULE;
 	} else if (!strcasecmp(attrvalue, "MemCache")) {
 	  /* ignore possible future type */
@@ -1053,6 +1045,13 @@ hwloc__xml_import_object(hwloc_topology_t topology,
     /* end of 1.x specific checks */
   }
 
+  /* 2.0 backward compatibility */
+  if (obj->type == HWLOC_OBJ_GROUP) {
+    if (obj->attr->group.kind == HWLOC_GROUP_KIND_INTEL_DIE
+	|| (obj->subtype && !strcmp(obj->subtype, "Die")))
+      obj->type = HWLOC_OBJ_DIE;
+  }
+
   /* check that cache attributes are coherent with the actual type */
   if (hwloc__obj_type_is_cache(obj->type)
       && obj->type != hwloc_cache_type_by_depth_type(obj->attr->cache.depth, obj->attr->cache.type)) {
@@ -1212,19 +1211,24 @@ hwloc__xml_import_object(hwloc_topology_t topology,
 
 static int
 hwloc__xml_v2import_distances(hwloc_topology_t topology,
-			      hwloc__xml_import_state_t state)
+			      hwloc__xml_import_state_t state,
+			      int heterotypes)
 {
-  hwloc_obj_type_t type = HWLOC_OBJ_TYPE_NONE;
+  hwloc_obj_type_t unique_type = HWLOC_OBJ_TYPE_NONE;
+  hwloc_obj_type_t *different_types = NULL;
   unsigned nbobjs = 0;
-  int indexing = 0;
+  int indexing = heterotypes;
   int os_indexing = 0;
-  int gp_indexing = 0;
+  int gp_indexing = heterotypes;
+  char *name = NULL;
   unsigned long kind = 0;
   unsigned nr_indexes, nr_u64values;
   uint64_t *indexes;
   uint64_t *u64values;
   int ret;
 
+#define _TAG_NAME (heterotypes ? "distances2hetero" : "distances2")
+
   /* process attributes */
   while (1) {
     char *attrname, *attrvalue;
@@ -1233,8 +1237,12 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
     if (!strcmp(attrname, "nbobjs"))
       nbobjs = strtoul(attrvalue, NULL, 10);
     else if (!strcmp(attrname, "type")) {
-      if (hwloc_type_sscanf(attrvalue, &type, NULL, 0) < 0)
+      if (hwloc_type_sscanf(attrvalue, &unique_type, NULL, 0) < 0) {
+	if (hwloc__xml_verbose())
+	  fprintf(stderr, "%s: unrecognized %s type %s\n",
+		  state->global->msgprefix, _TAG_NAME, attrvalue);
 	goto out;
+      }
     }
     else if (!strcmp(attrname, "indexing")) {
       indexing = 1;
@@ -1246,27 +1254,32 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
     else if (!strcmp(attrname, "kind")) {
       kind = strtoul(attrvalue, NULL, 10);
     }
+    else if (!strcmp(attrname, "name")) {
+      name = attrvalue;
+    }
     else {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: ignoring unknown distance attribute %s\n",
+	fprintf(stderr, "%s: ignoring unknown %s attribute %s\n",
-		state->global->msgprefix, attrname);
+		state->global->msgprefix, _TAG_NAME, attrname);
     }
   }
 
   /* abort if missing attribute */
-  if (!nbobjs || type == HWLOC_OBJ_TYPE_NONE || !indexing || !kind) {
+  if (!nbobjs || (!heterotypes && unique_type == HWLOC_OBJ_TYPE_NONE) || !indexing || !kind) {
     if (hwloc__xml_verbose())
-      fprintf(stderr, "%s: distance2 missing some attributes\n",
+      fprintf(stderr, "%s: %s missing some attributes\n",
-	      state->global->msgprefix);
+	      state->global->msgprefix, _TAG_NAME);
     goto out;
   }
 
   indexes = malloc(nbobjs*sizeof(*indexes));
   u64values = malloc(nbobjs*nbobjs*sizeof(*u64values));
-  if (!indexes || !u64values) {
+  if (heterotypes)
+    different_types = malloc(nbobjs*sizeof(*different_types));
+  if (!indexes || !u64values || (heterotypes && !different_types)) {
     if (hwloc__xml_verbose())
-      fprintf(stderr, "%s: failed to allocate distances arrays for %u objects\n",
+      fprintf(stderr, "%s: failed to allocate %s arrays for %u objects\n",
-	      state->global->msgprefix, nbobjs);
+	      state->global->msgprefix, _TAG_NAME, nbobjs);
     goto out_with_arrays;
   }
 
@@ -1290,16 +1303,16 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
       is_u64values = 1;
     if (!is_index && !is_u64values) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: distance2 with unrecognized child %s\n",
+	fprintf(stderr, "%s: %s with unrecognized child %s\n",
-		state->global->msgprefix, tag);
+		state->global->msgprefix, _TAG_NAME, tag);
       goto out_with_arrays;
     }
 
     if (state->global->next_attr(&childstate, &attrname, &attrvalue) < 0
 	|| strcmp(attrname, "length")) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: distance2 child must have length attribute\n",
+	fprintf(stderr, "%s: %s child must have length attribute\n",
-		state->global->msgprefix);
+		state->global->msgprefix, _TAG_NAME);
       goto out_with_arrays;
     }
     length = atoi(attrvalue);
@@ -1307,24 +1320,43 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
     ret = state->global->get_content(&childstate, &buffer, length);
     if (ret < 0) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: distance2 child needs content of length %d\n",
+	fprintf(stderr, "%s: %s child needs content of length %d\n",
-		state->global->msgprefix, length);
+		state->global->msgprefix, _TAG_NAME, length);
       goto out_with_arrays;
     }
 
     if (is_index) {
       /* get indexes */
-      char *tmp;
+      char *tmp, *tmp2;
       if (nr_indexes >= nbobjs) {
 	if (hwloc__xml_verbose())
-	  fprintf(stderr, "%s: distance2 with more than %u indexes\n",
+	  fprintf(stderr, "%s: %s with more than %u indexes\n",
-		  state->global->msgprefix, nbobjs);
+		  state->global->msgprefix, _TAG_NAME, nbobjs);
 	goto out_with_arrays;
       }
       tmp = buffer;
       while (1) {
 	char *next;
-	unsigned long long u = strtoull(tmp, &next, 0);
+	unsigned long long u;
+	if (heterotypes) {
+	  hwloc_obj_type_t t = HWLOC_OBJ_TYPE_NONE;
+	  if (hwloc_type_sscanf(tmp, &t, NULL, 0) < 0) {
+	    if (hwloc__xml_verbose())
+	      fprintf(stderr, "%s: %s with unrecognized heterogeneous type %s\n",
+		      state->global->msgprefix, _TAG_NAME, tmp);
+	    goto out_with_arrays;
+	  }
+	  tmp2 = strchr(tmp, ':');
+	  if (!tmp2) {
+	    if (hwloc__xml_verbose())
+	      fprintf(stderr, "%s: %s with missing colon after heterogeneous type %s\n",
+		      state->global->msgprefix, _TAG_NAME, tmp);
+	    goto out_with_arrays;
+	  }
+	  tmp = tmp2+1;
+	  different_types[nr_indexes] = t;
+	}
+	u = strtoull(tmp, &next, 0);
 	if (next == tmp)
 	  break;
 	indexes[nr_indexes++] = u;
@@ -1340,8 +1372,8 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
       char *tmp;
       if (nr_u64values >= nbobjs*nbobjs) {
 	if (hwloc__xml_verbose())
-	  fprintf(stderr, "%s: distance2 with more than %u u64values\n",
+	  fprintf(stderr, "%s: %s with more than %u u64values\n",
-		  state->global->msgprefix, nbobjs*nbobjs);
+		  state->global->msgprefix, _TAG_NAME, nbobjs*nbobjs);
 	goto out_with_arrays;
       }
       tmp = buffer;
@@ -1364,8 +1396,8 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
     ret = state->global->close_tag(&childstate);
     if (ret < 0) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: distance2 with more than %u indexes\n",
+	fprintf(stderr, "%s: %s with more than %u indexes\n",
-		state->global->msgprefix, nbobjs);
+		state->global->msgprefix, _TAG_NAME, nbobjs);
       goto out_with_arrays;
     }
 
@@ -1374,56 +1406,60 @@ hwloc__xml_v2import_distances(hwloc_topology_t topology,
 
   if (nr_indexes != nbobjs) {
     if (hwloc__xml_verbose())
-      fprintf(stderr, "%s: distance2 with less than %u indexes\n",
+      fprintf(stderr, "%s: %s with less than %u indexes\n",
-	      state->global->msgprefix, nbobjs);
+	      state->global->msgprefix, _TAG_NAME, nbobjs);
     goto out_with_arrays;
   }
   if (nr_u64values != nbobjs*nbobjs) {
     if (hwloc__xml_verbose())
-      fprintf(stderr, "%s: distance2 with less than %u u64values\n",
+      fprintf(stderr, "%s: %s with less than %u u64values\n",
-	      state->global->msgprefix, nbobjs*nbobjs);
+	      state->global->msgprefix, _TAG_NAME, nbobjs*nbobjs);
     goto out_with_arrays;
   }
 
   if (nbobjs < 2) {
     /* distances with a single object are useless, even if the XML isn't invalid */
     if (hwloc__xml_verbose())
-      fprintf(stderr, "%s: ignoring distances2 with only %u objects\n",
+      fprintf(stderr, "%s: ignoring %s with only %u objects\n",
-	      state->global->msgprefix, nbobjs);
+	      state->global->msgprefix, _TAG_NAME, nbobjs);
     goto out_ignore;
   }
-  if (type == HWLOC_OBJ_PU || type == HWLOC_OBJ_NUMANODE) {
+  if (unique_type == HWLOC_OBJ_PU || unique_type == HWLOC_OBJ_NUMANODE) {
     if (!os_indexing) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: ignoring PU or NUMA distances2 without os_indexing\n",
+	fprintf(stderr, "%s: ignoring PU or NUMA %s without os_indexing\n",
-		state->global->msgprefix);
+		state->global->msgprefix, _TAG_NAME);
       goto out_ignore;
     }
   } else {
     if (!gp_indexing) {
       if (hwloc__xml_verbose())
-	fprintf(stderr, "%s: ignoring !PU or !NUMA distances2 without gp_indexing\n",
+	fprintf(stderr, "%s: ignoring !PU or !NUMA %s without gp_indexing\n",
-		state->global->msgprefix);
+		state->global->msgprefix, _TAG_NAME);
       goto out_ignore;
     }
   }
 
-  hwloc_internal_distances_add_by_index(topology, type, nbobjs, indexes, u64values, kind, 0);
+  hwloc_internal_distances_add_by_index(topology, name, unique_type, different_types, nbobjs, indexes, u64values, kind, 0);
 
   /* prevent freeing below */
   indexes = NULL;
   u64values = NULL;
+  different_types = NULL;
 
  out_ignore:
+  free(different_types);
   free(indexes);
   free(u64values);
   return state->global->close_tag(state);
 
  out_with_arrays:
+  free(different_types);
   free(indexes);
   free(u64values);
  out:
   return -1;
+#undef _TAG_NAME
 }
 
 static int
@@ -1625,8 +1661,12 @@ hwloc_convert_from_v1dist_floats(hwloc_topology_t topology, unsigned nbobjs, flo
 
 /* this canNOT be the first XML call */
 static int
-hwloc_look_xml(struct hwloc_backend *backend)
+hwloc_look_xml(struct hwloc_backend *backend, struct hwloc_disc_status *dstatus)
 {
+  /*
+   * This backend enforces !topology->is_thissystem by default.
+   */
+
   struct hwloc_topology *topology = backend->topology;
   struct hwloc_xml_backend_data_s *data = backend->private_data;
   struct hwloc__xml_import_state_s state, childstate;
@@ -1634,9 +1674,10 @@ hwloc_look_xml(struct hwloc_backend *backend)
   char *tag;
   int gotignored = 0;
   hwloc_localeswitch_declare;
-  char *env;
   int ret;
 
+  assert(dstatus->phase == HWLOC_DISC_PHASE_GLOBAL);
+
   state.global = data;
 
   assert(!root->cpuset);
@@ -1647,9 +1688,6 @@ hwloc_look_xml(struct hwloc_backend *backend)
   data->first_numanode = data->last_numanode = NULL;
   data->first_v1dist = data->last_v1dist = NULL;
 
-  env = getenv("HWLOC_DONT_MERGE_DIE_GROUPS");
-  data->dont_merge_die_groups = env && atoi(env);
-
   ret = data->look_init(data, &state);
   if (ret < 0)
     goto failed;
@@ -1684,15 +1722,20 @@ hwloc_look_xml(struct hwloc_backend *backend)
 	goto failed;
       if (!ret)
 	break;
-      if (strcmp(tag, "distances2")) {
+      if (!strcmp(tag, "distances2")) {
+	ret = hwloc__xml_v2import_distances(topology, &childstate, 0);
+	if (ret < 0)
+	  goto failed;
+      } else if (!strcmp(tag, "distances2hetero")) {
+	ret = hwloc__xml_v2import_distances(topology, &childstate, 1);
+	if (ret < 0)
+	  goto failed;
+      } else {
 	if (hwloc__xml_verbose())
 	  fprintf(stderr, "%s: ignoring unknown tag `%s' after root object, expected `distances2'\n",
 		  data->msgprefix, tag);
 	goto done;
       }
-      ret = hwloc__xml_v2import_distances(topology, &childstate);
-      if (ret < 0)
-	goto failed;
       state.global->close_child(&childstate);
     }
   }
@@ -1742,8 +1785,8 @@ done:
 	      i<nbobjs;
 	      i++, node = node->next_cousin)
 	    objs[i] = node;
-hwloc_convert_from_v1dist_floats(topology, nbobjs, v1dist->floats, values);
+	  hwloc_convert_from_v1dist_floats(topology, nbobjs, v1dist->floats, values);
-	  hwloc_internal_distances_add(topology, nbobjs, objs, values, v1dist->kind, 0);
+	  hwloc_internal_distances_add(topology, NULL, nbobjs, objs, values, v1dist->kind, 0);
 	} else {
 	  free(objs);
 	  free(values);
@@ -1791,9 +1834,11 @@ hwloc_convert_from_v1dist_floats(topology, nbobjs, v1dist->floats, values);
   /* we could add "BackendSource=XML" to notify that XML was used between the actual backend and here */
 
   topology->support.discovery->pu = 1;
+  topology->support.discovery->disallowed_pu = 1;
   if (data->nbnumanodes) {
     topology->support.discovery->numa = 1;
     topology->support.discovery->numa_memory = 1; // FIXME
+    topology->support.discovery->disallowed_numa = 1;
   }
 
   if (data->look_done)
@@ -1936,6 +1981,9 @@ hwloc__xml_export_safestrdup(const char *old)
   char *new = malloc(strlen(old)+1);
   char *dst = new;
   const char *src = old;
+  if (!new)
+    return NULL;
+
   while (*src) {
     if (HWLOC_XML_CHAR_VALID(*src))
       *(dst++) = *src;
@@ -1955,6 +2003,8 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
 
   if (v1export && obj->type == HWLOC_OBJ_PACKAGE)
     state->new_prop(state, "type", "Socket");
+  else if (v1export && obj->type == HWLOC_OBJ_DIE)
+    state->new_prop(state, "type", "Group");
   else if (v1export && hwloc__obj_type_is_cache(obj->type))
     state->new_prop(state, "type", "Cache");
   else
@@ -1966,8 +2016,23 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
   }
 
   if (obj->cpuset) {
-    if (v1export && obj->type == HWLOC_OBJ_NUMANODE && obj->sibling_rank > 0) {
+    int empty_cpusets = 0;
-      /* v1 non-first NUMA nodes have empty cpusets */
+
+    if (v1export && obj->type == HWLOC_OBJ_NUMANODE) {
+      /* walk up this memory hierarchy to find-out if we are the first numa node.
+       * v1 non-first NUMA nodes have empty cpusets.
+       */
+      hwloc_obj_t parent = obj;
+      while (!hwloc_obj_type_is_normal(parent->type)) {
+	if (parent->sibling_rank > 0) {
+	  empty_cpusets = 1;
+	  break;
+	}
+	parent = parent->parent;
+      }
+    }
+
+    if (empty_cpusets) {
       state->new_prop(state, "cpuset", "0x0");
       state->new_prop(state, "online_cpuset", "0x0");
       state->new_prop(state, "complete_cpuset", "0x0");
@@ -2024,13 +2089,17 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
 
   if (obj->name) {
     char *name = hwloc__xml_export_safestrdup(obj->name);
-    state->new_prop(state, "name", name);
+    if (name) {
-    free(name);
+      state->new_prop(state, "name", name);
+      free(name);
+    }
   }
   if (!v1export && obj->subtype) {
     char *subtype = hwloc__xml_export_safestrdup(obj->subtype);
-    state->new_prop(state, "subtype", subtype);
+    if (subtype) {
-    free(subtype);
+      state->new_prop(state, "subtype", subtype);
+      free(subtype);
+    }
   }
 
   switch (obj->type) {
@@ -2057,6 +2126,7 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
   case HWLOC_OBJ_L1ICACHE:
   case HWLOC_OBJ_L2ICACHE:
   case HWLOC_OBJ_L3ICACHE:
+  case HWLOC_OBJ_MEMCACHE:
     sprintf(tmp, "%llu", (unsigned long long) obj->attr->cache.size);
     state->new_prop(state, "cache_size", tmp);
     sprintf(tmp, "%u", obj->attr->cache.depth);
@@ -2125,23 +2195,34 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
   for(i=0; i<obj->infos_count; i++) {
     char *name = hwloc__xml_export_safestrdup(obj->infos[i].name);
     char *value = hwloc__xml_export_safestrdup(obj->infos[i].value);
-    struct hwloc__xml_export_state_s childstate;
+    if (name && value) {
-    state->new_child(state, &childstate, "info");
+      struct hwloc__xml_export_state_s childstate;
-    childstate.new_prop(&childstate, "name", name);
+      state->new_child(state, &childstate, "info");
-    childstate.new_prop(&childstate, "value", value);
+      childstate.new_prop(&childstate, "name", name);
-    childstate.end_object(&childstate, "info");
+      childstate.new_prop(&childstate, "value", value);
+      childstate.end_object(&childstate, "info");
+    }
     free(name);
     free(value);
   }
   if (v1export && obj->subtype) {
     char *subtype = hwloc__xml_export_safestrdup(obj->subtype);
+    if (subtype) {
+      struct hwloc__xml_export_state_s childstate;
+      int is_coproctype = (obj->type == HWLOC_OBJ_OS_DEVICE && obj->attr->osdev.type == HWLOC_OBJ_OSDEV_COPROC);
+      state->new_child(state, &childstate, "info");
+      childstate.new_prop(&childstate, "name", is_coproctype ? "CoProcType" : "Type");
+      childstate.new_prop(&childstate, "value", subtype);
+      childstate.end_object(&childstate, "info");
+      free(subtype);
+    }
+  }
+  if (v1export && obj->type == HWLOC_OBJ_DIE) {
     struct hwloc__xml_export_state_s childstate;
-    int is_coproctype = (obj->type == HWLOC_OBJ_OS_DEVICE && obj->attr->osdev.type == HWLOC_OBJ_OSDEV_COPROC);
     state->new_child(state, &childstate, "info");
-    childstate.new_prop(&childstate, "name", is_coproctype ? "CoProcType" : "Type");
+    childstate.new_prop(&childstate, "name", "Type");
-    childstate.new_prop(&childstate, "value", subtype);
+    childstate.new_prop(&childstate, "value", "Die");
     childstate.end_object(&childstate, "info");
-    free(subtype);
   }
 
   if (v1export && !obj->parent) {
@@ -2152,19 +2233,27 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
     for(dist = topology->first_dist; dist; dist = dist->next) {
       struct hwloc__xml_export_state_s childstate;
       unsigned nbobjs = dist->nbobjs;
+      unsigned *logical_to_v2array;
       int depth;
 
-      if (nbobjs != (unsigned) hwloc_get_nbobjs_by_type(topology, dist->type))
+      if (nbobjs != (unsigned) hwloc_get_nbobjs_by_type(topology, dist->unique_type))
 	continue;
       if (!(dist->kind & HWLOC_DISTANCES_KIND_MEANS_LATENCY))
 	continue;
-     {
+      if (dist->kind & HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES)
-      HWLOC_VLA(unsigned, logical_to_v2array, nbobjs);
+	continue;
+
+      logical_to_v2array = malloc(nbobjs * sizeof(*logical_to_v2array));
+      if (!logical_to_v2array) {
+	fprintf(stderr, "xml/export/v1: failed to allocated logical_to_v2array\n");
+	continue;
+      }
+
       for(i=0; i<nbobjs; i++)
 	logical_to_v2array[dist->objs[i]->logical_index] = i;
 
       /* compute the relative depth */
-      if (dist->type == HWLOC_OBJ_NUMANODE) {
+      if (dist->unique_type == HWLOC_OBJ_NUMANODE) {
 	/* for NUMA nodes, use the highest normal-parent depth + 1 */
 	depth = -1;
 	for(i=0; i<nbobjs; i++) {
@@ -2188,7 +2277,7 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
 	  }
 	}
       done:
-	depth = hwloc_get_type_depth(topology, dist->type) + parent_with_memory;
+	depth = hwloc_get_type_depth(topology, dist->unique_type) + parent_with_memory;
       }
 
       state->new_child(state, &childstate, "distances");
@@ -2210,7 +2299,7 @@ hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, hwloc_topolo
 	}
       }
       childstate.end_object(&childstate, "distances");
-     }
+      free(logical_to_v2array);
     }
   }
 
@@ -2243,13 +2332,90 @@ hwloc__xml_v2export_object (hwloc__xml_export_state_t parentstate, hwloc_topolog
 static void
 hwloc__xml_v1export_object (hwloc__xml_export_state_t parentstate, hwloc_topology_t topology, hwloc_obj_t obj, unsigned long flags);
 
+static hwloc_obj_t
+hwloc__xml_v1export_object_next_numanode(hwloc_obj_t obj, hwloc_obj_t cur)
+{
+  hwloc_obj_t parent;
+
+  if (!cur) {
+    /* first numa node is on the very bottom left */
+    cur = obj->memory_first_child;
+    goto find_first;
+  }
+
+  /* walk-up until there's a next sibling */
+  parent = cur;
+  while (1) {
+    if (parent->next_sibling) {
+      /* found a next sibling, we'll walk down-left from there */
+      cur = parent->next_sibling;
+      break;
+    }
+    parent = parent->parent;
+    if (parent == obj)
+      return NULL;
+  }
+
+ find_first:
+  while (cur->type != HWLOC_OBJ_NUMANODE)
+    cur = cur->memory_first_child;
+  assert(cur);
+  return cur;
+}
+
+static unsigned
+hwloc__xml_v1export_object_list_numanodes(hwloc_obj_t obj, hwloc_obj_t *first_p, hwloc_obj_t **nodes_p)
+{
+  hwloc_obj_t *nodes, cur;
+  int nr;
+
+  if (!obj->memory_first_child) {
+    *first_p = NULL;
+    *nodes_p = NULL;
+    return 0;
+  }
+  /* we're sure there's at least one numa node */
+
+  nr = hwloc_bitmap_weight(obj->nodeset);
+  assert(nr > 0);
+  /* these are local nodes, but some of them may be attached above instead of here */
+
+  nodes = calloc(nr, sizeof(*nodes));
+  if (!nodes) {
+    /* only return the first node */
+    cur = hwloc__xml_v1export_object_next_numanode(obj, NULL);
+    assert(cur);
+    *first_p = cur;
+    *nodes_p = NULL;
+    return 1;
+  }
+
+  nr = 0;
+  cur = NULL;
+  while (1) {
+    cur = hwloc__xml_v1export_object_next_numanode(obj, cur);
+    if (!cur)
+      break;
+    nodes[nr++] = cur;
+  }
+
+  *first_p = nodes[0];
+  *nodes_p = nodes;
+  return nr;
+}
+
 static void
 hwloc__xml_v1export_object_with_memory(hwloc__xml_export_state_t parentstate, hwloc_topology_t topology, hwloc_obj_t obj, unsigned long flags)
 {
   struct hwloc__xml_export_state_s gstate, mstate, ostate, *state = parentstate;
   hwloc_obj_t child;
+  unsigned nr_numanodes;
+  hwloc_obj_t *numanodes, first_numanode;
+  unsigned i;
 
-  if (obj->parent->arity > 1 && obj->memory_arity > 1 && parentstate->global->v1_memory_group) {
+  nr_numanodes = hwloc__xml_v1export_object_list_numanodes(obj, &first_numanode, &numanodes);
+
+  if (obj->parent->arity > 1 && nr_numanodes > 1 && parentstate->global->v1_memory_group) {
     /* child has sibling, we must add a Group around those memory children */
     hwloc_obj_t group = parentstate->global->v1_memory_group;
     parentstate->new_child(parentstate, &gstate, "object");
@@ -2266,10 +2432,8 @@ hwloc__xml_v1export_object_with_memory(hwloc__xml_export_state_t parentstate, hw
   }
 
   /* export first memory child */
-  child = obj->memory_first_child;
-  assert(child->type == HWLOC_OBJ_NUMANODE);
   state->new_child(state, &mstate, "object");
-  hwloc__xml_export_object_contents (&mstate, topology, child, flags);
+  hwloc__xml_export_object_contents (&mstate, topology, first_numanode, flags);
 
   /* then the actual object */
   mstate.new_child(&mstate, &ostate, "object");
@@ -2288,9 +2452,10 @@ hwloc__xml_v1export_object_with_memory(hwloc__xml_export_state_t parentstate, hw
   mstate.end_object(&mstate, "object");
 
   /* now other memory children */
-  for_each_memory_child(child, obj)
+  for(i=1; i<nr_numanodes; i++)
-    if (child->sibling_rank > 0)
+    hwloc__xml_v1export_object (state, topology, numanodes[i], flags);
-      hwloc__xml_v1export_object (state, topology, child, flags);
+
+  free(numanodes);
 
   if (state == &gstate) {
     /* close group if any */
@@ -2346,30 +2511,74 @@ hwloc__xml_v1export_object (hwloc__xml_export_state_t parentstate, hwloc_topolog
   } \
 } while (0)
 
+#define EXPORT_TYPE_GPINDEX_ARRAY(state, nr, objs, tagname, maxperline) do { \
+  unsigned _i = 0; \
+  while (_i<(nr)) { \
+    char _tmp[255]; /* enough for (snprintf(type+index)+space) x maxperline */ \
+    char _tmp2[16]; \
+    size_t _len = 0; \
+    unsigned _j; \
+    struct hwloc__xml_export_state_s _childstate; \
+    (state)->new_child(state, &_childstate, tagname); \
+    for(_j=0; \
+	_i+_j<(nr) && _j<maxperline; \
+	_j++) \
+      _len += sprintf(_tmp+_len, "%s:%llu ", hwloc_obj_type_string((objs)[_i+_j]->type), (unsigned long long) (objs)[_i+_j]->gp_index); \
+    _i += _j; \
+    sprintf(_tmp2, "%lu", (unsigned long) _len); \
+    _childstate.new_prop(&_childstate, "length", _tmp2); \
+    _childstate.add_content(&_childstate, _tmp, _len); \
+    _childstate.end_object(&_childstate, tagname); \
+  } \
+} while (0)
+
+static void
+hwloc___xml_v2export_distances(hwloc__xml_export_state_t parentstate, struct hwloc_internal_distances_s *dist)
+{
+  char tmp[255];
+  unsigned nbobjs = dist->nbobjs;
+  struct hwloc__xml_export_state_s state;
+
+  if (dist->different_types) {
+    parentstate->new_child(parentstate, &state, "distances2hetero");
+  } else {
+    parentstate->new_child(parentstate, &state, "distances2");
+    state.new_prop(&state, "type", hwloc_obj_type_string(dist->unique_type));
+  }
+
+  sprintf(tmp, "%u", nbobjs);
+  state.new_prop(&state, "nbobjs", tmp);
+  sprintf(tmp, "%lu", dist->kind);
+  state.new_prop(&state, "kind", tmp);
+  if (dist->name)
+    state.new_prop(&state, "name", dist->name);
+
+  if (!dist->different_types) {
+    state.new_prop(&state, "indexing",
+		   HWLOC_DIST_TYPE_USE_OS_INDEX(dist->unique_type) ? "os" : "gp");
+  }
+
+  /* TODO don't hardwire 10 below. either snprintf the max to guess it, or just append until the end of the buffer */
+  if (dist->different_types) {
+    EXPORT_TYPE_GPINDEX_ARRAY(&state, nbobjs, dist->objs, "indexes", 10);
+  } else {
+    EXPORT_ARRAY(&state, unsigned long long, nbobjs, dist->indexes, "indexes", "%llu", 10);
+  }
+  EXPORT_ARRAY(&state, unsigned long long, nbobjs*nbobjs, dist->values, "u64values", "%llu", 10);
+  state.end_object(&state, dist->different_types ? "distances2hetero" : "distances2");
+}
+
 static void
 hwloc__xml_v2export_distances(hwloc__xml_export_state_t parentstate, hwloc_topology_t topology)
 {
   struct hwloc_internal_distances_s *dist;
-  for(dist = topology->first_dist; dist; dist = dist->next) {
+  for(dist = topology->first_dist; dist; dist = dist->next)
-    char tmp[255];
+    if (!dist->different_types)
-    unsigned nbobjs = dist->nbobjs;
+      hwloc___xml_v2export_distances(parentstate, dist);
-    struct hwloc__xml_export_state_s state;
+  /* export homogeneous distances first in case the importer doesn't support heterogeneous and stops there */
-
+  for(dist = topology->first_dist; dist; dist = dist->next)
-    parentstate->new_child(parentstate, &state, "distances2");
+    if (dist->different_types)
-
+      hwloc___xml_v2export_distances(parentstate, dist);
-    state.new_prop(&state, "type", hwloc_obj_type_string(dist->type));
-    sprintf(tmp, "%u", nbobjs);
-    state.new_prop(&state, "nbobjs", tmp);
-    sprintf(tmp, "%lu", dist->kind);
-    state.new_prop(&state, "kind", tmp);
-
-    state.new_prop(&state, "indexing",
-		   (dist->type == HWLOC_OBJ_NUMANODE || dist->type == HWLOC_OBJ_PU) ? "os" : "gp");
-    /* TODO don't hardwire 10 below. either snprintf the max to guess it, or just append until the end of the buffer */
-    EXPORT_ARRAY(&state, unsigned long long, nbobjs, dist->indexes, "indexes", "%llu", 10);
-    EXPORT_ARRAY(&state, unsigned long long, nbobjs*nbobjs, dist->values, "u64values", "%llu", 10);
-    state.end_object(&state, "distances2");
-  }
 }
 
 void
@@ -2378,18 +2587,22 @@ hwloc__xml_export_topology(hwloc__xml_export_state_t state, hwloc_topology_t top
   hwloc_obj_t root = hwloc_get_root_obj(topology);
 
   if (flags & HWLOC_TOPOLOGY_EXPORT_XML_FLAG_V1) {
-    if (root->memory_first_child) {
+    hwloc_obj_t *numanodes, first_numanode;
+    unsigned nr_numanodes;
+
+    nr_numanodes = hwloc__xml_v1export_object_list_numanodes(root, &first_numanode, &numanodes);
+
+    if (nr_numanodes) {
       /* we don't use hwloc__xml_v1export_object_with_memory() because we want/can keep root above the numa node */
       struct hwloc__xml_export_state_s rstate, mstate;
       hwloc_obj_t child;
+      unsigned i;
       /* export the root */
       state->new_child(state, &rstate, "object");
       hwloc__xml_export_object_contents (&rstate, topology, root, flags);
       /* export first memory child */
-      child = root->memory_first_child;
-      assert(child->type == HWLOC_OBJ_NUMANODE);
       rstate.new_child(&rstate, &mstate, "object");
-      hwloc__xml_export_object_contents (&mstate, topology, child, flags);
+      hwloc__xml_export_object_contents (&mstate, topology, first_numanode, flags);
       /* then its normal/io/misc children */
       for_each_child(child, root)
 	hwloc__xml_v1export_object (&mstate, topology, child, flags);
@@ -2400,15 +2613,16 @@ hwloc__xml_export_topology(hwloc__xml_export_state_t state, hwloc_topology_t top
       /* close first memory child */
       mstate.end_object(&mstate, "object");
       /* now other memory children */
-      for_each_memory_child(child, root)
+      for(i=1; i<nr_numanodes; i++)
-	if (child->sibling_rank > 0)
+	hwloc__xml_v1export_object (&rstate, topology, numanodes[i], flags);
-	  hwloc__xml_v1export_object (&rstate, topology, child, flags);
       /* close the root */
       rstate.end_object(&rstate, "object");
     } else {
       hwloc__xml_v1export_object(state, topology, root, flags);
     }
 
+    free(numanodes);
+
   } else {
     hwloc__xml_v2export_object (state, topology, root, flags);
     hwloc__xml_v2export_distances (state, topology);
@@ -2788,7 +3002,9 @@ hwloc_xml_backend_disable(struct hwloc_backend *backend)
 }
 
 static struct hwloc_backend *
-hwloc_xml_component_instantiate(struct hwloc_disc_component *component,
+hwloc_xml_component_instantiate(struct hwloc_topology *topology,
+				struct hwloc_disc_component *component,
+				unsigned excluded_phases __hwloc_attribute_unused,
 				const void *_data1,
 				const void *_data2,
 				const void *_data3)
@@ -2816,7 +3032,7 @@ hwloc_xml_component_instantiate(struct hwloc_disc_component *component,
     }
   }
 
-  backend = hwloc_backend_alloc(component);
+  backend = hwloc_backend_alloc(topology, component);
   if (!backend)
     goto out;
 
@@ -2868,8 +3084,8 @@ retry:
 }
 
 static struct hwloc_disc_component hwloc_xml_disc_component = {
-  HWLOC_DISC_COMPONENT_TYPE_GLOBAL,
   "xml",
+  HWLOC_DISC_PHASE_GLOBAL,
   ~0,
   hwloc_xml_component_instantiate,
   30,

src/3rdparty/hwloc/src/traversal.c

@@ -1,16 +1,17 @@
 /*
  * Copyright © 2009 CNRS
- * Copyright © 2009-2018 Inria.  All rights reserved.
+ * Copyright © 2009-2019 Inria.  All rights reserved.
  * Copyright © 2009-2010 Université Bordeaux
  * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
  * See COPYING in top-level directory.
  */
 
-#include <private/autogen/config.h>
+#include "private/autogen/config.h"
-#include <hwloc.h>
+#include "hwloc.h"
-#include <private/private.h>
+#include "private/private.h"
-#include <private/misc.h>
+#include "private/misc.h"
-#include <private/debug.h>
+#include "private/debug.h"
+
 #ifdef HAVE_STRINGS_H
 #include <strings.h>
 #endif /* HAVE_STRINGS_H */
@@ -40,6 +41,8 @@ hwloc_get_depth_type (hwloc_topology_t topology, int depth)
       return HWLOC_OBJ_OS_DEVICE;
     case HWLOC_TYPE_DEPTH_MISC:
       return HWLOC_OBJ_MISC;
+    case HWLOC_TYPE_DEPTH_MEMCACHE:
+      return HWLOC_OBJ_MEMCACHE;
     default:
       return HWLOC_OBJ_TYPE_NONE;
     }
@@ -237,8 +240,10 @@ hwloc_obj_type_string (hwloc_obj_type_t obj)
     case HWLOC_OBJ_MACHINE: return "Machine";
     case HWLOC_OBJ_MISC: return "Misc";
     case HWLOC_OBJ_GROUP: return "Group";
+    case HWLOC_OBJ_MEMCACHE: return "MemCache";
     case HWLOC_OBJ_NUMANODE: return "NUMANode";
     case HWLOC_OBJ_PACKAGE: return "Package";
+    case HWLOC_OBJ_DIE: return "Die";
     case HWLOC_OBJ_L1CACHE: return "L1Cache";
     case HWLOC_OBJ_L2CACHE: return "L2Cache";
     case HWLOC_OBJ_L3CACHE: return "L3Cache";
@@ -256,6 +261,41 @@ hwloc_obj_type_string (hwloc_obj_type_t obj)
     }
 }
 
+/* Check if string matches the given type at least on minmatch chars.
+ * On success, return the address of where matching stop, either pointing to \0 or to a suffix (digits, colon, etc)
+ * On error, return NULL;
+ */
+static __hwloc_inline const char *
+hwloc__type_match(const char *string,
+		  const char *type, /* type must be lowercase */
+		  size_t minmatch)
+{
+  const char *s, *t;
+  unsigned i;
+  for(i=0, s=string, t=type; ; i++, s++, t++) {
+    if (!*s) {
+      /* string ends before type */
+      if (i<minmatch)
+	return NULL;
+      else
+	return s;
+    }
+    if (*s != *t && *s != *t + 'A' - 'a') {
+      /* string is different */
+      if ((*s >= 'a' && *s <= 'z') || (*s >= 'A' && *s <= 'Z') || *s == '-')
+	/* valid character that doesn't match */
+	return NULL;
+      /* invalid character, we reached the end of the type namein string, stop matching here */
+      if (i<minmatch)
+	return NULL;
+      else
+	return s;
+    }
+  }
+
+  return NULL;
+}
+
 int
 hwloc_type_sscanf(const char *string, hwloc_obj_type_t *typep,
 		  union hwloc_obj_attr_u *attrp, size_t attrsize)
@@ -267,86 +307,102 @@ hwloc_type_sscanf(const char *string, hwloc_obj_type_t *typep,
   hwloc_obj_osdev_type_t ostype = (hwloc_obj_osdev_type_t) -1;
   char *end;
 
-  /* never match the ending \0 since we want to match things like core:2 too.
+  /* Never match the ending \0 since we want to match things like core:2 too.
-   * just use hwloc_strncasecmp() everywhere.
+   * We'll only compare the beginning substring only made of letters and dash.
    */
 
   /* types without a custom depth */
 
   /* osdev subtype first to avoid conflicts coproc/core etc */
-  if (!hwloc_strncasecmp(string, "os", 2)) {
+  if (hwloc__type_match(string, "osdev", 2)) {
     type = HWLOC_OBJ_OS_DEVICE;
-  } else if (!hwloc_strncasecmp(string, "bloc", 4)) {
+  } else if (hwloc__type_match(string, "block", 4)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_BLOCK;
-  } else if (!hwloc_strncasecmp(string, "net", 3)) {
+  } else if (hwloc__type_match(string, "network", 3)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_NETWORK;
-  } else if (!hwloc_strncasecmp(string, "openfab", 7)) {
+  } else if (hwloc__type_match(string, "openfabrics", 7)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_OPENFABRICS;
-  } else if (!hwloc_strncasecmp(string, "dma", 3)) {
+  } else if (hwloc__type_match(string, "dma", 3)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_DMA;
-  } else if (!hwloc_strncasecmp(string, "gpu", 3)) {
+  } else if (hwloc__type_match(string, "gpu", 3)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_GPU;
-  } else if (!hwloc_strncasecmp(string, "copro", 5)
+  } else if (hwloc__type_match(string, "coproc", 5)
-	     || !hwloc_strncasecmp(string, "co-pro", 6)) {
+	     || hwloc__type_match(string, "co-processor", 6)) {
     type = HWLOC_OBJ_OS_DEVICE;
     ostype = HWLOC_OBJ_OSDEV_COPROC;
 
-  } else if (!hwloc_strncasecmp(string, "machine", 2)) {
+  } else if (hwloc__type_match(string, "machine", 2)) {
     type = HWLOC_OBJ_MACHINE;
-  } else if (!hwloc_strncasecmp(string, "node", 2)
+  } else if (hwloc__type_match(string, "numanode", 2)
-	     || !hwloc_strncasecmp(string, "numa", 2)) { /* matches node and numanode */
+	     || hwloc__type_match(string, "node", 2)) { /* for convenience */
     type = HWLOC_OBJ_NUMANODE;
-  } else if (!hwloc_strncasecmp(string, "package", 2)
+  } else if (hwloc__type_match(string, "memcache", 5)
-	     || !hwloc_strncasecmp(string, "socket", 2)) { /* backward compat with v1.10 */
+	     || hwloc__type_match(string, "memory-side cache", 8)) {
+    type = HWLOC_OBJ_MEMCACHE;
+  } else if (hwloc__type_match(string, "package", 2)
+	     || hwloc__type_match(string, "socket", 2)) { /* backward compat with v1.10 */
     type = HWLOC_OBJ_PACKAGE;
-  } else if (!hwloc_strncasecmp(string, "core", 2)) {
+  } else if (hwloc__type_match(string, "die", 2)) {
+    type = HWLOC_OBJ_DIE;
+  } else if (hwloc__type_match(string, "core", 2)) {
     type = HWLOC_OBJ_CORE;
-  } else if (!hwloc_strncasecmp(string, "pu", 2)) {
+  } else if (hwloc__type_match(string, "pu", 2)) {
     type = HWLOC_OBJ_PU;
-  } else if (!hwloc_strncasecmp(string, "misc", 4)) {
+  } else if (hwloc__type_match(string, "misc", 4)) {
     type = HWLOC_OBJ_MISC;
 
-  } else if (!hwloc_strncasecmp(string, "bridge", 4)) {
+  } else if (hwloc__type_match(string, "bridge", 4)) {
     type = HWLOC_OBJ_BRIDGE;
-  } else if (!hwloc_strncasecmp(string, "hostbridge", 6)) {
+  } else if (hwloc__type_match(string, "hostbridge", 6)) {
     type = HWLOC_OBJ_BRIDGE;
     ubtype = HWLOC_OBJ_BRIDGE_HOST;
-  } else if (!hwloc_strncasecmp(string, "pcibridge", 5)) {
+  } else if (hwloc__type_match(string, "pcibridge", 5)) {
     type = HWLOC_OBJ_BRIDGE;
     ubtype = HWLOC_OBJ_BRIDGE_PCI;
 
-  } else if (!hwloc_strncasecmp(string, "pci", 3)) {
+  } else if (hwloc__type_match(string, "pcidev", 3)) {
     type = HWLOC_OBJ_PCI_DEVICE;
 
   /* types with depthattr */
   } else if ((string[0] == 'l' || string[0] == 'L') && string[1] >= '0' && string[1] <= '9') {
+    char *suffix;
     depthattr = strtol(string+1, &end, 10);
-    if (*end == 'i') {
+    if (*end == 'i' || *end == 'I') {
       if (depthattr >= 1 && depthattr <= 3) {
 	type = HWLOC_OBJ_L1ICACHE + depthattr-1;
 	cachetypeattr = HWLOC_OBJ_CACHE_INSTRUCTION;
+	suffix = end+1;
       } else
 	return -1;
     } else {
       if (depthattr >= 1 && depthattr <= 5) {
 	type = HWLOC_OBJ_L1CACHE + depthattr-1;
-	cachetypeattr = *end == 'd' ? HWLOC_OBJ_CACHE_DATA : HWLOC_OBJ_CACHE_UNIFIED;
+	if (*end == 'd' || *end == 'D') {
+	  cachetypeattr = HWLOC_OBJ_CACHE_DATA;
+	  suffix = end+1;
+	} else if (*end == 'u' || *end == 'U') {
+	  cachetypeattr = HWLOC_OBJ_CACHE_UNIFIED;
+	  suffix = end+1;
+	} else {
+	  cachetypeattr = HWLOC_OBJ_CACHE_UNIFIED;
+	  suffix = end;
+	}
       } else
 	return -1;
     }
+    /* check whether the optional suffix matches "cache" */
+    if (!hwloc__type_match(suffix, "cache", 0))
+      return -1;
 
-  } else if (!hwloc_strncasecmp(string, "group", 2)) {
+  } else if ((end = (char *) hwloc__type_match(string, "group", 2)) != NULL) {
-    size_t length;
     type = HWLOC_OBJ_GROUP;
-    length = strcspn(string, "0123456789");
+    if (*end >= '0' && *end <= '9') {
-    if (length <= 5 && !hwloc_strncasecmp(string, "group", length)
+      depthattr = strtol(end, &end, 10);
-	&& string[length] >= '0' && string[length] <= '9') {
-      depthattr = strtol(string+length, &end, 10);
     }
 
   } else
@@ -421,7 +477,9 @@ hwloc_obj_type_snprintf(char * __hwloc_restrict string, size_t size, hwloc_obj_t
   case HWLOC_OBJ_MISC:
   case HWLOC_OBJ_MACHINE:
   case HWLOC_OBJ_NUMANODE:
+  case HWLOC_OBJ_MEMCACHE:
   case HWLOC_OBJ_PACKAGE:
+  case HWLOC_OBJ_DIE:
   case HWLOC_OBJ_CORE:
   case HWLOC_OBJ_PU:
     return hwloc_snprintf(string, size, "%s", hwloc_obj_type_string(type));
@@ -523,6 +581,7 @@ hwloc_obj_attr_snprintf(char * __hwloc_restrict string, size_t size, hwloc_obj_t
   case HWLOC_OBJ_L1ICACHE:
   case HWLOC_OBJ_L2ICACHE:
   case HWLOC_OBJ_L3ICACHE:
+  case HWLOC_OBJ_MEMCACHE:
     if (verbose) {
       char assoc[32];
       if (obj->attr->cache.associativity == -1)

src/App.cpp

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -33,6 +33,7 @@
 #include "base/io/Console.h"
 #include "base/io/log/Log.h"
 #include "base/kernel/Signals.h"
+#include "base/kernel/Platform.h"
 #include "core/config/Config.h"
 #include "core/Controller.h"
 #include "core/Miner.h"
@@ -49,6 +50,8 @@ xmrig::App::App(Process *process)
 
 xmrig::App::~App()
 {
+    Cpu::release();
+
     delete m_signals;
     delete m_console;
     delete m_controller;
@@ -58,7 +61,7 @@ xmrig::App::~App()
 int xmrig::App::exec()
 {
     if (!m_controller->isReady()) {
-        LOG_EMERG("no valid configuration found.");
+        LOG_EMERG("no valid configuration found, try https://xmrig.com/wizard");
 
         return 2;
     }

src/App.h

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by

src/Summary.cpp

@@ -39,6 +39,11 @@
 #include "version.h"
 
 
+#ifdef XMRIG_ALGO_RANDOMX
+#   include "crypto/rx/RxConfig.h"
+#endif
+
+
 namespace xmrig {
 
 
@@ -59,24 +64,36 @@ inline static const char *asmName(Assembly::Id assembly)
 #endif
 
 
-static void print_memory(Config *config) {
+static void print_memory(Config *config)
-#   ifdef _WIN32
+{
+#   ifdef XMRIG_OS_WIN
     Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") "%s",
                "HUGE PAGES", config->cpu().isHugePages() ? (VirtualMemory::isHugepagesAvailable() ? GREEN_BOLD("permission granted") : RED_BOLD("unavailable")) : RED_BOLD("disabled"));
+#   else
+    Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") "%s", "HUGE PAGES",  config->cpu().isHugePages() ? GREEN_BOLD("supported") : RED_BOLD("disabled"));
+#   endif
+
+#   ifdef XMRIG_ALGO_RANDOMX
+#   ifdef XMRIG_OS_LINUX
+    Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") "%s",
+               "1GB PAGES", (VirtualMemory::isOneGbPagesAvailable() ? (config->rx().isOneGbPages() ? GREEN_BOLD("supported") : YELLOW_BOLD("disabled")) : YELLOW_BOLD("unavailable")));
+#   else
+    Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") "%s", "1GB PAGES", YELLOW_BOLD("unavailable"));
+#   endif
 #   endif
 }
 
 
 static void print_cpu(Config *)
 {
-    const ICpuInfo *info = Cpu::info();
+    const auto info = Cpu::info();
 
     Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s%s (%zu)") " %sx64 %sAES",
                "CPU",
                info->brand(),
                info->packages(),
-               info->isX64()   ? GREEN_BOLD_S : RED_BOLD_S "-",
+               info->isX64()          ? GREEN_BOLD_S : RED_BOLD_S "-",
-               info->hasAES()  ? GREEN_BOLD_S : RED_BOLD_S "-"
+               info->hasAES()         ? GREEN_BOLD_S : RED_BOLD_S "-"
                );
 #   if defined(XMRIG_FEATURE_LIBCPUID) || defined (XMRIG_FEATURE_HWLOC)
     Log::print(WHITE_BOLD("   %-13s") BLACK_BOLD("L2:") WHITE_BOLD("%.1f MB") BLACK_BOLD(" L3:") WHITE_BOLD("%.1f MB")
@@ -102,6 +119,23 @@ static void print_cpu(Config *)
 }
 
 
+static void print_memory()
+{
+    constexpr size_t oneGiB = 1024U * 1024U * 1024U;
+    const auto freeMem      = static_cast<double>(uv_get_free_memory());
+    const auto totalMem     = static_cast<double>(uv_get_total_memory());
+
+    const double percent = freeMem > 0 ? ((totalMem - freeMem) / totalMem * 100.0) : 100.0;
+
+    Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") CYAN_BOLD("%.1f/%.1f GB") BLACK_BOLD(" (%.0f%%)"),
+               "MEMORY",
+               (totalMem - freeMem) / oneGiB,
+               totalMem / oneGiB,
+               percent
+               );
+}
+
+
 static void print_threads(Config *config)
 {
     Log::print(GREEN_BOLD(" * ") WHITE_BOLD("%-13s") WHITE_BOLD("%s%d%%"),
@@ -125,7 +159,7 @@ static void print_threads(Config *config)
 
 static void print_commands(Config *)
 {
-    if (Log::colors) {
+    if (Log::isColors()) {
         Log::print(GREEN_BOLD(" * ") WHITE_BOLD("COMMANDS     ") MAGENTA_BG(WHITE_BOLD_S "h") WHITE_BOLD("ashrate, ")
                                                                      MAGENTA_BG(WHITE_BOLD_S "p") WHITE_BOLD("ause, ")
                                                                      MAGENTA_BG(WHITE_BOLD_S "r") WHITE_BOLD("esume"));
@@ -144,6 +178,7 @@ void xmrig::Summary::print(Controller *controller)
     controller->config()->printVersions();
     print_memory(controller->config());
     print_cpu(controller->config());
+    print_memory();
     print_threads(controller->config());
     controller->config()->pools().print();
 

src/backend/common/Hashrate.cpp

@@ -30,6 +30,7 @@
 
 
 #include "backend/common/Hashrate.h"
+#include "base/io/json/Json.h"
 #include "base/tools/Chrono.h"
 #include "base/tools/Handle.h"
 #include "rapidjson/document.h"
@@ -157,13 +158,7 @@ const char *xmrig::Hashrate::format(double h, char *buf, size_t size)
 
 rapidjson::Value xmrig::Hashrate::normalize(double d)
 {
-    using namespace rapidjson;
+    return Json::normalize(d, false);
-
-    if (!std::isnormal(d)) {
-        return Value(kNullType);
-    }
-
-    return Value(floor(d * 100.0) / 100.0);
 }
 
 

src/backend/common/Threads.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by

src/backend/common/Threads.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -30,8 +30,8 @@
 #include <set>
 
 
+#include "base/crypto/Algorithm.h"
 #include "base/tools/String.h"
-#include "crypto/common/Algorithm.h"
 #include "rapidjson/fwd.h"
 
 

src/backend/common/WorkerJob.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -64,13 +64,17 @@ public:
     }
 
 
-    inline void nextRound(uint32_t rounds, uint32_t roundSize)
+    inline bool nextRound(uint32_t rounds, uint32_t roundSize)
     {
+        bool ok = true;
         m_rounds[index()]++;
 
         if ((m_rounds[index()] % rounds) == 0) {
             for (size_t i = 0; i < N; ++i) {
-                *nonce(i) = Nonce::next(index(), *nonce(i), rounds * roundSize, currentJob().isNicehash());
+                *nonce(i) = Nonce::next(index(), *nonce(i), rounds * roundSize, currentJob().isNicehash(), &ok);
+                if (!ok) {
+                    break;
+                }
             }
         }
         else {
@@ -78,6 +82,8 @@ public:
                 *nonce(i) += roundSize;
             }
         }
+
+        return ok;
     }
 
 
@@ -114,16 +120,19 @@ inline uint32_t *xmrig::WorkerJob<1>::nonce(size_t)
 
 
 template<>
-inline void xmrig::WorkerJob<1>::nextRound(uint32_t rounds, uint32_t roundSize)
+inline bool xmrig::WorkerJob<1>::nextRound(uint32_t rounds, uint32_t roundSize)
 {
+    bool ok = true;
     m_rounds[index()]++;
 
     if ((m_rounds[index()] % rounds) == 0) {
-        *nonce() = Nonce::next(index(), *nonce(), rounds * roundSize, currentJob().isNicehash());
+        *nonce() = Nonce::next(index(), *nonce(), rounds * roundSize, currentJob().isNicehash(), &ok);
     }
     else {
         *nonce() += roundSize;
     }
+
+    return ok;
 }
 
 

src/backend/common/Workers.cpp

@@ -109,6 +109,11 @@ void xmrig::Workers<T>::start(const std::vector<T> &data)
 
     for (Thread<T> *worker : m_workers) {
         worker->start(Workers<T>::onReady);
+
+        // This sleep is important for optimal caching!
+        // Threads must allocate scratchpads in order so that adjacent cores will use adjacent scratchpads
+        // Sub-optimal caching can result in up to 0.5% hashrate penalty
+        std::this_thread::sleep_for(std::chrono::milliseconds(20));
     }
 }
 
@@ -163,7 +168,7 @@ void xmrig::Workers<T>::onReady(void *arg)
     assert(worker != nullptr);
 
     if (!worker || !worker->selfTest()) {
-        LOG_ERR("%s " RED("thread ") RED_BOLD("#%zu") RED(" self-test failed"), T::tag(), worker->id());
+        LOG_ERR("%s " RED("thread ") RED_BOLD("#%zu") RED(" self-test failed"), T::tag(), worker ? worker->id() : 0);
 
         handle->backend()->start(worker, false);
         delete worker;

src/backend/common/interfaces/IBackend.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -56,6 +56,7 @@ public:
     virtual void execCommand(char command)                              = 0;
     virtual void prepare(const Job &nextJob)                            = 0;
     virtual void printHashrate(bool details)                            = 0;
+    virtual void printHealth()                                          = 0;
     virtual void setJob(const Job &job)                                 = 0;
     virtual void start(IWorker *worker, bool ready)                     = 0;
     virtual void stop()                                                 = 0;

src/backend/common/interfaces/IRxStorage.h

@@ -24,6 +24,10 @@
 #define XMRIG_IRXSTORAGE_H
 
 
+#include "crypto/rx/RxConfig.h"
+#include "crypto/common/HugePagesInfo.h"
+
+
 #include <cstdint>
 #include <utility>
 
@@ -41,9 +45,10 @@ class IRxStorage
 public:
     virtual ~IRxStorage() = default;
 
-    virtual RxDataset *dataset(const Job &job, uint32_t nodeId) const       = 0;
+    virtual bool isAllocated() const                                                                                            = 0;
-    virtual std::pair<uint32_t, uint32_t> hugePages() const                 = 0;
+    virtual HugePagesInfo hugePages() const                                                                                     = 0;
-    virtual void init(const RxSeed &seed, uint32_t threads, bool hugePages) = 0;
+    virtual RxDataset *dataset(const Job &job, uint32_t nodeId) const                                                           = 0;
+    virtual void init(const RxSeed &seed, uint32_t threads, bool hugePages, bool oneGbPages, RxConfig::Mode mode, int priority) = 0;
 };
 
 

src/backend/cpu/Cpu.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -91,8 +91,6 @@ rapidjson::Value xmrig::Cpu::toJSON(rapidjson::Document &doc)
 
 void xmrig::Cpu::release()
 {
-    assert(cpuInfo != nullptr);
-
     delete cpuInfo;
     cpuInfo = nullptr;
 }

src/backend/cpu/Cpu.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by

src/backend/cpu/CpuBackend.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -68,17 +68,15 @@ static std::mutex mutex;
 struct CpuLaunchStatus
 {
 public:
-    inline size_t hugePages() const     { return m_hugePages; }
+    inline const HugePagesInfo &hugePages() const   { return m_hugePages; }
-    inline size_t memory() const        { return m_ways * m_memory; }
+    inline size_t memory() const                    { return m_ways * m_memory; }
-    inline size_t pages() const         { return m_pages; }
+    inline size_t threads() const                   { return m_threads; }
-    inline size_t threads() const       { return m_threads; }
+    inline size_t ways() const                      { return m_ways; }
-    inline size_t ways() const          { return m_ways; }
 
     inline void start(const std::vector<CpuLaunchData> &threads, size_t memory)
     {
-        m_hugePages = 0;
+        m_hugePages.reset();
         m_memory    = memory;
-        m_pages     = 0;
         m_started   = 0;
         m_errors    = 0;
         m_threads   = threads.size();
@@ -89,11 +87,9 @@ public:
     inline bool started(IWorker *worker, bool ready)
     {
         if (ready) {
-            auto hugePages = worker->memory()->hugePages();
-
             m_started++;
-            m_hugePages += hugePages.first;
+
-            m_pages     += hugePages.second;
+            m_hugePages += worker->memory()->hugePages();
             m_ways      += worker->intensity();
         }
         else {
@@ -115,19 +111,18 @@ public:
                  tag,
                  m_errors == 0 ? CYAN_BOLD_S : YELLOW_BOLD_S,
                  m_started, m_threads, m_ways,
-                 (m_hugePages == m_pages ? GREEN_BOLD_S : (m_hugePages == 0 ? RED_BOLD_S : YELLOW_BOLD_S)),
+                 (m_hugePages.isFullyAllocated() ? GREEN_BOLD_S : (m_hugePages.allocated == 0 ? RED_BOLD_S : YELLOW_BOLD_S)),
-                 m_hugePages == 0 ? 0.0 : static_cast<double>(m_hugePages) / m_pages * 100.0,
+                 m_hugePages.percent(),
-                 m_hugePages, m_pages,
+                 m_hugePages.allocated, m_hugePages.total,
                  memory() / 1024,
                  Chrono::steadyMSecs() - m_ts
                  );
     }
 
 private:
+    HugePagesInfo m_hugePages;
     size_t m_errors       = 0;
-    size_t m_hugePages    = 0;
     size_t m_memory       = 0;
-    size_t m_pages        = 0;
     size_t m_started      = 0;
     size_t m_threads      = 0;
     size_t m_ways         = 0;
@@ -169,18 +164,17 @@ public:
 
     rapidjson::Value hugePages(int version, rapidjson::Document &doc)
     {
-        std::pair<unsigned, unsigned> pages(0, 0);
+        HugePagesInfo pages;
 
     #   ifdef XMRIG_ALGO_RANDOMX
         if (algo.family() == Algorithm::RANDOM_X) {
-            pages = Rx::hugePages();
+            pages += Rx::hugePages();
         }
     #   endif
 
         mutex.lock();
 
-        pages.first  += status.hugePages();
+        pages += status.hugePages();
-        pages.second += status.pages();
 
         mutex.unlock();
 
@@ -188,11 +182,11 @@ public:
 
         if (version > 1) {
             hugepages.SetArray();
-            hugepages.PushBack(pages.first, doc.GetAllocator());
+            hugepages.PushBack(static_cast<uint64_t>(pages.allocated), doc.GetAllocator());
-            hugepages.PushBack(pages.second, doc.GetAllocator());
+            hugepages.PushBack(static_cast<uint64_t>(pages.total), doc.GetAllocator());
         }
         else {
-            hugepages = pages.first == pages.second;
+            hugepages = pages.isFullyAllocated();
         }
 
         return hugepages;
@@ -325,6 +319,11 @@ void xmrig::CpuBackend::printHashrate(bool details)
 }
 
 
+void xmrig::CpuBackend::printHealth()
+{
+}
+
+
 void xmrig::CpuBackend::setJob(const Job &job)
 {
     if (!isEnabled()) {
@@ -417,6 +416,10 @@ rapidjson::Value xmrig::CpuBackend::toJSON(rapidjson::Document &doc) const
     out.AddMember("argon2-impl", argon2::Impl::name().toJSON(), allocator);
 #   endif
 
+#   ifdef XMRIG_ALGO_ASTROBWT
+    out.AddMember("astrobwt-max-size", cpu.astrobwtMaxSize(), allocator);
+#   endif
+
     out.AddMember("hugepages", d_ptr->hugePages(2, doc), allocator);
     out.AddMember("memory",    static_cast<uint64_t>(d_ptr->algo.isValid() ? (d_ptr->ways() * d_ptr->algo.l3()) : 0), allocator);
 

src/backend/cpu/CpuBackend.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -59,6 +59,7 @@ protected:
     const String &type() const override;
     void prepare(const Job &nextJob) override;
     void printHashrate(bool details) override;
+    void printHealth() override;
     void setJob(const Job &job) override;
     void start(IWorker *worker, bool ready) override;
     void stop() override;

src/backend/cpu/CpuConfig.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -29,6 +29,8 @@
 #include "base/io/json/Json.h"
 #include "rapidjson/document.h"
 
+#include <algorithm>
+
 
 namespace xmrig {
 
@@ -38,6 +40,7 @@ static const char *kHwAes               = "hw-aes";
 static const char *kMaxThreadsHint      = "max-threads-hint";
 static const char *kMemoryPool          = "memory-pool";
 static const char *kPriority            = "priority";
+static const char *kYield               = "yield";
 
 #ifdef XMRIG_FEATURE_ASM
 static const char *kAsm = "asm";
@@ -47,6 +50,12 @@ static const char *kAsm = "asm";
 static const char *kArgon2Impl = "argon2-impl";
 #endif
 
+#ifdef XMRIG_ALGO_ASTROBWT
+static const char* kAstroBWTMaxSize = "astrobwt-max-size";
+static const char* kAstroBWTAVX2    = "astrobwt-avx2";
+#endif
+
+
 extern template class Threads<CpuThreads>;
 
 }
@@ -70,6 +79,7 @@ rapidjson::Value xmrig::CpuConfig::toJSON(rapidjson::Document &doc) const
     obj.AddMember(StringRef(kHwAes),        m_aes == AES_AUTO ? Value(kNullType) : Value(m_aes == AES_HW), allocator);
     obj.AddMember(StringRef(kPriority),     priority() != -1 ? Value(priority()) : Value(kNullType), allocator);
     obj.AddMember(StringRef(kMemoryPool),   m_memoryPool < 1 ? Value(m_memoryPool < 0) : Value(m_memoryPool), allocator);
+    obj.AddMember(StringRef(kYield),        m_yield, allocator);
 
     if (m_threads.isEmpty()) {
         obj.AddMember(StringRef(kMaxThreadsHint), m_limit, allocator);
@@ -83,6 +93,11 @@ rapidjson::Value xmrig::CpuConfig::toJSON(rapidjson::Document &doc) const
     obj.AddMember(StringRef(kArgon2Impl), m_argon2Impl.toJSON(), allocator);
 #   endif
 
+#   ifdef XMRIG_ALGO_ASTROBWT
+    obj.AddMember(StringRef(kAstroBWTMaxSize),  m_astrobwtMaxSize, allocator);
+    obj.AddMember(StringRef(kAstroBWTAVX2),     m_astrobwtAVX2, allocator);
+#   endif
+
     m_threads.toJSON(obj, doc);
 
     return obj;
@@ -117,9 +132,10 @@ std::vector<xmrig::CpuLaunchData> xmrig::CpuConfig::get(const Miner *miner, cons
 void xmrig::CpuConfig::read(const rapidjson::Value &value)
 {
     if (value.IsObject()) {
-        m_enabled   = Json::getBool(value, kEnabled, m_enabled);
+        m_enabled    = Json::getBool(value, kEnabled, m_enabled);
-        m_hugePages = Json::getBool(value, kHugePages, m_hugePages);
+        m_hugePages  = Json::getBool(value, kHugePages, m_hugePages);
-        m_limit     = Json::getUint(value, kMaxThreadsHint, m_limit);
+        m_limit      = Json::getUint(value, kMaxThreadsHint, m_limit);
+        m_yield      = Json::getBool(value, kYield, m_yield);
 
         setAesMode(Json::getValue(value, kHwAes));
         setPriority(Json::getInt(value,  kPriority, -1));
@@ -133,6 +149,24 @@ void xmrig::CpuConfig::read(const rapidjson::Value &value)
         m_argon2Impl = Json::getString(value, kArgon2Impl);
 #       endif
 
+#       ifdef XMRIG_ALGO_ASTROBWT
+        const auto& astroBWTMaxSize = Json::getValue(value, kAstroBWTMaxSize);
+        if (astroBWTMaxSize.IsNull() || !astroBWTMaxSize.IsInt()) {
+            m_shouldSave = true;
+        }
+        else {
+            m_astrobwtMaxSize = std::min(std::max(astroBWTMaxSize.GetInt(), 400), 1200);
+        }
+
+        const auto& astroBWTAVX2 = Json::getValue(value, kAstroBWTAVX2);
+        if (astroBWTAVX2.IsNull() || !astroBWTAVX2.IsBool()) {
+            m_shouldSave = true;
+        }
+        else {
+            m_astrobwtAVX2 = astroBWTAVX2.GetBool();
+        }
+#       endif
+
         m_threads.read(value);
 
         generate();
@@ -162,8 +196,9 @@ void xmrig::CpuConfig::generate()
     count += xmrig::generate<Algorithm::CN_PICO>(m_threads, m_limit);
     count += xmrig::generate<Algorithm::RANDOM_X>(m_threads, m_limit);
     count += xmrig::generate<Algorithm::ARGON2>(m_threads, m_limit);
+    count += xmrig::generate<Algorithm::ASTROBWT>(m_threads, m_limit);
 
-    m_shouldSave = count > 0;
+    m_shouldSave |= count > 0;
 }
 
 

src/backend/cpu/CpuConfig.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -55,10 +55,14 @@ public:
     inline bool isEnabled() const                       { return m_enabled; }
     inline bool isHugePages() const                     { return m_hugePages; }
     inline bool isShouldSave() const                    { return m_shouldSave; }
+    inline bool isYield() const                         { return m_yield; }
     inline const Assembly &assembly() const             { return m_assembly; }
     inline const String &argon2Impl() const             { return m_argon2Impl; }
     inline const Threads<CpuThreads> &threads() const   { return m_threads; }
+    inline int astrobwtMaxSize() const                  { return m_astrobwtMaxSize; }
+    inline bool astrobwtAVX2() const                    { return m_astrobwtAVX2; }
     inline int priority() const                         { return m_priority; }
+    inline uint32_t limit() const                       { return m_limit; }
 
 private:
     void generate();
@@ -67,16 +71,19 @@ private:
 
     inline void setPriority(int priority)   { m_priority = (priority >= -1 && priority <= 5) ? priority : -1; }
 
-    AesMode m_aes        = AES_AUTO;
+    AesMode m_aes           = AES_AUTO;
     Assembly m_assembly;
-    bool m_enabled       = true;
+    bool m_astrobwtAVX2     = false;
-    bool m_hugePages     = true;
+    bool m_enabled          = true;
-    bool m_shouldSave    = false;
+    bool m_hugePages        = true;
-    int m_memoryPool     = 0;
+    bool m_shouldSave       = false;
-    int m_priority       = -1;
+    bool m_yield            = true;
+    int m_astrobwtMaxSize   = 550;
+    int m_memoryPool        = 0;
+    int m_priority          = -1;
     String m_argon2Impl;
     Threads<CpuThreads> m_threads;
-    uint32_t m_limit     = 100;
+    uint32_t m_limit        = 100;
 };
 
 

src/backend/cpu/CpuConfig_gen.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -109,11 +109,11 @@ template<>
 size_t inline generate<Algorithm::RANDOM_X>(Threads<CpuThreads> &threads, uint32_t limit)
 {
     size_t count = 0;
-
+    auto cpuInfo = Cpu::info();
-    auto wow = Cpu::info()->threads(Algorithm::RX_WOW, limit);
+    auto wow     = cpuInfo->threads(Algorithm::RX_WOW, limit);
 
     if (!threads.isExist(Algorithm::RX_ARQ)) {
-        auto arq = Cpu::info()->threads(Algorithm::RX_ARQ, limit);
+        auto arq = cpuInfo->threads(Algorithm::RX_ARQ, limit);
         if (arq == wow) {
             threads.setAlias(Algorithm::RX_ARQ, "rx/wow");
             ++count;
@@ -123,6 +123,17 @@ size_t inline generate<Algorithm::RANDOM_X>(Threads<CpuThreads> &threads, uint32
         }
     }
 
+    if (!threads.isExist(Algorithm::RX_KEVA)) {
+        auto keva = cpuInfo->threads(Algorithm::RX_KEVA, limit);
+        if (keva == wow) {
+            threads.setAlias(Algorithm::RX_KEVA, "rx/wow");
+            ++count;
+        }
+        else {
+            count += threads.move("rx/keva", std::move(keva));
+        }
+    }
+
     if (!threads.isExist(Algorithm::RX_WOW)) {
         count += threads.move("rx/wow", std::move(wow));
     }
@@ -143,6 +154,14 @@ size_t inline generate<Algorithm::ARGON2>(Threads<CpuThreads> &threads, uint32_t
 #endif
 
 
+#ifdef XMRIG_ALGO_ASTROBWT
+template<>
+size_t inline generate<Algorithm::ASTROBWT>(Threads<CpuThreads>& threads, uint32_t limit)
+{
+    return generate("astrobwt", threads, Algorithm::ASTROBWT_DERO, limit);
+}
+#endif
+
 } /* namespace xmrig */
 
 

src/backend/cpu/CpuLaunchData.cpp

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -25,7 +25,6 @@
 
 
 #include "backend/cpu/CpuLaunchData.h"
-
 #include "backend/common/Tags.h"
 #include "backend/cpu/CpuConfig.h"
 
@@ -36,8 +35,11 @@
 xmrig::CpuLaunchData::CpuLaunchData(const Miner *miner, const Algorithm &algorithm, const CpuConfig &config, const CpuThread &thread) :
     algorithm(algorithm),
     assembly(config.assembly()),
+    astrobwtAVX2(config.astrobwtAVX2()),
     hugePages(config.isHugePages()),
     hwAES(config.isHwAES()),
+    yield(config.isYield()),
+    astrobwtMaxSize(config.astrobwtMaxSize()),    
     priority(config.priority()),
     affinity(thread.affinity()),
     miner(miner),

src/backend/cpu/CpuLaunchData.h

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -27,8 +27,8 @@
 #define XMRIG_CPULAUNCHDATA_H
 
 
+#include "base/crypto/Algorithm.h"
 #include "crypto/cn/CnHash.h"
-#include "crypto/common/Algorithm.h"
 #include "crypto/common/Assembly.h"
 #include "crypto/common/Nonce.h"
 
@@ -58,8 +58,11 @@ public:
 
     const Algorithm algorithm;
     const Assembly assembly;
+    const bool astrobwtAVX2;
     const bool hugePages;
     const bool hwAES;
+    const bool yield;
+    const int astrobwtMaxSize;
     const int priority;
     const int64_t affinity;
     const Miner *miner;

src/backend/cpu/CpuWorker.cpp

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -32,6 +32,7 @@
 #include "core/Miner.h"
 #include "crypto/cn/CnCtx.h"
 #include "crypto/cn/CryptoNight_test.h"
+#include "crypto/cn/CryptoNight.h"
 #include "crypto/common/Nonce.h"
 #include "crypto/common/VirtualMemory.h"
 #include "crypto/rx/Rx.h"
@@ -44,10 +45,29 @@
 #endif
 
 
+#ifdef XMRIG_ALGO_ASTROBWT
+#   include "crypto/astrobwt/AstroBWT.h"
+#endif
+
+
 namespace xmrig {
 
 static constexpr uint32_t kReserveCount = 32768;
 
+
+template<size_t N>
+inline bool nextRound(WorkerJob<N> &job)
+{
+    if (!job.nextRound(kReserveCount, 1)) {
+        JobResults::done(job.currentJob());
+
+        return false;
+    }
+
+    return true;
+}
+
+
 } // namespace xmrig
 
 
@@ -57,12 +77,15 @@ xmrig::CpuWorker<N>::CpuWorker(size_t id, const CpuLaunchData &data) :
     Worker(id, data.affinity, data.priority),
     m_algorithm(data.algorithm),
     m_assembly(data.assembly),
+    m_astrobwtAVX2(data.astrobwtAVX2),
     m_hwAES(data.hwAES),
+    m_yield(data.yield),
     m_av(data.av()),
+    m_astrobwtMaxSize(data.astrobwtMaxSize * 1000),
     m_miner(data.miner),
     m_ctx()
 {
-    m_memory = new VirtualMemory(m_algorithm.l3() * N, data.hugePages, true, m_node);
+    m_memory = new VirtualMemory(m_algorithm.l3() * N, data.hugePages, false, true, m_node);
 }
 
 
@@ -70,7 +93,7 @@ template<size_t N>
 xmrig::CpuWorker<N>::~CpuWorker()
 {
 #   ifdef XMRIG_ALGO_RANDOMX
-    delete m_vm;
+    RxVm::destroy(m_vm);
 #   endif
 
     CnCtx::release(m_ctx, N);
@@ -95,7 +118,7 @@ void xmrig::CpuWorker<N>::allocateRandomX_VM()
     }
 
     if (!m_vm) {
-        m_vm = new RxVm(dataset, m_memory->scratchpad(), !m_hwAES);
+        m_vm = RxVm::create(dataset, m_memory->scratchpad(), !m_hwAES, m_assembly, m_node);
     }
 }
 #endif
@@ -153,7 +176,8 @@ bool xmrig::CpuWorker<N>::selfTest()
 
 #   ifdef XMRIG_ALGO_CN_PICO
     if (m_algorithm.family() == Algorithm::CN_PICO) {
-        return verify(Algorithm::CN_PICO_0, test_output_pico_trtl);
+        return verify(Algorithm::CN_PICO_0, test_output_pico_trtl) &&
+               verify(Algorithm::CN_PICO_TLO, test_output_pico_tlo);
     }
 #   endif
 
@@ -164,6 +188,12 @@ bool xmrig::CpuWorker<N>::selfTest()
     }
 #   endif
 
+#   ifdef XMRIG_ALGO_ASTROBWT
+    if (m_algorithm.family() == Algorithm::ASTROBWT) {
+        return verify(Algorithm::ASTROBWT_DERO, astrobwt_dero_test_out);
+    }
+#   endif
+
     return false;
 }
 
@@ -185,8 +215,20 @@ void xmrig::CpuWorker<N>::start()
             consumeJob();
         }
 
+        uint64_t storeStatsMask = 7;
+
+#       ifdef XMRIG_ALGO_RANDOMX
+        bool first = true;
+        uint64_t tempHash[8] = {};
+
+        // RandomX is faster, we don't need to store stats so often
+        if (m_job.currentJob().algorithm().family() == Algorithm::RANDOM_X) {
+            storeStatsMask = 63;
+        }
+#       endif
+
         while (!Nonce::isOutdated(Nonce::CPU, m_job.sequence())) {
-            if ((m_count & 0x7) == 0) {
+            if ((m_count & storeStatsMask) == 0) {
                 storeStats();
             }
 
@@ -196,26 +238,57 @@ void xmrig::CpuWorker<N>::start()
                 break;
             }
 
+            uint32_t current_job_nonces[N];
+            for (size_t i = 0; i < N; ++i) {
+                current_job_nonces[i] = *m_job.nonce(i);
+            }
+
+            bool valid = true;
+
 #           ifdef XMRIG_ALGO_RANDOMX
             if (job.algorithm().family() == Algorithm::RANDOM_X) {
-                randomx_calculate_hash(m_vm->get(), m_job.blob(), job.size(), m_hash);
+                if (first) {
+                    first = false;
+                    randomx_calculate_hash_first(m_vm, tempHash, m_job.blob(), job.size());
+                }
+
+                if (!nextRound(m_job)) {
+                    break;
+                }
+
+                randomx_calculate_hash_next(m_vm, tempHash, m_job.blob(), job.size(), m_hash);
             }
             else
 #           endif
             {
-                fn(job.algorithm())(m_job.blob(), job.size(), m_hash, m_ctx, job.height());
+#               ifdef XMRIG_ALGO_ASTROBWT
-            }
+                if (job.algorithm().family() == Algorithm::ASTROBWT) {
-
+                    if (!astrobwt::astrobwt_dero(m_job.blob(), job.size(), m_ctx[0]->memory, m_hash, m_astrobwtMaxSize, m_astrobwtAVX2))
-            for (size_t i = 0; i < N; ++i) {
+                        valid = false;
-                if (*reinterpret_cast<uint64_t*>(m_hash + (i * 32) + 24) < job.target()) {
-                    JobResults::submit(job, *m_job.nonce(i), m_hash + (i * 32));
                 }
+                else
+#               endif
+                {
+                    fn(job.algorithm())(m_job.blob(), job.size(), m_hash, m_ctx, job.height());
+                }
+
+                if (!nextRound(m_job)) {
+                    break;
+                };
             }
 
-            m_job.nextRound(kReserveCount, 1);
+            if (valid) {
-            m_count += N;
+                for (size_t i = 0; i < N; ++i) {
+                    if (*reinterpret_cast<uint64_t*>(m_hash + (i * 32) + 24) < job.target()) {
+                        JobResults::submit(job, current_job_nonces[i], m_hash + (i * 32));
+                    }
+                }
+                m_count += N;
+            }
 
-            std::this_thread::yield();
+            if (m_yield) {
+                std::this_thread::yield();
+            }
         }
 
         consumeJob();

src/backend/cpu/CpuWorker.h

@@ -6,8 +6,8 @@
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2018 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
  * Copyright 2018      Lee Clagett <https://github.com/vtnerd>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <https://github.com/xmrig>, <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -34,6 +34,11 @@
 #include "net/JobResult.h"
 
 
+#ifdef XMRIG_ALGO_RANDOMX
+class randomx_vm;
+#endif
+
+
 namespace xmrig {
 
 
@@ -70,8 +75,11 @@ private:
 
     const Algorithm m_algorithm;
     const Assembly m_assembly;
+    const bool m_astrobwtAVX2;
     const bool m_hwAES;
+    const bool m_yield;
     const CnHash::AlgoVariant m_av;
+    const int m_astrobwtMaxSize;
     const Miner *m_miner;
     cryptonight_ctx *m_ctx[N];
     uint8_t m_hash[N * 32]{ 0 };
@@ -79,7 +87,7 @@ private:
     WorkerJob<N> m_job;
 
 #   ifdef XMRIG_ALGO_RANDOMX
-    RxVm *m_vm = nullptr;
+    randomx_vm *m_vm = nullptr;
 #   endif
 };
 

src/backend/cpu/interfaces/ICpuInfo.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2019 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -27,8 +27,8 @@
 
 
 #include "backend/cpu/CpuThreads.h"
+#include "base/crypto/Algorithm.h"
 #include "crypto/common/Assembly.h"
-#include "crypto/common/Algorithm.h"
 
 
 namespace xmrig {
@@ -37,6 +37,20 @@ namespace xmrig {
 class ICpuInfo
 {
 public:
+    enum Vendor : uint32_t {
+        VENDOR_UNKNOWN,
+        VENDOR_INTEL,
+        VENDOR_AMD
+    };
+
+    enum MsrMod : uint32_t {
+        MSR_MOD_NONE,
+        MSR_MOD_RYZEN,
+        MSR_MOD_INTEL,
+        MSR_MOD_CUSTOM,
+        MSR_MOD_MAX
+    };
+
     virtual ~ICpuInfo() = default;
 
 #   if defined(__x86_64__) || defined(_M_AMD64) || defined (__arm64__) || defined (__aarch64__)
@@ -48,15 +62,19 @@ public:
     virtual Assembly::Id assembly() const                                           = 0;
     virtual bool hasAES() const                                                     = 0;
     virtual bool hasAVX2() const                                                    = 0;
+    virtual bool hasBMI2() const                                                    = 0;
+    virtual bool hasOneGbPages() const                                              = 0;
     virtual const char *backend() const                                             = 0;
     virtual const char *brand() const                                               = 0;
     virtual CpuThreads threads(const Algorithm &algorithm, uint32_t limit) const    = 0;
+    virtual MsrMod msrMod() const                                                   = 0;
     virtual size_t cores() const                                                    = 0;
     virtual size_t L2() const                                                       = 0;
     virtual size_t L3() const                                                       = 0;
     virtual size_t nodes() const                                                    = 0;
     virtual size_t packages() const                                                 = 0;
     virtual size_t threads() const                                                  = 0;
+    virtual Vendor vendor() const                                                   = 0;
 };
 
 

src/backend/cpu/platform/AdvancedCpuInfo.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2019 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -22,6 +22,17 @@
  *   along with this program. If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include "backend/cpu/platform/AdvancedCpuInfo.h"
+#include "3rdparty/libcpuid/libcpuid.h"
+
+
+#ifdef _MSC_VER
+#   include <intrin.h>
+#else
+#   include <cpuid.h>
+#endif
+
+
 #include <algorithm>
 #include <cassert>
 #include <cmath>
@@ -29,10 +40,6 @@
 #include <cstring>
 
 
-#include "3rdparty/libcpuid/libcpuid.h"
-#include "backend/cpu/platform/AdvancedCpuInfo.h"
-
-
 namespace xmrig {
 
 
@@ -54,11 +61,38 @@ static inline void cpu_brand_string(char out[64], const char *in) {
 }
 
 
+static inline void cpuid(uint32_t level, int32_t output[4])
+{
+    memset(output, 0, sizeof(int32_t) * 4);
+
+#   ifdef _MSC_VER
+    __cpuid(output, static_cast<int>(level));
+#   else
+    __cpuid_count(level, 0, output[0], output[1], output[2], output[3]);
+#   endif
+}
+
+
+static inline bool has_feature(uint32_t level, uint32_t reg, int32_t bit)
+{
+    int32_t cpu_info[4] = { 0 };
+    cpuid(level, cpu_info);
+
+    return (cpu_info[reg] & bit) != 0;
+}
+
+
+static inline bool has_pdpe1gb()
+{
+    return has_feature(0x80000001, 3, 1 << 26);
+}
+
+
 } // namespace xmrig
 
 
 xmrig::AdvancedCpuInfo::AdvancedCpuInfo() :
-    m_brand()
+    m_pdpe1gb(has_pdpe1gb())
 {
     struct cpu_raw_data_t raw = {};
     struct cpu_id_t data      = {};
@@ -69,21 +103,28 @@ xmrig::AdvancedCpuInfo::AdvancedCpuInfo() :
     cpu_brand_string(m_brand, data.brand_str);
     snprintf(m_backend, sizeof m_backend, "libcpuid/%s", cpuid_lib_version());
 
+    if (data.vendor == ::VENDOR_INTEL) {
+        m_vendor = VENDOR_INTEL;
+    }
+    else if (data.vendor == ::VENDOR_AMD) {
+        m_vendor = VENDOR_AMD;
+    }
+
     m_threads  = static_cast<size_t>(data.total_logical_cpus);
     m_packages = std::max<size_t>(threads() / static_cast<size_t>(data.num_logical_cpus), 1);
     m_cores    = static_cast<size_t>(data.num_cores) * m_packages;
     m_L3       = data.l3_cache > 0 ? static_cast<size_t>(data.l3_cache) * m_packages : 0;
 
-    const size_t l2 = static_cast<size_t>(data.l2_cache);
+    const auto l2 = static_cast<size_t>(data.l2_cache);
 
     // Workaround for AMD CPUs https://github.com/anrieff/libcpuid/issues/97
-    if (data.vendor == VENDOR_AMD && data.ext_family >= 0x15 && data.ext_family < 0x17) {
+    if (m_vendor == VENDOR_AMD && data.ext_family >= 0x15 && data.ext_family < 0x17) {
         m_L2 = l2 * (cores() / 2) * m_packages;
         m_L2_exclusive = true;
     }
     // Workaround for Intel Pentium Dual-Core, Core Duo, Core 2 Duo, Core 2 Quad and their Xeon homologue
     // These processors have L2 cache shared by 2 cores.
-    else if (data.vendor == VENDOR_INTEL && data.ext_family == 0x06 && (data.ext_model == 0x0E || data.ext_model == 0x0F || data.ext_model == 0x17)) {
+    else if (m_vendor == VENDOR_INTEL && data.ext_family == 0x06 && (data.ext_model == 0x0E || data.ext_model == 0x0F || data.ext_model == 0x17)) {
         size_t l2_count_per_socket = cores() > 1 ? cores() / 2 : 1;
         m_L2 = data.l2_cache > 0 ? l2 * l2_count_per_socket * m_packages : 0;
     }
@@ -97,15 +138,22 @@ xmrig::AdvancedCpuInfo::AdvancedCpuInfo() :
     if (data.flags[CPU_FEATURE_AES]) {
         m_aes = true;
 
-        if (data.vendor == VENDOR_AMD) {
+        if (m_vendor == VENDOR_AMD) {
-            m_assembly = (data.ext_family >= 23) ? Assembly::RYZEN : Assembly::BULLDOZER;
+            if (data.ext_family >= 23) {
+                m_assembly = Assembly::RYZEN;
+                m_msrMod   = MSR_MOD_RYZEN;
+            }
+            else {
+                m_assembly = Assembly::BULLDOZER;
+            }
         }
-        else if (data.vendor == VENDOR_INTEL) {
+        else if (m_vendor == VENDOR_INTEL) {
             m_assembly = Assembly::INTEL;
         }
     }
 
     m_avx2 = data.flags[CPU_FEATURE_AVX2] && data.flags[CPU_FEATURE_OSXSAVE];
+    m_bmi2 = data.flags[CPU_FEATURE_BMI2];
 }
 
 
@@ -124,6 +172,17 @@ xmrig::CpuThreads xmrig::AdvancedCpuInfo::threads(const Algorithm &algorithm, ui
     size_t cache = 0;
     size_t count = 0;
 
+#   ifdef XMRIG_ALGO_ASTROBWT
+    if (algorithm == Algorithm::ASTROBWT_DERO) {
+        CpuThreads t;
+        count = threads();
+        for (size_t i = 0; i < count; ++i) {
+            t.add(i, 0);
+        }
+        return t;
+    }
+#   endif
+
     if (m_L3) {
         cache = m_L2_exclusive ? (m_L2 + m_L3) : m_L3;
     }

src/backend/cpu/platform/AdvancedCpuInfo.h

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2019 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -43,27 +43,35 @@ protected:
     inline Assembly::Id assembly() const override   { return m_assembly; }
     inline bool hasAES() const override             { return m_aes; }
     inline bool hasAVX2() const override            { return m_avx2; }
+    inline bool hasBMI2() const override            { return m_bmi2; }
+    inline bool hasOneGbPages() const override      { return m_pdpe1gb; }
     inline const char *backend() const override     { return m_backend; }
     inline const char *brand() const override       { return m_brand; }
+    inline MsrMod msrMod() const override           { return m_msrMod; }
     inline size_t cores() const override            { return m_cores; }
     inline size_t L2() const override               { return m_L2; }
     inline size_t L3() const override               { return m_L3; }
     inline size_t nodes() const override            { return 0; }
     inline size_t packages() const override         { return m_packages; }
     inline size_t threads() const override          { return m_threads; }
+    inline Vendor vendor() const override           { return m_vendor; }
 
 private:
     Assembly m_assembly;
     bool m_aes            = false;
     bool m_avx2           = false;
+    bool m_bmi2           = false;
     bool m_L2_exclusive   = false;
-    char m_backend[32];
+    char m_backend[32]{};
-    char m_brand[64 + 5];
+    char m_brand[64 + 5]{};
+    const bool m_pdpe1gb  = false;
+    MsrMod m_msrMod       = MSR_MOD_NONE;
     size_t m_cores        = 0;
     size_t m_L2           = 0;
     size_t m_L3           = 0;
     size_t m_packages     = 1;
     size_t m_threads      = 0;
+    Vendor m_vendor       = VENDOR_UNKNOWN;
 };
 
 

src/backend/cpu/platform/BasicCpuInfo.cpp

@@ -5,8 +5,8 @@
  * Copyright 2014-2016 Wolf9466    <https://github.com/OhGodAPet>
  * Copyright 2016      Jay D Dee   <jayddee246@gmail.com>
  * Copyright 2017-2019 XMR-Stak    <https://github.com/fireice-uk>, <https://github.com/psychocrypt>
- * Copyright 2018-2019 SChernykh   <https://github.com/SChernykh>
+ * Copyright 2018-2020 SChernykh   <https://github.com/SChernykh>
- * Copyright 2016-2019 XMRig       <support@xmrig.com>
+ * Copyright 2016-2020 XMRig       <support@xmrig.com>
  *
  *   This program is free software: you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -23,7 +23,7 @@
  */
 
 #include <algorithm>
-#include <string.h>
+#include <cstring>
 #include <thread>
 
 
@@ -45,6 +45,14 @@
 #   define bit_AVX2 (1 << 5)
 #endif
 
+#ifndef bit_BMI2
+#   define bit_BMI2 (1 << 8)
+#endif
+
+#ifndef bit_PDPE1GB
+#   define bit_PDPE1GB (1 << 26)
+#endif
+
 
 #include "backend/cpu/platform/BasicCpuInfo.h"
 #include "crypto/common/Assembly.h"
@@ -53,6 +61,7 @@
 #define VENDOR_ID                  (0)
 #define PROCESSOR_INFO             (1)
 #define EXTENDED_FEATURES          (7)
+#define PROCESSOR_EXT_INFO         (0x80000001)
 #define PROCESSOR_BRAND_STRING_1   (0x80000002)
 #define PROCESSOR_BRAND_STRING_2   (0x80000003)
 #define PROCESSOR_BRAND_STRING_3   (0x80000004)
@@ -108,7 +117,7 @@ static void cpu_brand_string(char out[64 + 6]) {
 }
 
 
-static bool has_feature(uint32_t level, uint32_t reg, int32_t bit)
+static inline bool has_feature(uint32_t level, uint32_t reg, int32_t bit)
 {
     int32_t cpu_info[4] = { 0 };
     cpuid(level, cpu_info);
@@ -136,15 +145,27 @@ static inline bool has_avx2()
 }
 
 
+static inline bool has_bmi2()
+{
+    return has_feature(EXTENDED_FEATURES, EBX_Reg, bit_BMI2);
+}
+
+
+static inline bool has_pdpe1gb()
+{
+    return has_feature(PROCESSOR_EXT_INFO, EDX_Reg, bit_PDPE1GB);
+}
+
+
 } // namespace xmrig
 
 
 xmrig::BasicCpuInfo::BasicCpuInfo() :
-    m_brand(),
     m_threads(std::thread::hardware_concurrency()),
-    m_assembly(Assembly::NONE),
     m_aes(has_aes_ni()),
-    m_avx2(has_avx2())
+    m_avx2(has_avx2()),
+    m_bmi2(has_bmi2()),
+    m_pdpe1gb(has_pdpe1gb())
 {
     cpu_brand_string(m_brand);
 
@@ -160,13 +181,23 @@ xmrig::BasicCpuInfo::BasicCpuInfo() :
         memcpy(vendor + 8, &data[2], 4);
 
         if (memcmp(vendor, "AuthenticAMD", 12) == 0) {
+            m_vendor = VENDOR_AMD;
+
             cpuid(PROCESSOR_INFO, data);
             const int32_t family = get_masked(data[EAX_Reg], 12, 8) + get_masked(data[EAX_Reg], 28, 20);
 
-            m_assembly = family >= 23 ? Assembly::RYZEN : Assembly::BULLDOZER;
+            if (family >= 23) {
+                m_assembly = Assembly::RYZEN;
+                m_msrMod   = MSR_MOD_RYZEN;
+            }
+            else {
+                m_assembly = Assembly::BULLDOZER;
+            }
         }
-        else {
+        else if (memcmp(vendor, "GenuineIntel", 12) == 0) {
+            m_vendor   = VENDOR_INTEL;
             m_assembly = Assembly::INTEL;
+            m_msrMod   = MSR_MOD_INTEL;
         }
     }
 #   endif
@@ -179,7 +210,7 @@ const char *xmrig::BasicCpuInfo::backend() const
 }
 
 
-xmrig::CpuThreads xmrig::BasicCpuInfo::threads(const Algorithm &algorithm, uint32_t limit) const
+xmrig::CpuThreads xmrig::BasicCpuInfo::threads(const Algorithm &algorithm, uint32_t) const
 {
     const size_t count = std::thread::hardware_concurrency();
 
@@ -227,5 +258,15 @@ xmrig::CpuThreads xmrig::BasicCpuInfo::threads(const Algorithm &algorithm, uint3
     }
 #   endif
 
+#   ifdef XMRIG_ALGO_ASTROBWT
+    if (algorithm.family() == Algorithm::ASTROBWT) {
+        CpuThreads threads;
+        for (size_t i = 0; i < count; ++i) {
+            threads.add(i, 0);
+        }
+        return threads;
+    }
+#   endif
+
     return CpuThreads(std::max<size_t>(count / 2, 1), 1);
 }