Stockfish 16.1

Official release version of Stockfish 16.1

Bench: 1303971

---

Stockfish 16.1

Today, we have the pleasure to announce Stockfish 16.1. As always, you can
freely download it at https://stockfishchess.org/download and use it in the GUI
of your choice[1].

Don't forget to join our Discord server[2] to get in touch with the community of
developers and users of the project!

*Quality of chess play*

In our testing against its predecessor, Stockfish 16.1 shows a notable
improvement in performance, with an Elo gain of up to 27 points and winning over
2 times more game pairs[3] than it loses.

*Update highlights*

*Improved evaluation*

- Updated neural network architecture: The neural network architecture has
  undergone two updates and is currently in its 8th version[4].
- Removal of handcrafted evaluation (HCE): This release marks the removal of the
  traditional handcrafted evaluation and the transition to a fully neural
  network-based approach[5].
- Dual NNUE: For the first time, Stockfish includes a secondary neural
  network[6], used to quickly evaluate positions that are easily decided.

*UCI Options removed*

`Use NNUE` and `UCI_AnalyseMode`[7] have been removed as they no longer had any
effect. `SlowMover`[8] has also been removed in favor of `Move Overhead`.

*More binaries*

We now offer 13 new binaries. These new binaries include `avx512`, `vnni256`,
`vnni512`, `m1-apple-silicon`, and `armv8-dotprod`, which take advantage of
specific CPU instructions for improved performance.
For most users, using `sse41-popcnt` (formerly `modern`), `avx2`, or `bmi2`
should be enough, but if your CPU supports these new instructions, feel free to
try them!

*Development changes*

- Updated testing book: This new book[9], now derived exclusively from the open
  Lichess database[10], is 10 times larger than its predecessor, and has been
  used to test potential improvements to Stockfish over the past few months.
- Consolidation of repositories: Aiming to simplify access to our resources, we
  have moved most Stockfish-related repositories into the official Stockfish
  organization[11] on GitHub.
- Growing maintainer team: We welcome Disservin[12] to the team of maintainers
  of the project! This extra pair of hands will ensure the lasting success of
  Stockfish.

*Thank you*

The Stockfish project builds on a thriving community of enthusiasts (thanks
everybody!) who contribute their expertise, time, and resources to build a free
and open-source chess engine that is robust, widely available, and very strong.

We would like to express our gratitude for the 10k stars[13] that light up our
GitHub project! Thank you for your support and encouragement – your recognition
means a lot to us.

We invite our chess fans to join the Fishtest testing framework[14], and
programmers to contribute to the project either directly to Stockfish[15] (C++),
to Fishtest[16] (HTML, CSS, JavaScript, and Python), to our trainer
nnue-pytorch[17] (C++ and Python), or to our website[18] (HTML, CSS/SCSS, and
JavaScript).

The Stockfish team

[1] https://github.com/official-stockfish/Stockfish/wiki/Download-and-usage#download-a-chess-gui
[2] https://discord.gg/GWDRS3kU6R
[3] https://tests.stockfishchess.org/tests/view/65d666051d8e83c78bfddbd8
[4] https://github.com/official-stockfish/nnue-pytorch/blob/master/docs/nnue.md#sfnnv8-architecture
[5] https://github.com/official-stockfish/Stockfish/commit/af110e0
[6] https://github.com/official-stockfish/Stockfish/commit/584d9ef
[7] https://github.com/official-stockfish/Stockfish/commit/c53d2ec
[8] https://github.com/official-stockfish/Stockfish/commit/536d692
[9] https://github.com/official-stockfish/books/commit/426eca4
[10] https://database.lichess.org/
[11] https://github.com/official-stockfish/
[12] https://github.com/Disservin
[13] https://github.com/official-stockfish/Stockfish/stargazers
[14] https://github.com/official-stockfish/fishtest/wiki/Running-the-worker
[15] https://github.com/official-stockfish/Stockfish
[16] https://github.com/official-stockfish/fishtest
[17] https://github.com/official-stockfish/nnue-pytorch
[18] https://github.com/official-stockfish/stockfish-web

.clang-format

@@ -0,0 +1,44 @@
+AccessModifierOffset: -1
+AlignAfterOpenBracket: Align
+AlignConsecutiveAssignments: Consecutive
+AlignConsecutiveDeclarations: Consecutive
+AlignEscapedNewlines: DontAlign
+AlignOperands: AlignAfterOperator
+AlignTrailingComments: true
+AllowAllParametersOfDeclarationOnNextLine: true
+AllowShortCaseLabelsOnASingleLine: false
+AllowShortEnumsOnASingleLine: false
+AllowShortIfStatementsOnASingleLine: false
+AlwaysBreakTemplateDeclarations: Yes
+BasedOnStyle: WebKit
+BitFieldColonSpacing: After
+BinPackParameters: false
+BreakBeforeBinaryOperators: NonAssignment
+BreakBeforeBraces: Custom
+BraceWrapping:
+  AfterFunction: false 
+  AfterClass: false
+  AfterControlStatement: true
+  BeforeElse: true
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializers: AfterColon
+BreakStringLiterals: false
+ColumnLimit: 100
+ContinuationIndentWidth: 2
+Cpp11BracedListStyle: true
+IndentGotoLabels: false
+IndentPPDirectives: BeforeHash
+IndentWidth: 4
+MaxEmptyLinesToKeep: 2
+NamespaceIndentation: None
+PackConstructorInitializers: Never
+ReflowComments: false
+SortIncludes: false
+SortUsingDeclarations: false
+SpaceAfterCStyleCast: true
+SpaceAfterTemplateKeyword: false
+SpaceBeforeCaseColon: true
+SpaceBeforeCpp11BracedList: false
+SpaceBeforeInheritanceColon: false
+SpaceInEmptyBlock: false
+SpacesBeforeTrailingComments: 2

.git-blame-ignore-revs

@@ -0,0 +1,7 @@
+# .git-blame-ignore-revs
+# Ignore commit which added clang-format
+2d0237db3f0e596fb06e3ffbadba84dcc4e018f6
+
+# Post commit formatting fixes
+0fca5605fa2e5e7240fde5e1aae50952b2612231
+08ed4c90db31959521b7ef3186c026edd1e90307

.github/ISSUE_TEMPLATE/BUG-REPORT.yml

@@ -0,0 +1,65 @@
+name: Report issue
+description: Create a report to help us fix issues with the engine
+body:
+- type: textarea
+  attributes:
+    label: Describe the issue
+    description: A clear and concise description of what you're experiencing.
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Expected behavior
+    description: A clear and concise description of what you expected to happen.
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Steps to reproduce
+    description: |
+      Steps to reproduce the behavior.
+      You can also use this section to paste the command line output.
+    placeholder: |
+      ```
+      position startpos moves g2g4 e7e5 f2f3
+      go mate 1
+      info string NNUE evaluation using nn-6877cd24400e.nnue enabled
+      info depth 1 seldepth 1 multipv 1 score mate 1 nodes 33 nps 11000 tbhits 0 time 3 pv d8h4
+      bestmove d8h4
+      ```
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Anything else?
+    description: |
+      Anything that will give us more context about the issue you are encountering.
+      You can also use this section to propose ideas on how to solve the issue. 
+  validations:
+    required: false
+
+- type: dropdown
+  attributes:
+    label: Operating system
+    options:
+      - All
+      - Windows
+      - Linux
+      - MacOS
+      - Android
+      - Other or N/A
+  validations:
+    required: true
+
+- type: input
+  attributes:
+    label: Stockfish version
+    description: |
+      This can be found by running the engine.
+      You can also use the commit ID.
+    placeholder: Stockfish 15 / e6e324e
+  validations:
+    required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Discord server
+    url: https://discord.gg/GWDRS3kU6R
+    about: Feel free to ask for support or have a chat with us on our Discord server!
+  - name: Discussions, Q&A, ideas, show us something...
+    url: https://github.com/official-stockfish/Stockfish/discussions/new
+    about: Do you have an idea for Stockfish? Do you want to show something that you made? Please open a discussion about it!

.github/ci/arm_matrix.json

@@ -0,0 +1,51 @@
+{
+  "config": [
+    {
+      "name": "Android NDK aarch64",
+      "os": "ubuntu-22.04",
+      "simple_name": "android",
+      "compiler": "aarch64-linux-android21-clang++",
+      "emu": "qemu-aarch64",
+      "comp": "ndk",
+      "shell": "bash",
+      "archive_ext": "tar"
+    },
+    {
+      "name": "Android NDK arm",
+      "os": "ubuntu-22.04",
+      "simple_name": "android",
+      "compiler": "armv7a-linux-androideabi21-clang++",
+      "emu": "qemu-arm",
+      "comp": "ndk",
+      "shell": "bash",
+      "archive_ext": "tar"
+    }
+  ],
+  "binaries": ["armv8-dotprod", "armv8", "armv7", "armv7-neon"],
+  "exclude": [
+    {
+      "binaries": "armv8-dotprod",
+      "config": {
+        "compiler": "armv7a-linux-androideabi21-clang++"
+      }
+    },
+    {
+      "binaries": "armv8",
+      "config": {
+        "compiler": "armv7a-linux-androideabi21-clang++"
+      }
+    },
+    {
+      "binaries": "armv7",
+      "config": {
+        "compiler": "aarch64-linux-android21-clang++"
+      }
+    },
+    {
+      "binaries": "armv7-neon",
+      "config": {
+        "compiler": "aarch64-linux-android21-clang++"
+      }
+    }
+  ]
+}

.github/ci/libcxx17.imp

@@ -0,0 +1,21 @@
+[
+    # Mappings for libcxx's internal headers
+    { include: [ "<__fwd/fstream.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/ios.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/istream.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/ostream.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/sstream.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/streambuf.h>", private, "<iosfwd>", public ] },
+    { include: [ "<__fwd/string_view.h>", private, "<string_view>", public ] },
+
+    # Mappings for includes between public headers
+    { include: [ "<ios>", public, "<iostream>", public ] },
+    { include: [ "<streambuf>", public, "<iostream>", public ] },
+    { include: [ "<istream>", public, "<iostream>", public ] },
+    { include: [ "<ostream>", public, "<iostream>", public ] },
+    { include: [ "<iosfwd>", public, "<iostream>", public ] },
+
+    # Missing mappings in include-what-you-use's libcxx.imp
+    { include: ["@<__condition_variable/.*>", private, "<condition_variable>", public ] },
+    { include: ["@<__mutex/.*>", private, "<mutex>", public ] },
+]

.github/ci/matrix.json

@@ -0,0 +1,160 @@
+{
+  "config": [
+    {
+      "name": "Ubuntu 20.04 GCC",
+      "os": "ubuntu-20.04",
+      "simple_name": "ubuntu",
+      "compiler": "g++",
+      "comp": "gcc",
+      "shell": "bash",
+      "archive_ext": "tar",
+      "sde": "/home/runner/work/Stockfish/Stockfish/.output/sde-temp-files/sde-external-9.27.0-2023-09-13-lin/sde -future --"
+    },
+    {
+      "name": "MacOS 13 Apple Clang",
+      "os": "macos-13",
+      "simple_name": "macos",
+      "compiler": "clang++",
+      "comp": "clang",
+      "shell": "bash",
+      "archive_ext": "tar"
+    },
+    {
+      "name": "MacOS 14 Apple Clang M1",
+      "os": "macos-14",
+      "simple_name": "macos-m1",
+      "compiler": "clang++",
+      "comp": "clang",
+      "shell": "bash",
+      "archive_ext": "tar"
+    },
+    {
+      "name": "Windows 2022 Mingw-w64 GCC x86_64",
+      "os": "windows-2022",
+      "simple_name": "windows",
+      "compiler": "g++",
+      "comp": "mingw",
+      "msys_sys": "mingw64",
+      "msys_env": "x86_64-gcc",
+      "shell": "msys2 {0}",
+      "ext": ".exe",
+      "sde": "/d/a/Stockfish/Stockfish/.output/sde-temp-files/sde-external-9.27.0-2023-09-13-win/sde.exe -future --",
+      "archive_ext": "zip"
+    }
+  ],
+  "binaries": [
+    "x86-64",
+    "x86-64-sse41-popcnt",
+    "x86-64-avx2",
+    "x86-64-bmi2",
+    "x86-64-avxvnni",
+    "x86-64-avx512",
+    "x86-64-vnni256",
+    "x86-64-vnni512",
+    "apple-silicon"
+  ],
+  "exclude": [
+    {
+      "binaries": "x86-64",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-sse41-popcnt",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-avx2",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-bmi2",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-avxvnni",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-avxvnni",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-avx512",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-vnni256",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-vnni512",
+      "config": {
+        "os": "macos-14"
+      }
+    },
+    {
+      "binaries": "x86-64-avxvnni",
+      "config": {
+        "ubuntu-20.04": null
+      }
+    },
+    {
+      "binaries": "x86-64-avxvnni",
+      "config": {
+        "os": "macos-13"
+      }
+    },
+    {
+      "binaries": "x86-64-avx512",
+      "config": {
+        "os": "macos-13"
+      }
+    },
+    {
+      "binaries": "x86-64-vnni256",
+      "config": {
+        "os": "macos-13"
+      }
+    },
+    {
+      "binaries": "x86-64-vnni512",
+      "config": {
+        "os": "macos-13"
+      }
+    },
+    {
+      "binaries": "apple-silicon",
+      "config": {
+        "os": "windows-2022"
+      }
+    },
+    {
+      "binaries": "apple-silicon",
+      "config": {
+        "os": "macos-13"
+      }
+    },
+    {
+      "binaries": "apple-silicon",
+      "config": {
+        "os": "ubuntu-20.04"
+      }
+    }
+  ]
+}

.github/workflows/arm_compilation.yml

@@ -0,0 +1,94 @@
+name: Compilation
+on:
+  workflow_call:
+    inputs:
+      matrix:
+        type: string
+        required: true
+jobs:
+  Compilation:
+    name: ${{ matrix.config.name }} ${{ matrix.binaries }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      COMPILER: ${{ matrix.config.compiler }}
+      COMP: ${{ matrix.config.comp }}
+      EMU: ${{ matrix.config.emu }}
+      EXT: ${{ matrix.config.ext }}
+      BINARY: ${{ matrix.binaries }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(inputs.matrix) }}
+    defaults:
+      run:
+        working-directory: src
+        shell: ${{ matrix.config.shell }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Download required linux packages
+        if: runner.os == 'Linux'
+        run: |
+          sudo apt update
+          sudo apt install qemu-user
+
+      - name: Install NDK
+        if: runner.os == 'Linux'
+        run: |
+          if [ $COMP == ndk ]; then
+            NDKV="21.4.7075529"
+            ANDROID_ROOT=/usr/local/lib/android
+            ANDROID_SDK_ROOT=$ANDROID_ROOT/sdk
+            SDKMANAGER=$ANDROID_SDK_ROOT/cmdline-tools/latest/bin/sdkmanager
+            echo "y" | $SDKMANAGER "ndk;$NDKV"
+            ANDROID_NDK_ROOT=$ANDROID_SDK_ROOT/ndk/$NDKV
+            ANDROID_NDK_BIN=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin
+            echo "ANDROID_NDK_BIN=$ANDROID_NDK_BIN" >> $GITHUB_ENV
+          fi
+
+      - name: Extract the bench number from the commit history
+        run: |
+          for hash in $(git rev-list -100 HEAD); do
+            benchref=$(git show -s $hash | tac | grep -m 1 -o -x '[[:space:]]*\b[Bb]ench[ :]\+[1-9][0-9]\{5,7\}\b[[:space:]]*' | sed 's/[^0-9]//g') && break || true
+          done
+          [[ -n "$benchref" ]] && echo "benchref=$benchref" >> $GITHUB_ENV && echo "From commit: $hash" && echo "Reference bench: $benchref" || echo "No bench found"
+
+      - name: Download the used network from the fishtest framework
+        run: make net
+
+      - name: Check compiler
+        run: |
+          if [ $COMP == ndk ]; then
+            export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+          fi
+          $COMPILER -v
+
+      - name: Test help target
+        run: make help
+
+      - name: Check git
+        run: git --version
+
+      # Compile profile guided builds
+
+      - name: Compile ${{ matrix.binaries }} build
+        run: |
+          if [ $COMP == ndk ]; then
+            export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+            export LDFLAGS="-static -Wno-unused-command-line-argument"
+          fi
+          make clean
+          make -j4 profile-build ARCH=$BINARY COMP=$COMP WINE_PATH=$EMU
+          make strip ARCH=$BINARY COMP=$COMP
+          WINE_PATH=$EMU ../tests/signature.sh $benchref
+          mv ./stockfish$EXT ../stockfish-android-$BINARY$EXT
+
+      - name: Remove non src files
+        run: git clean -fx
+
+      - name: Upload artifact for (pre)-release
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ matrix.config.simple_name }} ${{ matrix.binaries }}
+          path: .

.github/workflows/clang-format.yml

@@ -0,0 +1,51 @@
+# This workflow will run clang-format and comment on the PR.
+# Because of security reasons, it is crucial that this workflow
+# executes no shell script nor runs make.
+# Read this before editing: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
+
+name: Clang-Format
+on:
+  pull_request_target:
+    branches:
+      - "master"
+    paths:
+      - "**.cpp"
+      - "**.h"
+jobs:
+  Clang-Format:
+    name: Clang-Format
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: Run clang-format style check
+        uses: jidicula/clang-format-action@f62da5e3d3a2d88ff364771d9d938773a618ab5e # @v4.11.0
+        id: clang-format
+        continue-on-error: true
+        with:
+          clang-format-version: "17"
+          exclude-regex: "incbin"
+
+      - name: Comment on PR
+        if: steps.clang-format.outcome == 'failure'
+        uses: thollander/actions-comment-pull-request@1d3973dc4b8e1399c0620d3f2b1aa5e795465308 # @v2.4.3
+        with:
+          message: |
+            clang-format 17 needs to be run on this PR.
+            If you do not have clang-format installed, the maintainer will run it when merging.
+            For the exact version please see https://packages.ubuntu.com/mantic/clang-format-17.
+
+            _(execution **${{ github.run_id }}** / attempt **${{ github.run_attempt }}**)_
+          comment_tag: execution
+
+      - name: Comment on PR
+        if: steps.clang-format.outcome != 'failure'
+        uses: thollander/actions-comment-pull-request@1d3973dc4b8e1399c0620d3f2b1aa5e795465308 # @v2.4.3
+        with:
+          message: |
+            _(execution **${{ github.run_id }}** / attempt **${{ github.run_attempt }}**)_
+          create_if_not_exists: false
+          comment_tag: execution
+          mode: delete

.github/workflows/codeql.yml

@@ -0,0 +1,53 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: ["master"]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: ["master"]
+  schedule:
+    - cron: "17 18 * * 1"
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: ["cpp"]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Use only 'java' to analyze code written in Java, Kotlin, or both
+        # Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
+        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      # Initializes the CodeQL tools for scanning.
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v3
+        with:
+          languages: ${{ matrix.language }}
+          # If you wish to specify custom queries, you can do so here or in a config file.
+          # By default, queries listed here will override any specified in a config file.
+          # Prefix the list here with "+" to use these queries and those in the config file.
+
+          # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+          # queries: security-extended,security-and-quality
+
+      - name: Build
+        working-directory: src
+        run: make -j build ARCH=x86-64-modern
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v3
+        with:
+          category: "/language:${{matrix.language}}"

.github/workflows/compilation.yml

@@ -0,0 +1,89 @@
+name: Compilation
+on:
+  workflow_call:
+    inputs:
+      matrix:
+        type: string
+        required: true
+jobs:
+  Compilation:
+    name: ${{ matrix.config.name }} ${{ matrix.binaries }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      COMPILER: ${{ matrix.config.compiler }}
+      COMP: ${{ matrix.config.comp }}
+      EXT: ${{ matrix.config.ext }}
+      NAME: ${{ matrix.config.simple_name }}
+      BINARY: ${{ matrix.binaries }}
+      SDE: ${{ matrix.config.sde }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(inputs.matrix) }}
+    defaults:
+      run:
+        working-directory: src
+        shell: ${{ matrix.config.shell }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install fixed GCC on Linux
+        if: runner.os == 'Linux'
+        uses: egor-tensin/setup-gcc@eaa888eb19115a521fa72b65cd94fe1f25bbcaac # @v1.3
+        with:
+          version: 11
+
+      - name: Setup msys and install required packages
+        if: runner.os == 'Windows'
+        uses: msys2/setup-msys2@v2
+        with:
+          msystem: ${{ matrix.config.msys_sys }}
+          install: mingw-w64-${{ matrix.config.msys_env }} make git zip
+
+      - name: Download SDE package
+        if: runner.os == 'Linux' || runner.os == 'Windows'
+        uses: petarpetrovt/setup-sde@91a1a03434384e064706634125a15f7446d2aafb # @v2.3
+        with:
+          environmentVariableName: SDE_DIR
+          sdeVersion: 9.27.0
+
+      - name: Download the used network from the fishtest framework
+        run: make net
+
+      - name: Check compiler
+        run: $COMPILER -v
+
+      - name: Test help target
+        run: make help
+
+      - name: Check git
+        run: git --version
+
+      - name: Check compiler
+        run: $COMPILER -v
+
+      - name: Show g++ cpu info
+        if: runner.os != 'macOS'
+        run: g++ -Q -march=native --help=target
+
+      - name: Show clang++ cpu info
+        if: runner.os == 'macOS'
+        run: clang++ -E - -march=native -###
+
+      # x86-64 with newer extensions tests
+
+      - name: Compile ${{ matrix.config.binaries }} build
+        run: |
+          make clean
+          make -j4 profile-build ARCH=$BINARY COMP=$COMP WINE_PATH="$SDE"
+          make strip ARCH=$BINARY COMP=$COMP
+          WINE_PATH="$SDE" ../tests/signature.sh $benchref
+          mv ./stockfish$EXT ../stockfish-$NAME-$BINARY$EXT
+
+      - name: Remove non src files
+        run: git clean -fx
+
+      - name: Upload artifact for (pre)-release
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ matrix.config.simple_name }} ${{ matrix.binaries }}
+          path: .

.github/workflows/iwyu.yml

@@ -0,0 +1,47 @@
+name: IWYU
+on:
+  workflow_call:
+jobs:
+  Analyzers:
+    name: Check includes
+    runs-on: ubuntu-22.04
+    defaults:
+      run:
+        working-directory: Stockfish/src
+        shell: bash
+    steps:
+      - name: Checkout Stockfish
+        uses: actions/checkout@v4
+        with:
+          path: Stockfish
+
+      - name: Checkout include-what-you-use
+        uses: actions/checkout@v4
+        with:
+          repository: include-what-you-use/include-what-you-use
+          ref: f25caa280dc3277c4086ec345ad279a2463fea0f
+          path: include-what-you-use
+
+      - name: Download required linux packages
+        run: |
+          sudo add-apt-repository 'deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-17 main'
+          wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
+          sudo apt update
+          sudo apt install -y libclang-17-dev clang-17 libc++-17-dev
+
+      - name: Set up include-what-you-use
+        run: |
+          mkdir build && cd build
+          cmake -G "Unix Makefiles" -DCMAKE_PREFIX_PATH="/usr/lib/llvm-17" ..
+          sudo make install
+        working-directory: include-what-you-use
+
+      - name: Check include-what-you-use
+        run: include-what-you-use --version
+
+      - name: Check includes
+        run: >
+          make analyze
+          COMP=clang
+          CXX=include-what-you-use
+          CXXFLAGS="-stdlib=libc++ -Xiwyu --comment_style=long -Xiwyu --mapping='${{ github.workspace }}/Stockfish/.github/ci/libcxx17.imp' -Xiwyu --error"

.github/workflows/sanitizers.yml

@@ -0,0 +1,65 @@
+name: Sanitizers
+on:
+  workflow_call:
+jobs:
+  Test-under-sanitizers:
+    name: ${{ matrix.sanitizers.name }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      COMPILER: ${{ matrix.config.compiler }}
+      COMP: ${{ matrix.config.comp }}
+      CXXFLAGS: "-Werror"
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - name: Ubuntu 22.04 GCC
+            os: ubuntu-22.04
+            compiler: g++
+            comp: gcc
+            shell: bash
+        sanitizers:
+          - name: Run with thread sanitizer
+            make_option: sanitize=thread
+            instrumented_option: sanitizer-thread
+          - name: Run with UB sanitizer
+            make_option: sanitize=undefined
+            instrumented_option: sanitizer-undefined
+          - name: Run under valgrind
+            make_option: ""
+            instrumented_option: valgrind
+          - name: Run under valgrind-thread
+            make_option: ""
+            instrumented_option: valgrind-thread
+    defaults:
+      run:
+        working-directory: src
+        shell: ${{ matrix.config.shell }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Download required linux packages
+        run: |
+          sudo apt update
+          sudo apt install expect valgrind g++-multilib
+
+      - name: Download the used network from the fishtest framework
+        run: make net
+
+      - name: Check compiler
+        run: $COMPILER -v
+
+      - name: Test help target
+        run: make help
+
+      - name: Check git
+        run: git --version
+
+      # Sanitizers
+
+      - name: ${{ matrix.sanitizers.name }}
+        run: |
+          export CXXFLAGS="-O1 -fno-inline"
+          make clean
+          make -j4 ARCH=x86-64-sse41-popcnt ${{ matrix.sanitizers.make_option }} debug=yes optimize=no build > /dev/null
+          ../tests/instrumented.sh --${{ matrix.sanitizers.instrumented_option }}

.github/workflows/stockfish.yml

@@ -1,260 +1,76 @@
 name: Stockfish
 on:
   push:
+    tags:
+      - "*"
     branches:
       - master
       - tools
       - github_ci
-      - github_ci_armv7
   pull_request:
     branches:
       - master
       - tools
 jobs:
-  Stockfish:
+  Prerelease:
-    name: ${{ matrix.config.name }}
+    if: github.repository == 'official-stockfish/Stockfish' && (github.ref == 'refs/heads/master' || (startsWith(github.ref_name, 'sf_') && github.ref_type == 'tag'))
-    runs-on: ${{ matrix.config.os }}
+    runs-on: ubuntu-latest
-    env:
-      COMPILER: ${{ matrix.config.compiler }}
-      COMP: ${{ matrix.config.comp }}
-    strategy:
-      matrix:
-        config:
-          # set the variable for the required tests:
-          # run_expensive_tests: true
-          # run_32bit_tests: true
-          # run_64bit_tests: true
-          - {
-              name: "Ubuntu 20.04 GCC",
-              os: ubuntu-20.04,
-              compiler: g++,
-              comp: gcc,
-              run_expensive_tests: true,
-              run_64bit_tests: true,
-              shell: 'bash {0}'
-            }
-          - {
-              name: "Ubuntu 20.04 Clang",
-              os: ubuntu-20.04,
-              compiler: clang++,
-              comp: clang,
-              run_64bit_tests: true,
-              shell: 'bash {0}'
-            }
-          - {
-              name: "MacOS 10.15 Apple Clang",
-              os: macos-10.15,
-              compiler: clang++,
-              comp: clang,
-              run_64bit_tests: true,
-              shell: 'bash {0}'
-            }
-          - {
-              name: "MacOS 10.15 GCC 10",
-              os: macos-10.15,
-              compiler: g++-10,
-              comp: gcc,
-              run_64bit_tests: true,
-              shell: 'bash {0}'
-            }
-          - {
-              name: "Windows 2022 Mingw-w64 GCC x86_64",
-              os: windows-2022,
-              compiler: g++,
-              comp: mingw,
-              run_64bit_tests: true,
-              msys_sys: 'mingw64',
-              msys_env: 'x86_64-gcc',
-              shell: 'msys2 {0}'
-            }
-
-    defaults:
-      run:
-        working-directory: src
-        shell: ${{ matrix.config.shell }}
     steps:
-      - uses: actions/checkout@v2
+      # returns null if no pre-release exists
-        with:
+      - name: Get Commit SHA of Latest Pre-release
-          fetch-depth: 0
-
-      - name: Download required linux packages
-        if: runner.os == 'Linux'
         run: |
-          sudo apt update
+          # Install required packages
-          sudo apt install expect valgrind g++-multilib
+          sudo apt-get update
+          sudo apt-get install -y curl jq
 
-      - name: Setup msys and install required packages
+          echo "COMMIT_SHA=$(jq -r 'map(select(.prerelease)) | first | .tag_name' <<< $(curl -s https://api.github.com/repos/${{ github.repository_owner }}/Stockfish/releases))" >> $GITHUB_ENV
-        if: runner.os == 'Windows'
-        uses: msys2/setup-msys2@v2
-        with:
-          msystem: ${{matrix.config.msys_sys}}
-          install: mingw-w64-${{matrix.config.msys_env}} make git expect
 
-      - name: Download the used network from the fishtest framework
+        # delete old previous pre-release and tag
+      - uses: actions/checkout@v4
+      - run: gh release delete ${{ env.COMMIT_SHA }} --cleanup-tag
+        if: env.COMMIT_SHA != 'null'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  Matrix:
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+      arm_matrix: ${{ steps.set-arm-matrix.outputs.arm_matrix }}
+    steps:
+      - uses: actions/checkout@v4
+      - id: set-matrix
         run: |
-          make net
+          TASKS=$(echo $(cat .github/ci/matrix.json) )
-
+          echo "MATRIX=$TASKS" >> $GITHUB_OUTPUT
-      - name: Extract the bench number from the commit history
+      - id: set-arm-matrix
         run: |
-          git log HEAD | grep "\b[Bb]ench[ :]\+[0-9]\{7\}" | head -n 1 | sed "s/[^0-9]*\([0-9]*\).*/\1/g" > git_sig
+          TASKS_ARM=$(echo $(cat .github/ci/arm_matrix.json) )
-          [ -s git_sig ] && echo "benchref=$(cat git_sig)" >> $GITHUB_ENV && echo "Reference bench:" $(cat git_sig) || echo "No bench found"
+          echo "ARM_MATRIX=$TASKS_ARM" >> $GITHUB_OUTPUT
-
+  Compilation:
-      - name: Check compiler
+    needs: [Matrix]
-        run: |
+    uses: ./.github/workflows/compilation.yml
-          $COMPILER -v
+    with:
-
+      matrix: ${{ needs.Matrix.outputs.matrix }}
-      - name: Test help target
+  ARMCompilation:
-        run: |
+    needs: [Matrix]
-          make help
+    uses: ./.github/workflows/arm_compilation.yml
-
+    with:
-      # x86-32 tests
+      matrix: ${{ needs.Matrix.outputs.arm_matrix }}
-
+  IWYU:
-      - name: Test debug x86-32 build
+    uses: ./.github/workflows/iwyu.yml
-        if: ${{ matrix.config.run_32bit_tests }}
+  Sanitizers:
-        run: |
+    uses: ./.github/workflows/sanitizers.yml
-          export CXXFLAGS="-D_GLIBCXX_DEBUG"
+  Tests:
-          make clean
+    uses: ./.github/workflows/tests.yml
-          make -j2 ARCH=x86-32 optimize=no debug=yes build
+  Binaries:
-          ../tests/signature.sh $benchref
+    if: github.repository == 'official-stockfish/Stockfish'
-
+    needs: [Matrix, Prerelease, Compilation]
-      - name: Test x86-32 build
+    uses: ./.github/workflows/upload_binaries.yml
-        if: ${{ matrix.config.run_32bit_tests }}
+    with:
-        run: |
+      matrix: ${{ needs.Matrix.outputs.matrix }}
-          make clean
+  ARM_Binaries:
-          make -j2 ARCH=x86-32 build
+    if: github.repository == 'official-stockfish/Stockfish'
-          ../tests/signature.sh $benchref
+    needs: [Matrix, Prerelease, ARMCompilation]
-
+    uses: ./.github/workflows/upload_binaries.yml
-      - name: Test x86-32-sse41-popcnt build
+    with:
-        if: ${{ matrix.config.run_32bit_tests }}
+      matrix: ${{ needs.Matrix.outputs.arm_matrix }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-32-sse41-popcnt build
-          ../tests/signature.sh $benchref
-
-      - name: Test x86-32-sse2 build
-        if: ${{ matrix.config.run_32bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-32-sse2 build
-          ../tests/signature.sh $benchref
-
-      - name: Test general-32 build
-        if: ${{ matrix.config.run_32bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=general-32 build
-          ../tests/signature.sh $benchref
-
-      # x86-64 tests
-
-      - name: Test debug x86-64-modern build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          export CXXFLAGS="-D_GLIBCXX_DEBUG"
-          make clean
-          make -j2 ARCH=x86-64-modern optimize=no debug=yes build
-          ../tests/signature.sh $benchref
-
-      - name: Test x86-64-modern build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-modern build
-          ../tests/signature.sh $benchref
-
-      - name: Test x86-64-ssse3 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-ssse3 build
-          ../tests/signature.sh $benchref
-
-      - name: Test x86-64-sse3-popcnt build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-sse3-popcnt build
-          ../tests/signature.sh $benchref
-
-      - name: Test x86-64 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64 build
-          ../tests/signature.sh $benchref
-
-      - name: Test general-64 build
-        if: matrix.config.run_64bit_tests
-        run: |
-          make clean
-          make -j2 ARCH=general-64 build
-          ../tests/signature.sh $benchref
-
-      # x86-64 with newer extensions tests
-
-      - name: Compile x86-64-avx2 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-avx2 build
-
-      - name: Compile x86-64-bmi2 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-bmi2 build
-
-      - name: Compile x86-64-avx512 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-avx512 build
-
-      - name: Compile x86-64-vnni512 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-vnni512 build
-
-      - name: Compile x86-64-vnni256 build
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-vnni256 build
-
-      # Other tests
-
-      - name: Check perft and search reproducibility
-        if: ${{ matrix.config.run_64bit_tests }}
-        run: |
-          make clean
-          make -j2 ARCH=x86-64-modern build
-          ../tests/perft.sh
-          ../tests/reprosearch.sh
-
-      # Sanitizers
-
-      - name: Run under valgrind
-        if: ${{ matrix.config.run_expensive_tests }}
-        run: |
-          export CXXFLAGS="-O1 -fno-inline"
-          make clean
-          make -j2 ARCH=x86-64-modern debug=yes optimize=no build > /dev/null
-          ../tests/instrumented.sh --valgrind
-          ../tests/instrumented.sh --valgrind-thread
-
-      - name: Run with UB sanitizer
-        if: ${{ matrix.config.run_expensive_tests }}
-        run: |
-          export CXXFLAGS="-O1 -fno-inline"
-          make clean
-          make -j2 ARCH=x86-64-modern sanitize=undefined optimize=no debug=yes build > /dev/null
-          ../tests/instrumented.sh --sanitizer-undefined
-
-      - name: Run with thread sanitizer
-        if: ${{ matrix.config.run_expensive_tests }}
-        run: |
-          export CXXFLAGS="-O1 -fno-inline"
-          make clean
-          make -j2 ARCH=x86-64-modern sanitize=thread optimize=no debug=yes build > /dev/null
-          ../tests/instrumented.sh --sanitizer-thread

.github/workflows/tests.yml

@@ -0,0 +1,365 @@
+name: Tests
+on:
+  workflow_call:
+jobs:
+  Test-Targets:
+    name: ${{ matrix.config.name }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      COMPILER: ${{ matrix.config.compiler }}
+      COMP: ${{ matrix.config.comp }}
+      CXXFLAGS: "-Werror"
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - name: Ubuntu 20.04 GCC
+            os: ubuntu-20.04
+            compiler: g++
+            comp: gcc
+            run_32bit_tests: true
+            run_64bit_tests: true
+            shell: bash
+          - name: Ubuntu 20.04 Clang
+            os: ubuntu-20.04
+            compiler: clang++
+            comp: clang
+            run_32bit_tests: true
+            run_64bit_tests: true
+            shell: bash
+          - name: Android NDK aarch64
+            os: ubuntu-22.04
+            compiler: aarch64-linux-android21-clang++
+            comp: ndk
+            run_armv8_tests: true
+            shell: bash
+          - name: Android NDK arm
+            os: ubuntu-22.04
+            compiler: armv7a-linux-androideabi21-clang++
+            comp: ndk
+            run_armv7_tests: true
+            shell: bash
+          - name: Linux GCC riscv64
+            os: ubuntu-22.04
+            compiler: g++
+            comp: gcc
+            run_riscv64_tests: true
+            base_image: "riscv64/alpine:edge"
+            platform: linux/riscv64
+            shell: bash
+          - name: Linux GCC ppc64
+            os: ubuntu-22.04
+            compiler: g++
+            comp: gcc
+            run_ppc64_tests: true
+            base_image: "ppc64le/alpine:latest"
+            platform: linux/ppc64le
+            shell: bash
+          - name: MacOS 13 Apple Clang
+            os: macos-13
+            compiler: clang++
+            comp: clang
+            run_64bit_tests: true
+            shell: bash
+          - name: MacOS 14 Apple Clang M1
+            os: macos-14
+            compiler: clang++
+            comp: clang
+            run_64bit_tests: false
+            run_m1_tests: true
+            shell: bash
+          - name: MacOS 13 GCC 11
+            os: macos-13
+            compiler: g++-11
+            comp: gcc
+            run_64bit_tests: true
+            shell: bash
+          - name: Windows 2022 Mingw-w64 GCC x86_64
+            os: windows-2022
+            compiler: g++
+            comp: mingw
+            run_64bit_tests: true
+            msys_sys: mingw64
+            msys_env: x86_64-gcc
+            shell: msys2 {0}
+          - name: Windows 2022 Mingw-w64 GCC i686
+            os: windows-2022
+            compiler: g++
+            comp: mingw
+            run_32bit_tests: true
+            msys_sys: mingw32
+            msys_env: i686-gcc
+            shell: msys2 {0}
+          - name: Windows 2022 Mingw-w64 Clang x86_64
+            os: windows-2022
+            compiler: clang++
+            comp: clang
+            run_64bit_tests: true
+            msys_sys: clang64
+            msys_env: clang-x86_64-clang
+            shell: msys2 {0}
+    defaults:
+      run:
+        working-directory: src
+        shell: ${{ matrix.config.shell }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Download required linux packages
+        if: runner.os == 'Linux'
+        run: |
+          sudo apt update
+          sudo apt install expect valgrind g++-multilib qemu-user-static
+
+      - name: Install NDK
+        if: runner.os == 'Linux'
+        run: |
+          if [ $COMP == ndk ]; then
+            NDKV="21.4.7075529"
+            ANDROID_ROOT=/usr/local/lib/android
+            ANDROID_SDK_ROOT=$ANDROID_ROOT/sdk
+            SDKMANAGER=$ANDROID_SDK_ROOT/cmdline-tools/latest/bin/sdkmanager
+            echo "y" | $SDKMANAGER "ndk;$NDKV"
+            ANDROID_NDK_ROOT=$ANDROID_SDK_ROOT/ndk/$NDKV
+            ANDROID_NDK_BIN=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin
+            echo "ANDROID_NDK_BIN=$ANDROID_NDK_BIN" >> $GITHUB_ENV
+          fi
+
+      - name: Set up QEMU
+        if: matrix.config.base_image
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        if: matrix.config.base_image
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build Docker container
+        if: matrix.config.base_image
+        run: |
+          docker buildx build --load -t sf_builder - << EOF
+          FROM ${{ matrix.config.base_image }}
+          WORKDIR /app
+          RUN apk update && apk add make g++
+          CMD ["sh", "script.sh"]
+          EOF
+
+      - name: Download required macOS packages
+        if: runner.os == 'macOS'
+        run: brew install coreutils
+
+      - name: Setup msys and install required packages
+        if: runner.os == 'Windows'
+        uses: msys2/setup-msys2@v2
+        with:
+          msystem: ${{ matrix.config.msys_sys }}
+          install: mingw-w64-${{ matrix.config.msys_env }} make git expect
+
+      - name: Download the used network from the fishtest framework
+        run: make net
+
+      - name: Extract the bench number from the commit history
+        run: |
+          for hash in $(git rev-list -100 HEAD); do
+            benchref=$(git show -s $hash | tac | grep -m 1 -o -x '[[:space:]]*\b[Bb]ench[ :]\+[1-9][0-9]\{5,7\}\b[[:space:]]*' | sed 's/[^0-9]//g') && break || true
+          done
+          [[ -n "$benchref" ]] && echo "benchref=$benchref" >> $GITHUB_ENV && echo "From commit: $hash" && echo "Reference bench: $benchref" || echo "No bench found"
+
+      - name: Check compiler
+        run: |
+          if [ -z "${{ matrix.config.base_image }}" ]; then
+            if [ $COMP == ndk ]; then
+              export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+            fi
+            $COMPILER -v
+          else
+            echo "$COMPILER -v" > script.sh
+            docker run --rm --platform ${{ matrix.config.platform }} -v ${{ github.workspace }}/src:/app sf_builder
+          fi
+
+      - name: Test help target
+        run: make help
+
+      - name: Check git
+        run: git --version
+
+      # x86-32 tests
+
+      - name: Test debug x86-32 build
+        if: matrix.config.run_32bit_tests
+        run: |
+          export CXXFLAGS="-Werror -D_GLIBCXX_DEBUG"
+          make clean
+          make -j4 ARCH=x86-32 optimize=no debug=yes build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-32 build
+        if: matrix.config.run_32bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-32 build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-32-sse41-popcnt build
+        if: matrix.config.run_32bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-32-sse41-popcnt build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-32-sse2 build
+        if: matrix.config.run_32bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-32-sse2 build
+          ../tests/signature.sh $benchref
+
+      - name: Test general-32 build
+        if: matrix.config.run_32bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=general-32 build
+          ../tests/signature.sh $benchref
+
+      # x86-64 tests
+
+      - name: Test debug x86-64-avx2 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          export CXXFLAGS="-Werror -D_GLIBCXX_DEBUG"
+          make clean
+          make -j4 ARCH=x86-64-avx2 optimize=no debug=yes build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64-bmi2 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-bmi2 build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64-avx2 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-avx2 build
+          ../tests/signature.sh $benchref
+
+      # Test a deprecated arch
+      - name: Test x86-64-modern build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-modern build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64-sse41-popcnt build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-sse41-popcnt build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64-ssse3 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-ssse3 build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64-sse3-popcnt build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-sse3-popcnt build
+          ../tests/signature.sh $benchref
+
+      - name: Test x86-64 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64 build
+          ../tests/signature.sh $benchref
+
+      - name: Test general-64 build
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=general-64 build
+          ../tests/signature.sh $benchref
+
+      - name: Test apple-silicon build
+        if: matrix.config.run_m1_tests
+        run: |
+          make clean
+          make -j4 ARCH=apple-silicon build
+          ../tests/signature.sh $benchref
+
+      # armv8 tests
+
+      - name: Test armv8 build
+        if: matrix.config.run_armv8_tests
+        run: |
+          export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+          export LDFLAGS="-static -Wno-unused-command-line-argument"
+          make clean
+          make -j4 ARCH=armv8 build
+          ../tests/signature.sh $benchref
+
+      - name: Test armv8-dotprod build
+        if: matrix.config.run_armv8_tests
+        run: |
+          export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+          export LDFLAGS="-static -Wno-unused-command-line-argument"
+          make clean
+          make -j4 ARCH=armv8-dotprod build
+          ../tests/signature.sh $benchref
+
+      # armv7 tests
+
+      - name: Test armv7 build
+        if: matrix.config.run_armv7_tests
+        run: |
+          export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+          export LDFLAGS="-static -Wno-unused-command-line-argument"
+          make clean
+          make -j4 ARCH=armv7 build
+          ../tests/signature.sh $benchref
+
+      - name: Test armv7-neon build
+        if: matrix.config.run_armv7_tests
+        run: |
+          export PATH=${{ env.ANDROID_NDK_BIN }}:$PATH
+          export LDFLAGS="-static -Wno-unused-command-line-argument"
+          make clean
+          make -j4 ARCH=armv7-neon build
+          ../tests/signature.sh $benchref
+
+      # riscv64 tests
+
+      - name: Test riscv64 build
+        if: matrix.config.run_riscv64_tests
+        run: |
+          echo "export LDFLAGS='-static' && make clean && make -j4 ARCH=riscv64 build" > script.sh
+          docker run --rm --platform ${{ matrix.config.platform }} -v ${{ github.workspace }}/src:/app sf_builder
+          ../tests/signature.sh $benchref
+
+      # ppc64 tests
+
+      - name: Test ppc64 build
+        if: matrix.config.run_ppc64_tests
+        run: |
+          echo "export LDFLAGS='-static' && make clean && make -j4 ARCH=ppc-64 build" > script.sh
+          docker run --rm --platform ${{ matrix.config.platform }} -v ${{ github.workspace }}/src:/app sf_builder
+          ../tests/signature.sh $benchref
+
+      # Other tests
+
+      - name: Check perft and search reproducibility
+        if: matrix.config.run_64bit_tests
+        run: |
+          make clean
+          make -j4 ARCH=x86-64-avx2 build
+          ../tests/perft.sh
+          ../tests/reprosearch.sh

.github/workflows/upload_binaries.yml

@@ -0,0 +1,105 @@
+name: Upload Binaries
+on:
+  workflow_call:
+    inputs:
+      matrix:
+        type: string
+        required: true
+
+jobs:
+  Artifacts:
+    name: ${{ matrix.config.name }} ${{ matrix.binaries }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      COMPILER: ${{ matrix.config.compiler }}
+      COMP: ${{ matrix.config.comp }}
+      EXT: ${{ matrix.config.ext }}
+      NAME: ${{ matrix.config.simple_name }}
+      BINARY: ${{ matrix.binaries }}
+      SDE: ${{ matrix.config.sde }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(inputs.matrix) }}
+    defaults:
+      run:
+        shell: ${{ matrix.config.shell }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Download artifact from compilation
+        uses: actions/download-artifact@v4
+        with:
+          name: ${{ matrix.config.simple_name }} ${{ matrix.binaries }}
+          path: ${{ matrix.config.simple_name }} ${{ matrix.binaries }}
+
+      - name: Setup msys and install required packages
+        if: runner.os == 'Windows'
+        uses: msys2/setup-msys2@v2
+        with:
+          msystem: ${{ matrix.config.msys_sys }}
+          install: mingw-w64-${{ matrix.config.msys_env }} make git zip
+
+      - name: Create Package
+        run: |
+          mkdir stockfish
+
+      - name: Download wiki
+        run: |
+          git clone https://github.com/official-stockfish/Stockfish.wiki.git wiki
+          rm -rf wiki/.git
+          mv wiki stockfish/
+
+      - name: Copy files
+        run: |
+          mv "${{ matrix.config.simple_name }} ${{ matrix.binaries }}" stockfish-workflow
+          cd stockfish-workflow
+          cp -r src ../stockfish/
+          cp stockfish-$NAME-$BINARY$EXT ../stockfish/
+          cp "Top CPU Contributors.txt" ../stockfish/
+          cp Copying.txt ../stockfish/
+          cp AUTHORS ../stockfish/
+          cp CITATION.cff ../stockfish/
+          cp README.md ../stockfish/
+          cp CONTRIBUTING.md ../stockfish/
+
+      - name: Create tar
+        if: runner.os != 'Windows'
+        run: |
+          tar -cvf stockfish-$NAME-$BINARY.tar stockfish
+
+      - name: Create zip
+        if: runner.os == 'Windows'
+        run: |
+          zip -r stockfish-$NAME-$BINARY.zip stockfish
+
+      - name: Release
+        if: startsWith(github.ref_name, 'sf_') && github.ref_type == 'tag'
+        uses: softprops/action-gh-release@4634c16e79c963813287e889244c50009e7f0981
+        with:
+          files: stockfish-${{ matrix.config.simple_name }}-${{ matrix.binaries }}.${{ matrix.config.archive_ext }}
+
+      - name: Get last commit sha
+        id: last_commit
+        run: echo "COMMIT_SHA=$(git rev-parse HEAD | cut -c 1-8)" >> $GITHUB_ENV
+
+      - name: Get commit date
+        id: commit_date
+        run: echo "COMMIT_DATE=$(git show -s --date=format:'%Y%m%d' --format=%cd HEAD)" >> $GITHUB_ENV
+
+      # Make sure that an old ci that still runs on master doesn't recreate a prerelease
+      - name: Check Pullable Commits
+        id: check_commits
+        run: |
+          git fetch
+          CHANGES=$(git rev-list HEAD..origin/master --count)
+          echo "CHANGES=$CHANGES" >> $GITHUB_ENV
+
+      - name: Prerelease
+        if: github.ref_name == 'master' && env.CHANGES == '0'
+        continue-on-error: true
+        uses: softprops/action-gh-release@de2c0eb89ae2a093876385947365aca7b0e5f844 # @v1
+        with:
+          name: Stockfish dev-${{ env.COMMIT_DATE }}-${{ env.COMMIT_SHA }}
+          tag_name: stockfish-dev-${{ env.COMMIT_DATE }}-${{ env.COMMIT_SHA }}
+          prerelease: true
+          files: stockfish-${{ matrix.config.simple_name }}-${{ matrix.binaries }}.${{ matrix.config.archive_ext }}

AUTHORS

@@ -1,19 +1,18 @@
-# List of authors for Stockfish
+# Founders of the Stockfish project and Fishtest infrastructure
-
-# Founders of the Stockfish project and fishtest infrastructure
 Tord Romstad (romstad)
 Marco Costalba (mcostalba)
 Joona Kiiski (zamar)
 Gary Linscott (glinscott)
 
-# Authors and inventors of NNUE, training, NNUE port
+# Authors and inventors of NNUE, training, and NNUE port
 Yu Nasu (ynasu87)
 Motohiro Isozaki (yaneurao)
 Hisayori Noda (nodchip)
 
-# all other authors of the code in alphabetical order
+# All other authors of Stockfish code (in alphabetical order)
 Aditya (absimaldata)
 Adrian Petrescu (apetresc)
+Ahmed Kerimov (wcdbmv)
 Ajith Chandy Jose (ajithcj)
 Alain Savard (Rocky640)
 Alayan Feh (Alayan-stk-2)
@@ -21,6 +20,7 @@ Alexander Kure
 Alexander Pagel (Lolligerhans)
 Alfredo Menezes (lonfom169)
 Ali AlZhrani (Cooffe)
+Andreas Matthies (Matthies)
 Andrei Vetrov (proukornew)
 Andrew Grant (AndyGrant)
 Andrey Neporada (nepal)
@@ -30,23 +30,32 @@ Aram Tumanian (atumanian)
 Arjun Temurnikar
 Artem Solopiy (EntityFX)
 Auguste Pop
+Balazs Szilagyi
 Balint Pfliegel
 Ben Chaney (Chaneybenjamini)
 Ben Koshy (BKSpurgeon)
 Bill Henry (VoyagerOne)
 Bojun Guo (noobpwnftw, Nooby)
+borg323
+Boštjan Mejak (PedanticHacker)
 braich
 Brian Sheppard (SapphireBrand, briansheppard-toast)
 Bruno de Melo Costa (BM123499)
+Bruno Pellanda (pellanda)
 Bryan Cross (crossbr)
 candirufish
 Chess13234
 Chris Cain (ceebo)
+clefrks
+Clemens L. (rn5f107s2)
+Cody Ho (aesrentai)
 Dale Weiler (graphitemaster)
-Dan Schmidt (dfannius)
 Daniel Axtens (daxtens)
 Daniel Dugovic (ddugovic)
+Daniel Monroe (Ergodice)
+Dan Schmidt (dfannius)
 Dariusz Orzechowski (dorzechowski)
+David (dav1312)
 David Zar
 Daylen Yang (daylen)
 Deshawn Mohan-Smith (GoldenRare)
@@ -61,45 +70,50 @@ Eelco de Groot (KingDefender)
 Elvin Liu (solarlight2)
 erbsenzaehler
 Ernesto Gatti
-Linmiao Xu (linrock)
 Fabian Beuke (madnight)
 Fabian Fichter (ianfab)
 Fanael Linithien (Fanael)
 fanon
-Fauzi Akram Dabat (FauziAkram)
+Fauzi Akram Dabat (fauzi2)
 Felix Wittmann
 gamander
+Gabriele Lombardo (gabe)
+Gahtan Nahdi
 Gary Heckman (gheckman)
 George Sobala (gsobala)
 gguliash
 Giacomo Lorenzetti (G-Lorenz)
 Gian-Carlo Pascutto (gcp)
+Goh CJ (cj5716)
 Gontran Lemaire (gonlem)
 Goodkov Vasiliy Aleksandrovich (goodkov)
 Gregor Cramer
 GuardianRM
-Günther Demetz (pb00067, pb00068)
 Guy Vreuls (gvreuls)
+Günther Demetz (pb00067, pb00068)
 Henri Wiechers
 Hiraoka Takuya (HiraokaTakuya)
 homoSapiensSapiens
 Hongzhi Cheng
 Ivan Ivec (IIvec)
 Jacques B. (Timshel)
+Jake Senne (w1wwwwww)
 Jan Ondruš (hxim)
-Jared Kish (Kurtbusch)
+Jared Kish (Kurtbusch, kurt22i)
 Jarrod Torriero (DU-jdto)
-Jean Gauthier (OuaisBla)
+Jasper Shovelton (Beanie496)
 Jean-Francois Romang (jromang)
+Jean Gauthier (OuaisBla)
 Jekaa
 Jerry Donald Watson (jerrydonaldwatson)
 jjoshua2
-Jonathan Calovski (Mysseno)
 Jonathan Buladas Dumale (SFisGOD)
+Jonathan Calovski (Mysseno)
+Jonathan McDermid (jonathanmcdermid)
 Joost VandeVondele (vondele)
-Jörg Oster (joergoster)
 Joseph Ellis (jhellis3)
 Joseph R. Prostko
+Jörg Oster (joergoster)
 Julian Willemer (NightlyKing)
 jundery
 Justin Blanchard (UncombedCoconut)
@@ -113,6 +127,7 @@ Krystian Kuzniarek (kuzkry)
 Leonardo Ljubičić (ICCF World Champion)
 Leonid Pechenik (lp--)
 Liam Keegan (lkeegan)
+Linmiao Xu (linrock)
 Linus Arver (listx)
 loco-loco
 Lub van den Berg (ElbertoOne)
@@ -127,6 +142,7 @@ Matt Ginsberg (mattginsberg)
 Matthew Lai (matthewlai)
 Matthew Sullivan (Matt14916)
 Max A. (Disservin)
+Maxim Masiutin (maximmasiutin)
 Maxim Molchanov (Maxim)
 Michael An (man)
 Michael Byrne (MichaelB7)
@@ -141,35 +157,40 @@ Mira
 Miroslav Fontán (Hexik)
 Moez Jellouli (MJZ1977)
 Mohammed Li (tthsqe12)
+Muzhen J (XInTheDark)
 Nathan Rugg (nmrugg)
-Nick Pelling (nickpelling)
+Nguyen Pham (nguyenpham)
 Nicklas Persson (NicklasPersson)
+Nick Pelling (nickpelling)
 Niklas Fiekas (niklasf)
 Nikolay Kostov (NikolayIT)
-Nguyen Pham (nguyenpham)
 Norman Schmidt (FireFather)
 notruck
 Ofek Shochat (OfekShochat, ghostway)
 Ondrej Mosnáček (WOnder93)
+Ondřej Mišina (AndrovT)
 Oskar Werkelin Ahlin
 Pablo Vazquez
 Panthee
 Pascal Romaret
 Pasquale Pigazzini (ppigazzini)
 Patrick Jansen (mibere)
-pellanda
 Peter Schneider (pschneider1968)
 Peter Zsifkovits (CoffeeOne)
+PikaCat
 Praveen Kumar Tummala (praveentml)
+Prokop Randáček (ProkopRandacek)
 Rahul Dsilva (silversolver1)
 Ralph Stößer (Ralph Stoesser)
 Raminder Singh
 renouve
-Reuven Peleg
+Reuven Peleg (R-Peleg)
-Richard Lloyd
+Richard Lloyd (Richard-Lloyd)
+Robert Nürnberg (robertnurnberg)
 Rodrigo Exterckötter Tjäder
-Ron Britvich (Britvich)
+Rodrigo Roim (roim)
 Ronald de Man (syzygy1, syzygy)
+Ron Britvich (Britvich)
 rqs
 Rui Coelho (ruicoelhopedro)
 Ryan Schmitt
@@ -180,20 +201,27 @@ Sergei Antonov (saproj)
 Sergei Ivanov (svivanov72)
 Sergio Vieri (sergiovieri)
 sf-x
+Shahin M. Shahin (peregrine)
 Shane Booth (shane31)
 Shawn Varghese (xXH4CKST3RXx)
 Siad Daboul (Topologist)
 Stefan Geschwentner (locutus2)
 Stefano Cardanobile (Stefano80)
+Stefano Di Martino (StefanoD)
 Steinar Gunderson (sesse)
 Stéphane Nicolet (snicolet)
-Prokop Randáček (ProkopRandacek)
+Stephen Touset (stouset)
+Syine Mineta (MinetaS)
+Taras Vuk (TarasVuk)
 Thanar2
 thaspel
 theo77186
+TierynnB
+Ting-Hsuan Huang (fffelix-huang)
+Tobias Steinmann
+Tomasz Sobczyk (Sopel97)
 Tom Truscott
 Tom Vijlbrief (tomtor)
-Tomasz Sobczyk (Sopel97)
 Torsten Franz (torfranz, tfranzer)
 Torsten Hellwig (Torom)
 Tracey Emery (basepr1me)
@@ -201,11 +229,13 @@ tttak
 Unai Corzo (unaiic)
 Uri Blass (uriblass)
 Vince Negri (cuddlestmonkey)
+Viren
+windfishballad
 xefoci7612
+Xiang Wang (KatyushaScarlet)
 zz4032
 
-
 # Additionally, we acknowledge the authors and maintainers of fishtest,
-# an amazing and essential framework for the development of Stockfish!
+# an amazing and essential framework for Stockfish development!
 #
-# https://github.com/glinscott/fishtest/blob/master/AUTHORS
+# https://github.com/official-stockfish/fishtest/blob/master/AUTHORS

CITATION.cff

@@ -0,0 +1,23 @@
+# This CITATION.cff file was generated with cffinit.
+# Visit https://bit.ly/cffinit to generate yours today!
+
+cff-version: 1.2.0
+title: Stockfish
+message: >-
+  Please cite this software using the metadata from this
+  file.
+type: software
+authors:
+  - name: The Stockfish developers (see AUTHORS file)
+repository-code: 'https://github.com/official-stockfish/Stockfish'
+url: 'https://stockfishchess.org/'
+repository-artifact: 'https://stockfishchess.org/download/'
+abstract: Stockfish is a free and strong UCI chess engine.
+keywords:
+  - chess
+  - artificial intelligence (AI)
+  - tree search
+  - alpha-beta search
+  - neural networks (NN)
+  - efficiently updatable neural networks (NNUE)
+license: GPL-3.0

CONTRIBUTING.md

@@ -0,0 +1,97 @@
+# Contributing to Stockfish
+
+Welcome to the Stockfish project! We are excited that you are interested in
+contributing. This document outlines the guidelines and steps to follow when
+making contributions to Stockfish.
+
+## Table of Contents
+
+- [Building Stockfish](#building-stockfish)
+- [Making Contributions](#making-contributions)
+  - [Reporting Issues](#reporting-issues)
+  - [Submitting Pull Requests](#submitting-pull-requests)
+- [Code Style](#code-style)
+- [Community and Communication](#community-and-communication)
+- [License](#license)
+
+## Building Stockfish
+
+In case you do not have a C++ compiler installed, you can follow the
+instructions from our wiki.
+
+- [Ubuntu][ubuntu-compiling-link]
+- [Windows][windows-compiling-link]
+- [macOS][macos-compiling-link]
+
+## Making Contributions
+
+### Reporting Issues
+
+If you find a bug, please open an issue on the
+[issue tracker][issue-tracker-link]. Be sure to include relevant information
+like your operating system, build environment, and a detailed description of the
+problem.
+
+_Please note that Stockfish's development is not focused on adding new features.
+Thus any issue regarding missing features will potentially be closed without
+further discussion._
+
+### Submitting Pull Requests
+
+- Functional changes need to be tested on fishtest. See
+  [Creating my First Test][creating-my-first-test] for more details.
+  The accompanying pull request should include a link to the test results and
+  the new bench.
+
+- Non-functional changes (e.g. refactoring, code style, documentation) do not
+  need to be tested on fishtest, unless they might impact performance.
+
+- Provide a clear and concise description of the changes in the pull request
+  description.
+
+_First time contributors should add their name to [AUTHORS](../AUTHORS)._
+
+_Stockfish's development is not focused on adding new features. Thus any pull
+request introducing new features will potentially be closed without further
+discussion._
+
+## Code Style
+
+Changes to Stockfish C++ code should respect our coding style defined by
+[.clang-format](.clang-format). You can format your changes by running
+`make format`. This requires clang-format version 17 to be installed on your system.
+
+## Navigate
+
+For experienced Git users who frequently use git blame, it is recommended to
+configure the blame.ignoreRevsFile setting.
+This setting is useful for excluding noisy formatting commits.
+
+```bash
+git config blame.ignoreRevsFile .git-blame-ignore-revs
+```
+
+## Community and Communication
+
+- Join the [Stockfish discord][discord-link] to discuss ideas, issues, and
+  development.
+- Participate in the [Stockfish GitHub discussions][discussions-link] for
+  broader conversations.
+
+## License
+
+By contributing to Stockfish, you agree that your contributions will be licensed
+under the GNU General Public License v3.0. See [Copying.txt][copying-link] for
+more details.
+
+Thank you for contributing to Stockfish and helping us make it even better!
+
+
+[copying-link]:           https://github.com/official-stockfish/Stockfish/blob/master/Copying.txt
+[discord-link]:           https://discord.gg/GWDRS3kU6R
+[discussions-link]:       https://github.com/official-stockfish/Stockfish/discussions/new
+[creating-my-first-test]: https://github.com/official-stockfish/fishtest/wiki/Creating-my-first-test#create-your-test
+[issue-tracker-link]:     https://github.com/official-stockfish/Stockfish/issues
+[ubuntu-compiling-link]:  https://github.com/official-stockfish/Stockfish/wiki/Developers#user-content-installing-a-compiler-1
+[windows-compiling-link]: https://github.com/official-stockfish/Stockfish/wiki/Developers#user-content-installing-a-compiler
+[macos-compiling-link]:   https://github.com/official-stockfish/Stockfish/wiki/Developers#user-content-installing-a-compiler-2

README.md

@@ -1,371 +1,154 @@
+<div align="center">
+
+  [![Stockfish][stockfish128-logo]][website-link]
+
+  <h3>Stockfish</h3>
+
+  A free and strong UCI chess engine.
+  <br>
+  <strong>[Explore Stockfish docs »][wiki-link]</strong>
+  <br>
+  <br>
+  [Report bug][issue-link]
+  ·
+  [Open a discussion][discussions-link]
+  ·
+  [Discord][discord-link]
+  ·
+  [Blog][website-blog-link]
+
+  [![Build][build-badge]][build-link]
+  [![License][license-badge]][license-link]
+  <br>
+  [![Release][release-badge]][release-link]
+  [![Commits][commits-badge]][commits-link]
+  <br>
+  [![Website][website-badge]][website-link]
+  [![Fishtest][fishtest-badge]][fishtest-link]
+  [![Discord][discord-badge]][discord-link]
+
+</div>
+
 ## Overview
 
-[![Build Status](https://github.com/official-stockfish/Stockfish/actions/workflows/stockfish.yml/badge.svg)](https://github.com/official-stockfish/Stockfish/actions)
+[Stockfish][website-link] is a **free and strong UCI chess engine** derived from
+Glaurung 2.1 that analyzes chess positions and computes the optimal moves.
 
-[Stockfish](https://stockfishchess.org) is a free, powerful UCI chess engine
+Stockfish **does not include a graphical user interface** (GUI) that is required
-derived from Glaurung 2.1. Stockfish is not a complete chess program and requires a
+to display a chessboard and to make it easy to input moves. These GUIs are
-UCI-compatible graphical user interface (GUI) (e.g. XBoard with PolyGlot, Scid,
+developed independently from Stockfish and are available online. **Read the
-Cute Chess, eboard, Arena, Sigma Chess, Shredder, Chess Partner or Fritz) in order
+documentation for your GUI** of choice for information about how to use
-to be used comfortably. Read the documentation for your GUI of choice for information
+Stockfish with it.
-about how to use Stockfish with it.
 
-The Stockfish engine features two evaluation functions for chess. The efficiently 
+See also the Stockfish [documentation][wiki-usage-link] for further usage help.
-updatable neural network (NNUE) based evaluation is the default and by far the strongest.
-The classical evaluation based on handcrafted terms remains available. The strongest 
-network is integrated in the binary and downloaded automatically during the build process.
-The NNUE evaluation benefits from the vector intrinsics available on most CPUs (sse2, 
-avx2, neon, or similar).
 
 ## Files
 
 This distribution of Stockfish consists of the following files:
 
-  * [README.md](https://github.com/official-stockfish/Stockfish/blob/master/README.md),
+  * [README.md][readme-link], the file you are currently reading.
-    the file you are currently reading.
 
-  * [Copying.txt](https://github.com/official-stockfish/Stockfish/blob/master/Copying.txt),
+  * [Copying.txt][license-link], a text file containing the GNU General Public
-    a text file containing the GNU General Public License version 3.
+    License version 3.
 
-  * [AUTHORS](https://github.com/official-stockfish/Stockfish/blob/master/AUTHORS), 
+  * [AUTHORS][authors-link], a text file with the list of authors for the project.
-    a text file with the list of authors for the project
 
-  * [src](https://github.com/official-stockfish/Stockfish/tree/master/src),
+  * [src][src-link], a subdirectory containing the full source code, including a
-    a subdirectory containing the full source code, including a Makefile
+    Makefile that can be used to compile Stockfish on Unix-like systems.
-    that can be used to compile Stockfish on Unix-like systems.
 
   * a file with the .nnue extension, storing the neural network for the NNUE
     evaluation. Binary distributions will have this file embedded.
 
-## The UCI protocol and available options
+## Contributing
 
-The Universal Chess Interface (UCI) is a standard protocol used to communicate with
+__See [Contributing Guide](CONTRIBUTING.md).__
-a chess engine, and is the recommended way to do so for typical graphical user interfaces
-(GUI) or chess tools. Stockfish implements the majority of its options as described
-in [the UCI protocol](https://www.shredderchess.com/download/div/uci.zip).
-
-Developers can see the default values for UCI options available in Stockfish by typing
-`./stockfish uci` in a terminal, but the majority of users will typically see them and
-change them via a chess GUI. This is a list of available UCI options in Stockfish:
-
-  * #### Threads
-    The number of CPU threads used for searching a position. For best performance, set
-    this equal to the number of CPU cores available.
-
-  * #### Hash
-    The size of the hash table in MB. It is recommended to set Hash after setting Threads.
-
-  * #### Clear Hash
-    Clear the hash table.
-
-  * #### Ponder
-    Let Stockfish ponder its next move while the opponent is thinking.
-
-  * #### MultiPV
-    Output the N best lines (principal variations, PVs) when searching.
-    Leave at 1 for best performance.
-
-  * #### Use NNUE
-    Toggle between the NNUE and classical evaluation functions. If set to "true",
-    the network parameters must be available to load from file (see also EvalFile),
-    if they are not embedded in the binary.
-
-  * #### EvalFile
-    The name of the file of the NNUE evaluation parameters. Depending on the GUI the
-    filename might have to include the full path to the folder/directory that contains
-    the file. Other locations, such as the directory that contains the binary and the
-    working directory, are also searched.
-
-  * #### UCI_AnalyseMode
-    An option handled by your GUI.
-
-  * #### UCI_Chess960
-    An option handled by your GUI. If true, Stockfish will play Chess960.
-
-  * #### UCI_ShowWDL
-    If enabled, show approximate WDL statistics as part of the engine output.
-    These WDL numbers model expected game outcomes for a given evaluation and
-    game ply for engine self-play at fishtest LTC conditions (60+0.6s per game).
-
-  * #### UCI_LimitStrength
-    Enable weaker play aiming for an Elo rating as set by UCI_Elo. This option overrides Skill Level.
-
-  * #### UCI_Elo
-    If enabled by UCI_LimitStrength, aim for an engine strength of the given Elo.
-    This Elo rating has been calibrated at a time control of 60s+0.6s and anchored to CCRL 40/4.
-
-  * #### Skill Level
-    Lower the Skill Level in order to make Stockfish play weaker (see also UCI_LimitStrength).
-    Internally, MultiPV is enabled, and with a certain probability depending on the Skill Level a
-    weaker move will be played.
-
-  * #### SyzygyPath
-    Path to the folders/directories storing the Syzygy tablebase files. Multiple
-    directories are to be separated by ";" on Windows and by ":" on Unix-based
-    operating systems. Do not use spaces around the ";" or ":".
-
-    Example: `C:\tablebases\wdl345;C:\tablebases\wdl6;D:\tablebases\dtz345;D:\tablebases\dtz6`
-
-    It is recommended to store .rtbw files on an SSD. There is no loss in storing
-    the .rtbz files on a regular HDD. It is recommended to verify all md5 checksums
-    of the downloaded tablebase files (`md5sum -c checksum.md5`) as corruption will
-    lead to engine crashes.
-
-  * #### SyzygyProbeDepth
-    Minimum remaining search depth for which a position is probed. Set this option
-    to a higher value to probe less aggressively if you experience too much slowdown
-    (in terms of nps) due to tablebase probing.
-
-  * #### Syzygy50MoveRule
-    Disable to let fifty-move rule draws detected by Syzygy tablebase probes count
-    as wins or losses. This is useful for ICCF correspondence games.
-
-  * #### SyzygyProbeLimit
-    Limit Syzygy tablebase probing to positions with at most this many pieces left
-    (including kings and pawns).
-
-  * #### Move Overhead
-    Assume a time delay of x ms due to network and GUI overheads. This is useful to
-    avoid losses on time in those cases.
-
-  * #### Slow Mover
-    Lower values will make Stockfish take less time in games, higher values will
-    make it think longer.
-
-  * #### nodestime
-    Tells the engine to use nodes searched instead of wall time to account for
-    elapsed time. Useful for engine testing.
-
-  * #### Debug Log File
-    Write all communication to and from the engine into a text file.
-
-For developers the following non-standard commands might be of interest, mainly useful for debugging:
-
-  * #### bench *ttSize threads limit fenFile limitType evalType*
-    Performs a standard benchmark using various options. The signature of a version 
-    (standard node count) is obtained using all defaults. `bench` is currently 
-    `bench 16 1 13 default depth mixed`.
-
-  * #### compiler
-    Give information about the compiler and environment used for building a binary.
-
-  * #### d
-    Display the current position, with ascii art and fen.
-
-  * #### eval
-    Return the evaluation of the current position.
-
-  * #### export_net [filename]
-    Exports the currently loaded network to a file.
-    If the currently loaded network is the embedded network and the filename
-    is not specified then the network is saved to the file matching the name
-    of the embedded network, as defined in evaluate.h.
-    If the currently loaded network is not the embedded network (some net set
-    through the UCI setoption) then the filename parameter is required and the
-    network is saved into that file.
-
-  * #### flip
-    Flips the side to move.
-
-### Generating Training Data
-
-To generate training data from the classic eval, use the generate_training_data command with the setting "Use NNUE" set to "false". The given example is generation in its simplest form. There are more commands.
-
-```
-uci
-setoption name PruneAtShallowDepth value false
-setoption name Use NNUE value false
-setoption name Threads value X
-setoption name Hash value Y
-setoption name SyzygyPath value path
-isready
-generate_training_data depth A count B keep_draws 1 eval_limit 32000
-```
-
-- `A` is the searched depth per move, or how far the engine looks forward. This value is an integer.
-- `B` is the amount of positions generated. This value is also an integer.
-
-Specify how many threads and how much memory you would like to use with the `x` and `y` values. The option SyzygyPath is not necessary, but if you would like to use it, you must first have Syzygy endgame tablebases on your computer, which you can find [here](http://oics.olympuschess.com/tracker/index.php). You will need to have a torrent client to download these tablebases, as that is probably the fastest way to obtain them. The `path` is the path to the folder containing those tablebases. It does not have to be surrounded in quotes.
-
-This will create a file named "training_data.binpack" in the same folder as the binary containing the generated training data. Once generation is done, you can rename the file to something like "1billiondepth12.binpack" to remember the depth and quantity of the positions and move it to a folder named "trainingdata" in the same directory as the binaries.
-
-You will also need validation data that is used for loss calculation and accuracy computation. Validation data is generated in the same way as training data, but generally at most 1 million positions should be used as there's no need for more and it would just slow the learning process down. It may also be better to slightly increase the depth for validation data. After generation you can rename the validation data file to "val.binpack" and drop it in a folder named "validationdata" in the same directory to make it easier.
-
-## Training data formats.
-
-Currently there are 3 training data formats. Two of them are supported directly.
-
-- `.bin` - the original training data format. Uses 40 bytes per entry. Is supported directly by the `generate_training_data` command.
-- `.plain` - a human readable training data format. This one is not supported directly by the `generate_training_data` command. It should not be used for data exchange because it's less compact than other formats. It is mostly useful for inspection of the data.
-- `.binpack` - a compact binary training data format that exploits positions chains to further reduce size. It uses on average between 2 to 3 bytes per entry when generating data with `generate_training_data`. It is supported directly by `generate_training_data` command. It is currently the default for the `generate_training_data` command. A more in depth description can be found [here](docs/binpack.md)
-
-### Conversion between formats.
-
-There is a builting converted that support all 3 formats described above. Any of them can be converted to any other. For more information and usage guide see [here](docs/convert.md).
-
-## A note on classical evaluation versus NNUE evaluation
-
-Both approaches assign a value to a position that is used in alpha-beta (PVS) search
-to find the best move. The classical evaluation computes this value as a function
-of various chess concepts, handcrafted by experts, tested and tuned using fishtest.
-The NNUE evaluation computes this value with a neural network based on basic
-inputs (e.g. piece positions only). The network is optimized and trained
-on the evaluations of millions of positions at moderate search depth.
-
-The NNUE evaluation was first introduced in shogi, and ported to Stockfish afterward.
-It can be evaluated efficiently on CPUs, and exploits the fact that only parts
-of the neural network need to be updated after a typical chess move.
-[The nodchip repository](https://github.com/nodchip/Stockfish) provided the first 
-version of the needed tools to train and develop the NNUE networks. Today, more 
-advanced training tools are available in 
-[the nnue-pytorch repository](https://github.com/glinscott/nnue-pytorch/), 
-while data generation tools are available in 
-[a dedicated branch](https://github.com/official-stockfish/Stockfish/tree/tools).
-
-On CPUs supporting modern vector instructions (avx2 and similar), the NNUE evaluation
-results in much stronger playing strength, even if the nodes per second computed by
-the engine is somewhat lower (roughly 80% of nps is typical).
-
-Notes:
-
-1) the NNUE evaluation depends on the Stockfish binary and the network parameter file 
-(see the EvalFile UCI option). Not every parameter file is compatible with a given
-Stockfish binary, but the default value of the EvalFile UCI option is the name of a 
-network that is guaranteed to be compatible with that binary.
-
-2) to use the NNUE evaluation, the additional data file with neural network parameters
-needs to be available. Normally, this file is already embedded in the binary or it can
-be downloaded. The filename for the default (recommended) net can be found as the default
-value of the `EvalFile` UCI option, with the format `nn-[SHA256 first 12 digits].nnue`
-(for instance, `nn-c157e0a5755b.nnue`). This file can be downloaded from
-```
-https://tests.stockfishchess.org/api/nn/[filename]
-```
-replacing `[filename]` as needed.
-
-## What to expect from the Syzygy tablebases?
-
-If the engine is searching a position that is not in the tablebases (e.g.
-a position with 8 pieces), it will access the tablebases during the search.
-If the engine reports a very large score (typically 153.xx), this means
-it has found a winning line into a tablebase position.
-
-If the engine is given a position to search that is in the tablebases, it
-will use the tablebases at the beginning of the search to preselect all
-good moves, i.e. all moves that preserve the win or preserve the draw while
-taking into account the 50-move rule.
-It will then perform a search only on those moves. **The engine will not move
-immediately**, unless there is only a single good move. **The engine likely
-will not report a mate score, even if the position is known to be won.**
-
-It is therefore clear that this behaviour is not identical to what one might
-be used to with Nalimov tablebases. There are technical reasons for this
-difference, the main technical reason being that Nalimov tablebases use the
-DTM metric (distance-to-mate), while the Syzygy tablebases use a variation of the
-DTZ metric (distance-to-zero, zero meaning any move that resets the 50-move
-counter). This special metric is one of the reasons that the Syzygy tablebases are
-more compact than Nalimov tablebases, while still storing all information
-needed for optimal play and in addition being able to take into account
-the 50-move rule.
-
-## Large Pages
-
-Stockfish supports large pages on Linux and Windows. Large pages make
-the hash access more efficient, improving the engine speed, especially
-on large hash sizes. Typical increases are 5..10% in terms of nodes per
-second, but speed increases up to 30% have been measured. The support is
-automatic. Stockfish attempts to use large pages when available and
-will fall back to regular memory allocation when this is not the case.
-
-### Support on Linux
-
-Large page support on Linux is obtained by the Linux kernel
-transparent huge pages functionality. Typically, transparent huge pages
-are already enabled, and no configuration is needed.
-
-### Support on Windows
-
-The use of large pages requires "Lock Pages in Memory" privilege. See
-[Enable the Lock Pages in Memory Option (Windows)](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/enable-the-lock-pages-in-memory-option-windows)
-on how to enable this privilege, then run [RAMMap](https://docs.microsoft.com/en-us/sysinternals/downloads/rammap)
-to double-check that large pages are used. We suggest that you reboot
-your computer after you have enabled large pages, because long Windows
-sessions suffer from memory fragmentation, which may prevent Stockfish
-from getting large pages: a fresh session is better in this regard.
-
-## Compiling Stockfish yourself from the sources
-
-Stockfish has support for 32 or 64-bit CPUs, certain hardware
-instructions, big-endian machines such as Power PC, and other platforms.
-
-On Unix-like systems, it should be easy to compile Stockfish
-directly from the source code with the included Makefile in the folder
-`src`. In general it is recommended to run `make help` to see a list of make
-targets with corresponding descriptions.
-
-```
-    cd src
-    make help
-    make net
-    make build ARCH=x86-64-modern
-```
-
-When not using the Makefile to compile (for instance, with Microsoft MSVC) you
-need to manually set/unset some switches in the compiler command line; see
-file *types.h* for a quick reference.
-
-When reporting an issue or a bug, please tell us which Stockfish version
-and which compiler you used to create your executable. This information
-can be found by typing the following command in a console:
-
-```
-    ./stockfish compiler
-```
-
-## Understanding the code base and participating in the project
-
-Stockfish's improvement over the last decade has been a great community
-effort. There are a few ways to help contribute to its growth.
 
 ### Donating hardware
 
-Improving Stockfish requires a massive amount of testing. You can donate
+Improving Stockfish requires a massive amount of testing. You can donate your
-your hardware resources by installing the [Fishtest Worker](https://github.com/glinscott/fishtest/wiki/Running-the-worker:-overview)
+hardware resources by installing the [Fishtest Worker][worker-link] and viewing
-and view the current tests on [Fishtest](https://tests.stockfishchess.org/tests).
+the current tests on [Fishtest][fishtest-link].
 
 ### Improving the code
 
-If you want to help improve the code, there are several valuable resources:
+In the [chessprogramming wiki][programming-link], many techniques used in
-
-* [In this wiki,](https://www.chessprogramming.org) many techniques used in
 Stockfish are explained with a lot of background information.
+The [section on Stockfish][programmingsf-link] describes many features
+and techniques used by Stockfish. However, it is generic rather than
+focused on Stockfish's precise implementation.
 
-* [The section on Stockfish](https://www.chessprogramming.org/Stockfish)
+The engine testing is done on [Fishtest][fishtest-link].
-describes many features and techniques used by Stockfish. However, it is
+If you want to help improve Stockfish, please read this [guideline][guideline-link]
-generic rather than being focused on Stockfish's precise implementation.
-Nevertheless, a helpful resource.
-
-* The latest source can always be found on [GitHub](https://github.com/official-stockfish/Stockfish).
-Discussions about Stockfish take place these days mainly in the [FishCooking](https://groups.google.com/forum/#!forum/fishcooking)
-group and on the [Stockfish Discord channel](https://discord.gg/nv8gDtt).
-The engine testing is done on [Fishtest](https://tests.stockfishchess.org/tests).
-If you want to help improve Stockfish, please read this [guideline](https://github.com/glinscott/fishtest/wiki/Creating-my-first-test)
 first, where the basics of Stockfish development are explained.
 
+Discussions about Stockfish take place these days mainly in the Stockfish
+[Discord server][discord-link]. This is also the best place to ask questions
+about the codebase and how to improve it.
+
+## Compiling Stockfish
+
+Stockfish has support for 32 or 64-bit CPUs, certain hardware instructions,
+big-endian machines such as Power PC, and other platforms.
+
+On Unix-like systems, it should be easy to compile Stockfish directly from the
+source code with the included Makefile in the folder `src`. In general, it is
+recommended to run `make help` to see a list of make targets with corresponding
+descriptions. An example suitable for most Intel and AMD chips:
+
+```
+cd src
+make -j profile-build ARCH=x86-64-avx2
+```
+
+Detailed compilation instructions for all platforms can be found in our
+[documentation][wiki-compile-link]. Our wiki also has information about
+the [UCI commands][wiki-uci-link] supported by Stockfish.
 
 ## Terms of use
 
-Stockfish is free, and distributed under the **GNU General Public License version 3**
+Stockfish is free and distributed under the
-(GPL v3). Essentially, this means you are free to do almost exactly
+[**GNU General Public License version 3**][license-link] (GPL v3). Essentially,
-what you want with the program, including distributing it among your
+this means you are free to do almost exactly what you want with the program,
-friends, making it available for download from your website, selling
+including distributing it among your friends, making it available for download
-it (either by itself or as part of some bigger software package), or
+from your website, selling it (either by itself or as part of some bigger
-using it as the starting point for a software project of your own.
+software package), or using it as the starting point for a software project of
+your own.
 
-The only real limitation is that whenever you distribute Stockfish in
+The only real limitation is that whenever you distribute Stockfish in some way,
-some way, you MUST always include the license and the full source code
+you MUST always include the license and the full source code (or a pointer to
-(or a pointer to where the source code can be found) to generate the 
+where the source code can be found) to generate the exact binary you are
-exact binary you are distributing. If you make any changes to the
+distributing. If you make any changes to the source code, these changes must
-source code, these changes must also be made available under the GPL v3.
+also be made available under GPL v3.
 
-For full details, read the copy of the GPL v3 found in the file named
+
-[*Copying.txt*](https://github.com/official-stockfish/Stockfish/blob/master/Copying.txt).
+[authors-link]:       https://github.com/official-stockfish/Stockfish/blob/master/AUTHORS
+[build-link]:         https://github.com/official-stockfish/Stockfish/actions/workflows/stockfish.yml
+[commits-link]:       https://github.com/official-stockfish/Stockfish/commits/master
+[discord-link]:       https://discord.gg/GWDRS3kU6R
+[issue-link]:         https://github.com/official-stockfish/Stockfish/issues/new?assignees=&labels=&template=BUG-REPORT.yml
+[discussions-link]:   https://github.com/official-stockfish/Stockfish/discussions/new
+[fishtest-link]:      https://tests.stockfishchess.org/tests
+[guideline-link]:     https://github.com/official-stockfish/fishtest/wiki/Creating-my-first-test
+[license-link]:       https://github.com/official-stockfish/Stockfish/blob/master/Copying.txt
+[programming-link]:   https://www.chessprogramming.org/Main_Page
+[programmingsf-link]: https://www.chessprogramming.org/Stockfish
+[readme-link]:        https://github.com/official-stockfish/Stockfish/blob/master/README.md
+[release-link]:       https://github.com/official-stockfish/Stockfish/releases/latest
+[src-link]:           https://github.com/official-stockfish/Stockfish/tree/master/src
+[stockfish128-logo]:  https://stockfishchess.org/images/logo/icon_128x128.png
+[uci-link]:           https://backscattering.de/chess/uci/
+[website-link]:       https://stockfishchess.org
+[website-blog-link]:  https://stockfishchess.org/blog/
+[wiki-link]:          https://github.com/official-stockfish/Stockfish/wiki
+[wiki-compile-link]:  https://github.com/official-stockfish/Stockfish/wiki/Compiling-from-source
+[wiki-uci-link]:      https://github.com/official-stockfish/Stockfish/wiki/UCI-&-Commands
+[wiki-usage-link]:    https://github.com/official-stockfish/Stockfish/wiki/Download-and-usage
+[worker-link]:        https://github.com/official-stockfish/fishtest/wiki/Running-the-worker
+
+[build-badge]:        https://img.shields.io/github/actions/workflow/status/official-stockfish/Stockfish/stockfish.yml?branch=master&style=for-the-badge&label=stockfish&logo=github
+[commits-badge]:      https://img.shields.io/github/commits-since/official-stockfish/Stockfish/latest?style=for-the-badge
+[discord-badge]:      https://img.shields.io/discord/435943710472011776?style=for-the-badge&label=discord&logo=Discord
+[fishtest-badge]:     https://img.shields.io/website?style=for-the-badge&down_color=red&down_message=Offline&label=Fishtest&up_color=success&up_message=Online&url=https%3A%2F%2Ftests.stockfishchess.org%2Ftests%2Ffinished
+[license-badge]:      https://img.shields.io/github/license/official-stockfish/Stockfish?style=for-the-badge&label=license&color=success
+[release-badge]:      https://img.shields.io/github/v/release/official-stockfish/Stockfish?style=for-the-badge&label=official%20release
+[website-badge]:      https://img.shields.io/website?style=for-the-badge&down_color=red&down_message=Offline&label=website&up_color=success&up_message=Online&url=https%3A%2F%2Fstockfishchess.org

Top CPU Contributors.txt

@@ -1,100 +1,126 @@
-Contributors to Fishtest with >10,000 CPU hours, as of 2022-04-14.
+Contributors to Fishtest with >10,000 CPU hours, as of 2024-02-24.
 Thank you!
 
 Username                                CPU Hours     Games played
 ------------------------------------------------------------------
-noobpwnftw                               31714850       2267266129
+noobpwnftw                               39302472       3055513453
-mlang                                     2954099        198421098
+technologov                              20845762        994893444
-technologov                               2324150        102449398
+linrock                                   8616428        560281417
-dew                                       1670874         99276012
+mlang                                     3026000        200065824
-grandphish2                               1134273         68070459
+okrout                                    2332151        222639518
-okrout                                     901194         77738874
+pemo                                      1800019         60274069
-TueRens                                    821388         50207666
+dew                                       1689162        100033738
-tvijlbrief                                 795993         51894442
+TueRens                                   1474943         75121774
-pemo                                       744463         32486677
+grandphish2                               1463002         91616949
-JojoM                                      724378         43660674
+JojoM                                     1109702         72927902
+olafm                                      978631         71037944
+sebastronomy                               939955         44920556
+tvijlbrief                                 796125         51897690
+gvreuls                                    711320         49142318
 mibere                                     703840         46867607
-linrock                                    626939         17408017
+oz                                         646268         46293638
-gvreuls                                    534079         34352532
+rpngn                                      572571         38928563
-cw                                         507221         34006775
+leszek                                     531858         39316505
-fastgm                                     489749         29344518
+cw                                         518116         34894291
-crunchy                                    427035         27344275
+fastgm                                     503862         30260818
-CSU_Dynasty                                424643         28525220
+CSU_Dynasty                                468784         31385034
-ctoks                                      415771         27364603
+ctoks                                      434591         28520597
-oz                                         369200         27017658
+maximmasiutin                              429983         27066286
-bcross                                     342642         23671289
+crunchy                                    427414         27371625
+bcross                                     415724         29061187
+velislav                                   342588         22140902
+mgrabiak                                   338763         23999170
 Fisherman                                  327231         21829379
-velislav                                   325670         20911076
+robal                                      299836         20213182
-leszek                                     321295         19874113
+Dantist                                    296386         18031762
-Dantist                                    274747         16910258
+ncfish1                                    267604         17881149
-mgrabiak                                   237604         15418700
+nordlandia                                 249322         16420192
-robal                                      217959         13840386
+marrco                                     234581         17714473
-glinscott                                  217799         13780820
+tolkki963                                  233490         19773930
-nordlandia                                 211692         13484886
+glinscott                                  208125         13277240
-drabel                                     201967         13798360
+drabel                                     204167         13930674
+mhoram                                     202894         12601997
 bking_US                                   198894         11876016
-mhoram                                     194862         12261809
+Calis007                                   188631         12795784
 Thanar                                     179852         12365359
+Fifis                                      176209         10638245
 vdv                                        175544          9904472
 spams                                      157128         10319326
-rpngn                                      154081          9652139
+DesolatedDodo                              156659         10210328
-marrco                                     150300          9402229
+armo9494                                   155355         10566898
 sqrt2                                      147963          9724586
-vdbergh                                    137430          8955097
+jcAEie                                     140086         10603658
+vdbergh                                    139746          9172061
 CoffeeOne                                  137100          5024116
 malala                                     136182          8002293
 xoto                                       133759          9159372
-davar                                      125240          8117121
+davar                                      129023          8376525
+DMBK                                       122960          8980062
 dsmith                                     122059          7570238
-amicic                                     119659          7937885
+javran                                     121564         10144656
+amicic                                     119661          7938029
+sschnee                                    118107          7389266
+Wolfgang                                   114616          8070494
 Data                                       113305          8220352
 BrunoBanani                                112960          7436849
-CypressChess                               108321          7759588
+Wencey                                     111502          5991676
-DesolatedDodo                              106811          6776980
+cuistot                                    108503          7006992
+CypressChess                               108331          7759788
+skiminki                                   107583          7218170
 MaZePallas                                 102823          6633619
 sterni1971                                 100532          5880772
 sunu                                       100167          7040199
+zeryl                                       99331          6221261
+thirdlife                                   99156          2245320
 ElbertoOne                                  99028          7023771
-skiminki                                    98123          6478402
+Dubslow                                     98600          6903242
+markkulix                                   97010          7643900
+bigpen0r                                    94809          6529203
 brabos                                      92118          6186135
-cuistot                                     90358          5351004
+Maxim                                       90818          3283364
 psk                                         89957          5984901
-racerschmacer                               85712          6119648
+megaman7de                                  88822          6052132
+racerschmacer                               85805          6122790
+maposora                                    85710          7778146
 Vizvezdenec                                 83761          5344740
-zeryl                                       83680          5250995
-sschnee                                     83003          4840890
 0x3C33                                      82614          5271253
 BRAVONE                                     81239          5054681
 nssy                                        76497          5259388
+jromang                                     76106          5236025
 teddybaer                                   75125          5407666
-jromang                                     74796          5175825
 Pking_cda                                   73776          5293873
-Calis007                                    72477          4088576
+yurikvelo                                   73516          5036928
+MarcusTullius                               71053          4803477
+Bobo1239                                    70579          4794999
 solarlight                                  70517          5028306
 dv8silencer                                 70287          3883992
-Bobo1239                                    68515          4652287
+Spprtr                                      69646          4806763
+Mineta                                      66325          4537742
 manap                                       66273          4121774
-yurikvelo                                   65716          4457300
+szupaw                                      65468          5669742
 tinker                                      64333          4268790
-Wolfgang                                    62644          3817410
 qurashee                                    61208          3429862
+woutboat                                    59496          4906352
+AGI                                         58195          4329580
 robnjr                                      57262          4053117
 Freja                                       56938          3733019
+MaxKlaxxMiner                               56879          3423958
 ttruscott                                   56010          3680085
 rkl                                         55132          4164467
+jmdana                                      54697          4012593
 renouve                                     53811          3501516
-megaman7de                                  52434          3243016
+notchris                                    52433          4044590
-MaxKlaxxMiner                               51977          3153032
 finfish                                     51360          3370515
 eva42                                       51272          3599691
-eastorwest                                  51058          3451555
+eastorwest                                  51117          3454811
+Goatminola                                  51004          4432492
 rap                                         49985          3219146
-pb00067                                     49727          3298270
+pb00067                                     49733          3298934
-Spprtr                                      48920          3161711
+GPUex                                       48686          3684998
-bigpen0r                                    47667          3336927
+OuaisBla                                    48626          3445134
 ronaldjerum                                 47654          3240695
 biffhero                                    46564          3111352
-Fifis                                       45843          3088497
+oryx                                        45533          3539290
 VoyagerOne                                  45476          3452465
 speedycpu                                   43842          3003273
 jbwiebe                                     43305          2805433
@@ -102,134 +128,159 @@ Antihistamine                               41788          2761312
 mhunt                                       41735          2691355
 homyur                                      39893          2850481
 gri                                         39871          2515779
-armo9494                                    39064          2832326
+Garf                                        37741          2999686
-oryx                                        38867          2976992
 SC                                          37299          2731694
-Garf                                        37213          2986270
+Sylvain27                                   36520          1467082
-tolkki963                                   37059          2154330
 csnodgrass                                  36207          2688994
-jmdana                                      36157          2210661
+Gaster319                                   35655          3149442
 strelock                                    34716          2074055
-DMBK                                        34010          2482916
 EthanOConnor                                33370          2090311
 slakovv                                     32915          2021889
-gopeto                                      30993          2028106
+gopeto                                      31884          2076712
+Gelma                                       31771          1551204
+kdave                                       31157          2198362
 manapbk                                     30987          1810399
+ZacHFX                                      30551          2238078
 Prcuvu                                      30377          2170122
 anst                                        30301          2190091
 jkiiski                                     30136          1904470
+spcc                                        29925          1901692
 hyperbolic.tom                              29840          2017394
 chuckstablers                               29659          2093438
 Pyafue                                      29650          1902349
-ncfish1                                     29105          1704011
+belzedar94                                  28846          1811530
-belzedar94                                  27935          1789106
+votoanthuan                                 27978          2285818
-OuaisBla                                    27636          1578800
+shawnxu                                     27438          2465810
 chriswk                                     26902          1868317
+xwziegtm                                    26897          2124586
 achambord                                   26582          1767323
 Patrick_G                                   26276          1801617
 yorkman                                     26193          1992080
+Ulysses                                     25397          1701264
+Jopo12321                                   25227          1652482
 SFTUser                                     25182          1675689
-nabildanial                                 24942          1519409
+nabildanial                                 25068          1531665
 Sharaf_DG                                   24765          1786697
-rodneyc                                     24275          1410450
+rodneyc                                     24376          1416402
+jsys14                                      24297          1721230
 agg177                                      23890          1395014
+srowen                                      23842          1342508
+Ente                                        23752          1678188
+jojo2357                                    23479          2061238
 JanErik                                     23408          1703875
 Isidor                                      23388          1680691
-Norabor                                     23339          1602636
+Norabor                                     23371          1603244
-Ente                                        23270          1651432
+cisco2015                                   22920          1763301
-cisco2015                                   22897          1762669
-MarcusTullius                               22688          1274821
 Zirie                                       22542          1472937
+Nullvalue                                   22490          1970374
+AndreasKrug                                 22485          1769491
 team-oh                                     22272          1636708
+Roady                                       22220          1465606
 MazeOfGalious                               21978          1629593
-sg4032                                      21947          1643265
+sg4032                                      21947          1643353
 ianh2105                                    21725          1632562
 xor12                                       21628          1680365
 dex                                         21612          1467203
 nesoneg                                     21494          1463031
-Roady                                       21323          1433822
+user213718                                  21454          1404128
 sphinx                                      21211          1384728
-user213718                                  21196          1397710
+qoo_charly_cai                              21135          1514907
-spcc                                        21065          1311338
 jjoshua2                                    21001          1423089
+Zake9298                                    20938          1565848
 horst.prack                                 20878          1465656
 0xB00B1ES                                   20590          1208666
+Serpensin                                   20487          1729674
+Dinde                                       20440          1292390
 j3corre                                     20405           941444
-kdave                                       20364          1389254
 Adrian.Schmidt123                           20316          1281436
-Ulysses                                     20217          1351500
-markkulix                                   19976          1115258
 wei                                         19973          1745989
+fishtester                                  19617          1257388
 rstoesser                                   19569          1293588
 eudhan                                      19274          1283717
-fishtester                                  18995          1238686
 vulcan                                      18871          1729392
+Karpovbot                                   18766          1053178
+WoodMan777                                  18556          1628264
 jundery                                     18445          1115855
-iisiraider                                  18247          1101015
 ville                                       17883          1384026
 chris                                       17698          1487385
 purplefishies                               17595          1092533
-dju                                         17353           978595
+dju                                         17414           981289
-Wencey                                      17125           805964
+ols                                         17291          1042003
+iisiraider                                  17275          1049015
+Skiff84                                     17111           950248
 DragonLord                                  17014          1162790
-thirdlife                                   16996           447356
+redstone59                                  16842          1461780
+Karby                                       16839          1010124
+Alb11747                                    16787          1213990
+pirt                                        16493          1237199
+Naven94                                     16414           951718
+wizardassassin                              16392          1148672
 IgorLeMasson                                16064          1147232
+scuzzi                                      15757           968735
 ako027ako                                   15671          1173203
-AndreasKrug                                 15550          1194497
 Nikolay.IT                                  15154          1068349
 Andrew Grant                                15114           895539
-scuzzi                                      14928           953313
 OssumOpossum                                14857          1007129
-Karby                                       14808           867120
+LunaticBFF57                                14525          1190310
-jsys14                                      14652           855642
 enedene                                     14476           905279
-bpfliegel                                   14298           884523
+IslandLambda                                14393           958196
+bpfliegel                                   14233           882523
+YELNAMRON                                   14230          1128094
 mpx86                                       14019           759568
 jpulman                                     13982           870599
+getraideBFF                                 13871          1172846
+Nesa92                                      13806          1116101
 crocogoat                                   13803          1117422
-joster                                      13794           950160
+joster                                      13710           946160
-Nesa92                                      13786          1114691
 mbeier                                      13650          1044928
 Hjax                                        13535           915487
 Dark_wizzie                                 13422          1007152
-Jopo12321                                   13367           678852
 Rudolphous                                  13244           883140
 Machariel                                   13010           863104
+infinigon                                   12991           943216
 mabichito                                   12903           749391
 thijsk                                      12886           722107
 AdrianSA                                    12860           804972
-infinigon                                   12807           937332
 Flopzee                                     12698           894821
+mschmidt                                    12644           863193
+korposzczur                                 12606           838168
+tsim67                                      12570           890180
+Jackfish                                    12553           836958
 fatmurphy                                   12547           853210
+Oakwen                                      12503           853105
 SapphireBrand                               12416           969604
+deflectooor                                 12386           579392
 modolief                                    12386           896470
+TataneSan                                   12358           609332
 Farseer                                     12249           694108
 pgontarz                                    12151           848794
-pirt                                        12008           923149
+dbernier                                    12103           860824
+FormazChar                                  11989           907809
 stocky                                      11954           699440
-mschmidt                                    11941           803401
+somethingintheshadows                       11940           989472
-dbernier                                    11609           818636
+MooTheCow                                   11892           776126
-Maxim                                       11543           836024
+3cho                                        11842          1036786
+whelanh                                     11557           245188
 infinity                                    11470           727027
-aga                                         11409           695071
+aga                                         11412           695127
 torbjo                                      11395           729145
 Thomas A. Anderson                          11372           732094
 savage84                                    11358           670860
-FormazChar                                  11349           850327
 d64                                         11263           789184
-MooTheCow                                   11237           720174
+ali-al-zhrani                               11245           779246
+ckaz                                        11170           680866
 snicolet                                    11106           869170
-ali-al-zhrani                               11098           768494
+dapper                                      11032           771402
-whelanh                                     11067           235676
+Ethnikoi                                    10993           945906
-Jackfish                                    10978           720078
+Snuuka                                      10938           435504
-deflectooor                                 10886           520116
+Karmatron                                   10859           678058
 basepi                                      10637           744851
+jibarbosa                                   10628           857100
 Cubox                                       10621           826448
+mecevdimitar                                10609           787318
 michaelrpg                                  10509           739239
+Def9Infinity                                10427           686978
 OIVAS7572                                   10420           995586
+wxt9861                                     10412          1013864
+Garruk                                      10365           706465
 dzjp                                        10343           732529
-Garruk                                      10334           704065
-ols                                         10259           570669
-lbraesch                                    10252           647825
-qoo_charly_cai                              10212           620407
-Naven94                                     10069           503192

docs/binpack.md

@@ -1,42 +0,0 @@
-# Binpack
-
-Binpack is a binary training data storage format designed to take advantage of position chains differing by a single move. Therefore it is very good at compactly storing data generated from real games (as opposed to random positions for example sourced from an opening book).
-
-It is currently implemented through a single header library in `extra/nnue_data_binpack_format.h`.
-
-Below follows a rough description of the format in a BNF-like notation.
-
-```
-[[nodiscard]] std::uint16_t signedToUnsigned(std::int16_t a) {
-    std::uint16_t r;
-    std::memcpy(&r, &a, sizeof(std::uint16_t));
-    if (r & 0x8000) r ^= 0x7FFF; // flip value bits if negative
-    r = (r << 1) | (r >> 15); // store sign bit at bit 0
-    return r;
-}
-
-file := <block>*
-block := BINP<chain>*
-chain := <stem><movetext>
-stem := <pos><move><score><ply_and_result><rule50> (32 bytes)
-pos := https://github.com/Sopel97/nnue_data_compress/blob/master/src/chess/Position.h#L1166 (24 bytes)
-move := https://github.com/Sopel97/nnue_data_compress/blob/master/src/chess/Chess.h#L1044 (2 bytes)
-score := signedToUnsigned(score) (2 bytes, big endian)
-ply_and_result := ply bitwise_or (signedToUnsigned(result) << 14) (2 bytes, big endian)
-rule50 := rule_50_counter (2 bytes, big endian)
-    // this is a small defect from old version,
-    I didn't want to break backwards compatibility. Effectively means that there's
-    one byte left for something else in the future because rule50 always fits in one byte.
-
-movetext := <count><move_and_score>*
-count := number of plies in the movetext (2 bytes, big endian). Can be 0.
-move_and_score := <encoded_move><encoded_score> (~2 bytes)
-encoded_move := oof this one is complicated to explain.
-    https://github.com/Sopel97/nnue_data_compress/blob/master/src/compress_file.cpp#L827.
-    https://github.com/Sopel97/chess_pos_db/blob/master/docs/bcgn/variable_length.md
-
-encoded_score := https://en.wikipedia.org/wiki/Variable-width_encoding
-    with block size of 4 bits + 1 bit for extension bit.
-    Encoded value is signedToUnsigned(-prev_score - current_score)
-    (scores are always seen from the perspective of side to move in <pos>, that's why the '-' before prev_score)
-```

docs/convert.md

@@ -1,18 +0,0 @@
-# Convert
-
-`convert` allows conversion of training data between any of `.plain`, `.bin`, and `.binpack`.
-
-As all commands in stockfish `convert` can be invoked either from command line (as `stockfish.exe convert ...`) or in the interactive prompt.
-
-The syntax of this command is as follows:
-```
-convert from_path to_path [append] [validate]
-```
-
-`from_path` is the path to the file to convert from. The type of the data is deduced based on its extension (one of `.plain`, `.bin`, `.binpack`).
-`to_path` is the path to an output file. The type of the data is deduced from its extension. If the file does not exist it is created.
-
-`append` and `validate` can come in any order and are optional.
-If `append` not specified then the output file will be truncated prior to any writes. If `append` is specified then the converted training data will be appended to the end of the output file.
-
-If `validate` is specified then the conversion will stop on the first illegal move found and a diagnostic will be shown.

docs/generate_training_data.md

@@ -1,67 +0,0 @@
-# generate_training_data
-
-`generate_training_data` command allows generation of training data from self-play in a manner that suits training better than traditional games. It introduces random moves to diversify openings, and fixed depth evaluation.
-
-As all commands in stockfish `generate_training_data` can be invoked either from command line (as `stockfish.exe generate_training_data ...`, but this is not recommended because it's not possible to specify UCI options before `generate_training_data` executes) or in the interactive prompt.
-
-It is recommended to set the `PruneAtShallowDepth` UCI option to `false` as it will increase the quality of fixed depth searches.
-
-It is recommended to keep the `EnableTranspositionTable` UCI option at the default `true` value as it will make the generation process faster without noticably harming the uniformity of the data.
-
-`generate_training_data` takes named parameters in the form of `generate_training_data param_1_name param_1_value param_2_name param_2_value ...`.
-
-Currently the following options are available:
-
-`set_recommended_uci_options` - this is a modifier not a parameter, no value follows it. If specified then some UCI options are set to recommended values.
-
-`depth` - sets minimum and maximum depth of evaluation of each position. Default: 3.
-
-`mindepth` - minimum depth of evaluation of each position. If not specified then the same as `depth`.
-
-`maxdepth` - minimum depth of evaluation of each position. If not specified then the same as `depth`.
-
-`nodes` - the number of nodes to use for evaluation of each position. This number is multiplied by the number of PVs of the current search. This does NOT override the `depth` and `depth2` options. If specified then whichever of depth or nodes limit is reached first applies.
-
-`count` - the number of training data entries to generate. 1 entry == 1 position. If both `count` and `max_time_*` are specified the data generation process will end when any of conditions is fullfilled. Default: 8000000000 (8B).
-
-`max_time_seconds`, `max_time_minutes`, `max_time_hours` - specifies the maximum runtime for the data generation. The data generation will NOT be interrupted while a self-play game is in progress. If both `count` and `max_time_*` are specified the data generation process will end when any of conditions is fullfilled. Default: \~250 years.
-
-`output_file_name` - the name of the file to output to. If the extension is not present or doesn't match the selected training data format the right extension will be appened. Default: generated_kifu
-
-`eval_limit` - evaluations with higher absolute value than this will not be written and will terminate a self-play game. Should not exceed 10000 which is VALUE_KNOWN_WIN, but is only hardcapped at mate in 2 (\~30000). Default: 3000
-
-`random_move_min_ply` - the minimal ply at which a random move may be executed instead of a move chosen by search. Default: 1.
-
-`random_move_max_ply` - the maximal ply at which a random move may be executed instead of a move chosen by search. Default: 24.
-
-`random_move_count` - maximum number of random moves in a single self-play game. Default: 5.
-
-`random_move_like_apery` - either 0 or 1. If 1 then random king moves will be followed by a random king move from the opponent whenever possible with 50% probability. Default: 0.
-
-`random_multi_pv` - the number of PVs used for determining the random move. If not specified then a truly random move will be chosen. If specified then a multiPV search will be performed the random move will be one of the moves chosen by the search.
-
-`random_multi_pv_diff` - Makes the multiPV random move selection consider only moves that are at most `random_multi_pv_diff` worse than the next best move. Default: 30000 (all multiPV moves).
-
-`random_multi_pv_depth` - the depth to use for multiPV search for random move. Default: `depth2`.
-
-`random_multi_pv_nodes` - the maximum number of nodes for a multiPV search for random move. Default: `nodes`.
-
-`write_min_ply` - minimum ply for which the training data entry will be emitted. Default: 16.
-
-`write_max_ply` - maximum ply for which the training data entry will be emitted. Default: 400.
-
-`book` - a path to an opening book to use for the starting positions. Currently only .epd format is supported. If not specified then the starting position is always the standard chess starting position.
-
-`save_every` - the number of training data entries per file. If not specified then there will be always one file. If specified there may be more than one file generated (each having at most `save_every` training data entries) and each file will have a unique number attached.
-
-`random_file_name` - if specified then the output filename will be chosen randomly. Overrides `output_file_name`.
-
-`keep_draws` - either 0 or 1. If 1 then training data from drawn games will be emitted too. Default: 1.
-
-`adjudicate_draws_by_score` - either 0 or 1. If 1 then drawn games will be adjudicated when the score remains 0 for at least 8 plies after ply 80. Default: 1.
-
-`adjudicate_draws_by_insufficient_mating_material` - either 0 or 1. If 1 then position with insufficient material will be adjudicated as draws. Default: 1.
-
-`data_format` - format of the training data to use. Either `bin` or `binpack`. Default: `binpack`.
-
-`seed` - seed for the PRNG. Can be either a number or a string. If it's a string then its hash will be used. If not specified then the current time will be used.

docs/generate_training_data_nonpv.md

@@ -1,41 +0,0 @@
-# generate_training_data_nonpv
-
-`generate_training_data_nonpv` command allows generation of training data from self-play in a manner that suits training better than traditional games. It plays fixed nodes self play games for exploration and records [some of] the evaluated positions. Then rescores them with fixed depth search.
-
-As all commands in stockfish `generate_training_data_nonpv` can be invoked either from command line (as `stockfish.exe generate_training_data_nonpv ...`, but this is not recommended because it's not possible to specify UCI options before `generate_training_data_nonpv` executes) or in the interactive prompt.
-
-It is recommended to set the `PruneAtShallowDepth` UCI option to `false` as it will increase the quality of fixed depth searches.
-
-It is recommended to keep the `EnableTranspositionTable` UCI option at the default `true` value as it will make the generation process faster without noticably harming the uniformity of the data.
-
-`generate_training_data_nonpv` takes named parameters in the form of `generate_training_data_nonpv param_1_name param_1_value param_2_name param_2_value ...`.
-
-Currently the following options are available:
-
-`depth` - the search depth to use for rescoring. Default: 3.
-
-`count` - the number of training data entries to generate. 1 entry == 1 position. Default: 1000000 (1M).
-
-`exploration_min_nodes` - the min number of nodes to use for exploraton during selfplay. Default: 5000.
-
-`exploration_max_nodes` - the max number of nodes to use for exploraton during selfplay. The number of nodes is chosen from a uniform distribution between min and max. Default: 15000.
-
-`exploration_save_rate` - the ratio of positions seen during exploration self play games that are saved for later rescoring. Default: 0.01 (meaning 1 in 100 positions seen during search get saved for rescoring).
-
-`output_file` - the name of the file to output to. If the extension is not present or doesn't match the selected training data format the right extension will be appened. Default: generated_gensfen_nonpv
-
-`eval_limit` - evaluations with higher absolute value than this will not be written and will terminate a self-play game. Should not exceed 10000 which is VALUE_KNOWN_WIN, but is only hardcapped at mate in 2 (\~30000). Default: 4000
-
-`exploration_eval_limit` - same as `eval_limit` but used during exploration with a value from fixed depth search.
-
-`exploration_min_pieces` - the min number of pieces in the self play games to start the fixed depth search. Note that even if there's N pieces on the board the fixed nodes search usually reaches positions with less pieces and they are saved too. Default: 8.
-
-`exploration_max_ply` the max ply for the exploration self play. Default: 200.
-
-`smart_fen_skipping` - this is a flag option. When specified some position that are not good candidates for teaching are removed from the output. This includes positions where the best move is a capture or promotion, and position where a king is in check.
-
-`book` - a path to an opening book to use for the starting positions. Currently only .epd format is supported. If not specified then the starting position is always the standard chess starting position.
-
-`data_format` - format of the training data to use. Either `bin` or `binpack`. Default: `binpack`.
-
-`seed` - seed for the PRNG. Can be either a number or a string. If it's a string then its hash will be used. If not specified then the current time will be used.

docs/stats.md

@@ -1,41 +0,0 @@
-# Stats
-
-`gather_statistics` command allows gathering various statistics from a .bin or a .binpack file. The syntax is `gather_statistics (GROUP)* input_file FILENAME`. There can be many groups specified. Any statistic gatherer that belongs to at least one of the specified groups will be used.
-
-Simplest usage: `stockfish.exe gather_statistics all input_file a.binpack`
-
-Any name that doesn't designate an argument name or is not an argument will be interpreted as a group name.
-
-## Parameters
-
-`input_file` - the path to the .bin or .binpack input file to read
-
-`output_file` - optional path to the output file to write the results too. Results are always written on the console, so if this is specified the results will be written in both places.
-
-`max_count` - the maximum number of positions to process. Default: no limit.
-
-## Groups
-
-`all` - a special group designating all statistics gatherers available.
-
-`position_count` - the total number of positions in the file.
-
-`king`, `king_square_count` - the number of times a king was on each square. Output is layed out as a chessboard, with the 8th rank being the topmost. Separate values for white and black kings.
-
-`move`, `move_from_count` - same as `king_square_count` but for from_sq(move)
-
-`move`, `move_to_count` - same as `king_square_count` but for to_sq(move)
-
-`move`, `move_type` - the number of moves with each type. Includes normal, captures, castling, promotions, enpassant. The groups are not disjoint.
-
-`move`, `moved_piece_type` - the number of times a piece of each type was moved
-
-`piece_count` - the histogram of the number of pieces on the board
-
-`ply_discontinuities` - the number of times the ply jumped by a value different than 1 between two consecutive positions. Usually the number of games.
-
-`material_imbalance` - the histogram of imbalances, with values computed using "simple eval", that is pawn=1, bishop=knight=3, rook=5, queen=9
-
-`results` - the distribution of game results
-
-`endgames_6man` - distribution of endgame configurations for <=6 pieces (including kings)

docs/transform.md

@@ -1,42 +0,0 @@
-# Transform
-
-`transform` command exposes subcommands that perform some specific transformation over data. The call syntax is `transform <subcommand>`. Currently implemented subcommands are listed and described below.
-
-## `nudged_static`
-
-`transform nudged_static` takes named parameters in the form of `transform nudged_static param_1_name param_1_value param_2_name param_2_value ...` and flag parameters which don't require values.
-
-This command goes through positions in the input files and replaces the scores with new ones - generated from static eval - but slightly adjusted based on the scores in the original input file.
-
-Currently the following options are available:
-
-`input_file` - path to the input file. Supports bin and binpack formats. Default: in.binpack.
-
-`output_file` - path to the output file. Supports bin and binpack formats. Default: out.binpack.
-
-`absolute` - states that the adjustment should be bounded by an absolute value. After this token follows the maximum absolute adjustment. Values are always adjusted towards scores in the input file. This is the default mode. Default maximum adjustement: 5.
-
-`relative` - states that the adjustment should be bounded by a value relative in magnitude to the static eval value. After this token follows the maximum relative change - a floating point value greater than 0. For example a value of 0.1 only allows changing the static eval by at most 10% towards the score from the input file.
-
-`interpolate` states that the output score should be a value interpolated between static eval and the score from the input file. After this token follows the interpolation constant `t`. `t` of 0 means that only static eval is used. `t` of 1 means that only score from the input file is used. `t` of 0.5 means that the static eval and input score are averaged. It accepts values outside of range `<0, 1>`, but the usefulness is questionable.
-
-## `rescore`
-
-`transform rescore` takes named parameters in the form of `transform rescore param_1_name param_1_value param_2_name param_2_value ...` and flag parameters which don't require values.
-
-This tool respects the UCI option `Threads` and uses all available threads.
-
-This command takes a path to the input file that is either a .epd file which contains one FEN per line or a .bin or .binpack file and outputs a .bin or .binpack file with these positions rescored with specified depth search.
-
-Currently the following options are available:
-
-`input_file` - path to the input file. Default: in.binpack.
-
-`output_file` - path to the output .bin or .binpack file. The file is opened in append mode. Default: out.binpack.
-
-`depth` - the search depth to use for rescoring. Default: 3.
-
-`keep_moves` - whether to keep moves from the input file if available. Allows to keep compression in .binpack. Default: 1.
-
-`research_count` - number of additional searches of depth N done on the same position before using the eval. Default: 0.
-

docs/validate_training_data.md

@@ -1,12 +0,0 @@
-# validate_training_data
-
-`validate_training_data` allows validation of training data of types `.plain`, `.bin`, and `.binpack`.
-
-As all commands in stockfish `validate_training_data` can be invoked either from command line (as `stockfish.exe validate_training_data ...`) or in the interactive prompt.
-
-The syntax of this command is as follows:
-```
-validate_training_data in_path
-```
-
-`in_path` is the path to the file to validate. The type of the data is deduced based on its extension (one of `.plain`, `.bin`, `.binpack`).

script/README.md

@@ -1,52 +0,0 @@
-# `pgn_to_plain`
-This script converts pgn files into text file to apply `learn convert_bin` command. You need to import [python-chess](https://pypi.org/project/python-chess/) to use this script.
-
-
-    pip install python-chess
-	
-
-# Example of Qhapaq's finetune using `pgn_to_plain`
-
-## Download data
-You can download data from [here](http://rebel13.nl/index.html)
-
-## Convert pgn files
-
-**Important : convert text will be superheavy (approx 200 byte / position)**
-
-    python pgn_to_plain.py --pgn "pgn/*.pgn" --start_ply 1 --output converted_pgn.txt
-
-
-`--pgn` option supports wildcard. When you use pgn files with elo >= 3300, You will get 1.7 GB text file.
-	
-	
-## Convert into training data
-
-
-### Example build command
-
-    make nnue-learn ARCH=x86-64
-
-See `src/Makefile` for detail.
-
-
-### Convert
-
-    ./stockfish
-    learn convert_bin converted_pgn.txt output_file_name pgn_bin.bin
-	learn shuffle pgn_bin.bin
-	
-You also need to prepare validation data for training like following.
-	
-	python pgn_to_plain.py --pgn "pgn/ccrl-40-15-3400.pgn" --start_ply 1 --output ccrl-40-15-3400.txt
-	./stockfish
-    learn convert_bin ccrl-40-15-3400.txt ccrl-40-15-3400_plain.bin
-	
-	
-### Learn
-
-    ./stockfish
-	setoption name Threads value 8
-    learn shuffled_sfen.bin newbob_decay 0.5  validation_set_file_name ccrl-40-15-3400_plain.bin  nn_batch_size 50000 batchsize 1000000 eval_save_interval 8000000 eta 0.05 lambda 0.0 eval_limit 3000 mirror_percentage 0 use_draw_in_training 1
-
-

script/extract_bin.py

@@ -1,42 +0,0 @@
-import sys
-
-ENTRY_SIZE = 40
-NUM_ENTRIES_IN_CHUNK = 1024*1024
-
-def copy(infile, outfile, count, times):
-    if times > 1:
-        outfile.write(infile.read(count*ENTRY_SIZE)*times)
-    else:
-        offset = 0
-        while offset < count:
-            to_read = NUM_ENTRIES_IN_CHUNK if offset + NUM_ENTRIES_IN_CHUNK <= count else count - offset
-
-            outfile.write(infile.read(to_read*ENTRY_SIZE))
-
-            offset += NUM_ENTRIES_IN_CHUNK
-
-def work():
-    filename = sys.argv[1]
-    offset = int(sys.argv[2])
-    count = int(sys.argv[3])
-    times = int(sys.argv[4]) if len(sys.argv) >= 5 else 1
-
-    with open(filename, 'rb') as infile:
-        infile.seek(offset * ENTRY_SIZE)
-        filename_parts = filename.split('.')
-        out_path = '.'.join(filename_parts[:-1]) + '_' + str(offset) + '_' + str(count) + '_' + str(times) + '.' + filename_parts[-1]
-        with open(out_path, 'wb') as outfile:
-            copy(infile, outfile, count, times)
-
-def show_help():
-    print('Usage: python extract_bin.py filename offset count [times]')
-    print('filename - the path to the .bin file to process')
-    print('offset - the number of sfens to skip')
-    print('count - the number of sfens to extract')
-    print('times - the number of times to repeat the extracted sfens. Default = 1')
-    print('The result is saved in a new file named `filename.stem`_`offset`_`count`_`times`.bin')
-
-if len(sys.argv) < 4:
-    show_help()
-else:
-    work()

script/interleave_binpacks.py

@@ -1,86 +0,0 @@
-import struct
-import sys
-import os
-import random
-from pathlib import Path
-
-
-def copy_next_chunk(in_file, out_file):
-    chunk_header = in_file.read(8)
-    assert chunk_header[0:4] == b"BINP"
-    size = struct.unpack("<I", chunk_header[4:])[0]
-
-    out_file.write(chunk_header)
-    data = in_file.read(size)
-    out_file.write(data)
-
-    return size + 8
-
-
-def main():
-    if len(sys.argv) < 4:
-        print("Usage: python interleave_binpacks.py infile1 ... infileN outfile")
-        print("       The output binpack, will contain all data from the input files.")
-        print("       Data is read sequentially from the input, randomly alternating between files.")
-        return
-
-    # open last arg as output file name
-    out_filename = sys.argv[-1]
-    print("outfile: ", out_filename)
-
-    if Path(out_filename).exists():
-        print(
-            "Output path {} already exists. Please specify a path to a file that does not exist.".format(
-                out_filename
-            )
-        )
-        return
-
-    out_file = open(out_filename, "wb")
-
-    # open other args as input file names, and get their sizes
-    in_filenames = []
-    for i in range(1, len(sys.argv) - 1):
-        in_filenames.append(sys.argv[i])
-    print("infiles: ", in_filenames)
-
-    in_files = []
-    in_files_remaining = []
-    for in_filename in in_filenames:
-        in_file = open(in_filename, "rb")
-        in_files.append(in_file)
-        file_size = os.path.getsize(in_filename)
-        in_files_remaining.append(file_size)
-
-    # randomly pick a file, with a probability related to their sizes.
-    # copy from the front and keep track of remaining sizes
-    total_remaining = sum(in_files_remaining)
-    print("Merging {} bytes ".format(total_remaining))
-
-    total_size = 0
-    report_every = 100
-    prev_mib = -report_every
-
-    while total_remaining > 0:
-        where = random.randrange(total_remaining)
-        i = 0
-        while where >= in_files_remaining[i]:
-            where -= in_files_remaining[i]
-            i += 1
-        size = copy_next_chunk(in_files[i], out_file)
-        in_files_remaining[i] -= size
-        total_remaining -= size
-        total_size += size
-        mib = total_size // 1024 // 1024
-        if mib // 100 != prev_mib // 100:
-            print("Copied {} MiB".format(mib))
-            prev_mib = mib
-
-    out_file.close()
-    for in_file in in_files:
-        in_file.close()
-
-    print("Merged  {} bytes".format(total_size))
-
-
-main()

script/pgn_to_plain.py

@@ -1,110 +0,0 @@
-import chess.pgn
-import argparse
-import glob
-import re
-from typing import List
-
-# todo close in c++ tools using pgn-extract
-# https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/help.html#-w
-
-commentRe = re.compile("([+-]*M*[0-9.]*)/([0-9]*)")
-mateRe = re.compile("([+-])M([0-9]*)")
-flip_black = False
-
-def parse_result(result_str:str, board:chess.Board) -> int:
-    if result_str == "1/2-1/2":
-        return 0
-    if result_str == "0-1":
-        if board.turn == chess.WHITE:
-            return -1
-        else:
-            return 1
-    elif result_str == "1-0":
-        if board.turn == chess.WHITE:
-            return 1
-        else:
-            return -1
-    else:
-        print("illegal result", result_str)
-        raise ValueError
-
-def game_sanity_check(game: chess.pgn.Game) -> bool:
-    if not game.headers["Result"] in ["1/2-1/2", "0-1", "1-0"]:
-        print("invalid result", game.headers["Result"])
-        return False
-    return True
-
-def parse_comment_for_score(comment_str: str, board: chess.Board) -> int:
-    global commentRe
-    global mateRe
-    global flip_black
-
-    try:
-      m = commentRe.search(comment_str)
-      if m:
-         score = m.group(1)
-         # depth = int(m.group(2))
-         m = mateRe.search(score)
-         if m:
-            if m.group(1) == "+":
-               score =  32000 - int(m.group(2))
-            else:
-               score = -32000 + int(m.group(2))
-         else:
-            score = int(float(score) * 208) # pawn to SF PawnValueEg
-
-         if flip_black and board.turn == chess.BLACK:
-            score = -score
-      else:
-         score = 0
-    except:
-      score = 0
-
-    return score
-
-def parse_game(game: chess.pgn.Game, writer, start_play: int=1)->None:
-    board: chess.Board = game.board()
-    if not game_sanity_check(game):
-        return
-
-    result: str = game.headers["Result"]
-    ply = 0
-    for node in game.mainline():
-        move = node.move
-        if ply >= start_play:
-            comment: str = node.comment
-            writer.write("fen " + board.fen() + "\n")
-            writer.write("move " + str(move) + "\n")
-            writer.write("score " + str(parse_comment_for_score(comment, board)) + "\n")
-            writer.write("ply " + str(ply)+"\n")
-            writer.write("result " + str(parse_result(result, board)) +"\n")
-            writer.write("e\n")
-        ply += 1
-        board.push(move)
-
-def main():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--pgn", type=str, required=True)
-    parser.add_argument("--start_ply", type=int, default=1)
-    parser.add_argument("--output", type=str, default="plain.txt")
-    parser.add_argument("--flip_black_score", action='store_true', dest='flip_black_score', help="Flip black score. Default: False")
-    args = parser.parse_args()
-
-    global flip_black
-    flip_black = args.flip_black_score
-
-    pgn_files: List[str] = glob.glob(args.pgn)
-    pgn_files = sorted(pgn_files, key=lambda x:float(re.findall("-(\d+).pgn",x)[0] if re.findall("-(\d+).pgn",x) else 0.0))
-    f = open(args.output, 'w')
-    for pgn_file in pgn_files:
-        print("parse", pgn_file)
-        pgn_loader = open(pgn_file)
-        while True:
-            game = chess.pgn.read_game(pgn_loader)
-            if game is None:
-                break
-            parse_game(game, f, args.start_ply)
-    f.close()
-
-if __name__=="__main__":
-    main()

script/shuffle_binpack.py

@@ -1,89 +0,0 @@
-import struct
-import sys
-import os
-import random
-from pathlib import Path
-
-def index_binpack(file):
-    print('Indexing...')
-    index = []
-    offset = 0
-    report_every = 100
-    prev_mib = -report_every
-    while file.peek():
-        chunk_header = file.read(8)
-        assert chunk_header[0:4] == b'BINP'
-        size = struct.unpack('<I', chunk_header[4:])[0]
-        file.seek(size, os.SEEK_CUR)
-        index.append((offset, size + 8))
-        offset += size + 8
-
-        mib = offset // 1024 // 1024
-        if mib // 100 != prev_mib // 100:
-            print('Indexed {} MiB'.format(mib))
-            prev_mib = mib
-
-    return index
-
-def copy_binpack_indexed(in_file, index, out_files):
-    print('Copying...')
-    total_size = 0
-    report_every = 100
-    prev_mib = -report_every
-    nextfile = 0
-    for offset, size in index:
-        in_file.seek(offset, os.SEEK_SET)
-        data = in_file.read(size)
-        assert len(data) == size
-        out_files[nextfile].write(data)
-        nextfile = (nextfile + 1) % len(out_files)
-
-        total_size += size
-        mib = total_size // 1024 // 1024
-        if mib // 100 != prev_mib // 100:
-            print('Copied {} MiB'.format(mib))
-            prev_mib = mib
-
-def main():
-    if len(sys.argv) < 3:
-        print('Usage: python shuffle_binpack.py infile outfile [split_count]')
-        return
-
-    in_filename = sys.argv[1]
-
-    if len(sys.argv) > 3:
-       # split the infile in split_count pieces, creating new outfile names based on the provided name
-       basefile = sys.argv[2]
-       split_count = int(sys.argv[3])
-       base=os.path.splitext(basefile)[0]
-       ext=os.path.splitext(basefile)[1]
-       out_filenames = []
-       for i in range(split_count):
-           out_filenames.append(base+"_{}".format(i)+ext)
-    else:
-       out_filenames = [sys.argv[2]]
-
-    for out_filename in out_filenames:
-      if (Path(out_filename).exists()):
-          print('Output path {} already exists. Please specify a path to a file that does not exist.'.format(out_filename))
-          return
-
-    print(out_filenames)
-
-    in_file = open(in_filename, 'rb')
-    index = index_binpack(in_file)
-
-    print('Shuffling...')
-    random.shuffle(index)
-
-    out_files = []
-    for out_filename in out_filenames:
-        out_files.append(open(out_filename, 'wb'))
-
-    copy_binpack_indexed(in_file, index, out_files)
-
-    in_file.close()
-    for out_file in out_files:
-        out_file.close()
-
-main()

scripts/get_native_properties.sh

@@ -0,0 +1,120 @@
+#!/bin/sh
+
+#
+# Returns properties of the native system.
+# best architecture as supported by the CPU
+# filename of the best binary uploaded as an artifact during CI
+#
+
+# Check if all the given flags are present in the CPU flags list
+check_flags() {
+  for flag; do
+    printf '%s\n' "$flags" | grep -q -w "$flag" || return 1
+  done
+}
+
+# Set the CPU flags list
+# remove underscores and points from flags, e.g. gcc uses avx512vnni, while some cpuinfo can have avx512_vnni, some systems use sse4_1 others sse4.1
+get_flags() {
+  flags=$(awk '/^flags[ \t]*:|^Features[ \t]*:/{gsub(/^flags[ \t]*:[ \t]*|^Features[ \t]*:[ \t]*|[_.]/, ""); line=$0} END{print line}' /proc/cpuinfo)
+}
+
+# Check for gcc march "znver1" or "znver2" https://en.wikichip.org/wiki/amd/cpuid
+check_znver_1_2() {
+  vendor_id=$(awk '/^vendor_id/{print $3; exit}' /proc/cpuinfo)
+  cpu_family=$(awk '/^cpu family/{print $4; exit}' /proc/cpuinfo)
+  [ "$vendor_id" = "AuthenticAMD" ] && [ "$cpu_family" = "23" ] && znver_1_2=true
+}
+
+# Set the file CPU x86_64 architecture
+set_arch_x86_64() {
+  if check_flags 'avx512vnni' 'avx512dq' 'avx512f' 'avx512bw' 'avx512vl'; then
+    true_arch='x86-64-vnni256'
+  elif check_flags 'avx512f' 'avx512bw'; then
+    true_arch='x86-64-avx512'
+  elif [ -z "${znver_1_2+1}" ] && check_flags 'bmi2'; then
+    true_arch='x86-64-bmi2'
+  elif check_flags 'avx2'; then
+    true_arch='x86-64-avx2'
+  elif check_flags 'sse41' && check_flags 'popcnt'; then
+    true_arch='x86-64-sse41-popcnt'
+  else
+    true_arch='x86-64'
+  fi
+}
+
+# Check the system type
+uname_s=$(uname -s)
+uname_m=$(uname -m)
+case $uname_s in
+  'Darwin') # Mac OSX system
+    case $uname_m in
+      'arm64')
+        true_arch='apple-silicon'
+        file_arch='x86-64-sse41-popcnt' # Supported by Rosetta 2
+        ;;
+      'x86_64')
+        flags=$(sysctl -n machdep.cpu.features machdep.cpu.leaf7_features | tr '\n' ' ' | tr '[:upper:]' '[:lower:]' | tr -d '_.')
+        set_arch_x86_64
+        if [ "$true_arch" = 'x86-64-vnni256' ] || [ "$true_arch" = 'x86-64-avx512' ]; then
+           file_arch='x86-64-bmi2'
+        fi
+        ;;
+    esac
+    file_os='macos'
+    file_ext='tar'
+    ;;
+  'Linux') # Linux system
+    get_flags
+    case $uname_m in
+      'x86_64')
+        file_os='ubuntu'
+        check_znver_1_2
+        set_arch_x86_64
+        ;;
+      'i686')
+        file_os='ubuntu'
+        true_arch='x86-32'
+        ;;
+      'aarch64')
+        file_os='android'
+        true_arch='armv8'
+        if check_flags 'asimddp'; then
+          true_arch="$true_arch-dotprod"
+        fi
+        ;;
+      'armv7'*)
+        file_os='android'
+        true_arch='armv7'
+        if check_flags 'neon'; then
+          true_arch="$true_arch-neon"
+        fi
+        ;;
+      *) # Unsupported machine type, exit with error
+        printf 'Unsupported machine type: %s\n' "$uname_m"
+        exit 1
+        ;;
+    esac
+    file_ext='tar'
+    ;;
+  'CYGWIN'*|'MINGW'*|'MSYS'*) # Windows system with POSIX compatibility layer
+    get_flags
+    check_znver_1_2
+    set_arch_x86_64
+    file_os='windows'
+    file_ext='zip'
+    ;;
+  *)
+    # Unknown system type, exit with error
+    printf 'Unsupported system type: %s\n' "$uname_s"
+    exit 1
+    ;;
+esac
+
+if [ -z "$file_arch" ]; then
+  file_arch=$true_arch
+fi
+
+file_name="stockfish-$file_os-$file_arch.$file_ext"
+
+printf '%s %s\n' "$true_arch" "$file_name"

src/Makefile

@@ -1,5 +1,5 @@
 # Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-# Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+# Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 #
 # Stockfish is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,9 +20,9 @@
 ### ==========================================================================
 
 ### Establish the operating system name
-KERNEL = $(shell uname -s)
+KERNEL := $(shell uname -s)
 ifeq ($(KERNEL),Linux)
-	OS = $(shell uname -o)
+	OS := $(shell uname -o)
 endif
 
 ### Target Windows OS
@@ -33,7 +33,7 @@ ifeq ($(OS),Windows_NT)
 else ifeq ($(COMP),mingw)
 	target_windows = yes
 	ifeq ($(WINE_PATH),)
-		WINE_PATH = $(shell which wine)
+		WINE_PATH := $(shell which wine)
 	endif
 endif
 
@@ -44,74 +44,62 @@ else
 	EXE = stockfish
 endif
 
-### Establish the operating system name
-KERNEL = $(shell uname -s)
-ifeq ($(KERNEL),Linux)
-	OS = $(shell uname -o)
-endif
-
 ### Installation dir definitions
 PREFIX = /usr/local
 BINDIR = $(PREFIX)/bin
 
 ### Built-in benchmark for pgo-builds
-ifeq ($(SDE_PATH),)
+PGOBENCH = $(WINE_PATH) ./$(EXE) bench
-	PGOBENCH = $(WINE_PATH) ./$(EXE) bench
-	PGOGENSFEN = $(WINE_PATH) ./$(EXE) gensfen depth 3 loop 1000 sfen_format bin output_file_name $(PGO_TRAINING_DATA_FILE)
-else
-	PGOBENCH = $(SDE_PATH) -- $(WINE_PATH) ./$(EXE) bench
-	PGOGENSFEN = $(SDE_PATH) -- $(WINE_PATH) ./$(EXE) gensfen depth 3 loop 1000 sfen_format bin output_file_name $(PGO_TRAINING_DATA_FILE)
-endif
 
 ### Source and object files
-SRCS = benchmark.cpp bitbase.cpp bitboard.cpp endgame.cpp evaluate.cpp main.cpp \
+SRCS = benchmark.cpp bitboard.cpp evaluate.cpp main.cpp \
-	material.cpp misc.cpp movegen.cpp movepick.cpp pawns.cpp position.cpp psqt.cpp \
+	misc.cpp movegen.cpp movepick.cpp position.cpp \
 	search.cpp thread.cpp timeman.cpp tt.cpp uci.cpp ucioption.cpp tune.cpp syzygy/tbprobe.cpp \
-	nnue/evaluate_nnue.cpp \
+	nnue/evaluate_nnue.cpp nnue/features/half_ka_v2_hm.cpp
-	nnue/features/half_ka_v2_hm.cpp \
+
-	tools/validate_training_data.cpp \
+HEADERS = benchmark.h bitboard.h evaluate.h misc.h movegen.h movepick.h \
-	tools/sfen_packer.cpp \
+		nnue/evaluate_nnue.h nnue/features/half_ka_v2_hm.h nnue/layers/affine_transform.h \
-	tools/training_data_generator.cpp \
+		nnue/layers/affine_transform_sparse_input.h nnue/layers/clipped_relu.h nnue/layers/simd.h \
-	tools/training_data_generator_nonpv.cpp \
+		nnue/layers/sqr_clipped_relu.h nnue/nnue_accumulator.h nnue/nnue_architecture.h \
-	tools/opening_book.cpp \
+		nnue/nnue_common.h nnue/nnue_feature_transformer.h position.h \
-	tools/convert.cpp \
+		search.h syzygy/tbprobe.h thread.h thread_win32_osx.h timeman.h \
-	tools/transform.cpp \
+		tt.h tune.h types.h uci.h ucioption.h perft.h
-	tools/stats.cpp
 
 OBJS = $(notdir $(SRCS:.cpp=.o))
 
-VPATH = syzygy:nnue:nnue/features:eval:extra:tools
+VPATH = syzygy:nnue:nnue/features
 
 ### ==========================================================================
 ### Section 2. High-level Configuration
 ### ==========================================================================
 #
-# flag                --- Comp switch      --- Description
+# flag                --- Comp switch        --- Description
 # ----------------------------------------------------------------------------
 #
-# debug = yes/no      --- -DNDEBUG         --- Enable/Disable debug mode
+# debug = yes/no      --- -DNDEBUG           --- Enable/Disable debug mode
 # sanitize = none/<sanitizer> ... (-fsanitize )
-#                     --- ( undefined )    --- enable undefined behavior checks
+#                     --- ( undefined )      --- enable undefined behavior checks
-#                     --- ( thread    )    --- enable threading error checks
+#                     --- ( thread    )      --- enable threading error checks
-#                     --- ( address   )    --- enable memory access checks
+#                     --- ( address   )      --- enable memory access checks
-#                     --- ...etc...        --- see compiler documentation for supported sanitizers
+#                     --- ...etc...          --- see compiler documentation for supported sanitizers
-# optimize = yes/no   --- (-O3/-fast etc.) --- Enable/Disable optimizations
+# optimize = yes/no   --- (-O3/-fast etc.)   --- Enable/Disable optimizations
-# arch = (name)       --- (-arch)          --- Target architecture
+# arch = (name)       --- (-arch)            --- Target architecture
-# bits = 64/32        --- -DIS_64BIT       --- 64-/32-bit operating system
+# bits = 64/32        --- -DIS_64BIT         --- 64-/32-bit operating system
-# prefetch = yes/no   --- -DUSE_PREFETCH   --- Use prefetch asm-instruction
+# prefetch = yes/no   --- -DUSE_PREFETCH     --- Use prefetch asm-instruction
-# popcnt = yes/no     --- -DUSE_POPCNT     --- Use popcnt asm-instruction
+# popcnt = yes/no     --- -DUSE_POPCNT       --- Use popcnt asm-instruction
-# pext = yes/no       --- -DUSE_PEXT       --- Use pext x86_64 asm-instruction
+# pext = yes/no       --- -DUSE_PEXT         --- Use pext x86_64 asm-instruction
-# sse = yes/no        --- -msse            --- Use Intel Streaming SIMD Extensions
+# sse = yes/no        --- -msse              --- Use Intel Streaming SIMD Extensions
-# mmx = yes/no        --- -mmmx            --- Use Intel MMX instructions
+# mmx = yes/no        --- -mmmx              --- Use Intel MMX instructions
-# sse2 = yes/no       --- -msse2           --- Use Intel Streaming SIMD Extensions 2
+# sse2 = yes/no       --- -msse2             --- Use Intel Streaming SIMD Extensions 2
-# ssse3 = yes/no      --- -mssse3          --- Use Intel Supplemental Streaming SIMD Extensions 3
+# ssse3 = yes/no      --- -mssse3            --- Use Intel Supplemental Streaming SIMD Extensions 3
-# sse41 = yes/no      --- -msse4.1         --- Use Intel Streaming SIMD Extensions 4.1
+# sse41 = yes/no      --- -msse4.1           --- Use Intel Streaming SIMD Extensions 4.1
-# avx2 = yes/no       --- -mavx2           --- Use Intel Advanced Vector Extensions 2
+# avx2 = yes/no       --- -mavx2             --- Use Intel Advanced Vector Extensions 2
-# avxvnni = yes/no    --- -mavxvnni        --- Use Intel Vector Neural Network Instructions AVX
+# avxvnni = yes/no    --- -mavxvnni          --- Use Intel Vector Neural Network Instructions AVX
-# avx512 = yes/no     --- -mavx512bw       --- Use Intel Advanced Vector Extensions 512
+# avx512 = yes/no     --- -mavx512bw         --- Use Intel Advanced Vector Extensions 512
-# vnni256 = yes/no    --- -mavx512vnni     --- Use Intel Vector Neural Network Instructions 256
+# vnni256 = yes/no    --- -mavx256vnni       --- Use Intel Vector Neural Network Instructions 512 with 256bit operands
-# vnni512 = yes/no    --- -mavx512vnni     --- Use Intel Vector Neural Network Instructions 512
+# vnni512 = yes/no    --- -mavx512vnni       --- Use Intel Vector Neural Network Instructions 512
-# neon = yes/no       --- -DUSE_NEON       --- Use ARM SIMD architecture
+# neon = yes/no       --- -DUSE_NEON         --- Use ARM SIMD architecture
+# dotprod = yes/no    --- -DUSE_NEON_DOTPROD --- Use ARM advanced SIMD Int8 dot product instructions
 #
 # Note that Makefile is space sensitive, so when adding new architectures
 # or modifying existing flags, you have to make sure there are no extra spaces
@@ -124,16 +112,20 @@ VPATH = syzygy:nnue:nnue/features:eval:extra:tools
 ### 2.1. General and architecture defaults
 
 ifeq ($(ARCH),)
-   ARCH = x86-64-modern
+   ARCH = native
-   help_skip_sanity = yes
 endif
+
+ifeq ($(ARCH), native)
+   override ARCH := $(shell $(SHELL) ../scripts/get_native_properties.sh | cut -d " " -f 1)
+endif
+
 # explicitly check for the list of supported architectures (as listed with make help),
 # the user can override with `make ARCH=x86-32-vnni256 SUPPORTED_ARCH=true`
 ifeq ($(ARCH), $(filter $(ARCH), \
                  x86-64-vnni512 x86-64-vnni256 x86-64-avx512 x86-64-avxvnni x86-64-bmi2 \
                  x86-64-avx2 x86-64-sse41-popcnt x86-64-modern x86-64-ssse3 x86-64-sse3-popcnt \
                  x86-64 x86-32-sse41-popcnt x86-32-sse2 x86-32 ppc-64 ppc-32 e2k \
-                 armv7 armv7-neon armv8 apple-silicon general-64 general-32))
+                 armv7 armv7-neon armv8 armv8-dotprod apple-silicon general-64 general-32 riscv64 loongarch64))
    SUPPORTED_ARCH=true
 else
    SUPPORTED_ARCH=false
@@ -157,9 +149,16 @@ avx512 = no
 vnni256 = no
 vnni512 = no
 neon = no
+dotprod = no
 arm_version = 0
 STRIP = strip
 
+ifneq ($(shell which clang-format-17 2> /dev/null),)
+	CLANG-FORMAT = clang-format-17
+else
+	CLANG-FORMAT = clang-format
+endif
+
 ### 2.2 Architecture specific
 
 ifeq ($(findstring x86,$(ARCH)),x86)
@@ -208,6 +207,8 @@ ifeq ($(findstring -sse41,$(ARCH)),-sse41)
 endif
 
 ifeq ($(findstring -modern,$(ARCH)),-modern)
+        $(warning *** ARCH=$(ARCH) is deprecated, defaulting to ARCH=x86-64-sse41-popcnt. Execute `make help` for a list of available architectures. ***)
+        $(shell sleep 5)
 	popcnt = yes
 	sse = yes
 	sse2 = yes
@@ -325,11 +326,21 @@ ifeq ($(ARCH),armv8)
 	arm_version = 8
 endif
 
+ifeq ($(ARCH),armv8-dotprod)
+	arch = armv8
+	prefetch = yes
+	popcnt = yes
+	neon = yes
+	dotprod = yes
+	arm_version = 8
+endif
+
 ifeq ($(ARCH),apple-silicon)
 	arch = arm64
 	prefetch = yes
 	popcnt = yes
 	neon = yes
+	dotprod = yes
 	arm_version = 8
 endif
 
@@ -355,8 +366,16 @@ ifeq ($(findstring e2k,$(ARCH)),e2k)
 	popcnt = yes
 endif
 
+ifeq ($(ARCH),riscv64)
+	arch = riscv64
 endif
 
+ifeq ($(ARCH),loongarch64)
+	arch = loongarch64
+endif
+endif
+
+
 ### ==========================================================================
 ### Section 3. Low-level Configuration
 ### ==========================================================================
@@ -368,9 +387,8 @@ ifeq ($(MAKELEVEL),0)
        export ENV_LDFLAGS := $(LDFLAGS)
 endif
 
-ADDITIONAL_INCLUDE_DIRECTORIES = -I.
+CXXFLAGS = $(ENV_CXXFLAGS) -Wall -Wcast-qual -fno-exceptions -std=c++17 $(EXTRACXXFLAGS)
-CXXFLAGS = $(ENV_CXXFLAGS) -Wall -Wcast-qual -fno-exceptions -std=c++17 $(ADDITIONAL_INCLUDE_DIRECTORIES) $(EXTRACXXFLAGS)
+DEPENDFLAGS = $(ENV_DEPENDFLAGS) -std=c++17
-DEPENDFLAGS = $(ENV_DEPENDFLAGS) -std=c++17 $(ADDITIONAL_INCLUDE_DIRECTORIES)
 LDFLAGS = $(ENV_LDFLAGS) $(EXTRALDFLAGS)
 
 ifeq ($(COMP),)
@@ -380,13 +398,18 @@ endif
 ifeq ($(COMP),gcc)
 	comp=gcc
 	CXX=g++
-	CXXFLAGS += -pedantic -Wextra -Wshadow -lstdc++fs
+	CXXFLAGS += -pedantic -Wextra -Wshadow -Wmissing-declarations
 
-	ifeq ($(arch),$(filter $(arch),armv7 armv8))
+	ifeq ($(arch),$(filter $(arch),armv7 armv8 riscv64))
 		ifeq ($(OS),Android)
 			CXXFLAGS += -m$(bits)
 			LDFLAGS += -m$(bits)
 		endif
+		ifeq ($(ARCH),riscv64)
+			CXXFLAGS += -latomic
+		endif
+	else ifeq ($(ARCH),loongarch64)
+		CXXFLAGS += -latomic
 	else
 		CXXFLAGS += -m$(bits)
 		LDFLAGS += -m$(bits)
@@ -421,13 +444,14 @@ ifeq ($(COMP),mingw)
 			CXX=i686-w64-mingw32-c++-posix
 		endif
 	endif
-	CXXFLAGS += -pedantic -Wextra -Wshadow
+	CXXFLAGS += -pedantic -Wextra -Wshadow -Wmissing-declarations
 endif
 
-ifeq ($(COMP),icc)
+ifeq ($(COMP),icx)
-	comp=icc
+	comp=icx
-	CXX=icpc
+	CXX=icpx
-	CXXFLAGS += -diag-disable 1476,10120 -Wcheck -Wabi -Wdeprecated -strict-ansi
+	CXXFLAGS += --intel -pedantic -Wextra -Wshadow -Wmissing-prototypes \
+		-Wconditional-uninitialized -Wabi -Wdeprecated
 endif
 
 ifeq ($(COMP),clang)
@@ -437,7 +461,8 @@ ifeq ($(COMP),clang)
 		CXX=x86_64-w64-mingw32-clang++
 	endif
 
-	CXXFLAGS += -pedantic -Wextra -Wshadow
+	CXXFLAGS += -pedantic -Wextra -Wshadow -Wmissing-prototypes \
+	            -Wconditional-uninitialized
 
 	ifeq ($(filter $(KERNEL),Darwin OpenBSD FreeBSD),)
 	ifeq ($(target_windows),)
@@ -447,11 +472,16 @@ ifeq ($(COMP),clang)
 	endif
 	endif
 
-	ifeq ($(arch),$(filter $(arch),armv7 armv8))
+	ifeq ($(arch),$(filter $(arch),armv7 armv8 riscv64))
 		ifeq ($(OS),Android)
 			CXXFLAGS += -m$(bits)
 			LDFLAGS += -m$(bits)
 		endif
+		ifeq ($(ARCH),riscv64)
+			CXXFLAGS += -latomic
+		endif
+	else ifeq ($(ARCH),loongarch64)
+		CXXFLAGS += -latomic
 	else
 		CXXFLAGS += -m$(bits)
 		LDFLAGS += -m$(bits)
@@ -459,8 +489,8 @@ ifeq ($(COMP),clang)
 endif
 
 ifeq ($(KERNEL),Darwin)
-	CXXFLAGS += -mmacosx-version-min=10.15
+	CXXFLAGS += -mmacosx-version-min=10.14
-	LDFLAGS += -mmacosx-version-min=10.15
+	LDFLAGS += -mmacosx-version-min=10.14
 	ifneq ($(arch),any)
 		CXXFLAGS += -arch $(arch)
 		LDFLAGS += -arch $(arch)
@@ -491,12 +521,20 @@ ifeq ($(COMP),ndk)
 			STRIP=llvm-strip
 		endif
 	endif
+	ifeq ($(arch),x86_64)
+		CXX=x86_64-linux-android21-clang++
+		ifneq ($(shell which x86_64-linux-android-strip 2>/dev/null),)
+			STRIP=x86_64-linux-android-strip
+		else
+			STRIP=llvm-strip
+		endif
+	endif
 	LDFLAGS += -static-libstdc++ -pie -lm -latomic
 endif
 
-ifeq ($(comp),icc)
+ifeq ($(comp),icx)
-	profile_make = icc-profile-make
+	profile_make = icx-profile-make
-	profile_use = icc-profile-use
+	profile_use = icx-profile-use
 else ifeq ($(comp),clang)
 	profile_make = clang-profile-make
 	profile_use = clang-profile-use
@@ -520,8 +558,8 @@ endif
 
 ### Sometimes gcc is really clang
 ifeq ($(COMP),gcc)
-	gccversion = $(shell $(CXX) --version)
+	gccversion := $(shell $(CXX) --version 2>/dev/null)
-	gccisclang = $(findstring clang,$(gccversion))
+	gccisclang := $(findstring clang,$(gccversion))
 	ifneq ($(gccisclang),)
 		profile_make = clang-profile-make
 		profile_use = clang-profile-use
@@ -536,13 +574,13 @@ ifneq ($(comp),mingw)
 		# Haiku has pthreads in its libroot, so only link it in on other platforms
 		ifneq ($(KERNEL),Haiku)
 			ifneq ($(COMP),ndk)
-			LDFLAGS += -lpthread
+				LDFLAGS += -lpthread
+			endif
 		endif
 	endif
 endif
-endif
 
-### 3.2.2 Debugging
+### 3.2.1 Debugging
 ifeq ($(debug),no)
 	CXXFLAGS += -DNDEBUG
 else
@@ -558,7 +596,7 @@ endif
 ### 3.3 Optimization
 ifeq ($(optimize),yes)
 
-	CXXFLAGS += -O3
+	CXXFLAGS += -O3 -funroll-loops
 
 	ifeq ($(comp),gcc)
 		ifeq ($(OS), Android)
@@ -567,7 +605,7 @@ ifeq ($(optimize),yes)
 	endif
 
 	ifeq ($(KERNEL),Darwin)
-		ifeq ($(comp),$(filter $(comp),clang icc))
+		ifeq ($(comp),$(filter $(comp),clang icx))
 			CXXFLAGS += -mdynamic-no-pic
 		endif
 
@@ -579,7 +617,10 @@ ifeq ($(optimize),yes)
 	endif
 
 	ifeq ($(comp),clang)
-		CXXFLAGS += -fexperimental-new-pass-manager
+		clangmajorversion := $(shell $(CXX) -dumpversion 2>/dev/null | cut -f1 -d.)
+		ifeq ($(shell expr $(clangmajorversion) \< 16),1)
+			CXXFLAGS += -fexperimental-new-pass-manager
+		endif
 	endif
 endif
 
@@ -600,8 +641,6 @@ endif
 ifeq ($(popcnt),yes)
 	ifeq ($(arch),$(filter $(arch),ppc64 armv7 armv8 arm64))
 		CXXFLAGS += -DUSE_POPCNT
-	else ifeq ($(comp),icc)
-		CXXFLAGS += -msse3 -DUSE_POPCNT
 	else
 		CXXFLAGS += -msse3 -mpopcnt -DUSE_POPCNT
 	endif
@@ -610,63 +649,62 @@ endif
 ### 3.6 SIMD architectures
 ifeq ($(avx2),yes)
 	CXXFLAGS += -DUSE_AVX2
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
-		CXXFLAGS += -mavx2
+		CXXFLAGS += -mavx2 -mbmi
 	endif
 endif
 
 ifeq ($(avxvnni),yes)
 	CXXFLAGS += -DUSE_VNNI -DUSE_AVXVNNI
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -mavxvnni
 	endif
 endif
 
 ifeq ($(avx512),yes)
 	CXXFLAGS += -DUSE_AVX512
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -mavx512f -mavx512bw
 	endif
 endif
 
 ifeq ($(vnni256),yes)
 	CXXFLAGS += -DUSE_VNNI
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -mavx512f -mavx512bw -mavx512vnni -mavx512dq -mavx512vl -mprefer-vector-width=256
 	endif
 endif
 
 ifeq ($(vnni512),yes)
 	CXXFLAGS += -DUSE_VNNI
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
-		CXXFLAGS += -mavx512vnni -mavx512dq -mavx512vl
+		CXXFLAGS += -mavx512f -mavx512bw -mavx512vnni -mavx512dq -mavx512vl -mprefer-vector-width=512
 	endif
 endif
 
 ifeq ($(sse41),yes)
 	CXXFLAGS += -DUSE_SSE41
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -msse4.1
 	endif
 endif
 
 ifeq ($(ssse3),yes)
 	CXXFLAGS += -DUSE_SSSE3
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -mssse3
 	endif
 endif
 
 ifeq ($(sse2),yes)
 	CXXFLAGS += -DUSE_SSE2
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -msse2
 	endif
 endif
 
 ifeq ($(mmx),yes)
-	CXXFLAGS += -DUSE_MMX
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
 		CXXFLAGS += -mmmx
 	endif
 endif
@@ -679,24 +717,48 @@ ifeq ($(neon),yes)
 		CXXFLAGS += -mfpu=neon
 	endif
 	endif
+	endif
 endif
+
+ifeq ($(dotprod),yes)
+	CXXFLAGS += -march=armv8.2-a+dotprod -DUSE_NEON_DOTPROD
 endif
 
 ### 3.7 pext
 ifeq ($(pext),yes)
 	CXXFLAGS += -DUSE_PEXT
-	ifeq ($(comp),$(filter $(comp),gcc clang mingw))
+	ifeq ($(comp),$(filter $(comp),gcc clang mingw icx))
 		CXXFLAGS += -mbmi2
 	endif
 endif
 
-### 3.8 Link Time Optimization
+### 3.8.1 Try to include git commit sha for versioning
+GIT_SHA := $(shell git rev-parse HEAD 2>/dev/null | cut -c 1-8)
+ifneq ($(GIT_SHA), )
+	CXXFLAGS += -DGIT_SHA=$(GIT_SHA)
+endif
+
+### 3.8.2 Try to include git commit date for versioning
+GIT_DATE := $(shell git show -s --date=format:'%Y%m%d' --format=%cd HEAD 2>/dev/null)
+ifneq ($(GIT_DATE), )
+	CXXFLAGS += -DGIT_DATE=$(GIT_DATE)
+endif
+
+### 3.8.3 Try to include architecture
+ifneq ($(ARCH), )
+	CXXFLAGS += -DARCH=$(ARCH)
+endif
+
+### 3.9 Link Time Optimization
 ### This is a mix of compile and link time options because the lto link phase
 ### needs access to the optimization flags.
 ifeq ($(optimize),yes)
 ifeq ($(debug), no)
-	ifeq ($(comp),clang)
+	ifeq ($(comp),$(filter $(comp),clang icx))
-		CXXFLAGS += -flto
+		CXXFLAGS += -flto=full
+		ifeq ($(comp),icx)
+			CXXFLAGS += -fwhole-program-vtables
+                endif
 		ifeq ($(target_windows),yes)
 			CXXFLAGS += -fuse-ld=lld
 		endif
@@ -706,25 +768,23 @@ ifeq ($(debug), no)
 # GCC on some systems.
 	else ifeq ($(comp),gcc)
 	ifeq ($(gccisclang),)
-		CXXFLAGS += -flto
+		CXXFLAGS += -flto -flto-partition=one
 		LDFLAGS += $(CXXFLAGS) -flto=jobserver
 	else
-		CXXFLAGS += -flto
+		CXXFLAGS += -flto=full
 		LDFLAGS += $(CXXFLAGS)
 	endif
 
 # To use LTO and static linking on Windows,
 # the tool chain requires gcc version 10.1 or later.
 	else ifeq ($(comp),mingw)
-	ifneq ($(arch),i386)
+		CXXFLAGS += -flto -flto-partition=one
-		CXXFLAGS += -flto
 		LDFLAGS += $(CXXFLAGS) -save-temps
 	endif
-	endif
 endif
 endif
 
-### 3.9 Android 5 can only run position independent executables. Note that this
+### 3.10 Android 5 can only run position independent executables. Note that this
 ### breaks Android 4.0 and earlier.
 ifeq ($(OS), Android)
 	CXXFLAGS += -fPIE
@@ -735,81 +795,87 @@ endif
 ### Section 4. Public Targets
 ### ==========================================================================
 
-
 help:
 	@echo ""
 	@echo "To compile stockfish, type: "
 	@echo ""
-	@echo "make target ARCH=arch [COMP=compiler] [COMPCXX=cxx]"
+	@echo "make -j target [ARCH=arch] [COMP=compiler] [COMPCXX=cxx]"
 	@echo ""
 	@echo "Supported targets:"
 	@echo ""
 	@echo "help                    > Display architecture details"
-	@echo "build                   > Standard build"
+	@echo "profile-build           > standard build with profile-guided optimization"
-	@echo "net                     > Download the default nnue net"
+	@echo "build                   > skip profile-guided optimization"
-	@echo "profile-build           > Faster build (with profile-guided optimization)"
+	@echo "net                     > Download the default nnue nets"
 	@echo "strip                   > Strip executable"
 	@echo "install                 > Install executable"
 	@echo "clean                   > Clean up"
 	@echo ""
 	@echo "Supported archs:"
 	@echo ""
-	@echo "x86-64-vnni512          > x86 64-bit with vnni support 512bit wide"
+	@echo "native                  > select the best architecture for the host processor (default)"
-	@echo "x86-64-vnni256          > x86 64-bit with vnni support 256bit wide"
+	@echo "x86-64-vnni512          > x86 64-bit with vnni 512bit support"
+	@echo "x86-64-vnni256          > x86 64-bit with vnni 512bit support, limit operands to 256bit wide"
 	@echo "x86-64-avx512           > x86 64-bit with avx512 support"
-	@echo "x86-64-avxvnni          > x86 64-bit with avxvnni support"
+	@echo "x86-64-avxvnni          > x86 64-bit with vnni 256bit support"
 	@echo "x86-64-bmi2             > x86 64-bit with bmi2 support"
 	@echo "x86-64-avx2             > x86 64-bit with avx2 support"
 	@echo "x86-64-sse41-popcnt     > x86 64-bit with sse41 and popcnt support"
-	@echo "x86-64-modern           > common modern CPU, currently x86-64-sse41-popcnt"
+	@echo "x86-64-modern           > deprecated, currently x86-64-sse41-popcnt"
 	@echo "x86-64-ssse3            > x86 64-bit with ssse3 support"
-	@echo "x86-64-sse3-popcnt      > x86 64-bit with sse3 and popcnt support"
+	@echo "x86-64-sse3-popcnt      > x86 64-bit with sse3 compile and popcnt support"
 	@echo "x86-64                  > x86 64-bit generic (with sse2 support)"
 	@echo "x86-32-sse41-popcnt     > x86 32-bit with sse41 and popcnt support"
 	@echo "x86-32-sse2             > x86 32-bit with sse2 support"
-	@echo "x86-32                  > x86 32-bit generic (with mmx and sse support)"
+	@echo "x86-32                  > x86 32-bit generic (with mmx compile support)"
 	@echo "ppc-64                  > PPC 64-bit"
 	@echo "ppc-32                  > PPC 32-bit"
 	@echo "armv7                   > ARMv7 32-bit"
 	@echo "armv7-neon              > ARMv7 32-bit with popcnt and neon"
 	@echo "armv8                   > ARMv8 64-bit with popcnt and neon"
+	@echo "armv8-dotprod           > ARMv8 64-bit with popcnt, neon and dot product support"
 	@echo "e2k                     > Elbrus 2000"
 	@echo "apple-silicon           > Apple silicon ARM64"
 	@echo "general-64              > unspecified 64-bit"
 	@echo "general-32              > unspecified 32-bit"
+	@echo "riscv64                 > RISC-V 64-bit"
+	@echo "loongarch64             > LoongArch 64-bit"
 	@echo ""
 	@echo "Supported compilers:"
 	@echo ""
-	@echo "gcc                     > Gnu compiler (default)"
+	@echo "gcc                     > GNU compiler (default)"
-	@echo "mingw                   > Gnu compiler with MinGW under Windows"
+	@echo "mingw                   > GNU compiler with MinGW under Windows"
 	@echo "clang                   > LLVM Clang compiler"
-	@echo "icc                     > Intel compiler"
+	@echo "icx                     > Intel oneAPI DPC++/C++ Compiler"
 	@echo "ndk                     > Google NDK to cross-compile for Android"
 	@echo ""
-	@echo "Simple examples. If you don't know what to do, you likely want to run: "
+	@echo "Simple examples. If you don't know what to do, you likely want to run one of: "
 	@echo ""
-	@echo "make -j build ARCH=x86-64  (A portable, slow compile for 64-bit systems)"
+	@echo "make -j profile-build ARCH=x86-64-avx2    # typically a fast compile for common systems "
-	@echo "make -j build ARCH=x86-32  (A portable, slow compile for 32-bit systems)"
+	@echo "make -j profile-build ARCH=x86-64-sse41-popcnt  # A more portable compile for 64-bit systems "
+	@echo "make -j profile-build ARCH=x86-64         # A portable compile for 64-bit systems "
 	@echo ""
-	@echo "Advanced examples, for experienced users looking for performance: "
+	@echo "Advanced examples, for experienced users: "
 	@echo ""
-	@echo "make    help  ARCH=x86-64-bmi2"
+	@echo "make -j profile-build ARCH=x86-64-avxvnni"
-	@echo "make -j profile-build ARCH=x86-64-bmi2 COMP=gcc COMPCXX=g++-9.0"
+	@echo "make -j profile-build ARCH=x86-64-avxvnni COMP=gcc COMPCXX=g++-12.0"
 	@echo "make -j build ARCH=x86-64-ssse3 COMP=clang"
 	@echo ""
-	@echo "-------------------------------"
+ifneq ($(SUPPORTED_ARCH), true)
-ifeq ($(SUPPORTED_ARCH)$(help_skip_sanity), true)
-	@echo "The selected architecture $(ARCH) will enable the following configuration: "
-	@$(MAKE) ARCH=$(ARCH) COMP=$(COMP) config-sanity
-else
 	@echo "Specify a supported architecture with the ARCH option for more details"
 	@echo ""
 endif
 
 
-.PHONY: help build profile-build strip install clean net objclean profileclean \
+.PHONY: help analyze build profile-build strip install clean net \
-        config-sanity icc-profile-use icc-profile-make gcc-profile-use gcc-profile-make \
+	objclean profileclean config-sanity \
-        clang-profile-use clang-profile-make
+	icx-profile-use icx-profile-make \
+	gcc-profile-use gcc-profile-make \
+	clang-profile-use clang-profile-make FORCE \
+	format analyze
+
+analyze: net config-sanity objclean
+	$(MAKE) -k ARCH=$(ARCH) COMP=$(COMP) $(OBJS)
 
 build: net config-sanity
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) all
@@ -820,8 +886,8 @@ profile-build: net config-sanity objclean profileclean
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) $(profile_make)
 	@echo ""
 	@echo "Step 2/4. Running benchmark for pgo-build ..."
-	$(PGOBENCH) 2>&1 | tail -n 4
+	$(PGOBENCH) > PGOBENCH.out 2>&1
-	$(PGOGENSFEN) 2>&1 | tail -n 4
+	tail -n 4 PGOBENCH.out
 	@echo ""
 	@echo "Step 3/4. Building optimized executable ..."
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) objclean
@@ -842,44 +908,78 @@ install:
 clean: objclean profileclean
 	@rm -f .depend *~ core
 
-# evaluation network (nnue)
-net:
-	$(eval nnuenet := $(shell grep EvalFileDefaultName evaluate.h | grep define | sed 's/.*\(nn-[a-z0-9]\{12\}.nnue\).*/\1/'))
-	@echo "Default net: $(nnuenet)"
-	$(eval nnuedownloadurl := https://tests.stockfishchess.org/api/nn/$(nnuenet))
-	$(eval curl_or_wget := $(shell if hash curl 2>/dev/null; then echo "curl -skL"; elif hash wget 2>/dev/null; then echo "wget -qO-"; fi))
-	@if test -f "$(nnuenet)"; then \
-		echo "Already available."; \
-	else \
-		if [ "x$(curl_or_wget)" = "x" ]; then \
-			echo "Automatic download failed: neither curl nor wget is installed. Install one of these tools or download the net manually"; exit 1; \
-		else \
-			echo "Downloading $(nnuedownloadurl)"; $(curl_or_wget) $(nnuedownloadurl) > $(nnuenet);\
-		fi; \
-	fi;
-	$(eval shasum_command := $(shell if hash shasum 2>/dev/null; then echo "shasum -a 256 "; elif hash sha256sum 2>/dev/null; then echo "sha256sum "; fi))
-	@if [ "x$(shasum_command)" != "x" ]; then \
-		if [ "$(nnuenet)" != "nn-"`$(shasum_command) $(nnuenet) | cut -c1-12`".nnue" ]; then \
-			echo "Failed download or $(nnuenet) corrupted, please delete!"; exit 1; \
-		fi \
-	else \
-		echo "shasum / sha256sum not found, skipping net validation"; \
-	fi
-
 # clean binaries and objects
 objclean:
-	@rm -f $(EXE) *.o ./syzygy/*.o ./nnue/*.o ./nnue/features/*.o ./tools/*.o ./extra/*.o ./eval/*.o
+	@rm -f stockfish stockfish.exe *.o ./syzygy/*.o ./nnue/*.o ./nnue/features/*.o
 
 # clean auxiliary profiling files
 profileclean:
 	@rm -rf profdir
-	@rm -f bench.txt *.gcda *.gcno ./syzygy/*.gcda ./nnue/*.gcda ./nnue/features/*.gcda *.s ./tools/*.gcda ./extra/*.gcda ./eval/*.gcda
+	@rm -f bench.txt *.gcda *.gcno ./syzygy/*.gcda ./nnue/*.gcda ./nnue/features/*.gcda *.s PGOBENCH.out
 	@rm -f stockfish.profdata *.profraw
 	@rm -f stockfish.*args*
 	@rm -f stockfish.*lt*
 	@rm -f stockfish.res
 	@rm -f ./-lstdc++.res
 
+define fetch_network
+	@echo "Default net: $(nnuenet)"
+	@if [ "x$(curl_or_wget)" = "x" ]; then \
+		echo "Neither curl nor wget is installed. Install one of these tools unless the net has been downloaded manually"; \
+	fi
+	@if [ "x$(shasum_command)" = "x" ]; then \
+		echo "shasum / sha256sum not found, skipping net validation"; \
+	elif test -f "$(nnuenet)"; then \
+		if [ "$(nnuenet)" != "nn-"`$(shasum_command) $(nnuenet) | cut -c1-12`".nnue" ]; then \
+			echo "Removing invalid network"; rm -f $(nnuenet); \
+		fi; \
+	fi;
+	@for nnuedownloadurl in "$(nnuedownloadurl1)" "$(nnuedownloadurl2)"; do \
+		if test -f "$(nnuenet)"; then \
+			echo "$(nnuenet) available : OK"; break; \
+		else \
+			if [ "x$(curl_or_wget)" != "x" ]; then \
+				echo "Downloading $${nnuedownloadurl}"; $(curl_or_wget) $${nnuedownloadurl} > $(nnuenet);\
+			else \
+				echo "No net found and download not possible"; exit 1;\
+			fi; \
+		fi; \
+		if [ "x$(shasum_command)" != "x" ]; then \
+			if [ "$(nnuenet)" != "nn-"`$(shasum_command) $(nnuenet) | cut -c1-12`".nnue" ]; then \
+				echo "Removing failed download"; rm -f $(nnuenet); \
+			fi; \
+		fi; \
+	done
+	@if ! test -f "$(nnuenet)"; then \
+		echo "Failed to download $(nnuenet)."; \
+	fi;
+	@if [ "x$(shasum_command)" != "x" ]; then \
+		if [ "$(nnuenet)" = "nn-"`$(shasum_command) $(nnuenet) | cut -c1-12`".nnue" ]; then \
+			echo "Network validated"; break; \
+		fi; \
+	fi;
+endef
+
+# set up shell variables for the net stuff
+define netvariables
+$(eval nnuenet := $(shell grep $(1) evaluate.h | grep define | sed 's/.*\(nn-[a-z0-9]\{12\}.nnue\).*/\1/'))
+$(eval nnuedownloadurl1 := https://tests.stockfishchess.org/api/nn/$(nnuenet))
+$(eval nnuedownloadurl2 := https://github.com/official-stockfish/networks/raw/master/$(nnuenet))
+$(eval curl_or_wget := $(shell if hash curl 2>/dev/null; then echo "curl -skL"; elif hash wget 2>/dev/null; then echo "wget -qO-"; fi))
+$(eval shasum_command := $(shell if hash shasum 2>/dev/null; then echo "shasum -a 256 "; elif hash sha256sum 2>/dev/null; then echo "sha256sum "; fi))
+endef
+
+# evaluation network (nnue)
+net:
+	$(call netvariables, EvalFileDefaultNameBig)
+	$(call fetch_network)
+	$(call netvariables, EvalFileDefaultNameSmall)
+	$(call fetch_network)
+
+format:
+	$(CLANG-FORMAT) -i $(SRCS) $(HEADERS) -style=file
+
+# default target
 default:
 	help
 
@@ -913,7 +1013,9 @@ config-sanity: net
 	@echo "vnni256: '$(vnni256)'"
 	@echo "vnni512: '$(vnni512)'"
 	@echo "neon: '$(neon)'"
+	@echo "dotprod: '$(dotprod)'"
 	@echo "arm_version: '$(arm_version)'"
+	@echo "target_windows: '$(target_windows)'"
 	@echo ""
 	@echo "Flags:"
 	@echo "CXX: $(CXX)"
@@ -927,7 +1029,7 @@ config-sanity: net
 	@test "$(SUPPORTED_ARCH)" = "true"
 	@test "$(arch)" = "any" || test "$(arch)" = "x86_64" || test "$(arch)" = "i386" || \
 	 test "$(arch)" = "ppc64" || test "$(arch)" = "ppc" || test "$(arch)" = "e2k" || \
-	 test "$(arch)" = "armv7" || test "$(arch)" = "armv8" || test "$(arch)" = "arm64"
+	 test "$(arch)" = "armv7" || test "$(arch)" = "armv8" || test "$(arch)" = "arm64" || test "$(arch)" = "riscv64" || test "$(arch)" = "loongarch64"
 	@test "$(bits)" = "32" || test "$(bits)" = "64"
 	@test "$(prefetch)" = "yes" || test "$(prefetch)" = "no"
 	@test "$(popcnt)" = "yes" || test "$(popcnt)" = "no"
@@ -942,12 +1044,16 @@ config-sanity: net
 	@test "$(vnni256)" = "yes" || test "$(vnni256)" = "no"
 	@test "$(vnni512)" = "yes" || test "$(vnni512)" = "no"
 	@test "$(neon)" = "yes" || test "$(neon)" = "no"
-	@test "$(comp)" = "gcc" || test "$(comp)" = "icc" || test "$(comp)" = "mingw" || test "$(comp)" = "clang" \
+	@test "$(comp)" = "gcc" || test "$(comp)" = "icx" || test "$(comp)" = "mingw" || test "$(comp)" = "clang" \
 	|| test "$(comp)" = "armv7a-linux-androideabi16-clang"  || test "$(comp)" = "aarch64-linux-android21-clang"
 
 $(EXE): $(OBJS)
 	+$(CXX) -o $@ $(OBJS) $(LDFLAGS)
 
+# Force recompilation to ensure version info is up-to-date
+misc.o: FORCE
+FORCE:
+
 clang-profile-make:
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) \
 	EXTRACXXFLAGS='-fprofile-instr-generate ' \
@@ -976,18 +1082,22 @@ gcc-profile-use:
 	EXTRALDFLAGS='-lgcov' \
 	all
 
-icc-profile-make:
+icx-profile-make:
-	@mkdir -p profdir
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) \
-	EXTRACXXFLAGS='-prof-gen=srcpos -prof_dir ./profdir' \
+	EXTRACXXFLAGS='-fprofile-instr-generate ' \
+	EXTRALDFLAGS=' -fprofile-instr-generate' \
 	all
 
-icc-profile-use:
+icx-profile-use:
+	$(XCRUN) llvm-profdata merge -output=stockfish.profdata *.profraw
 	$(MAKE) ARCH=$(ARCH) COMP=$(COMP) \
-	EXTRACXXFLAGS='-prof_use -prof_dir ./profdir' \
+	EXTRACXXFLAGS='-fprofile-instr-use=stockfish.profdata' \
+	EXTRALDFLAGS='-fprofile-use ' \
 	all
 
 .depend: $(SRCS)
 	-@$(CXX) $(DEPENDFLAGS) -MM $(SRCS) > $@ 2> /dev/null
 
+ifeq (, $(filter $(MAKECMDGOALS), help strip install clean net objclean profileclean config-sanity))
 -include .depend
+endif

src/benchmark.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,18 +16,19 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "benchmark.h"
+
+#include <cstdlib>
 #include <fstream>
 #include <iostream>
-#include <istream>
 #include <vector>
 
 #include "position.h"
 
-using namespace std;
-
 namespace {
 
-const vector<string> Defaults = {
+// clang-format off
+const std::vector<std::string> Defaults = {
   "setoption name UCI_Chess960 value false",
   "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1",
   "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 10",
@@ -90,86 +91,75 @@ const vector<string> Defaults = {
   "nqbnrkrb/pppppppp/8/8/8/8/PPPPPPPP/NQBNRKRB w KQkq - 0 1",
   "setoption name UCI_Chess960 value false"
 };
+// clang-format on
 
-} // namespace
+}  // namespace
 
 namespace Stockfish {
 
-/// setup_bench() builds a list of UCI commands to be run by bench. There
+// Builds a list of UCI commands to be run by bench. There
-/// are five parameters: TT size in MB, number of search threads that
+// are five parameters: TT size in MB, number of search threads that
-/// should be used, the limit value spent for each position, a file name
+// should be used, the limit value spent for each position, a file name
-/// where to look for positions in FEN format, the type of the limit:
+// where to look for positions in FEN format, and the type of the limit:
-/// depth, perft, nodes and movetime (in millisecs), and evaluation type
+// depth, perft, nodes and movetime (in milliseconds). Examples:
-/// mixed (default), classical, NNUE.
+//
-///
+// bench                            : search default positions up to depth 13
-/// bench -> search default positions up to depth 13
+// bench 64 1 15                    : search default positions up to depth 15 (TT = 64MB)
-/// bench 64 1 15 -> search default positions up to depth 15 (TT = 64MB)
+// bench 64 1 100000 default nodes  : search default positions for 100K nodes each
-/// bench 64 4 5000 current movetime -> search current position with 4 threads for 5 sec
+// bench 64 4 5000 current movetime : search current position with 4 threads for 5 sec
-/// bench 64 1 100000 default nodes -> search default positions for 100K nodes each
+// bench 16 1 5 blah perft          : run a perft 5 on positions in file "blah"
-/// bench 16 1 5 default perft -> run a perft 5 on default positions
+std::vector<std::string> setup_bench(const Position& current, std::istream& is) {
 
-vector<string> setup_bench(const Position& current, istream& is) {
+    std::vector<std::string> fens, list;
+    std::string              go, token;
 
-  vector<string> fens, list;
+    // Assign default values to missing arguments
-  string go, token;
+    std::string ttSize    = (is >> token) ? token : "16";
+    std::string threads   = (is >> token) ? token : "1";
+    std::string limit     = (is >> token) ? token : "13";
+    std::string fenFile   = (is >> token) ? token : "default";
+    std::string limitType = (is >> token) ? token : "depth";
 
-  // Assign default values to missing arguments
+    go = limitType == "eval" ? "eval" : "go " + limitType + " " + limit;
-  string ttSize    = (is >> token) ? token : "16";
-  string threads   = (is >> token) ? token : "1";
-  string limit     = (is >> token) ? token : "13";
-  string fenFile   = (is >> token) ? token : "default";
-  string limitType = (is >> token) ? token : "depth";
-  string evalType  = (is >> token) ? token : "mixed";
 
-  go = limitType == "eval" ? "eval" : "go " + limitType + " " + limit;
+    if (fenFile == "default")
+        fens = Defaults;
 
-  if (fenFile == "default")
+    else if (fenFile == "current")
-      fens = Defaults;
+        fens.push_back(current.fen());
 
-  else if (fenFile == "current")
+    else
-      fens.push_back(current.fen());
+    {
+        std::string   fen;
+        std::ifstream file(fenFile);
 
-  else
+        if (!file.is_open())
-  {
+        {
-      string fen;
+            std::cerr << "Unable to open file " << fenFile << std::endl;
-      ifstream file(fenFile);
+            exit(EXIT_FAILURE);
+        }
 
-      if (!file.is_open())
+        while (getline(file, fen))
-      {
+            if (!fen.empty())
-          cerr << "Unable to open file " << fenFile << endl;
+                fens.push_back(fen);
-          exit(EXIT_FAILURE);
-      }
 
-      while (getline(file, fen))
+        file.close();
-          if (!fen.empty())
+    }
-              fens.push_back(fen);
 
-      file.close();
+    list.emplace_back("setoption name Threads value " + threads);
-  }
+    list.emplace_back("setoption name Hash value " + ttSize);
+    list.emplace_back("ucinewgame");
 
-  list.emplace_back("setoption name Threads value " + threads);
+    for (const std::string& fen : fens)
-  list.emplace_back("setoption name Hash value " + ttSize);
+        if (fen.find("setoption") != std::string::npos)
-  list.emplace_back("ucinewgame");
+            list.emplace_back(fen);
+        else
+        {
+            list.emplace_back("position fen " + fen);
+            list.emplace_back(go);
+        }
 
-  size_t posCounter = 0;
+    return list;
-
-  for (const string& fen : fens)
-      if (fen.find("setoption") != string::npos)
-          list.emplace_back(fen);
-      else
-      {
-          if (evalType == "classical" || (evalType == "mixed" && posCounter % 2 == 0))
-              list.emplace_back("setoption name Use NNUE value false");
-          else if (evalType == "NNUE" || (evalType == "mixed" && posCounter % 2 != 0))
-              list.emplace_back("setoption name Use NNUE value true");
-          list.emplace_back("position fen " + fen);
-          list.emplace_back(go);
-          ++posCounter;
-      }
-
-  list.emplace_back("setoption name Use NNUE value true");
-
-  return list;
 }
 
-} // namespace Stockfish
+}  // namespace Stockfish

src/benchmark.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,23 +16,19 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#ifndef BENCHMARK_H_INCLUDED
+#define BENCHMARK_H_INCLUDED
 
-#ifndef PSQT_H_INCLUDED
+#include <iosfwd>
-#define PSQT_H_INCLUDED
+#include <string>
+#include <vector>
 
+namespace Stockfish {
 
-#include "types.h"
+class Position;
 
+std::vector<std::string> setup_bench(const Position&, std::istream&);
 
-namespace Stockfish::PSQT
+}  // namespace Stockfish
-{
 
-extern Score psq[PIECE_NB][SQUARE_NB];
+#endif  // #ifndef BENCHMARK_H_INCLUDED
-
-// Fill psqt array from a set of internally linked parameters
-extern void init();
-
-} // namespace Stockfish::PSQT
-
-
-#endif // PSQT_H_INCLUDED

src/bitbase.cpp

@@ -1,172 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <cassert>
-#include <vector>
-#include <bitset>
-
-#include "bitboard.h"
-#include "types.h"
-
-namespace Stockfish {
-
-namespace {
-
-  // There are 24 possible pawn squares: files A to D and ranks from 2 to 7.
-  // Positions with the pawn on files E to H will be mirrored before probing.
-  constexpr unsigned MAX_INDEX = 2*24*64*64; // stm * psq * wksq * bksq = 196608
-
-  std::bitset<MAX_INDEX> KPKBitbase;
-
-  // A KPK bitbase index is an integer in [0, IndexMax] range
-  //
-  // Information is mapped in a way that minimizes the number of iterations:
-  //
-  // bit  0- 5: white king square (from SQ_A1 to SQ_H8)
-  // bit  6-11: black king square (from SQ_A1 to SQ_H8)
-  // bit    12: side to move (WHITE or BLACK)
-  // bit 13-14: white pawn file (from FILE_A to FILE_D)
-  // bit 15-17: white pawn RANK_7 - rank (from RANK_7 - RANK_7 to RANK_7 - RANK_2)
-  unsigned index(Color stm, Square bksq, Square wksq, Square psq) {
-    return int(wksq) | (bksq << 6) | (stm << 12) | (file_of(psq) << 13) | ((RANK_7 - rank_of(psq)) << 15);
-  }
-
-  enum Result {
-    INVALID = 0,
-    UNKNOWN = 1,
-    DRAW    = 2,
-    WIN     = 4
-  };
-
-  Result& operator|=(Result& r, Result v) { return r = Result(r | v); }
-
-  struct KPKPosition {
-    KPKPosition() = default;
-    explicit KPKPosition(unsigned idx);
-    operator Result() const { return result; }
-    Result classify(const std::vector<KPKPosition>& db);
-
-    Color stm;
-    Square ksq[COLOR_NB], psq;
-    Result result;
-  };
-
-} // namespace
-
-bool Bitbases::probe(Square wksq, Square wpsq, Square bksq, Color stm) {
-
-  assert(file_of(wpsq) <= FILE_D);
-
-  return KPKBitbase[index(stm, bksq, wksq, wpsq)];
-}
-
-
-void Bitbases::init() {
-
-  std::vector<KPKPosition> db(MAX_INDEX);
-  unsigned idx, repeat = 1;
-
-  // Initialize db with known win / draw positions
-  for (idx = 0; idx < MAX_INDEX; ++idx)
-      db[idx] = KPKPosition(idx);
-
-  // Iterate through the positions until none of the unknown positions can be
-  // changed to either wins or draws (15 cycles needed).
-  while (repeat)
-      for (repeat = idx = 0; idx < MAX_INDEX; ++idx)
-          repeat |= (db[idx] == UNKNOWN && db[idx].classify(db) != UNKNOWN);
-
-  // Fill the bitbase with the decisive results
-  for (idx = 0; idx < MAX_INDEX; ++idx)
-      if (db[idx] == WIN)
-          KPKBitbase.set(idx);
-}
-
-namespace {
-
-  KPKPosition::KPKPosition(unsigned idx) {
-
-    ksq[WHITE] = Square((idx >>  0) & 0x3F);
-    ksq[BLACK] = Square((idx >>  6) & 0x3F);
-    stm        = Color ((idx >> 12) & 0x01);
-    psq        = make_square(File((idx >> 13) & 0x3), Rank(RANK_7 - ((idx >> 15) & 0x7)));
-
-    // Invalid if two pieces are on the same square or if a king can be captured
-    if (   distance(ksq[WHITE], ksq[BLACK]) <= 1
-        || ksq[WHITE] == psq
-        || ksq[BLACK] == psq
-        || (stm == WHITE && (pawn_attacks_bb(WHITE, psq) & ksq[BLACK])))
-        result = INVALID;
-
-    // Win if the pawn can be promoted without getting captured
-    else if (   stm == WHITE
-             && rank_of(psq) == RANK_7
-             && ksq[WHITE] != psq + NORTH
-             && (    distance(ksq[BLACK], psq + NORTH) > 1
-                 || (distance(ksq[WHITE], psq + NORTH) == 1)))
-        result = WIN;
-
-    // Draw if it is stalemate or the black king can capture the pawn
-    else if (   stm == BLACK
-             && (  !(attacks_bb<KING>(ksq[BLACK]) & ~(attacks_bb<KING>(ksq[WHITE]) | pawn_attacks_bb(WHITE, psq)))
-                 || (attacks_bb<KING>(ksq[BLACK]) & ~attacks_bb<KING>(ksq[WHITE]) & psq)))
-        result = DRAW;
-
-    // Position will be classified later
-    else
-        result = UNKNOWN;
-  }
-
-  Result KPKPosition::classify(const std::vector<KPKPosition>& db) {
-
-    // White to move: If one move leads to a position classified as WIN, the result
-    // of the current position is WIN. If all moves lead to positions classified
-    // as DRAW, the current position is classified as DRAW, otherwise the current
-    // position is classified as UNKNOWN.
-    //
-    // Black to move: If one move leads to a position classified as DRAW, the result
-    // of the current position is DRAW. If all moves lead to positions classified
-    // as WIN, the position is classified as WIN, otherwise the current position is
-    // classified as UNKNOWN.
-    const Result Good = (stm == WHITE ? WIN   : DRAW);
-    const Result Bad  = (stm == WHITE ? DRAW  : WIN);
-
-    Result r = INVALID;
-    Bitboard b = attacks_bb<KING>(ksq[stm]);
-
-    while (b)
-        r |= stm == WHITE ? db[index(BLACK, ksq[BLACK], pop_lsb(b), psq)]
-                          : db[index(WHITE, pop_lsb(b), ksq[WHITE], psq)];
-
-    if (stm == WHITE)
-    {
-        if (rank_of(psq) < RANK_7)      // Single push
-            r |= db[index(BLACK, ksq[BLACK], ksq[WHITE], psq + NORTH)];
-
-        if (   rank_of(psq) == RANK_2   // Double push
-            && psq + NORTH != ksq[WHITE]
-            && psq + NORTH != ksq[BLACK])
-            r |= db[index(BLACK, ksq[BLACK], ksq[WHITE], psq + NORTH + NORTH)];
-    }
-
-    return result = r & Good  ? Good  : r & UNKNOWN ? UNKNOWN : Bad;
-  }
-
-} // namespace
-
-} // namespace Stockfish

src/bitboard.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,10 +16,12 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "bitboard.h"
+
 #include <algorithm>
 #include <bitset>
+#include <initializer_list>
 
-#include "bitboard.h"
 #include "misc.h"
 
 namespace Stockfish {
@@ -27,7 +29,6 @@ namespace Stockfish {
 uint8_t PopCnt16[1 << 16];
 uint8_t SquareDistance[SQUARE_NB][SQUARE_NB];
 
-Bitboard SquareBB[SQUARE_NB];
 Bitboard LineBB[SQUARE_NB][SQUARE_NB];
 Bitboard BetweenBB[SQUARE_NB][SQUARE_NB];
 Bitboard PseudoAttacks[PIECE_TYPE_NB][SQUARE_NB];
@@ -38,93 +39,86 @@ Magic BishopMagics[SQUARE_NB];
 
 namespace {
 
-  Bitboard RookTable[0x19000];  // To store rook attacks
+Bitboard RookTable[0x19000];   // To store rook attacks
-  Bitboard BishopTable[0x1480]; // To store bishop attacks
+Bitboard BishopTable[0x1480];  // To store bishop attacks
 
-  void init_magics(PieceType pt, Bitboard table[], Magic magics[]);
+void init_magics(PieceType pt, Bitboard table[], Magic magics[]);
 
-}
+// Returns the bitboard of target square for the given step
-
+// from the given square. If the step is off the board, returns empty bitboard.
-/// safe_destination() returns the bitboard of target square for the given step
+Bitboard safe_destination(Square s, int step) {
-/// from the given square. If the step is off the board, returns empty bitboard.
-
-inline Bitboard safe_destination(Square s, int step) {
     Square to = Square(s + step);
     return is_ok(to) && distance(s, to) <= 2 ? square_bb(to) : Bitboard(0);
 }
+}
 
-
+// Returns an ASCII representation of a bitboard suitable
-/// Bitboards::pretty() returns an ASCII representation of a bitboard suitable
+// to be printed to standard output. Useful for debugging.
-/// to be printed to standard output. Useful for debugging.
-
 std::string Bitboards::pretty(Bitboard b) {
 
-  std::string s = "+---+---+---+---+---+---+---+---+\n";
+    std::string s = "+---+---+---+---+---+---+---+---+\n";
 
-  for (Rank r = RANK_8; r >= RANK_1; --r)
+    for (Rank r = RANK_8; r >= RANK_1; --r)
-  {
+    {
-      for (File f = FILE_A; f <= FILE_H; ++f)
+        for (File f = FILE_A; f <= FILE_H; ++f)
-          s += b & make_square(f, r) ? "| X " : "|   ";
+            s += b & make_square(f, r) ? "| X " : "|   ";
 
-      s += "| " + std::to_string(1 + r) + "\n+---+---+---+---+---+---+---+---+\n";
+        s += "| " + std::to_string(1 + r) + "\n+---+---+---+---+---+---+---+---+\n";
-  }
+    }
-  s += "  a   b   c   d   e   f   g   h\n";
+    s += "  a   b   c   d   e   f   g   h\n";
 
-  return s;
+    return s;
 }
 
 
-/// Bitboards::init() initializes various bitboard tables. It is called at
+// Initializes various bitboard tables. It is called at
-/// startup and relies on global objects to be already zero-initialized.
+// startup and relies on global objects to be already zero-initialized.
-
 void Bitboards::init() {
 
-  for (unsigned i = 0; i < (1 << 16); ++i)
+    for (unsigned i = 0; i < (1 << 16); ++i)
-      PopCnt16[i] = uint8_t(std::bitset<16>(i).count());
+        PopCnt16[i] = uint8_t(std::bitset<16>(i).count());
 
-  for (Square s = SQ_A1; s <= SQ_H8; ++s)
+    for (Square s1 = SQ_A1; s1 <= SQ_H8; ++s1)
-      SquareBB[s] = (1ULL << s);
+        for (Square s2 = SQ_A1; s2 <= SQ_H8; ++s2)
+            SquareDistance[s1][s2] = std::max(distance<File>(s1, s2), distance<Rank>(s1, s2));
 
-  for (Square s1 = SQ_A1; s1 <= SQ_H8; ++s1)
+    init_magics(ROOK, RookTable, RookMagics);
-      for (Square s2 = SQ_A1; s2 <= SQ_H8; ++s2)
+    init_magics(BISHOP, BishopTable, BishopMagics);
-          SquareDistance[s1][s2] = std::max(distance<File>(s1, s2), distance<Rank>(s1, s2));
 
-  init_magics(ROOK, RookTable, RookMagics);
+    for (Square s1 = SQ_A1; s1 <= SQ_H8; ++s1)
-  init_magics(BISHOP, BishopTable, BishopMagics);
+    {
+        PawnAttacks[WHITE][s1] = pawn_attacks_bb<WHITE>(square_bb(s1));
+        PawnAttacks[BLACK][s1] = pawn_attacks_bb<BLACK>(square_bb(s1));
 
-  for (Square s1 = SQ_A1; s1 <= SQ_H8; ++s1)
+        for (int step : {-9, -8, -7, -1, 1, 7, 8, 9})
-  {
+            PseudoAttacks[KING][s1] |= safe_destination(s1, step);
-      PawnAttacks[WHITE][s1] = pawn_attacks_bb<WHITE>(square_bb(s1));
-      PawnAttacks[BLACK][s1] = pawn_attacks_bb<BLACK>(square_bb(s1));
 
-      for (int step : {-9, -8, -7, -1, 1, 7, 8, 9} )
+        for (int step : {-17, -15, -10, -6, 6, 10, 15, 17})
-         PseudoAttacks[KING][s1] |= safe_destination(s1, step);
+            PseudoAttacks[KNIGHT][s1] |= safe_destination(s1, step);
 
-      for (int step : {-17, -15, -10, -6, 6, 10, 15, 17} )
+        PseudoAttacks[QUEEN][s1] = PseudoAttacks[BISHOP][s1] = attacks_bb<BISHOP>(s1, 0);
-         PseudoAttacks[KNIGHT][s1] |= safe_destination(s1, step);
+        PseudoAttacks[QUEEN][s1] |= PseudoAttacks[ROOK][s1]  = attacks_bb<ROOK>(s1, 0);
 
-      PseudoAttacks[QUEEN][s1]  = PseudoAttacks[BISHOP][s1] = attacks_bb<BISHOP>(s1, 0);
+        for (PieceType pt : {BISHOP, ROOK})
-      PseudoAttacks[QUEEN][s1] |= PseudoAttacks[  ROOK][s1] = attacks_bb<  ROOK>(s1, 0);
+            for (Square s2 = SQ_A1; s2 <= SQ_H8; ++s2)
-
+            {
-      for (PieceType pt : { BISHOP, ROOK })
+                if (PseudoAttacks[pt][s1] & s2)
-          for (Square s2 = SQ_A1; s2 <= SQ_H8; ++s2)
+                {
-          {
+                    LineBB[s1][s2] = (attacks_bb(pt, s1, 0) & attacks_bb(pt, s2, 0)) | s1 | s2;
-              if (PseudoAttacks[pt][s1] & s2)
+                    BetweenBB[s1][s2] =
-              {
+                      (attacks_bb(pt, s1, square_bb(s2)) & attacks_bb(pt, s2, square_bb(s1)));
-                  LineBB[s1][s2]    = (attacks_bb(pt, s1, 0) & attacks_bb(pt, s2, 0)) | s1 | s2;
+                }
-                  BetweenBB[s1][s2] = (attacks_bb(pt, s1, square_bb(s2)) & attacks_bb(pt, s2, square_bb(s1)));
+                BetweenBB[s1][s2] |= s2;
-              }
+            }
-              BetweenBB[s1][s2] |= s2;
+    }
-          }
-  }
 }
 
 namespace {
 
-  Bitboard sliding_attack(PieceType pt, Square sq, Bitboard occupied) {
+Bitboard sliding_attack(PieceType pt, Square sq, Bitboard occupied) {
 
-    Bitboard attacks = 0;
+    Bitboard  attacks             = 0;
-    Direction   RookDirections[4] = {NORTH, SOUTH, EAST, WEST};
+    Direction RookDirections[4]   = {NORTH, SOUTH, EAST, WEST};
     Direction BishopDirections[4] = {NORTH_EAST, SOUTH_EAST, SOUTH_WEST, NORTH_WEST};
 
     for (Direction d : (pt == ROOK ? RookDirections : BishopDirections))
@@ -135,22 +129,21 @@ namespace {
     }
 
     return attacks;
-  }
+}
 
 
-  // init_magics() computes all rook and bishop attacks at startup. Magic
+// Computes all rook and bishop attacks at startup. Magic
-  // bitboards are used to look up attacks of sliding pieces. As a reference see
+// bitboards are used to look up attacks of sliding pieces. As a reference see
-  // www.chessprogramming.org/Magic_Bitboards. In particular, here we use the so
+// www.chessprogramming.org/Magic_Bitboards. In particular, here we use the so
-  // called "fancy" approach.
+// called "fancy" approach.
-
+void init_magics(PieceType pt, Bitboard table[], Magic magics[]) {
-  void init_magics(PieceType pt, Bitboard table[], Magic magics[]) {
 
     // Optimal PRNG seeds to pick the correct magics in the shortest time
-    int seeds[][RANK_NB] = { { 8977, 44560, 54343, 38998,  5731, 95205, 104912, 17020 },
+    int seeds[][RANK_NB] = {{8977, 44560, 54343, 38998, 5731, 95205, 104912, 17020},
-                             {  728, 10316, 55013, 32803, 12281, 15100,  16645,   255 } };
+                            {728, 10316, 55013, 32803, 12281, 15100, 16645, 255}};
 
     Bitboard occupancy[4096], reference[4096], edges, b;
-    int epoch[4096] = {}, cnt = 0, size = 0;
+    int      epoch[4096] = {}, cnt = 0, size = 0;
 
     for (Square s = SQ_A1; s <= SQ_H8; ++s)
     {
@@ -163,8 +156,8 @@ namespace {
         // the number of 1s of the mask. Hence we deduce the size of the shift to
         // apply to the 64 or 32 bits word to get the index.
         Magic& m = magics[s];
-        m.mask  = sliding_attack(pt, s, 0) & ~edges;
+        m.mask   = sliding_attack(pt, s, 0) & ~edges;
-        m.shift = (Is64Bit ? 64 : 32) - popcount(m.mask);
+        m.shift  = (Is64Bit ? 64 : 32) - popcount(m.mask);
 
         // Set the offset for the attacks table of the square. We have individual
         // table sizes for each square with "Fancy Magic Bitboards".
@@ -173,7 +166,8 @@ namespace {
         // Use Carry-Rippler trick to enumerate all subsets of masks[s] and
         // store the corresponding sliding attack bitboard in reference[].
         b = size = 0;
-        do {
+        do
+        {
             occupancy[size] = b;
             reference[size] = sliding_attack(pt, s, b);
 
@@ -191,9 +185,9 @@ namespace {
 
         // Find a magic for square 's' picking up an (almost) random number
         // until we find the one that passes the verification test.
-        for (int i = 0; i < size; )
+        for (int i = 0; i < size;)
         {
-            for (m.magic = 0; popcount((m.magic * m.mask) >> 56) < 6; )
+            for (m.magic = 0; popcount((m.magic * m.mask) >> 56) < 6;)
                 m.magic = rng.sparse_rand<Bitboard>();
 
             // A good magic must map every possible occupancy to an index that
@@ -208,7 +202,7 @@ namespace {
 
                 if (epoch[idx] < cnt)
                 {
-                    epoch[idx] = cnt;
+                    epoch[idx]     = cnt;
                     m.attacks[idx] = reference[i];
                 }
                 else if (m.attacks[idx] != reference[i])
@@ -216,7 +210,7 @@ namespace {
             }
         }
     }
-  }
+}
 }
 
-} // namespace Stockfish
+}  // namespace Stockfish

src/bitboard.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -19,28 +19,23 @@
 #ifndef BITBOARD_H_INCLUDED
 #define BITBOARD_H_INCLUDED
 
+#include <algorithm>
+#include <cassert>
+#include <cmath>
+#include <cstdint>
+#include <cstdlib>
 #include <string>
 
 #include "types.h"
 
 namespace Stockfish {
 
-namespace Bitbases {
-
-void init();
-bool probe(Square wksq, Square wpsq, Square bksq, Color us);
-
-} // namespace Stockfish::Bitbases
-
 namespace Bitboards {
 
-void init();
+void        init();
 std::string pretty(Bitboard b);
 
-} // namespace Stockfish::Bitboards
+}  // namespace Stockfish::Bitboards
-
-constexpr Bitboard AllSquares = ~Bitboard(0);
-constexpr Bitboard DarkSquares = 0xAA55AA55AA55AA55ULL;
 
 constexpr Bitboard FileABB = 0x0101010101010101ULL;
 constexpr Bitboard FileBBB = FileABB << 1;
@@ -60,392 +55,317 @@ constexpr Bitboard Rank6BB = Rank1BB << (8 * 5);
 constexpr Bitboard Rank7BB = Rank1BB << (8 * 6);
 constexpr Bitboard Rank8BB = Rank1BB << (8 * 7);
 
-constexpr Bitboard QueenSide   = FileABB | FileBBB | FileCBB | FileDBB;
-constexpr Bitboard CenterFiles = FileCBB | FileDBB | FileEBB | FileFBB;
-constexpr Bitboard KingSide    = FileEBB | FileFBB | FileGBB | FileHBB;
-constexpr Bitboard Center      = (FileDBB | FileEBB) & (Rank4BB | Rank5BB);
-
-constexpr Bitboard KingFlank[FILE_NB] = {
-  QueenSide ^ FileDBB, QueenSide, QueenSide,
-  CenterFiles, CenterFiles,
-  KingSide, KingSide, KingSide ^ FileEBB
-};
-
 extern uint8_t PopCnt16[1 << 16];
 extern uint8_t SquareDistance[SQUARE_NB][SQUARE_NB];
 
-extern Bitboard SquareBB[SQUARE_NB];
 extern Bitboard BetweenBB[SQUARE_NB][SQUARE_NB];
 extern Bitboard LineBB[SQUARE_NB][SQUARE_NB];
 extern Bitboard PseudoAttacks[PIECE_TYPE_NB][SQUARE_NB];
 extern Bitboard PawnAttacks[COLOR_NB][SQUARE_NB];
 
 
-/// Magic holds all magic bitboards relevant data for a single square
+// Magic holds all magic bitboards relevant data for a single square
 struct Magic {
-  Bitboard  mask;
+    Bitboard  mask;
-  Bitboard  magic;
+    Bitboard  magic;
-  Bitboard* attacks;
+    Bitboard* attacks;
-  unsigned  shift;
+    unsigned  shift;
 
-  // Compute the attack's index using the 'magic bitboards' approach
+    // Compute the attack's index using the 'magic bitboards' approach
-  unsigned index(Bitboard occupied) const {
+    unsigned index(Bitboard occupied) const {
 
-    if (HasPext)
+        if (HasPext)
-        return unsigned(pext(occupied, mask));
+            return unsigned(pext(occupied, mask));
 
-    if (Is64Bit)
+        if (Is64Bit)
-        return unsigned(((occupied & mask) * magic) >> shift);
+            return unsigned(((occupied & mask) * magic) >> shift);
 
-    unsigned lo = unsigned(occupied) & unsigned(mask);
+        unsigned lo = unsigned(occupied) & unsigned(mask);
-    unsigned hi = unsigned(occupied >> 32) & unsigned(mask >> 32);
+        unsigned hi = unsigned(occupied >> 32) & unsigned(mask >> 32);
-    return (lo * unsigned(magic) ^ hi * unsigned(magic >> 32)) >> shift;
+        return (lo * unsigned(magic) ^ hi * unsigned(magic >> 32)) >> shift;
-  }
+    }
 };
 
 extern Magic RookMagics[SQUARE_NB];
 extern Magic BishopMagics[SQUARE_NB];
 
-inline Bitboard square_bb(Square s) {
+constexpr Bitboard square_bb(Square s) {
-  assert(is_ok(s));
+    assert(is_ok(s));
-  return SquareBB[s];
+    return (1ULL << s);
 }
 
 
-/// Overloads of bitwise operators between a Bitboard and a Square for testing
+// Overloads of bitwise operators between a Bitboard and a Square for testing
-/// whether a given bit is set in a bitboard, and for setting and clearing bits.
+// whether a given bit is set in a bitboard, and for setting and clearing bits.
 
-inline Bitboard  operator&( Bitboard  b, Square s) { return b &  square_bb(s); }
+inline Bitboard  operator&(Bitboard b, Square s) { return b & square_bb(s); }
-inline Bitboard  operator|( Bitboard  b, Square s) { return b |  square_bb(s); }
+inline Bitboard  operator|(Bitboard b, Square s) { return b | square_bb(s); }
-inline Bitboard  operator^( Bitboard  b, Square s) { return b ^  square_bb(s); }
+inline Bitboard  operator^(Bitboard b, Square s) { return b ^ square_bb(s); }
 inline Bitboard& operator|=(Bitboard& b, Square s) { return b |= square_bb(s); }
 inline Bitboard& operator^=(Bitboard& b, Square s) { return b ^= square_bb(s); }
 
-inline Bitboard  operator&(Square s, Bitboard b) { return b & s; }
+inline Bitboard operator&(Square s, Bitboard b) { return b & s; }
-inline Bitboard  operator|(Square s, Bitboard b) { return b | s; }
+inline Bitboard operator|(Square s, Bitboard b) { return b | s; }
-inline Bitboard  operator^(Square s, Bitboard b) { return b ^ s; }
+inline Bitboard operator^(Square s, Bitboard b) { return b ^ s; }
 
-inline Bitboard  operator|(Square s1, Square s2) { return square_bb(s1) | s2; }
+inline Bitboard operator|(Square s1, Square s2) { return square_bb(s1) | s2; }
 
-constexpr bool more_than_one(Bitboard b) {
+constexpr bool more_than_one(Bitboard b) { return b & (b - 1); }
-  return b & (b - 1);
-}
 
 
-constexpr bool opposite_colors(Square s1, Square s2) {
+// rank_bb() and file_bb() return a bitboard representing all the squares on
-  return (s1 + rank_of(s1) + s2 + rank_of(s2)) & 1;
+// the given file or rank.
-}
+
+constexpr Bitboard rank_bb(Rank r) { return Rank1BB << (8 * r); }
+
+constexpr Bitboard rank_bb(Square s) { return rank_bb(rank_of(s)); }
+
+constexpr Bitboard file_bb(File f) { return FileABB << f; }
+
+constexpr Bitboard file_bb(Square s) { return file_bb(file_of(s)); }
 
 
-/// rank_bb() and file_bb() return a bitboard representing all the squares on
+// Moves a bitboard one or two steps as specified by the direction D
-/// the given file or rank.
-
-constexpr Bitboard rank_bb(Rank r) {
-  return Rank1BB << (8 * r);
-}
-
-constexpr Bitboard rank_bb(Square s) {
-  return rank_bb(rank_of(s));
-}
-
-constexpr Bitboard file_bb(File f) {
-  return FileABB << f;
-}
-
-constexpr Bitboard file_bb(Square s) {
-  return file_bb(file_of(s));
-}
-
-
-/// shift() moves a bitboard one or two steps as specified by the direction D
-
 template<Direction D>
 constexpr Bitboard shift(Bitboard b) {
-  return  D == NORTH      ?  b             << 8 : D == SOUTH      ?  b             >> 8
+    return D == NORTH         ? b << 8
-        : D == NORTH+NORTH?  b             <<16 : D == SOUTH+SOUTH?  b             >>16
+         : D == SOUTH         ? b >> 8
-        : D == EAST       ? (b & ~FileHBB) << 1 : D == WEST       ? (b & ~FileABB) >> 1
+         : D == NORTH + NORTH ? b << 16
-        : D == NORTH_EAST ? (b & ~FileHBB) << 9 : D == NORTH_WEST ? (b & ~FileABB) << 7
+         : D == SOUTH + SOUTH ? b >> 16
-        : D == SOUTH_EAST ? (b & ~FileHBB) >> 7 : D == SOUTH_WEST ? (b & ~FileABB) >> 9
+         : D == EAST          ? (b & ~FileHBB) << 1
-        : 0;
+         : D == WEST          ? (b & ~FileABB) >> 1
+         : D == NORTH_EAST    ? (b & ~FileHBB) << 9
+         : D == NORTH_WEST    ? (b & ~FileABB) << 7
+         : D == SOUTH_EAST    ? (b & ~FileHBB) >> 7
+         : D == SOUTH_WEST    ? (b & ~FileABB) >> 9
+                              : 0;
 }
 
 
-/// pawn_attacks_bb() returns the squares attacked by pawns of the given color
+// Returns the squares attacked by pawns of the given color
-/// from the squares in the given bitboard.
+// from the squares in the given bitboard.
-
 template<Color C>
 constexpr Bitboard pawn_attacks_bb(Bitboard b) {
-  return C == WHITE ? shift<NORTH_WEST>(b) | shift<NORTH_EAST>(b)
+    return C == WHITE ? shift<NORTH_WEST>(b) | shift<NORTH_EAST>(b)
-                    : shift<SOUTH_WEST>(b) | shift<SOUTH_EAST>(b);
+                      : shift<SOUTH_WEST>(b) | shift<SOUTH_EAST>(b);
 }
 
 inline Bitboard pawn_attacks_bb(Color c, Square s) {
 
-  assert(is_ok(s));
+    assert(is_ok(s));
-  return PawnAttacks[c][s];
+    return PawnAttacks[c][s];
 }
 
-
+// Returns a bitboard representing an entire line (from board edge
-/// pawn_double_attacks_bb() returns the squares doubly attacked by pawns of the
+// to board edge) that intersects the two given squares. If the given squares
-/// given color from the squares in the given bitboard.
+// are not on a same file/rank/diagonal, the function returns 0. For instance,
-
+// line_bb(SQ_C4, SQ_F7) will return a bitboard with the A2-G8 diagonal.
-template<Color C>
-constexpr Bitboard pawn_double_attacks_bb(Bitboard b) {
-  return C == WHITE ? shift<NORTH_WEST>(b) & shift<NORTH_EAST>(b)
-                    : shift<SOUTH_WEST>(b) & shift<SOUTH_EAST>(b);
-}
-
-
-/// adjacent_files_bb() returns a bitboard representing all the squares on the
-/// adjacent files of a given square.
-
-constexpr Bitboard adjacent_files_bb(Square s) {
-  return shift<EAST>(file_bb(s)) | shift<WEST>(file_bb(s));
-}
-
-
-/// line_bb() returns a bitboard representing an entire line (from board edge
-/// to board edge) that intersects the two given squares. If the given squares
-/// are not on a same file/rank/diagonal, the function returns 0. For instance,
-/// line_bb(SQ_C4, SQ_F7) will return a bitboard with the A2-G8 diagonal.
-
 inline Bitboard line_bb(Square s1, Square s2) {
 
-  assert(is_ok(s1) && is_ok(s2));
+    assert(is_ok(s1) && is_ok(s2));
-
+    return LineBB[s1][s2];
-  return LineBB[s1][s2];
 }
 
 
-/// between_bb(s1, s2) returns a bitboard representing the squares in the semi-open
+// Returns a bitboard representing the squares in the semi-open
-/// segment between the squares s1 and s2 (excluding s1 but including s2). If the
+// segment between the squares s1 and s2 (excluding s1 but including s2). If the
-/// given squares are not on a same file/rank/diagonal, it returns s2. For instance,
+// given squares are not on a same file/rank/diagonal, it returns s2. For instance,
-/// between_bb(SQ_C4, SQ_F7) will return a bitboard with squares D5, E6 and F7, but
+// between_bb(SQ_C4, SQ_F7) will return a bitboard with squares D5, E6 and F7, but
-/// between_bb(SQ_E6, SQ_F8) will return a bitboard with the square F8. This trick
+// between_bb(SQ_E6, SQ_F8) will return a bitboard with the square F8. This trick
-/// allows to generate non-king evasion moves faster: the defending piece must either
+// allows to generate non-king evasion moves faster: the defending piece must either
-/// interpose itself to cover the check or capture the checking piece.
+// interpose itself to cover the check or capture the checking piece.
-
 inline Bitboard between_bb(Square s1, Square s2) {
 
-  assert(is_ok(s1) && is_ok(s2));
+    assert(is_ok(s1) && is_ok(s2));
-
+    return BetweenBB[s1][s2];
-  return BetweenBB[s1][s2];
 }
 
+// Returns true if the squares s1, s2 and s3 are aligned either on a
+// straight or on a diagonal line.
+inline bool aligned(Square s1, Square s2, Square s3) { return line_bb(s1, s2) & s3; }
 
-/// forward_ranks_bb() returns a bitboard representing the squares on the ranks in
-/// front of the given one, from the point of view of the given color. For instance,
-/// forward_ranks_bb(BLACK, SQ_D3) will return the 16 squares on ranks 1 and 2.
 
-constexpr Bitboard forward_ranks_bb(Color c, Square s) {
+// distance() functions return the distance between x and y, defined as the
-  return c == WHITE ? ~Rank1BB << 8 * relative_rank(WHITE, s)
+// number of steps for a king in x to reach y.
-                    : ~Rank8BB >> 8 * relative_rank(BLACK, s);
+
+template<typename T1 = Square>
+inline int distance(Square x, Square y);
+
+template<>
+inline int distance<File>(Square x, Square y) {
+    return std::abs(file_of(x) - file_of(y));
 }
 
-
+template<>
-/// forward_file_bb() returns a bitboard representing all the squares along the
+inline int distance<Rank>(Square x, Square y) {
-/// line in front of the given one, from the point of view of the given color.
+    return std::abs(rank_of(x) - rank_of(y));
-
-constexpr Bitboard forward_file_bb(Color c, Square s) {
-  return forward_ranks_bb(c, s) & file_bb(s);
 }
 
-
+template<>
-/// pawn_attack_span() returns a bitboard representing all the squares that can
+inline int distance<Square>(Square x, Square y) {
-/// be attacked by a pawn of the given color when it moves along its file, starting
+    return SquareDistance[x][y];
-/// from the given square.
-
-constexpr Bitboard pawn_attack_span(Color c, Square s) {
-  return forward_ranks_bb(c, s) & adjacent_files_bb(s);
 }
 
-
-/// passed_pawn_span() returns a bitboard which can be used to test if a pawn of
-/// the given color and on the given square is a passed pawn.
-
-constexpr Bitboard passed_pawn_span(Color c, Square s) {
-  return pawn_attack_span(c, s) | forward_file_bb(c, s);
-}
-
-
-/// aligned() returns true if the squares s1, s2 and s3 are aligned either on a
-/// straight or on a diagonal line.
-
-inline bool aligned(Square s1, Square s2, Square s3) {
-  return line_bb(s1, s2) & s3;
-}
-
-
-/// distance() functions return the distance between x and y, defined as the
-/// number of steps for a king in x to reach y.
-
-template<typename T1 = Square> inline int distance(Square x, Square y);
-template<> inline int distance<File>(Square x, Square y) { return std::abs(file_of(x) - file_of(y)); }
-template<> inline int distance<Rank>(Square x, Square y) { return std::abs(rank_of(x) - rank_of(y)); }
-template<> inline int distance<Square>(Square x, Square y) { return SquareDistance[x][y]; }
-
 inline int edge_distance(File f) { return std::min(f, File(FILE_H - f)); }
-inline int edge_distance(Rank r) { return std::min(r, Rank(RANK_8 - r)); }
-
-
-/// attacks_bb(Square) returns the pseudo attacks of the give piece type
-/// assuming an empty board.
 
+// Returns the pseudo attacks of the given piece type
+// assuming an empty board.
 template<PieceType Pt>
 inline Bitboard attacks_bb(Square s) {
 
-  assert((Pt != PAWN) && (is_ok(s)));
+    assert((Pt != PAWN) && (is_ok(s)));
-
+    return PseudoAttacks[Pt][s];
-  return PseudoAttacks[Pt][s];
 }
 
 
-/// attacks_bb(Square, Bitboard) returns the attacks by the given piece
+// Returns the attacks by the given piece
-/// assuming the board is occupied according to the passed Bitboard.
+// assuming the board is occupied according to the passed Bitboard.
-/// Sliding piece attacks do not continue passed an occupied square.
+// Sliding piece attacks do not continue passed an occupied square.
-
 template<PieceType Pt>
 inline Bitboard attacks_bb(Square s, Bitboard occupied) {
 
-  assert((Pt != PAWN) && (is_ok(s)));
+    assert((Pt != PAWN) && (is_ok(s)));
 
-  switch (Pt)
+    switch (Pt)
-  {
+    {
-  case BISHOP: return BishopMagics[s].attacks[BishopMagics[s].index(occupied)];
+    case BISHOP :
-  case ROOK  : return   RookMagics[s].attacks[  RookMagics[s].index(occupied)];
+        return BishopMagics[s].attacks[BishopMagics[s].index(occupied)];
-  case QUEEN : return attacks_bb<BISHOP>(s, occupied) | attacks_bb<ROOK>(s, occupied);
+    case ROOK :
-  default    : return PseudoAttacks[Pt][s];
+        return RookMagics[s].attacks[RookMagics[s].index(occupied)];
-  }
+    case QUEEN :
+        return attacks_bb<BISHOP>(s, occupied) | attacks_bb<ROOK>(s, occupied);
+    default :
+        return PseudoAttacks[Pt][s];
+    }
 }
 
+// Returns the attacks by the given piece
+// assuming the board is occupied according to the passed Bitboard.
+// Sliding piece attacks do not continue passed an occupied square.
 inline Bitboard attacks_bb(PieceType pt, Square s, Bitboard occupied) {
 
-  assert((pt != PAWN) && (is_ok(s)));
+    assert((pt != PAWN) && (is_ok(s)));
 
-  switch (pt)
+    switch (pt)
-  {
+    {
-  case BISHOP: return attacks_bb<BISHOP>(s, occupied);
+    case BISHOP :
-  case ROOK  : return attacks_bb<  ROOK>(s, occupied);
+        return attacks_bb<BISHOP>(s, occupied);
-  case QUEEN : return attacks_bb<BISHOP>(s, occupied) | attacks_bb<ROOK>(s, occupied);
+    case ROOK :
-  default    : return PseudoAttacks[pt][s];
+        return attacks_bb<ROOK>(s, occupied);
-  }
+    case QUEEN :
+        return attacks_bb<BISHOP>(s, occupied) | attacks_bb<ROOK>(s, occupied);
+    default :
+        return PseudoAttacks[pt][s];
+    }
 }
 
 
-/// popcount() counts the number of non-zero bits in a bitboard
+// Counts the number of non-zero bits in a bitboard.
-
 inline int popcount(Bitboard b) {
 
 #ifndef USE_POPCNT
 
-  union { Bitboard bb; uint16_t u[4]; } v = { b };
+    union {
-  return PopCnt16[v.u[0]] + PopCnt16[v.u[1]] + PopCnt16[v.u[2]] + PopCnt16[v.u[3]];
+        Bitboard bb;
+        uint16_t u[4];
+    } v = {b};
+    return PopCnt16[v.u[0]] + PopCnt16[v.u[1]] + PopCnt16[v.u[2]] + PopCnt16[v.u[3]];
 
-#elif defined(_MSC_VER) || defined(__INTEL_COMPILER)
+#elif defined(_MSC_VER)
 
-  return (int)_mm_popcnt_u64(b);
+    return int(_mm_popcnt_u64(b));
 
-#else // Assumed gcc or compatible compiler
+#else  // Assumed gcc or compatible compiler
 
-  return __builtin_popcountll(b);
+    return __builtin_popcountll(b);
 
 #endif
 }
 
-
+// Returns the least significant bit in a non-zero bitboard.
-/// lsb() and msb() return the least/most significant bit in a non-zero bitboard
-
-#if defined(__GNUC__)  // GCC, Clang, ICC
-
 inline Square lsb(Bitboard b) {
-  assert(b);
+    assert(b);
-  return Square(__builtin_ctzll(b));
-}
 
-inline Square msb(Bitboard b) {
+#if defined(__GNUC__)  // GCC, Clang, ICX
-  assert(b);
-  return Square(63 ^ __builtin_clzll(b));
-}
 
-#elif defined(_MSC_VER)  // MSVC
+    return Square(__builtin_ctzll(b));
 
-#ifdef _WIN64  // MSVC, WIN64
+#elif defined(_MSC_VER)
+    #ifdef _WIN64  // MSVC, WIN64
 
-inline Square lsb(Bitboard b) {
+    unsigned long idx;
-  assert(b);
+    _BitScanForward64(&idx, b);
-  unsigned long idx;
+    return Square(idx);
-  _BitScanForward64(&idx, b);
-  return (Square) idx;
-}
 
-inline Square msb(Bitboard b) {
+    #else  // MSVC, WIN32
-  assert(b);
+    unsigned long idx;
-  unsigned long idx;
-  _BitScanReverse64(&idx, b);
-  return (Square) idx;
-}
-
-#else  // MSVC, WIN32
-
-inline Square lsb(Bitboard b) {
-  assert(b);
-  unsigned long idx;
-
-  if (b & 0xffffffff) {
-      _BitScanForward(&idx, int32_t(b));
-      return Square(idx);
-  } else {
-      _BitScanForward(&idx, int32_t(b >> 32));
-      return Square(idx + 32);
-  }
-}
-
-inline Square msb(Bitboard b) {
-  assert(b);
-  unsigned long idx;
-
-  if (b >> 32) {
-      _BitScanReverse(&idx, int32_t(b >> 32));
-      return Square(idx + 32);
-  } else {
-      _BitScanReverse(&idx, int32_t(b));
-      return Square(idx);
-  }
-}
-
-#endif
 
+    if (b & 0xffffffff)
+    {
+        _BitScanForward(&idx, int32_t(b));
+        return Square(idx);
+    }
+    else
+    {
+        _BitScanForward(&idx, int32_t(b >> 32));
+        return Square(idx + 32);
+    }
+    #endif
 #else  // Compiler is neither GCC nor MSVC compatible
-
+    #error "Compiler not supported."
-#error "Compiler not supported."
-
 #endif
+}
 
-/// least_significant_square_bb() returns the bitboard of the least significant
+// Returns the most significant bit in a non-zero bitboard.
-/// square of a non-zero bitboard. It is equivalent to square_bb(lsb(bb)).
+inline Square msb(Bitboard b) {
+    assert(b);
 
+#if defined(__GNUC__)  // GCC, Clang, ICX
+
+    return Square(63 ^ __builtin_clzll(b));
+
+#elif defined(_MSC_VER)
+    #ifdef _WIN64  // MSVC, WIN64
+
+    unsigned long idx;
+    _BitScanReverse64(&idx, b);
+    return Square(idx);
+
+    #else  // MSVC, WIN32
+
+    unsigned long idx;
+
+    if (b >> 32)
+    {
+        _BitScanReverse(&idx, int32_t(b >> 32));
+        return Square(idx + 32);
+    }
+    else
+    {
+        _BitScanReverse(&idx, int32_t(b));
+        return Square(idx);
+    }
+    #endif
+#else  // Compiler is neither GCC nor MSVC compatible
+    #error "Compiler not supported."
+#endif
+}
+
+// Returns the bitboard of the least significant
+// square of a non-zero bitboard. It is equivalent to square_bb(lsb(bb)).
 inline Bitboard least_significant_square_bb(Bitboard b) {
-  assert(b);
+    assert(b);
-  return b & -b;
+    return b & -b;
 }
 
-/// pop_lsb() finds and clears the least significant bit in a non-zero bitboard
+// Finds and clears the least significant bit in a non-zero bitboard.
-
 inline Square pop_lsb(Bitboard& b) {
-  assert(b);
+    assert(b);
-  const Square s = lsb(b);
+    const Square s = lsb(b);
-  b &= b - 1;
+    b &= b - 1;
-  return s;
+    return s;
 }
 
+}  // namespace Stockfish
 
-/// frontmost_sq() returns the most advanced square for the given color,
+#endif  // #ifndef BITBOARD_H_INCLUDED
-/// requires a non-zero bitboard.
-inline Square frontmost_sq(Color c, Bitboard b) {
-  assert(b);
-  return c == WHITE ? msb(b) : lsb(b);
-}
-
-} // namespace Stockfish
-
-#endif // #ifndef BITBOARD_H_INCLUDED

src/endgame.cpp

@@ -1,747 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <cassert>
-
-#include "bitboard.h"
-#include "endgame.h"
-#include "movegen.h"
-
-namespace Stockfish {
-
-namespace {
-
-  // Used to drive the king towards the edge of the board
-  // in KX vs K and KQ vs KR endgames.
-  // Values range from 27 (center squares) to 90 (in the corners)
-  inline int push_to_edge(Square s) {
-      int rd = edge_distance(rank_of(s)), fd = edge_distance(file_of(s));
-      return 90 - (7 * fd * fd / 2 + 7 * rd * rd / 2);
-  }
-
-  // Used to drive the king towards A1H8 corners in KBN vs K endgames.
-  // Values range from 0 on A8H1 diagonal to 7 in A1H8 corners
-  inline int push_to_corner(Square s) {
-      return abs(7 - rank_of(s) - file_of(s));
-  }
-
-  // Drive a piece close to or away from another piece
-  inline int push_close(Square s1, Square s2) { return 140 - 20 * distance(s1, s2); }
-  inline int push_away(Square s1, Square s2) { return 120 - push_close(s1, s2); }
-
-#ifndef NDEBUG
-  bool verify_material(const Position& pos, Color c, Value npm, int pawnsCnt) {
-    return pos.non_pawn_material(c) == npm && pos.count<PAWN>(c) == pawnsCnt;
-  }
-#endif
-
-  // Map the square as if strongSide is white and strongSide's only pawn
-  // is on the left half of the board.
-  Square normalize(const Position& pos, Color strongSide, Square sq) {
-
-    assert(pos.count<PAWN>(strongSide) == 1);
-
-    if (file_of(pos.square<PAWN>(strongSide)) >= FILE_E)
-        sq = flip_file(sq);
-
-    return strongSide == WHITE ? sq : flip_rank(sq);
-  }
-
-} // namespace
-
-
-namespace Endgames {
-
-  std::pair<Map<Value>, Map<ScaleFactor>> maps;
-
-  void init() {
-
-    add<KPK>("KPK");
-    add<KNNK>("KNNK");
-    add<KBNK>("KBNK");
-    add<KRKP>("KRKP");
-    add<KRKB>("KRKB");
-    add<KRKN>("KRKN");
-    add<KQKP>("KQKP");
-    add<KQKR>("KQKR");
-    add<KNNKP>("KNNKP");
-
-    add<KRPKR>("KRPKR");
-    add<KRPKB>("KRPKB");
-    add<KBPKB>("KBPKB");
-    add<KBPKN>("KBPKN");
-    add<KBPPKB>("KBPPKB");
-    add<KRPPKRP>("KRPPKRP");
-  }
-}
-
-
-/// Mate with KX vs K. This function is used to evaluate positions with
-/// king and plenty of material vs a lone king. It simply gives the
-/// attacking side a bonus for driving the defending king towards the edge
-/// of the board, and for keeping the distance between the two kings small.
-template<>
-Value Endgame<KXK>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 0));
-  assert(!pos.checkers()); // Eval is never called when in check
-
-  // Stalemate detection with lone king
-  if (pos.side_to_move() == weakSide && !MoveList<LEGAL>(pos).size())
-      return VALUE_DRAW;
-
-  Square strongKing = pos.square<KING>(strongSide);
-  Square weakKing   = pos.square<KING>(weakSide);
-
-  Value result =  pos.non_pawn_material(strongSide)
-                + pos.count<PAWN>(strongSide) * PawnValueEg
-                + push_to_edge(weakKing)
-                + push_close(strongKing, weakKing);
-
-  if (   pos.count<QUEEN>(strongSide)
-      || pos.count<ROOK>(strongSide)
-      ||(pos.count<BISHOP>(strongSide) && pos.count<KNIGHT>(strongSide))
-      || (   (pos.pieces(strongSide, BISHOP) & ~DarkSquares)
-          && (pos.pieces(strongSide, BISHOP) &  DarkSquares)))
-      result = std::min(result + VALUE_KNOWN_WIN, VALUE_TB_WIN_IN_MAX_PLY - 1);
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// Mate with KBN vs K. This is similar to KX vs K, but we have to drive the
-/// defending king towards a corner square that our bishop attacks.
-template<>
-Value Endgame<KBNK>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, KnightValueMg + BishopValueMg, 0));
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 0));
-
-  Square strongKing   = pos.square<KING>(strongSide);
-  Square strongBishop = pos.square<BISHOP>(strongSide);
-  Square weakKing     = pos.square<KING>(weakSide);
-
-  // If our bishop does not attack A1/H8, we flip the enemy king square
-  // to drive to opposite corners (A8/H1).
-
-  Value result =  (VALUE_KNOWN_WIN + 3520)
-                + push_close(strongKing, weakKing)
-                + 420 * push_to_corner(opposite_colors(strongBishop, SQ_A1) ? flip_file(weakKing) : weakKing);
-
-  assert(abs(result) < VALUE_TB_WIN_IN_MAX_PLY);
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KP vs K. This endgame is evaluated with the help of a bitbase
-template<>
-Value Endgame<KPK>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, VALUE_ZERO, 1));
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 0));
-
-  // Assume strongSide is white and the pawn is on files A-D
-  Square strongKing = normalize(pos, strongSide, pos.square<KING>(strongSide));
-  Square strongPawn = normalize(pos, strongSide, pos.square<PAWN>(strongSide));
-  Square weakKing   = normalize(pos, strongSide, pos.square<KING>(weakSide));
-
-  Color us = strongSide == pos.side_to_move() ? WHITE : BLACK;
-
-  if (!Bitbases::probe(strongKing, strongPawn, weakKing, us))
-      return VALUE_DRAW;
-
-  Value result = VALUE_KNOWN_WIN + PawnValueEg + Value(rank_of(strongPawn));
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KR vs KP. This is a somewhat tricky endgame to evaluate precisely without
-/// a bitbase. The function below returns drawish scores when the pawn is
-/// far advanced with support of the king, while the attacking king is far
-/// away.
-template<>
-Value Endgame<KRKP>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 0));
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 1));
-
-  Square strongKing = pos.square<KING>(strongSide);
-  Square weakKing   = pos.square<KING>(weakSide);
-  Square strongRook = pos.square<ROOK>(strongSide);
-  Square weakPawn   = pos.square<PAWN>(weakSide);
-  Square queeningSquare = make_square(file_of(weakPawn), relative_rank(weakSide, RANK_8));
-  Value result;
-
-  // If the stronger side's king is in front of the pawn, it's a win
-  if (forward_file_bb(strongSide, strongKing) & weakPawn)
-      result = RookValueEg - distance(strongKing, weakPawn);
-
-  // If the weaker side's king is too far from the pawn and the rook,
-  // it's a win.
-  else if (   distance(weakKing, weakPawn) >= 3 + (pos.side_to_move() == weakSide)
-           && distance(weakKing, strongRook) >= 3)
-      result = RookValueEg - distance(strongKing, weakPawn);
-
-  // If the pawn is far advanced and supported by the defending king,
-  // the position is drawish
-  else if (   relative_rank(strongSide, weakKing) <= RANK_3
-           && distance(weakKing, weakPawn) == 1
-           && relative_rank(strongSide, strongKing) >= RANK_4
-           && distance(strongKing, weakPawn) > 2 + (pos.side_to_move() == strongSide))
-      result = Value(80) - 8 * distance(strongKing, weakPawn);
-
-  else
-      result =  Value(200) - 8 * (  distance(strongKing, weakPawn + pawn_push(weakSide))
-                                  - distance(weakKing, weakPawn + pawn_push(weakSide))
-                                  - distance(weakPawn, queeningSquare));
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KR vs KB. This is very simple, and always returns drawish scores. The
-/// score is slightly bigger when the defending king is close to the edge.
-template<>
-Value Endgame<KRKB>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 0));
-  assert(verify_material(pos, weakSide, BishopValueMg, 0));
-
-  Value result = Value(push_to_edge(pos.square<KING>(weakSide)));
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KR vs KN. The attacking side has slightly better winning chances than
-/// in KR vs KB, particularly if the king and the knight are far apart.
-template<>
-Value Endgame<KRKN>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 0));
-  assert(verify_material(pos, weakSide, KnightValueMg, 0));
-
-  Square weakKing   = pos.square<KING>(weakSide);
-  Square weakKnight = pos.square<KNIGHT>(weakSide);
-  Value result = Value(push_to_edge(weakKing) + push_away(weakKing, weakKnight));
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KQ vs KP. In general, this is a win for the stronger side, but there are a
-/// few important exceptions. A pawn on 7th rank and on the A,C,F or H files
-/// with a king positioned next to it can be a draw, so in that case, we only
-/// use the distance between the kings.
-template<>
-Value Endgame<KQKP>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, QueenValueMg, 0));
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 1));
-
-  Square strongKing = pos.square<KING>(strongSide);
-  Square weakKing   = pos.square<KING>(weakSide);
-  Square weakPawn   = pos.square<PAWN>(weakSide);
-
-  Value result = Value(push_close(strongKing, weakKing));
-
-  if (   relative_rank(weakSide, weakPawn) != RANK_7
-      || distance(weakKing, weakPawn) != 1
-      || ((FileBBB | FileDBB | FileEBB | FileGBB) & weakPawn))
-      result += QueenValueEg - PawnValueEg;
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KQ vs KR. This is almost identical to KX vs K: we give the attacking
-/// king a bonus for having the kings close together, and for forcing the
-/// defending king towards the edge. If we also take care to avoid null move for
-/// the defending side in the search, this is usually sufficient to win KQ vs KR.
-template<>
-Value Endgame<KQKR>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, QueenValueMg, 0));
-  assert(verify_material(pos, weakSide, RookValueMg, 0));
-
-  Square strongKing = pos.square<KING>(strongSide);
-  Square weakKing   = pos.square<KING>(weakSide);
-
-  Value result =  QueenValueEg
-                - RookValueEg
-                + push_to_edge(weakKing)
-                + push_close(strongKing, weakKing);
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// KNN vs KP. Very drawish, but there are some mate opportunities if we can
-/// press the weakSide King to a corner before the pawn advances too much.
-template<>
-Value Endgame<KNNKP>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, 2 * KnightValueMg, 0));
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 1));
-
-  Square weakKing = pos.square<KING>(weakSide);
-  Square weakPawn = pos.square<PAWN>(weakSide);
-
-  Value result =      PawnValueEg
-               +  2 * push_to_edge(weakKing)
-               - 10 * relative_rank(weakSide, weakPawn);
-
-  return strongSide == pos.side_to_move() ? result : -result;
-}
-
-
-/// Some cases of trivial draws
-template<> Value Endgame<KNNK>::operator()(const Position&) const { return VALUE_DRAW; }
-
-
-/// KB and one or more pawns vs K. It checks for draws with rook pawns and
-/// a bishop of the wrong color. If such a draw is detected, SCALE_FACTOR_DRAW
-/// is returned. If not, the return value is SCALE_FACTOR_NONE, i.e. no scaling
-/// will be used.
-template<>
-ScaleFactor Endgame<KBPsK>::operator()(const Position& pos) const {
-
-  assert(pos.non_pawn_material(strongSide) == BishopValueMg);
-  assert(pos.count<PAWN>(strongSide) >= 1);
-
-  // No assertions about the material of weakSide, because we want draws to
-  // be detected even when the weaker side has some pawns.
-
-  Bitboard strongPawns = pos.pieces(strongSide, PAWN);
-  Bitboard allPawns = pos.pieces(PAWN);
-
-  Square strongBishop = pos.square<BISHOP>(strongSide);
-  Square weakKing = pos.square<KING>(weakSide);
-  Square strongKing = pos.square<KING>(strongSide);
-
-  // All strongSide pawns are on a single rook file?
-  if (!(strongPawns & ~FileABB) || !(strongPawns & ~FileHBB))
-  {
-      Square queeningSquare = relative_square(strongSide, make_square(file_of(lsb(strongPawns)), RANK_8));
-
-      if (   opposite_colors(queeningSquare, strongBishop)
-          && distance(queeningSquare, weakKing) <= 1)
-          return SCALE_FACTOR_DRAW;
-  }
-
-  // If all the pawns are on the same B or G file, then it's potentially a draw
-  if ((!(allPawns & ~FileBBB) || !(allPawns & ~FileGBB))
-      && pos.non_pawn_material(weakSide) == 0
-      && pos.count<PAWN>(weakSide) >= 1)
-  {
-      // Get the least advanced weakSide pawn
-      Square weakPawn = frontmost_sq(strongSide, pos.pieces(weakSide, PAWN));
-
-      // There's potential for a draw if our pawn is blocked on the 7th rank,
-      // the bishop cannot attack it or they only have one pawn left.
-      if (   relative_rank(strongSide, weakPawn) == RANK_7
-          && (strongPawns & (weakPawn + pawn_push(weakSide)))
-          && (opposite_colors(strongBishop, weakPawn) || !more_than_one(strongPawns)))
-      {
-          int strongKingDist = distance(weakPawn, strongKing);
-          int weakKingDist = distance(weakPawn, weakKing);
-
-          // It's a draw if the weak king is on its back two ranks, within 2
-          // squares of the blocking pawn and the strong king is not
-          // closer. (I think this rule only fails in practically
-          // unreachable positions such as 5k1K/6p1/6P1/8/8/3B4/8/8 w
-          // and positions where qsearch will immediately correct the
-          // problem such as 8/4k1p1/6P1/1K6/3B4/8/8/8 w).
-          if (   relative_rank(strongSide, weakKing) >= RANK_7
-              && weakKingDist <= 2
-              && weakKingDist <= strongKingDist)
-              return SCALE_FACTOR_DRAW;
-      }
-  }
-
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// KQ vs KR and one or more pawns. It tests for fortress draws with a rook on
-/// the third rank defended by a pawn.
-template<>
-ScaleFactor Endgame<KQKRPs>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, QueenValueMg, 0));
-  assert(pos.count<ROOK>(weakSide) == 1);
-  assert(pos.count<PAWN>(weakSide) >= 1);
-
-  Square strongKing = pos.square<KING>(strongSide);
-  Square weakKing   = pos.square<KING>(weakSide);
-  Square weakRook   = pos.square<ROOK>(weakSide);
-
-  if (    relative_rank(weakSide,   weakKing) <= RANK_2
-      &&  relative_rank(weakSide, strongKing) >= RANK_4
-      &&  relative_rank(weakSide,   weakRook) == RANK_3
-      && (  pos.pieces(weakSide, PAWN)
-          & attacks_bb<KING>(weakKing)
-          & pawn_attacks_bb(strongSide, weakRook)))
-          return SCALE_FACTOR_DRAW;
-
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// KRP vs KR. This function knows a handful of the most important classes of
-/// drawn positions, but is far from perfect. It would probably be a good idea
-/// to add more knowledge in the future.
-///
-/// It would also be nice to rewrite the actual code for this function,
-/// which is mostly copied from Glaurung 1.x, and isn't very pretty.
-template<>
-ScaleFactor Endgame<KRPKR>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 1));
-  assert(verify_material(pos, weakSide,   RookValueMg, 0));
-
-  // Assume strongSide is white and the pawn is on files A-D
-  Square strongKing = normalize(pos, strongSide, pos.square<KING>(strongSide));
-  Square strongRook = normalize(pos, strongSide, pos.square<ROOK>(strongSide));
-  Square strongPawn = normalize(pos, strongSide, pos.square<PAWN>(strongSide));
-  Square weakKing = normalize(pos, strongSide, pos.square<KING>(weakSide));
-  Square weakRook = normalize(pos, strongSide, pos.square<ROOK>(weakSide));
-
-  File pawnFile = file_of(strongPawn);
-  Rank pawnRank = rank_of(strongPawn);
-  Square queeningSquare = make_square(pawnFile, RANK_8);
-  int tempo = (pos.side_to_move() == strongSide);
-
-  // If the pawn is not too far advanced and the defending king defends the
-  // queening square, use the third-rank defence.
-  if (   pawnRank <= RANK_5
-      && distance(weakKing, queeningSquare) <= 1
-      && strongKing <= SQ_H5
-      && (rank_of(weakRook) == RANK_6 || (pawnRank <= RANK_3 && rank_of(strongRook) != RANK_6)))
-      return SCALE_FACTOR_DRAW;
-
-  // The defending side saves a draw by checking from behind in case the pawn
-  // has advanced to the 6th rank with the king behind.
-  if (   pawnRank == RANK_6
-      && distance(weakKing, queeningSquare) <= 1
-      && rank_of(strongKing) + tempo <= RANK_6
-      && (rank_of(weakRook) == RANK_1 || (!tempo && distance<File>(weakRook, strongPawn) >= 3)))
-      return SCALE_FACTOR_DRAW;
-
-  if (   pawnRank >= RANK_6
-      && weakKing == queeningSquare
-      && rank_of(weakRook) == RANK_1
-      && (!tempo || distance(strongKing, strongPawn) >= 2))
-      return SCALE_FACTOR_DRAW;
-
-  // White pawn on a7 and rook on a8 is a draw if black's king is on g7 or h7
-  // and the black rook is behind the pawn.
-  if (   strongPawn == SQ_A7
-      && strongRook == SQ_A8
-      && (weakKing == SQ_H7 || weakKing == SQ_G7)
-      && file_of(weakRook) == FILE_A
-      && (rank_of(weakRook) <= RANK_3 || file_of(strongKing) >= FILE_D || rank_of(strongKing) <= RANK_5))
-      return SCALE_FACTOR_DRAW;
-
-  // If the defending king blocks the pawn and the attacking king is too far
-  // away, it's a draw.
-  if (   pawnRank <= RANK_5
-      && weakKing == strongPawn + NORTH
-      && distance(strongKing, strongPawn) - tempo >= 2
-      && distance(strongKing, weakRook) - tempo >= 2)
-      return SCALE_FACTOR_DRAW;
-
-  // Pawn on the 7th rank supported by the rook from behind usually wins if the
-  // attacking king is closer to the queening square than the defending king,
-  // and the defending king cannot gain tempi by threatening the attacking rook.
-  if (   pawnRank == RANK_7
-      && pawnFile != FILE_A
-      && file_of(strongRook) == pawnFile
-      && strongRook != queeningSquare
-      && (distance(strongKing, queeningSquare) < distance(weakKing, queeningSquare) - 2 + tempo)
-      && (distance(strongKing, queeningSquare) < distance(weakKing, strongRook) + tempo))
-      return ScaleFactor(SCALE_FACTOR_MAX - 2 * distance(strongKing, queeningSquare));
-
-  // Similar to the above, but with the pawn further back
-  if (   pawnFile != FILE_A
-      && file_of(strongRook) == pawnFile
-      && strongRook < strongPawn
-      && (distance(strongKing, queeningSquare) < distance(weakKing, queeningSquare) - 2 + tempo)
-      && (distance(strongKing, strongPawn + NORTH) < distance(weakKing, strongPawn + NORTH) - 2 + tempo)
-      && (  distance(weakKing, strongRook) + tempo >= 3
-          || (    distance(strongKing, queeningSquare) < distance(weakKing, strongRook) + tempo
-              && (distance(strongKing, strongPawn + NORTH) < distance(weakKing, strongPawn) + tempo))))
-      return ScaleFactor(  SCALE_FACTOR_MAX
-                         - 8 * distance(strongPawn, queeningSquare)
-                         - 2 * distance(strongKing, queeningSquare));
-
-  // If the pawn is not far advanced and the defending king is somewhere in
-  // the pawn's path, it's probably a draw.
-  if (pawnRank <= RANK_4 && weakKing > strongPawn)
-  {
-      if (file_of(weakKing) == file_of(strongPawn))
-          return ScaleFactor(10);
-      if (   distance<File>(weakKing, strongPawn) == 1
-          && distance(strongKing, weakKing) > 2)
-          return ScaleFactor(24 - 2 * distance(strongKing, weakKing));
-  }
-  return SCALE_FACTOR_NONE;
-}
-
-template<>
-ScaleFactor Endgame<KRPKB>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 1));
-  assert(verify_material(pos, weakSide, BishopValueMg, 0));
-
-  // Test for a rook pawn
-  if (pos.pieces(PAWN) & (FileABB | FileHBB))
-  {
-      Square weakKing = pos.square<KING>(weakSide);
-      Square weakBishop = pos.square<BISHOP>(weakSide);
-      Square strongKing = pos.square<KING>(strongSide);
-      Square strongPawn = pos.square<PAWN>(strongSide);
-      Rank pawnRank = relative_rank(strongSide, strongPawn);
-      Direction push = pawn_push(strongSide);
-
-      // If the pawn is on the 5th rank and the pawn (currently) is on
-      // the same color square as the bishop then there is a chance of
-      // a fortress. Depending on the king position give a moderate
-      // reduction or a stronger one if the defending king is near the
-      // corner but not trapped there.
-      if (pawnRank == RANK_5 && !opposite_colors(weakBishop, strongPawn))
-      {
-          int d = distance(strongPawn + 3 * push, weakKing);
-
-          if (d <= 2 && !(d == 0 && weakKing == strongKing + 2 * push))
-              return ScaleFactor(24);
-          else
-              return ScaleFactor(48);
-      }
-
-      // When the pawn has moved to the 6th rank we can be fairly sure
-      // it's drawn if the bishop attacks the square in front of the
-      // pawn from a reasonable distance and the defending king is near
-      // the corner
-      if (   pawnRank == RANK_6
-          && distance(strongPawn + 2 * push, weakKing) <= 1
-          && (attacks_bb<BISHOP>(weakBishop) & (strongPawn + push))
-          && distance<File>(weakBishop, strongPawn) >= 2)
-          return ScaleFactor(8);
-  }
-
-  return SCALE_FACTOR_NONE;
-}
-
-/// KRPP vs KRP. There is just a single rule: if the stronger side has no passed
-/// pawns and the defending king is actively placed, the position is drawish.
-template<>
-ScaleFactor Endgame<KRPPKRP>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, RookValueMg, 2));
-  assert(verify_material(pos, weakSide,   RookValueMg, 1));
-
-  Square strongPawn1 = lsb(pos.pieces(strongSide, PAWN));
-  Square strongPawn2 = msb(pos.pieces(strongSide, PAWN));
-  Square weakKing = pos.square<KING>(weakSide);
-
-  // Does the stronger side have a passed pawn?
-  if (pos.pawn_passed(strongSide, strongPawn1) || pos.pawn_passed(strongSide, strongPawn2))
-      return SCALE_FACTOR_NONE;
-
-  Rank pawnRank = std::max(relative_rank(strongSide, strongPawn1), relative_rank(strongSide, strongPawn2));
-
-  if (   distance<File>(weakKing, strongPawn1) <= 1
-      && distance<File>(weakKing, strongPawn2) <= 1
-      && relative_rank(strongSide, weakKing) > pawnRank)
-  {
-      assert(pawnRank > RANK_1 && pawnRank < RANK_7);
-      return ScaleFactor(7 * pawnRank);
-  }
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// K and two or more pawns vs K. There is just a single rule here: if all pawns
-/// are on the same rook file and are blocked by the defending king, it's a draw.
-template<>
-ScaleFactor Endgame<KPsK>::operator()(const Position& pos) const {
-
-  assert(pos.non_pawn_material(strongSide) == VALUE_ZERO);
-  assert(pos.count<PAWN>(strongSide) >= 2);
-  assert(verify_material(pos, weakSide, VALUE_ZERO, 0));
-
-  Square weakKing = pos.square<KING>(weakSide);
-  Bitboard strongPawns = pos.pieces(strongSide, PAWN);
-
-  // If all pawns are ahead of the king on a single rook file, it's a draw.
-  if (   !(strongPawns & ~(FileABB | FileHBB))
-      && !(strongPawns & ~passed_pawn_span(weakSide, weakKing)))
-      return SCALE_FACTOR_DRAW;
-
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// KBP vs KB. There are two rules: if the defending king is somewhere along the
-/// path of the pawn, and the square of the king is not of the same color as the
-/// stronger side's bishop, it's a draw. If the two bishops have opposite color,
-/// it's almost always a draw.
-template<>
-ScaleFactor Endgame<KBPKB>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, BishopValueMg, 1));
-  assert(verify_material(pos, weakSide,   BishopValueMg, 0));
-
-  Square strongPawn = pos.square<PAWN>(strongSide);
-  Square strongBishop = pos.square<BISHOP>(strongSide);
-  Square weakBishop = pos.square<BISHOP>(weakSide);
-  Square weakKing = pos.square<KING>(weakSide);
-
-  // Case 1: Defending king blocks the pawn, and cannot be driven away
-  if (   (forward_file_bb(strongSide, strongPawn) & weakKing)
-      && (   opposite_colors(weakKing, strongBishop)
-          || relative_rank(strongSide, weakKing) <= RANK_6))
-      return SCALE_FACTOR_DRAW;
-
-  // Case 2: Opposite colored bishops
-  if (opposite_colors(strongBishop, weakBishop))
-      return SCALE_FACTOR_DRAW;
-
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// KBPP vs KB. It detects a few basic draws with opposite-colored bishops
-template<>
-ScaleFactor Endgame<KBPPKB>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, BishopValueMg, 2));
-  assert(verify_material(pos, weakSide,   BishopValueMg, 0));
-
-  Square strongBishop = pos.square<BISHOP>(strongSide);
-  Square weakBishop   = pos.square<BISHOP>(weakSide);
-
-  if (!opposite_colors(strongBishop, weakBishop))
-      return SCALE_FACTOR_NONE;
-
-  Square weakKing = pos.square<KING>(weakSide);
-  Square strongPawn1 = lsb(pos.pieces(strongSide, PAWN));
-  Square strongPawn2 = msb(pos.pieces(strongSide, PAWN));
-  Square blockSq1, blockSq2;
-
-  if (relative_rank(strongSide, strongPawn1) > relative_rank(strongSide, strongPawn2))
-  {
-      blockSq1 = strongPawn1 + pawn_push(strongSide);
-      blockSq2 = make_square(file_of(strongPawn2), rank_of(strongPawn1));
-  }
-  else
-  {
-      blockSq1 = strongPawn2 + pawn_push(strongSide);
-      blockSq2 = make_square(file_of(strongPawn1), rank_of(strongPawn2));
-  }
-
-  switch (distance<File>(strongPawn1, strongPawn2))
-  {
-  case 0:
-    // Both pawns are on the same file. It's an easy draw if the defender firmly
-    // controls some square in the frontmost pawn's path.
-    if (   file_of(weakKing) == file_of(blockSq1)
-        && relative_rank(strongSide, weakKing) >= relative_rank(strongSide, blockSq1)
-        && opposite_colors(weakKing, strongBishop))
-        return SCALE_FACTOR_DRAW;
-    else
-        return SCALE_FACTOR_NONE;
-
-  case 1:
-    // Pawns on adjacent files. It's a draw if the defender firmly controls the
-    // square in front of the frontmost pawn's path, and the square diagonally
-    // behind this square on the file of the other pawn.
-    if (   weakKing == blockSq1
-        && opposite_colors(weakKing, strongBishop)
-        && (   weakBishop == blockSq2
-            || (attacks_bb<BISHOP>(blockSq2, pos.pieces()) & pos.pieces(weakSide, BISHOP))
-            || distance<Rank>(strongPawn1, strongPawn2) >= 2))
-        return SCALE_FACTOR_DRAW;
-
-    else if (   weakKing == blockSq2
-             && opposite_colors(weakKing, strongBishop)
-             && (   weakBishop == blockSq1
-                 || (attacks_bb<BISHOP>(blockSq1, pos.pieces()) & pos.pieces(weakSide, BISHOP))))
-        return SCALE_FACTOR_DRAW;
-    else
-        return SCALE_FACTOR_NONE;
-
-  default:
-    // The pawns are not on the same file or adjacent files. No scaling.
-    return SCALE_FACTOR_NONE;
-  }
-}
-
-
-/// KBP vs KN. There is a single rule: if the defending king is somewhere along
-/// the path of the pawn, and the square of the king is not of the same color as
-/// the stronger side's bishop, it's a draw.
-template<>
-ScaleFactor Endgame<KBPKN>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, BishopValueMg, 1));
-  assert(verify_material(pos, weakSide, KnightValueMg, 0));
-
-  Square strongPawn = pos.square<PAWN>(strongSide);
-  Square strongBishop = pos.square<BISHOP>(strongSide);
-  Square weakKing = pos.square<KING>(weakSide);
-
-  if (   file_of(weakKing) == file_of(strongPawn)
-      && relative_rank(strongSide, strongPawn) < relative_rank(strongSide, weakKing)
-      && (   opposite_colors(weakKing, strongBishop)
-          || relative_rank(strongSide, weakKing) <= RANK_6))
-      return SCALE_FACTOR_DRAW;
-
-  return SCALE_FACTOR_NONE;
-}
-
-
-/// KP vs KP. This is done by removing the weakest side's pawn and probing the
-/// KP vs K bitbase: if the weakest side has a draw without the pawn, it probably
-/// has at least a draw with the pawn as well. The exception is when the stronger
-/// side's pawn is far advanced and not on a rook file; in this case it is often
-/// possible to win (e.g. 8/4k3/3p4/3P4/6K1/8/8/8 w - - 0 1).
-template<>
-ScaleFactor Endgame<KPKP>::operator()(const Position& pos) const {
-
-  assert(verify_material(pos, strongSide, VALUE_ZERO, 1));
-  assert(verify_material(pos, weakSide,   VALUE_ZERO, 1));
-
-  // Assume strongSide is white and the pawn is on files A-D
-  Square strongKing = normalize(pos, strongSide, pos.square<KING>(strongSide));
-  Square weakKing   = normalize(pos, strongSide, pos.square<KING>(weakSide));
-  Square strongPawn = normalize(pos, strongSide, pos.square<PAWN>(strongSide));
-
-  Color us = strongSide == pos.side_to_move() ? WHITE : BLACK;
-
-  // If the pawn has advanced to the fifth rank or further, and is not a
-  // rook pawn, it's too dangerous to assume that it's at least a draw.
-  if (rank_of(strongPawn) >= RANK_5 && file_of(strongPawn) != FILE_A)
-      return SCALE_FACTOR_NONE;
-
-  // Probe the KPK bitbase with the weakest side's pawn removed. If it's a draw,
-  // it's probably at least a draw even with the pawn.
-  return Bitbases::probe(strongKing, strongPawn, weakKing, us) ? SCALE_FACTOR_NONE : SCALE_FACTOR_DRAW;
-}
-
-} // namespace Stockfish

src/endgame.h

@@ -1,126 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifndef ENDGAME_H_INCLUDED
-#define ENDGAME_H_INCLUDED
-
-#include <memory>
-#include <string>
-#include <type_traits>
-#include <unordered_map>
-#include <utility>
-
-#include "position.h"
-#include "types.h"
-
-namespace Stockfish {
-
-/// EndgameCode lists all supported endgame functions by corresponding codes
-
-enum EndgameCode {
-
-  EVALUATION_FUNCTIONS,
-  KNNK,  // KNN vs K
-  KNNKP, // KNN vs KP
-  KXK,   // Generic "mate lone king" eval
-  KBNK,  // KBN vs K
-  KPK,   // KP vs K
-  KRKP,  // KR vs KP
-  KRKB,  // KR vs KB
-  KRKN,  // KR vs KN
-  KQKP,  // KQ vs KP
-  KQKR,  // KQ vs KR
-
-  SCALING_FUNCTIONS,
-  KBPsK,   // KB and pawns vs K
-  KQKRPs,  // KQ vs KR and pawns
-  KRPKR,   // KRP vs KR
-  KRPKB,   // KRP vs KB
-  KRPPKRP, // KRPP vs KRP
-  KPsK,    // K and pawns vs K
-  KBPKB,   // KBP vs KB
-  KBPPKB,  // KBPP vs KB
-  KBPKN,   // KBP vs KN
-  KPKP     // KP vs KP
-};
-
-
-/// Endgame functions can be of two types depending on whether they return a
-/// Value or a ScaleFactor.
-
-template<EndgameCode E> using
-eg_type = typename std::conditional<(E < SCALING_FUNCTIONS), Value, ScaleFactor>::type;
-
-
-/// Base and derived functors for endgame evaluation and scaling functions
-
-template<typename T>
-struct EndgameBase {
-
-  explicit EndgameBase(Color c) : strongSide(c), weakSide(~c) {}
-  virtual ~EndgameBase() = default;
-  virtual T operator()(const Position&) const = 0;
-
-  const Color strongSide, weakSide;
-};
-
-
-template<EndgameCode E, typename T = eg_type<E>>
-struct Endgame : public EndgameBase<T> {
-
-  explicit Endgame(Color c) : EndgameBase<T>(c) {}
-  T operator()(const Position&) const override;
-};
-
-
-/// The Endgames namespace handles the pointers to endgame evaluation and scaling
-/// base objects in two std::map. We use polymorphism to invoke the actual
-/// endgame function by calling its virtual operator().
-
-namespace Endgames {
-
-  template<typename T> using Ptr = std::unique_ptr<EndgameBase<T>>;
-  template<typename T> using Map = std::unordered_map<Key, Ptr<T>>;
-
-  extern std::pair<Map<Value>, Map<ScaleFactor>> maps;
-
-  void init();
-
-  template<typename T>
-  Map<T>& map() {
-    return std::get<std::is_same<T, ScaleFactor>::value>(maps);
-  }
-
-  template<EndgameCode E, typename T = eg_type<E>>
-  void add(const std::string& code) {
-
-    StateInfo st;
-    map<T>()[Position().set(code, WHITE, &st).material_key()] = Ptr<T>(new Endgame<E>(WHITE));
-    map<T>()[Position().set(code, BLACK, &st).material_key()] = Ptr<T>(new Endgame<E>(BLACK));
-  }
-
-  template<typename T>
-  const EndgameBase<T>* probe(Key key) {
-    auto it = map<T>().find(key);
-    return it != map<T>().end() ? it->second.get() : nullptr;
-  }
-}
-
-} // namespace Stockfish
-
-#endif // #ifndef ENDGAME_H_INCLUDED

src/evaluate.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -20,49 +20,52 @@
 #define EVALUATE_H_INCLUDED
 
 #include <string>
-#include <optional>
+#include <unordered_map>
 
 #include "types.h"
 
 namespace Stockfish {
 
 class Position;
+class OptionsMap;
 
 namespace Eval {
 
-  std::string trace(Position& pos);
+std::string trace(Position& pos);
-  Value evaluate(const Position& pos);
 
-  // The default net name MUST follow the format nn-[SHA256 first 12 digits].nnue
+int   simple_eval(const Position& pos, Color c);
-  // for the build process (profile-build and fishtest) to work. Do not change the
+Value evaluate(const Position& pos, int optimism);
-  // name of the macro, as it is used in the Makefile.
-  #define EvalFileDefaultName   "nn-3c0aa92af1da.nnue"
 
-  namespace NNUE {
+// The default net name MUST follow the format nn-[SHA256 first 12 digits].nnue
-    enum struct UseNNUEMode
+// for the build process (profile-build and fishtest) to work. Do not change the
-    {
+// name of the macro, as it is used in the Makefile.
-      False,
+#define EvalFileDefaultNameBig "nn-b1a57edbea57.nnue"
-      True,
+#define EvalFileDefaultNameSmall "nn-baff1ede1f90.nnue"
-      Pure
-    };
 
-    extern UseNNUEMode useNNUE;
+struct EvalFile {
-    extern std::string currentEvalFileName;
+    // UCI option name
+    std::string optionName;
+    // Default net name, will use one of the macros above
+    std::string defaultName;
+    // Selected net name, either via uci option or default
+    std::string current;
+    // Net description extracted from the net file
+    std::string netDescription;
+};
 
-    std::string trace(Position& pos);
+namespace NNUE {
-    Value evaluate(const Position& pos, bool adjusted = false);
 
-    void init();
+enum NetSize : int;
-    void verify();
 
-    bool load_eval(std::string name, std::istream& stream);
+using EvalFiles = std::unordered_map<Eval::NNUE::NetSize, EvalFile>;
-    bool save_eval(std::ostream& stream);
-    bool save_eval(const std::optional<std::string>& filename);
 
-  } // namespace NNUE
+EvalFiles load_networks(const std::string&, const OptionsMap&, EvalFiles);
+void      verify(const OptionsMap&, const EvalFiles&);
 
-} // namespace Eval
+}  // namespace NNUE
 
-} // namespace Stockfish
+}  // namespace Eval
 
-#endif // #ifndef EVALUATE_H_INCLUDED
+}  // namespace Stockfish
+
+#endif  // #ifndef EVALUATE_H_INCLUDED

src/incbin/incbin.h

@@ -3,8 +3,8 @@
  * @author Dale Weiler
  * @brief Utility for including binary files
  *
- * Facilities for including binary files into the current translation unit and
+ * Facilities for including binary files into the current translation unit
- * making use from them externally in other translation units.
+ * and making use of them externally in other translation units.
  */
 #ifndef INCBIN_HDR
 #define INCBIN_HDR
@@ -139,7 +139,7 @@
 #endif
 
 #if defined(__APPLE__)
-/* The directives are different for Apple branded compilers */
+/* The directives are different for Apple-branded compilers */
 #  define INCBIN_SECTION         INCBIN_OUTPUT_SECTION "\n"
 #  define INCBIN_GLOBAL(NAME)    ".globl " INCBIN_MANGLE INCBIN_STRINGIZE(INCBIN_PREFIX) #NAME "\n"
 #  define INCBIN_INT             ".long "
@@ -261,8 +261,8 @@
     INCBIN_STRINGIZE( \
         INCBIN_STYLE_IDENT(TYPE)) \
 
-/* Generate the global labels by indirectly invoking the macro with our style
+/* Generate the global labels by indirectly invoking the macro
- * type and concatenating the name against them. */
+ * with our style type and concatenate the name against them. */
 #define INCBIN_GLOBAL_LABELS(NAME, TYPE) \
     INCBIN_INVOKE( \
         INCBIN_GLOBAL, \

src/main.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -17,39 +17,32 @@
 */
 
 #include <iostream>
-
+#include <unordered_map>
-#include "nnue/evaluate_nnue.h"
 
 #include "bitboard.h"
-#include "endgame.h"
+#include "evaluate.h"
+#include "misc.h"
 #include "position.h"
-#include "psqt.h"
+#include "tune.h"
-#include "search.h"
+#include "types.h"
-#include "syzygy/tbprobe.h"
-#include "thread.h"
-#include "tt.h"
 #include "uci.h"
 
 using namespace Stockfish;
 
 int main(int argc, char* argv[]) {
 
-  std::cout << engine_info() << std::endl;
+    std::cout << engine_info() << std::endl;
 
-  CommandLine::init(argc, argv);
+    Bitboards::init();
-  UCI::init(Options);
+    Position::init();
-  Tune::init();
-  PSQT::init();
-  Bitboards::init();
-  Position::init();
-  Bitbases::init();
-  Endgames::init();
-  Threads.set(size_t(Options["Threads"]));
-  Search::clear(); // After threads are up
-  Eval::NNUE::init();
 
-  UCI::loop(argc, argv);
+    UCI uci(argc, argv);
 
-  Threads.set(0);
+    Tune::init(uci.options);
-  return 0;
+
+    uci.evalFiles = Eval::NNUE::load_networks(uci.workingDirectory(), uci.options, uci.evalFiles);
+
+    uci.loop();
+
+    return 0;
 }

src/material.cpp

@@ -1,229 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <cassert>
-#include <cstring>   // For std::memset
-
-#include "material.h"
-#include "thread.h"
-
-using namespace std;
-
-namespace Stockfish {
-
-namespace {
-  #define S(mg, eg) make_score(mg, eg)
-
-  // Polynomial material imbalance parameters
-
-  // One Score parameter for each pair (our piece, another of our pieces)
-  constexpr Score QuadraticOurs[][PIECE_TYPE_NB] = {
-    // OUR PIECE 2
-    // bishop pair    pawn         knight       bishop       rook           queen
-    {S(1419, 1455)                                                                  }, // Bishop pair
-    {S( 101,   28), S( 37,  39)                                                     }, // Pawn
-    {S(  57,   64), S(249, 187), S(-49, -62)                                        }, // Knight      OUR PIECE 1
-    {S(   0,    0), S(118, 137), S( 10,  27), S(  0,   0)                           }, // Bishop
-    {S( -63,  -68), S( -5,   3), S(100,  81), S(132, 118), S(-246, -244)            }, // Rook
-    {S(-210, -211), S( 37,  14), S(147, 141), S(161, 105), S(-158, -174), S(-9,-31) }  // Queen
-  };
-
-  // One Score parameter for each pair (our piece, their piece)
-  constexpr Score QuadraticTheirs[][PIECE_TYPE_NB] = {
-    // THEIR PIECE
-    // bishop pair   pawn         knight       bishop       rook         queen
-    {                                                                               }, // Bishop pair
-    {S(  33,  30)                                                                   }, // Pawn
-    {S(  46,  18), S(106,  84)                                                      }, // Knight      OUR PIECE
-    {S(  75,  35), S( 59,  44), S( 60,  15)                                         }, // Bishop
-    {S(  26,  35), S(  6,  22), S( 38,  39), S(-12,  -2)                            }, // Rook
-    {S(  97,  93), S(100, 163), S(-58, -91), S(112, 192), S(276, 225)               }  // Queen
-  };
-
-  #undef S
-
-  // Endgame evaluation and scaling functions are accessed directly and not through
-  // the function maps because they correspond to more than one material hash key.
-  Endgame<KXK>    EvaluateKXK[] = { Endgame<KXK>(WHITE),    Endgame<KXK>(BLACK) };
-
-  Endgame<KBPsK>  ScaleKBPsK[]  = { Endgame<KBPsK>(WHITE),  Endgame<KBPsK>(BLACK) };
-  Endgame<KQKRPs> ScaleKQKRPs[] = { Endgame<KQKRPs>(WHITE), Endgame<KQKRPs>(BLACK) };
-  Endgame<KPsK>   ScaleKPsK[]   = { Endgame<KPsK>(WHITE),   Endgame<KPsK>(BLACK) };
-  Endgame<KPKP>   ScaleKPKP[]   = { Endgame<KPKP>(WHITE),   Endgame<KPKP>(BLACK) };
-
-  // Helper used to detect a given material distribution
-  bool is_KXK(const Position& pos, Color us) {
-    return  !more_than_one(pos.pieces(~us))
-          && pos.non_pawn_material(us) >= RookValueMg;
-  }
-
-  bool is_KBPsK(const Position& pos, Color us) {
-    return   pos.non_pawn_material(us) == BishopValueMg
-          && pos.count<PAWN>(us) >= 1;
-  }
-
-  bool is_KQKRPs(const Position& pos, Color us) {
-    return  !pos.count<PAWN>(us)
-          && pos.non_pawn_material(us) == QueenValueMg
-          && pos.count<ROOK>(~us) == 1
-          && pos.count<PAWN>(~us) >= 1;
-  }
-
-
-  /// imbalance() calculates the imbalance by comparing the piece count of each
-  /// piece type for both colors.
-
-  template<Color Us>
-  Score imbalance(const int pieceCount[][PIECE_TYPE_NB]) {
-
-    constexpr Color Them = ~Us;
-
-    Score bonus = SCORE_ZERO;
-
-    // Second-degree polynomial material imbalance, by Tord Romstad
-    for (int pt1 = NO_PIECE_TYPE; pt1 <= QUEEN; ++pt1)
-    {
-        if (!pieceCount[Us][pt1])
-            continue;
-
-        int v = QuadraticOurs[pt1][pt1] * pieceCount[Us][pt1];
-
-        for (int pt2 = NO_PIECE_TYPE; pt2 < pt1; ++pt2)
-            v +=  QuadraticOurs[pt1][pt2] * pieceCount[Us][pt2]
-                + QuadraticTheirs[pt1][pt2] * pieceCount[Them][pt2];
-
-        bonus += pieceCount[Us][pt1] * v;
-    }
-
-    return bonus;
-  }
-
-} // namespace
-
-namespace Material {
-
-
-/// Material::probe() looks up the current position's material configuration in
-/// the material hash table. It returns a pointer to the Entry if the position
-/// is found. Otherwise a new Entry is computed and stored there, so we don't
-/// have to recompute all when the same material configuration occurs again.
-
-Entry* probe(const Position& pos) {
-
-  Key key = pos.material_key();
-  Entry* e = pos.this_thread()->materialTable[key];
-
-  if (e->key == key)
-      return e;
-
-  std::memset(e, 0, sizeof(Entry));
-  e->key = key;
-  e->factor[WHITE] = e->factor[BLACK] = (uint8_t)SCALE_FACTOR_NORMAL;
-
-  Value npm_w = pos.non_pawn_material(WHITE);
-  Value npm_b = pos.non_pawn_material(BLACK);
-  Value npm   = std::clamp(npm_w + npm_b, EndgameLimit, MidgameLimit);
-
-  // Map total non-pawn material into [PHASE_ENDGAME, PHASE_MIDGAME]
-  e->gamePhase = Phase(((npm - EndgameLimit) * PHASE_MIDGAME) / (MidgameLimit - EndgameLimit));
-
-  // Let's look if we have a specialized evaluation function for this particular
-  // material configuration. Firstly we look for a fixed configuration one, then
-  // for a generic one if the previous search failed.
-  if ((e->evaluationFunction = Endgames::probe<Value>(key)) != nullptr)
-      return e;
-
-  for (Color c : { WHITE, BLACK })
-      if (is_KXK(pos, c))
-      {
-          e->evaluationFunction = &EvaluateKXK[c];
-          return e;
-      }
-
-  // OK, we didn't find any special evaluation function for the current material
-  // configuration. Is there a suitable specialized scaling function?
-  const auto* sf = Endgames::probe<ScaleFactor>(key);
-
-  if (sf)
-  {
-      e->scalingFunction[sf->strongSide] = sf; // Only strong color assigned
-      return e;
-  }
-
-  // We didn't find any specialized scaling function, so fall back on generic
-  // ones that refer to more than one material distribution. Note that in this
-  // case we don't return after setting the function.
-  for (Color c : { WHITE, BLACK })
-  {
-    if (is_KBPsK(pos, c))
-        e->scalingFunction[c] = &ScaleKBPsK[c];
-
-    else if (is_KQKRPs(pos, c))
-        e->scalingFunction[c] = &ScaleKQKRPs[c];
-  }
-
-  if (npm_w + npm_b == VALUE_ZERO && pos.pieces(PAWN)) // Only pawns on the board
-  {
-      if (!pos.count<PAWN>(BLACK))
-      {
-          assert(pos.count<PAWN>(WHITE) >= 2);
-
-          e->scalingFunction[WHITE] = &ScaleKPsK[WHITE];
-      }
-      else if (!pos.count<PAWN>(WHITE))
-      {
-          assert(pos.count<PAWN>(BLACK) >= 2);
-
-          e->scalingFunction[BLACK] = &ScaleKPsK[BLACK];
-      }
-      else if (pos.count<PAWN>(WHITE) == 1 && pos.count<PAWN>(BLACK) == 1)
-      {
-          // This is a special case because we set scaling functions
-          // for both colors instead of only one.
-          e->scalingFunction[WHITE] = &ScaleKPKP[WHITE];
-          e->scalingFunction[BLACK] = &ScaleKPKP[BLACK];
-      }
-  }
-
-  // Zero or just one pawn makes it difficult to win, even with a small material
-  // advantage. This catches some trivial draws like KK, KBK and KNK and gives a
-  // drawish scale factor for cases such as KRKBP and KmmKm (except for KBBKN).
-  if (!pos.count<PAWN>(WHITE) && npm_w - npm_b <= BishopValueMg)
-      e->factor[WHITE] = uint8_t(npm_w <  RookValueMg   ? SCALE_FACTOR_DRAW :
-                                 npm_b <= BishopValueMg ? 4 : 14);
-
-  if (!pos.count<PAWN>(BLACK) && npm_b - npm_w <= BishopValueMg)
-      e->factor[BLACK] = uint8_t(npm_b <  RookValueMg   ? SCALE_FACTOR_DRAW :
-                                 npm_w <= BishopValueMg ? 4 : 14);
-
-  // Evaluate the material imbalance. We use PIECE_TYPE_NONE as a place holder
-  // for the bishop pair "extended piece", which allows us to be more flexible
-  // in defining bishop pair bonuses.
-  const int pieceCount[COLOR_NB][PIECE_TYPE_NB] = {
-  { pos.count<BISHOP>(WHITE) > 1, pos.count<PAWN>(WHITE), pos.count<KNIGHT>(WHITE),
-    pos.count<BISHOP>(WHITE)    , pos.count<ROOK>(WHITE), pos.count<QUEEN >(WHITE) },
-  { pos.count<BISHOP>(BLACK) > 1, pos.count<PAWN>(BLACK), pos.count<KNIGHT>(BLACK),
-    pos.count<BISHOP>(BLACK)    , pos.count<ROOK>(BLACK), pos.count<QUEEN >(BLACK) } };
-
-  e->score = (imbalance<WHITE>(pieceCount) - imbalance<BLACK>(pieceCount)) / 16;
-  return e;
-}
-
-} // namespace Material
-
-} // namespace Stockfish

src/material.h

@@ -1,71 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifndef MATERIAL_H_INCLUDED
-#define MATERIAL_H_INCLUDED
-
-#include "endgame.h"
-#include "misc.h"
-#include "position.h"
-#include "types.h"
-
-namespace Stockfish::Material {
-
-/// Material::Entry contains various information about a material configuration.
-/// It contains a material imbalance evaluation, a function pointer to a special
-/// endgame evaluation function (which in most cases is NULL, meaning that the
-/// standard evaluation function will be used), and scale factors.
-///
-/// The scale factors are used to scale the evaluation score up or down. For
-/// instance, in KRB vs KR endgames, the score is scaled down by a factor of 4,
-/// which will result in scores of absolute value less than one pawn.
-
-struct Entry {
-
-  Score imbalance() const { return score; }
-  Phase game_phase() const { return (Phase)gamePhase; }
-  bool specialized_eval_exists() const { return evaluationFunction != nullptr; }
-  Value evaluate(const Position& pos) const { return (*evaluationFunction)(pos); }
-
-  // scale_factor() takes a position and a color as input and returns a scale factor
-  // for the given color. We have to provide the position in addition to the color
-  // because the scale factor may also be a function which should be applied to
-  // the position. For instance, in KBP vs K endgames, the scaling function looks
-  // for rook pawns and wrong-colored bishops.
-  ScaleFactor scale_factor(const Position& pos, Color c) const {
-    ScaleFactor sf = scalingFunction[c] ? (*scalingFunction[c])(pos)
-                                        :  SCALE_FACTOR_NONE;
-    return sf != SCALE_FACTOR_NONE ? sf : ScaleFactor(factor[c]);
-  }
-
-  Key key;
-  const EndgameBase<Value>* evaluationFunction;
-  const EndgameBase<ScaleFactor>* scalingFunction[COLOR_NB]; // Could be one for each
-                                                             // side (e.g. KPKP, KBPsK)
-  Score score;
-  int16_t gamePhase;
-  uint8_t factor[COLOR_NB];
-};
-
-typedef HashTable<Entry, 8192> Table;
-
-Entry* probe(const Position& pos);
-
-} // namespace Stockfish::Material
-
-#endif // #ifndef MATERIAL_H_INCLUDED

src/misc.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -22,687 +22,187 @@
 #include <algorithm>
 #include <cassert>
 #include <chrono>
-#include <functional>
+#include <cstddef>
-#include <mutex>
+#include <cstdint>
-#include <ostream>
+#include <iosfwd>
 #include <string>
 #include <vector>
-#include <iostream>
 
-#include <cstdint>
+#define stringify2(x) #x
-#include <cmath>
+#define stringify(x) stringify2(x)
-#include <cctype>
-#include <sstream>
-#include <deque>
-
-#include "types.h"
 
 namespace Stockfish {
 
 std::string engine_info(bool to_uci = false);
 std::string compiler_info();
-void prefetch(void* addr);
-void start_logger(const std::string& fname);
-void* std_aligned_alloc(size_t alignment, size_t size);
-void std_aligned_free(void* ptr);
-void* aligned_large_pages_alloc(size_t size); // memory aligned by page size, min alignment: 4096 bytes
-void aligned_large_pages_free(void* mem); // nop if mem == nullptr
 
-void dbg_hit_on(bool b);
+// Preloads the given address in L1/L2 cache. This is a non-blocking
-void dbg_hit_on(bool c, bool b);
+// function that doesn't stall the CPU waiting for data to be loaded from memory,
-void dbg_mean_of(int v);
+// which can be quite slow.
+void prefetch(void* addr);
+
+void  start_logger(const std::string& fname);
+void* std_aligned_alloc(size_t alignment, size_t size);
+void  std_aligned_free(void* ptr);
+// memory aligned by page size, min alignment: 4096 bytes
+void* aligned_large_pages_alloc(size_t size);
+// nop if mem == nullptr
+void aligned_large_pages_free(void* mem);
+
+void dbg_hit_on(bool cond, int slot = 0);
+void dbg_mean_of(int64_t value, int slot = 0);
+void dbg_stdev_of(int64_t value, int slot = 0);
+void dbg_correl_of(int64_t value1, int64_t value2, int slot = 0);
 void dbg_print();
 
-/// Debug macro to write to std::err if NDEBUG flag is set, and do nothing otherwise
+using TimePoint = std::chrono::milliseconds::rep;  // A value in milliseconds
-#if defined(NDEBUG)
-#define debug 1 && std::cerr
-#else
-#define debug 0 && std::cerr
-#endif
-
-inline void hit_any_key() {
-#ifndef NDEBUG
-  debug << "Hit any key to continue..." << std::endl << std::flush;
-  system("read");   // on Windows, should be system("pause");
-#endif
-}
-
-typedef std::chrono::milliseconds::rep TimePoint; // A value in milliseconds
 static_assert(sizeof(TimePoint) == sizeof(int64_t), "TimePoint should be 64 bits");
 inline TimePoint now() {
-  return std::chrono::duration_cast<std::chrono::milliseconds>
+    return std::chrono::duration_cast<std::chrono::milliseconds>(
-        (std::chrono::steady_clock::now().time_since_epoch()).count();
+             std::chrono::steady_clock::now().time_since_epoch())
+      .count();
 }
 
-template<class Entry, int Size>
-struct HashTable {
-  Entry* operator[](Key key) { return &table[(uint32_t)key & (Size - 1)]; }
 
-private:
+enum SyncCout {
-  std::vector<Entry> table = std::vector<Entry>(Size); // Allocate on the heap
+    IO_LOCK,
+    IO_UNLOCK
 };
-
-
-enum SyncCout { IO_LOCK, IO_UNLOCK };
 std::ostream& operator<<(std::ostream&, SyncCout);
 
 #define sync_cout std::cout << IO_LOCK
 #define sync_endl std::endl << IO_UNLOCK
 
 
-// align_ptr_up() : get the first aligned element of an array.
+// Get the first aligned element of an array.
 // ptr must point to an array of size at least `sizeof(T) * N + alignment` bytes,
 // where N is the number of elements in the array.
-template <uintptr_t Alignment, typename T>
+template<uintptr_t Alignment, typename T>
-T* align_ptr_up(T* ptr)
+T* align_ptr_up(T* ptr) {
-{
+    static_assert(alignof(T) < Alignment);
-  static_assert(alignof(T) < Alignment);
 
-  const uintptr_t ptrint = reinterpret_cast<uintptr_t>(reinterpret_cast<char*>(ptr));
+    const uintptr_t ptrint = reinterpret_cast<uintptr_t>(reinterpret_cast<char*>(ptr));
-  return reinterpret_cast<T*>(reinterpret_cast<char*>((ptrint + (Alignment - 1)) / Alignment * Alignment));
+    return reinterpret_cast<T*>(
+      reinterpret_cast<char*>((ptrint + (Alignment - 1)) / Alignment * Alignment));
 }
 
 
-// IsLittleEndian : true if and only if the binary is compiled on a little endian machine
+// True if and only if the binary is compiled on a little-endian machine
-static inline const union { uint32_t i; char c[4]; } Le = { 0x01020304 };
+static inline const union {
+    uint32_t i;
+    char     c[4];
+} Le                                    = {0x01020304};
 static inline const bool IsLittleEndian = (Le.c[0] == 4);
 
 
-// RunningAverage : a class to calculate a running average of a series of values.
+template<typename T, std::size_t MaxSize>
-// For efficiency, all computations are done with integers.
-class RunningAverage {
-  public:
-
-      // Reset the running average to rational value p / q
-      void set(int64_t p, int64_t q)
-        { average = p * PERIOD * RESOLUTION / q; }
-
-      // Update average with value v
-      void update(int64_t v)
-        { average = RESOLUTION * v + (PERIOD - 1) * average / PERIOD; }
-
-      // Test if average is strictly greater than rational a / b
-      bool is_greater(int64_t a, int64_t b) const
-        { return b * average > a * (PERIOD * RESOLUTION); }
-
-      int64_t value() const
-        { return average / (PERIOD * RESOLUTION); }
-
-  private :
-      static constexpr int64_t PERIOD     = 4096;
-      static constexpr int64_t RESOLUTION = 1024;
-      int64_t average;
-};
-
-template <typename T, std::size_t MaxSize>
 class ValueList {
 
-public:
+   public:
-  std::size_t size() const { return size_; }
+    std::size_t size() const { return size_; }
-  void resize(std::size_t newSize) { size_ = newSize; }
+    void        push_back(const T& value) { values_[size_++] = value; }
-  void push_back(const T& value) { values_[size_++] = value; }
+    const T*    begin() const { return values_; }
-  T& operator[](std::size_t index) { return values_[index]; }
+    const T*    end() const { return values_ + size_; }
-  T* begin() { return values_; }
+    const T&    operator[](int index) const { return values_[index]; }
-  T* end() { return values_ + size_; }
-  const T& operator[](std::size_t index) const { return values_[index]; }
-  const T* begin() const { return values_; }
-  const T* end() const { return values_ + size_; }
 
-  void swap(ValueList& other) {
+   private:
-    const std::size_t maxSize = std::max(size_, other.size_);
+    T           values_[MaxSize];
-    for (std::size_t i = 0; i < maxSize; ++i) {
+    std::size_t size_ = 0;
-      std::swap(values_[i], other.values_[i]);
-    }
-    std::swap(size_, other.size_);
-  }
-
-private:
-  T values_[MaxSize];
-  std::size_t size_ = 0;
 };
 
-// This logger allows printing many parts in a region atomically
-// but doesn't block the threads trying to append to other regions.
-// Instead if some region tries to pring while other region holds
-// the lock the messages are queued to be printed as soon as the
-// current region releases the lock.
-struct SynchronizedRegionLogger
-{
-  using RegionId = std::uint64_t;
 
-  struct Region
+// xorshift64star Pseudo-Random Number Generator
-  {
+// This class is based on original code written and dedicated
-    friend struct SynchronizedRegionLogger;
+// to the public domain by Sebastiano Vigna (2014).
-
+// It has the following characteristics:
-    Region() :
+//
-      logger(nullptr), region_id(0), is_held(false)
+//  -  Outputs 64-bit numbers
-    {
+//  -  Passes Dieharder and SmallCrush test batteries
-    }
+//  -  Does not require warm-up, no zeroland to escape
-
+//  -  Internal state is a single 64-bit integer
-    Region(const Region&) = delete;
+//  -  Period is 2^64 - 1
-    Region& operator=(const Region&) = delete;
+//  -  Speed: 1.60 ns/call (Core i7 @3.40GHz)
-
+//
-    Region(Region&& other) :
+// For further analysis see
-      logger(other.logger), region_id(other.region_id), is_held(other.is_held)
+//   <http://vigna.di.unimi.it/ftp/papers/xorshift.pdf>
-    {
-      other.logger = nullptr;
-      other.is_held = false;
-    }
-
-    Region& operator=(Region&& other) {
-      if (is_held && logger != nullptr)
-      {
-        logger->release_region(region_id);
-      }
-
-      logger = other.logger;
-      region_id = other.region_id;
-      is_held = other.is_held;
-
-      other.is_held = false;
-
-      return *this;
-    }
-
-    ~Region() { unlock(); }
-
-    void unlock() {
-      if (is_held) {
-        is_held = false;
-
-        if (logger != nullptr)
-          logger->release_region(region_id);
-      }
-    }
-
-    Region& operator << (std::ostream&(*pManip)(std::ostream&)) {
-      if (logger != nullptr)
-        logger->write(region_id, pManip);
-
-      return *this;
-    }
-
-    template <typename T>
-    Region& operator << (const T& value) {
-      if (logger != nullptr)
-        logger->write(region_id, value);
-
-      return *this;
-    }
-
-  private:
-    SynchronizedRegionLogger* logger;
-    RegionId region_id;
-    bool is_held;
-
-    Region(SynchronizedRegionLogger& log, RegionId id) :
-      logger(&log), region_id(id), is_held(true)
-    {
-    }
-  };
-
-private:
-  struct RegionBookkeeping
-  {
-    RegionBookkeeping(RegionId rid) : id(rid), is_held(true) {}
-
-    std::vector<std::string> pending_parts;
-    RegionId id;
-    bool is_held;
-  };
-
-  RegionId init_next_region()
-  {
-    static RegionId next_id = 0;
-
-    std::lock_guard lock(mutex);
-
-    const auto id = next_id++;
-    regions.emplace_back(id);
-
-    return id;
-  }
-
-  void write(RegionId id, std::ostream&(*pManip)(std::ostream&)) {
-    std::lock_guard lock(mutex);
-
-    if (regions.empty())
-      return;
-
-    if (id == regions.front().id) {
-      // We can just directly print to the output because
-      // we are at the front of the region queue.
-      out << *pManip;
-    } else {
-      // We have to schedule the print until previous regions are
-      // processed
-      auto* region = find_region_nolock(id);
-      if (region == nullptr)
-        return;
-
-      std::stringstream ss;
-      ss << *pManip;
-      region->pending_parts.emplace_back(std::move(ss).str());
-    }
-  }
-
-  template <typename T>
-  void write(RegionId id, const T& value) {
-    std::lock_guard lock(mutex);
-
-    if (regions.empty())
-      return;
-
-    if (id == regions.front().id) {
-      // We can just directly print to the output because
-      // we are at the front of the region queue.
-      out << value;
-    } else {
-      // We have to schedule the print until previous regions are
-      // processed
-      auto* region = find_region_nolock(id);
-      if (region == nullptr)
-        return;
-
-      std::stringstream ss;
-      ss << value;
-      region->pending_parts.emplace_back(std::move(ss).str());
-    }
-  }
-
-  std::ostream& out;
-
-  std::deque<RegionBookkeeping> regions;
-
-  std::mutex mutex;
-
-  RegionBookkeeping* find_region_nolock(RegionId id) {
-    // Linear search because the amount of concurrent regions should be small.
-    auto it = std::find_if(
-      regions.begin(),
-      regions.end(),
-      [id](const RegionBookkeeping& r) { return r.id == id; });
-
-    if (it == regions.end())
-      return nullptr;
-    else
-      return &*it;
-  }
-
-  void release_region(RegionId id) {
-    std::lock_guard lock(mutex);
-
-    auto* region = find_region_nolock(id);
-    if (region == nullptr)
-      return;
-
-    region->is_held = false;
-
-    process_backlog_nolock();
-  }
-
-  void process_backlog_nolock()
-  {
-    while(!regions.empty()) {
-      auto& region = regions.front();
-
-      for(auto& part : region.pending_parts) {
-        out << part;
-      }
-
-      // If the region is still held then we don't
-      // want to start printing stuff from the next region.
-      if (region.is_held)
-        break;
-
-      regions.pop_front();
-    }
-  }
-
-public:
-
-  SynchronizedRegionLogger(std::ostream& s) :
-    out(s)
-  {
-  }
-
-  [[nodiscard]] Region new_region() {
-    const auto id = init_next_region();
-    return Region(*this, id);
-  }
-
-};
-
-extern SynchronizedRegionLogger sync_region_cout;
-
-/// sigmoid(t, x0, y0, C, P, Q) implements a sigmoid-like function using only integers,
-/// with the following properties:
-///
-///  -  sigmoid is centered in (x0, y0)
-///  -  sigmoid has amplitude [-P/Q , P/Q] instead of [-1 , +1]
-///  -  limit is (y0 - P/Q) when t tends to -infinity
-///  -  limit is (y0 + P/Q) when t tends to +infinity
-///  -  the slope can be adjusted using C > 0, smaller C giving a steeper sigmoid
-///  -  the slope of the sigmoid when t = x0 is P/(Q*C)
-///  -  sigmoid is increasing with t when P > 0 and Q > 0
-///  -  to get a decreasing sigmoid, change sign of P
-///  -  mean value of the sigmoid is y0
-///
-/// Use <https://www.desmos.com/calculator/jhh83sqq92> to draw the sigmoid
-
-inline int64_t sigmoid(int64_t t, int64_t x0,
-                                  int64_t y0,
-                                  int64_t  C,
-                                  int64_t  P,
-                                  int64_t  Q)
-{
-   assert(C > 0);
-   assert(Q != 0);
-   return y0 + P * (t-x0) / (Q * (std::abs(t-x0) + C)) ;
-}
-
-
-/// xorshift64star Pseudo-Random Number Generator
-/// This class is based on original code written and dedicated
-/// to the public domain by Sebastiano Vigna (2014).
-/// It has the following characteristics:
-///
-///  -  Outputs 64-bit numbers
-///  -  Passes Dieharder and SmallCrush test batteries
-///  -  Does not require warm-up, no zeroland to escape
-///  -  Internal state is a single 64-bit integer
-///  -  Period is 2^64 - 1
-///  -  Speed: 1.60 ns/call (Core i7 @3.40GHz)
-///
-/// For further analysis see
-///   <http://vigna.di.unimi.it/ftp/papers/xorshift.pdf>
-
-static uint64_t string_hash(const std::string& str)
-{
-  uint64_t h = 525201411107845655ull;
-
-  for (auto c : str) {
-    h ^= static_cast<uint64_t>(c);
-    h *= 0x5bd1e9955bd1e995ull;
-    h ^= h >> 47;
-  }
-
-  return h;
-}
 
 class PRNG {
 
-  uint64_t s;
+    uint64_t s;
 
-  uint64_t rand64() {
+    uint64_t rand64() {
 
-    s ^= s >> 12, s ^= s << 25, s ^= s >> 27;
+        s ^= s >> 12, s ^= s << 25, s ^= s >> 27;
-    return s * 2685821657736338717LL;
+        return s * 2685821657736338717LL;
-  }
-
-public:
-  PRNG() { set_seed_from_time(); }
-  PRNG(uint64_t seed) : s(seed) { assert(seed); }
-  PRNG(const std::string& seed) { set_seed(seed); }
-
-  template<typename T> T rand() { return T(rand64()); }
-
-  /// Special generator used to fast init magic numbers.
-  /// Output values only have 1/8th of their bits set on average.
-  template<typename T> T sparse_rand()
-  { return T(rand64() & rand64() & rand64()); }
-  // Returns a random number from 0 to n-1. (Not uniform distribution, but this is enough in reality)
-  uint64_t rand(uint64_t n) { return rand<uint64_t>() % n; }
-
-  // Return the random seed used internally.
-  uint64_t get_seed() const { return s; }
-
-  void set_seed(uint64_t seed) { s = seed; }
-
-  uint64_t next_random_seed()
-  {
-    uint64_t seed = 0;
-    for(int i = 0; i < 64; ++i)
-    {
-      const auto off = rand64() % 64;
-      seed |= (rand64() & (uint64_t(1) << off)) >> off;
-      seed <<= 1;
     }
-    return seed;
-  }
 
-  void set_seed_from_time()
+   public:
-  {
+    PRNG(uint64_t seed) :
-      set_seed(std::chrono::system_clock::now().time_since_epoch().count());
+        s(seed) {
-  }
+        assert(seed);
+    }
 
-  void set_seed(const std::string& str)
+    template<typename T>
-  {
+    T rand() {
-    if (str.empty())
+        return T(rand64());
-    {
-      set_seed_from_time();
     }
-    else if (std::all_of(str.begin(), str.end(), [](char c) { return std::isdigit(c);} )) {
+
-      set_seed(std::stoull(str));
+    // Special generator used to fast init magic numbers.
+    // Output values only have 1/8th of their bits set on average.
+    template<typename T>
+    T sparse_rand() {
+        return T(rand64() & rand64() & rand64());
     }
-    else
-    {
-      set_seed(string_hash(str));
-    }
-  }
 };
 
-// Display a random seed. (For debugging)
-inline std::ostream& operator<<(std::ostream& os, PRNG& prng)
-{
-  os << "PRNG::seed = " << std::hex << prng.get_seed() << std::dec;
-  return os;
-}
-
 inline uint64_t mul_hi64(uint64_t a, uint64_t b) {
 #if defined(__GNUC__) && defined(IS_64BIT)
-    __extension__ typedef unsigned __int128 uint128;
+    __extension__ using uint128 = unsigned __int128;
-    return ((uint128)a * (uint128)b) >> 64;
+    return (uint128(a) * uint128(b)) >> 64;
 #else
-    uint64_t aL = (uint32_t)a, aH = a >> 32;
+    uint64_t aL = uint32_t(a), aH = a >> 32;
-    uint64_t bL = (uint32_t)b, bH = b >> 32;
+    uint64_t bL = uint32_t(b), bH = b >> 32;
     uint64_t c1 = (aL * bL) >> 32;
     uint64_t c2 = aH * bL + c1;
-    uint64_t c3 = aL * bH + (uint32_t)c2;
+    uint64_t c3 = aL * bH + uint32_t(c2);
     return aH * bH + (c2 >> 32) + (c3 >> 32);
 #endif
 }
 
-// This bitset can be accessed concurrently, provided
+// Under Windows it is not possible for a process to run on more than one
-// the concurrent accesses are performed on distinct
+// logical processor group. This usually means being limited to using max 64
-// instances of underlying type. That means the cuncurrent
+// cores. To overcome this, some special platform-specific API should be
-// accesses need to be spaced by at least
+// called to set group affinity for each thread. Original code from Texel by
-// bits_per_bucket bits.
+// Peter Österlund.
-// But at least best_concurrent_access_stride bits
-// is recommended to prevent false sharing.
-template <uint64_t N>
-struct LargeBitset
-{
-private:
-    constexpr static uint64_t cache_line_size = 64;
-
-public:
-    using UnderlyingType = uint64_t;
-
-    constexpr static uint64_t num_bits = N;
-    constexpr static uint64_t bits_per_bucket = 8 * sizeof(uint64_t);
-    constexpr static uint64_t num_buckets = (num_bits + bits_per_bucket - 1) / bits_per_bucket;
-    constexpr static uint64_t best_concurrent_access_stride = 8 * cache_line_size;
-
-    LargeBitset()
-    {
-        std::fill(std::begin(bits), std::end(bits), 0);
-    }
-
-    void set(uint64_t idx)
-    {
-        const uint64_t bucket = idx / bits_per_bucket;
-        const uint64_t bit = uint64_t(1) << (idx % bits_per_bucket);
-        bits[bucket] |= bit;
-    }
-
-    bool test(uint64_t idx) const
-    {
-        const uint64_t bucket = idx / bits_per_bucket;
-        const uint64_t bit = uint64_t(1) << (idx % bits_per_bucket);
-        return bits[bucket] & bit;
-    }
-
-    uint64_t count() const
-    {
-        uint64_t c = 0;
-        uint64_t i = 0;
-
-        for (; i < num_buckets - 3; i += 4)
-        {
-            uint64_t c0 = popcount(bits[i+0]);
-            uint64_t c1 = popcount(bits[i+1]);
-            uint64_t c2 = popcount(bits[i+2]);
-            uint64_t c3 = popcount(bits[i+3]);
-            c0 += c1;
-            c2 += c3;
-            c += c0 + c2;
-        }
-
-        for (; i < num_buckets; ++i)
-        {
-            c += popcount(bits[i]);
-        }
-
-        return c;
-    }
-
-private:
-    alignas(cache_line_size) UnderlyingType bits[num_buckets];
-};
-
-/// Under Windows it is not possible for a process to run on more than one
-/// logical processor group. This usually means to be limited to use max 64
-/// cores. To overcome this, some special platform specific API should be
-/// called to set group affinity for each thread. Original code from Texel by
-/// Peter Österlund.
-
 namespace WinProcGroup {
-  void bindThisThread(size_t idx);
+void bindThisThread(size_t idx);
 }
 
-// Returns a string that represents the current time. (Used for log output when learning evaluation function)
-std::string now_string();
-void sleep(int ms);
 
-namespace Algo {
+struct CommandLine {
-    // Fisher-Yates
+   public:
-    template <typename Rng, typename T>
+    CommandLine(int, char**);
-    void shuffle(std::vector<T>& buf, Rng&& prng)
+
+    int    argc;
+    char** argv;
+
+    std::string binaryDirectory;   // path of the executable directory
+    std::string workingDirectory;  // path of the working directory
+};
+
+namespace Utility {
+
+template<typename T, typename Predicate>
+void move_to_front(std::vector<T>& vec, Predicate pred) {
+    auto it = std::find_if(vec.begin(), vec.end(), pred);
+
+    if (it != vec.end())
     {
-        const auto size = buf.size();
+        std::rotate(vec.begin(), it, it + 1);
-        for (uint64_t i = 0; i < size; ++i)
-            std::swap(buf[i], buf[prng.rand(size - i) + i]);
-    }
-
-    // split the string
-    inline std::vector<std::string> split(const std::string& input, char delimiter) {
-        std::istringstream stream(input);
-        std::string field;
-        std::vector<std::string> fields;
-
-        while (std::getline(stream, field, delimiter)) {
-            fields.push_back(field);
-        }
-
-        return fields;
     }
 }
-
-// --------------------
-//       Path
-// --------------------
-
-// Something like Path class in C#. File name manipulation.
-// Match with the C# method name.
-struct Path
-{
-	// Combine the path name and file name and return it.
-	// If the folder name is not an empty string, append it if there is no'/' or'\\' at the end.
-	static std::string combine(const std::string& folder, const std::string& filename)
-	{
-		if (folder.length() >= 1 && *folder.rbegin() != '/' && *folder.rbegin() != '\\')
-			return folder + "/" + filename;
-
-		return folder + filename;
-	}
-
-	// Get the file name part (excluding the folder name) from the full path expression.
-	static std::string get_file_name(const std::string& path)
-	{
-		// I don't know which "\" or "/" is used.
-		auto path_index1 = path.find_last_of("\\") + 1;
-		auto path_index2 = path.find_last_of("/") + 1;
-		auto path_index = std::max(path_index1, path_index2);
-
-		return path.substr(path_index);
-	}
-};
-
-// It is ignored when new even though alignas is specified & because it is ignored when the STL container allocates memory,
-// A custom allocator used for that.
-template <typename T>
-class AlignedAllocator {
-public:
-  using value_type = T;
-
-  AlignedAllocator() {}
-  AlignedAllocator(const AlignedAllocator&) {}
-  AlignedAllocator(AlignedAllocator&&) {}
-
-  template <typename U> AlignedAllocator(const AlignedAllocator<U>&) {}
-
-  T* allocate(std::size_t n) { return (T*)std_aligned_alloc(alignof(T), n * sizeof(T)); }
-  void deallocate(T* p, std::size_t ) { std_aligned_free(p); }
-};
-
-template <typename T>
-class CacheLineAlignedAllocator {
-public:
-    using value_type = T;
-
-    constexpr static uint64_t cache_line_size = 64;
-
-    CacheLineAlignedAllocator() {}
-    CacheLineAlignedAllocator(const CacheLineAlignedAllocator&) {}
-    CacheLineAlignedAllocator(CacheLineAlignedAllocator&&) {}
-
-    template <typename U> CacheLineAlignedAllocator(const CacheLineAlignedAllocator<U>&) {}
-
-    T* allocate(std::size_t n) { return (T*)std_aligned_alloc(cache_line_size, n * sizeof(T)); }
-    void deallocate(T* p, std::size_t) { std_aligned_free(p); }
-};
-
-// --------------------
-//  Dependency Wrapper
-// --------------------
-
-namespace Dependency
-{
-  // In the Linux environment, if you getline() the text file is'\r\n'
-  // Since'\r' remains at the end, write a wrapper to remove this'\r'.
-  // So when calling getline() on fstream,
-  // just write getline() instead of std::getline() and use this function.
-  extern bool getline(std::ifstream& fs, std::string& s);
 }
 
-namespace CommandLine {
+}  // namespace Stockfish
-  void init(int argc, char* argv[]);
 
-  extern std::string binaryDirectory;  // path of the executable directory
+#endif  // #ifndef MISC_H_INCLUDED
-  extern std::string workingDirectory; // path of the working directory
-}
-
-} // namespace Stockfish
-
-#endif // #ifndef MISC_H_INCLUDED

src/movegen.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,82 +16,86 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
-#include <cassert>
-
 #include "movegen.h"
+
+#include <cassert>
+#include <initializer_list>
+
+#include "bitboard.h"
 #include "position.h"
 
 namespace Stockfish {
 
 namespace {
 
-  template<GenType Type, Direction D>
+template<GenType Type, Direction D, bool Enemy>
-  ExtMove* make_promotions(ExtMove* moveList, Square to) {
+ExtMove* make_promotions(ExtMove* moveList, [[maybe_unused]] Square to) {
 
-    if (Type == CAPTURES || Type == EVASIONS || Type == NON_EVASIONS)
+    constexpr bool all = Type == EVASIONS || Type == NON_EVASIONS;
-        *moveList++ = make<PROMOTION>(to - D, to, QUEEN);
 
-    if (Type == QUIETS || Type == EVASIONS || Type == NON_EVASIONS)
+    if constexpr (Type == CAPTURES || all)
+        *moveList++ = Move::make<PROMOTION>(to - D, to, QUEEN);
+
+    if constexpr ((Type == CAPTURES && Enemy) || (Type == QUIETS && !Enemy) || all)
     {
-        *moveList++ = make<PROMOTION>(to - D, to, ROOK);
+        *moveList++ = Move::make<PROMOTION>(to - D, to, ROOK);
-        *moveList++ = make<PROMOTION>(to - D, to, BISHOP);
+        *moveList++ = Move::make<PROMOTION>(to - D, to, BISHOP);
-        *moveList++ = make<PROMOTION>(to - D, to, KNIGHT);
+        *moveList++ = Move::make<PROMOTION>(to - D, to, KNIGHT);
     }
 
     return moveList;
-  }
+}
 
 
-  template<Color Us, GenType Type>
+template<Color Us, GenType Type>
-  ExtMove* generate_pawn_moves(const Position& pos, ExtMove* moveList, Bitboard target) {
+ExtMove* generate_pawn_moves(const Position& pos, ExtMove* moveList, Bitboard target) {
 
     constexpr Color     Them     = ~Us;
-    constexpr Bitboard  TRank7BB = (Us == WHITE ? Rank7BB    : Rank2BB);
+    constexpr Bitboard  TRank7BB = (Us == WHITE ? Rank7BB : Rank2BB);
-    constexpr Bitboard  TRank3BB = (Us == WHITE ? Rank3BB    : Rank6BB);
+    constexpr Bitboard  TRank3BB = (Us == WHITE ? Rank3BB : Rank6BB);
     constexpr Direction Up       = pawn_push(Us);
     constexpr Direction UpRight  = (Us == WHITE ? NORTH_EAST : SOUTH_WEST);
     constexpr Direction UpLeft   = (Us == WHITE ? NORTH_WEST : SOUTH_EAST);
 
     const Bitboard emptySquares = ~pos.pieces();
-    const Bitboard enemies      =  Type == EVASIONS ? pos.checkers()
+    const Bitboard enemies      = Type == EVASIONS ? pos.checkers() : pos.pieces(Them);
-                                                    : pos.pieces(Them);
 
-    Bitboard pawnsOn7    = pos.pieces(Us, PAWN) &  TRank7BB;
+    Bitboard pawnsOn7    = pos.pieces(Us, PAWN) & TRank7BB;
     Bitboard pawnsNotOn7 = pos.pieces(Us, PAWN) & ~TRank7BB;
 
     // Single and double pawn pushes, no promotions
-    if (Type != CAPTURES)
+    if constexpr (Type != CAPTURES)
     {
-        Bitboard b1 = shift<Up>(pawnsNotOn7)   & emptySquares;
+        Bitboard b1 = shift<Up>(pawnsNotOn7) & emptySquares;
         Bitboard b2 = shift<Up>(b1 & TRank3BB) & emptySquares;
 
-        if (Type == EVASIONS) // Consider only blocking squares
+        if constexpr (Type == EVASIONS)  // Consider only blocking squares
         {
             b1 &= target;
             b2 &= target;
         }
 
-        if (Type == QUIET_CHECKS)
+        if constexpr (Type == QUIET_CHECKS)
         {
             // To make a quiet check, you either make a direct check by pushing a pawn
             // or push a blocker pawn that is not on the same file as the enemy king.
             // Discovered check promotion has been already generated amongst the captures.
-            Square ksq = pos.square<KING>(Them);
+            Square   ksq              = pos.square<KING>(Them);
             Bitboard dcCandidatePawns = pos.blockers_for_king(Them) & ~file_bb(ksq);
-            b1 &= pawn_attacks_bb(Them, ksq) | shift<   Up>(dcCandidatePawns);
+            b1 &= pawn_attacks_bb(Them, ksq) | shift<Up>(dcCandidatePawns);
-            b2 &= pawn_attacks_bb(Them, ksq) | shift<Up+Up>(dcCandidatePawns);
+            b2 &= pawn_attacks_bb(Them, ksq) | shift<Up + Up>(dcCandidatePawns);
         }
 
         while (b1)
         {
-            Square to = pop_lsb(b1);
+            Square to   = pop_lsb(b1);
-            *moveList++ = make_move(to - Up, to);
+            *moveList++ = Move(to - Up, to);
         }
 
         while (b2)
         {
-            Square to = pop_lsb(b2);
+            Square to   = pop_lsb(b2);
-            *moveList++ = make_move(to - Up - Up, to);
+            *moveList++ = Move(to - Up - Up, to);
         }
     }
 
@@ -99,38 +103,38 @@ namespace {
     if (pawnsOn7)
     {
         Bitboard b1 = shift<UpRight>(pawnsOn7) & enemies;
-        Bitboard b2 = shift<UpLeft >(pawnsOn7) & enemies;
+        Bitboard b2 = shift<UpLeft>(pawnsOn7) & enemies;
-        Bitboard b3 = shift<Up     >(pawnsOn7) & emptySquares;
+        Bitboard b3 = shift<Up>(pawnsOn7) & emptySquares;
 
-        if (Type == EVASIONS)
+        if constexpr (Type == EVASIONS)
             b3 &= target;
 
         while (b1)
-            moveList = make_promotions<Type, UpRight>(moveList, pop_lsb(b1));
+            moveList = make_promotions<Type, UpRight, true>(moveList, pop_lsb(b1));
 
         while (b2)
-            moveList = make_promotions<Type, UpLeft >(moveList, pop_lsb(b2));
+            moveList = make_promotions<Type, UpLeft, true>(moveList, pop_lsb(b2));
 
         while (b3)
-            moveList = make_promotions<Type, Up     >(moveList, pop_lsb(b3));
+            moveList = make_promotions<Type, Up, false>(moveList, pop_lsb(b3));
     }
 
     // Standard and en passant captures
-    if (Type == CAPTURES || Type == EVASIONS || Type == NON_EVASIONS)
+    if constexpr (Type == CAPTURES || Type == EVASIONS || Type == NON_EVASIONS)
     {
         Bitboard b1 = shift<UpRight>(pawnsNotOn7) & enemies;
-        Bitboard b2 = shift<UpLeft >(pawnsNotOn7) & enemies;
+        Bitboard b2 = shift<UpLeft>(pawnsNotOn7) & enemies;
 
         while (b1)
         {
-            Square to = pop_lsb(b1);
+            Square to   = pop_lsb(b1);
-            *moveList++ = make_move(to - UpRight, to);
+            *moveList++ = Move(to - UpRight, to);
         }
 
         while (b2)
         {
-            Square to = pop_lsb(b2);
+            Square to   = pop_lsb(b2);
-            *moveList++ = make_move(to - UpLeft, to);
+            *moveList++ = Move(to - UpLeft, to);
         }
 
         if (pos.ep_square() != SQ_NONE)
@@ -146,16 +150,16 @@ namespace {
             assert(b1);
 
             while (b1)
-                *moveList++ = make<EN_PASSANT>(pop_lsb(b1), pos.ep_square());
+                *moveList++ = Move::make<EN_PASSANT>(pop_lsb(b1), pos.ep_square());
         }
     }
 
     return moveList;
-  }
+}
 
 
-  template<Color Us, PieceType Pt, bool Checks>
+template<Color Us, PieceType Pt, bool Checks>
-  ExtMove* generate_moves(const Position& pos, ExtMove* moveList, Bitboard target) {
+ExtMove* generate_moves(const Position& pos, ExtMove* moveList, Bitboard target) {
 
     static_assert(Pt != KING && Pt != PAWN, "Unsupported piece type in generate_moves()");
 
@@ -163,43 +167,43 @@ namespace {
 
     while (bb)
     {
-        Square from = pop_lsb(bb);
+        Square   from = pop_lsb(bb);
-        Bitboard b = attacks_bb<Pt>(from, pos.pieces()) & target;
+        Bitboard b    = attacks_bb<Pt>(from, pos.pieces()) & target;
 
         // To check, you either move freely a blocker or make a direct check.
         if (Checks && (Pt == QUEEN || !(pos.blockers_for_king(~Us) & from)))
             b &= pos.check_squares(Pt);
 
         while (b)
-            *moveList++ = make_move(from, pop_lsb(b));
+            *moveList++ = Move(from, pop_lsb(b));
     }
 
     return moveList;
-  }
+}
 
 
-  template<Color Us, GenType Type>
+template<Color Us, GenType Type>
-  ExtMove* generate_all(const Position& pos, ExtMove* moveList) {
+ExtMove* generate_all(const Position& pos, ExtMove* moveList) {
 
     static_assert(Type != LEGAL, "Unsupported type in generate_all()");
 
-    constexpr bool Checks = Type == QUIET_CHECKS; // Reduce template instantiations
+    constexpr bool Checks = Type == QUIET_CHECKS;  // Reduce template instantiations
-    const Square ksq = pos.square<KING>(Us);
+    const Square   ksq    = pos.square<KING>(Us);
-    Bitboard target;
+    Bitboard       target;
 
     // Skip generating non-king moves when in double check
     if (Type != EVASIONS || !more_than_one(pos.checkers()))
     {
-        target = Type == EVASIONS     ?  between_bb(ksq, lsb(pos.checkers()))
+        target = Type == EVASIONS     ? between_bb(ksq, lsb(pos.checkers()))
-               : Type == NON_EVASIONS ? ~pos.pieces( Us)
+               : Type == NON_EVASIONS ? ~pos.pieces(Us)
-               : Type == CAPTURES     ?  pos.pieces(~Us)
+               : Type == CAPTURES     ? pos.pieces(~Us)
-                                      : ~pos.pieces(   ); // QUIETS || QUIET_CHECKS
+                                      : ~pos.pieces();  // QUIETS || QUIET_CHECKS
 
         moveList = generate_pawn_moves<Us, Type>(pos, moveList, target);
         moveList = generate_moves<Us, KNIGHT, Checks>(pos, moveList, target);
         moveList = generate_moves<Us, BISHOP, Checks>(pos, moveList, target);
-        moveList = generate_moves<Us,   ROOK, Checks>(pos, moveList, target);
+        moveList = generate_moves<Us, ROOK, Checks>(pos, moveList, target);
-        moveList = generate_moves<Us,  QUEEN, Checks>(pos, moveList, target);
+        moveList = generate_moves<Us, QUEEN, Checks>(pos, moveList, target);
     }
 
     if (!Checks || pos.blockers_for_king(~Us) & ksq)
@@ -209,38 +213,38 @@ namespace {
             b &= ~attacks_bb<QUEEN>(pos.square<KING>(~Us));
 
         while (b)
-            *moveList++ = make_move(ksq, pop_lsb(b));
+            *moveList++ = Move(ksq, pop_lsb(b));
 
         if ((Type == QUIETS || Type == NON_EVASIONS) && pos.can_castle(Us & ANY_CASTLING))
-            for (CastlingRights cr : { Us & KING_SIDE, Us & QUEEN_SIDE } )
+            for (CastlingRights cr : {Us & KING_SIDE, Us & QUEEN_SIDE})
                 if (!pos.castling_impeded(cr) && pos.can_castle(cr))
-                    *moveList++ = make<CASTLING>(ksq, pos.castling_rook_square(cr));
+                    *moveList++ = Move::make<CASTLING>(ksq, pos.castling_rook_square(cr));
     }
 
     return moveList;
-  }
+}
 
-} // namespace
+}  // namespace
 
 
-/// <CAPTURES>     Generates all pseudo-legal captures plus queen promotions
+// <CAPTURES>     Generates all pseudo-legal captures plus queen promotions
-/// <QUIETS>       Generates all pseudo-legal non-captures and underpromotions
+// <QUIETS>       Generates all pseudo-legal non-captures and underpromotions
-/// <EVASIONS>     Generates all pseudo-legal check evasions when the side to move is in check
+// <EVASIONS>     Generates all pseudo-legal check evasions
-/// <QUIET_CHECKS> Generates all pseudo-legal non-captures giving check, except castling and promotions
+// <NON_EVASIONS> Generates all pseudo-legal captures and non-captures
-/// <NON_EVASIONS> Generates all pseudo-legal captures and non-captures
+// <QUIET_CHECKS> Generates all pseudo-legal non-captures giving check,
-///
+//                except castling and promotions
-/// Returns a pointer to the end of the move list.
+//
-
+// Returns a pointer to the end of the move list.
 template<GenType Type>
 ExtMove* generate(const Position& pos, ExtMove* moveList) {
 
-  static_assert(Type != LEGAL, "Unsupported type in generate()");
+    static_assert(Type != LEGAL, "Unsupported type in generate()");
-  assert((Type == EVASIONS) == (bool)pos.checkers());
+    assert((Type == EVASIONS) == bool(pos.checkers()));
 
-  Color us = pos.side_to_move();
+    Color us = pos.side_to_move();
 
-  return us == WHITE ? generate_all<WHITE, Type>(pos, moveList)
+    return us == WHITE ? generate_all<WHITE, Type>(pos, moveList)
-                     : generate_all<BLACK, Type>(pos, moveList);
+                       : generate_all<BLACK, Type>(pos, moveList);
 }
 
 // Explicit template instantiations
@@ -251,26 +255,26 @@ template ExtMove* generate<QUIET_CHECKS>(const Position&, ExtMove*);
 template ExtMove* generate<NON_EVASIONS>(const Position&, ExtMove*);
 
 
-/// generate<LEGAL> generates all the legal moves in the given position
+// generate<LEGAL> generates all the legal moves in the given position
 
 template<>
 ExtMove* generate<LEGAL>(const Position& pos, ExtMove* moveList) {
 
-  Color us = pos.side_to_move();
+    Color    us     = pos.side_to_move();
-  Bitboard pinned = pos.blockers_for_king(us) & pos.pieces(us);
+    Bitboard pinned = pos.blockers_for_king(us) & pos.pieces(us);
-  Square ksq = pos.square<KING>(us);
+    Square   ksq    = pos.square<KING>(us);
-  ExtMove* cur = moveList;
+    ExtMove* cur    = moveList;
 
-  moveList = pos.checkers() ? generate<EVASIONS    >(pos, moveList)
+    moveList =
-                            : generate<NON_EVASIONS>(pos, moveList);
+      pos.checkers() ? generate<EVASIONS>(pos, moveList) : generate<NON_EVASIONS>(pos, moveList);
-  while (cur != moveList)
+    while (cur != moveList)
-      if (  ((pinned && pinned & from_sq(*cur)) || from_sq(*cur) == ksq || type_of(*cur) == EN_PASSANT)
+        if (((pinned & cur->from_sq()) || cur->from_sq() == ksq || cur->type_of() == EN_PASSANT)
-          && !pos.legal(*cur))
+            && !pos.legal(*cur))
-          *cur = (--moveList)->move;
+            *cur = *(--moveList);
-      else
+        else
-          ++cur;
+            ++cur;
 
-  return moveList;
+    return moveList;
 }
 
-} // namespace Stockfish
+}  // namespace Stockfish

src/movegen.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -19,7 +19,8 @@
 #ifndef MOVEGEN_H_INCLUDED
 #define MOVEGEN_H_INCLUDED
 
-#include <algorithm>
+#include <algorithm>  // IWYU pragma: keep
+#include <cstddef>
 
 #include "types.h"
 
@@ -28,53 +29,46 @@ namespace Stockfish {
 class Position;
 
 enum GenType {
-  CAPTURES,
+    CAPTURES,
-  QUIETS,
+    QUIETS,
-  QUIET_CHECKS,
+    QUIET_CHECKS,
-  EVASIONS,
+    EVASIONS,
-  NON_EVASIONS,
+    NON_EVASIONS,
-  LEGAL
+    LEGAL
 };
 
-struct ExtMove {
+struct ExtMove: public Move {
-  Move move;
+    int value;
-  int value;
 
-  operator Move() const { return move; }
+    void operator=(Move m) { data = m.raw(); }
-  void operator=(Move m) { move = m; }
 
-  // Inhibit unwanted implicit conversions to Move
+    // Inhibit unwanted implicit conversions to Move
-  // with an ambiguity that yields to a compile error.
+    // with an ambiguity that yields to a compile error.
-  operator float() const = delete;
+    operator float() const = delete;
 };
 
-inline bool operator<(const ExtMove& f, const ExtMove& s) {
+inline bool operator<(const ExtMove& f, const ExtMove& s) { return f.value < s.value; }
-  return f.value < s.value;
-}
 
 template<GenType>
 ExtMove* generate(const Position& pos, ExtMove* moveList);
 
-/// The MoveList struct is a simple wrapper around generate(). It sometimes comes
+// The MoveList struct wraps the generate() function and returns a convenient
-/// in handy to use this class instead of the low level generate() function.
+// list of moves. Using MoveList is sometimes preferable to directly calling
+// the lower level generate() function.
 template<GenType T>
 struct MoveList {
 
-  explicit MoveList(const Position& pos) : last(generate<T>(pos, moveList)) {}
+    explicit MoveList(const Position& pos) :
-  const ExtMove* begin() const { return moveList; }
+        last(generate<T>(pos, moveList)) {}
-  const ExtMove* end() const { return last; }
+    const ExtMove* begin() const { return moveList; }
-  size_t size() const { return last - moveList; }
+    const ExtMove* end() const { return last; }
-  bool contains(Move move) const {
+    size_t         size() const { return last - moveList; }
-    return std::find(begin(), end(), move) != end();
+    bool           contains(Move move) const { return std::find(begin(), end(), move) != end(); }
-  }
 
-  // returns the i th element
+   private:
-  const ExtMove at(size_t i) const { assert(0 <= i && i < size()); return begin()[i]; }
+    ExtMove moveList[MAX_MOVES], *last;
-
-private:
-  ExtMove moveList[MAX_MOVES], *last;
 };
 
-} // namespace Stockfish
+}  // namespace Stockfish
 
-#endif // #ifndef MOVEGEN_H_INCLUDED
+#endif  // #ifndef MOVEGEN_H_INCLUDED

src/movepick.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,291 +16,371 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "movepick.h"
+
+#include <algorithm>
 #include <cassert>
+#include <iterator>
+#include <utility>
 
 #include "bitboard.h"
-#include "movepick.h"
+#include "position.h"
 
 namespace Stockfish {
 
 namespace {
 
-  enum Stages {
+enum Stages {
-    MAIN_TT, CAPTURE_INIT, GOOD_CAPTURE, REFUTATION, QUIET_INIT, QUIET, BAD_CAPTURE,
+    // generate main search moves
-    EVASION_TT, EVASION_INIT, EVASION,
+    MAIN_TT,
-    PROBCUT_TT, PROBCUT_INIT, PROBCUT,
+    CAPTURE_INIT,
-    QSEARCH_TT, QCAPTURE_INIT, QCAPTURE, QCHECK_INIT, QCHECK
+    GOOD_CAPTURE,
-  };
+    REFUTATION,
+    QUIET_INIT,
+    GOOD_QUIET,
+    BAD_CAPTURE,
+    BAD_QUIET,
 
-  // partial_insertion_sort() sorts moves in descending order up to and including
+    // generate evasion moves
-  // a given limit. The order of moves smaller than the limit is left unspecified.
+    EVASION_TT,
-  void partial_insertion_sort(ExtMove* begin, ExtMove* end, int limit) {
+    EVASION_INIT,
+    EVASION,
+
+    // generate probcut moves
+    PROBCUT_TT,
+    PROBCUT_INIT,
+    PROBCUT,
+
+    // generate qsearch moves
+    QSEARCH_TT,
+    QCAPTURE_INIT,
+    QCAPTURE,
+    QCHECK_INIT,
+    QCHECK
+};
+
+// Sort moves in descending order up to and including
+// a given limit. The order of moves smaller than the limit is left unspecified.
+void partial_insertion_sort(ExtMove* begin, ExtMove* end, int limit) {
 
     for (ExtMove *sortedEnd = begin, *p = begin + 1; p < end; ++p)
         if (p->value >= limit)
         {
             ExtMove tmp = *p, *q;
-            *p = *++sortedEnd;
+            *p          = *++sortedEnd;
             for (q = sortedEnd; q != begin && *(q - 1) < tmp; --q)
                 *q = *(q - 1);
             *q = tmp;
         }
-  }
-
-} // namespace
-
-
-/// Constructors of the MovePicker class. As arguments we pass information
-/// to help it to return the (presumably) good moves first, to decide which
-/// moves to return (in the quiescence search, for instance, we only want to
-/// search captures, promotions, and some checks) and how important good move
-/// ordering is at the current node.
-
-/// MovePicker constructor for the main search
-MovePicker::MovePicker(const Position& p, Move ttm, Depth d, const ButterflyHistory* mh,
-                                                             const CapturePieceToHistory* cph,
-                                                             const PieceToHistory** ch,
-                                                             Move cm,
-                                                             const Move* killers)
-           : pos(p), mainHistory(mh), captureHistory(cph), continuationHistory(ch),
-             ttMove(ttm), refutations{{killers[0], 0}, {killers[1], 0}, {cm, 0}}, depth(d)
-{
-  assert(d > 0);
-
-  stage = (pos.checkers() ? EVASION_TT : MAIN_TT) +
-          !(ttm && pos.pseudo_legal(ttm));
 }
 
-/// MovePicker constructor for quiescence search
+}  // namespace
-MovePicker::MovePicker(const Position& p, Move ttm, Depth d, const ButterflyHistory* mh,
-                                                             const CapturePieceToHistory* cph,
-                                                             const PieceToHistory** ch,
-                                                             Square rs)
-           : pos(p), mainHistory(mh), captureHistory(cph), continuationHistory(ch), ttMove(ttm), recaptureSquare(rs), depth(d)
-{
-  assert(d <= 0);
 
-  stage = (pos.checkers() ? EVASION_TT : QSEARCH_TT) +
+
-          !(   ttm
+// Constructors of the MovePicker class. As arguments, we pass information
-            && (pos.checkers() || depth > DEPTH_QS_RECAPTURES || to_sq(ttm) == recaptureSquare)
+// to help it return the (presumably) good moves first, to decide which
-            && pos.pseudo_legal(ttm));
+// moves to return (in the quiescence search, for instance, we only want to
+// search captures, promotions, and some checks) and how important a good
+// move ordering is at the current node.
+
+// MovePicker constructor for the main search
+MovePicker::MovePicker(const Position&              p,
+                       Move                         ttm,
+                       Depth                        d,
+                       const ButterflyHistory*      mh,
+                       const CapturePieceToHistory* cph,
+                       const PieceToHistory**       ch,
+                       const PawnHistory*           ph,
+                       Move                         cm,
+                       const Move*                  killers) :
+    pos(p),
+    mainHistory(mh),
+    captureHistory(cph),
+    continuationHistory(ch),
+    pawnHistory(ph),
+    ttMove(ttm),
+    refutations{{killers[0], 0}, {killers[1], 0}, {cm, 0}},
+    depth(d) {
+    assert(d > 0);
+
+    stage = (pos.checkers() ? EVASION_TT : MAIN_TT) + !(ttm && pos.pseudo_legal(ttm));
 }
 
-/// MovePicker constructor for ProbCut: we generate captures with SEE greater
+// Constructor for quiescence search
-/// than or equal to the given threshold.
+MovePicker::MovePicker(const Position&              p,
-MovePicker::MovePicker(const Position& p, Move ttm, Value th, Depth d, const CapturePieceToHistory* cph)
+                       Move                         ttm,
-           : pos(p), captureHistory(cph), ttMove(ttm), threshold(th), depth(d)
+                       Depth                        d,
-{
+                       const ButterflyHistory*      mh,
-  assert(!pos.checkers());
+                       const CapturePieceToHistory* cph,
+                       const PieceToHistory**       ch,
+                       const PawnHistory*           ph) :
+    pos(p),
+    mainHistory(mh),
+    captureHistory(cph),
+    continuationHistory(ch),
+    pawnHistory(ph),
+    ttMove(ttm),
+    depth(d) {
+    assert(d <= 0);
 
-  stage = PROBCUT_TT + !(ttm && pos.capture(ttm)
+    stage = (pos.checkers() ? EVASION_TT : QSEARCH_TT) + !(ttm && pos.pseudo_legal(ttm));
-                             && pos.pseudo_legal(ttm)
-                             && pos.see_ge(ttm, threshold));
 }
 
-/// MovePicker::score() assigns a numerical value to each move in a list, used
+// Constructor for ProbCut: we generate captures with SEE greater
-/// for sorting. Captures are ordered by Most Valuable Victim (MVV), preferring
+// than or equal to the given threshold.
-/// captures with a good history. Quiets moves are ordered using the histories.
+MovePicker::MovePicker(const Position& p, Move ttm, int th, const CapturePieceToHistory* cph) :
+    pos(p),
+    captureHistory(cph),
+    ttMove(ttm),
+    threshold(th) {
+    assert(!pos.checkers());
+
+    stage = PROBCUT_TT
+          + !(ttm && pos.capture_stage(ttm) && pos.pseudo_legal(ttm) && pos.see_ge(ttm, threshold));
+}
+
+// Assigns a numerical value to each move in a list, used
+// for sorting. Captures are ordered by Most Valuable Victim (MVV), preferring
+// captures with a good history. Quiets moves are ordered using the history tables.
 template<GenType Type>
 void MovePicker::score() {
 
-  static_assert(Type == CAPTURES || Type == QUIETS || Type == EVASIONS, "Wrong type");
+    static_assert(Type == CAPTURES || Type == QUIETS || Type == EVASIONS, "Wrong type");
 
-  Bitboard threatened, threatenedByPawn, threatenedByMinor, threatenedByRook;
+    [[maybe_unused]] Bitboard threatenedByPawn, threatenedByMinor, threatenedByRook,
-  if constexpr (Type == QUIETS)
+      threatenedPieces;
-  {
+    if constexpr (Type == QUIETS)
-      Color us = pos.side_to_move();
+    {
-      // squares threatened by pawns
+        Color us = pos.side_to_move();
-      threatenedByPawn  = pos.attacks_by<PAWN>(~us);
-      // squares threatened by minors or pawns
-      threatenedByMinor = pos.attacks_by<KNIGHT>(~us) | pos.attacks_by<BISHOP>(~us) | threatenedByPawn;
-      // squares threatened by rooks, minors or pawns
-      threatenedByRook  = pos.attacks_by<ROOK>(~us) | threatenedByMinor;
 
-      // pieces threatened by pieces of lesser material value
+        threatenedByPawn = pos.attacks_by<PAWN>(~us);
-      threatened =  (pos.pieces(us, QUEEN) & threatenedByRook)
+        threatenedByMinor =
-                  | (pos.pieces(us, ROOK)  & threatenedByMinor)
+          pos.attacks_by<KNIGHT>(~us) | pos.attacks_by<BISHOP>(~us) | threatenedByPawn;
-                  | (pos.pieces(us, KNIGHT, BISHOP) & threatenedByPawn);
+        threatenedByRook = pos.attacks_by<ROOK>(~us) | threatenedByMinor;
-  }
-  else
-  {
-      // Silence unused variable warnings
-      (void) threatened;
-      (void) threatenedByPawn;
-      (void) threatenedByMinor;
-      (void) threatenedByRook;
-  }
 
-  for (auto& m : *this)
+        // Pieces threatened by pieces of lesser material value
-      if constexpr (Type == CAPTURES)
+        threatenedPieces = (pos.pieces(us, QUEEN) & threatenedByRook)
-          m.value =  6 * int(PieceValue[MG][pos.piece_on(to_sq(m))])
+                         | (pos.pieces(us, ROOK) & threatenedByMinor)
-                   +     (*captureHistory)[pos.moved_piece(m)][to_sq(m)][type_of(pos.piece_on(to_sq(m)))];
+                         | (pos.pieces(us, KNIGHT, BISHOP) & threatenedByPawn);
+    }
 
-      else if constexpr (Type == QUIETS)
+    for (auto& m : *this)
-          m.value =      (*mainHistory)[pos.side_to_move()][from_to(m)]
+        if constexpr (Type == CAPTURES)
-                   + 2 * (*continuationHistory[0])[pos.moved_piece(m)][to_sq(m)]
+            m.value =
-                   +     (*continuationHistory[1])[pos.moved_piece(m)][to_sq(m)]
+              7 * int(PieceValue[pos.piece_on(m.to_sq())])
-                   +     (*continuationHistory[3])[pos.moved_piece(m)][to_sq(m)]
+              + (*captureHistory)[pos.moved_piece(m)][m.to_sq()][type_of(pos.piece_on(m.to_sq()))];
-                   +     (*continuationHistory[5])[pos.moved_piece(m)][to_sq(m)]
-                   +     (threatened & from_sq(m) ?
-                           (type_of(pos.moved_piece(m)) == QUEEN && !(to_sq(m) & threatenedByRook)  ? 50000
-                          : type_of(pos.moved_piece(m)) == ROOK  && !(to_sq(m) & threatenedByMinor) ? 25000
-                          :                                         !(to_sq(m) & threatenedByPawn)  ? 15000
-                          :                                                                           0)
-                          :                                                                           0);
 
-      else // Type == EVASIONS
+        else if constexpr (Type == QUIETS)
-      {
+        {
-          if (pos.capture(m))
+            Piece     pc   = pos.moved_piece(m);
-              m.value =  PieceValue[MG][pos.piece_on(to_sq(m))]
+            PieceType pt   = type_of(pc);
-                       - Value(type_of(pos.moved_piece(m)));
+            Square    from = m.from_sq();
-          else
+            Square    to   = m.to_sq();
-              m.value =      (*mainHistory)[pos.side_to_move()][from_to(m)]
+
-                       + 2 * (*continuationHistory[0])[pos.moved_piece(m)][to_sq(m)]
+            // histories
-                       - (1 << 28);
+            m.value = 2 * (*mainHistory)[pos.side_to_move()][m.from_to()];
-      }
+            m.value += 2 * (*pawnHistory)[pawn_structure_index(pos)][pc][to];
+            m.value += 2 * (*continuationHistory[0])[pc][to];
+            m.value += (*continuationHistory[1])[pc][to];
+            m.value += (*continuationHistory[2])[pc][to] / 4;
+            m.value += (*continuationHistory[3])[pc][to];
+            m.value += (*continuationHistory[5])[pc][to];
+
+            // bonus for checks
+            m.value += bool(pos.check_squares(pt) & to) * 16384;
+
+            // bonus for escaping from capture
+            m.value += threatenedPieces & from ? (pt == QUEEN && !(to & threatenedByRook)   ? 50000
+                                                  : pt == ROOK && !(to & threatenedByMinor) ? 25000
+                                                  : !(to & threatenedByPawn)                ? 15000
+                                                                                            : 0)
+                                               : 0;
+
+            // malus for putting piece en prise
+            m.value -= !(threatenedPieces & from)
+                       ? (pt == QUEEN ? bool(to & threatenedByRook) * 50000
+                                          + bool(to & threatenedByMinor) * 10000
+                          : pt == ROOK ? bool(to & threatenedByMinor) * 25000
+                          : pt != PAWN ? bool(to & threatenedByPawn) * 15000
+                                       : 0)
+                       : 0;
+        }
+
+        else  // Type == EVASIONS
+        {
+            if (pos.capture_stage(m))
+                m.value =
+                  PieceValue[pos.piece_on(m.to_sq())] - type_of(pos.moved_piece(m)) + (1 << 28);
+            else
+                m.value = (*mainHistory)[pos.side_to_move()][m.from_to()]
+                        + (*continuationHistory[0])[pos.moved_piece(m)][m.to_sq()]
+                        + (*pawnHistory)[pawn_structure_index(pos)][pos.moved_piece(m)][m.to_sq()];
+        }
 }
 
-/// MovePicker::select() returns the next move satisfying a predicate function.
+// Returns the next move satisfying a predicate function.
-/// It never returns the TT move.
+// It never returns the TT move.
 template<MovePicker::PickType T, typename Pred>
 Move MovePicker::select(Pred filter) {
 
-  while (cur < endMoves)
+    while (cur < endMoves)
-  {
+    {
-      if (T == Best)
+        if constexpr (T == Best)
-          std::swap(*cur, *std::max_element(cur, endMoves));
+            std::swap(*cur, *std::max_element(cur, endMoves));
 
-      if (*cur != ttMove && filter())
+        if (*cur != ttMove && filter())
-          return *cur++;
+            return *cur++;
 
-      cur++;
+        cur++;
-  }
+    }
-  return MOVE_NONE;
+    return Move::none();
 }
 
-/// MovePicker::next_move() is the most important method of the MovePicker class. It
+// Most important method of the MovePicker class. It
-/// returns a new pseudo-legal move every time it is called until there are no more
+// returns a new pseudo-legal move every time it is called until there are no more
-/// moves left, picking the move with the highest score from a list of generated moves.
+// moves left, picking the move with the highest score from a list of generated moves.
 Move MovePicker::next_move(bool skipQuiets) {
 
+    auto quiet_threshold = [](Depth d) { return -3330 * d; };
+
 top:
-  switch (stage) {
+    switch (stage)
+    {
 
-  case MAIN_TT:
+    case MAIN_TT :
-  case EVASION_TT:
+    case EVASION_TT :
-  case QSEARCH_TT:
+    case QSEARCH_TT :
-  case PROBCUT_TT:
+    case PROBCUT_TT :
-      ++stage;
+        ++stage;
-      return ttMove;
+        return ttMove;
 
-  case CAPTURE_INIT:
+    case CAPTURE_INIT :
-  case PROBCUT_INIT:
+    case PROBCUT_INIT :
-  case QCAPTURE_INIT:
+    case QCAPTURE_INIT :
-      cur = endBadCaptures = moves;
+        cur = endBadCaptures = moves;
-      endMoves = generate<CAPTURES>(pos, cur);
+        endMoves             = generate<CAPTURES>(pos, cur);
 
-      score<CAPTURES>();
+        score<CAPTURES>();
-      partial_insertion_sort(cur, endMoves, -3000 * depth);
+        partial_insertion_sort(cur, endMoves, std::numeric_limits<int>::min());
-      ++stage;
+        ++stage;
-      goto top;
+        goto top;
 
-  case GOOD_CAPTURE:
+    case GOOD_CAPTURE :
-      if (select<Next>([&](){
+        if (select<Next>([&]() {
-                       return pos.see_ge(*cur, Value(-69 * cur->value / 1024)) ?
+                // Move losing capture to endBadCaptures to be tried later
-                              // Move losing capture to endBadCaptures to be tried later
+                return pos.see_ge(*cur, -cur->value / 18) ? true
-                              true : (*endBadCaptures++ = *cur, false); }))
+                                                          : (*endBadCaptures++ = *cur, false);
-          return *(cur - 1);
+            }))
+            return *(cur - 1);
 
-      // Prepare the pointers to loop over the refutations array
+        // Prepare the pointers to loop over the refutations array
-      cur = std::begin(refutations);
+        cur      = std::begin(refutations);
-      endMoves = std::end(refutations);
+        endMoves = std::end(refutations);
 
-      // If the countermove is the same as a killer, skip it
+        // If the countermove is the same as a killer, skip it
-      if (   refutations[0].move == refutations[2].move
+        if (refutations[0] == refutations[2] || refutations[1] == refutations[2])
-          || refutations[1].move == refutations[2].move)
+            --endMoves;
-          --endMoves;
 
-      ++stage;
+        ++stage;
-      [[fallthrough]];
+        [[fallthrough]];
 
-  case REFUTATION:
+    case REFUTATION :
-      if (select<Next>([&](){ return    *cur != MOVE_NONE
+        if (select<Next>([&]() {
-                                    && !pos.capture(*cur)
+                return *cur != Move::none() && !pos.capture_stage(*cur) && pos.pseudo_legal(*cur);
-                                    &&  pos.pseudo_legal(*cur); }))
+            }))
-          return *(cur - 1);
+            return *(cur - 1);
-      ++stage;
+        ++stage;
-      [[fallthrough]];
+        [[fallthrough]];
 
-  case QUIET_INIT:
+    case QUIET_INIT :
-      if (!skipQuiets)
+        if (!skipQuiets)
-      {
+        {
-          cur = endBadCaptures;
+            cur      = endBadCaptures;
-          endMoves = generate<QUIETS>(pos, cur);
+            endMoves = beginBadQuiets = endBadQuiets = generate<QUIETS>(pos, cur);
 
-          score<QUIETS>();
+            score<QUIETS>();
-          partial_insertion_sort(cur, endMoves, -3000 * depth);
+            partial_insertion_sort(cur, endMoves, quiet_threshold(depth));
-      }
+        }
 
-      ++stage;
+        ++stage;
-      [[fallthrough]];
+        [[fallthrough]];
 
-  case QUIET:
+    case GOOD_QUIET :
-      if (   !skipQuiets
+        if (!skipQuiets && select<Next>([&]() {
-          && select<Next>([&](){return   *cur != refutations[0].move
+                return *cur != refutations[0] && *cur != refutations[1] && *cur != refutations[2];
-                                      && *cur != refutations[1].move
+            }))
-                                      && *cur != refutations[2].move;}))
+        {
-          return *(cur - 1);
+            if ((cur - 1)->value > -8000 || (cur - 1)->value <= quiet_threshold(depth))
+                return *(cur - 1);
 
-      // Prepare the pointers to loop over the bad captures
+            // Remaining quiets are bad
-      cur = moves;
+            beginBadQuiets = cur - 1;
-      endMoves = endBadCaptures;
+        }
 
-      ++stage;
+        // Prepare the pointers to loop over the bad captures
-      [[fallthrough]];
+        cur      = moves;
+        endMoves = endBadCaptures;
 
-  case BAD_CAPTURE:
+        ++stage;
-      return select<Next>([](){ return true; });
+        [[fallthrough]];
 
-  case EVASION_INIT:
+    case BAD_CAPTURE :
-      cur = moves;
+        if (select<Next>([]() { return true; }))
-      endMoves = generate<EVASIONS>(pos, cur);
+            return *(cur - 1);
 
-      score<EVASIONS>();
+        // Prepare the pointers to loop over the bad quiets
-      ++stage;
+        cur      = beginBadQuiets;
-      [[fallthrough]];
+        endMoves = endBadQuiets;
 
-  case EVASION:
+        ++stage;
-      return select<Best>([](){ return true; });
+        [[fallthrough]];
 
-  case PROBCUT:
+    case BAD_QUIET :
-      return select<Next>([&](){ return pos.see_ge(*cur, threshold); });
+        if (!skipQuiets)
+            return select<Next>([&]() {
+                return *cur != refutations[0] && *cur != refutations[1] && *cur != refutations[2];
+            });
 
-  case QCAPTURE:
+        return Move::none();
-      if (select<Next>([&](){ return   depth > DEPTH_QS_RECAPTURES
-                                    || to_sq(*cur) == recaptureSquare; }))
-          return *(cur - 1);
 
-      // If we did not find any move and we do not try checks, we have finished
+    case EVASION_INIT :
-      if (depth != DEPTH_QS_CHECKS)
+        cur      = moves;
-          return MOVE_NONE;
+        endMoves = generate<EVASIONS>(pos, cur);
 
-      ++stage;
+        score<EVASIONS>();
-      [[fallthrough]];
+        ++stage;
+        [[fallthrough]];
 
-  case QCHECK_INIT:
+    case EVASION :
-      cur = moves;
+        return select<Best>([]() { return true; });
-      endMoves = generate<QUIET_CHECKS>(pos, cur);
 
-      ++stage;
+    case PROBCUT :
-      [[fallthrough]];
+        return select<Next>([&]() { return pos.see_ge(*cur, threshold); });
 
-  case QCHECK:
+    case QCAPTURE :
-      return select<Next>([](){ return true; });
+        if (select<Next>([]() { return true; }))
-  }
+            return *(cur - 1);
 
-  assert(false);
+        // If we did not find any move and we do not try checks, we have finished
-  return MOVE_NONE; // Silence warning
+        if (depth != DEPTH_QS_CHECKS)
+            return Move::none();
+
+        ++stage;
+        [[fallthrough]];
+
+    case QCHECK_INIT :
+        cur      = moves;
+        endMoves = generate<QUIET_CHECKS>(pos, cur);
+
+        ++stage;
+        [[fallthrough]];
+
+    case QCHECK :
+        return select<Next>([]() { return true; });
+    }
+
+    assert(false);
+    return Move::none();  // Silence warning
 }
 
-} // namespace Stockfish
+}  // namespace Stockfish

src/movepick.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -20,134 +20,181 @@
 #define MOVEPICK_H_INCLUDED
 
 #include <array>
+#include <cassert>
+#include <cmath>
+#include <cstdint>
+#include <cstdlib>
 #include <limits>
-#include <type_traits>
+#include <type_traits>  // IWYU pragma: keep
 
 #include "movegen.h"
-#include "position.h"
 #include "types.h"
+#include "position.h"
 
 namespace Stockfish {
 
-/// StatsEntry stores the stat table value. It is usually a number but could
+constexpr int PAWN_HISTORY_SIZE        = 512;    // has to be a power of 2
-/// be a move or even a nested history. We use a class instead of naked value
+constexpr int CORRECTION_HISTORY_SIZE  = 16384;  // has to be a power of 2
-/// to directly call history update operator<<() on the entry so to use stats
+constexpr int CORRECTION_HISTORY_LIMIT = 1024;
-/// tables at caller sites as simple multi-dim arrays.
+
+static_assert((PAWN_HISTORY_SIZE & (PAWN_HISTORY_SIZE - 1)) == 0,
+              "PAWN_HISTORY_SIZE has to be a power of 2");
+
+static_assert((CORRECTION_HISTORY_SIZE & (CORRECTION_HISTORY_SIZE - 1)) == 0,
+              "CORRECTION_HISTORY_SIZE has to be a power of 2");
+
+enum PawnHistoryType {
+    Normal,
+    Correction
+};
+
+template<PawnHistoryType T = Normal>
+inline int pawn_structure_index(const Position& pos) {
+    return pos.pawn_key() & ((T == Normal ? PAWN_HISTORY_SIZE : CORRECTION_HISTORY_SIZE) - 1);
+}
+
+// StatsEntry stores the stat table value. It is usually a number but could
+// be a move or even a nested history. We use a class instead of a naked value
+// to directly call history update operator<<() on the entry so to use stats
+// tables at caller sites as simple multi-dim arrays.
 template<typename T, int D>
 class StatsEntry {
 
-  T entry;
+    T entry;
 
-public:
+   public:
-  void operator=(const T& v) { entry = v; }
+    void operator=(const T& v) { entry = v; }
-  T* operator&() { return &entry; }
+    T*   operator&() { return &entry; }
-  T* operator->() { return &entry; }
+    T*   operator->() { return &entry; }
-  operator const T&() const { return entry; }
+    operator const T&() const { return entry; }
 
-  void operator<<(int bonus) {
+    void operator<<(int bonus) {
-    assert(abs(bonus) <= D); // Ensure range is [-D, D]
+        assert(std::abs(bonus) <= D);  // Ensure range is [-D, D]
-    static_assert(D <= std::numeric_limits<T>::max(), "D overflows T");
+        static_assert(D <= std::numeric_limits<T>::max(), "D overflows T");
 
-    entry += bonus - entry * abs(bonus) / D;
+        entry += bonus - entry * std::abs(bonus) / D;
 
-    assert(abs(entry) <= D);
+        assert(std::abs(entry) <= D);
-  }
+    }
 };
 
-/// Stats is a generic N-dimensional array used to store various statistics.
+// Stats is a generic N-dimensional array used to store various statistics.
-/// The first template parameter T is the base type of the array, the second
+// The first template parameter T is the base type of the array, and the second
-/// template parameter D limits the range of updates in [-D, D] when we update
+// template parameter D limits the range of updates in [-D, D] when we update
-/// values with the << operator, while the last parameters (Size and Sizes)
+// values with the << operator, while the last parameters (Size and Sizes)
-/// encode the dimensions of the array.
+// encode the dimensions of the array.
-template <typename T, int D, int Size, int... Sizes>
+template<typename T, int D, int Size, int... Sizes>
-struct Stats : public std::array<Stats<T, D, Sizes...>, Size>
+struct Stats: public std::array<Stats<T, D, Sizes...>, Size> {
-{
+    using stats = Stats<T, D, Size, Sizes...>;
-  typedef Stats<T, D, Size, Sizes...> stats;
 
-  void fill(const T& v) {
+    void fill(const T& v) {
 
-    // For standard-layout 'this' points to first struct member
+        // For standard-layout 'this' points to the first struct member
-    assert(std::is_standard_layout<stats>::value);
+        assert(std::is_standard_layout_v<stats>);
 
-    typedef StatsEntry<T, D> entry;
+        using entry = StatsEntry<T, D>;
-    entry* p = reinterpret_cast<entry*>(this);
+        entry* p    = reinterpret_cast<entry*>(this);
-    std::fill(p, p + sizeof(*this) / sizeof(entry), v);
+        std::fill(p, p + sizeof(*this) / sizeof(entry), v);
-  }
+    }
 };
 
-template <typename T, int D, int Size>
+template<typename T, int D, int Size>
-struct Stats<T, D, Size> : public std::array<StatsEntry<T, D>, Size> {};
+struct Stats<T, D, Size>: public std::array<StatsEntry<T, D>, Size> {};
 
-/// In stats table, D=0 means that the template parameter is not used
+// In stats table, D=0 means that the template parameter is not used
-enum StatsParams { NOT_USED = 0 };
+enum StatsParams {
-enum StatsType { NoCaptures, Captures };
+    NOT_USED = 0
+};
+enum StatsType {
+    NoCaptures,
+    Captures
+};
 
-/// ButterflyHistory records how often quiet moves have been successful or
+// ButterflyHistory records how often quiet moves have been successful or unsuccessful
-/// unsuccessful during the current search, and is used for reduction and move
+// during the current search, and is used for reduction and move ordering decisions.
-/// ordering decisions. It uses 2 tables (one for each color) indexed by
+// It uses 2 tables (one for each color) indexed by the move's from and to squares,
-/// the move's from and to squares, see www.chessprogramming.org/Butterfly_Boards
+// see www.chessprogramming.org/Butterfly_Boards (~11 elo)
-typedef Stats<int16_t, 14365, COLOR_NB, int(SQUARE_NB) * int(SQUARE_NB)> ButterflyHistory;
+using ButterflyHistory = Stats<int16_t, 7183, COLOR_NB, int(SQUARE_NB) * int(SQUARE_NB)>;
 
-/// CounterMoveHistory stores counter moves indexed by [piece][to] of the previous
+// CounterMoveHistory stores counter moves indexed by [piece][to] of the previous
-/// move, see www.chessprogramming.org/Countermove_Heuristic
+// move, see www.chessprogramming.org/Countermove_Heuristic
-typedef Stats<Move, NOT_USED, PIECE_NB, SQUARE_NB> CounterMoveHistory;
+using CounterMoveHistory = Stats<Move, NOT_USED, PIECE_NB, SQUARE_NB>;
 
-/// CapturePieceToHistory is addressed by a move's [piece][to][captured piece type]
+// CapturePieceToHistory is addressed by a move's [piece][to][captured piece type]
-typedef Stats<int16_t, 10692, PIECE_NB, SQUARE_NB, PIECE_TYPE_NB> CapturePieceToHistory;
+using CapturePieceToHistory = Stats<int16_t, 10692, PIECE_NB, SQUARE_NB, PIECE_TYPE_NB>;
 
-/// PieceToHistory is like ButterflyHistory but is addressed by a move's [piece][to]
+// PieceToHistory is like ButterflyHistory but is addressed by a move's [piece][to]
-typedef Stats<int16_t, 29952, PIECE_NB, SQUARE_NB> PieceToHistory;
+using PieceToHistory = Stats<int16_t, 29952, PIECE_NB, SQUARE_NB>;
 
-/// ContinuationHistory is the combined history of a given pair of moves, usually
+// ContinuationHistory is the combined history of a given pair of moves, usually
-/// the current one given a previous one. The nested history table is based on
+// the current one given a previous one. The nested history table is based on
-/// PieceToHistory instead of ButterflyBoards.
+// PieceToHistory instead of ButterflyBoards.
-typedef Stats<PieceToHistory, NOT_USED, PIECE_NB, SQUARE_NB> ContinuationHistory;
+// (~63 elo)
+using ContinuationHistory = Stats<PieceToHistory, NOT_USED, PIECE_NB, SQUARE_NB>;
 
+// PawnHistory is addressed by the pawn structure and a move's [piece][to]
+using PawnHistory = Stats<int16_t, 8192, PAWN_HISTORY_SIZE, PIECE_NB, SQUARE_NB>;
 
-/// MovePicker class is used to pick one pseudo-legal move at a time from the
+// CorrectionHistory is addressed by color and pawn structure
-/// current position. The most important method is next_move(), which returns a
+using CorrectionHistory =
-/// new pseudo-legal move each time it is called, until there are no moves left,
+  Stats<int16_t, CORRECTION_HISTORY_LIMIT, COLOR_NB, CORRECTION_HISTORY_SIZE>;
-/// when MOVE_NONE is returned. In order to improve the efficiency of the
+
-/// alpha-beta algorithm, MovePicker attempts to return the moves which are most
+// MovePicker class is used to pick one pseudo-legal move at a time from the
-/// likely to get a cut-off first.
+// current position. The most important method is next_move(), which returns a
+// new pseudo-legal move each time it is called, until there are no moves left,
+// when Move::none() is returned. In order to improve the efficiency of the
+// alpha-beta algorithm, MovePicker attempts to return the moves which are most
+// likely to get a cut-off first.
 class MovePicker {
 
-  enum PickType { Next, Best };
+    enum PickType {
+        Next,
+        Best
+    };
 
-public:
+   public:
-  MovePicker(const MovePicker&) = delete;
+    MovePicker(const MovePicker&)            = delete;
-  MovePicker& operator=(const MovePicker&) = delete;
+    MovePicker& operator=(const MovePicker&) = delete;
-  MovePicker(const Position&, Move, Depth, const ButterflyHistory*,
+    MovePicker(const Position&,
-                                           const CapturePieceToHistory*,
+               Move,
-                                           const PieceToHistory**,
+               Depth,
-                                           Move,
+               const ButterflyHistory*,
-                                           const Move*);
+               const CapturePieceToHistory*,
-  MovePicker(const Position&, Move, Depth, const ButterflyHistory*,
+               const PieceToHistory**,
-                                           const CapturePieceToHistory*,
+               const PawnHistory*,
-                                           const PieceToHistory**,
+               Move,
-                                           Square);
+               const Move*);
-  MovePicker(const Position&, Move, Value, Depth, const CapturePieceToHistory*);
+    MovePicker(const Position&,
-  Move next_move(bool skipQuiets = false);
+               Move,
+               Depth,
+               const ButterflyHistory*,
+               const CapturePieceToHistory*,
+               const PieceToHistory**,
+               const PawnHistory*);
+    MovePicker(const Position&, Move, int, const CapturePieceToHistory*);
+    Move next_move(bool skipQuiets = false);
 
-private:
+   private:
-  template<PickType T, typename Pred> Move select(Pred);
+    template<PickType T, typename Pred>
-  template<GenType> void score();
+    Move select(Pred);
-  ExtMove* begin() { return cur; }
+    template<GenType>
-  ExtMove* end() { return endMoves; }
+    void     score();
+    ExtMove* begin() { return cur; }
+    ExtMove* end() { return endMoves; }
 
-  const Position& pos;
+    const Position&              pos;
-  const ButterflyHistory* mainHistory;
+    const ButterflyHistory*      mainHistory;
-  const CapturePieceToHistory* captureHistory;
+    const CapturePieceToHistory* captureHistory;
-  const PieceToHistory** continuationHistory;
+    const PieceToHistory**       continuationHistory;
-  Move ttMove;
+    const PawnHistory*           pawnHistory;
-  ExtMove refutations[3], *cur, *endMoves, *endBadCaptures;
+    Move                         ttMove;
-  int stage;
+    ExtMove refutations[3], *cur, *endMoves, *endBadCaptures, *beginBadQuiets, *endBadQuiets;
-  Square recaptureSquare;
+    int     stage;
-  Value threshold;
+    int     threshold;
-  Depth depth;
+    Depth   depth;
-  ExtMove moves[MAX_MOVES];
+    ExtMove moves[MAX_MOVES];
 };
 
-} // namespace Stockfish
+}  // namespace Stockfish
 
-#endif // #ifndef MOVEPICK_H_INCLUDED
+#endif  // #ifndef MOVEPICK_H_INCLUDED

src/nnue/evaluate_nnue.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -18,303 +18,375 @@
 
 // Code for calculating NNUE evaluation function
 
-#include <iostream>
+#include "evaluate_nnue.h"
-#include <set>
+
-#include <sstream>
+#include <cmath>
-#include <iomanip>
+#include <cstdlib>
+#include <cstring>
 #include <fstream>
+#include <iomanip>
+#include <iostream>
+#include <optional>
+#include <sstream>
+#include <string_view>
+#include <type_traits>
+#include <unordered_map>
 
 #include "../evaluate.h"
-#include "../position.h"
 #include "../misc.h"
-#include "../uci.h"
+#include "../position.h"
 #include "../types.h"
-
+#include "../uci.h"
-#include "evaluate_nnue.h"
+#include "nnue_accumulator.h"
+#include "nnue_common.h"
 
 namespace Stockfish::Eval::NNUE {
 
-  // Input feature converter
+// Input feature converter
-  LargePagePtr<FeatureTransformer> featureTransformer;
+LargePagePtr<FeatureTransformer<TransformedFeatureDimensionsBig, &StateInfo::accumulatorBig>>
+  featureTransformerBig;
+LargePagePtr<FeatureTransformer<TransformedFeatureDimensionsSmall, &StateInfo::accumulatorSmall>>
+  featureTransformerSmall;
 
-  // Evaluation function
+// Evaluation function
-  AlignedPtr<Network> network[LayerStacks];
+AlignedPtr<Network<TransformedFeatureDimensionsBig, L2Big, L3Big>>       networkBig[LayerStacks];
+AlignedPtr<Network<TransformedFeatureDimensionsSmall, L2Small, L3Small>> networkSmall[LayerStacks];
 
-  // Evaluation function file name
+// Evaluation function file names
-  std::string fileName;
-  std::string netDescription;
 
-  namespace Detail {
+namespace Detail {
 
-  // Initialize the evaluation function parameters
+// Initialize the evaluation function parameters
-  template <typename T>
+template<typename T>
-  void initialize(AlignedPtr<T>& pointer) {
+void initialize(AlignedPtr<T>& pointer) {
 
     pointer.reset(reinterpret_cast<T*>(std_aligned_alloc(alignof(T), sizeof(T))));
     std::memset(pointer.get(), 0, sizeof(T));
-  }
+}
 
-  template <typename T>
+template<typename T>
-  void initialize(LargePagePtr<T>& pointer) {
+void initialize(LargePagePtr<T>& pointer) {
 
-    static_assert(alignof(T) <= 4096, "aligned_large_pages_alloc() may fail for such a big alignment requirement of T");
+    static_assert(alignof(T) <= 4096,
+                  "aligned_large_pages_alloc() may fail for such a big alignment requirement of T");
     pointer.reset(reinterpret_cast<T*>(aligned_large_pages_alloc(sizeof(T))));
     std::memset(pointer.get(), 0, sizeof(T));
-  }
+}
 
-  // Read evaluation function parameters
+// Read evaluation function parameters
-  template <typename T>
+template<typename T>
-  bool read_parameters(std::istream& stream, T& reference) {
+bool read_parameters(std::istream& stream, T& reference) {
 
     std::uint32_t header;
     header = read_little_endian<std::uint32_t>(stream);
-    if (!stream || header != T::get_hash_value()) return false;
+    if (!stream || header != T::get_hash_value())
+        return false;
     return reference.read_parameters(stream);
-  }
+}
 
-  // Write evaluation function parameters
+// Write evaluation function parameters
-  template <typename T>
+template<typename T>
-  bool write_parameters(std::ostream& stream, const T& reference) {
+bool write_parameters(std::ostream& stream, const T& reference) {
 
     write_little_endian<std::uint32_t>(stream, T::get_hash_value());
     return reference.write_parameters(stream);
-  }
+}
 
-  }  // namespace Detail
+}  // namespace Detail
 
-  // Initialize the evaluation function parameters
-  void initialize() {
 
-    Detail::initialize(featureTransformer);
+// Initialize the evaluation function parameters
-    for (std::size_t i = 0; i < LayerStacks; ++i)
+static void initialize(NetSize netSize) {
-      Detail::initialize(network[i]);
-  }
 
-  // Read network header
+    if (netSize == Small)
-  bool read_header(std::istream& stream, std::uint32_t* hashValue, std::string* desc)
+    {
-  {
+        Detail::initialize(featureTransformerSmall);
+        for (std::size_t i = 0; i < LayerStacks; ++i)
+            Detail::initialize(networkSmall[i]);
+    }
+    else
+    {
+        Detail::initialize(featureTransformerBig);
+        for (std::size_t i = 0; i < LayerStacks; ++i)
+            Detail::initialize(networkBig[i]);
+    }
+}
+
+// Read network header
+static bool read_header(std::istream& stream, std::uint32_t* hashValue, std::string* desc) {
     std::uint32_t version, size;
 
-    version     = read_little_endian<std::uint32_t>(stream);
+    version    = read_little_endian<std::uint32_t>(stream);
-    *hashValue  = read_little_endian<std::uint32_t>(stream);
+    *hashValue = read_little_endian<std::uint32_t>(stream);
-    size        = read_little_endian<std::uint32_t>(stream);
+    size       = read_little_endian<std::uint32_t>(stream);
-    if (!stream || version != Version) return false;
+    if (!stream || version != Version)
+        return false;
     desc->resize(size);
     stream.read(&(*desc)[0], size);
     return !stream.fail();
-  }
+}
 
-  // Write network header
+// Write network header
-  bool write_header(std::ostream& stream, std::uint32_t hashValue, const std::string& desc)
+static bool write_header(std::ostream& stream, std::uint32_t hashValue, const std::string& desc) {
-  {
     write_little_endian<std::uint32_t>(stream, Version);
     write_little_endian<std::uint32_t>(stream, hashValue);
-    write_little_endian<std::uint32_t>(stream, (std::uint32_t)desc.size());
+    write_little_endian<std::uint32_t>(stream, std::uint32_t(desc.size()));
     stream.write(&desc[0], desc.size());
     return !stream.fail();
-  }
+}
 
-  // Read network parameters
+// Read network parameters
-  bool read_parameters(std::istream& stream) {
+static bool read_parameters(std::istream& stream, NetSize netSize, std::string& netDescription) {
 
     std::uint32_t hashValue;
-    if (!read_header(stream, &hashValue, &netDescription)) return false;
+    if (!read_header(stream, &hashValue, &netDescription))
-    if (hashValue != HashValue) return false;
+        return false;
-    if (!Detail::read_parameters(stream, *featureTransformer)) return false;
+    if (hashValue != HashValue[netSize])
+        return false;
+    if (netSize == Big && !Detail::read_parameters(stream, *featureTransformerBig))
+        return false;
+    if (netSize == Small && !Detail::read_parameters(stream, *featureTransformerSmall))
+        return false;
     for (std::size_t i = 0; i < LayerStacks; ++i)
-      if (!Detail::read_parameters(stream, *(network[i]))) return false;
+    {
+        if (netSize == Big && !Detail::read_parameters(stream, *(networkBig[i])))
+            return false;
+        if (netSize == Small && !Detail::read_parameters(stream, *(networkSmall[i])))
+            return false;
+    }
     return stream && stream.peek() == std::ios::traits_type::eof();
-  }
+}
 
-  // Write network parameters
+// Write network parameters
-  bool write_parameters(std::ostream& stream) {
+static bool
+write_parameters(std::ostream& stream, NetSize netSize, const std::string& netDescription) {
 
-    if (!write_header(stream, HashValue, netDescription)) return false;
+    if (!write_header(stream, HashValue[netSize], netDescription))
-    if (!Detail::write_parameters(stream, *featureTransformer)) return false;
+        return false;
+    if (netSize == Big && !Detail::write_parameters(stream, *featureTransformerBig))
+        return false;
+    if (netSize == Small && !Detail::write_parameters(stream, *featureTransformerSmall))
+        return false;
     for (std::size_t i = 0; i < LayerStacks; ++i)
-      if (!Detail::write_parameters(stream, *(network[i]))) return false;
+    {
-    return (bool)stream;
+        if (netSize == Big && !Detail::write_parameters(stream, *(networkBig[i])))
-  }
+            return false;
+        if (netSize == Small && !Detail::write_parameters(stream, *(networkSmall[i])))
+            return false;
+    }
+    return bool(stream);
+}
 
-  // Evaluation function. Perform differential calculation.
+void hint_common_parent_position(const Position& pos) {
-  Value evaluate(const Position& pos, bool adjusted) {
+
+    int simpleEval = simple_eval(pos, pos.side_to_move());
+    if (std::abs(simpleEval) > 1050)
+        featureTransformerSmall->hint_common_access(pos);
+    else
+        featureTransformerBig->hint_common_access(pos);
+}
+
+// Evaluation function. Perform differential calculation.
+template<NetSize Net_Size>
+Value evaluate(const Position& pos, bool adjusted, int* complexity) {
 
     // We manually align the arrays on the stack because with gcc < 9.3
     // overaligning stack variables with alignas() doesn't work correctly.
 
     constexpr uint64_t alignment = CacheLineSize;
-    int delta = 10 - pos.non_pawn_material() / 1515;
+    constexpr int      delta     = 24;
 
 #if defined(ALIGNAS_ON_STACK_VARIABLES_BROKEN)
-    TransformedFeatureType transformedFeaturesUnaligned[
+    TransformedFeatureType transformedFeaturesUnaligned
-      FeatureTransformer::BufferSize + alignment / sizeof(TransformedFeatureType)];
+      [FeatureTransformer < Net_Size == Small ? TransformedFeatureDimensionsSmall
+                                              : TransformedFeatureDimensionsBig,
+       nullptr > ::BufferSize + alignment / sizeof(TransformedFeatureType)];
 
     auto* transformedFeatures = align_ptr_up<alignment>(&transformedFeaturesUnaligned[0]);
 #else
-    alignas(alignment)
+
-      TransformedFeatureType transformedFeatures[FeatureTransformer::BufferSize];
+    alignas(alignment) TransformedFeatureType
+      transformedFeatures[FeatureTransformer < Net_Size == Small ? TransformedFeatureDimensionsSmall
+                                                                 : TransformedFeatureDimensionsBig,
+                          nullptr > ::BufferSize];
 #endif
 
     ASSERT_ALIGNED(transformedFeatures, alignment);
 
-    const int bucket = (pos.count<ALL_PIECES>() - 1) / 4;
+    const int  bucket     = (pos.count<ALL_PIECES>() - 1) / 4;
-    const auto psqt = featureTransformer->transform(pos, transformedFeatures, bucket);
+    const auto psqt       = Net_Size == Small
-    const auto positional = network[bucket]->propagate(transformedFeatures);
+                            ? featureTransformerSmall->transform(pos, transformedFeatures, bucket)
+                            : featureTransformerBig->transform(pos, transformedFeatures, bucket);
+    const auto positional = Net_Size == Small ? networkSmall[bucket]->propagate(transformedFeatures)
+                                              : networkBig[bucket]->propagate(transformedFeatures);
+
+    if (complexity)
+        *complexity = std::abs(psqt - positional) / OutputScale;
 
     // Give more value to positional evaluation when adjusted flag is set
     if (adjusted)
-        return static_cast<Value>(((128 - delta) * psqt + (128 + delta) * positional) / 128 / OutputScale);
+        return static_cast<Value>(((1024 - delta) * psqt + (1024 + delta) * positional)
+                                  / (1024 * OutputScale));
     else
         return static_cast<Value>((psqt + positional) / OutputScale);
-  }
+}
 
-  struct NnueEvalTrace {
+template Value evaluate<Big>(const Position& pos, bool adjusted, int* complexity);
+template Value evaluate<Small>(const Position& pos, bool adjusted, int* complexity);
+
+struct NnueEvalTrace {
     static_assert(LayerStacks == PSQTBuckets);
 
-    Value psqt[LayerStacks];
+    Value       psqt[LayerStacks];
-    Value positional[LayerStacks];
+    Value       positional[LayerStacks];
     std::size_t correctBucket;
-  };
+};
 
-  static NnueEvalTrace trace_evaluate(const Position& pos) {
+static NnueEvalTrace trace_evaluate(const Position& pos) {
 
     // We manually align the arrays on the stack because with gcc < 9.3
     // overaligning stack variables with alignas() doesn't work correctly.
-
     constexpr uint64_t alignment = CacheLineSize;
 
 #if defined(ALIGNAS_ON_STACK_VARIABLES_BROKEN)
-    TransformedFeatureType transformedFeaturesUnaligned[
+    TransformedFeatureType transformedFeaturesUnaligned
-      FeatureTransformer::BufferSize + alignment / sizeof(TransformedFeatureType)];
+      [FeatureTransformer<TransformedFeatureDimensionsBig, nullptr>::BufferSize
+       + alignment / sizeof(TransformedFeatureType)];
 
     auto* transformedFeatures = align_ptr_up<alignment>(&transformedFeaturesUnaligned[0]);
 #else
-    alignas(alignment)
+    alignas(alignment) TransformedFeatureType
-      TransformedFeatureType transformedFeatures[FeatureTransformer::BufferSize];
+      transformedFeatures[FeatureTransformer<TransformedFeatureDimensionsBig, nullptr>::BufferSize];
 #endif
 
     ASSERT_ALIGNED(transformedFeatures, alignment);
 
     NnueEvalTrace t{};
     t.correctBucket = (pos.count<ALL_PIECES>() - 1) / 4;
-    for (IndexType bucket = 0; bucket < LayerStacks; ++bucket) {
+    for (IndexType bucket = 0; bucket < LayerStacks; ++bucket)
-      const auto materialist = featureTransformer->transform(pos, transformedFeatures, bucket);
+    {
-      const auto positional = network[bucket]->propagate(transformedFeatures);
+        const auto materialist = featureTransformerBig->transform(pos, transformedFeatures, bucket);
+        const auto positional  = networkBig[bucket]->propagate(transformedFeatures);
 
-      t.psqt[bucket] = static_cast<Value>( materialist / OutputScale );
+        t.psqt[bucket]       = static_cast<Value>(materialist / OutputScale);
-      t.positional[bucket] = static_cast<Value>( positional / OutputScale );
+        t.positional[bucket] = static_cast<Value>(positional / OutputScale);
     }
 
     return t;
-  }
+}
 
-  static const std::string PieceToChar(" PNBRQK  pnbrqk");
+constexpr std::string_view PieceToChar(" PNBRQK  pnbrqk");
 
 
-  // format_cp_compact() converts a Value into (centi)pawns and writes it in a buffer.
+// Converts a Value into (centi)pawns and writes it in a buffer.
-  // The buffer must have capacity for at least 5 chars.
+// The buffer must have capacity for at least 5 chars.
-  static void format_cp_compact(Value v, char* buffer) {
+static void format_cp_compact(Value v, char* buffer) {
 
     buffer[0] = (v < 0 ? '-' : v > 0 ? '+' : ' ');
 
-    int cp = std::abs(100 * v / PawnValueEg);
+    int cp = std::abs(UCI::to_cp(v));
     if (cp >= 10000)
     {
-        buffer[1] = '0' + cp / 10000; cp %= 10000;
+        buffer[1] = '0' + cp / 10000;
-        buffer[2] = '0' + cp / 1000; cp %= 1000;
+        cp %= 10000;
+        buffer[2] = '0' + cp / 1000;
+        cp %= 1000;
         buffer[3] = '0' + cp / 100;
         buffer[4] = ' ';
     }
     else if (cp >= 1000)
     {
-        buffer[1] = '0' + cp / 1000; cp %= 1000;
+        buffer[1] = '0' + cp / 1000;
-        buffer[2] = '0' + cp / 100; cp %= 100;
+        cp %= 1000;
+        buffer[2] = '0' + cp / 100;
+        cp %= 100;
         buffer[3] = '.';
         buffer[4] = '0' + cp / 10;
     }
     else
     {
-        buffer[1] = '0' + cp / 100; cp %= 100;
+        buffer[1] = '0' + cp / 100;
+        cp %= 100;
         buffer[2] = '.';
-        buffer[3] = '0' + cp / 10; cp %= 10;
+        buffer[3] = '0' + cp / 10;
+        cp %= 10;
         buffer[4] = '0' + cp / 1;
     }
-  }
+}
 
 
-  // format_cp_aligned_dot() converts a Value into (centi)pawns and writes it in a buffer,
+// Converts a Value into pawns, always keeping two decimals
-  // always keeping two decimals. The buffer must have capacity for at least 7 chars.
+static void format_cp_aligned_dot(Value v, std::stringstream& stream) {
-  static void format_cp_aligned_dot(Value v, char* buffer) {
 
-    buffer[0] = (v < 0 ? '-' : v > 0 ? '+' : ' ');
+    const double pawns = std::abs(0.01 * UCI::to_cp(v));
 
-    double cp = 1.0 * std::abs(int(v)) / PawnValueEg;
+    stream << (v < 0   ? '-'
-    sprintf(&buffer[1], "%6.2f", cp);
+               : v > 0 ? '+'
-  }
+                       : ' ')
+           << std::setiosflags(std::ios::fixed) << std::setw(6) << std::setprecision(2) << pawns;
+}
 
 
-  // trace() returns a string with the value of each piece on a board,
+// Returns a string with the value of each piece on a board,
-  // and a table for (PSQT, Layers) values bucket by bucket.
+// and a table for (PSQT, Layers) values bucket by bucket.
-
+std::string trace(Position& pos) {
-  std::string trace(Position& pos) {
 
     std::stringstream ss;
 
-    char board[3*8+1][8*8+2];
+    char board[3 * 8 + 1][8 * 8 + 2];
     std::memset(board, ' ', sizeof(board));
-    for (int row = 0; row < 3*8+1; ++row)
+    for (int row = 0; row < 3 * 8 + 1; ++row)
-      board[row][8*8+1] = '\0';
+        board[row][8 * 8 + 1] = '\0';
 
     // A lambda to output one box of the board
     auto writeSquare = [&board](File file, Rank rank, Piece pc, Value value) {
-
+        const int x = int(file) * 8;
-      const int x = ((int)file) * 8;
+        const int y = (7 - int(rank)) * 3;
-      const int y = (7 - (int)rank) * 3;
+        for (int i = 1; i < 8; ++i)
-      for (int i = 1; i < 8; ++i)
+            board[y][x + i] = board[y + 3][x + i] = '-';
-         board[y][x+i] = board[y+3][x+i] = '-';
+        for (int i = 1; i < 3; ++i)
-      for (int i = 1; i < 3; ++i)
+            board[y + i][x] = board[y + i][x + 8] = '|';
-         board[y+i][x] = board[y+i][x+8] = '|';
+        board[y][x] = board[y][x + 8] = board[y + 3][x + 8] = board[y + 3][x] = '+';
-      board[y][x] = board[y][x+8] = board[y+3][x+8] = board[y+3][x] = '+';
+        if (pc != NO_PIECE)
-      if (pc != NO_PIECE)
+            board[y + 1][x + 4] = PieceToChar[pc];
-        board[y+1][x+4] = PieceToChar[pc];
+        if (value != VALUE_NONE)
-      if (value != VALUE_NONE)
+            format_cp_compact(value, &board[y + 2][x + 2]);
-        format_cp_compact(value, &board[y+2][x+2]);
     };
 
     // We estimate the value of each piece by doing a differential evaluation from
     // the current base eval, simulating the removal of the piece from its square.
-    Value base = evaluate(pos);
+    Value base = evaluate<NNUE::Big>(pos);
-    base = pos.side_to_move() == WHITE ? base : -base;
+    base       = pos.side_to_move() == WHITE ? base : -base;
 
     for (File f = FILE_A; f <= FILE_H; ++f)
-      for (Rank r = RANK_1; r <= RANK_8; ++r)
+        for (Rank r = RANK_1; r <= RANK_8; ++r)
-      {
-        Square sq = make_square(f, r);
-        Piece pc = pos.piece_on(sq);
-        Value v = VALUE_NONE;
-
-        if (pc != NO_PIECE && type_of(pc) != KING)
         {
-          auto st = pos.state();
+            Square sq = make_square(f, r);
+            Piece  pc = pos.piece_on(sq);
+            Value  v  = VALUE_NONE;
 
-          pos.remove_piece(sq);
+            if (pc != NO_PIECE && type_of(pc) != KING)
-          st->accumulator.computed[WHITE] = false;
+            {
-          st->accumulator.computed[BLACK] = false;
+                auto st = pos.state();
 
-          Value eval = evaluate(pos);
+                pos.remove_piece(sq);
-          eval = pos.side_to_move() == WHITE ? eval : -eval;
+                st->accumulatorBig.computed[WHITE] = false;
-          v = base - eval;
+                st->accumulatorBig.computed[BLACK] = false;
 
-          pos.put_piece(pc, sq);
+                Value eval = evaluate<NNUE::Big>(pos);
-          st->accumulator.computed[WHITE] = false;
+                eval       = pos.side_to_move() == WHITE ? eval : -eval;
-          st->accumulator.computed[BLACK] = false;
+                v          = base - eval;
+
+                pos.put_piece(pc, sq);
+                st->accumulatorBig.computed[WHITE] = false;
+                st->accumulatorBig.computed[BLACK] = false;
+            }
+
+            writeSquare(f, r, pc, v);
         }
 
-        writeSquare(f, r, pc, v);
-      }
-
     ss << " NNUE derived piece values:\n";
-    for (int row = 0; row < 3*8+1; ++row)
+    for (int row = 0; row < 3 * 8 + 1; ++row)
         ss << board[row] << '\n';
     ss << '\n';
 
@@ -329,48 +401,53 @@ namespace Stockfish::Eval::NNUE {
 
     for (std::size_t bucket = 0; bucket < LayerStacks; ++bucket)
     {
-      char buffer[3][8];
+        ss << "|  " << bucket << "        ";
-      std::memset(buffer, '\0', sizeof(buffer));
+        ss << " |  ";
-
+        format_cp_aligned_dot(t.psqt[bucket], ss);
-      format_cp_aligned_dot(t.psqt[bucket], buffer[0]);
+        ss << "  "
-      format_cp_aligned_dot(t.positional[bucket], buffer[1]);
+           << " |  ";
-      format_cp_aligned_dot(t.psqt[bucket] + t.positional[bucket], buffer[2]);
+        format_cp_aligned_dot(t.positional[bucket], ss);
-
+        ss << "  "
-      ss <<  "|  " << bucket    << "        "
+           << " |  ";
-         << " |  " << buffer[0] << "  "
+        format_cp_aligned_dot(t.psqt[bucket] + t.positional[bucket], ss);
-         << " |  " << buffer[1] << "  "
+        ss << "  "
-         << " |  " << buffer[2] << "  "
+           << " |";
-         << " |";
+        if (bucket == t.correctBucket)
-      if (bucket == t.correctBucket)
+            ss << " <-- this bucket is used";
-          ss << " <-- this bucket is used";
+        ss << '\n';
-      ss << '\n';
     }
 
     ss << "+------------+------------+------------+------------+\n";
 
     return ss.str();
-  }
+}
 
 
-  // Load eval, from a file stream or a memory stream
+// Load eval, from a file stream or a memory stream
-  bool load_eval(std::string name, std::istream& stream) {
+std::optional<std::string> load_eval(std::istream& stream, NetSize netSize) {
 
-    initialize();
+    initialize(netSize);
-    fileName = name;
+    std::string netDescription;
-    return read_parameters(stream);
+    return read_parameters(stream, netSize, netDescription) ? std::make_optional(netDescription)
-  }
+                                                            : std::nullopt;
+}
 
-  // Save eval, to a file stream or a memory stream
+// Save eval, to a file stream or a memory stream
-  bool save_eval(std::ostream& stream) {
+bool save_eval(std::ostream&      stream,
+               NetSize            netSize,
+               const std::string& name,
+               const std::string& netDescription) {
 
-    if (fileName.empty())
+    if (name.empty() || name == "None")
-      return false;
+        return false;
 
-    return write_parameters(stream);
+    return write_parameters(stream, netSize, netDescription);
-  }
+}
 
-  /// Save eval, to a file given by its name
+// Save eval, to a file given by its name
-  bool save_eval(const std::optional<std::string>& filename) {
+bool save_eval(const std::optional<std::string>&                              filename,
+               NetSize                                                        netSize,
+               const std::unordered_map<Eval::NNUE::NetSize, Eval::EvalFile>& evalFiles) {
 
     std::string actualFilename;
     std::string msg;
@@ -379,25 +456,27 @@ namespace Stockfish::Eval::NNUE {
         actualFilename = filename.value();
     else
     {
-        if (currentEvalFileName != EvalFileDefaultName)
+        if (evalFiles.at(netSize).current
+            != (netSize == Small ? EvalFileDefaultNameSmall : EvalFileDefaultNameBig))
         {
-             msg = "Failed to export a net. A non-embedded net can only be saved if the filename is specified";
+            msg = "Failed to export a net. "
+                  "A non-embedded net can only be saved if the filename is specified";
 
-             sync_cout << msg << sync_endl;
+            sync_cout << msg << sync_endl;
-             return false;
+            return false;
         }
-        actualFilename = EvalFileDefaultName;
+        actualFilename = (netSize == Small ? EvalFileDefaultNameSmall : EvalFileDefaultNameBig);
     }
 
     std::ofstream stream(actualFilename, std::ios_base::binary);
-    bool saved = save_eval(stream);
+    bool          saved = save_eval(stream, netSize, evalFiles.at(netSize).current,
+                                    evalFiles.at(netSize).netDescription);
 
-    msg = saved ? "Network saved successfully to " + actualFilename
+    msg = saved ? "Network saved successfully to " + actualFilename : "Failed to export a net";
-                : "Failed to export a net";
 
     sync_cout << msg << sync_endl;
     return saved;
-  }
+}
 
 
-} // namespace Stockfish::Eval::NNUE
+}  // namespace Stockfish::Eval::NNUE

src/nnue/evaluate_nnue.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,39 +21,73 @@
 #ifndef NNUE_EVALUATE_NNUE_H_INCLUDED
 #define NNUE_EVALUATE_NNUE_H_INCLUDED
 
+#include <cstdint>
+#include <iosfwd>
+#include <memory>
+#include <optional>
+#include <string>
+#include <unordered_map>
+
+#include "../misc.h"
+#include "../types.h"
+#include "nnue_architecture.h"
 #include "nnue_feature_transformer.h"
 
-#include <memory>
+namespace Stockfish {
+class Position;
+
+namespace Eval {
+struct EvalFile;
+}
+
+}
 
 namespace Stockfish::Eval::NNUE {
 
-  // Hash value of evaluation function structure
+// Hash value of evaluation function structure
-  constexpr std::uint32_t HashValue =
+constexpr std::uint32_t HashValue[2] = {
-      FeatureTransformer::get_hash_value() ^ Network::get_hash_value();
+  FeatureTransformer<TransformedFeatureDimensionsBig, nullptr>::get_hash_value()
+    ^ Network<TransformedFeatureDimensionsBig, L2Big, L3Big>::get_hash_value(),
+  FeatureTransformer<TransformedFeatureDimensionsSmall, nullptr>::get_hash_value()
+    ^ Network<TransformedFeatureDimensionsSmall, L2Small, L3Small>::get_hash_value()};
 
-  // Deleter for automating release of memory area
+// Deleter for automating release of memory area
-  template <typename T>
+template<typename T>
-  struct AlignedDeleter {
+struct AlignedDeleter {
     void operator()(T* ptr) const {
-      ptr->~T();
+        ptr->~T();
-      std_aligned_free(ptr);
+        std_aligned_free(ptr);
     }
-  };
+};
 
-  template <typename T>
+template<typename T>
-  struct LargePageDeleter {
+struct LargePageDeleter {
     void operator()(T* ptr) const {
-      ptr->~T();
+        ptr->~T();
-      aligned_large_pages_free(ptr);
+        aligned_large_pages_free(ptr);
     }
-  };
+};
 
-  template <typename T>
+template<typename T>
-  using AlignedPtr = std::unique_ptr<T, AlignedDeleter<T>>;
+using AlignedPtr = std::unique_ptr<T, AlignedDeleter<T>>;
 
-  template <typename T>
+template<typename T>
-  using LargePagePtr = std::unique_ptr<T, LargePageDeleter<T>>;
+using LargePagePtr = std::unique_ptr<T, LargePageDeleter<T>>;
+
+std::string trace(Position& pos);
+template<NetSize Net_Size>
+Value evaluate(const Position& pos, bool adjusted = false, int* complexity = nullptr);
+void  hint_common_parent_position(const Position& pos);
+
+std::optional<std::string> load_eval(std::istream& stream, NetSize netSize);
+bool                       save_eval(std::ostream&      stream,
+                                     NetSize            netSize,
+                                     const std::string& name,
+                                     const std::string& netDescription);
+bool                       save_eval(const std::optional<std::string>& filename,
+                                     NetSize                           netSize,
+                                     const std::unordered_map<Eval::NNUE::NetSize, Eval::EvalFile>&);
 
 }  // namespace Stockfish::Eval::NNUE
 
-#endif // #ifndef NNUE_EVALUATE_NNUE_H_INCLUDED
+#endif  // #ifndef NNUE_EVALUATE_NNUE_H_INCLUDED

src/nnue/features/half_ka_v2_hm.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -20,64 +20,67 @@
 
 #include "half_ka_v2_hm.h"
 
+#include "../../bitboard.h"
 #include "../../position.h"
+#include "../../types.h"
+#include "../nnue_common.h"
 
 namespace Stockfish::Eval::NNUE::Features {
 
-  // Orient a square according to perspective (rotates by 180 for black)
+// Index of a feature for a given king position and another piece on some square
-  inline Square HalfKAv2_hm::orient(Color perspective, Square s, Square ksq) {
+template<Color Perspective>
-    return Square(int(s) ^ (bool(perspective) * SQ_A8) ^ ((file_of(ksq) < FILE_E) * SQ_H1));
+inline IndexType HalfKAv2_hm::make_index(Square s, Piece pc, Square ksq) {
-  }
+    return IndexType((int(s) ^ OrientTBL[Perspective][ksq]) + PieceSquareIndex[Perspective][pc]
+                     + KingBuckets[Perspective][ksq]);
+}
 
-  // Index of a feature for a given king position and another piece on some square
+// Get a list of indices for active features
-  inline IndexType HalfKAv2_hm::make_index(Color perspective, Square s, Piece pc, Square ksq) {
+template<Color Perspective>
-    Square o_ksq = orient(perspective, ksq, ksq);
+void HalfKAv2_hm::append_active_indices(const Position& pos, IndexList& active) {
-    return IndexType(orient(perspective, s, ksq) + PieceSquareIndex[perspective][pc] + PS_NB * KingBuckets[o_ksq]);
+    Square   ksq = pos.square<KING>(Perspective);
-  }
+    Bitboard bb  = pos.pieces();
-
-  // Get a list of indices for active features
-  void HalfKAv2_hm::append_active_indices(
-    const Position& pos,
-    Color perspective,
-    IndexList& active
-  ) {
-    Square ksq = pos.square<KING>(perspective);
-    Bitboard bb = pos.pieces();
     while (bb)
     {
-      Square s = pop_lsb(bb);
+        Square s = pop_lsb(bb);
-      active.push_back(make_index(perspective, s, pos.piece_on(s), ksq));
+        active.push_back(make_index<Perspective>(s, pos.piece_on(s), ksq));
     }
-  }
+}
 
+// Explicit template instantiations
+template void HalfKAv2_hm::append_active_indices<WHITE>(const Position& pos, IndexList& active);
+template void HalfKAv2_hm::append_active_indices<BLACK>(const Position& pos, IndexList& active);
 
-  // append_changed_indices() : get a list of indices for recently changed features
+// Get a list of indices for recently changed features
-
+template<Color Perspective>
-  void HalfKAv2_hm::append_changed_indices(
+void HalfKAv2_hm::append_changed_indices(Square            ksq,
-    Square ksq,
+                                         const DirtyPiece& dp,
-    const DirtyPiece& dp,
+                                         IndexList&        removed,
-    Color perspective,
+                                         IndexList&        added) {
-    IndexList& removed,
+    for (int i = 0; i < dp.dirty_num; ++i)
-    IndexList& added
+    {
-  ) {
+        if (dp.from[i] != SQ_NONE)
-    for (int i = 0; i < dp.dirty_num; ++i) {
+            removed.push_back(make_index<Perspective>(dp.from[i], dp.piece[i], ksq));
-      if (dp.from[i] != SQ_NONE)
+        if (dp.to[i] != SQ_NONE)
-        removed.push_back(make_index(perspective, dp.from[i], dp.piece[i], ksq));
+            added.push_back(make_index<Perspective>(dp.to[i], dp.piece[i], ksq));
-      if (dp.to[i] != SQ_NONE)
-        added.push_back(make_index(perspective, dp.to[i], dp.piece[i], ksq));
     }
-  }
+}
 
-  int HalfKAv2_hm::update_cost(const StateInfo* st) {
+// Explicit template instantiations
-    return st->dirtyPiece.dirty_num;
+template void HalfKAv2_hm::append_changed_indices<WHITE>(Square            ksq,
-  }
+                                                         const DirtyPiece& dp,
+                                                         IndexList&        removed,
+                                                         IndexList&        added);
+template void HalfKAv2_hm::append_changed_indices<BLACK>(Square            ksq,
+                                                         const DirtyPiece& dp,
+                                                         IndexList&        removed,
+                                                         IndexList&        added);
 
-  int HalfKAv2_hm::refresh_cost(const Position& pos) {
+int HalfKAv2_hm::update_cost(const StateInfo* st) { return st->dirtyPiece.dirty_num; }
-    return pos.count<ALL_PIECES>();
-  }
 
-  bool HalfKAv2_hm::requires_refresh(const StateInfo* st, Color perspective) {
+int HalfKAv2_hm::refresh_cost(const Position& pos) { return pos.count<ALL_PIECES>(); }
+
+bool HalfKAv2_hm::requires_refresh(const StateInfo* st, Color perspective) {
     return st->dirtyPiece.piece[0] == make_piece(perspective, KING);
-  }
+}
 
 }  // namespace Stockfish::Eval::NNUE::Features

src/nnue/features/half_ka_v2_hm.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,52 +21,51 @@
 #ifndef NNUE_FEATURES_HALF_KA_V2_HM_H_INCLUDED
 #define NNUE_FEATURES_HALF_KA_V2_HM_H_INCLUDED
 
+#include <cstdint>
+
+#include "../../misc.h"
+#include "../../types.h"
 #include "../nnue_common.h"
 
-#include "../../evaluate.h"
-#include "../../misc.h"
-
 namespace Stockfish {
-  struct StateInfo;
+struct StateInfo;
+class Position;
 }
 
 namespace Stockfish::Eval::NNUE::Features {
 
-  // Feature HalfKAv2_hm: Combination of the position of own king
+// Feature HalfKAv2_hm: Combination of the position of own king and the
-  // and the position of pieces. Position mirrored such that king always on e..h files.
+// position of pieces. Position mirrored such that king is always on e..h files.
-  class HalfKAv2_hm {
+class HalfKAv2_hm {
 
-    // unique number for each piece type on each square
+    // Unique number for each piece type on each square
     enum {
-      PS_NONE     =  0,
+        PS_NONE     = 0,
-      PS_W_PAWN   =  0,
+        PS_W_PAWN   = 0,
-      PS_B_PAWN   =  1 * SQUARE_NB,
+        PS_B_PAWN   = 1 * SQUARE_NB,
-      PS_W_KNIGHT =  2 * SQUARE_NB,
+        PS_W_KNIGHT = 2 * SQUARE_NB,
-      PS_B_KNIGHT =  3 * SQUARE_NB,
+        PS_B_KNIGHT = 3 * SQUARE_NB,
-      PS_W_BISHOP =  4 * SQUARE_NB,
+        PS_W_BISHOP = 4 * SQUARE_NB,
-      PS_B_BISHOP =  5 * SQUARE_NB,
+        PS_B_BISHOP = 5 * SQUARE_NB,
-      PS_W_ROOK   =  6 * SQUARE_NB,
+        PS_W_ROOK   = 6 * SQUARE_NB,
-      PS_B_ROOK   =  7 * SQUARE_NB,
+        PS_B_ROOK   = 7 * SQUARE_NB,
-      PS_W_QUEEN  =  8 * SQUARE_NB,
+        PS_W_QUEEN  = 8 * SQUARE_NB,
-      PS_B_QUEEN  =  9 * SQUARE_NB,
+        PS_B_QUEEN  = 9 * SQUARE_NB,
-      PS_KING     =  10 * SQUARE_NB,
+        PS_KING     = 10 * SQUARE_NB,
-      PS_NB       =  11 * SQUARE_NB
+        PS_NB       = 11 * SQUARE_NB
     };
 
     static constexpr IndexType PieceSquareIndex[COLOR_NB][PIECE_NB] = {
-      // convention: W - us, B - them
+      // Convention: W - us, B - them
-      // viewed from other side, W and B are reversed
+      // Viewed from other side, W and B are reversed
-      { PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE,
+      {PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE,
-        PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE },
+       PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE},
-      { PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE,
+      {PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE,
-        PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE }
+       PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE}};
-    };
-
-    // Orient a square according to perspective (rotates by 180 for black)
-    static Square orient(Color perspective, Square s, Square ksq);
 
     // Index of a feature for a given king position and another piece on some square
-    static IndexType make_index(Color perspective, Square s, Piece pc, Square ksq);
+    template<Color Perspective>
+    static IndexType make_index(Square s, Piece pc, Square ksq);
 
    public:
     // Feature name
@@ -77,48 +76,75 @@ namespace Stockfish::Eval::NNUE::Features {
 
     // Number of feature dimensions
     static constexpr IndexType Dimensions =
-        static_cast<IndexType>(SQUARE_NB) * static_cast<IndexType>(PS_NB) / 2;
+      static_cast<IndexType>(SQUARE_NB) * static_cast<IndexType>(PS_NB) / 2;
 
-    static constexpr int KingBuckets[64] = {
+#define B(v) (v * PS_NB)
-      -1, -1, -1, -1, 31, 30, 29, 28,
+    // clang-format off
-      -1, -1, -1, -1, 27, 26, 25, 24,
+    static constexpr int KingBuckets[COLOR_NB][SQUARE_NB] = {
-      -1, -1, -1, -1, 23, 22, 21, 20,
+      { B(28), B(29), B(30), B(31), B(31), B(30), B(29), B(28),
-      -1, -1, -1, -1, 19, 18, 17, 16,
+        B(24), B(25), B(26), B(27), B(27), B(26), B(25), B(24),
-      -1, -1, -1, -1, 15, 14, 13, 12,
+        B(20), B(21), B(22), B(23), B(23), B(22), B(21), B(20),
-      -1, -1, -1, -1, 11, 10,  9,  8,
+        B(16), B(17), B(18), B(19), B(19), B(18), B(17), B(16),
-      -1, -1, -1, -1,  7,  6,  5,  4,
+        B(12), B(13), B(14), B(15), B(15), B(14), B(13), B(12),
-      -1, -1, -1, -1,  3,  2,  1,  0
+        B( 8), B( 9), B(10), B(11), B(11), B(10), B( 9), B( 8),
+        B( 4), B( 5), B( 6), B( 7), B( 7), B( 6), B( 5), B( 4),
+        B( 0), B( 1), B( 2), B( 3), B( 3), B( 2), B( 1), B( 0) },
+      { B( 0), B( 1), B( 2), B( 3), B( 3), B( 2), B( 1), B( 0),
+        B( 4), B( 5), B( 6), B( 7), B( 7), B( 6), B( 5), B( 4),
+        B( 8), B( 9), B(10), B(11), B(11), B(10), B( 9), B( 8),
+        B(12), B(13), B(14), B(15), B(15), B(14), B(13), B(12),
+        B(16), B(17), B(18), B(19), B(19), B(18), B(17), B(16),
+        B(20), B(21), B(22), B(23), B(23), B(22), B(21), B(20),
+        B(24), B(25), B(26), B(27), B(27), B(26), B(25), B(24),
+        B(28), B(29), B(30), B(31), B(31), B(30), B(29), B(28) }
     };
+    // clang-format on
+#undef B
+    // clang-format off
+    // Orient a square according to perspective (rotates by 180 for black)
+    static constexpr int OrientTBL[COLOR_NB][SQUARE_NB] = {
+      { SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1,
+        SQ_H1, SQ_H1, SQ_H1, SQ_H1, SQ_A1, SQ_A1, SQ_A1, SQ_A1 },
+      { SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8,
+        SQ_H8, SQ_H8, SQ_H8, SQ_H8, SQ_A8, SQ_A8, SQ_A8, SQ_A8 }
+    };
+    // clang-format on
 
     // Maximum number of simultaneously active features.
     static constexpr IndexType MaxActiveDimensions = 32;
-    using IndexList = ValueList<IndexType, MaxActiveDimensions>;
+    using IndexList                                = ValueList<IndexType, MaxActiveDimensions>;
 
     // Get a list of indices for active features
-    static void append_active_indices(
+    template<Color Perspective>
-      const Position& pos,
+    static void append_active_indices(const Position& pos, IndexList& active);
-      Color perspective,
-      IndexList& active);
 
     // Get a list of indices for recently changed features
-    static void append_changed_indices(
+    template<Color Perspective>
-      Square ksq,
+    static void
-      const DirtyPiece& dp,
+    append_changed_indices(Square ksq, const DirtyPiece& dp, IndexList& removed, IndexList& added);
-      Color perspective,
-      IndexList& removed,
-      IndexList& added
-    );
 
     // Returns the cost of updating one perspective, the most costly one.
     // Assumes no refresh needed.
     static int update_cost(const StateInfo* st);
     static int refresh_cost(const Position& pos);
 
-    // Returns whether the change stored in this StateInfo means that
+    // Returns whether the change stored in this StateInfo means
-    // a full accumulator refresh is required.
+    // that a full accumulator refresh is required.
     static bool requires_refresh(const StateInfo* st, Color perspective);
-  };
+};
 
 }  // namespace Stockfish::Eval::NNUE::Features
 
-#endif // #ifndef NNUE_FEATURES_HALF_KA_V2_HM_H_INCLUDED
+#endif  // #ifndef NNUE_FEATURES_HALF_KA_V2_HM_H_INCLUDED

src/nnue/layers/affine_transform.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,34 +21,18 @@
 #ifndef NNUE_LAYERS_AFFINE_TRANSFORM_H_INCLUDED
 #define NNUE_LAYERS_AFFINE_TRANSFORM_H_INCLUDED
 
+#include <cstdint>
 #include <iostream>
-#include <algorithm>
+
-#include <type_traits>
 #include "../nnue_common.h"
-#include "../../simd.h"
+#include "simd.h"
 
 /*
   This file contains the definition for a fully connected layer (aka affine transform).
-  Two approaches are employed, depending on the sizes of the transform.
 
-  Approach 1:
-    - used when the PaddedInputDimensions >= 128
-    - uses AVX512 if possible
-    - processes inputs in batches of 2*InputSimdWidth
-      - so in batches of 128 for AVX512
-    - the weight blocks of size InputSimdWidth are transposed such that
-      access is sequential
-    - N columns of the weight matrix are processed a time, where N
-      depends on the architecture (the amount of registers)
-    - accumulate + hadd is used
-
-  Approach 2:
-    - used when the PaddedInputDimensions < 128
-    - does not use AVX512
     - expected use-case is for when PaddedInputDimensions == 32 and InputDimensions <= 32.
       - that's why AVX512 is hard to implement
     - expected use-case is small layers
-      - not optimized as well as the approach 1
     - inputs are processed in chunks of 4, weights are respectively transposed
     - accumulation happens directly to int32s
 */
@@ -56,111 +40,104 @@
 namespace Stockfish::Eval::NNUE::Layers {
 
 // Fallback implementation for older/other architectures.
-// Identical for both approaches. Requires the input to be padded to at least 16 values.
+// Requires the input to be padded to at least 16 values.
 #if !defined(USE_SSSE3)
-  template <IndexType InputDimensions, IndexType PaddedInputDimensions, IndexType OutputDimensions>
+template<IndexType InputDimensions, IndexType PaddedInputDimensions, IndexType OutputDimensions>
-  static void affine_transform_non_ssse3(std::int32_t* output, const std::int8_t* weights, const std::int32_t* biases, const std::uint8_t* input)
+static void affine_transform_non_ssse3(std::int32_t*       output,
-  {
+                                       const std::int8_t*  weights,
-# if defined(USE_SSE2)
+                                       const std::int32_t* biases,
+                                       const std::uint8_t* input) {
+    #if defined(USE_SSE2) || defined(USE_NEON_DOTPROD) || defined(USE_NEON)
+        #if defined(USE_SSE2)
     // At least a multiple of 16, with SSE2.
-    constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 16) / 16;
+    constexpr IndexType NumChunks   = ceil_to_multiple<IndexType>(InputDimensions, 16) / 16;
-    const __m128i Zeros = _mm_setzero_si128();
+    const __m128i       Zeros       = _mm_setzero_si128();
-    const auto inputVector = reinterpret_cast<const __m128i*>(input);
+    const auto          inputVector = reinterpret_cast<const __m128i*>(input);
 
-# elif defined(USE_MMX)
+        #elif defined(USE_NEON_DOTPROD)
-    constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 8) / 8;
+    constexpr IndexType NumChunks   = ceil_to_multiple<IndexType>(InputDimensions, 16) / 16;
-    const __m64 Zeros = _mm_setzero_si64();
+    const auto          inputVector = reinterpret_cast<const int8x16_t*>(input);
-    const auto inputVector = reinterpret_cast<const __m64*>(input);
 
-# elif defined(USE_NEON)
+        #elif defined(USE_NEON)
-    constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 16) / 16;
+    constexpr IndexType NumChunks   = ceil_to_multiple<IndexType>(InputDimensions, 16) / 16;
-    const auto inputVector = reinterpret_cast<const int8x8_t*>(input);
+    const auto          inputVector = reinterpret_cast<const int8x8_t*>(input);
-# endif
+        #endif
 
-    for (IndexType i = 0; i < OutputDimensions; ++i) {
+    for (IndexType i = 0; i < OutputDimensions; ++i)
-      const IndexType offset = i * PaddedInputDimensions;
+    {
+        const IndexType offset = i * PaddedInputDimensions;
 
-# if defined(USE_SSE2)
+        #if defined(USE_SSE2)
-      __m128i sumLo = _mm_cvtsi32_si128(biases[i]);
+        __m128i    sumLo = _mm_cvtsi32_si128(biases[i]);
-      __m128i sumHi = Zeros;
+        __m128i    sumHi = Zeros;
-      const auto row = reinterpret_cast<const __m128i*>(&weights[offset]);
+        const auto row   = reinterpret_cast<const __m128i*>(&weights[offset]);
-      for (IndexType j = 0; j < NumChunks; ++j) {
+        for (IndexType j = 0; j < NumChunks; ++j)
-        __m128i row_j = _mm_load_si128(&row[j]);
+        {
-        __m128i input_j = _mm_load_si128(&inputVector[j]);
+            __m128i row_j           = _mm_load_si128(&row[j]);
-        __m128i extendedRowLo = _mm_srai_epi16(_mm_unpacklo_epi8(row_j, row_j), 8);
+            __m128i input_j         = _mm_load_si128(&inputVector[j]);
-        __m128i extendedRowHi = _mm_srai_epi16(_mm_unpackhi_epi8(row_j, row_j), 8);
+            __m128i extendedRowLo   = _mm_srai_epi16(_mm_unpacklo_epi8(row_j, row_j), 8);
-        __m128i extendedInputLo = _mm_unpacklo_epi8(input_j, Zeros);
+            __m128i extendedRowHi   = _mm_srai_epi16(_mm_unpackhi_epi8(row_j, row_j), 8);
-        __m128i extendedInputHi = _mm_unpackhi_epi8(input_j, Zeros);
+            __m128i extendedInputLo = _mm_unpacklo_epi8(input_j, Zeros);
-        __m128i productLo = _mm_madd_epi16(extendedRowLo, extendedInputLo);
+            __m128i extendedInputHi = _mm_unpackhi_epi8(input_j, Zeros);
-        __m128i productHi = _mm_madd_epi16(extendedRowHi, extendedInputHi);
+            __m128i productLo       = _mm_madd_epi16(extendedRowLo, extendedInputLo);
-        sumLo = _mm_add_epi32(sumLo, productLo);
+            __m128i productHi       = _mm_madd_epi16(extendedRowHi, extendedInputHi);
-        sumHi = _mm_add_epi32(sumHi, productHi);
+            sumLo                   = _mm_add_epi32(sumLo, productLo);
-      }
+            sumHi                   = _mm_add_epi32(sumHi, productHi);
-      __m128i sum = _mm_add_epi32(sumLo, sumHi);
+        }
-      __m128i sumHigh_64 = _mm_shuffle_epi32(sum, _MM_SHUFFLE(1, 0, 3, 2));
+        __m128i sum           = _mm_add_epi32(sumLo, sumHi);
-      sum = _mm_add_epi32(sum, sumHigh_64);
+        __m128i sumHigh_64    = _mm_shuffle_epi32(sum, _MM_SHUFFLE(1, 0, 3, 2));
-      __m128i sum_second_32 = _mm_shufflelo_epi16(sum, _MM_SHUFFLE(1, 0, 3, 2));
+        sum                   = _mm_add_epi32(sum, sumHigh_64);
-      sum = _mm_add_epi32(sum, sum_second_32);
+        __m128i sum_second_32 = _mm_shufflelo_epi16(sum, _MM_SHUFFLE(1, 0, 3, 2));
-      output[i] = _mm_cvtsi128_si32(sum);
+        sum                   = _mm_add_epi32(sum, sum_second_32);
+        output[i]             = _mm_cvtsi128_si32(sum);
 
-# elif defined(USE_MMX)
+        #elif defined(USE_NEON_DOTPROD)
-      __m64 sumLo = _mm_cvtsi32_si64(biases[i]);
+        int32x4_t  sum = {biases[i]};
-      __m64 sumHi = Zeros;
+        const auto row = reinterpret_cast<const int8x16_t*>(&weights[offset]);
-      const auto row = reinterpret_cast<const __m64*>(&weights[offset]);
+        for (IndexType j = 0; j < NumChunks; ++j)
-      for (IndexType j = 0; j < NumChunks; ++j) {
+        {
-        __m64 row_j = row[j];
+            sum = vdotq_s32(sum, inputVector[j], row[j]);
-        __m64 input_j = inputVector[j];
+        }
-        __m64 extendedRowLo = _mm_srai_pi16(_mm_unpacklo_pi8(row_j, row_j), 8);
+        output[i] = vaddvq_s32(sum);
-        __m64 extendedRowHi = _mm_srai_pi16(_mm_unpackhi_pi8(row_j, row_j), 8);
-        __m64 extendedInputLo = _mm_unpacklo_pi8(input_j, Zeros);
-        __m64 extendedInputHi = _mm_unpackhi_pi8(input_j, Zeros);
-        __m64 productLo = _mm_madd_pi16(extendedRowLo, extendedInputLo);
-        __m64 productHi = _mm_madd_pi16(extendedRowHi, extendedInputHi);
-        sumLo = _mm_add_pi32(sumLo, productLo);
-        sumHi = _mm_add_pi32(sumHi, productHi);
-      }
-      __m64 sum = _mm_add_pi32(sumLo, sumHi);
-      sum = _mm_add_pi32(sum, _mm_unpackhi_pi32(sum, sum));
-      output[i] = _mm_cvtsi64_si32(sum);
 
-# elif defined(USE_NEON)
+        #elif defined(USE_NEON)
-      int32x4_t sum = {biases[i]};
+        int32x4_t  sum = {biases[i]};
-      const auto row = reinterpret_cast<const int8x8_t*>(&weights[offset]);
+        const auto row = reinterpret_cast<const int8x8_t*>(&weights[offset]);
-      for (IndexType j = 0; j < NumChunks; ++j) {
+        for (IndexType j = 0; j < NumChunks; ++j)
-        int16x8_t product = vmull_s8(inputVector[j * 2], row[j * 2]);
+        {
-        product = vmlal_s8(product, inputVector[j * 2 + 1], row[j * 2 + 1]);
+            int16x8_t product = vmull_s8(inputVector[j * 2], row[j * 2]);
-        sum = vpadalq_s16(sum, product);
+            product           = vmlal_s8(product, inputVector[j * 2 + 1], row[j * 2 + 1]);
-      }
+            sum               = vpadalq_s16(sum, product);
-      output[i] = sum[0] + sum[1] + sum[2] + sum[3];
+        }
+        output[i] = sum[0] + sum[1] + sum[2] + sum[3];
 
-# else
+        #endif
-      std::int32_t sum = biases[i];
-      for (IndexType j = 0; j < InputDimensions; ++j) {
-        sum += weights[offset + j] * input[j];
-      }
-      output[i] = sum;
-# endif
     }
+    #else
+    std::memcpy(output, biases, sizeof(std::int32_t) * OutputDimensions);
 
-# if defined(USE_MMX)
+    // Traverse weights in transpose order to take advantage of input sparsity
-    _mm_empty();
+    for (IndexType i = 0; i < InputDimensions; ++i)
-# endif
+        if (input[i])
-  }
+        {
+            const std::int8_t* w  = &weights[i];
+            const int          in = input[i];
+            for (IndexType j = 0; j < OutputDimensions; ++j)
+                output[j] += w[j * PaddedInputDimensions] * in;
+        }
+    #endif
+}
 #endif
 
-  template <IndexType InDims, IndexType OutDims, typename Enabled = void>
+template<IndexType InDims, IndexType OutDims>
-  class AffineTransform;
+class AffineTransform {
-
-  // A specialization for large inputs.
-  template <IndexType InDims, IndexType OutDims>
-  class AffineTransform<InDims, OutDims, std::enable_if_t<(ceil_to_multiple<IndexType>(InDims, MaxSimdWidth) >= 2*64)>> {
    public:
     // Input/output type
-    using InputType = std::uint8_t;
+    using InputType  = std::uint8_t;
     using OutputType = std::int32_t;
 
     // Number of input/output dimensions
-    static constexpr IndexType InputDimensions = InDims;
+    static constexpr IndexType InputDimensions  = InDims;
     static constexpr IndexType OutputDimensions = OutDims;
 
     static constexpr IndexType PaddedInputDimensions =
@@ -170,370 +147,162 @@ namespace Stockfish::Eval::NNUE::Layers {
 
     using OutputBuffer = OutputType[PaddedOutputDimensions];
 
-    static_assert(PaddedInputDimensions >= 128, "Something went wrong. This specialization should not have been chosen.");
-
-#if defined (USE_AVX512)
-    static constexpr const IndexType InputSimdWidth = 64;
-    static constexpr const IndexType MaxNumOutputRegs = 16;
-#elif defined (USE_AVX2)
-    static constexpr const IndexType InputSimdWidth = 32;
-    static constexpr const IndexType MaxNumOutputRegs = 8;
-#elif defined (USE_SSSE3)
-    static constexpr const IndexType InputSimdWidth = 16;
-    static constexpr const IndexType MaxNumOutputRegs = 8;
-#elif defined (USE_NEON)
-    static constexpr const IndexType InputSimdWidth = 8;
-    static constexpr const IndexType MaxNumOutputRegs = 8;
-#else
-    // The fallback implementation will not have permuted weights.
-    // We define these to avoid a lot of ifdefs later.
-    static constexpr const IndexType InputSimdWidth = 1;
-    static constexpr const IndexType MaxNumOutputRegs = 1;
-#endif
-
-    // A big block is a region in the weight matrix of the size [PaddedInputDimensions, NumOutputRegs].
-    // A small block is a region of size [InputSimdWidth, 1]
-
-    static constexpr const IndexType NumOutputRegs = std::min(MaxNumOutputRegs, OutputDimensions);
-    static constexpr const IndexType SmallBlockSize = InputSimdWidth;
-    static constexpr const IndexType BigBlockSize = NumOutputRegs * PaddedInputDimensions;
-    static constexpr const IndexType NumSmallBlocksInBigBlock = BigBlockSize / SmallBlockSize;
-    static constexpr const IndexType NumSmallBlocksPerOutput = PaddedInputDimensions / SmallBlockSize;
-    static constexpr const IndexType NumBigBlocks = OutputDimensions / NumOutputRegs;
-
-    static_assert(OutputDimensions % NumOutputRegs == 0);
-
     // Hash value embedded in the evaluation file
     static constexpr std::uint32_t get_hash_value(std::uint32_t prevHash) {
-      std::uint32_t hashValue = 0xCC03DAE4u;
+        std::uint32_t hashValue = 0xCC03DAE4u;
-      hashValue += OutputDimensions;
+        hashValue += OutputDimensions;
-      hashValue ^= prevHash >> 1;
+        hashValue ^= prevHash >> 1;
-      hashValue ^= prevHash << 31;
+        hashValue ^= prevHash << 31;
-      return hashValue;
+        return hashValue;
     }
 
-    /*
+    static constexpr IndexType get_weight_index_scrambled(IndexType i) {
-      Transposes the small blocks within a block.
+        return (i / 4) % (PaddedInputDimensions / 4) * OutputDimensions * 4
-      Effectively means that weights can be traversed sequentially during inference.
+             + i / PaddedInputDimensions * 4 + i % 4;
-    */
-    static IndexType get_weight_index(IndexType i)
-    {
-      const IndexType smallBlock = (i / SmallBlockSize) % NumSmallBlocksInBigBlock;
-      const IndexType smallBlockCol = smallBlock / NumSmallBlocksPerOutput;
-      const IndexType smallBlockRow = smallBlock % NumSmallBlocksPerOutput;
-      const IndexType bigBlock   = i / BigBlockSize;
-      const IndexType rest       = i % SmallBlockSize;
-
-      const IndexType idx =
-          bigBlock * BigBlockSize
-        + smallBlockRow * SmallBlockSize * NumOutputRegs
-        + smallBlockCol * SmallBlockSize
-        + rest;
-
-      return idx;
     }
 
-    // Read network parameters
+    static constexpr IndexType get_weight_index(IndexType i) {
-    bool read_parameters(std::istream& stream) {
+#if defined(USE_SSSE3)
-      for (IndexType i = 0; i < OutputDimensions; ++i)
+        return get_weight_index_scrambled(i);
-        biases[i] = read_little_endian<BiasType>(stream);
-
-      for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
-        weights[get_weight_index(i)] = read_little_endian<WeightType>(stream);
-
-      return !stream.fail();
-    }
-
-    // Write network parameters
-    bool write_parameters(std::ostream& stream) const {
-      for (IndexType i = 0; i < OutputDimensions; ++i)
-          write_little_endian<BiasType>(stream, biases[i]);
-
-      for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
-        write_little_endian<WeightType>(stream, weights[get_weight_index(i)]);
-
-      return !stream.fail();
-    }
-
-    // Forward propagation
-    const OutputType* propagate(
-        const InputType* input, OutputType* output) const {
-
-#if defined (USE_AVX512)
-      using acc_vec_t = __m512i;
-      using bias_vec_t = __m128i;
-      using weight_vec_t = __m512i;
-      using in_vec_t = __m512i;
-      #define vec_zero _mm512_setzero_si512()
-      #define vec_add_dpbusd_32x2 Simd::m512_add_dpbusd_epi32x2
-      #define vec_hadd Simd::m512_hadd
-      #define vec_haddx4 Simd::m512_haddx4
-#elif defined (USE_AVX2)
-      using acc_vec_t = __m256i;
-      using bias_vec_t = __m128i;
-      using weight_vec_t = __m256i;
-      using in_vec_t = __m256i;
-      #define vec_zero _mm256_setzero_si256()
-      #define vec_add_dpbusd_32x2 Simd::m256_add_dpbusd_epi32x2
-      #define vec_hadd Simd::m256_hadd
-      #define vec_haddx4 Simd::m256_haddx4
-#elif defined (USE_SSSE3)
-      using acc_vec_t = __m128i;
-      using bias_vec_t = __m128i;
-      using weight_vec_t = __m128i;
-      using in_vec_t = __m128i;
-      #define vec_zero _mm_setzero_si128()
-      #define vec_add_dpbusd_32x2 Simd::m128_add_dpbusd_epi32x2
-      #define vec_hadd Simd::m128_hadd
-      #define vec_haddx4 Simd::m128_haddx4
-#elif defined (USE_NEON)
-      using acc_vec_t = int32x4_t;
-      using bias_vec_t = int32x4_t;
-      using weight_vec_t = int8x8_t;
-      using in_vec_t = int8x8_t;
-      #define vec_zero {0}
-      #define vec_add_dpbusd_32x2 Simd::neon_m128_add_dpbusd_epi32x2
-      #define vec_hadd Simd::neon_m128_hadd
-      #define vec_haddx4 Simd::neon_m128_haddx4
-#endif
-
-#if defined (USE_SSSE3) || defined (USE_NEON)
-      const in_vec_t* invec = reinterpret_cast<const in_vec_t*>(input);
-
-      // Perform accumulation to registers for each big block
-      for (IndexType bigBlock = 0; bigBlock < NumBigBlocks; ++bigBlock)
-      {
-        acc_vec_t acc[NumOutputRegs] = { vec_zero };
-
-        // Each big block has NumOutputRegs small blocks in each "row", one per register.
-        // We process two small blocks at a time to save on one addition without VNNI.
-        for (IndexType smallBlock = 0; smallBlock < NumSmallBlocksPerOutput; smallBlock += 2)
-        {
-          const weight_vec_t* weightvec =
-            reinterpret_cast<const weight_vec_t*>(
-                weights
-              + bigBlock * BigBlockSize
-              + smallBlock * SmallBlockSize * NumOutputRegs);
-
-          const in_vec_t in0 = invec[smallBlock + 0];
-          const in_vec_t in1 = invec[smallBlock + 1];
-
-          for (IndexType k = 0; k < NumOutputRegs; ++k)
-            vec_add_dpbusd_32x2(acc[k], in0, weightvec[k], in1, weightvec[k + NumOutputRegs]);
-        }
-
-        // Horizontally add all accumulators.
-        if constexpr (NumOutputRegs % 4 == 0)
-        {
-          bias_vec_t* outputvec = reinterpret_cast<bias_vec_t*>(output);
-          const bias_vec_t* biasvec = reinterpret_cast<const bias_vec_t*>(biases);
-
-          for (IndexType k = 0; k < NumOutputRegs; k += 4)
-          {
-            const IndexType idx = (bigBlock * NumOutputRegs + k) / 4;
-            outputvec[idx] = vec_haddx4(acc[k+0], acc[k+1], acc[k+2], acc[k+3], biasvec[idx]);
-          }
-        }
-        else
-        {
-          for (IndexType k = 0; k < NumOutputRegs; ++k)
-          {
-            const IndexType idx = (bigBlock * NumOutputRegs + k);
-            output[idx] = vec_hadd(acc[k], biases[idx]);
-          }
-        }
-      }
-
-# undef vec_zero
-# undef vec_add_dpbusd_32x2
-# undef vec_hadd
-# undef vec_haddx4
 #else
-      // Use old implementation for the other architectures.
+        return i;
-      affine_transform_non_ssse3<
-        InputDimensions,
-        PaddedInputDimensions,
-        OutputDimensions>(output, weights, biases, input);
-
-#endif
-
-      return output;
-    }
-
-   private:
-    using BiasType = OutputType;
-    using WeightType = std::int8_t;
-
-    alignas(CacheLineSize) BiasType biases[OutputDimensions];
-    alignas(CacheLineSize) WeightType weights[OutputDimensions * PaddedInputDimensions];
-  };
-
-  template <IndexType InDims, IndexType OutDims>
-  class AffineTransform<InDims, OutDims, std::enable_if_t<(ceil_to_multiple<IndexType>(InDims, MaxSimdWidth) < 2*64)>> {
-   public:
-    // Input/output type
-    // Input/output type
-    using InputType = std::uint8_t;
-    using OutputType = std::int32_t;
-
-    // Number of input/output dimensions
-    static constexpr IndexType InputDimensions = InDims;
-    static constexpr IndexType OutputDimensions = OutDims;
-
-    static constexpr IndexType PaddedInputDimensions =
-      ceil_to_multiple<IndexType>(InputDimensions, MaxSimdWidth);
-    static constexpr IndexType PaddedOutputDimensions =
-      ceil_to_multiple<IndexType>(OutputDimensions, MaxSimdWidth);
-
-    using OutputBuffer = OutputType[PaddedOutputDimensions];
-
-    static_assert(PaddedInputDimensions < 128, "Something went wrong. This specialization should not have been chosen.");
-
-#if defined (USE_SSSE3)
-    static constexpr const IndexType OutputSimdWidth = SimdWidth / 4;
-    static constexpr const IndexType InputSimdWidth = SimdWidth;
-#endif
-
-    // Hash value embedded in the evaluation file
-    static constexpr std::uint32_t get_hash_value(std::uint32_t prevHash) {
-      std::uint32_t hashValue = 0xCC03DAE4u;
-      hashValue += OutputDimensions;
-      hashValue ^= prevHash >> 1;
-      hashValue ^= prevHash << 31;
-      return hashValue;
-    }
-
-    static IndexType get_weight_index_scrambled(IndexType i)
-    {
-      return
-        (i / 4) % (PaddedInputDimensions / 4) * OutputDimensions * 4 +
-        i / PaddedInputDimensions * 4 +
-        i % 4;
-    }
-
-    static IndexType get_weight_index(IndexType i)
-    {
-#if defined (USE_SSSE3)
-      return get_weight_index_scrambled(i);
-#else
-      return i;
 #endif
     }
 
     // Read network parameters
     bool read_parameters(std::istream& stream) {
-      for (IndexType i = 0; i < OutputDimensions; ++i)
+        read_little_endian<BiasType>(stream, biases, OutputDimensions);
-        biases[i] = read_little_endian<BiasType>(stream);
+        for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
-      for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
+            weights[get_weight_index(i)] = read_little_endian<WeightType>(stream);
-        weights[get_weight_index(i)] = read_little_endian<WeightType>(stream);
 
-      return !stream.fail();
+        return !stream.fail();
     }
 
     // Write network parameters
     bool write_parameters(std::ostream& stream) const {
-      for (IndexType i = 0; i < OutputDimensions; ++i)
+        write_little_endian<BiasType>(stream, biases, OutputDimensions);
-        write_little_endian<BiasType>(stream, biases[i]);
 
-      for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
+        for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
-        write_little_endian<WeightType>(stream, weights[get_weight_index(i)]);
+            write_little_endian<WeightType>(stream, weights[get_weight_index(i)]);
 
-      return !stream.fail();
+        return !stream.fail();
     }
     // Forward propagation
-    const OutputType* propagate(
+    void propagate(const InputType* input, OutputType* output) const {
-        const InputType* input, OutputType* output) const {
 
-#if defined (USE_AVX2)
+#if defined(USE_SSSE3)
-      using vec_t = __m256i;
-      #define vec_setzero _mm256_setzero_si256
-      #define vec_set_32 _mm256_set1_epi32
-      #define vec_add_dpbusd_32 Simd::m256_add_dpbusd_epi32
-      #define vec_add_dpbusd_32x2 Simd::m256_add_dpbusd_epi32x2
-      #define vec_add_dpbusd_32x4 Simd::m256_add_dpbusd_epi32x4
-      #define vec_hadd Simd::m256_hadd
-      #define vec_haddx4 Simd::m256_haddx4
-#elif defined (USE_SSSE3)
-      using vec_t = __m128i;
-      #define vec_setzero _mm_setzero_si128
-      #define vec_set_32 _mm_set1_epi32
-      #define vec_add_dpbusd_32 Simd::m128_add_dpbusd_epi32
-      #define vec_add_dpbusd_32x2 Simd::m128_add_dpbusd_epi32x2
-      #define vec_add_dpbusd_32x4 Simd::m128_add_dpbusd_epi32x4
-      #define vec_hadd Simd::m128_hadd
-      #define vec_haddx4 Simd::m128_haddx4
-#endif
 
-#if defined (USE_SSSE3)
+        if constexpr (OutputDimensions > 1)
-      const auto inputVector = reinterpret_cast<const vec_t*>(input);
-
-      static_assert(OutputDimensions % OutputSimdWidth == 0 || OutputDimensions == 1);
-
-      if constexpr (OutputDimensions % OutputSimdWidth == 0)
-      {
-        constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 8) / 4;
-        constexpr IndexType NumRegs = OutputDimensions / OutputSimdWidth;
-
-        const auto input32 = reinterpret_cast<const std::int32_t*>(input);
-        const vec_t* biasvec = reinterpret_cast<const vec_t*>(biases);
-        vec_t acc[NumRegs];
-        for (IndexType k = 0; k < NumRegs; ++k)
-          acc[k] = biasvec[k];
-
-        for (IndexType i = 0; i < NumChunks; i += 2)
         {
-          const vec_t in0 = vec_set_32(input32[i + 0]);
+
-          const vec_t in1 = vec_set_32(input32[i + 1]);
+    #if defined(USE_AVX512)
-          const auto col0 = reinterpret_cast<const vec_t*>(&weights[(i + 0) * OutputDimensions * 4]);
+            using vec_t = __m512i;
-          const auto col1 = reinterpret_cast<const vec_t*>(&weights[(i + 1) * OutputDimensions * 4]);
+        #define vec_setzero _mm512_setzero_si512
-          for (IndexType k = 0; k < NumRegs; ++k)
+        #define vec_set_32 _mm512_set1_epi32
-            vec_add_dpbusd_32x2(acc[k], in0, col0[k], in1, col1[k]);
+        #define vec_add_dpbusd_32 Simd::m512_add_dpbusd_epi32
+        #define vec_hadd Simd::m512_hadd
+    #elif defined(USE_AVX2)
+            using vec_t = __m256i;
+        #define vec_setzero _mm256_setzero_si256
+        #define vec_set_32 _mm256_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m256_add_dpbusd_epi32
+        #define vec_hadd Simd::m256_hadd
+    #elif defined(USE_SSSE3)
+            using vec_t = __m128i;
+        #define vec_setzero _mm_setzero_si128
+        #define vec_set_32 _mm_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m128_add_dpbusd_epi32
+        #define vec_hadd Simd::m128_hadd
+    #endif
+
+            static constexpr IndexType OutputSimdWidth = sizeof(vec_t) / sizeof(OutputType);
+
+            static_assert(OutputDimensions % OutputSimdWidth == 0);
+
+            constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 8) / 4;
+            constexpr IndexType NumRegs   = OutputDimensions / OutputSimdWidth;
+
+            const auto   input32 = reinterpret_cast<const std::int32_t*>(input);
+            const vec_t* biasvec = reinterpret_cast<const vec_t*>(biases);
+            vec_t        acc[NumRegs];
+            for (IndexType k = 0; k < NumRegs; ++k)
+                acc[k] = biasvec[k];
+
+            for (IndexType i = 0; i < NumChunks; ++i)
+            {
+                const vec_t in0 = vec_set_32(input32[i]);
+                const auto  col0 =
+                  reinterpret_cast<const vec_t*>(&weights[i * OutputDimensions * 4]);
+
+                for (IndexType k = 0; k < NumRegs; ++k)
+                    vec_add_dpbusd_32(acc[k], in0, col0[k]);
+            }
+
+            vec_t* outptr = reinterpret_cast<vec_t*>(output);
+            for (IndexType k = 0; k < NumRegs; ++k)
+                outptr[k] = acc[k];
+
+    #undef vec_setzero
+    #undef vec_set_32
+    #undef vec_add_dpbusd_32
+    #undef vec_hadd
         }
-
+        else if constexpr (OutputDimensions == 1)
-        vec_t* outptr = reinterpret_cast<vec_t*>(output);
-        for (IndexType k = 0; k < NumRegs; ++k)
-          outptr[k] = acc[k];
-      }
-      else if constexpr (OutputDimensions == 1)
-      {
-        constexpr IndexType NumChunks = PaddedInputDimensions / SimdWidth;
-        vec_t sum0 = vec_setzero();
-        const auto row0 = reinterpret_cast<const vec_t*>(&weights[0]);
-
-        for (int j = 0; j < (int)NumChunks; ++j)
         {
-          const vec_t in = inputVector[j];
-          vec_add_dpbusd_32(sum0, in, row0[j]);
-        }
-        output[0] = vec_hadd(sum0, biases[0]);
-      }
 
-# undef vec_setzero
+    // We cannot use AVX512 for the last layer because there are only 32 inputs
-# undef vec_set_32
+    // and the buffer is not padded to 64 elements.
-# undef vec_add_dpbusd_32
+    #if defined(USE_AVX2)
-# undef vec_add_dpbusd_32x2
+            using vec_t = __m256i;
-# undef vec_add_dpbusd_32x4
+        #define vec_setzero _mm256_setzero_si256
-# undef vec_hadd
+        #define vec_set_32 _mm256_set1_epi32
-# undef vec_haddx4
+        #define vec_add_dpbusd_32 Simd::m256_add_dpbusd_epi32
+        #define vec_hadd Simd::m256_hadd
+    #elif defined(USE_SSSE3)
+            using vec_t = __m128i;
+        #define vec_setzero _mm_setzero_si128
+        #define vec_set_32 _mm_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m128_add_dpbusd_epi32
+        #define vec_hadd Simd::m128_hadd
+    #endif
+
+            const auto inputVector = reinterpret_cast<const vec_t*>(input);
+
+            static constexpr IndexType InputSimdWidth = sizeof(vec_t) / sizeof(InputType);
+
+            static_assert(PaddedInputDimensions % InputSimdWidth == 0);
+
+            constexpr IndexType NumChunks = PaddedInputDimensions / InputSimdWidth;
+            vec_t               sum0      = vec_setzero();
+            const auto          row0      = reinterpret_cast<const vec_t*>(&weights[0]);
+
+            for (int j = 0; j < int(NumChunks); ++j)
+            {
+                const vec_t in = inputVector[j];
+                vec_add_dpbusd_32(sum0, in, row0[j]);
+            }
+            output[0] = vec_hadd(sum0, biases[0]);
+
+    #undef vec_setzero
+    #undef vec_set_32
+    #undef vec_add_dpbusd_32
+    #undef vec_hadd
+        }
 #else
-      // Use old implementation for the other architectures.
+        // Use old implementation for the other architectures.
-      affine_transform_non_ssse3<
+        affine_transform_non_ssse3<InputDimensions, PaddedInputDimensions, OutputDimensions>(
-        InputDimensions,
+          output, weights, biases, input);
-        PaddedInputDimensions,
-        OutputDimensions>(output, weights, biases, input);
 #endif
-
-      return output;
     }
 
    private:
-    using BiasType = OutputType;
+    using BiasType   = OutputType;
     using WeightType = std::int8_t;
 
     alignas(CacheLineSize) BiasType biases[OutputDimensions];
     alignas(CacheLineSize) WeightType weights[OutputDimensions * PaddedInputDimensions];
-  };
+};
 
 }  // namespace Stockfish::Eval::NNUE::Layers
 
-#endif // #ifndef NNUE_LAYERS_AFFINE_TRANSFORM_H_INCLUDED
+#endif  // #ifndef NNUE_LAYERS_AFFINE_TRANSFORM_H_INCLUDED

src/nnue/layers/affine_transform_sparse_input.h

@@ -0,0 +1,278 @@
+/*
+  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
+
+  Stockfish is free software: you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation, either version 3 of the License, or
+  (at your option) any later version.
+
+  Stockfish is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+// Definition of layer AffineTransformSparseInput of NNUE evaluation function
+
+#ifndef NNUE_LAYERS_AFFINE_TRANSFORM_SPARSE_INPUT_H_INCLUDED
+#define NNUE_LAYERS_AFFINE_TRANSFORM_SPARSE_INPUT_H_INCLUDED
+
+#include <algorithm>
+#include <array>
+#include <cstdint>
+#include <iostream>
+
+#include "../../bitboard.h"
+#include "../nnue_common.h"
+#include "affine_transform.h"
+#include "simd.h"
+
+/*
+  This file contains the definition for a fully connected layer (aka affine transform) with block sparse input.
+*/
+
+namespace Stockfish::Eval::NNUE::Layers {
+
+#if (USE_SSSE3 | (USE_NEON >= 8))
+alignas(CacheLineSize) static inline const
+  std::array<std::array<std::uint16_t, 8>, 256> lookup_indices = []() {
+      std::array<std::array<std::uint16_t, 8>, 256> v{};
+      for (unsigned i = 0; i < 256; ++i)
+      {
+          std::uint64_t j = i, k = 0;
+          while (j)
+              v[i][k++] = pop_lsb(j);
+      }
+      return v;
+  }();
+
+// Find indices of nonzero numbers in an int32_t array
+template<const IndexType InputDimensions>
+void find_nnz(const std::int32_t* input, std::uint16_t* out, IndexType& count_out) {
+    #if defined(USE_SSSE3)
+        #if defined(USE_AVX512)
+    using vec_t = __m512i;
+            #define vec_nnz(a) _mm512_cmpgt_epi32_mask(a, _mm512_setzero_si512())
+        #elif defined(USE_AVX2)
+    using vec_t = __m256i;
+            #if defined(USE_VNNI) && !defined(USE_AVXVNNI)
+                #define vec_nnz(a) _mm256_cmpgt_epi32_mask(a, _mm256_setzero_si256())
+            #else
+                #define vec_nnz(a) \
+                    _mm256_movemask_ps( \
+                      _mm256_castsi256_ps(_mm256_cmpgt_epi32(a, _mm256_setzero_si256())))
+            #endif
+        #elif defined(USE_SSSE3)
+    using vec_t = __m128i;
+            #define vec_nnz(a) \
+                _mm_movemask_ps(_mm_castsi128_ps(_mm_cmpgt_epi32(a, _mm_setzero_si128())))
+        #endif
+    using vec128_t = __m128i;
+        #define vec128_zero _mm_setzero_si128()
+        #define vec128_set_16(a) _mm_set1_epi16(a)
+        #define vec128_load(a) _mm_load_si128(a)
+        #define vec128_storeu(a, b) _mm_storeu_si128(a, b)
+        #define vec128_add(a, b) _mm_add_epi16(a, b)
+    #elif defined(USE_NEON)
+    using vec_t                        = uint32x4_t;
+    static const std::uint32_t Mask[4] = {1, 2, 4, 8};
+        #define vec_nnz(a) vaddvq_u32(vandq_u32(vtstq_u32(a, a), vld1q_u32(Mask)))
+    using vec128_t                     = uint16x8_t;
+        #define vec128_zero vdupq_n_u16(0)
+        #define vec128_set_16(a) vdupq_n_u16(a)
+        #define vec128_load(a) vld1q_u16(reinterpret_cast<const std::uint16_t*>(a))
+        #define vec128_storeu(a, b) vst1q_u16(reinterpret_cast<std::uint16_t*>(a), b)
+        #define vec128_add(a, b) vaddq_u16(a, b)
+    #endif
+    constexpr IndexType InputSimdWidth = sizeof(vec_t) / sizeof(std::int32_t);
+    // Inputs are processed InputSimdWidth at a time and outputs are processed 8 at a time so we process in chunks of max(InputSimdWidth, 8)
+    constexpr IndexType ChunkSize       = std::max<IndexType>(InputSimdWidth, 8);
+    constexpr IndexType NumChunks       = InputDimensions / ChunkSize;
+    constexpr IndexType InputsPerChunk  = ChunkSize / InputSimdWidth;
+    constexpr IndexType OutputsPerChunk = ChunkSize / 8;
+
+    const auto     inputVector = reinterpret_cast<const vec_t*>(input);
+    IndexType      count       = 0;
+    vec128_t       base        = vec128_zero;
+    const vec128_t increment   = vec128_set_16(8);
+    for (IndexType i = 0; i < NumChunks; ++i)
+    {
+        // bitmask of nonzero values in this chunk
+        unsigned nnz = 0;
+        for (IndexType j = 0; j < InputsPerChunk; ++j)
+        {
+            const vec_t inputChunk = inputVector[i * InputsPerChunk + j];
+            nnz |= unsigned(vec_nnz(inputChunk)) << (j * InputSimdWidth);
+        }
+        for (IndexType j = 0; j < OutputsPerChunk; ++j)
+        {
+            const auto lookup = (nnz >> (j * 8)) & 0xFF;
+            const auto offsets =
+              vec128_load(reinterpret_cast<const vec128_t*>(&lookup_indices[lookup]));
+            vec128_storeu(reinterpret_cast<vec128_t*>(out + count), vec128_add(base, offsets));
+            count += popcount(lookup);
+            base = vec128_add(base, increment);
+        }
+    }
+    count_out = count;
+}
+    #undef vec_nnz
+    #undef vec128_zero
+    #undef vec128_set_16
+    #undef vec128_load
+    #undef vec128_storeu
+    #undef vec128_add
+#endif
+
+// Sparse input implementation
+template<IndexType InDims, IndexType OutDims>
+class AffineTransformSparseInput {
+   public:
+    // Input/output type
+    using InputType  = std::uint8_t;
+    using OutputType = std::int32_t;
+
+    // Number of input/output dimensions
+    static constexpr IndexType InputDimensions  = InDims;
+    static constexpr IndexType OutputDimensions = OutDims;
+
+    static_assert(OutputDimensions % 16 == 0,
+                  "Only implemented for OutputDimensions divisible by 16.");
+
+    static constexpr IndexType PaddedInputDimensions =
+      ceil_to_multiple<IndexType>(InputDimensions, MaxSimdWidth);
+    static constexpr IndexType PaddedOutputDimensions =
+      ceil_to_multiple<IndexType>(OutputDimensions, MaxSimdWidth);
+
+#if (USE_SSSE3 | (USE_NEON >= 8))
+    static constexpr IndexType ChunkSize = 4;
+#else
+    static constexpr IndexType ChunkSize = 1;
+#endif
+
+    using OutputBuffer = OutputType[PaddedOutputDimensions];
+
+    // Hash value embedded in the evaluation file
+    static constexpr std::uint32_t get_hash_value(std::uint32_t prevHash) {
+        std::uint32_t hashValue = 0xCC03DAE4u;
+        hashValue += OutputDimensions;
+        hashValue ^= prevHash >> 1;
+        hashValue ^= prevHash << 31;
+        return hashValue;
+    }
+
+    static constexpr IndexType get_weight_index_scrambled(IndexType i) {
+        return (i / ChunkSize) % (PaddedInputDimensions / ChunkSize) * OutputDimensions * ChunkSize
+             + i / PaddedInputDimensions * ChunkSize + i % ChunkSize;
+    }
+
+    static constexpr IndexType get_weight_index(IndexType i) {
+#if (USE_SSSE3 | (USE_NEON >= 8))
+        return get_weight_index_scrambled(i);
+#else
+        return i;
+#endif
+    }
+
+    // Read network parameters
+    bool read_parameters(std::istream& stream) {
+        read_little_endian<BiasType>(stream, biases, OutputDimensions);
+        for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
+            weights[get_weight_index(i)] = read_little_endian<WeightType>(stream);
+
+        return !stream.fail();
+    }
+
+    // Write network parameters
+    bool write_parameters(std::ostream& stream) const {
+        write_little_endian<BiasType>(stream, biases, OutputDimensions);
+
+        for (IndexType i = 0; i < OutputDimensions * PaddedInputDimensions; ++i)
+            write_little_endian<WeightType>(stream, weights[get_weight_index(i)]);
+
+        return !stream.fail();
+    }
+    // Forward propagation
+    void propagate(const InputType* input, OutputType* output) const {
+
+#if (USE_SSSE3 | (USE_NEON >= 8))
+    #if defined(USE_AVX512)
+        using invec_t  = __m512i;
+        using outvec_t = __m512i;
+        #define vec_set_32 _mm512_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m512_add_dpbusd_epi32
+    #elif defined(USE_AVX2)
+        using invec_t  = __m256i;
+        using outvec_t = __m256i;
+        #define vec_set_32 _mm256_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m256_add_dpbusd_epi32
+    #elif defined(USE_SSSE3)
+        using invec_t  = __m128i;
+        using outvec_t = __m128i;
+        #define vec_set_32 _mm_set1_epi32
+        #define vec_add_dpbusd_32 Simd::m128_add_dpbusd_epi32
+    #elif defined(USE_NEON_DOTPROD)
+        using invec_t  = int8x16_t;
+        using outvec_t = int32x4_t;
+        #define vec_set_32(a) vreinterpretq_s8_u32(vdupq_n_u32(a))
+        #define vec_add_dpbusd_32 Simd::dotprod_m128_add_dpbusd_epi32
+    #elif defined(USE_NEON)
+        using invec_t  = int8x16_t;
+        using outvec_t = int32x4_t;
+        #define vec_set_32(a) vreinterpretq_s8_u32(vdupq_n_u32(a))
+        #define vec_add_dpbusd_32 Simd::neon_m128_add_dpbusd_epi32
+    #endif
+        static constexpr IndexType OutputSimdWidth = sizeof(outvec_t) / sizeof(OutputType);
+
+        constexpr IndexType NumChunks = ceil_to_multiple<IndexType>(InputDimensions, 8) / ChunkSize;
+        constexpr IndexType NumRegs   = OutputDimensions / OutputSimdWidth;
+        std::uint16_t       nnz[NumChunks];
+        IndexType           count;
+
+        const auto input32 = reinterpret_cast<const std::int32_t*>(input);
+
+        // Find indices of nonzero 32-bit blocks
+        find_nnz<NumChunks>(input32, nnz, count);
+
+        const outvec_t* biasvec = reinterpret_cast<const outvec_t*>(biases);
+        outvec_t        acc[NumRegs];
+        for (IndexType k = 0; k < NumRegs; ++k)
+            acc[k] = biasvec[k];
+
+        for (IndexType j = 0; j < count; ++j)
+        {
+            const auto    i  = nnz[j];
+            const invec_t in = vec_set_32(input32[i]);
+            const auto    col =
+              reinterpret_cast<const invec_t*>(&weights[i * OutputDimensions * ChunkSize]);
+            for (IndexType k = 0; k < NumRegs; ++k)
+                vec_add_dpbusd_32(acc[k], in, col[k]);
+        }
+
+        outvec_t* outptr = reinterpret_cast<outvec_t*>(output);
+        for (IndexType k = 0; k < NumRegs; ++k)
+            outptr[k] = acc[k];
+    #undef vec_set_32
+    #undef vec_add_dpbusd_32
+#else
+        // Use dense implementation for the other architectures.
+        affine_transform_non_ssse3<InputDimensions, PaddedInputDimensions, OutputDimensions>(
+          output, weights, biases, input);
+#endif
+    }
+
+   private:
+    using BiasType   = OutputType;
+    using WeightType = std::int8_t;
+
+    alignas(CacheLineSize) BiasType biases[OutputDimensions];
+    alignas(CacheLineSize) WeightType weights[OutputDimensions * PaddedInputDimensions];
+};
+
+}  // namespace Stockfish::Eval::NNUE::Layers
+
+#endif  // #ifndef NNUE_LAYERS_AFFINE_TRANSFORM_SPARSE_INPUT_H_INCLUDED

src/nnue/layers/clipped_relu.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,160 +21,148 @@
 #ifndef NNUE_LAYERS_CLIPPED_RELU_H_INCLUDED
 #define NNUE_LAYERS_CLIPPED_RELU_H_INCLUDED
 
+#include <algorithm>
+#include <cstdint>
+#include <iosfwd>
+
 #include "../nnue_common.h"
 
 namespace Stockfish::Eval::NNUE::Layers {
 
-  // Clipped ReLU
+// Clipped ReLU
-  template <IndexType InDims>
+template<IndexType InDims>
-  class ClippedReLU {
+class ClippedReLU {
    public:
     // Input/output type
-    using InputType = std::int32_t;
+    using InputType  = std::int32_t;
     using OutputType = std::uint8_t;
 
     // Number of input/output dimensions
-    static constexpr IndexType InputDimensions = InDims;
+    static constexpr IndexType InputDimensions  = InDims;
     static constexpr IndexType OutputDimensions = InputDimensions;
     static constexpr IndexType PaddedOutputDimensions =
-        ceil_to_multiple<IndexType>(OutputDimensions, 32);
+      ceil_to_multiple<IndexType>(OutputDimensions, 32);
 
     using OutputBuffer = OutputType[PaddedOutputDimensions];
 
     // Hash value embedded in the evaluation file
     static constexpr std::uint32_t get_hash_value(std::uint32_t prevHash) {
-      std::uint32_t hashValue = 0x538D24C7u;
+        std::uint32_t hashValue = 0x538D24C7u;
-      hashValue += prevHash;
+        hashValue += prevHash;
-      return hashValue;
+        return hashValue;
     }
 
     // Read network parameters
-    bool read_parameters(std::istream&) {
+    bool read_parameters(std::istream&) { return true; }
-      return true;
-    }
 
     // Write network parameters
-    bool write_parameters(std::ostream&) const {
+    bool write_parameters(std::ostream&) const { return true; }
-      return true;
-    }
 
     // Forward propagation
-    const OutputType* propagate(
+    void propagate(const InputType* input, OutputType* output) const {
-        const InputType* input, OutputType* output) const {
 
-  #if defined(USE_AVX2)
+#if defined(USE_AVX2)
-      if constexpr (InputDimensions % SimdWidth == 0) {
+        if constexpr (InputDimensions % SimdWidth == 0)
+        {
+            constexpr IndexType NumChunks = InputDimensions / SimdWidth;
+            const __m256i       Zero      = _mm256_setzero_si256();
+            const __m256i       Offsets   = _mm256_set_epi32(7, 3, 6, 2, 5, 1, 4, 0);
+            const auto          in        = reinterpret_cast<const __m256i*>(input);
+            const auto          out       = reinterpret_cast<__m256i*>(output);
+            for (IndexType i = 0; i < NumChunks; ++i)
+            {
+                const __m256i words0 =
+                  _mm256_srai_epi16(_mm256_packs_epi32(_mm256_load_si256(&in[i * 4 + 0]),
+                                                       _mm256_load_si256(&in[i * 4 + 1])),
+                                    WeightScaleBits);
+                const __m256i words1 =
+                  _mm256_srai_epi16(_mm256_packs_epi32(_mm256_load_si256(&in[i * 4 + 2]),
+                                                       _mm256_load_si256(&in[i * 4 + 3])),
+                                    WeightScaleBits);
+                _mm256_store_si256(
+                  &out[i], _mm256_permutevar8x32_epi32(
+                             _mm256_max_epi8(_mm256_packs_epi16(words0, words1), Zero), Offsets));
+            }
+        }
+        else
+        {
+            constexpr IndexType NumChunks = InputDimensions / (SimdWidth / 2);
+            const __m128i       Zero      = _mm_setzero_si128();
+            const auto          in        = reinterpret_cast<const __m128i*>(input);
+            const auto          out       = reinterpret_cast<__m128i*>(output);
+            for (IndexType i = 0; i < NumChunks; ++i)
+            {
+                const __m128i words0 = _mm_srai_epi16(
+                  _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 0]), _mm_load_si128(&in[i * 4 + 1])),
+                  WeightScaleBits);
+                const __m128i words1 = _mm_srai_epi16(
+                  _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 2]), _mm_load_si128(&in[i * 4 + 3])),
+                  WeightScaleBits);
+                const __m128i packedbytes = _mm_packs_epi16(words0, words1);
+                _mm_store_si128(&out[i], _mm_max_epi8(packedbytes, Zero));
+            }
+        }
+        constexpr IndexType Start = InputDimensions % SimdWidth == 0
+                                    ? InputDimensions / SimdWidth * SimdWidth
+                                    : InputDimensions / (SimdWidth / 2) * (SimdWidth / 2);
+
+#elif defined(USE_SSE2)
         constexpr IndexType NumChunks = InputDimensions / SimdWidth;
-        const __m256i Zero = _mm256_setzero_si256();
+
-        const __m256i Offsets = _mm256_set_epi32(7, 3, 6, 2, 5, 1, 4, 0);
+    #ifdef USE_SSE41
-        const auto in = reinterpret_cast<const __m256i*>(input);
-        const auto out = reinterpret_cast<__m256i*>(output);
-        for (IndexType i = 0; i < NumChunks; ++i) {
-          const __m256i words0 = _mm256_srai_epi16(_mm256_packs_epi32(
-              _mm256_load_si256(&in[i * 4 + 0]),
-              _mm256_load_si256(&in[i * 4 + 1])), WeightScaleBits);
-          const __m256i words1 = _mm256_srai_epi16(_mm256_packs_epi32(
-              _mm256_load_si256(&in[i * 4 + 2]),
-              _mm256_load_si256(&in[i * 4 + 3])), WeightScaleBits);
-          _mm256_store_si256(&out[i], _mm256_permutevar8x32_epi32(_mm256_max_epi8(
-              _mm256_packs_epi16(words0, words1), Zero), Offsets));
-        }
-      } else {
-        constexpr IndexType NumChunks = InputDimensions / (SimdWidth / 2);
         const __m128i Zero = _mm_setzero_si128();
-        const auto in = reinterpret_cast<const __m128i*>(input);
+    #else
+        const __m128i k0x80s = _mm_set1_epi8(-128);
+    #endif
+
+        const auto in  = reinterpret_cast<const __m128i*>(input);
         const auto out = reinterpret_cast<__m128i*>(output);
-        for (IndexType i = 0; i < NumChunks; ++i) {
+        for (IndexType i = 0; i < NumChunks; ++i)
-          const __m128i words0 = _mm_srai_epi16(_mm_packs_epi32(
+        {
-              _mm_load_si128(&in[i * 4 + 0]),
+            const __m128i words0 = _mm_srai_epi16(
-              _mm_load_si128(&in[i * 4 + 1])), WeightScaleBits);
+              _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 0]), _mm_load_si128(&in[i * 4 + 1])),
-          const __m128i words1 = _mm_srai_epi16(_mm_packs_epi32(
+              WeightScaleBits);
-              _mm_load_si128(&in[i * 4 + 2]),
+            const __m128i words1 = _mm_srai_epi16(
-              _mm_load_si128(&in[i * 4 + 3])), WeightScaleBits);
+              _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 2]), _mm_load_si128(&in[i * 4 + 3])),
-          const __m128i packedbytes = _mm_packs_epi16(words0, words1);
+              WeightScaleBits);
-          _mm_store_si128(&out[i], _mm_max_epi8(packedbytes, Zero));
+            const __m128i packedbytes = _mm_packs_epi16(words0, words1);
+            _mm_store_si128(&out[i],
+
+    #ifdef USE_SSE41
+                            _mm_max_epi8(packedbytes, Zero)
+    #else
+                            _mm_subs_epi8(_mm_adds_epi8(packedbytes, k0x80s), k0x80s)
+    #endif
+
+            );
         }
-      }
+        constexpr IndexType Start = NumChunks * SimdWidth;
-      constexpr IndexType Start =
-        InputDimensions % SimdWidth == 0
-        ? InputDimensions / SimdWidth * SimdWidth
-        : InputDimensions / (SimdWidth / 2) * (SimdWidth / 2);
 
-  #elif defined(USE_SSE2)
+#elif defined(USE_NEON)
-      constexpr IndexType NumChunks = InputDimensions / SimdWidth;
+        constexpr IndexType NumChunks = InputDimensions / (SimdWidth / 2);
+        const int8x8_t      Zero      = {0};
+        const auto          in        = reinterpret_cast<const int32x4_t*>(input);
+        const auto          out       = reinterpret_cast<int8x8_t*>(output);
+        for (IndexType i = 0; i < NumChunks; ++i)
+        {
+            int16x8_t  shifted;
+            const auto pack = reinterpret_cast<int16x4_t*>(&shifted);
+            pack[0]         = vqshrn_n_s32(in[i * 2 + 0], WeightScaleBits);
+            pack[1]         = vqshrn_n_s32(in[i * 2 + 1], WeightScaleBits);
+            out[i]          = vmax_s8(vqmovn_s16(shifted), Zero);
+        }
+        constexpr IndexType Start = NumChunks * (SimdWidth / 2);
+#else
+        constexpr IndexType Start = 0;
+#endif
 
-  #ifdef USE_SSE41
+        for (IndexType i = Start; i < InputDimensions; ++i)
-      const __m128i Zero = _mm_setzero_si128();
+        {
-  #else
+            output[i] = static_cast<OutputType>(std::clamp(input[i] >> WeightScaleBits, 0, 127));
-      const __m128i k0x80s = _mm_set1_epi8(-128);
+        }
-  #endif
-
-      const auto in = reinterpret_cast<const __m128i*>(input);
-      const auto out = reinterpret_cast<__m128i*>(output);
-      for (IndexType i = 0; i < NumChunks; ++i) {
-        const __m128i words0 = _mm_srai_epi16(_mm_packs_epi32(
-            _mm_load_si128(&in[i * 4 + 0]),
-            _mm_load_si128(&in[i * 4 + 1])), WeightScaleBits);
-        const __m128i words1 = _mm_srai_epi16(_mm_packs_epi32(
-            _mm_load_si128(&in[i * 4 + 2]),
-            _mm_load_si128(&in[i * 4 + 3])), WeightScaleBits);
-        const __m128i packedbytes = _mm_packs_epi16(words0, words1);
-        _mm_store_si128(&out[i],
-
-  #ifdef USE_SSE41
-          _mm_max_epi8(packedbytes, Zero)
-  #else
-          _mm_subs_epi8(_mm_adds_epi8(packedbytes, k0x80s), k0x80s)
-  #endif
-
-        );
-      }
-      constexpr IndexType Start = NumChunks * SimdWidth;
-
-  #elif defined(USE_MMX)
-      constexpr IndexType NumChunks = InputDimensions / SimdWidth;
-      const __m64 k0x80s = _mm_set1_pi8(-128);
-      const auto in = reinterpret_cast<const __m64*>(input);
-      const auto out = reinterpret_cast<__m64*>(output);
-      for (IndexType i = 0; i < NumChunks; ++i) {
-        const __m64 words0 = _mm_srai_pi16(
-            _mm_packs_pi32(in[i * 4 + 0], in[i * 4 + 1]),
-            WeightScaleBits);
-        const __m64 words1 = _mm_srai_pi16(
-            _mm_packs_pi32(in[i * 4 + 2], in[i * 4 + 3]),
-            WeightScaleBits);
-        const __m64 packedbytes = _mm_packs_pi16(words0, words1);
-        out[i] = _mm_subs_pi8(_mm_adds_pi8(packedbytes, k0x80s), k0x80s);
-      }
-      _mm_empty();
-      constexpr IndexType Start = NumChunks * SimdWidth;
-
-  #elif defined(USE_NEON)
-      constexpr IndexType NumChunks = InputDimensions / (SimdWidth / 2);
-      const int8x8_t Zero = {0};
-      const auto in = reinterpret_cast<const int32x4_t*>(input);
-      const auto out = reinterpret_cast<int8x8_t*>(output);
-      for (IndexType i = 0; i < NumChunks; ++i) {
-        int16x8_t shifted;
-        const auto pack = reinterpret_cast<int16x4_t*>(&shifted);
-        pack[0] = vqshrn_n_s32(in[i * 2 + 0], WeightScaleBits);
-        pack[1] = vqshrn_n_s32(in[i * 2 + 1], WeightScaleBits);
-        out[i] = vmax_s8(vqmovn_s16(shifted), Zero);
-      }
-      constexpr IndexType Start = NumChunks * (SimdWidth / 2);
-  #else
-      constexpr IndexType Start = 0;
-  #endif
-
-      for (IndexType i = Start; i < InputDimensions; ++i) {
-        output[i] = static_cast<OutputType>(
-            std::max(0, std::min(127, input[i] >> WeightScaleBits)));
-      }
-
-      return output;
     }
-  };
+};
 
 }  // namespace Stockfish::Eval::NNUE::Layers
 
-#endif // NNUE_LAYERS_CLIPPED_RELU_H_INCLUDED
+#endif  // NNUE_LAYERS_CLIPPED_RELU_H_INCLUDED

src/nnue/layers/simd.h

@@ -0,0 +1,167 @@
+/*
+  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
+
+  Stockfish is free software: you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation, either version 3 of the License, or
+  (at your option) any later version.
+
+  Stockfish is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#ifndef STOCKFISH_SIMD_H_INCLUDED
+#define STOCKFISH_SIMD_H_INCLUDED
+
+#if defined(USE_AVX2)
+    #include <immintrin.h>
+
+#elif defined(USE_SSE41)
+    #include <smmintrin.h>
+
+#elif defined(USE_SSSE3)
+    #include <tmmintrin.h>
+
+#elif defined(USE_SSE2)
+    #include <emmintrin.h>
+
+#elif defined(USE_NEON)
+    #include <arm_neon.h>
+#endif
+
+namespace Stockfish::Simd {
+
+#if defined(USE_AVX512)
+
+[[maybe_unused]] static int m512_hadd(__m512i sum, int bias) {
+    return _mm512_reduce_add_epi32(sum) + bias;
+}
+
+/*
+      Parameters:
+        sum0 = [zmm0.i128[0], zmm0.i128[1], zmm0.i128[2], zmm0.i128[3]]
+        sum1 = [zmm1.i128[0], zmm1.i128[1], zmm1.i128[2], zmm1.i128[3]]
+        sum2 = [zmm2.i128[0], zmm2.i128[1], zmm2.i128[2], zmm2.i128[3]]
+        sum3 = [zmm3.i128[0], zmm3.i128[1], zmm3.i128[2], zmm3.i128[3]]
+
+      Returns:
+        ret = [
+          reduce_add_epi32(zmm0.i128[0]), reduce_add_epi32(zmm1.i128[0]), reduce_add_epi32(zmm2.i128[0]), reduce_add_epi32(zmm3.i128[0]),
+          reduce_add_epi32(zmm0.i128[1]), reduce_add_epi32(zmm1.i128[1]), reduce_add_epi32(zmm2.i128[1]), reduce_add_epi32(zmm3.i128[1]),
+          reduce_add_epi32(zmm0.i128[2]), reduce_add_epi32(zmm1.i128[2]), reduce_add_epi32(zmm2.i128[2]), reduce_add_epi32(zmm3.i128[2]),
+          reduce_add_epi32(zmm0.i128[3]), reduce_add_epi32(zmm1.i128[3]), reduce_add_epi32(zmm2.i128[3]), reduce_add_epi32(zmm3.i128[3])
+        ]
+    */
+[[maybe_unused]] static __m512i
+m512_hadd128x16_interleave(__m512i sum0, __m512i sum1, __m512i sum2, __m512i sum3) {
+
+    __m512i sum01a = _mm512_unpacklo_epi32(sum0, sum1);
+    __m512i sum01b = _mm512_unpackhi_epi32(sum0, sum1);
+
+    __m512i sum23a = _mm512_unpacklo_epi32(sum2, sum3);
+    __m512i sum23b = _mm512_unpackhi_epi32(sum2, sum3);
+
+    __m512i sum01 = _mm512_add_epi32(sum01a, sum01b);
+    __m512i sum23 = _mm512_add_epi32(sum23a, sum23b);
+
+    __m512i sum0123a = _mm512_unpacklo_epi64(sum01, sum23);
+    __m512i sum0123b = _mm512_unpackhi_epi64(sum01, sum23);
+
+    return _mm512_add_epi32(sum0123a, sum0123b);
+}
+
+[[maybe_unused]] static void m512_add_dpbusd_epi32(__m512i& acc, __m512i a, __m512i b) {
+
+    #if defined(USE_VNNI)
+    acc = _mm512_dpbusd_epi32(acc, a, b);
+    #else
+    __m512i product0 = _mm512_maddubs_epi16(a, b);
+    product0         = _mm512_madd_epi16(product0, _mm512_set1_epi16(1));
+    acc              = _mm512_add_epi32(acc, product0);
+    #endif
+}
+
+#endif
+
+#if defined(USE_AVX2)
+
+[[maybe_unused]] static int m256_hadd(__m256i sum, int bias) {
+    __m128i sum128 = _mm_add_epi32(_mm256_castsi256_si128(sum), _mm256_extracti128_si256(sum, 1));
+    sum128         = _mm_add_epi32(sum128, _mm_shuffle_epi32(sum128, _MM_PERM_BADC));
+    sum128         = _mm_add_epi32(sum128, _mm_shuffle_epi32(sum128, _MM_PERM_CDAB));
+    return _mm_cvtsi128_si32(sum128) + bias;
+}
+
+[[maybe_unused]] static void m256_add_dpbusd_epi32(__m256i& acc, __m256i a, __m256i b) {
+
+    #if defined(USE_VNNI)
+    acc = _mm256_dpbusd_epi32(acc, a, b);
+    #else
+    __m256i product0 = _mm256_maddubs_epi16(a, b);
+    product0         = _mm256_madd_epi16(product0, _mm256_set1_epi16(1));
+    acc              = _mm256_add_epi32(acc, product0);
+    #endif
+}
+
+#endif
+
+#if defined(USE_SSSE3)
+
+[[maybe_unused]] static int m128_hadd(__m128i sum, int bias) {
+    sum = _mm_add_epi32(sum, _mm_shuffle_epi32(sum, 0x4E));  //_MM_PERM_BADC
+    sum = _mm_add_epi32(sum, _mm_shuffle_epi32(sum, 0xB1));  //_MM_PERM_CDAB
+    return _mm_cvtsi128_si32(sum) + bias;
+}
+
+[[maybe_unused]] static void m128_add_dpbusd_epi32(__m128i& acc, __m128i a, __m128i b) {
+
+    __m128i product0 = _mm_maddubs_epi16(a, b);
+    product0         = _mm_madd_epi16(product0, _mm_set1_epi16(1));
+    acc              = _mm_add_epi32(acc, product0);
+}
+
+#endif
+
+#if defined(USE_NEON_DOTPROD)
+
+[[maybe_unused]] static void
+dotprod_m128_add_dpbusd_epi32(int32x4_t& acc, int8x16_t a, int8x16_t b) {
+
+    acc = vdotq_s32(acc, a, b);
+}
+#endif
+
+#if defined(USE_NEON)
+
+[[maybe_unused]] static int neon_m128_reduce_add_epi32(int32x4_t s) {
+    #if USE_NEON >= 8
+    return vaddvq_s32(s);
+    #else
+    return s[0] + s[1] + s[2] + s[3];
+    #endif
+}
+
+[[maybe_unused]] static int neon_m128_hadd(int32x4_t sum, int bias) {
+    return neon_m128_reduce_add_epi32(sum) + bias;
+}
+
+#endif
+
+#if USE_NEON >= 8
+[[maybe_unused]] static void neon_m128_add_dpbusd_epi32(int32x4_t& acc, int8x16_t a, int8x16_t b) {
+
+    int16x8_t product0 = vmull_s8(vget_low_s8(a), vget_low_s8(b));
+    int16x8_t product1 = vmull_high_s8(a, b);
+    int16x8_t sum      = vpaddq_s16(product0, product1);
+    acc                = vpadalq_s16(acc, sum);
+}
+#endif
+}
+
+#endif  // STOCKFISH_SIMD_H_INCLUDED

src/nnue/layers/sqr_clipped_relu.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,100 +21,83 @@
 #ifndef NNUE_LAYERS_SQR_CLIPPED_RELU_H_INCLUDED
 #define NNUE_LAYERS_SQR_CLIPPED_RELU_H_INCLUDED
 
+#include <algorithm>
+#include <cstdint>
+#include <iosfwd>
+
 #include "../nnue_common.h"
 
 namespace Stockfish::Eval::NNUE::Layers {
 
-  // Clipped ReLU
+// Clipped ReLU
-  template <IndexType InDims>
+template<IndexType InDims>
-  class SqrClippedReLU {
+class SqrClippedReLU {
    public:
     // Input/output type
-    using InputType = std::int32_t;
+    using InputType  = std::int32_t;
     using OutputType = std::uint8_t;
 
     // Number of input/output dimensions
-    static constexpr IndexType InputDimensions = InDims;
+    static constexpr IndexType InputDimensions  = InDims;
     static constexpr IndexType OutputDimensions = InputDimensions;
     static constexpr IndexType PaddedOutputDimensions =
-        ceil_to_multiple<IndexType>(OutputDimensions, 32);
+      ceil_to_multiple<IndexType>(OutputDimensions, 32);
 
     using OutputBuffer = OutputType[PaddedOutputDimensions];
 
     // Hash value embedded in the evaluation file
     static constexpr std::uint32_t get_hash_value(std::uint32_t prevHash) {
-      std::uint32_t hashValue = 0x538D24C7u;
+        std::uint32_t hashValue = 0x538D24C7u;
-      hashValue += prevHash;
+        hashValue += prevHash;
-      return hashValue;
+        return hashValue;
     }
 
     // Read network parameters
-    bool read_parameters(std::istream&) {
+    bool read_parameters(std::istream&) { return true; }
-      return true;
-    }
 
     // Write network parameters
-    bool write_parameters(std::ostream&) const {
+    bool write_parameters(std::ostream&) const { return true; }
-      return true;
-    }
 
     // Forward propagation
-    const OutputType* propagate(
+    void propagate(const InputType* input, OutputType* output) const {
-        const InputType* input, OutputType* output) const {
 
-  #if defined(USE_SSE2)
+#if defined(USE_SSE2)
-      constexpr IndexType NumChunks = InputDimensions / 16;
+        constexpr IndexType NumChunks = InputDimensions / 16;
 
-  #ifdef USE_SSE41
+        static_assert(WeightScaleBits == 6);
-      const __m128i Zero = _mm_setzero_si128();
+        const auto in  = reinterpret_cast<const __m128i*>(input);
-  #else
+        const auto out = reinterpret_cast<__m128i*>(output);
-      const __m128i k0x80s = _mm_set1_epi8(-128);
+        for (IndexType i = 0; i < NumChunks; ++i)
-  #endif
+        {
+            __m128i words0 =
+              _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 0]), _mm_load_si128(&in[i * 4 + 1]));
+            __m128i words1 =
+              _mm_packs_epi32(_mm_load_si128(&in[i * 4 + 2]), _mm_load_si128(&in[i * 4 + 3]));
 
-      static_assert(WeightScaleBits == 6);
+            // We shift by WeightScaleBits * 2 = 12 and divide by 128
-      const auto in = reinterpret_cast<const __m128i*>(input);
+            // which is an additional shift-right of 7, meaning 19 in total.
-      const auto out = reinterpret_cast<__m128i*>(output);
+            // MulHi strips the lower 16 bits so we need to shift out 3 more to match.
-      for (IndexType i = 0; i < NumChunks; ++i) {
+            words0 = _mm_srli_epi16(_mm_mulhi_epi16(words0, words0), 3);
-        __m128i words0 = _mm_packs_epi32(
+            words1 = _mm_srli_epi16(_mm_mulhi_epi16(words1, words1), 3);
-            _mm_load_si128(&in[i * 4 + 0]),
-            _mm_load_si128(&in[i * 4 + 1]));
-        __m128i words1 = _mm_packs_epi32(
-            _mm_load_si128(&in[i * 4 + 2]),
-            _mm_load_si128(&in[i * 4 + 3]));
 
-        // Not sure if
+            _mm_store_si128(&out[i], _mm_packs_epi16(words0, words1));
-        words0 = _mm_srli_epi16(_mm_mulhi_epi16(words0, words0), 3);
+        }
-        words1 = _mm_srli_epi16(_mm_mulhi_epi16(words1, words1), 3);
+        constexpr IndexType Start = NumChunks * 16;
 
-        const __m128i packedbytes = _mm_packs_epi16(words0, words1);
+#else
+        constexpr IndexType Start = 0;
+#endif
 
-        _mm_store_si128(&out[i],
+        for (IndexType i = Start; i < InputDimensions; ++i)
-
+        {
-  #ifdef USE_SSE41
+            output[i] = static_cast<OutputType>(
-          _mm_max_epi8(packedbytes, Zero)
+              // Really should be /127 but we need to make it fast so we right-shift
-  #else
+              // by an extra 7 bits instead. Needs to be accounted for in the trainer.
-          _mm_subs_epi8(_mm_adds_epi8(packedbytes, k0x80s), k0x80s)
+              std::min(127ll, ((long long) (input[i]) * input[i]) >> (2 * WeightScaleBits + 7)));
-  #endif
+        }
-
-        );
-      }
-      constexpr IndexType Start = NumChunks * 16;
-
-  #else
-      constexpr IndexType Start = 0;
-  #endif
-
-      for (IndexType i = Start; i < InputDimensions; ++i) {
-        output[i] = static_cast<OutputType>(
-            // realy should be /127 but we need to make it fast
-            // needs to be accounted for in the trainer
-            std::max(0ll, std::min(127ll, (((long long)input[i] * input[i]) >> (2 * WeightScaleBits)) / 128)));
-      }
-
-      return output;
     }
-  };
+};
 
 }  // namespace Stockfish::Eval::NNUE::Layers
 
-#endif // NNUE_LAYERS_SQR_CLIPPED_RELU_H_INCLUDED
+#endif  // NNUE_LAYERS_SQR_CLIPPED_RELU_H_INCLUDED

src/nnue/nnue_accumulator.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,17 +21,21 @@
 #ifndef NNUE_ACCUMULATOR_H_INCLUDED
 #define NNUE_ACCUMULATOR_H_INCLUDED
 
+#include <cstdint>
+
 #include "nnue_architecture.h"
+#include "nnue_common.h"
 
 namespace Stockfish::Eval::NNUE {
 
-  // Class that holds the result of affine transformation of input features
+// Class that holds the result of affine transformation of input features
-  struct alignas(CacheLineSize) Accumulator {
+template<IndexType Size>
-    std::int16_t accumulation[2][TransformedFeatureDimensions];
+struct alignas(CacheLineSize) Accumulator {
+    std::int16_t accumulation[2][Size];
     std::int32_t psqtAccumulation[2][PSQTBuckets];
-    bool computed[2];
+    bool         computed[2];
-  };
+};
 
 }  // namespace Stockfish::Eval::NNUE
 
-#endif // NNUE_ACCUMULATOR_H_INCLUDED
+#endif  // NNUE_ACCUMULATOR_H_INCLUDED

src/nnue/nnue_architecture.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,118 +21,122 @@
 #ifndef NNUE_ARCHITECTURE_H_INCLUDED
 #define NNUE_ARCHITECTURE_H_INCLUDED
 
-#include <memory>
+#include <cstdint>
-
+#include <cstring>
-#include "nnue_common.h"
+#include <iosfwd>
 
 #include "features/half_ka_v2_hm.h"
-
 #include "layers/affine_transform.h"
+#include "layers/affine_transform_sparse_input.h"
 #include "layers/clipped_relu.h"
 #include "layers/sqr_clipped_relu.h"
-
+#include "nnue_common.h"
-#include "../misc.h"
 
 namespace Stockfish::Eval::NNUE {
 
 // Input features used in evaluation function
 using FeatureSet = Features::HalfKAv2_hm;
 
+enum NetSize : int {
+    Big,
+    Small
+};
+
 // Number of input feature dimensions after conversion
-constexpr IndexType TransformedFeatureDimensions = 1024;
+constexpr IndexType TransformedFeatureDimensionsBig = 2560;
+constexpr int       L2Big                           = 15;
+constexpr int       L3Big                           = 32;
+
+constexpr IndexType TransformedFeatureDimensionsSmall = 128;
+constexpr int       L2Small                           = 15;
+constexpr int       L3Small                           = 32;
+
 constexpr IndexType PSQTBuckets = 8;
 constexpr IndexType LayerStacks = 8;
 
-struct Network
+template<IndexType L1, int L2, int L3>
-{
+struct Network {
-  static constexpr int FC_0_OUTPUTS = 15;
+    static constexpr IndexType TransformedFeatureDimensions = L1;
-  static constexpr int FC_1_OUTPUTS = 32;
+    static constexpr int       FC_0_OUTPUTS                 = L2;
+    static constexpr int       FC_1_OUTPUTS                 = L3;
 
-  Layers::AffineTransform<TransformedFeatureDimensions, FC_0_OUTPUTS + 1> fc_0;
+    Layers::AffineTransformSparseInput<TransformedFeatureDimensions, FC_0_OUTPUTS + 1> fc_0;
-  Layers::SqrClippedReLU<FC_0_OUTPUTS + 1> ac_sqr_0;
+    Layers::SqrClippedReLU<FC_0_OUTPUTS + 1>                                           ac_sqr_0;
-  Layers::ClippedReLU<FC_0_OUTPUTS + 1> ac_0;
+    Layers::ClippedReLU<FC_0_OUTPUTS + 1>                                              ac_0;
-  Layers::AffineTransform<FC_0_OUTPUTS * 2, FC_1_OUTPUTS> fc_1;
+    Layers::AffineTransform<FC_0_OUTPUTS * 2, FC_1_OUTPUTS>                            fc_1;
-  Layers::ClippedReLU<FC_1_OUTPUTS> ac_1;
+    Layers::ClippedReLU<FC_1_OUTPUTS>                                                  ac_1;
-  Layers::AffineTransform<FC_1_OUTPUTS, 1> fc_2;
+    Layers::AffineTransform<FC_1_OUTPUTS, 1>                                           fc_2;
 
-  // Hash value embedded in the evaluation file
+    // Hash value embedded in the evaluation file
-  static constexpr std::uint32_t get_hash_value() {
+    static constexpr std::uint32_t get_hash_value() {
-    // input slice hash
+        // input slice hash
-    std::uint32_t hashValue = 0xEC42E90Du;
+        std::uint32_t hashValue = 0xEC42E90Du;
-    hashValue ^= TransformedFeatureDimensions * 2;
+        hashValue ^= TransformedFeatureDimensions * 2;
 
-    hashValue = decltype(fc_0)::get_hash_value(hashValue);
+        hashValue = decltype(fc_0)::get_hash_value(hashValue);
-    hashValue = decltype(ac_0)::get_hash_value(hashValue);
+        hashValue = decltype(ac_0)::get_hash_value(hashValue);
-    hashValue = decltype(fc_1)::get_hash_value(hashValue);
+        hashValue = decltype(fc_1)::get_hash_value(hashValue);
-    hashValue = decltype(ac_1)::get_hash_value(hashValue);
+        hashValue = decltype(ac_1)::get_hash_value(hashValue);
-    hashValue = decltype(fc_2)::get_hash_value(hashValue);
+        hashValue = decltype(fc_2)::get_hash_value(hashValue);
 
-    return hashValue;
+        return hashValue;
-  }
+    }
 
-  // Read network parameters
+    // Read network parameters
-  bool read_parameters(std::istream& stream) {
+    bool read_parameters(std::istream& stream) {
-    if (!fc_0.read_parameters(stream)) return false;
+        return fc_0.read_parameters(stream) && ac_0.read_parameters(stream)
-    if (!ac_0.read_parameters(stream)) return false;
+            && fc_1.read_parameters(stream) && ac_1.read_parameters(stream)
-    if (!fc_1.read_parameters(stream)) return false;
+            && fc_2.read_parameters(stream);
-    if (!ac_1.read_parameters(stream)) return false;
+    }
-    if (!fc_2.read_parameters(stream)) return false;
-    return true;
-  }
 
-  // Read network parameters
+    // Write network parameters
-  bool write_parameters(std::ostream& stream) const {
+    bool write_parameters(std::ostream& stream) const {
-    if (!fc_0.write_parameters(stream)) return false;
+        return fc_0.write_parameters(stream) && ac_0.write_parameters(stream)
-    if (!ac_0.write_parameters(stream)) return false;
+            && fc_1.write_parameters(stream) && ac_1.write_parameters(stream)
-    if (!fc_1.write_parameters(stream)) return false;
+            && fc_2.write_parameters(stream);
-    if (!ac_1.write_parameters(stream)) return false;
+    }
-    if (!fc_2.write_parameters(stream)) return false;
-    return true;
-  }
 
-  std::int32_t propagate(const TransformedFeatureType* transformedFeatures)
+    std::int32_t propagate(const TransformedFeatureType* transformedFeatures) {
-  {
+        struct alignas(CacheLineSize) Buffer {
-    struct alignas(CacheLineSize) Buffer
+            alignas(CacheLineSize) typename decltype(fc_0)::OutputBuffer fc_0_out;
-    {
+            alignas(CacheLineSize) typename decltype(ac_sqr_0)::OutputType
-      alignas(CacheLineSize) decltype(fc_0)::OutputBuffer fc_0_out;
+              ac_sqr_0_out[ceil_to_multiple<IndexType>(FC_0_OUTPUTS * 2, 32)];
-      alignas(CacheLineSize) decltype(ac_sqr_0)::OutputType ac_sqr_0_out[ceil_to_multiple<IndexType>(FC_0_OUTPUTS * 2, 32)];
+            alignas(CacheLineSize) typename decltype(ac_0)::OutputBuffer ac_0_out;
-      alignas(CacheLineSize) decltype(ac_0)::OutputBuffer ac_0_out;
+            alignas(CacheLineSize) typename decltype(fc_1)::OutputBuffer fc_1_out;
-      alignas(CacheLineSize) decltype(fc_1)::OutputBuffer fc_1_out;
+            alignas(CacheLineSize) typename decltype(ac_1)::OutputBuffer ac_1_out;
-      alignas(CacheLineSize) decltype(ac_1)::OutputBuffer ac_1_out;
+            alignas(CacheLineSize) typename decltype(fc_2)::OutputBuffer fc_2_out;
-      alignas(CacheLineSize) decltype(fc_2)::OutputBuffer fc_2_out;
 
-      Buffer()
+            Buffer() { std::memset(this, 0, sizeof(*this)); }
-      {
+        };
-          std::memset(this, 0, sizeof(*this));
-      }
-    };
 
 #if defined(__clang__) && (__APPLE__)
-    // workaround for a bug reported with xcode 12
+        // workaround for a bug reported with xcode 12
-    static thread_local auto tlsBuffer = std::make_unique<Buffer>();
+        static thread_local auto tlsBuffer = std::make_unique<Buffer>();
-    // Access TLS only once, cache result.
+        // Access TLS only once, cache result.
-    Buffer& buffer = *tlsBuffer;
+        Buffer& buffer = *tlsBuffer;
 #else
-    alignas(CacheLineSize) static thread_local Buffer buffer;
+        alignas(CacheLineSize) static thread_local Buffer buffer;
 #endif
 
-    fc_0.propagate(transformedFeatures, buffer.fc_0_out);
+        fc_0.propagate(transformedFeatures, buffer.fc_0_out);
-    ac_sqr_0.propagate(buffer.fc_0_out, buffer.ac_sqr_0_out);
+        ac_sqr_0.propagate(buffer.fc_0_out, buffer.ac_sqr_0_out);
-    ac_0.propagate(buffer.fc_0_out, buffer.ac_0_out);
+        ac_0.propagate(buffer.fc_0_out, buffer.ac_0_out);
-    std::memcpy(buffer.ac_sqr_0_out + FC_0_OUTPUTS, buffer.ac_0_out, FC_0_OUTPUTS * sizeof(decltype(ac_0)::OutputType));
+        std::memcpy(buffer.ac_sqr_0_out + FC_0_OUTPUTS, buffer.ac_0_out,
-    fc_1.propagate(buffer.ac_sqr_0_out, buffer.fc_1_out);
+                    FC_0_OUTPUTS * sizeof(typename decltype(ac_0)::OutputType));
-    ac_1.propagate(buffer.fc_1_out, buffer.ac_1_out);
+        fc_1.propagate(buffer.ac_sqr_0_out, buffer.fc_1_out);
-    fc_2.propagate(buffer.ac_1_out, buffer.fc_2_out);
+        ac_1.propagate(buffer.fc_1_out, buffer.ac_1_out);
+        fc_2.propagate(buffer.ac_1_out, buffer.fc_2_out);
 
-    // buffer.fc_0_out[FC_0_OUTPUTS] is such that 1.0 is equal to 127*(1<<WeightScaleBits) in quantized form
+        // buffer.fc_0_out[FC_0_OUTPUTS] is such that 1.0 is equal to 127*(1<<WeightScaleBits) in
-    // but we want 1.0 to be equal to 600*OutputScale
+        // quantized form, but we want 1.0 to be equal to 600*OutputScale
-    std::int32_t fwdOut = int(buffer.fc_0_out[FC_0_OUTPUTS]) * (600*OutputScale) / (127*(1<<WeightScaleBits));
+        std::int32_t fwdOut =
-    std::int32_t outputValue = buffer.fc_2_out[0] + fwdOut;
+          (buffer.fc_0_out[FC_0_OUTPUTS]) * (600 * OutputScale) / (127 * (1 << WeightScaleBits));
+        std::int32_t outputValue = buffer.fc_2_out[0] + fwdOut;
 
-    return outputValue;
+        return outputValue;
-  }
+    }
 };
 
 }  // namespace Stockfish::Eval::NNUE
 
-#endif // #ifndef NNUE_ARCHITECTURE_H_INCLUDED
+#endif  // #ifndef NNUE_ARCHITECTURE_H_INCLUDED

src/nnue/nnue_common.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,146 +21,264 @@
 #ifndef NNUE_COMMON_H_INCLUDED
 #define NNUE_COMMON_H_INCLUDED
 
-#include "../types.h"
+#include <algorithm>
-
+#include <cassert>
+#include <cstdint>
 #include <cstring>
 #include <iostream>
+#include <type_traits>
 
-#include "../misc.h"  // for IsLittleEndian
+#include "../misc.h"
 
 #if defined(USE_AVX2)
-#include <immintrin.h>
+    #include <immintrin.h>
 
 #elif defined(USE_SSE41)
-#include <smmintrin.h>
+    #include <smmintrin.h>
 
 #elif defined(USE_SSSE3)
-#include <tmmintrin.h>
+    #include <tmmintrin.h>
 
 #elif defined(USE_SSE2)
-#include <emmintrin.h>
+    #include <emmintrin.h>
-
-#elif defined(USE_MMX)
-#include <mmintrin.h>
 
 #elif defined(USE_NEON)
-#include <arm_neon.h>
+    #include <arm_neon.h>
 #endif
 
 namespace Stockfish::Eval::NNUE {
 
-  // Version of the evaluation file
+// Version of the evaluation file
-  constexpr std::uint32_t Version = 0x7AF32F20u;
+constexpr std::uint32_t Version = 0x7AF32F20u;
 
-  // Constant used in evaluation value calculation
+// Constant used in evaluation value calculation
-  constexpr int OutputScale = 16;
+constexpr int OutputScale     = 16;
-  constexpr int WeightScaleBits = 6;
+constexpr int WeightScaleBits = 6;
 
-  // Size of cache line (in bytes)
+// Size of cache line (in bytes)
-  constexpr std::size_t CacheLineSize = 64;
+constexpr std::size_t CacheLineSize = 64;
 
-  // SIMD width (in bytes)
+constexpr const char        Leb128MagicString[]   = "COMPRESSED_LEB128";
-  #if defined(USE_AVX2)
+constexpr const std::size_t Leb128MagicStringSize = sizeof(Leb128MagicString) - 1;
-  constexpr std::size_t SimdWidth = 32;
 
-  #elif defined(USE_SSE2)
+// SIMD width (in bytes)
-  constexpr std::size_t SimdWidth = 16;
+#if defined(USE_AVX2)
+constexpr std::size_t SimdWidth = 32;
 
-  #elif defined(USE_MMX)
+#elif defined(USE_SSE2)
-  constexpr std::size_t SimdWidth = 8;
+constexpr std::size_t SimdWidth = 16;
 
-  #elif defined(USE_NEON)
+#elif defined(USE_NEON)
-  constexpr std::size_t SimdWidth = 16;
+constexpr std::size_t SimdWidth = 16;
-  #endif
+#endif
 
-  constexpr std::size_t MaxSimdWidth = 32;
+constexpr std::size_t MaxSimdWidth = 32;
 
-  // Type of input feature after conversion
+// Type of input feature after conversion
-  using TransformedFeatureType = std::uint8_t;
+using TransformedFeatureType = std::uint8_t;
-  using IndexType = std::uint32_t;
+using IndexType              = std::uint32_t;
 
-  // Round n up to be a multiple of base
+// Round n up to be a multiple of base
-  template <typename IntType>
+template<typename IntType>
-  constexpr IntType ceil_to_multiple(IntType n, IntType base) {
+constexpr IntType ceil_to_multiple(IntType n, IntType base) {
-      return (n + base - 1) / base * base;
+    return (n + base - 1) / base * base;
-  }
+}
 
-  // read_little_endian() is our utility to read an integer (signed or unsigned, any size)
-  // from a stream in little-endian order. We swap the byte order after the read if
-  // necessary to return a result with the byte ordering of the compiling machine.
-  template <typename IntType>
-  inline IntType read_little_endian(std::istream& stream) {
-      IntType result;
 
-      if (IsLittleEndian)
+// Utility to read an integer (signed or unsigned, any size)
-          stream.read(reinterpret_cast<char*>(&result), sizeof(IntType));
+// from a stream in little-endian order. We swap the byte order after the read if
-      else
+// necessary to return a result with the byte ordering of the compiling machine.
-      {
+template<typename IntType>
-          std::uint8_t u[sizeof(IntType)];
+inline IntType read_little_endian(std::istream& stream) {
-          typename std::make_unsigned<IntType>::type v = 0;
+    IntType result;
 
-          stream.read(reinterpret_cast<char*>(u), sizeof(IntType));
+    if (IsLittleEndian)
-          for (std::size_t i = 0; i < sizeof(IntType); ++i)
+        stream.read(reinterpret_cast<char*>(&result), sizeof(IntType));
-              v = (v << 8) | u[sizeof(IntType) - i - 1];
+    else
+    {
+        std::uint8_t                  u[sizeof(IntType)];
+        std::make_unsigned_t<IntType> v = 0;
 
-          std::memcpy(&result, &v, sizeof(IntType));
+        stream.read(reinterpret_cast<char*>(u), sizeof(IntType));
-      }
+        for (std::size_t i = 0; i < sizeof(IntType); ++i)
+            v = (v << 8) | u[sizeof(IntType) - i - 1];
 
-      return result;
+        std::memcpy(&result, &v, sizeof(IntType));
-  }
+    }
 
-  // write_little_endian() is our utility to write an integer (signed or unsigned, any size)
+    return result;
-  // to a stream in little-endian order. We swap the byte order before the write if
+}
-  // necessary to always write in little endian order, independently of the byte
-  // ordering of the compiling machine.
-  template <typename IntType>
-  inline void write_little_endian(std::ostream& stream, IntType value) {
 
-      if (IsLittleEndian)
-          stream.write(reinterpret_cast<const char*>(&value), sizeof(IntType));
-      else
-      {
-          std::uint8_t u[sizeof(IntType)];
-          typename std::make_unsigned<IntType>::type v = value;
 
-          std::size_t i = 0;
+// Utility to write an integer (signed or unsigned, any size)
-          // if constexpr to silence the warning about shift by 8
+// to a stream in little-endian order. We swap the byte order before the write if
-          if constexpr (sizeof(IntType) > 1)
+// necessary to always write in little-endian order, independently of the byte
-          {
+// ordering of the compiling machine.
+template<typename IntType>
+inline void write_little_endian(std::ostream& stream, IntType value) {
+
+    if (IsLittleEndian)
+        stream.write(reinterpret_cast<const char*>(&value), sizeof(IntType));
+    else
+    {
+        std::uint8_t                  u[sizeof(IntType)];
+        std::make_unsigned_t<IntType> v = value;
+
+        std::size_t i = 0;
+        // if constexpr to silence the warning about shift by 8
+        if constexpr (sizeof(IntType) > 1)
+        {
             for (; i + 1 < sizeof(IntType); ++i)
             {
-                u[i] = (std::uint8_t)v;
+                u[i] = std::uint8_t(v);
                 v >>= 8;
             }
-          }
+        }
-          u[i] = (std::uint8_t)v;
+        u[i] = std::uint8_t(v);
 
-          stream.write(reinterpret_cast<char*>(u), sizeof(IntType));
+        stream.write(reinterpret_cast<char*>(u), sizeof(IntType));
-      }
+    }
-  }
+}
 
-  // read_little_endian(s, out, N) : read integers in bulk from a little indian stream.
-  // This reads N integers from stream s and put them in array out.
-  template <typename IntType>
-  inline void read_little_endian(std::istream& stream, IntType* out, std::size_t count) {
-      if (IsLittleEndian)
-          stream.read(reinterpret_cast<char*>(out), sizeof(IntType) * count);
-      else
-          for (std::size_t i = 0; i < count; ++i)
-              out[i] = read_little_endian<IntType>(stream);
-  }
 
-  // write_little_endian(s, values, N) : write integers in bulk to a little indian stream.
+// Read integers in bulk from a little-endian stream.
-  // This takes N integers from array values and writes them on stream s.
+// This reads N integers from stream s and puts them in array out.
-  template <typename IntType>
+template<typename IntType>
-  inline void write_little_endian(std::ostream& stream, const IntType* values, std::size_t count) {
+inline void read_little_endian(std::istream& stream, IntType* out, std::size_t count) {
-      if (IsLittleEndian)
+    if (IsLittleEndian)
-          stream.write(reinterpret_cast<const char*>(values), sizeof(IntType) * count);
+        stream.read(reinterpret_cast<char*>(out), sizeof(IntType) * count);
-      else
+    else
-          for (std::size_t i = 0; i < count; ++i)
+        for (std::size_t i = 0; i < count; ++i)
-              write_little_endian<IntType>(stream, values[i]);
+            out[i] = read_little_endian<IntType>(stream);
-  }
+}
+
+
+// Write integers in bulk to a little-endian stream.
+// This takes N integers from array values and writes them on stream s.
+template<typename IntType>
+inline void write_little_endian(std::ostream& stream, const IntType* values, std::size_t count) {
+    if (IsLittleEndian)
+        stream.write(reinterpret_cast<const char*>(values), sizeof(IntType) * count);
+    else
+        for (std::size_t i = 0; i < count; ++i)
+            write_little_endian<IntType>(stream, values[i]);
+}
+
+
+// Read N signed integers from the stream s, putting them in the array out.
+// The stream is assumed to be compressed using the signed LEB128 format.
+// See https://en.wikipedia.org/wiki/LEB128 for a description of the compression scheme.
+template<typename IntType>
+inline void read_leb_128(std::istream& stream, IntType* out, std::size_t count) {
+
+    // Check the presence of our LEB128 magic string
+    char leb128MagicString[Leb128MagicStringSize];
+    stream.read(leb128MagicString, Leb128MagicStringSize);
+    assert(strncmp(Leb128MagicString, leb128MagicString, Leb128MagicStringSize) == 0);
+
+    static_assert(std::is_signed_v<IntType>, "Not implemented for unsigned types");
+
+    const std::uint32_t BUF_SIZE = 4096;
+    std::uint8_t        buf[BUF_SIZE];
+
+    auto bytes_left = read_little_endian<std::uint32_t>(stream);
+
+    std::uint32_t buf_pos = BUF_SIZE;
+    for (std::size_t i = 0; i < count; ++i)
+    {
+        IntType result = 0;
+        size_t  shift  = 0;
+        do
+        {
+            if (buf_pos == BUF_SIZE)
+            {
+                stream.read(reinterpret_cast<char*>(buf), std::min(bytes_left, BUF_SIZE));
+                buf_pos = 0;
+            }
+
+            std::uint8_t byte = buf[buf_pos++];
+            --bytes_left;
+            result |= (byte & 0x7f) << shift;
+            shift += 7;
+
+            if ((byte & 0x80) == 0)
+            {
+                out[i] = (sizeof(IntType) * 8 <= shift || (byte & 0x40) == 0)
+                         ? result
+                         : result | ~((1 << shift) - 1);
+                break;
+            }
+        } while (shift < sizeof(IntType) * 8);
+    }
+
+    assert(bytes_left == 0);
+}
+
+
+// Write signed integers to a stream with LEB128 compression.
+// This takes N integers from array values, compresses them with
+// the LEB128 algorithm and writes the result on the stream s.
+// See https://en.wikipedia.org/wiki/LEB128 for a description of the compression scheme.
+template<typename IntType>
+inline void write_leb_128(std::ostream& stream, const IntType* values, std::size_t count) {
+
+    // Write our LEB128 magic string
+    stream.write(Leb128MagicString, Leb128MagicStringSize);
+
+    static_assert(std::is_signed_v<IntType>, "Not implemented for unsigned types");
+
+    std::uint32_t byte_count = 0;
+    for (std::size_t i = 0; i < count; ++i)
+    {
+        IntType      value = values[i];
+        std::uint8_t byte;
+        do
+        {
+            byte = value & 0x7f;
+            value >>= 7;
+            ++byte_count;
+        } while ((byte & 0x40) == 0 ? value != 0 : value != -1);
+    }
+
+    write_little_endian(stream, byte_count);
+
+    const std::uint32_t BUF_SIZE = 4096;
+    std::uint8_t        buf[BUF_SIZE];
+    std::uint32_t       buf_pos = 0;
+
+    auto flush = [&]() {
+        if (buf_pos > 0)
+        {
+            stream.write(reinterpret_cast<char*>(buf), buf_pos);
+            buf_pos = 0;
+        }
+    };
+
+    auto write = [&](std::uint8_t byte) {
+        buf[buf_pos++] = byte;
+        if (buf_pos == BUF_SIZE)
+            flush();
+    };
+
+    for (std::size_t i = 0; i < count; ++i)
+    {
+        IntType value = values[i];
+        while (true)
+        {
+            std::uint8_t byte = value & 0x7f;
+            value >>= 7;
+            if ((byte & 0x40) == 0 ? value == 0 : value == -1)
+            {
+                write(byte);
+                break;
+            }
+            write(byte | 0x80);
+        }
+    }
+
+    flush();
+}
 
 }  // namespace Stockfish::Eval::NNUE
 
-#endif // #ifndef NNUE_COMMON_H_INCLUDED
+#endif  // #ifndef NNUE_COMMON_H_INCLUDED

src/pawns.cpp

@@ -1,305 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#include <algorithm>
-#include <cassert>
-
-#include "bitboard.h"
-#include "pawns.h"
-#include "position.h"
-#include "thread.h"
-
-namespace Stockfish {
-
-namespace {
-
-  #define V Value
-  #define S(mg, eg) make_score(mg, eg)
-
-  // Pawn penalties
-  constexpr Score Backward      = S( 6, 19);
-  constexpr Score Doubled       = S(11, 51);
-  constexpr Score DoubledEarly  = S(17,  7);
-  constexpr Score Isolated      = S( 1, 20);
-  constexpr Score WeakLever     = S( 2, 57);
-  constexpr Score WeakUnopposed = S(15, 18);
-
-  // Bonus for blocked pawns at 5th or 6th rank
-  constexpr Score BlockedPawn[2] = { S(-19, -8), S(-7, 3) };
-
-  constexpr Score BlockedStorm[RANK_NB] = {
-    S(0, 0), S(0, 0), S(64, 75), S(-3, 14), S(-12, 19), S(-7, 4), S(-10, 5)
-  };
-
-  // Connected pawn bonus
-  constexpr int Connected[RANK_NB] = { 0, 3, 7, 7, 15, 54, 86 };
-
-  // Strength of pawn shelter for our king by [distance from edge][rank].
-  // RANK_1 = 0 is used for files where we have no pawn, or pawn is behind our king.
-  constexpr Value ShelterStrength[int(FILE_NB) / 2][RANK_NB] = {
-    { V(-2), V(85), V(95), V(53), V(39), V(23), V(25) },
-    { V(-55), V(64), V(32), V(-55), V(-30), V(-11), V(-61) },
-    { V(-11), V(75), V(19), V(-6), V(26), V(9), V(-47) },
-    { V(-41), V(-11), V(-27), V(-58), V(-42), V(-66), V(-163) }
-  };
-
-  // Danger of enemy pawns moving toward our king by [distance from edge][rank].
-  // RANK_1 = 0 is used for files where the enemy has no pawn, or their pawn
-  // is behind our king. Note that UnblockedStorm[0][1-2] accommodate opponent pawn
-  // on edge, likely blocked by our king.
-  constexpr Value UnblockedStorm[int(FILE_NB) / 2][RANK_NB] = {
-    { V(94), V(-280), V(-170), V(90), V(59), V(47), V(53) },
-    { V(43), V(-17), V(128), V(39), V(26), V(-17), V(15) },
-    { V(-9), V(62), V(170), V(34), V(-5), V(-20), V(-11) },
-    { V(-27), V(-19), V(106), V(10), V(2), V(-13), V(-24) }
-  };
-
-
-  // KingOnFile[semi-open Us][semi-open Them] contains bonuses/penalties
-  // for king when the king is on a semi-open or open file.
-  constexpr Score KingOnFile[2][2] = {{ S(-18,11), S(-6,-3)  },
-                                     {  S(  0, 0), S( 5,-4) }};
-
-  #undef S
-  #undef V
-
-
-  /// evaluate() calculates a score for the static pawn structure of the given position.
-  /// We cannot use the location of pieces or king in this function, as the evaluation
-  /// of the pawn structure will be stored in a small cache for speed reasons, and will
-  /// be re-used even when the pieces have moved.
-
-  template<Color Us>
-  Score evaluate(const Position& pos, Pawns::Entry* e) {
-
-    constexpr Color     Them = ~Us;
-    constexpr Direction Up   = pawn_push(Us);
-    constexpr Direction Down = -Up;
-
-    Bitboard neighbours, stoppers, support, phalanx, opposed;
-    Bitboard lever, leverPush, blocked;
-    Square s;
-    bool backward, passed, doubled;
-    Score score = SCORE_ZERO;
-    Bitboard b = pos.pieces(Us, PAWN);
-
-    Bitboard ourPawns   = pos.pieces(  Us, PAWN);
-    Bitboard theirPawns = pos.pieces(Them, PAWN);
-
-    Bitboard doubleAttackThem = pawn_double_attacks_bb<Them>(theirPawns);
-
-    e->passedPawns[Us] = 0;
-    e->kingSquares[Us] = SQ_NONE;
-    e->pawnAttacks[Us] = e->pawnAttacksSpan[Us] = pawn_attacks_bb<Us>(ourPawns);
-    e->blockedCount += popcount(shift<Up>(ourPawns) & (theirPawns | doubleAttackThem));
-
-    // Loop through all pawns of the current color and score each pawn
-    while (b)
-    {
-        s = pop_lsb(b);
-
-        assert(pos.piece_on(s) == make_piece(Us, PAWN));
-
-        Rank r = relative_rank(Us, s);
-
-        // Flag the pawn
-        opposed    = theirPawns & forward_file_bb(Us, s);
-        blocked    = theirPawns & (s + Up);
-        stoppers   = theirPawns & passed_pawn_span(Us, s);
-        lever      = theirPawns & pawn_attacks_bb(Us, s);
-        leverPush  = theirPawns & pawn_attacks_bb(Us, s + Up);
-        doubled    = ourPawns   & (s - Up);
-        neighbours = ourPawns   & adjacent_files_bb(s);
-        phalanx    = neighbours & rank_bb(s);
-        support    = neighbours & rank_bb(s - Up);
-
-        if (doubled)
-        {
-            // Additional doubled penalty if none of their pawns is fixed
-            if (!(ourPawns & shift<Down>(theirPawns | pawn_attacks_bb<Them>(theirPawns))))
-                score -= DoubledEarly;
-        }
-
-        // A pawn is backward when it is behind all pawns of the same color on
-        // the adjacent files and cannot safely advance.
-        backward =  !(neighbours & forward_ranks_bb(Them, s + Up))
-                  && (leverPush | blocked);
-
-        // Compute additional span if pawn is not backward nor blocked
-        if (!backward && !blocked)
-            e->pawnAttacksSpan[Us] |= pawn_attack_span(Us, s);
-
-        // A pawn is passed if one of the three following conditions is true:
-        // (a) there is no stoppers except some levers
-        // (b) the only stoppers are the leverPush, but we outnumber them
-        // (c) there is only one front stopper which can be levered.
-        //     (Refined in Evaluation::passed)
-        passed =   !(stoppers ^ lever)
-                || (   !(stoppers ^ leverPush)
-                    && popcount(phalanx) >= popcount(leverPush))
-                || (   stoppers == blocked && r >= RANK_5
-                    && (shift<Up>(support) & ~(theirPawns | doubleAttackThem)));
-
-        passed &= !(forward_file_bb(Us, s) & ourPawns);
-
-        // Passed pawns will be properly scored later in evaluation when we have
-        // full attack info.
-        if (passed)
-            e->passedPawns[Us] |= s;
-
-        // Score this pawn
-        if (support | phalanx)
-        {
-            int v =  Connected[r] * (2 + bool(phalanx) - bool(opposed))
-                   + 22 * popcount(support);
-
-            score += make_score(v, v * (r - 2) / 4);
-        }
-
-        else if (!neighbours)
-        {
-            if (     opposed
-                &&  (ourPawns & forward_file_bb(Them, s))
-                && !(theirPawns & adjacent_files_bb(s)))
-                score -= Doubled;
-            else
-                score -=  Isolated
-                        + WeakUnopposed * !opposed;
-        }
-
-        else if (backward)
-            score -=  Backward
-                    + WeakUnopposed * !opposed * bool(~(FileABB | FileHBB) & s);
-
-        if (!support)
-            score -=  Doubled * doubled
-                    + WeakLever * more_than_one(lever);
-
-        if (blocked && r >= RANK_5)
-            score += BlockedPawn[r - RANK_5];
-    }
-
-    return score;
-  }
-
-} // namespace
-
-namespace Pawns {
-
-
-/// Pawns::probe() looks up the current position's pawns configuration in
-/// the pawns hash table. It returns a pointer to the Entry if the position
-/// is found. Otherwise a new Entry is computed and stored there, so we don't
-/// have to recompute all when the same pawns configuration occurs again.
-
-Entry* probe(const Position& pos) {
-
-  Key key = pos.pawn_key();
-  Entry* e = pos.this_thread()->pawnsTable[key];
-
-  if (e->key == key)
-      return e;
-
-  e->key = key;
-  e->blockedCount = 0;
-  e->scores[WHITE] = evaluate<WHITE>(pos, e);
-  e->scores[BLACK] = evaluate<BLACK>(pos, e);
-
-  return e;
-}
-
-
-/// Entry::evaluate_shelter() calculates the shelter bonus and the storm
-/// penalty for a king, looking at the king file and the two closest files.
-
-template<Color Us>
-Score Entry::evaluate_shelter(const Position& pos, Square ksq) const {
-
-  constexpr Color Them = ~Us;
-
-  Bitboard b = pos.pieces(PAWN) & ~forward_ranks_bb(Them, ksq);
-  Bitboard ourPawns = b & pos.pieces(Us) & ~pawnAttacks[Them];
-  Bitboard theirPawns = b & pos.pieces(Them);
-
-  Score bonus = make_score(5, 5);
-
-  File center = std::clamp(file_of(ksq), FILE_B, FILE_G);
-  for (File f = File(center - 1); f <= File(center + 1); ++f)
-  {
-      b = ourPawns & file_bb(f);
-      int ourRank = b ? relative_rank(Us, frontmost_sq(Them, b)) : 0;
-
-      b = theirPawns & file_bb(f);
-      int theirRank = b ? relative_rank(Us, frontmost_sq(Them, b)) : 0;
-
-      int d = edge_distance(f);
-      bonus += make_score(ShelterStrength[d][ourRank], 0);
-
-      if (ourRank && (ourRank == theirRank - 1))
-          bonus -= BlockedStorm[theirRank];
-      else
-          bonus -= make_score(UnblockedStorm[d][theirRank], 0);
-  }
-
-  // King On File
-  bonus -= KingOnFile[pos.is_on_semiopen_file(Us, ksq)][pos.is_on_semiopen_file(Them, ksq)];
-
-  return bonus;
-}
-
-
-/// Entry::do_king_safety() calculates a bonus for king safety. It is called only
-/// when king square changes, which is about 20% of total king_safety() calls.
-
-template<Color Us>
-Score Entry::do_king_safety(const Position& pos) {
-
-  Square ksq = pos.square<KING>(Us);
-  kingSquares[Us] = ksq;
-  castlingRights[Us] = pos.castling_rights(Us);
-  auto compare = [](Score a, Score b) { return mg_value(a) < mg_value(b); };
-
-  Score shelter = evaluate_shelter<Us>(pos, ksq);
-
-  // If we can castle use the bonus after castling if it is bigger
-
-  if (pos.can_castle(Us & KING_SIDE))
-      shelter = std::max(shelter, evaluate_shelter<Us>(pos, relative_square(Us, SQ_G1)), compare);
-
-  if (pos.can_castle(Us & QUEEN_SIDE))
-      shelter = std::max(shelter, evaluate_shelter<Us>(pos, relative_square(Us, SQ_C1)), compare);
-
-  // In endgame we like to bring our king near our closest pawn
-  Bitboard pawns = pos.pieces(Us, PAWN);
-  int minPawnDist = 6;
-
-  if (pawns & attacks_bb<KING>(ksq))
-      minPawnDist = 1;
-  else while (pawns)
-      minPawnDist = std::min(minPawnDist, distance(ksq, pop_lsb(pawns)));
-
-  return shelter - make_score(0, 16 * minPawnDist);
-}
-
-// Explicit template instantiation
-template Score Entry::do_king_safety<WHITE>(const Position& pos);
-template Score Entry::do_king_safety<BLACK>(const Position& pos);
-
-} // namespace Pawns
-
-} // namespace Stockfish

src/pawns.h

@@ -1,70 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifndef PAWNS_H_INCLUDED
-#define PAWNS_H_INCLUDED
-
-#include "misc.h"
-#include "position.h"
-#include "types.h"
-
-namespace Stockfish::Pawns {
-
-/// Pawns::Entry contains various information about a pawn structure. A lookup
-/// to the pawn hash table (performed by calling the probe function) returns a
-/// pointer to an Entry object.
-
-struct Entry {
-
-  Score pawn_score(Color c) const { return scores[c]; }
-  Bitboard pawn_attacks(Color c) const { return pawnAttacks[c]; }
-  Bitboard passed_pawns(Color c) const { return passedPawns[c]; }
-  Bitboard pawn_attacks_span(Color c) const { return pawnAttacksSpan[c]; }
-  int passed_count() const { return popcount(passedPawns[WHITE] | passedPawns[BLACK]); }
-  int blocked_count() const { return blockedCount; }
-
-  template<Color Us>
-  Score king_safety(const Position& pos) {
-    return  kingSquares[Us] == pos.square<KING>(Us) && castlingRights[Us] == pos.castling_rights(Us)
-          ? kingSafety[Us] : (kingSafety[Us] = do_king_safety<Us>(pos));
-  }
-
-  template<Color Us>
-  Score do_king_safety(const Position& pos);
-
-  template<Color Us>
-  Score evaluate_shelter(const Position& pos, Square ksq) const;
-
-  Key key;
-  Score scores[COLOR_NB];
-  Bitboard passedPawns[COLOR_NB];
-  Bitboard pawnAttacks[COLOR_NB];
-  Bitboard pawnAttacksSpan[COLOR_NB];
-  Square kingSquares[COLOR_NB];
-  Score kingSafety[COLOR_NB];
-  int castlingRights[COLOR_NB];
-  int blockedCount;
-};
-
-typedef HashTable<Entry, 131072> Table;
-
-Entry* probe(const Position& pos);
-
-} // namespace Stockfish::Pawns
-
-#endif // #ifndef PAWNS_H_INCLUDED

src/perft.h

@@ -0,0 +1,69 @@
+/*
+  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
+
+  Stockfish is free software: you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation, either version 3 of the License, or
+  (at your option) any later version.
+
+  Stockfish is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#ifndef PERFT_H_INCLUDED
+#define PERFT_H_INCLUDED
+
+#include <cstdint>
+
+#include "movegen.h"
+#include "position.h"
+#include "types.h"
+#include "uci.h"
+
+namespace Stockfish {
+
+// Utility to verify move generation. All the leaf nodes up
+// to the given depth are generated and counted, and the sum is returned.
+template<bool Root>
+uint64_t perft(Position& pos, Depth depth) {
+
+    StateInfo st;
+    ASSERT_ALIGNED(&st, Eval::NNUE::CacheLineSize);
+
+    uint64_t   cnt, nodes = 0;
+    const bool leaf = (depth == 2);
+
+    for (const auto& m : MoveList<LEGAL>(pos))
+    {
+        if (Root && depth <= 1)
+            cnt = 1, nodes++;
+        else
+        {
+            pos.do_move(m, st);
+            cnt = leaf ? MoveList<LEGAL>(pos).size() : perft<false>(pos, depth - 1);
+            nodes += cnt;
+            pos.undo_move(m);
+        }
+        if (Root)
+            sync_cout << UCI::move(m, pos.is_chess960()) << ": " << cnt << sync_endl;
+    }
+    return nodes;
+}
+
+inline void perft(const std::string& fen, Depth depth, bool isChess960) {
+    StateListPtr states(new std::deque<StateInfo>(1));
+    Position     p;
+    p.set(fen, isChess960, &states->back());
+
+    uint64_t nodes = perft<true>(p, depth);
+    sync_cout << "\nNodes searched: " << nodes << "\n" << sync_endl;
+}
+}
+
+#endif  // PERFT_H_INCLUDED

src/position.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,445 +21,344 @@
 
 #include <cassert>
 #include <deque>
-#include <memory> // For std::unique_ptr
+#include <iosfwd>
+#include <memory>
 #include <string>
 
 #include "bitboard.h"
-#include "evaluate.h"
-#include "psqt.h"
-#include "types.h"
-
 #include "nnue/nnue_accumulator.h"
-
+#include "nnue/nnue_architecture.h"
-#include "tools/packed_sfen.h"
+#include "types.h"
-#include "tools/sfen_packer.h"
 
 namespace Stockfish {
 
-/// StateInfo struct stores information needed to restore a Position object to
+class TranspositionTable;
-/// its previous state when we retract a move. Whenever a move is made on the
+
-/// board (by calling Position::do_move), a StateInfo object must be passed.
+// StateInfo struct stores information needed to restore a Position object to
+// its previous state when we retract a move. Whenever a move is made on the
+// board (by calling Position::do_move), a StateInfo object must be passed.
 
 struct StateInfo {
 
-  // Copied when making a move
+    // Copied when making a move
-  Key    pawnKey;
+    Key    materialKey;
-  Key    materialKey;
+    Key    pawnKey;
-  Value  nonPawnMaterial[COLOR_NB];
+    Value  nonPawnMaterial[COLOR_NB];
-  int    castlingRights;
+    int    castlingRights;
-  int    rule50;
+    int    rule50;
-  int    pliesFromNull;
+    int    pliesFromNull;
-  Square epSquare;
+    Square epSquare;
 
-  // Not copied when making a move (will be recomputed anyhow)
+    // Not copied when making a move (will be recomputed anyhow)
-  Key        key;
+    Key        key;
-  Bitboard   checkersBB;
+    Bitboard   checkersBB;
-  StateInfo* previous;
+    StateInfo* previous;
-  Bitboard   blockersForKing[COLOR_NB];
+    Bitboard   blockersForKing[COLOR_NB];
-  Bitboard   pinners[COLOR_NB];
+    Bitboard   pinners[COLOR_NB];
-  Bitboard   checkSquares[PIECE_TYPE_NB];
+    Bitboard   checkSquares[PIECE_TYPE_NB];
-  Piece      capturedPiece;
+    Piece      capturedPiece;
-  int        repetition;
+    int        repetition;
 
-  // Used by NNUE
+    // Used by NNUE
-  Eval::NNUE::Accumulator accumulator;
+    Eval::NNUE::Accumulator<Eval::NNUE::TransformedFeatureDimensionsBig>   accumulatorBig;
-  DirtyPiece dirtyPiece;
+    Eval::NNUE::Accumulator<Eval::NNUE::TransformedFeatureDimensionsSmall> accumulatorSmall;
+    DirtyPiece                                                             dirtyPiece;
 };
 
 
-/// A list to keep track of the position states along the setup moves (from the
+// A list to keep track of the position states along the setup moves (from the
-/// start position to the position just before the search starts). Needed by
+// start position to the position just before the search starts). Needed by
-/// 'draw by repetition' detection. Use a std::deque because pointers to
+// 'draw by repetition' detection. Use a std::deque because pointers to
-/// elements are not invalidated upon list resizing.
+// elements are not invalidated upon list resizing.
-typedef std::unique_ptr<std::deque<StateInfo>> StateListPtr;
+using StateListPtr = std::unique_ptr<std::deque<StateInfo>>;
 
 
-/// Position class stores information regarding the board representation as
+// Position class stores information regarding the board representation as
-/// pieces, side to move, hash keys, castling info, etc. Important methods are
+// pieces, side to move, hash keys, castling info, etc. Important methods are
-/// do_move() and undo_move(), used by the search to update node info when
+// do_move() and undo_move(), used by the search to update node info when
-/// traversing the search tree.
+// traversing the search tree.
-class Thread;
-
 class Position {
-public:
+   public:
-  static void init();
+    static void init();
 
-  Position() = default;
+    Position()                           = default;
-  Position(const Position&) = delete;
+    Position(const Position&)            = delete;
-  Position& operator=(const Position&) = delete;
+    Position& operator=(const Position&) = delete;
 
-  // FEN string input/output
+    // FEN string input/output
-  Position& set(const std::string& fenStr, bool isChess960, StateInfo* si, Thread* th);
+    Position&   set(const std::string& fenStr, bool isChess960, StateInfo* si);
-  Position& set(const std::string& code, Color c, StateInfo* si);
+    Position&   set(const std::string& code, Color c, StateInfo* si);
-  std::string fen() const;
+    std::string fen() const;
 
-  // Position representation
+    // Position representation
-  Bitboard pieces(PieceType pt) const;
+    Bitboard pieces(PieceType pt = ALL_PIECES) const;
-  Bitboard pieces(PieceType pt1, PieceType pt2) const;
+    template<typename... PieceTypes>
-  Bitboard pieces(Color c) const;
+    Bitboard pieces(PieceType pt, PieceTypes... pts) const;
-  Bitboard pieces(Color c, PieceType pt) const;
+    Bitboard pieces(Color c) const;
-  Bitboard pieces(Color c, PieceType pt1, PieceType pt2) const;
+    template<typename... PieceTypes>
-  Piece piece_on(Square s) const;
+    Bitboard pieces(Color c, PieceTypes... pts) const;
-  Square ep_square() const;
+    Piece    piece_on(Square s) const;
-  bool empty(Square s) const;
+    Square   ep_square() const;
-  template<PieceType Pt> int count(Color c) const;
+    bool     empty(Square s) const;
-  template<PieceType Pt> int count() const;
+    template<PieceType Pt>
-  template<PieceType Pt> Square square(Color c) const;
+    int count(Color c) const;
-  bool is_on_semiopen_file(Color c, Square s) const;
+    template<PieceType Pt>
+    int count() const;
+    template<PieceType Pt>
+    Square square(Color c) const;
 
-  // Castling
+    // Castling
-  CastlingRights castling_rights(Color c) const;
+    CastlingRights castling_rights(Color c) const;
-  bool can_castle(CastlingRights cr) const;
+    bool           can_castle(CastlingRights cr) const;
-  bool castling_impeded(CastlingRights cr) const;
+    bool           castling_impeded(CastlingRights cr) const;
-  Square castling_rook_square(CastlingRights cr) const;
+    Square         castling_rook_square(CastlingRights cr) const;
 
-  // Checking
+    // Checking
-  Bitboard checkers() const;
+    Bitboard checkers() const;
-  Bitboard blockers_for_king(Color c) const;
+    Bitboard blockers_for_king(Color c) const;
-  Bitboard check_squares(PieceType pt) const;
+    Bitboard check_squares(PieceType pt) const;
-  Bitboard pinners(Color c) const;
+    Bitboard pinners(Color c) const;
 
-  // Attacks to/from a given square
+    // Attacks to/from a given square
-  Bitboard attackers_to(Square s) const;
+    Bitboard attackers_to(Square s) const;
-  Bitboard attackers_to(Square s, Bitboard occupied) const;
+    Bitboard attackers_to(Square s, Bitboard occupied) const;
-  Bitboard slider_blockers(Bitboard sliders, Square s, Bitboard& pinners) const;
+    void     update_slider_blockers(Color c) const;
-  template<PieceType Pt> Bitboard attacks_by(Color c) const;
+    template<PieceType Pt>
+    Bitboard attacks_by(Color c) const;
 
-  // Properties of moves
+    // Properties of moves
-  bool legal(Move m) const;
+    bool  legal(Move m) const;
-  bool pseudo_legal(const Move m) const;
+    bool  pseudo_legal(const Move m) const;
-  bool capture(Move m) const;
+    bool  capture(Move m) const;
-  bool capture_or_promotion(Move m) const;
+    bool  capture_stage(Move m) const;
-  bool gives_check(Move m) const;
+    bool  gives_check(Move m) const;
-  Piece moved_piece(Move m) const;
+    Piece moved_piece(Move m) const;
-  Piece captured_piece() const;
+    Piece captured_piece() const;
 
-  // Piece specific
+    // Doing and undoing moves
-  bool pawn_passed(Color c, Square s) const;
+    void do_move(Move m, StateInfo& newSt);
-  bool opposite_bishops() const;
+    void do_move(Move m, StateInfo& newSt, bool givesCheck);
-  int  pawns_on_same_color_squares(Color c, Square s) const;
+    void undo_move(Move m);
+    void do_null_move(StateInfo& newSt, TranspositionTable& tt);
+    void undo_null_move();
 
-  // Doing and undoing moves
+    // Static Exchange Evaluation
-  void do_move(Move m, StateInfo& newSt);
+    bool see_ge(Move m, int threshold = 0) const;
-  void do_move(Move m, StateInfo& newSt, bool givesCheck);
-  void undo_move(Move m);
-  void do_null_move(StateInfo& newSt);
-  void undo_null_move();
 
-  // Static Exchange Evaluation
+    // Accessing hash keys
-  bool see_ge(Move m, Value threshold = VALUE_ZERO) const;
+    Key key() const;
+    Key key_after(Move m) const;
+    Key material_key() const;
+    Key pawn_key() const;
 
-  // Accessing hash keys
+    // Other properties of the position
-  Key key() const;
+    Color side_to_move() const;
-  Key key_after(Move m) const;
+    int   game_ply() const;
-  Key material_key() const;
+    bool  is_chess960() const;
-  Key pawn_key() const;
+    bool  is_draw(int ply) const;
+    bool  has_game_cycle(int ply) const;
+    bool  has_repeated() const;
+    int   rule50_count() const;
+    Value non_pawn_material(Color c) const;
+    Value non_pawn_material() const;
 
-  // Other properties of the position
+    // Position consistency check, for debugging
-  Color side_to_move() const;
+    bool pos_is_ok() const;
-  int game_ply() const;
+    void flip();
-  bool is_chess960() const;
-  Thread* this_thread() const;
-  bool is_draw(int ply) const;
-  bool is_fifty_move_draw() const;
-  bool is_three_fold_repetition() const;
-  bool has_game_cycle(int ply) const;
-  bool has_repeated() const;
-  int rule50_count() const;
-  Score psq_score() const;
-  Value non_pawn_material(Color c) const;
-  Value non_pawn_material() const;
 
-  // Position consistency check, for debugging
+    // Used by NNUE
-  bool pos_is_ok() const;
+    StateInfo* state() const;
-  void flip();
 
-  // Used by NNUE
+    void put_piece(Piece pc, Square s);
-  StateInfo* state() const;
+    void remove_piece(Square s);
 
-  // --sfenization helper
+   private:
+    // Initialization helpers (used while setting up a position)
+    void set_castling_right(Color c, Square rfrom);
+    void set_state() const;
+    void set_check_info() const;
 
-  friend int Tools::set_from_packed_sfen(Position& pos, const Tools::PackedSfen& sfen, StateInfo* si, Thread* th, bool frc);
+    // Other helpers
+    void move_piece(Square from, Square to);
+    template<bool Do>
+    void do_castling(Color us, Square from, Square& to, Square& rfrom, Square& rto);
+    template<bool AfterMove>
+    Key adjust_key50(Key k) const;
 
-  // Get the packed sfen. Returns to the buffer specified in the argument.
+    // Data members
-  // Do not include gamePly in pack.
+    Piece      board[SQUARE_NB];
-  void sfen_pack(Tools::PackedSfen& sfen, bool resetCastlingRights);
+    Bitboard   byTypeBB[PIECE_TYPE_NB];
-
+    Bitboard   byColorBB[COLOR_NB];
-  // It is slow to go through sfen, so I made a function to set packed sfen directly.
+    int        pieceCount[PIECE_NB];
-  // Equivalent to pos.set(sfen_unpack(data),si,th);.
+    int        castlingRightsMask[SQUARE_NB];
-  // If there is a problem with the passed phase and there is an error, non-zero is returned.
+    Square     castlingRookSquare[CASTLING_RIGHT_NB];
-  // PackedSfen does not include gamePly so it cannot be restored. If you want to set it, specify it with an argument.
+    Bitboard   castlingPath[CASTLING_RIGHT_NB];
-  int set_from_packed_sfen(const Tools::PackedSfen& sfen, StateInfo* si, Thread* th, bool frc);
+    StateInfo* st;
-
+    int        gamePly;
-  void clear() { std::memset(this, 0, sizeof(Position)); }
+    Color      sideToMove;
-
+    bool       chess960;
-  // Give the board, hand piece, and turn, and return the sfen.
-  //static std::string sfen_from_rawdata(Piece board[81], Hand hands[2], Color turn, int gamePly);
-
-  // Returns the position of the ball on the c side.
-  Square king_square(Color c) const { return lsb(pieces(c, KING)); }
-  
-  void put_piece(Piece pc, Square s);
-  void remove_piece(Square s);
-
-private:
-  // Initialization helpers (used while setting up a position)
-  void set_castling_right(Color c, Square rfrom);
-  void set_state(StateInfo* si) const;
-  void set_check_info(StateInfo* si) const;
-
-  // Other helpers
-  void move_piece(Square from, Square to);
-  template<bool Do>
-  void do_castling(Color us, Square from, Square& to, Square& rfrom, Square& rto);
-
-  // Data members
-  Piece board[SQUARE_NB];
-  Bitboard byTypeBB[PIECE_TYPE_NB];
-  Bitboard byColorBB[COLOR_NB];
-  int pieceCount[PIECE_NB];
-  int castlingRightsMask[SQUARE_NB];
-  Square castlingRookSquare[CASTLING_RIGHT_NB];
-  Bitboard castlingPath[CASTLING_RIGHT_NB];
-  Thread* thisThread;
-  StateInfo* st;
-  int gamePly;
-  Color sideToMove;
-  Score psq;
-  bool chess960;
 };
 
-extern std::ostream& operator<<(std::ostream& os, const Position& pos);
+std::ostream& operator<<(std::ostream& os, const Position& pos);
 
-inline Color Position::side_to_move() const {
+inline Color Position::side_to_move() const { return sideToMove; }
-  return sideToMove;
-}
 
 inline Piece Position::piece_on(Square s) const {
-  assert(is_ok(s));
+    assert(is_ok(s));
-  return board[s];
+    return board[s];
 }
 
-inline bool Position::empty(Square s) const {
+inline bool Position::empty(Square s) const { return piece_on(s) == NO_PIECE; }
-  return piece_on(s) == NO_PIECE;
+
+inline Piece Position::moved_piece(Move m) const { return piece_on(m.from_sq()); }
+
+inline Bitboard Position::pieces(PieceType pt) const { return byTypeBB[pt]; }
+
+template<typename... PieceTypes>
+inline Bitboard Position::pieces(PieceType pt, PieceTypes... pts) const {
+    return pieces(pt) | pieces(pts...);
 }
 
-inline Piece Position::moved_piece(Move m) const {
+inline Bitboard Position::pieces(Color c) const { return byColorBB[c]; }
-  return piece_on(from_sq(m));
+
+template<typename... PieceTypes>
+inline Bitboard Position::pieces(Color c, PieceTypes... pts) const {
+    return pieces(c) & pieces(pts...);
 }
 
-inline Bitboard Position::pieces(PieceType pt = ALL_PIECES) const {
+template<PieceType Pt>
-  return byTypeBB[pt];
+inline int Position::count(Color c) const {
+    return pieceCount[make_piece(c, Pt)];
 }
 
-inline Bitboard Position::pieces(PieceType pt1, PieceType pt2) const {
+template<PieceType Pt>
-  return pieces(pt1) | pieces(pt2);
+inline int Position::count() const {
+    return count<Pt>(WHITE) + count<Pt>(BLACK);
 }
 
-inline Bitboard Position::pieces(Color c) const {
+template<PieceType Pt>
-  return byColorBB[c];
+inline Square Position::square(Color c) const {
+    assert(count<Pt>(c) == 1);
+    return lsb(pieces(c, Pt));
 }
 
-inline Bitboard Position::pieces(Color c, PieceType pt) const {
+inline Square Position::ep_square() const { return st->epSquare; }
-  return pieces(c) & pieces(pt);
-}
 
-inline Bitboard Position::pieces(Color c, PieceType pt1, PieceType pt2) const {
+inline bool Position::can_castle(CastlingRights cr) const { return st->castlingRights & cr; }
-  return pieces(c) & (pieces(pt1) | pieces(pt2));
-}
-
-template<PieceType Pt> inline int Position::count(Color c) const {
-  return pieceCount[make_piece(c, Pt)];
-}
-
-template<PieceType Pt> inline int Position::count() const {
-  return count<Pt>(WHITE) + count<Pt>(BLACK);
-}
-
-template<PieceType Pt> inline Square Position::square(Color c) const {
-  assert(count<Pt>(c) == 1);
-  return lsb(pieces(c, Pt));
-}
-
-inline Square Position::ep_square() const {
-  return st->epSquare;
-}
-
-inline bool Position::is_on_semiopen_file(Color c, Square s) const {
-  return !(pieces(c, PAWN) & file_bb(s));
-}
-
-inline bool Position::can_castle(CastlingRights cr) const {
-  return st->castlingRights & cr;
-}
 
 inline CastlingRights Position::castling_rights(Color c) const {
-  return c & CastlingRights(st->castlingRights);
+    return c & CastlingRights(st->castlingRights);
 }
 
 inline bool Position::castling_impeded(CastlingRights cr) const {
-  assert(cr == WHITE_OO || cr == WHITE_OOO || cr == BLACK_OO || cr == BLACK_OOO);
+    assert(cr == WHITE_OO || cr == WHITE_OOO || cr == BLACK_OO || cr == BLACK_OOO);
-
+    return pieces() & castlingPath[cr];
-  return pieces() & castlingPath[cr];
 }
 
 inline Square Position::castling_rook_square(CastlingRights cr) const {
-  assert(cr == WHITE_OO || cr == WHITE_OOO || cr == BLACK_OO || cr == BLACK_OOO);
+    assert(cr == WHITE_OO || cr == WHITE_OOO || cr == BLACK_OO || cr == BLACK_OOO);
-
+    return castlingRookSquare[cr];
-  return castlingRookSquare[cr];
 }
 
-inline Bitboard Position::attackers_to(Square s) const {
+inline Bitboard Position::attackers_to(Square s) const { return attackers_to(s, pieces()); }
-  return attackers_to(s, pieces());
-}
 
 template<PieceType Pt>
 inline Bitboard Position::attacks_by(Color c) const {
 
-  if constexpr (Pt == PAWN)
+    if constexpr (Pt == PAWN)
-      return c == WHITE ? pawn_attacks_bb<WHITE>(pieces(WHITE, PAWN))
+        return c == WHITE ? pawn_attacks_bb<WHITE>(pieces(WHITE, PAWN))
-                        : pawn_attacks_bb<BLACK>(pieces(BLACK, PAWN));
+                          : pawn_attacks_bb<BLACK>(pieces(BLACK, PAWN));
-  else
+    else
-  {
+    {
-      Bitboard threats = 0;
+        Bitboard threats   = 0;
-      Bitboard attackers = pieces(c, Pt);
+        Bitboard attackers = pieces(c, Pt);
-      while (attackers)
+        while (attackers)
-          threats |= attacks_bb<Pt>(pop_lsb(attackers), pieces());
+            threats |= attacks_bb<Pt>(pop_lsb(attackers), pieces());
-      return threats;
+        return threats;
-  }
+    }
 }
 
-inline Bitboard Position::checkers() const {
+inline Bitboard Position::checkers() const { return st->checkersBB; }
-  return st->checkersBB;
+
+inline Bitboard Position::blockers_for_king(Color c) const { return st->blockersForKing[c]; }
+
+inline Bitboard Position::pinners(Color c) const { return st->pinners[c]; }
+
+inline Bitboard Position::check_squares(PieceType pt) const { return st->checkSquares[pt]; }
+
+inline Key Position::key() const { return adjust_key50<false>(st->key); }
+
+template<bool AfterMove>
+inline Key Position::adjust_key50(Key k) const {
+    return st->rule50 < 14 - AfterMove ? k : k ^ make_key((st->rule50 - (14 - AfterMove)) / 8);
 }
 
-inline Bitboard Position::blockers_for_king(Color c) const {
+inline Key Position::pawn_key() const { return st->pawnKey; }
-  return st->blockersForKing[c];
-}
 
-inline Bitboard Position::pinners(Color c) const {
+inline Key Position::material_key() const { return st->materialKey; }
-  return st->pinners[c];
-}
 
-inline Bitboard Position::check_squares(PieceType pt) const {
+inline Value Position::non_pawn_material(Color c) const { return st->nonPawnMaterial[c]; }
-  return st->checkSquares[pt];
-}
-
-inline bool Position::pawn_passed(Color c, Square s) const {
-  return !(pieces(~c, PAWN) & passed_pawn_span(c, s));
-}
-
-inline int Position::pawns_on_same_color_squares(Color c, Square s) const {
-  return popcount(pieces(c, PAWN) & ((DarkSquares & s) ? DarkSquares : ~DarkSquares));
-}
-
-inline Key Position::key() const {
-  return st->rule50 < 14 ? st->key
-                         : st->key ^ make_key((st->rule50 - 14) / 8);
-}
-
-inline Key Position::pawn_key() const {
-  return st->pawnKey;
-}
-
-inline Key Position::material_key() const {
-  return st->materialKey;
-}
-
-inline Score Position::psq_score() const {
-  return psq;
-}
-
-inline Value Position::non_pawn_material(Color c) const {
-  return st->nonPawnMaterial[c];
-}
 
 inline Value Position::non_pawn_material() const {
-  return non_pawn_material(WHITE) + non_pawn_material(BLACK);
+    return non_pawn_material(WHITE) + non_pawn_material(BLACK);
 }
 
-inline int Position::game_ply() const {
+inline int Position::game_ply() const { return gamePly; }
-  return gamePly;
-}
 
-inline int Position::rule50_count() const {
+inline int Position::rule50_count() const { return st->rule50; }
-  return st->rule50;
-}
 
-inline bool Position::opposite_bishops() const {
+inline bool Position::is_chess960() const { return chess960; }
-  return   count<BISHOP>(WHITE) == 1
-        && count<BISHOP>(BLACK) == 1
-        && opposite_colors(square<BISHOP>(WHITE), square<BISHOP>(BLACK));
-}
-
-inline bool Position::is_chess960() const {
-  return chess960;
-}
-
-inline bool Position::capture_or_promotion(Move m) const {
-  assert(is_ok(m));
-  return type_of(m) != NORMAL ? type_of(m) != CASTLING : !empty(to_sq(m));
-}
 
 inline bool Position::capture(Move m) const {
-  assert(is_ok(m));
+    assert(m.is_ok());
-  // Castling is encoded as "king captures rook"
+    return (!empty(m.to_sq()) && m.type_of() != CASTLING) || m.type_of() == EN_PASSANT;
-  return (!empty(to_sq(m)) && type_of(m) != CASTLING) || type_of(m) == EN_PASSANT;
 }
 
-inline Piece Position::captured_piece() const {
+// Returns true if a move is generated from the capture stage, having also
-  return st->capturedPiece;
+// queen promotions covered, i.e. consistency with the capture stage move generation
+// is needed to avoid the generation of duplicate moves.
+inline bool Position::capture_stage(Move m) const {
+    assert(m.is_ok());
+    return capture(m) || m.promotion_type() == QUEEN;
 }
 
-inline Thread* Position::this_thread() const {
+inline Piece Position::captured_piece() const { return st->capturedPiece; }
-  return thisThread;
-}
 
 inline void Position::put_piece(Piece pc, Square s) {
 
-  board[s] = pc;
+    board[s] = pc;
-  byTypeBB[ALL_PIECES] |= byTypeBB[type_of(pc)] |= s;
+    byTypeBB[ALL_PIECES] |= byTypeBB[type_of(pc)] |= s;
-  byColorBB[color_of(pc)] |= s;
+    byColorBB[color_of(pc)] |= s;
-  pieceCount[pc]++;
+    pieceCount[pc]++;
-  pieceCount[make_piece(color_of(pc), ALL_PIECES)]++;
+    pieceCount[make_piece(color_of(pc), ALL_PIECES)]++;
-  psq += PSQT::psq[pc][s];
 }
 
 inline void Position::remove_piece(Square s) {
 
-  Piece pc = board[s];
+    Piece pc = board[s];
-  byTypeBB[ALL_PIECES] ^= s;
+    byTypeBB[ALL_PIECES] ^= s;
-  byTypeBB[type_of(pc)] ^= s;
+    byTypeBB[type_of(pc)] ^= s;
-  byColorBB[color_of(pc)] ^= s;
+    byColorBB[color_of(pc)] ^= s;
-  board[s] = NO_PIECE;
+    board[s] = NO_PIECE;
-  pieceCount[pc]--;
+    pieceCount[pc]--;
-  pieceCount[make_piece(color_of(pc), ALL_PIECES)]--;
+    pieceCount[make_piece(color_of(pc), ALL_PIECES)]--;
-  psq -= PSQT::psq[pc][s];
 }
 
 inline void Position::move_piece(Square from, Square to) {
 
-  Piece pc = board[from];
+    Piece    pc     = board[from];
-  Bitboard fromTo = from | to;
+    Bitboard fromTo = from | to;
-  byTypeBB[ALL_PIECES] ^= fromTo;
+    byTypeBB[ALL_PIECES] ^= fromTo;
-  byTypeBB[type_of(pc)] ^= fromTo;
+    byTypeBB[type_of(pc)] ^= fromTo;
-  byColorBB[color_of(pc)] ^= fromTo;
+    byColorBB[color_of(pc)] ^= fromTo;
-  board[from] = NO_PIECE;
+    board[from] = NO_PIECE;
-  board[to] = pc;
+    board[to]   = pc;
-  psq += PSQT::psq[pc][to] - PSQT::psq[pc][from];
 }
 
-inline void Position::do_move(Move m, StateInfo& newSt) {
+inline void Position::do_move(Move m, StateInfo& newSt) { do_move(m, newSt, gives_check(m)); }
-  do_move(m, newSt, gives_check(m));
-}
 
-inline StateInfo* Position::state() const {
+inline StateInfo* Position::state() const { return st; }
 
-  return st;
+}  // namespace Stockfish
-}
 
-static const char* const StartFEN = "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1";
+#endif  // #ifndef POSITION_H_INCLUDED
-
-} // namespace Stockfish
-
-#endif // #ifndef POSITION_H_INCLUDED

src/psqt.cpp

@@ -1,131 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-
-#include "psqt.h"
-
-#include <algorithm>
-
-#include "bitboard.h"
-#include "types.h"
-
-namespace Stockfish {
-
-namespace
-{
-
-auto constexpr S = make_score;
-
-// 'Bonus' contains Piece-Square parameters.
-// Scores are explicit for files A to D, implicitly mirrored for E to H.
-constexpr Score Bonus[][RANK_NB][int(FILE_NB) / 2] = {
-  { },
-  { },
-  { // Knight
-   { S(-175, -96), S(-92,-65), S(-74,-49), S(-73,-21) },
-   { S( -77, -67), S(-41,-54), S(-27,-18), S(-15,  8) },
-   { S( -61, -40), S(-17,-27), S(  6, -8), S( 12, 29) },
-   { S( -35, -35), S(  8, -2), S( 40, 13), S( 49, 28) },
-   { S( -34, -45), S( 13,-16), S( 44,  9), S( 51, 39) },
-   { S(  -9, -51), S( 22,-44), S( 58,-16), S( 53, 17) },
-   { S( -67, -69), S(-27,-50), S(  4,-51), S( 37, 12) },
-   { S(-201,-100), S(-83,-88), S(-56,-56), S(-26,-17) }
-  },
-  { // Bishop
-   { S(-37,-40), S(-4 ,-21), S( -6,-26), S(-16, -8) },
-   { S(-11,-26), S(  6, -9), S( 13,-12), S(  3,  1) },
-   { S(-5 ,-11), S( 15, -1), S( -4, -1), S( 12,  7) },
-   { S(-4 ,-14), S(  8, -4), S( 18,  0), S( 27, 12) },
-   { S(-8 ,-12), S( 20, -1), S( 15,-10), S( 22, 11) },
-   { S(-11,-21), S(  4,  4), S(  1,  3), S(  8,  4) },
-   { S(-12,-22), S(-10,-14), S(  4, -1), S(  0,  1) },
-   { S(-34,-32), S(  1,-29), S(-10,-26), S(-16,-17) }
-  },
-  { // Rook
-   { S(-31, -9), S(-20,-13), S(-14,-10), S(-5, -9) },
-   { S(-21,-12), S(-13, -9), S( -8, -1), S( 6, -2) },
-   { S(-25,  6), S(-11, -8), S( -1, -2), S( 3, -6) },
-   { S(-13, -6), S( -5,  1), S( -4, -9), S(-6,  7) },
-   { S(-27, -5), S(-15,  8), S( -4,  7), S( 3, -6) },
-   { S(-22,  6), S( -2,  1), S(  6, -7), S(12, 10) },
-   { S( -2,  4), S( 12,  5), S( 16, 20), S(18, -5) },
-   { S(-17, 18), S(-19,  0), S( -1, 19), S( 9, 13) }
-  },
-  { // Queen
-   { S( 3,-69), S(-5,-57), S(-5,-47), S( 4,-26) },
-   { S(-3,-54), S( 5,-31), S( 8,-22), S(12, -4) },
-   { S(-3,-39), S( 6,-18), S(13, -9), S( 7,  3) },
-   { S( 4,-23), S( 5, -3), S( 9, 13), S( 8, 24) },
-   { S( 0,-29), S(14, -6), S(12,  9), S( 5, 21) },
-   { S(-4,-38), S(10,-18), S( 6,-11), S( 8,  1) },
-   { S(-5,-50), S( 6,-27), S(10,-24), S( 8, -8) },
-   { S(-2,-74), S(-2,-52), S( 1,-43), S(-2,-34) }
-  },
-  { // King
-   { S(271,  1), S(327, 45), S(271, 85), S(198, 76) },
-   { S(278, 53), S(303,100), S(234,133), S(179,135) },
-   { S(195, 88), S(258,130), S(169,169), S(120,175) },
-   { S(164,103), S(190,156), S(138,172), S( 98,172) },
-   { S(154, 96), S(179,166), S(105,199), S( 70,199) },
-   { S(123, 92), S(145,172), S( 81,184), S( 31,191) },
-   { S( 88, 47), S(120,121), S( 65,116), S( 33,131) },
-   { S( 59, 11), S( 89, 59), S( 45, 73), S( -1, 78) }
-  }
-};
-
-constexpr Score PBonus[RANK_NB][FILE_NB] =
-  { // Pawn (asymmetric distribution)
-   { },
-   { S(  2, -8), S(  4, -6), S( 11,  9), S( 18,  5), S( 16, 16), S( 21,  6), S(  9, -6), S( -3,-18) },
-   { S( -9, -9), S(-15, -7), S( 11,-10), S( 15,  5), S( 31,  2), S( 23,  3), S(  6, -8), S(-20, -5) },
-   { S( -3,  7), S(-20,  1), S(  8, -8), S( 19, -2), S( 39,-14), S( 17,-13), S(  2,-11), S( -5, -6) },
-   { S( 11, 12), S( -4,  6), S(-11,  2), S(  2, -6), S( 11, -5), S(  0, -4), S(-12, 14), S(  5,  9) },
-   { S(  3, 27), S(-11, 18), S( -6, 19), S( 22, 29), S( -8, 30), S( -5,  9), S(-14,  8), S(-11, 14) },
-   { S( -7, -1), S(  6,-14), S( -2, 13), S(-11, 22), S(  4, 24), S(-14, 17), S( 10,  7), S( -9,  7) }
-  };
-
-} // namespace
-
-
-namespace PSQT
-{
-
-Score psq[PIECE_NB][SQUARE_NB];
-
-// PSQT::init() initializes piece-square tables: the white halves of the tables are
-// copied from Bonus[] and PBonus[], adding the piece value, then the black halves of
-// the tables are initialized by flipping and changing the sign of the white scores.
-void init() {
-
-  for (Piece pc : {W_PAWN, W_KNIGHT, W_BISHOP, W_ROOK, W_QUEEN, W_KING})
-  {
-    Score score = make_score(PieceValue[MG][pc], PieceValue[EG][pc]);
-
-    for (Square s = SQ_A1; s <= SQ_H8; ++s)
-    {
-      File f = File(edge_distance(file_of(s)));
-      psq[ pc][s] = score + (type_of(pc) == PAWN ? PBonus[rank_of(s)][file_of(s)]
-                                                 : Bonus[pc][rank_of(s)][f]);
-      psq[~pc][flip_rank(s)] = -psq[pc][s];
-    }
-  }
-}
-
-} // namespace PSQT
-
-} // namespace Stockfish

src/search.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -19,132 +19,243 @@
 #ifndef SEARCH_H_INCLUDED
 #define SEARCH_H_INCLUDED
 
+#include <array>
+#include <atomic>
+#include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
 #include <vector>
+#include <string>
 
 #include "misc.h"
 #include "movepick.h"
+#include "position.h"
+#include "syzygy/tbprobe.h"
+#include "timeman.h"
 #include "types.h"
-#include "uci.h"
 
 namespace Stockfish {
 
-class Position;
+// Different node types, used as a template parameter
+enum NodeType {
+    NonPV,
+    PV,
+    Root
+};
+
+class TranspositionTable;
+class ThreadPool;
+class OptionsMap;
 
 namespace Search {
 
-/// Threshold used for countermoves based pruning
+// Stack struct keeps track of the information we need to remember from nodes
-constexpr int CounterMovePruneThreshold = 0;
+// shallower and deeper in the tree during the search. Each search thread has
-
+// its own array of Stack objects, indexed by the current ply.
-extern bool prune_at_shallow_depth;
-
-/// Stack struct keeps track of the information we need to remember from nodes
-/// shallower and deeper in the tree during the search. Each search thread has
-/// its own array of Stack objects, indexed by the current ply.
-
 struct Stack {
-  Move* pv;
+    Move*           pv;
-  PieceToHistory* continuationHistory;
+    PieceToHistory* continuationHistory;
-  int ply;
+    int             ply;
-  Move currentMove;
+    Move            currentMove;
-  Move excludedMove;
+    Move            excludedMove;
-  Move killers[2];
+    Move            killers[2];
-  Value staticEval;
+    Value           staticEval;
-  int statScore;
+    int             statScore;
-  int moveCount;
+    int             moveCount;
-  bool inCheck;
+    bool            inCheck;
-  bool ttPv;
+    bool            ttPv;
-  bool ttHit;
+    bool            ttHit;
-  int doubleExtensions;
+    int             multipleExtensions;
-  int cutoffCnt;
+    int             cutoffCnt;
 };
 
 
-/// RootMove struct is used for moves at the root of the tree. For each root move
+// RootMove struct is used for moves at the root of the tree. For each root move
-/// we store a score and a PV (really a refutation in the case of moves which
+// we store a score and a PV (really a refutation in the case of moves which
-/// fail low). Score is normally set at -VALUE_INFINITE for all non-pv moves.
+// fail low). Score is normally set at -VALUE_INFINITE for all non-pv moves.
-
 struct RootMove {
 
-  explicit RootMove(Move m) : pv(1, m) {}
+    explicit RootMove(Move m) :
-  bool extract_ponder_from_tt(Position& pos);
+        pv(1, m) {}
-  bool operator==(const Move& m) const { return pv[0] == m; }
+    bool extract_ponder_from_tt(const TranspositionTable& tt, Position& pos);
-  bool operator<(const RootMove& m) const { // Sort in descending order
+    bool operator==(const Move& m) const { return pv[0] == m; }
-    return m.score != score ? m.score < score
+    // Sort in descending order
-                            : m.previousScore < previousScore;
+    bool operator<(const RootMove& m) const {
-  }
+        return m.score != score ? m.score < score : m.previousScore < previousScore;
+    }
 
-  Value score = -VALUE_INFINITE;
+    Value             score           = -VALUE_INFINITE;
-  Value previousScore = -VALUE_INFINITE;
+    Value             previousScore   = -VALUE_INFINITE;
-  Value averageScore = -VALUE_INFINITE;
+    Value             averageScore    = -VALUE_INFINITE;
-  int selDepth = 0;
+    Value             uciScore        = -VALUE_INFINITE;
-  int tbRank = 0;
+    bool              scoreLowerbound = false;
-  Value tbScore;
+    bool              scoreUpperbound = false;
-  std::vector<Move> pv;
+    int               selDepth        = 0;
+    int               tbRank          = 0;
+    Value             tbScore;
+    std::vector<Move> pv;
 };
 
-typedef std::vector<RootMove> RootMoves;
+using RootMoves = std::vector<RootMove>;
 
 
-/// LimitsType struct stores information sent by GUI about available time to
+// LimitsType struct stores information sent by GUI about available time to
-/// search the current move, maximum depth/time, or if we are in analysis mode.
+// search the current move, maximum depth/time, or if we are in analysis mode.
-
 struct LimitsType {
 
-  LimitsType() { // Init explicitly due to broken value-initialization of non POD in MSVC
+    // Init explicitly due to broken value-initialization of non POD in MSVC
-    time[WHITE] = time[BLACK] = inc[WHITE] = inc[BLACK] = npmsec = movetime = TimePoint(0);
+    LimitsType() {
-    movestogo = depth = mate = perft = infinite = 0;
+        time[WHITE] = time[BLACK] = inc[WHITE] = inc[BLACK] = npmsec = movetime = TimePoint(0);
-    nodes = 0;
+        movestogo = depth = mate = perft = infinite = 0;
-    silent = false;
+        nodes                                       = 0;
-  }
+    }
 
-  bool use_time_management() const {
+    bool use_time_management() const { return time[WHITE] || time[BLACK]; }
-    return time[WHITE] || time[BLACK];
-  }
 
-  std::vector<Move> searchmoves;
+    std::vector<Move> searchmoves;
-  TimePoint time[COLOR_NB], inc[COLOR_NB], npmsec, movetime, startTime;
+    TimePoint         time[COLOR_NB], inc[COLOR_NB], npmsec, movetime, startTime;
-  int movestogo, depth, mate, perft, infinite;
+    int               movestogo, depth, mate, perft, infinite;
-  int64_t nodes;
+    uint64_t          nodes;
-  // Silent mode that does not output to the screen (for continuous self-play in process)
-  // Do not output PV at this time.
-  bool silent;
 };
 
-extern LimitsType Limits;
 
-void init();
+// The UCI stores the uci options, thread pool, and transposition table.
-void clear();
+// This struct is used to easily forward data to the Search::Worker class.
+struct SharedState {
+    SharedState(const OptionsMap&   optionsMap,
+                ThreadPool&         threadPool,
+                TranspositionTable& transpositionTable) :
+        options(optionsMap),
+        threads(threadPool),
+        tt(transpositionTable) {}
 
-// A pair of reader and evaluation value. Returned by Tools::search(),Tools::qsearch().
+    const OptionsMap&   options;
-using ValueAndPV = std::pair<Value, std::vector<Move>>;
+    ThreadPool&         threads;
+    TranspositionTable& tt;
+};
 
-ValueAndPV qsearch(Position& pos);
+class Worker;
-ValueAndPV search(Position& pos, int depth_, size_t multiPV = 1, uint64_t nodesLimit = 0);
 
-namespace MCTS {
+// Null Object Pattern, implement a common interface for the SearchManagers.
+// A Null Object will be given to non-mainthread workers.
+class ISearchManager {
+   public:
+    virtual ~ISearchManager() {}
+    virtual void check_time(Search::Worker&) = 0;
+};
 
-  struct MctsContinuation {
+// SearchManager manages the search from the main thread. It is responsible for
-    std::uint64_t numVisits;
+// keeping track of the time, and storing data strictly related to the main thread.
-    Value value;
+class SearchManager: public ISearchManager {
-    float actionValue;
+   public:
-    std::vector<Move> pv;
+    void check_time(Search::Worker& worker) override;
-  };
 
-  ValueAndPV search_mcts(
+    std::string pv(const Search::Worker&     worker,
-    Position& pos,
+                   const ThreadPool&         threads,
-    std::uint64_t nodes,
+                   const TranspositionTable& tt,
-    Depth leafDepth,
+                   Depth                     depth) const;
-    float explorationFactor);
 
-  std::vector<MctsContinuation> search_mcts_multipv(
+    Stockfish::TimeManagement tm;
-    Position& pos,
+    int                       callsCnt;
-    std::uint64_t numPlayouts,
+    std::atomic_bool          ponder;
-    Depth leafDepth,
-    float explorationFactor);
-}
 
-}
+    std::array<Value, 4> iterValue;
+    double               previousTimeReduction;
+    Value                bestPreviousScore;
+    Value                bestPreviousAverageScore;
+    bool                 stopOnPonderhit;
 
-} // namespace Stockfish
+    size_t id;
+};
 
-#endif // #ifndef SEARCH_H_INCLUDED
+class NullSearchManager: public ISearchManager {
+   public:
+    void check_time(Search::Worker&) override {}
+};
+
+// Search::Worker is the class that does the actual search.
+// It is instantiated once per thread, and it is responsible for keeping track
+// of the search history, and storing data required for the search.
+class Worker {
+   public:
+    Worker(SharedState&, std::unique_ptr<ISearchManager>, size_t);
+
+    // Called at instantiation to initialize Reductions tables
+    // Reset histories, usually before a new game
+    void clear();
+
+    // Called when the program receives the UCI 'go' command.
+    // It searches from the root position and outputs the "bestmove".
+    void start_searching();
+
+    bool is_mainthread() const { return thread_idx == 0; }
+
+    // Public because they need to be updatable by the stats
+    CounterMoveHistory    counterMoves;
+    ButterflyHistory      mainHistory;
+    CapturePieceToHistory captureHistory;
+    ContinuationHistory   continuationHistory[2][2];
+    PawnHistory           pawnHistory;
+    CorrectionHistory     correctionHistory;
+
+   private:
+    void iterative_deepening();
+
+    // Main search function for both PV and non-PV nodes
+    template<NodeType nodeType>
+    Value search(Position& pos, Stack* ss, Value alpha, Value beta, Depth depth, bool cutNode);
+
+    // Quiescence search function, which is called by the main search
+    template<NodeType nodeType>
+    Value qsearch(Position& pos, Stack* ss, Value alpha, Value beta, Depth depth = 0);
+
+    Depth reduction(bool i, Depth d, int mn, int delta);
+
+    // Get a pointer to the search manager, only allowed to be called by the
+    // main thread.
+    SearchManager* main_manager() const {
+        assert(thread_idx == 0);
+        return static_cast<SearchManager*>(manager.get());
+    }
+
+    std::array<std::array<uint64_t, SQUARE_NB>, SQUARE_NB> effort;
+
+    LimitsType limits;
+
+    size_t                pvIdx, pvLast;
+    std::atomic<uint64_t> nodes, tbHits, bestMoveChanges;
+    int                   selDepth, nmpMinPly;
+
+    Value optimism[COLOR_NB];
+
+    Position  rootPos;
+    StateInfo rootState;
+    RootMoves rootMoves;
+    Depth     rootDepth, completedDepth;
+    Value     rootDelta;
+
+    size_t thread_idx;
+
+    // Reductions lookup table initialized at startup
+    std::array<int, MAX_MOVES> reductions;  // [depth or moveNumber]
+
+    // The main thread has a SearchManager, the others have a NullSearchManager
+    std::unique_ptr<ISearchManager> manager;
+
+    Tablebases::Config tbConfig;
+
+    const OptionsMap&   options;
+    ThreadPool&         threads;
+    TranspositionTable& tt;
+
+    friend class Stockfish::ThreadPool;
+    friend class SearchManager;
+};
+
+
+}  // namespace Search
+
+}  // namespace Stockfish
+
+#endif  // #ifndef SEARCH_H_INCLUDED

src/simd.h

@@ -1,387 +0,0 @@
-/*
-  Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
-
-  Stockfish is free software: you can redistribute it and/or modify
-  it under the terms of the GNU General Public License as published by
-  the Free Software Foundation, either version 3 of the License, or
-  (at your option) any later version.
-
-  Stockfish is distributed in the hope that it will be useful,
-  but WITHOUT ANY WARRANTY; without even the implied warranty of
-  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  GNU General Public License for more details.
-
-  You should have received a copy of the GNU General Public License
-  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-#ifndef STOCKFISH_SIMD_H_INCLUDED
-#define STOCKFISH_SIMD_H_INCLUDED
-
-#if defined(USE_AVX2)
-# include <immintrin.h>
-
-#elif defined(USE_SSE41)
-# include <smmintrin.h>
-
-#elif defined(USE_SSSE3)
-# include <tmmintrin.h>
-
-#elif defined(USE_SSE2)
-# include <emmintrin.h>
-
-#elif defined(USE_MMX)
-# include <mmintrin.h>
-
-#elif defined(USE_NEON)
-# include <arm_neon.h>
-#endif
-
-// The inline asm is only safe for GCC, where it is necessary to get good codegen.
-// See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101693
-// Clang does fine without it.
-// Play around here: https://godbolt.org/z/7EWqrYq51
-#if (defined(__GNUC__) && !defined(__clang__) && !defined(__INTEL_COMPILER))
-#define USE_INLINE_ASM
-#endif
-
-// Use either the AVX512 or AVX-VNNI version of the VNNI instructions.
-#if defined(USE_AVXVNNI)
-#define VNNI_PREFIX "%{vex%} "
-#else
-#define VNNI_PREFIX ""
-#endif
-
-namespace Stockfish::Simd {
-
-#if defined (USE_AVX512)
-
-    [[maybe_unused]] static int m512_hadd(__m512i sum, int bias) {
-      return _mm512_reduce_add_epi32(sum) + bias;
-    }
-
-    /*
-      Parameters:
-        sum0 = [zmm0.i128[0], zmm0.i128[1], zmm0.i128[2], zmm0.i128[3]]
-        sum1 = [zmm1.i128[0], zmm1.i128[1], zmm1.i128[2], zmm1.i128[3]]
-        sum2 = [zmm2.i128[0], zmm2.i128[1], zmm2.i128[2], zmm2.i128[3]]
-        sum3 = [zmm3.i128[0], zmm3.i128[1], zmm3.i128[2], zmm3.i128[3]]
-
-      Returns:
-        ret = [
-          reduce_add_epi32(zmm0.i128[0]), reduce_add_epi32(zmm1.i128[0]), reduce_add_epi32(zmm2.i128[0]), reduce_add_epi32(zmm3.i128[0]),
-          reduce_add_epi32(zmm0.i128[1]), reduce_add_epi32(zmm1.i128[1]), reduce_add_epi32(zmm2.i128[1]), reduce_add_epi32(zmm3.i128[1]),
-          reduce_add_epi32(zmm0.i128[2]), reduce_add_epi32(zmm1.i128[2]), reduce_add_epi32(zmm2.i128[2]), reduce_add_epi32(zmm3.i128[2]),
-          reduce_add_epi32(zmm0.i128[3]), reduce_add_epi32(zmm1.i128[3]), reduce_add_epi32(zmm2.i128[3]), reduce_add_epi32(zmm3.i128[3])
-        ]
-    */
-    [[maybe_unused]] static __m512i m512_hadd128x16_interleave(
-        __m512i sum0, __m512i sum1, __m512i sum2, __m512i sum3) {
-
-      __m512i sum01a = _mm512_unpacklo_epi32(sum0, sum1);
-      __m512i sum01b = _mm512_unpackhi_epi32(sum0, sum1);
-
-      __m512i sum23a = _mm512_unpacklo_epi32(sum2, sum3);
-      __m512i sum23b = _mm512_unpackhi_epi32(sum2, sum3);
-
-      __m512i sum01 = _mm512_add_epi32(sum01a, sum01b);
-      __m512i sum23 = _mm512_add_epi32(sum23a, sum23b);
-
-      __m512i sum0123a = _mm512_unpacklo_epi64(sum01, sum23);
-      __m512i sum0123b = _mm512_unpackhi_epi64(sum01, sum23);
-
-      return _mm512_add_epi32(sum0123a, sum0123b);
-    }
-
-    [[maybe_unused]] static __m128i m512_haddx4(
-        __m512i sum0, __m512i sum1, __m512i sum2, __m512i sum3,
-        __m128i bias) {
-
-      __m512i sum = m512_hadd128x16_interleave(sum0, sum1, sum2, sum3);
-
-      __m256i sum256lo = _mm512_castsi512_si256(sum);
-      __m256i sum256hi = _mm512_extracti64x4_epi64(sum, 1);
-
-      sum256lo = _mm256_add_epi32(sum256lo, sum256hi);
-
-      __m128i sum128lo = _mm256_castsi256_si128(sum256lo);
-      __m128i sum128hi = _mm256_extracti128_si256(sum256lo, 1);
-
-      return _mm_add_epi32(_mm_add_epi32(sum128lo, sum128hi), bias);
-    }
-
-    [[maybe_unused]] static void m512_add_dpbusd_epi32(
-        __m512i& acc,
-        __m512i a,
-        __m512i b) {
-
-# if defined (USE_VNNI)
-#   if defined (USE_INLINE_ASM)
-      asm(
-        "vpdpbusd %[b], %[a], %[acc]\n\t"
-        : [acc]"+v"(acc)
-        : [a]"v"(a), [b]"vm"(b)
-      );
-#   else
-      acc = _mm512_dpbusd_epi32(acc, a, b);
-#   endif
-# else
-#   if defined (USE_INLINE_ASM)
-      __m512i tmp = _mm512_maddubs_epi16(a, b);
-      asm(
-          "vpmaddwd    %[tmp], %[ones], %[tmp]\n\t"
-          "vpaddd      %[acc], %[tmp], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp]"+&v"(tmp)
-          : [ones]"v"(_mm512_set1_epi16(1))
-      );
-#   else
-      __m512i product0 = _mm512_maddubs_epi16(a, b);
-      product0 = _mm512_madd_epi16(product0, _mm512_set1_epi16(1));
-      acc = _mm512_add_epi32(acc, product0);
-#   endif
-# endif
-    }
-
-    [[maybe_unused]] static void m512_add_dpbusd_epi32x2(
-        __m512i& acc,
-        __m512i a0, __m512i b0,
-        __m512i a1, __m512i b1) {
-
-# if defined (USE_VNNI)
-#   if defined (USE_INLINE_ASM)
-      asm(
-        "vpdpbusd %[b0], %[a0], %[acc]\n\t"
-        "vpdpbusd %[b1], %[a1], %[acc]\n\t"
-        : [acc]"+v"(acc)
-        : [a0]"v"(a0), [b0]"vm"(b0), [a1]"v"(a1), [b1]"vm"(b1)
-      );
-#   else
-      acc = _mm512_dpbusd_epi32(acc, a0, b0);
-      acc = _mm512_dpbusd_epi32(acc, a1, b1);
-#   endif
-# else
-#   if defined (USE_INLINE_ASM)
-      __m512i tmp0 = _mm512_maddubs_epi16(a0, b0);
-      __m512i tmp1 = _mm512_maddubs_epi16(a1, b1);
-      asm(
-          "vpaddsw     %[tmp0], %[tmp1], %[tmp0]\n\t"
-          "vpmaddwd    %[tmp0], %[ones], %[tmp0]\n\t"
-          "vpaddd      %[acc], %[tmp0], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp0]"+&v"(tmp0)
-          : [tmp1]"v"(tmp1), [ones]"v"(_mm512_set1_epi16(1))
-      );
-#   else
-      __m512i product0 = _mm512_maddubs_epi16(a0, b0);
-      __m512i product1 = _mm512_maddubs_epi16(a1, b1);
-      product0 = _mm512_adds_epi16(product0, product1);
-      product0 = _mm512_madd_epi16(product0, _mm512_set1_epi16(1));
-      acc = _mm512_add_epi32(acc, product0);
-#   endif
-# endif
-    }
-
-#endif
-
-#if defined (USE_AVX2)
-
-    [[maybe_unused]] static int m256_hadd(__m256i sum, int bias) {
-      __m128i sum128 = _mm_add_epi32(_mm256_castsi256_si128(sum), _mm256_extracti128_si256(sum, 1));
-      sum128 = _mm_add_epi32(sum128, _mm_shuffle_epi32(sum128, _MM_PERM_BADC));
-      sum128 = _mm_add_epi32(sum128, _mm_shuffle_epi32(sum128, _MM_PERM_CDAB));
-      return _mm_cvtsi128_si32(sum128) + bias;
-    }
-
-    [[maybe_unused]] static __m128i m256_haddx4(
-        __m256i sum0, __m256i sum1, __m256i sum2, __m256i sum3,
-        __m128i bias) {
-
-      sum0 = _mm256_hadd_epi32(sum0, sum1);
-      sum2 = _mm256_hadd_epi32(sum2, sum3);
-
-      sum0 = _mm256_hadd_epi32(sum0, sum2);
-
-      __m128i sum128lo = _mm256_castsi256_si128(sum0);
-      __m128i sum128hi = _mm256_extracti128_si256(sum0, 1);
-
-      return _mm_add_epi32(_mm_add_epi32(sum128lo, sum128hi), bias);
-    }
-
-    [[maybe_unused]] static void m256_add_dpbusd_epi32(
-        __m256i& acc,
-        __m256i a,
-        __m256i b) {
-
-# if defined (USE_VNNI)
-#   if defined (USE_INLINE_ASM)
-      asm(
-        VNNI_PREFIX "vpdpbusd %[b], %[a], %[acc]\n\t"
-        : [acc]"+v"(acc)
-        : [a]"v"(a), [b]"vm"(b)
-      );
-#   else
-      acc = _mm256_dpbusd_epi32(acc, a, b);
-#   endif
-# else
-#   if defined (USE_INLINE_ASM)
-      __m256i tmp = _mm256_maddubs_epi16(a, b);
-      asm(
-          "vpmaddwd    %[tmp], %[ones], %[tmp]\n\t"
-          "vpaddd      %[acc], %[tmp], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp]"+&v"(tmp)
-          : [ones]"v"(_mm256_set1_epi16(1))
-      );
-#   else
-      __m256i product0 = _mm256_maddubs_epi16(a, b);
-      product0 = _mm256_madd_epi16(product0, _mm256_set1_epi16(1));
-      acc = _mm256_add_epi32(acc, product0);
-#   endif
-# endif
-    }
-
-    [[maybe_unused]] static void m256_add_dpbusd_epi32x2(
-        __m256i& acc,
-        __m256i a0, __m256i b0,
-        __m256i a1, __m256i b1) {
-
-# if defined (USE_VNNI)
-#   if defined (USE_INLINE_ASM)
-      asm(
-        VNNI_PREFIX "vpdpbusd %[b0], %[a0], %[acc]\n\t"
-        VNNI_PREFIX "vpdpbusd %[b1], %[a1], %[acc]\n\t"
-        : [acc]"+v"(acc)
-        : [a0]"v"(a0), [b0]"vm"(b0), [a1]"v"(a1), [b1]"vm"(b1)
-      );
-#   else
-      acc = _mm256_dpbusd_epi32(acc, a0, b0);
-      acc = _mm256_dpbusd_epi32(acc, a1, b1);
-#   endif
-# else
-#   if defined (USE_INLINE_ASM)
-      __m256i tmp0 = _mm256_maddubs_epi16(a0, b0);
-      __m256i tmp1 = _mm256_maddubs_epi16(a1, b1);
-      asm(
-          "vpaddsw     %[tmp0], %[tmp1], %[tmp0]\n\t"
-          "vpmaddwd    %[tmp0], %[ones], %[tmp0]\n\t"
-          "vpaddd      %[acc], %[tmp0], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp0]"+&v"(tmp0)
-          : [tmp1]"v"(tmp1), [ones]"v"(_mm256_set1_epi16(1))
-      );
-#   else
-      __m256i product0 = _mm256_maddubs_epi16(a0, b0);
-      __m256i product1 = _mm256_maddubs_epi16(a1, b1);
-      product0 = _mm256_adds_epi16(product0, product1);
-      product0 = _mm256_madd_epi16(product0, _mm256_set1_epi16(1));
-      acc = _mm256_add_epi32(acc, product0);
-#   endif
-# endif
-    }
-
-#endif
-
-#if defined (USE_SSSE3)
-
-    [[maybe_unused]] static int m128_hadd(__m128i sum, int bias) {
-      sum = _mm_add_epi32(sum, _mm_shuffle_epi32(sum, 0x4E)); //_MM_PERM_BADC
-      sum = _mm_add_epi32(sum, _mm_shuffle_epi32(sum, 0xB1)); //_MM_PERM_CDAB
-      return _mm_cvtsi128_si32(sum) + bias;
-    }
-
-    [[maybe_unused]] static __m128i m128_haddx4(
-        __m128i sum0, __m128i sum1, __m128i sum2, __m128i sum3,
-        __m128i bias) {
-
-      sum0 = _mm_hadd_epi32(sum0, sum1);
-      sum2 = _mm_hadd_epi32(sum2, sum3);
-      sum0 = _mm_hadd_epi32(sum0, sum2);
-      return _mm_add_epi32(sum0, bias);
-    }
-
-    [[maybe_unused]] static void m128_add_dpbusd_epi32(
-        __m128i& acc,
-        __m128i a,
-        __m128i b) {
-
-#   if defined (USE_INLINE_ASM)
-      __m128i tmp = _mm_maddubs_epi16(a, b);
-      asm(
-          "pmaddwd    %[ones], %[tmp]\n\t"
-          "paddd      %[tmp], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp]"+&v"(tmp)
-          : [ones]"v"(_mm_set1_epi16(1))
-      );
-#   else
-      __m128i product0 = _mm_maddubs_epi16(a, b);
-      product0 = _mm_madd_epi16(product0, _mm_set1_epi16(1));
-      acc = _mm_add_epi32(acc, product0);
-#   endif
-    }
-
-    [[maybe_unused]] static void m128_add_dpbusd_epi32x2(
-        __m128i& acc,
-        __m128i a0, __m128i b0,
-        __m128i a1, __m128i b1) {
-
-#   if defined (USE_INLINE_ASM)
-      __m128i tmp0 = _mm_maddubs_epi16(a0, b0);
-      __m128i tmp1 = _mm_maddubs_epi16(a1, b1);
-      asm(
-          "paddsw     %[tmp1], %[tmp0]\n\t"
-          "pmaddwd    %[ones], %[tmp0]\n\t"
-          "paddd      %[tmp0], %[acc]\n\t"
-          : [acc]"+v"(acc), [tmp0]"+&v"(tmp0)
-          : [tmp1]"v"(tmp1), [ones]"v"(_mm_set1_epi16(1))
-      );
-#   else
-      __m128i product0 = _mm_maddubs_epi16(a0, b0);
-      __m128i product1 = _mm_maddubs_epi16(a1, b1);
-      product0 = _mm_adds_epi16(product0, product1);
-      product0 = _mm_madd_epi16(product0, _mm_set1_epi16(1));
-      acc = _mm_add_epi32(acc, product0);
-#   endif
-    }
-
-#endif
-
-#if defined (USE_NEON)
-
-    [[maybe_unused]] static int neon_m128_reduce_add_epi32(int32x4_t s) {
-#   if USE_NEON >= 8
-      return vaddvq_s32(s);
-#   else
-      return s[0] + s[1] + s[2] + s[3];
-#   endif
-    }
-
-    [[maybe_unused]] static int neon_m128_hadd(int32x4_t sum, int bias) {
-      return neon_m128_reduce_add_epi32(sum) + bias;
-    }
-
-    [[maybe_unused]] static int32x4_t neon_m128_haddx4(
-        int32x4_t sum0, int32x4_t sum1, int32x4_t sum2, int32x4_t sum3,
-        int32x4_t bias) {
-
-      int32x4_t hsums {
-        neon_m128_reduce_add_epi32(sum0),
-        neon_m128_reduce_add_epi32(sum1),
-        neon_m128_reduce_add_epi32(sum2),
-        neon_m128_reduce_add_epi32(sum3)
-      };
-      return vaddq_s32(hsums, bias);
-    }
-
-    [[maybe_unused]] static void neon_m128_add_dpbusd_epi32x2(
-        int32x4_t& acc,
-        int8x8_t a0, int8x8_t b0,
-        int8x8_t a1, int8x8_t b1) {
-
-      int16x8_t product = vmull_s8(a0, b0);
-      product = vmlal_s8(product, a1, b1);
-      acc = vpadalq_s16(acc, product);
-    }
-
-#endif
-
-}
-
-#endif // STOCKFISH_SIMD_H_INCLUDED

src/syzygy/tbprobe.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -19,60 +19,57 @@
 #ifndef TBPROBE_H
 #define TBPROBE_H
 
-#include <ostream>
+#include <string>
+#include <vector>
 
-#include "../search.h"
+
+namespace Stockfish {
+class Position;
+class OptionsMap;
+
+using Depth = int;
+
+namespace Search {
+struct RootMove;
+using RootMoves = std::vector<RootMove>;
+}
+}
 
 namespace Stockfish::Tablebases {
 
-enum WDLScore {
+struct Config {
-    WDLLoss        = -2, // Loss
+    int   cardinality = 0;
-    WDLBlessedLoss = -1, // Loss, but draw under 50-move rule
+    bool  rootInTB    = false;
-    WDLDraw        =  0, // Draw
+    bool  useRule50   = false;
-    WDLCursedWin   =  1, // Win, but draw under 50-move rule
+    Depth probeDepth  = 0;
-    WDLWin         =  2, // Win
+};
 
-    WDLScoreNone  = -1000
+enum WDLScore {
+    WDLLoss        = -2,  // Loss
+    WDLBlessedLoss = -1,  // Loss, but draw under 50-move rule
+    WDLDraw        = 0,   // Draw
+    WDLCursedWin   = 1,   // Win, but draw under 50-move rule
+    WDLWin         = 2,   // Win
 };
 
 // Possible states after a probing operation
 enum ProbeState {
-    FAIL              =  0, // Probe failed (missing file table)
+    FAIL              = 0,   // Probe failed (missing file table)
-    OK                =  1, // Probe successful
+    OK                = 1,   // Probe successful
-    CHANGE_STM        = -1, // DTZ should check the other side
+    CHANGE_STM        = -1,  // DTZ should check the other side
-    ZEROING_BEST_MOVE =  2  // Best move zeroes DTZ (capture or pawn move)
+    ZEROING_BEST_MOVE = 2    // Best move zeroes DTZ (capture or pawn move)
 };
 
 extern int MaxCardinality;
 
-void init(const std::string& paths);
+
+void     init(const std::string& paths);
 WDLScore probe_wdl(Position& pos, ProbeState* result);
-int probe_dtz(Position& pos, ProbeState* result);
+int      probe_dtz(Position& pos, ProbeState* result);
-bool root_probe(Position& pos, Search::RootMoves& rootMoves);
+bool     root_probe(Position& pos, Search::RootMoves& rootMoves, bool rule50);
-bool root_probe_wdl(Position& pos, Search::RootMoves& rootMoves);
+bool     root_probe_wdl(Position& pos, Search::RootMoves& rootMoves, bool rule50);
-void rank_root_moves(Position& pos, Search::RootMoves& rootMoves);
+Config   rank_root_moves(const OptionsMap& options, Position& pos, Search::RootMoves& rootMoves);
 
-inline std::ostream& operator<<(std::ostream& os, const WDLScore v) {
+}  // namespace Stockfish::Tablebases
-
-    os << (v == WDLLoss        ? "Loss" :
-           v == WDLBlessedLoss ? "Blessed loss" :
-           v == WDLDraw        ? "Draw" :
-           v == WDLCursedWin   ? "Cursed win" :
-           v == WDLWin         ? "Win" : "None");
-
-    return os;
-}
-
-inline std::ostream& operator<<(std::ostream& os, const ProbeState v) {
-
-    os << (v == FAIL              ? "Failed" :
-           v == OK                ? "Success" :
-           v == CHANGE_STM        ? "Probed opponent side" :
-           v == ZEROING_BEST_MOVE ? "Best move zeroes DTZ" : "None");
-
-    return os;
-}
-
-} // namespace Stockfish::Tablebases
 
 #endif

src/thread.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,257 +16,260 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
-#include <cassert>
+#include "thread.h"
 
-#include <algorithm> // For std::count
+#include <algorithm>
+#include <cassert>
+#include <deque>
+#include <memory>
+#include <unordered_map>
+#include <utility>
+#include <array>
+
+#include "misc.h"
 #include "movegen.h"
 #include "search.h"
-#include "thread.h"
-#include "uci.h"
 #include "syzygy/tbprobe.h"
+#include "timeman.h"
 #include "tt.h"
+#include "types.h"
+#include "ucioption.h"
 
 namespace Stockfish {
 
-ThreadPool Threads; // Global object
+// Constructor launches the thread and waits until it goes to sleep
+// in idle_loop(). Note that 'searching' and 'exit' should be already set.
+Thread::Thread(Search::SharedState&                    sharedState,
+               std::unique_ptr<Search::ISearchManager> sm,
+               size_t                                  n) :
+    worker(std::make_unique<Search::Worker>(sharedState, std::move(sm), n)),
+    idx(n),
+    nthreads(sharedState.options["Threads"]),
+    stdThread(&Thread::idle_loop, this) {
 
-
+    wait_for_search_finished();
-/// Thread constructor launches the thread and waits until it goes to sleep
-/// in idle_loop(). Note that 'searching' and 'exit' should be already set.
-
-Thread::Thread(size_t n) : idx(n), stdThread(&Thread::idle_loop, this), maxNodes(0) {
-
-  wait_for_search_finished();
-  wait_for_worker_finished();
 }
 
 
-/// Thread destructor wakes up the thread in idle_loop() and waits
+// Destructor wakes up the thread in idle_loop() and waits
-/// for its termination. Thread should be already waiting.
+// for its termination. Thread should be already waiting.
-
 Thread::~Thread() {
 
-  assert(!searching);
+    assert(!searching);
 
-  exit = true;
+    exit = true;
-  start_searching();
+    start_searching();
-  stdThread.join();
+    stdThread.join();
 }
 
-
+// Wakes up the thread that will start the search
-/// Thread::clear() reset histories, usually before a new game
-
-void Thread::clear() {
-
-  counterMoves.fill(MOVE_NONE);
-  mainHistory.fill(0);
-  captureHistory.fill(0);
-  previousDepth = 0;
-  
-  for (bool inCheck : { false, true })
-      for (StatsType c : { NoCaptures, Captures })
-      {
-          for (auto& to : continuationHistory[inCheck][c])
-                for (auto& h : to)
-                      h->fill(-71);
-          continuationHistory[inCheck][c][NO_PIECE][0]->fill(Search::CounterMovePruneThreshold - 1);
-      }
-}
-
-
-/// Thread::start_searching() wakes up the thread that will start the search
-
 void Thread::start_searching() {
-
+    mutex.lock();
-  std::lock_guard<std::mutex> lk(mutex);
+    searching = true;
-  searching = true;
+    mutex.unlock();   // Unlock before notifying saves a few CPU-cycles
-  cv.notify_one(); // Wake up the thread in idle_loop()
+    cv.notify_one();  // Wake up the thread in idle_loop()
-}
-
-void Thread::execute_with_worker(std::function<void(Thread&)> t)
-{
-  std::lock_guard<std::mutex> lk(mutex);
-  worker = std::move(t);
-  searching = true;
-  cv.notify_one(); // Wake up the thread in idle_loop()
 }
 
 
-/// Thread::wait_for_search_finished() blocks on the condition variable
+// Blocks on the condition variable
-/// until the thread has finished searching.
+// until the thread has finished searching.
-
 void Thread::wait_for_search_finished() {
 
-  std::unique_lock<std::mutex> lk(mutex);
+    std::unique_lock<std::mutex> lk(mutex);
-  cv.wait(lk, [&]{ return !searching; });
+    cv.wait(lk, [&] { return !searching; });
 }
 
 
-void Thread::wait_for_worker_finished() {
+// Thread gets parked here, blocked on the
-
+// condition variable, when it has no work to do.
-  std::unique_lock<std::mutex> lk(mutex);
-  cv.wait(lk, [&]{ return !searching; });
-}
-
-/// Thread::idle_loop() is where the thread is parked, blocked on the
-/// condition variable, when it has no work to do.
 
 void Thread::idle_loop() {
 
-  // If OS already scheduled us on a different group than 0 then don't overwrite
+    // If OS already scheduled us on a different group than 0 then don't overwrite
-  // the choice, eventually we are one of many one-threaded processes running on
+    // the choice, eventually we are one of many one-threaded processes running on
-  // some Windows NUMA hardware, for instance in fishtest. To make it simple,
+    // some Windows NUMA hardware, for instance in fishtest. To make it simple,
-  // just check if running threads are below a threshold, in this case all this
+    // just check if running threads are below a threshold, in this case, all this
-  // NUMA machinery is not needed.
+    // NUMA machinery is not needed.
-  if (Options["Threads"] > 8)
+    if (nthreads > 8)
-      WinProcGroup::bindThisThread(idx);
+        WinProcGroup::bindThisThread(idx);
 
-  while (true)
+    while (true)
-  {
+    {
-      std::unique_lock<std::mutex> lk(mutex);
+        std::unique_lock<std::mutex> lk(mutex);
-      searching = false;
+        searching = false;
-      worker = nullptr;
+        cv.notify_one();  // Wake up anyone waiting for search finished
-      cv.notify_one(); // Wake up anyone waiting for search finished
+        cv.wait(lk, [&] { return searching; });
-      cv.wait(lk, [&]{ return searching; });
 
-      if (exit)
+        if (exit)
-          return;
+            return;
 
-      auto wrk = std::move(worker);
+        lk.unlock();
 
-      lk.unlock();
+        worker->start_searching();
-
+    }
-      if (wrk)
-      {
-        wrk(*this);
-      }
-      else
-      {
-        search();
-      }
-  }
 }
 
-/// ThreadPool::set() creates/destroys threads to match the requested number.
+// Creates/destroys threads to match the requested number.
-/// Created and launched threads will immediately go to sleep in idle_loop.
+// Created and launched threads will immediately go to sleep in idle_loop.
-/// Upon resizing, threads are recreated to allow for binding if necessary.
+// Upon resizing, threads are recreated to allow for binding if necessary.
+void ThreadPool::set(Search::SharedState sharedState) {
 
-void ThreadPool::set(size_t requested) {
+    if (threads.size() > 0)  // destroy any existing thread(s)
+    {
+        main_thread()->wait_for_search_finished();
 
-  if (size() > 0)   // destroy any existing thread(s)
+        while (threads.size() > 0)
-  {
+            delete threads.back(), threads.pop_back();
-      main()->wait_for_search_finished();
+    }
 
-      while (size() > 0)
+    const size_t requested = sharedState.options["Threads"];
-          delete back(), pop_back();
-  }
 
-  if (requested > 0)   // create new thread(s)
+    if (requested > 0)  // create new thread(s)
-  {
+    {
-      push_back(new MainThread(0));
+        threads.push_back(new Thread(
+          sharedState, std::unique_ptr<Search::ISearchManager>(new Search::SearchManager()), 0));
 
-      while (size() < requested)
-          push_back(new Thread(size()));
-      clear();
 
-      // Reallocate the hash with the new threadpool size
+        while (threads.size() < requested)
-      TT.resize(size_t(Options["Hash"]));
+            threads.push_back(new Thread(
+              sharedState, std::unique_ptr<Search::ISearchManager>(new Search::NullSearchManager()),
+              threads.size()));
+        clear();
 
-      // Init thread number dependent search params.
+        main_thread()->wait_for_search_finished();
-      Search::init();
+
-  }
+        // Reallocate the hash with the new threadpool size
+        sharedState.tt.resize(sharedState.options["Hash"], requested);
+    }
 }
 
 
-/// ThreadPool::clear() sets threadPool data to initial values
+// Sets threadPool data to initial values
-
 void ThreadPool::clear() {
 
-  for (Thread* th : *this)
+    for (Thread* th : threads)
-      th->clear();
+        th->worker->clear();
 
-  main()->callsCnt = 0;
+    main_manager()->callsCnt                 = 0;
-  main()->bestPreviousScore = VALUE_INFINITE;
+    main_manager()->bestPreviousScore        = VALUE_INFINITE;
-  main()->bestPreviousAverageScore = VALUE_INFINITE;
+    main_manager()->bestPreviousAverageScore = VALUE_INFINITE;
-  main()->previousTimeReduction = 1.0;
+    main_manager()->previousTimeReduction    = 1.0;
+    main_manager()->tm.clear();
 }
 
-void ThreadPool::execute_with_workers(const std::function<void(Thread&)>& worker)
-{
-  for(Thread* th : *this)
-  {
-    th->execute_with_worker(worker);
-  }
-}
 
-/// ThreadPool::start_thinking() wakes up main thread waiting in idle_loop() and
+// Wakes up main thread waiting in idle_loop() and
-/// returns immediately. Main thread will wake up other threads and start the search.
+// returns immediately. Main thread will wake up other threads and start the search.
+void ThreadPool::start_thinking(const OptionsMap&  options,
+                                Position&          pos,
+                                StateListPtr&      states,
+                                Search::LimitsType limits,
+                                bool               ponderMode) {
 
-void ThreadPool::start_thinking(Position& pos, StateListPtr& states,
+    main_thread()->wait_for_search_finished();
-                                const Search::LimitsType& limits, bool ponderMode) {
 
-  main()->wait_for_search_finished();
+    main_manager()->stopOnPonderhit = stop = abortedSearch = false;
+    main_manager()->ponder                                 = ponderMode;
 
-  main()->stopOnPonderhit = stop = false;
+    increaseDepth = true;
-  increaseDepth = true;
-  main()->ponder = ponderMode;
-  Search::Limits = limits;
-  Search::RootMoves rootMoves;
 
-  for (const auto& m : MoveList<LEGAL>(pos))
+    Search::RootMoves rootMoves;
-      if (   limits.searchmoves.empty()
-          || std::count(limits.searchmoves.begin(), limits.searchmoves.end(), m))
-          rootMoves.emplace_back(m);
 
-  // After ownership transfer 'states' becomes empty, so if we stop the search
+    for (const auto& m : MoveList<LEGAL>(pos))
-  // and call 'go' again without setting a new position states.get() == NULL.
+        if (limits.searchmoves.empty()
-  assert(states.get() || setupStates.get());
+            || std::count(limits.searchmoves.begin(), limits.searchmoves.end(), m))
+            rootMoves.emplace_back(m);
 
-  if (states.get())
+    Tablebases::Config tbConfig = Tablebases::rank_root_moves(options, pos, rootMoves);
-      setupStates = std::move(states); // Ownership transfer, states is now empty
 
-  // We use Position::set() to set root position across threads. But there are
+    // After ownership transfer 'states' becomes empty, so if we stop the search
-  // some StateInfo fields (previous, pliesFromNull, capturedPiece) that cannot
+    // and call 'go' again without setting a new position states.get() == nullptr.
-  // be deduced from a fen string, so set() clears them and they are set from
+    assert(states.get() || setupStates.get());
-  // setupStates->back() later. The rootState is per thread, earlier states are shared
-  // since they are read-only.
-  for (Thread* th : *this)
-  {
-      th->nodes = th->tbHits = th->nmpMinPly = th->bestMoveChanges = 0;
-      th->rootDepth = th->completedDepth = 0;
-      th->rootMoves = rootMoves;
-      th->rootPos.set(pos.fen(), pos.is_chess960(), &th->rootState, th);
-      th->rootState = setupStates->back();
-  }
 
-  main()->start_searching();
+    if (states.get())
+        setupStates = std::move(states);  // Ownership transfer, states is now empty
+
+    // We use Position::set() to set root position across threads. But there are
+    // some StateInfo fields (previous, pliesFromNull, capturedPiece) that cannot
+    // be deduced from a fen string, so set() clears them and they are set from
+    // setupStates->back() later. The rootState is per thread, earlier states are shared
+    // since they are read-only.
+    for (Thread* th : threads)
+    {
+        th->worker->limits = limits;
+        th->worker->nodes = th->worker->tbHits = th->worker->nmpMinPly =
+          th->worker->bestMoveChanges          = 0;
+        th->worker->rootDepth = th->worker->completedDepth = 0;
+        th->worker->rootMoves                              = rootMoves;
+        th->worker->rootPos.set(pos.fen(), pos.is_chess960(), &th->worker->rootState);
+        th->worker->rootState = setupStates->back();
+        th->worker->tbConfig  = tbConfig;
+        th->worker->effort    = {};
+    }
+
+    main_thread()->start_searching();
 }
 
 Thread* ThreadPool::get_best_thread() const {
 
-    Thread* bestThread = front();
+    Thread* bestThread = threads.front();
-    std::map<Move, int64_t> votes;
+    Value   minScore   = VALUE_NONE;
-    Value minScore = VALUE_NONE;
 
-    // Find minimum score of all threads
+    std::unordered_map<Move, int64_t, Move::MoveHash> votes(
-    for (Thread* th: *this)
+      2 * std::min(size(), bestThread->worker->rootMoves.size()));
-        minScore = std::min(minScore, th->rootMoves[0].score);
+
+    // Find the minimum score of all threads
+    for (Thread* th : threads)
+        minScore = std::min(minScore, th->worker->rootMoves[0].score);
 
     // Vote according to score and depth, and select the best thread
-    for (Thread* th : *this)
+    auto thread_voting_value = [minScore](Thread* th) {
-    {
+        return (th->worker->rootMoves[0].score - minScore + 14) * int(th->worker->completedDepth);
-        votes[th->rootMoves[0].pv[0]] +=
+    };
-            (th->rootMoves[0].score - minScore + 14) * int(th->completedDepth);
 
-        if (abs(bestThread->rootMoves[0].score) >= VALUE_TB_WIN_IN_MAX_PLY)
+    for (Thread* th : threads)
+        votes[th->worker->rootMoves[0].pv[0]] += thread_voting_value(th);
+
+    for (Thread* th : threads)
+    {
+        const auto bestThreadScore = bestThread->worker->rootMoves[0].score;
+        const auto newThreadScore  = th->worker->rootMoves[0].score;
+
+        const auto& bestThreadPV = bestThread->worker->rootMoves[0].pv;
+        const auto& newThreadPV  = th->worker->rootMoves[0].pv;
+
+        const auto bestThreadMoveVote = votes[bestThreadPV[0]];
+        const auto newThreadMoveVote  = votes[newThreadPV[0]];
+
+        const bool bestThreadInProvenWin = bestThreadScore >= VALUE_TB_WIN_IN_MAX_PLY;
+        const bool newThreadInProvenWin  = newThreadScore >= VALUE_TB_WIN_IN_MAX_PLY;
+
+        const bool bestThreadInProvenLoss =
+          bestThreadScore != -VALUE_INFINITE && bestThreadScore <= VALUE_TB_LOSS_IN_MAX_PLY;
+        const bool newThreadInProvenLoss =
+          newThreadScore != -VALUE_INFINITE && newThreadScore <= VALUE_TB_LOSS_IN_MAX_PLY;
+
+        // Note that we make sure not to pick a thread with truncated-PV for better viewer experience.
+        const bool betterVotingValue =
+          thread_voting_value(th) * int(newThreadPV.size() > 2)
+          > thread_voting_value(bestThread) * int(bestThreadPV.size() > 2);
+
+        if (bestThreadInProvenWin)
         {
-            // Make sure we pick the shortest mate / TB conversion or stave off mate the longest
+            // Make sure we pick the shortest mate / TB conversion
-            if (th->rootMoves[0].score > bestThread->rootMoves[0].score)
+            if (newThreadScore > bestThreadScore)
                 bestThread = th;
         }
-        else if (   th->rootMoves[0].score >= VALUE_TB_WIN_IN_MAX_PLY
+        else if (bestThreadInProvenLoss)
-                 || (   th->rootMoves[0].score > VALUE_TB_LOSS_IN_MAX_PLY
+        {
-                     && votes[th->rootMoves[0].pv[0]] > votes[bestThread->rootMoves[0].pv[0]]))
+            // Make sure we pick the shortest mated / TB conversion
+            if (newThreadInProvenLoss && newThreadScore < bestThreadScore)
+                bestThread = th;
+        }
+        else if (newThreadInProvenWin || newThreadInProvenLoss
+                 || (newThreadScore > VALUE_TB_LOSS_IN_MAX_PLY
+                     && (newThreadMoveVote > bestThreadMoveVote
+                         || (newThreadMoveVote == bestThreadMoveVote && betterVotingValue))))
             bestThread = th;
     }
 
@@ -274,30 +277,23 @@ Thread* ThreadPool::get_best_thread() const {
 }
 
 
-/// Start non-main threads
+// Start non-main threads
-
+// Will be invoked by main thread after it has started searching
 void ThreadPool::start_searching() {
 
-    for (Thread* th : *this)
+    for (Thread* th : threads)
-        if (th != front())
+        if (th != threads.front())
             th->start_searching();
 }
 
 
-/// Wait for non-main threads
+// Wait for non-main threads
 
 void ThreadPool::wait_for_search_finished() const {
 
-    for (Thread* th : *this)
+    for (Thread* th : threads)
-        if (th != front())
+        if (th != threads.front())
             th->wait_for_search_finished();
 }
 
-
+}  // namespace Stockfish
-void ThreadPool::wait_for_workers_finished() const {
-
-    for (Thread* th : *this)
-        th->wait_for_worker_finished();
-}
-
-} // namespace Stockfish

src/thread.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,206 +21,101 @@
 
 #include <atomic>
 #include <condition_variable>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
 #include <mutex>
-#include <thread>
 #include <vector>
-#include <functional>
 
-#include "material.h"
-#include "movepick.h"
-#include "pawns.h"
 #include "position.h"
 #include "search.h"
 #include "thread_win32_osx.h"
 
 namespace Stockfish {
 
-/// Thread class keeps together all the thread-related stuff. We use
+class OptionsMap;
-/// per-thread pawn and material hash tables so that once we get a
+using Value = int;
-/// pointer to an entry its life time is unlimited and we don't have
-/// to care about someone changing the entry under our feet.
-
-namespace Detail {
-
-  template <typename T>
-  struct TypeIdentity {
-    using Type = T;
-  };
-
-}
 
+// Abstraction of a thread. It contains a pointer to the worker and a native thread.
+// After construction, the native thread is started with idle_loop()
+// waiting for a signal to start searching.
+// When the signal is received, the thread starts searching and when
+// the search is finished, it goes back to idle_loop() waiting for a new signal.
 class Thread {
+   public:
+    Thread(Search::SharedState&, std::unique_ptr<Search::ISearchManager>, size_t);
+    virtual ~Thread();
 
-  std::mutex mutex;
+    void   idle_loop();
-  std::condition_variable cv;
+    void   start_searching();
-  size_t idx;
+    void   wait_for_search_finished();
-  bool exit = false, searching = true; // Set before starting std::thread
+    size_t id() const { return idx; }
-  std::function<void(Thread&)> worker;
-  std::function<void(Position&)> on_eval_callback;
-  NativeThread stdThread;
 
-public:
+    std::unique_ptr<Search::Worker> worker;
-  explicit Thread(size_t);
-  virtual ~Thread();
-  virtual void search();
 
-  // The function object to be executed is taken by value to remove
+   private:
-  // the need for separate lvalue and rvalue overloads.
+    std::mutex              mutex;
-  // The worker thread needs to have ownership of the task
+    std::condition_variable cv;
-  // to be executed because otherwise there's no way to manage its lifetime.
+    size_t                  idx, nthreads;
-  virtual void execute_with_worker(std::function<void(Thread&)> t);
+    bool                    exit = false, searching = true;  // Set before starting std::thread
-
+    NativeThread            stdThread;
-  void clear();
-  void idle_loop();
-  void start_searching();
-  void wait_for_search_finished();
-  size_t id() const { return idx; }
-
-  void wait_for_worker_finished();
-
-  template <typename FuncT>
-  void set_eval_callback(FuncT&& f) { on_eval_callback = std::forward<FuncT>(f); }
-
-  void clear_eval_callback() { on_eval_callback = nullptr; }
-
-  void on_eval() { if (on_eval_callback) on_eval_callback(rootPos); }
-
-  Pawns::Table pawnsTable;
-  Material::Table materialTable;
-  size_t pvIdx, pvLast;
-  RunningAverage complexityAverage;
-  std::atomic<uint64_t> nodes, tbHits, bestMoveChanges;
-  int selDepth, nmpMinPly;
-  Color nmpColor;
-  Value bestValue, optimism[COLOR_NB];
-  uint64_t maxNodes;
-
-  Position rootPos;
-  StateInfo rootState;
-  Search::RootMoves rootMoves;
-  Depth rootDepth, completedDepth, depth, previousDepth;
-  Value rootDelta;
-  CounterMoveHistory counterMoves;
-  ButterflyHistory mainHistory;
-  CapturePieceToHistory captureHistory;
-  ContinuationHistory continuationHistory[2][2];
-  Score trend;
-  int failedHighCnt;
-  bool rootInTB;
-  int Cardinality;
-  bool UseRule50;
-  Depth ProbeDepth;
 };
 
 
-/// MainThread is a derived class specific for main thread
+// ThreadPool struct handles all the threads-related stuff like init, starting,
+// parking and, most importantly, launching a thread. All the access to threads
+// is done through this class.
+class ThreadPool {
 
-struct MainThread : public Thread {
+   public:
+    ~ThreadPool() {
+        // destroy any existing thread(s)
+        if (threads.size() > 0)
+        {
+            main_thread()->wait_for_search_finished();
 
-  using Thread::Thread;
+            while (threads.size() > 0)
-
+                delete threads.back(), threads.pop_back();
-  void search() override;
-  void check_time();
-
-  double previousTimeReduction;
-  Value bestPreviousScore;
-  Value bestPreviousAverageScore;
-  Value iterValue[4];
-  int callsCnt;
-  bool stopOnPonderhit;
-  std::atomic_bool ponder;
-};
-
-
-/// ThreadPool struct handles all the threads-related stuff like init, starting,
-/// parking and, most importantly, launching a thread. All the access to threads
-/// is done through this class.
-
-struct ThreadPool : public std::vector<Thread*> {
-
-  // Each thread gets its own copy of the `worker` function object.
-  // This means that each worker thread will have exclusive access
-  // to the state of the `worker` function object.
-  void execute_with_workers(const std::function<void(Thread&)>& worker);
-
-  template <typename IndexT, typename FuncT>
-  void for_each_index_with_workers(
-    IndexT begin,
-    typename Detail::TypeIdentity<IndexT>::Type end,
-    FuncT func)
-  {
-    // This value must outlive the function call.
-    // It's fairly safe if we make it static
-    // because for_each_index_with_workers
-    // is not reentrant nor thread safe.
-    static std::atomic<IndexT> i_atomic;
-    i_atomic.store(begin);
-
-    execute_with_workers(
-      [end, func](Thread& th) mutable {
-        for(;;) {
-          const auto i = i_atomic.fetch_add(1);
-          if (i >= end)
-            break;
-
-          func(th, i);
         }
-      });
+    }
-  }
 
-  template <typename IndexT, typename FuncT>
+    void
-  void for_each_index_chunk_with_workers(
+    start_thinking(const OptionsMap&, Position&, StateListPtr&, Search::LimitsType, bool = false);
-    IndexT begin,
+    void clear();
-    typename Detail::TypeIdentity<IndexT>::Type end,
+    void set(Search::SharedState);
-    FuncT func)
-  {
-    // This value must outlive the function call.
-    // It's fairly safe if we make it static
-    // because for_each_index_with_workers
-    // is not reentrant nor thread safe.
-    const IndexT size = end - begin;
-    const IndexT chunk_size = (size + this->size()) / this->size();
 
-    execute_with_workers(
+    Search::SearchManager* main_manager() const {
-      [chunk_size, end, func](Thread& th) mutable {
+        return static_cast<Search::SearchManager*>(main_thread()->worker.get()->manager.get());
-        const IndexT thread_id = th.id();
+    };
-        const IndexT offset = chunk_size * thread_id;
+    Thread*  main_thread() const { return threads.front(); }
-        if (offset >= end)
+    uint64_t nodes_searched() const { return accumulate(&Search::Worker::nodes); }
-          return;
+    uint64_t tb_hits() const { return accumulate(&Search::Worker::tbHits); }
+    Thread*  get_best_thread() const;
+    void     start_searching();
+    void     wait_for_search_finished() const;
 
-        const IndexT count = offset + chunk_size > end ? end - offset : chunk_size;
+    std::atomic_bool stop, abortedSearch, increaseDepth;
-        func(th, offset, count);
-      });
-  }
 
-  void start_thinking(Position&, StateListPtr&, const Search::LimitsType&, bool = false);
+    auto cbegin() const noexcept { return threads.cbegin(); }
-  void clear();
+    auto begin() noexcept { return threads.begin(); }
-  void set(size_t);
+    auto end() noexcept { return threads.end(); }
+    auto cend() const noexcept { return threads.cend(); }
+    auto size() const noexcept { return threads.size(); }
+    auto empty() const noexcept { return threads.empty(); }
 
-  MainThread* main()        const { return static_cast<MainThread*>(front()); }
+   private:
-  uint64_t nodes_searched() const { return accumulate(&Thread::nodes); }
+    StateListPtr         setupStates;
-  uint64_t tb_hits()        const { return accumulate(&Thread::tbHits); }
+    std::vector<Thread*> threads;
-  Thread* get_best_thread() const;
-  void start_searching();
-  void wait_for_search_finished() const;
-  void wait_for_workers_finished() const;
 
-  std::atomic_bool stop, increaseDepth;
+    uint64_t accumulate(std::atomic<uint64_t> Search::Worker::*member) const {
 
-private:
+        uint64_t sum = 0;
-  StateListPtr setupStates;
+        for (Thread* th : threads)
-
+            sum += (th->worker.get()->*member).load(std::memory_order_relaxed);
-  uint64_t accumulate(std::atomic<uint64_t> Thread::* member) const {
+        return sum;
-
+    }
-    uint64_t sum = 0;
-    for (Thread* th : *this)
-        sum += (th->*member).load(std::memory_order_relaxed);
-    return sum;
-  }
 };
 
-extern ThreadPool Threads;
+}  // namespace Stockfish
 
-} // namespace Stockfish
+#endif  // #ifndef THREAD_H_INCLUDED
-
-#endif // #ifndef THREAD_H_INCLUDED

src/thread_win32_osx.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -21,54 +21,58 @@
 
 #include <thread>
 
-/// On OSX threads other than the main thread are created with a reduced stack
+// On OSX threads other than the main thread are created with a reduced stack
-/// size of 512KB by default, this is too low for deep searches, which require
+// size of 512KB by default, this is too low for deep searches, which require
-/// somewhat more than 1MB stack, so adjust it to TH_STACK_SIZE.
+// somewhat more than 1MB stack, so adjust it to TH_STACK_SIZE.
-/// The implementation calls pthread_create() with the stack size parameter
+// The implementation calls pthread_create() with the stack size parameter
-/// equal to the linux 8MB default, on platforms that support it.
+// equal to the Linux 8MB default, on platforms that support it.
 
 #if defined(__APPLE__) || defined(__MINGW32__) || defined(__MINGW64__) || defined(USE_PTHREADS)
 
-#include <pthread.h>
+    #include <pthread.h>
+    #include <functional>
 
 namespace Stockfish {
 
-static const size_t TH_STACK_SIZE = 8 * 1024 * 1024;
-
-template <class T, class P = std::pair<T*, void(T::*)()>>
-void* start_routine(void* ptr)
-{
-   P* p = reinterpret_cast<P*>(ptr);
-   (p->first->*(p->second))(); // Call member function pointer
-   delete p;
-   return NULL;
-}
-
 class NativeThread {
+    pthread_t thread;
 
-   pthread_t thread;
+    static constexpr size_t TH_STACK_SIZE = 8 * 1024 * 1024;
 
-public:
+   public:
-  template<class T, class P = std::pair<T*, void(T::*)()>>
+    template<class Function, class... Args>
-  explicit NativeThread(void(T::*fun)(), T* obj) {
+    explicit NativeThread(Function&& fun, Args&&... args) {
-    pthread_attr_t attr_storage, *attr = &attr_storage;
+        auto func = new std::function<void()>(
-    pthread_attr_init(attr);
+          std::bind(std::forward<Function>(fun), std::forward<Args>(args)...));
-    pthread_attr_setstacksize(attr, TH_STACK_SIZE);
+
-    pthread_create(&thread, attr, start_routine<T>, new P(obj, fun));
+        pthread_attr_t attr_storage, *attr = &attr_storage;
-  }
+        pthread_attr_init(attr);
-  void join() { pthread_join(thread, NULL); }
+        pthread_attr_setstacksize(attr, TH_STACK_SIZE);
+
+        auto start_routine = [](void* ptr) -> void* {
+            auto f = reinterpret_cast<std::function<void()>*>(ptr);
+            // Call the function
+            (*f)();
+            delete f;
+            return nullptr;
+        };
+
+        pthread_create(&thread, attr, start_routine, func);
+    }
+
+    void join() { pthread_join(thread, nullptr); }
 };
 
-} // namespace Stockfish
+}  // namespace Stockfish
 
-#else // Default case: use STL classes
+#else  // Default case: use STL classes
 
 namespace Stockfish {
 
-typedef std::thread NativeThread;
+using NativeThread = std::thread;
 
-} // namespace Stockfish
+}  // namespace Stockfish
 
 #endif
 
-#endif // #ifndef THREAD_WIN32_OSX_H_INCLUDED
+#endif  // #ifndef THREAD_WIN32_OSX_H_INCLUDED

src/timeman.cpp

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -16,90 +16,111 @@
   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "timeman.h"
+
 #include <algorithm>
-#include <cfloat>
+#include <cassert>
 #include <cmath>
+#include <cstdint>
 
 #include "search.h"
-#include "timeman.h"
+#include "ucioption.h"
-#include "uci.h"
 
 namespace Stockfish {
 
-TimeManagement Time; // Our global time management object
+TimePoint TimeManagement::optimum() const { return optimumTime; }
-
+TimePoint TimeManagement::maximum() const { return maximumTime; }
-
+TimePoint TimeManagement::elapsed(size_t nodes) const {
-/// TimeManagement::init() is called at the beginning of the search and calculates
+    return useNodesTime ? TimePoint(nodes) : now() - startTime;
-/// the bounds of time allowed for the current game ply. We currently support:
-//      1) x basetime (+ z increment)
-//      2) x moves in y seconds (+ z increment)
-
-void TimeManagement::init(Search::LimitsType& limits, Color us, int ply) {
-
-  TimePoint moveOverhead    = TimePoint(Options["Move Overhead"]);
-  TimePoint slowMover       = TimePoint(Options["Slow Mover"]);
-  TimePoint npmsec          = TimePoint(Options["nodestime"]);
-
-  // optScale is a percentage of available time to use for the current move.
-  // maxScale is a multiplier applied to optimumTime.
-  double optScale, maxScale;
-
-  // If we have to play in 'nodes as time' mode, then convert from time
-  // to nodes, and use resulting values in time management formulas.
-  // WARNING: to avoid time losses, the given npmsec (nodes per millisecond)
-  // must be much lower than the real engine speed.
-  if (npmsec)
-  {
-      if (!availableNodes) // Only once at game start
-          availableNodes = npmsec * limits.time[us]; // Time is in msec
-
-      // Convert from milliseconds to nodes
-      limits.time[us] = TimePoint(availableNodes);
-      limits.inc[us] *= npmsec;
-      limits.npmsec = npmsec;
-  }
-
-  startTime = limits.startTime;
-
-  // Maximum move horizon of 50 moves
-  int mtg = limits.movestogo ? std::min(limits.movestogo, 50) : 50;
-
-  // Make sure timeLeft is > 0 since we may use it as a divisor
-  TimePoint timeLeft =  std::max(TimePoint(1),
-      limits.time[us] + limits.inc[us] * (mtg - 1) - moveOverhead * (2 + mtg));
-
-  // Use extra time with larger increments
-  double optExtra = std::clamp(1.0 + 12.0 * limits.inc[us] / limits.time[us], 1.0, 1.12);
-
-  // A user may scale time usage by setting UCI option "Slow Mover"
-  // Default is 100 and changing this value will probably lose elo.
-  timeLeft = slowMover * timeLeft / 100;
-
-  // x basetime (+ z increment)
-  // If there is a healthy increment, timeLeft can exceed actual available
-  // game time for the current move, so also cap to 20% of available game time.
-  if (limits.movestogo == 0)
-  {
-      optScale = std::min(0.0084 + std::pow(ply + 3.0, 0.5) * 0.0042,
-                           0.2 * limits.time[us] / double(timeLeft))
-                 * optExtra;
-      maxScale = std::min(7.0, 4.0 + ply / 12.0);
-  }
-
-  // x moves in y seconds (+ z increment)
-  else
-  {
-      optScale = std::min((0.88 + ply / 116.4) / mtg,
-                            0.88 * limits.time[us] / double(timeLeft));
-      maxScale = std::min(6.3, 1.5 + 0.11 * mtg);
-  }
-
-  // Never use more than 80% of the available time for this move
-  optimumTime = TimePoint(optScale * timeLeft);
-  maximumTime = TimePoint(std::min(0.8 * limits.time[us] - moveOverhead, maxScale * optimumTime));
-
-  if (Options["Ponder"])
-      optimumTime += optimumTime / 4;
 }
 
-} // namespace Stockfish
+void TimeManagement::clear() {
+    availableNodes = 0;  // When in 'nodes as time' mode
+}
+
+void TimeManagement::advance_nodes_time(std::int64_t nodes) {
+    assert(useNodesTime);
+    availableNodes += nodes;
+}
+
+// Called at the beginning of the search and calculates
+// the bounds of time allowed for the current game ply. We currently support:
+//      1) x basetime (+ z increment)
+//      2) x moves in y seconds (+ z increment)
+void TimeManagement::init(Search::LimitsType& limits,
+                          Color               us,
+                          int                 ply,
+                          const OptionsMap&   options) {
+    // If we have no time, no need to initialize TM, except for the start time,
+    // which is used by movetime.
+    startTime = limits.startTime;
+    if (limits.time[us] == 0)
+        return;
+
+    TimePoint moveOverhead = TimePoint(options["Move Overhead"]);
+    TimePoint npmsec       = TimePoint(options["nodestime"]);
+
+    // optScale is a percentage of available time to use for the current move.
+    // maxScale is a multiplier applied to optimumTime.
+    double optScale, maxScale;
+
+    // If we have to play in 'nodes as time' mode, then convert from time
+    // to nodes, and use resulting values in time management formulas.
+    // WARNING: to avoid time losses, the given npmsec (nodes per millisecond)
+    // must be much lower than the real engine speed.
+    if (npmsec)
+    {
+        useNodesTime = true;
+
+        if (!availableNodes)                            // Only once at game start
+            availableNodes = npmsec * limits.time[us];  // Time is in msec
+
+        // Convert from milliseconds to nodes
+        limits.time[us] = TimePoint(availableNodes);
+        limits.inc[us] *= npmsec;
+        limits.npmsec = npmsec;
+    }
+
+    // Maximum move horizon of 50 moves
+    int mtg = limits.movestogo ? std::min(limits.movestogo, 50) : 50;
+
+    // Make sure timeLeft is > 0 since we may use it as a divisor
+    TimePoint timeLeft = std::max(TimePoint(1), limits.time[us] + limits.inc[us] * (mtg - 1)
+                                                  - moveOverhead * (2 + mtg));
+
+    // x basetime (+ z increment)
+    // If there is a healthy increment, timeLeft can exceed actual available
+    // game time for the current move, so also cap to 20% of available game time.
+    if (limits.movestogo == 0)
+    {
+        // Use extra time with larger increments
+        double optExtra = std::clamp(1.0 + 12.5 * limits.inc[us] / limits.time[us], 1.0, 1.11);
+
+        // Calculate time constants based on current time left.
+        double optConstant =
+          std::min(0.00334 + 0.0003 * std::log10(limits.time[us] / 1000.0), 0.0049);
+        double maxConstant = std::max(3.4 + 3.0 * std::log10(limits.time[us] / 1000.0), 2.76);
+
+        optScale = std::min(0.0120 + std::pow(ply + 3.1, 0.44) * optConstant,
+                            0.21 * limits.time[us] / double(timeLeft))
+                 * optExtra;
+        maxScale = std::min(6.9, maxConstant + ply / 12.2);
+    }
+
+    // x moves in y seconds (+ z increment)
+    else
+    {
+        optScale = std::min((0.88 + ply / 116.4) / mtg, 0.88 * limits.time[us] / double(timeLeft));
+        maxScale = std::min(6.3, 1.5 + 0.11 * mtg);
+    }
+
+    // Limit the maximum possible time for this move
+    optimumTime = TimePoint(optScale * timeLeft);
+    maximumTime =
+      TimePoint(std::min(0.84 * limits.time[us] - moveOverhead, maxScale * optimumTime)) - 10;
+
+    if (options["Ponder"])
+        optimumTime += optimumTime / 4;
+}
+
+}  // namespace Stockfish

src/timeman.h

@@ -1,6 +1,6 @@
 /*
   Stockfish, a UCI chess playing engine derived from Glaurung 2.1
-  Copyright (C) 2004-2022 The Stockfish developers (see AUTHORS file)
+  Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
 
   Stockfish is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
@@ -19,33 +19,42 @@
 #ifndef TIMEMAN_H_INCLUDED
 #define TIMEMAN_H_INCLUDED
 
+#include <cstddef>
+#include <cstdint>
+
 #include "misc.h"
-#include "search.h"
+#include "types.h"
-#include "thread.h"
 
 namespace Stockfish {
 
-/// The TimeManagement class computes the optimal time to think depending on
+class OptionsMap;
-/// the maximum available time, the game move number and other parameters.
 
+namespace Search {
+struct LimitsType;
+}
+
+// The TimeManagement class computes the optimal time to think depending on
+// the maximum available time, the game move number, and other parameters.
 class TimeManagement {
-public:
+   public:
-  void init(Search::LimitsType& limits, Color us, int ply);
+    void init(Search::LimitsType& limits, Color us, int ply, const OptionsMap& options);
-  TimePoint optimum() const { return optimumTime; }
-  TimePoint maximum() const { return maximumTime; }
-  TimePoint elapsed() const { return Search::Limits.npmsec ?
-                                     TimePoint(Threads.nodes_searched()) : now() - startTime; }
 
-  int64_t availableNodes; // When in 'nodes as time' mode
+    TimePoint optimum() const;
+    TimePoint maximum() const;
+    TimePoint elapsed(std::size_t nodes) const;
 
-private:
+    void clear();
-  TimePoint startTime;
+    void advance_nodes_time(std::int64_t nodes);
-  TimePoint optimumTime;
+
-  TimePoint maximumTime;
+   private:
+    TimePoint startTime;
+    TimePoint optimumTime;
+    TimePoint maximumTime;
+
+    std::int64_t availableNodes = 0;      // When in 'nodes as time' mode
+    bool         useNodesTime   = false;  // True if we are in 'nodes as time' mode
 };
 
-extern TimeManagement Time;
+}  // namespace Stockfish
 
-} // namespace Stockfish
+#endif  // #ifndef TIMEMAN_H_INCLUDED
-
-#endif // #ifndef TIMEMAN_H_INCLUDED

src/tools/convert.cpp

@@ -1,815 +0,0 @@
-#include "convert.h"
-
-#include "uci.h"
-#include "misc.h"
-#include "thread.h"
-#include "position.h"
-#include "tt.h"
-
-#include "extra/nnue_data_binpack_format.h"
-
-#include "nnue/evaluate_nnue.h"
-
-#include "syzygy/tbprobe.h"
-
-#include <sstream>
-#include <fstream>
-#include <unordered_set>
-#include <iomanip>
-#include <list>
-#include <cmath>    // std::exp(),std::pow(),std::log()
-#include <cstring>  // memcpy()
-#include <memory>
-#include <limits>
-#include <optional>
-#include <chrono>
-#include <random>
-#include <regex>
-#include <filesystem>
-
-using namespace std;
-namespace sys = std::filesystem;
-
-namespace Stockfish::Tools
-{
-    bool fen_is_ok(Position& pos, std::string input_fen) {
-        std::string pos_fen = pos.fen();
-        std::istringstream ss_input(input_fen);
-        std::istringstream ss_pos(pos_fen);
-
-        // example : "2r4r/4kpp1/nb1np3/p2p3p/B2P1BP1/PP6/4NPKP/2R1R3 w - h6 0 24"
-        //       --> "2r4r/4kpp1/nb1np3/p2p3p/B2P1BP1/PP6/4NPKP/2R1R3"
-        std::string str_input, str_pos;
-        ss_input >> str_input;
-        ss_pos >> str_pos;
-
-        // Only compare "Piece placement field" between input_fen and pos.fen().
-        return str_input == str_pos;
-    }
-
-    void convert_bin(
-        const vector<string>& filenames,
-        const string& output_file_name,
-        const int ply_minimum,
-        const int ply_maximum,
-        const int interpolate_eval,
-        const int src_score_min_value,
-        const int src_score_max_value,
-        const int dest_score_min_value,
-        const int dest_score_max_value,
-        const bool check_invalid_fen,
-        const bool check_illegal_move)
-    {
-        std::cout << "check_invalid_fen=" << check_invalid_fen << std::endl;
-        std::cout << "check_illegal_move=" << check_illegal_move << std::endl;
-
-        std::fstream fs;
-        uint64_t data_size = 0;
-        uint64_t filtered_size = 0;
-        uint64_t filtered_size_fen = 0;
-        uint64_t filtered_size_move = 0;
-        uint64_t filtered_size_ply = 0;
-        auto th = Threads.main();
-        auto& tpos = th->rootPos;
-        // convert plain rag to packed sfenvalue for Yaneura king
-        fs.open(output_file_name, ios::app | ios::binary);
-        StateListPtr states;
-        for (auto filename : filenames) {
-            std::cout << "convert " << filename << " ... ";
-            std::string line;
-            ifstream ifs;
-            ifs.open(filename);
-            PackedSfenValue p;
-            data_size = 0;
-            filtered_size = 0;
-            filtered_size_fen = 0;
-            filtered_size_move = 0;
-            filtered_size_ply = 0;
-            p.gamePly = 1; // Not included in apery format. Should be initialized
-            bool ignore_flag_fen = false;
-            bool ignore_flag_move = false;
-            bool ignore_flag_ply = false;
-            while (std::getline(ifs, line)) {
-                std::stringstream ss(line);
-                std::string token;
-                std::string value;
-                ss >> token;
-                if (token == "fen") {
-                    states = StateListPtr(new std::deque<StateInfo>(1)); // Drop old and create a new one
-                    std::string input_fen = line.substr(4);
-                    tpos.set(input_fen, false, &states->back(), Threads.main());
-                    if (check_invalid_fen && !fen_is_ok(tpos, input_fen)) {
-                        ignore_flag_fen = true;
-                        filtered_size_fen++;
-                    }
-                    else {
-                        tpos.sfen_pack(p.sfen, false);
-                    }
-                }
-                else if (token == "move") {
-                    ss >> value;
-                    Move move = UCI::to_move(tpos, value);
-                    if (check_illegal_move && move == MOVE_NONE) {
-                        ignore_flag_move = true;
-                        filtered_size_move++;
-                    }
-                    else {
-                        p.move = move;
-                    }
-                }
-                else if (token == "score") {
-                    double score;
-                    ss >> score;
-                    // Training Formula ?Issue #71 ?nodchip/Stockfish https://github.com/nodchip/Stockfish/issues/71
-                    // Normalize to [0.0, 1.0].
-                    score = (score - src_score_min_value) / (src_score_max_value - src_score_min_value);
-                    // Scale to [dest_score_min_value, dest_score_max_value].
-                    score = score * (dest_score_max_value - dest_score_min_value) + dest_score_min_value;
-                    p.score = std::clamp((int32_t)std::round(score), -(int32_t)VALUE_MATE, (int32_t)VALUE_MATE);
-                }
-                else if (token == "ply") {
-                    int temp;
-                    ss >> temp;
-                    if (temp < ply_minimum || temp > ply_maximum) {
-                        ignore_flag_ply = true;
-                        filtered_size_ply++;
-                    }
-                    p.gamePly = uint16_t(temp); // No cast here?
-                    if (interpolate_eval != 0) {
-                        p.score = min(3000, interpolate_eval * temp);
-                    }
-                }
-                else if (token == "result") {
-                    int temp;
-                    ss >> temp;
-                    p.game_result = int8_t(temp); // Do you need a cast here?
-                    if (interpolate_eval) {
-                        p.score = p.score * p.game_result;
-                    }
-                }
-                else if (token == "e") {
-                    if (!(ignore_flag_fen || ignore_flag_move || ignore_flag_ply)) {
-                        fs.write((char*)&p, sizeof(PackedSfenValue));
-                        data_size += 1;
-                        // debug
-                        // std::cout<<tpos<<std::endl;
-                        // std::cout<<p.score<<","<<int(p.gamePly)<<","<<int(p.game_result)<<std::endl;
-                    }
-                    else {
-                        filtered_size++;
-                    }
-                    ignore_flag_fen = false;
-                    ignore_flag_move = false;
-                    ignore_flag_ply = false;
-                }
-            }
-            std::cout << "done " << data_size << " parsed " << filtered_size << " is filtered"
-                << " (invalid fen:" << filtered_size_fen << ", illegal move:" << filtered_size_move << ", invalid ply:" << filtered_size_ply << ")" << std::endl;
-            ifs.close();
-        }
-        std::cout << "all done" << std::endl;
-        fs.close();
-    }
-
-    static inline void ltrim(std::string& s) {
-        s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](int ch) {
-            return !std::isspace(ch);
-            }));
-    }
-
-    static inline void rtrim(std::string& s) {
-        s.erase(std::find_if(s.rbegin(), s.rend(), [](int ch) {
-            return !std::isspace(ch);
-            }).base(), s.end());
-    }
-
-    static inline void trim(std::string& s) {
-        ltrim(s);
-        rtrim(s);
-    }
-
-    int parse_game_result_from_pgn_extract(std::string result) {
-        // White Win
-        if (result == "\"1-0\"") {
-            return 1;
-        }
-        // Black Win
-        else if (result == "\"0-1\"") {
-            return -1;
-        }
-        // Draw
-        else {
-            return 0;
-        }
-    }
-
-    // 0.25 -->  0.25 * PawnValueEg
-    // #-4  --> -mate_in(4)
-    // #3   -->  mate_in(3)
-    // -M4  --> -mate_in(4)
-    // +M3  -->  mate_in(3)
-    Value parse_score_from_pgn_extract(std::string eval, bool& success) {
-        success = true;
-
-        if (eval.substr(0, 1) == "#") {
-            if (eval.substr(1, 1) == "-") {
-                return -mate_in(stoi(eval.substr(2, eval.length() - 2)));
-            }
-            else {
-                return mate_in(stoi(eval.substr(1, eval.length() - 1)));
-            }
-        }
-        else if (eval.substr(0, 2) == "-M") {
-            //std::cout << "eval=" << eval << std::endl;
-            return -mate_in(stoi(eval.substr(2, eval.length() - 2)));
-        }
-        else if (eval.substr(0, 2) == "+M") {
-            //std::cout << "eval=" << eval << std::endl;
-            return mate_in(stoi(eval.substr(2, eval.length() - 2)));
-        }
-        else {
-            char* endptr;
-            double value = strtod(eval.c_str(), &endptr);
-
-            if (*endptr != '\0') {
-                success = false;
-                return VALUE_ZERO;
-            }
-            else {
-                return Value(value * static_cast<double>(PawnValueEg));
-            }
-        }
-    }
-
-    // for Debug
-    //#define DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT
-
-    bool is_like_fen(std::string fen) {
-        int count_space = std::count(fen.cbegin(), fen.cend(), ' ');
-        int count_slash = std::count(fen.cbegin(), fen.cend(), '/');
-
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-        //std::cout << "count_space=" << count_space << std::endl;
-        //std::cout << "count_slash=" << count_slash << std::endl;
-#endif
-
-        return count_space == 5 && count_slash == 7;
-    }
-
-    void convert_bin_from_pgn_extract(
-        const vector<string>& filenames,
-        const string& output_file_name,
-        const bool pgn_eval_side_to_move,
-        const bool convert_no_eval_fens_as_score_zero)
-    {
-        std::cout << "pgn_eval_side_to_move=" << pgn_eval_side_to_move << std::endl;
-        std::cout << "convert_no_eval_fens_as_score_zero=" << convert_no_eval_fens_as_score_zero << std::endl;
-
-        auto th = Threads.main();
-        auto& pos = th->rootPos;
-
-        std::fstream ofs;
-        ofs.open(output_file_name, ios::out | ios::binary);
-
-        int game_count = 0;
-        int fen_count = 0;
-
-        for (auto filename : filenames) {
-            std::cout << now_string() << " convert " << filename << std::endl;
-            ifstream ifs;
-            ifs.open(filename);
-
-            int game_result = 0;
-
-            std::string line;
-            while (std::getline(ifs, line)) {
-
-                if (line.empty()) {
-                    continue;
-                }
-
-                else if (line.substr(0, 1) == "[") {
-                    std::regex pattern_result(R"(\[Result (.+?)\])");
-                    std::smatch match;
-
-                    // example: [Result "1-0"]
-                    if (std::regex_search(line, match, pattern_result)) {
-                        game_result = parse_game_result_from_pgn_extract(match.str(1));
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-                        std::cout << "game_result=" << game_result << std::endl;
-#endif
-                        game_count++;
-                        if (game_count % 10000 == 0) {
-                            std::cout << now_string() << " game_count=" << game_count << ", fen_count=" << fen_count << std::endl;
-                        }
-                    }
-
-                    continue;
-                }
-
-                else {
-                    int gamePly = 1;
-                    auto itr = line.cbegin();
-
-                    while (true) {
-                        gamePly++;
-
-                        PackedSfenValue psv;
-                        memset((char*)&psv, 0, sizeof(PackedSfenValue));
-
-                        // fen
-                        {
-                            bool fen_found = false;
-
-                            while (!fen_found) {
-                                std::regex pattern_bracket(R"(\{(.+?)\})");
-                                std::smatch match;
-                                if (!std::regex_search(itr, line.cend(), match, pattern_bracket)) {
-                                    break;
-                                }
-
-                                itr += match.position(0) + match.length(0) - 1;
-                                std::string str_fen = match.str(1);
-                                trim(str_fen);
-
-                                if (is_like_fen(str_fen)) {
-                                    fen_found = true;
-
-                                    StateInfo si;
-                                    pos.set(str_fen, false, &si, th);
-                                    pos.sfen_pack(psv.sfen, false);
-                                }
-
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-                                std::cout << "str_fen=" << str_fen << std::endl;
-                                std::cout << "fen_found=" << fen_found << std::endl;
-#endif
-                            }
-
-                            if (!fen_found) {
-                                break;
-                            }
-                        }
-
-                        // move
-                        {
-                            std::regex pattern_move(R"(\}(.+?)\{)");
-                            std::smatch match;
-                            if (!std::regex_search(itr, line.cend(), match, pattern_move)) {
-                                break;
-                            }
-
-                            itr += match.position(0) + match.length(0) - 1;
-                            std::string str_move = match.str(1);
-                            trim(str_move);
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-                            std::cout << "str_move=" << str_move << std::endl;
-#endif
-                            psv.move = UCI::to_move(pos, str_move);
-                        }
-
-                        // eval
-                        bool eval_found = false;
-                        {
-                            std::regex pattern_bracket(R"(\{(.+?)\})");
-                            std::smatch match;
-                            if (!std::regex_search(itr, line.cend(), match, pattern_bracket)) {
-                                break;
-                            }
-
-                            std::string str_eval_clk = match.str(1);
-                            trim(str_eval_clk);
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-                            std::cout << "str_eval_clk=" << str_eval_clk << std::endl;
-#endif
-
-                            // example: { [%eval 0.25] [%clk 0:10:00] }
-                            // example: { [%eval #-4] [%clk 0:10:00] }
-                            // example: { [%eval #3] [%clk 0:10:00] }
-                            // example: { +0.71/22 1.2s }
-                            // example: { -M4/7 0.003s }
-                            // example: { M3/245 0.017s }
-                            // example: { +M1/245 0.010s, White mates }
-                            // example: { 0.60 }
-                            // example: { book }
-                            // example: { rnbqkb1r/pp3ppp/2p1pn2/3p4/2PP4/2N2N2/PP2PPPP/R1BQKB1R w KQkq - 0 5 }
-
-                            // Considering the absence of eval
-                            if (!is_like_fen(str_eval_clk)) {
-                                itr += match.position(0) + match.length(0) - 1;
-
-                                if (str_eval_clk != "book") {
-                                    std::regex pattern_eval1(R"(\[\%eval (.+?)\])");
-                                    std::regex pattern_eval2(R"((.+?)\/)");
-
-                                    std::string str_eval;
-                                    if (std::regex_search(str_eval_clk, match, pattern_eval1) ||
-                                        std::regex_search(str_eval_clk, match, pattern_eval2)) {
-                                        str_eval = match.str(1);
-                                        trim(str_eval);
-                                    }
-                                    else {
-                                        str_eval = str_eval_clk;
-                                    }
-
-                                    bool success = false;
-                                    Value value = parse_score_from_pgn_extract(str_eval, success);
-                                    if (success) {
-                                        eval_found = true;
-                                        psv.score = std::clamp(value, -VALUE_MATE, VALUE_MATE);
-                                    }
-
-#if defined(DEBUG_CONVERT_BIN_FROM_PGN_EXTRACT)
-                                    std::cout << "str_eval=" << str_eval << std::endl;
-                                    std::cout << "success=" << success << ", psv.score=" << psv.score << std::endl;
-#endif
-                                }
-                            }
-                        }
-
-                        // write
-                        if (eval_found || convert_no_eval_fens_as_score_zero) {
-                            if (!eval_found && convert_no_eval_fens_as_score_zero) {
-                                psv.score = 0;
-                            }
-
-                            psv.gamePly = gamePly;
-                            psv.game_result = game_result;
-
-                            if (pos.side_to_move() == BLACK) {
-                                if (!pgn_eval_side_to_move) {
-                                    psv.score *= -1;
-                                }
-                                psv.game_result *= -1;
-                            }
-
-                            ofs.write((char*)&psv, sizeof(PackedSfenValue));
-
-                            fen_count++;
-                        }
-                    }
-
-                    game_result = 0;
-                }
-            }
-        }
-
-        std::cout << now_string() << " game_count=" << game_count << ", fen_count=" << fen_count << std::endl;
-        std::cout << now_string() << " all done" << std::endl;
-        ofs.close();
-    }
-
-    void convert_plain(
-        const vector<string>& filenames,
-        const string& output_file_name)
-    {
-        Position tpos;
-        std::ofstream ofs;
-        ofs.open(output_file_name, ios::app);
-        auto th = Threads.main();
-        for (auto filename : filenames) {
-            std::cout << "convert " << filename << " ... ";
-
-            // Just convert packedsfenvalue to text
-            std::fstream fs;
-            fs.open(filename, ios::in | ios::binary);
-            PackedSfenValue p;
-            while (true)
-            {
-                if (fs.read((char*)&p, sizeof(PackedSfenValue))) {
-                    StateInfo si;
-                    tpos.set_from_packed_sfen(p.sfen, &si, th, false);
-
-                    // write as plain text
-                    ofs << "fen " << tpos.fen() << std::endl;
-                    ofs << "move " << UCI::move(Move(p.move), false) << std::endl;
-                    ofs << "score " << p.score << std::endl;
-                    ofs << "ply " << int(p.gamePly) << std::endl;
-                    ofs << "result " << int(p.game_result) << std::endl;
-                    ofs << "e" << std::endl;
-                }
-                else {
-                    break;
-                }
-            }
-            fs.close();
-            std::cout << "done" << std::endl;
-        }
-        ofs.close();
-        std::cout << "all done" << std::endl;
-    }
-
-    static inline const std::string plain_extension = ".plain";
-    static inline const std::string bin_extension = ".bin";
-    static inline const std::string binpack_extension = ".binpack";
-
-    static bool file_exists(const std::string& name)
-    {
-        std::ifstream f(name);
-        return f.good();
-    }
-
-    static bool ends_with(const std::string& lhs, const std::string& end)
-    {
-        if (end.size() > lhs.size()) return false;
-
-        return std::equal(end.rbegin(), end.rend(), lhs.rbegin());
-    }
-
-    static bool is_convert_of_type(
-        const std::string& input_path,
-        const std::string& output_path,
-        const std::string& expected_input_extension,
-        const std::string& expected_output_extension)
-    {
-        return ends_with(input_path, expected_input_extension)
-            && ends_with(output_path, expected_output_extension);
-    }
-
-    using ConvertFunctionType = void(std::string inputPath, std::string outputPath, std::ios_base::openmode om, bool validate);
-
-    static ConvertFunctionType* get_convert_function(const std::string& input_path, const std::string& output_path)
-    {
-        if (is_convert_of_type(input_path, output_path, plain_extension, bin_extension))
-            return binpack::convertPlainToBin;
-        if (is_convert_of_type(input_path, output_path, plain_extension, binpack_extension))
-            return binpack::convertPlainToBinpack;
-
-        if (is_convert_of_type(input_path, output_path, bin_extension, plain_extension))
-            return binpack::convertBinToPlain;
-        if (is_convert_of_type(input_path, output_path, bin_extension, binpack_extension))
-            return binpack::convertBinToBinpack;
-
-        if (is_convert_of_type(input_path, output_path, binpack_extension, plain_extension))
-            return binpack::convertBinpackToPlain;
-        if (is_convert_of_type(input_path, output_path, binpack_extension, bin_extension))
-            return binpack::convertBinpackToBin;
-
-        return nullptr;
-    }
-
-    static void convert(const std::string& input_path, const std::string& output_path, std::ios_base::openmode om, bool validate)
-    {
-        if(!file_exists(input_path))
-        {
-            std::cerr << "Input file does not exist.\n";
-            return;
-        }
-
-        auto func = get_convert_function(input_path, output_path);
-        if (func != nullptr)
-        {
-            func(input_path, output_path, om, validate);
-        }
-        else
-        {
-            std::cerr << "Conversion between files of these types is not supported.\n";
-        }
-    }
-
-    static void convert(const std::vector<std::string>& args)
-    {
-        if (args.size() < 2 || args.size() > 4)
-        {
-            std::cerr << "Invalid arguments.\n";
-            std::cerr << "Usage: convert from_path to_path [append] [validate]\n";
-            return;
-        }
-
-        const bool append = std::find(args.begin() + 2, args.end(), "append") != args.end();
-        const bool validate = std::find(args.begin() + 2, args.end(), "validate") != args.end();
-
-        const std::ios_base::openmode openmode =
-            append
-            ? std::ios_base::app
-            : std::ios_base::trunc;
-
-        convert(args[0], args[1], openmode, validate);
-    }
-
-    void convert(istringstream& is)
-    {
-        std::vector<std::string> args;
-
-        while (true)
-        {
-            std::string token = "";
-            is >> token;
-            if (token == "")
-                break;
-
-            args.push_back(token);
-        }
-
-        convert(args);
-    }
-
-    static void append_files_from_dir(
-        std::vector<std::string>& filenames,
-        const std::string& base_dir,
-        const std::string& target_dir)
-    {
-        string kif_base_dir = Path::combine(base_dir, target_dir);
-
-        sys::path p(kif_base_dir); // Origin of enumeration
-        std::for_each(sys::directory_iterator(p), sys::directory_iterator(),
-            [&](const sys::path& path) {
-                if (sys::is_regular_file(path))
-                    filenames.push_back(Path::combine(target_dir, path.filename().generic_string()));
-            });
-    }
-
-    static void rebase_files(
-        std::vector<std::string>& filenames,
-        const std::string& base_dir)
-    {
-        for (auto& file : filenames)
-        {
-            file = Path::combine(base_dir, file);
-        }
-    }
-
-    void convert_bin_from_pgn_extract(std::istringstream& is)
-    {
-        std::vector<std::string> filenames;
-
-        string base_dir;
-        string target_dir;
-
-        bool pgn_eval_side_to_move = false;
-        bool convert_no_eval_fens_as_score_zero = false;
-
-        string output_file_name = "shuffled_sfen.bin";
-
-        while (true)
-        {
-            string option;
-            is >> option;
-
-            if (option == "")
-                break;
-
-            if (option == "targetdir") is >> target_dir;
-            else if (option == "targetfile")
-            {
-                std::string filename;
-                is >> filename;
-                filenames.push_back(filename);
-            }
-
-            else if (option == "basedir")   is >> base_dir;
-
-            else if (option == "pgn_eval_side_to_move") is >> pgn_eval_side_to_move;
-            else if (option == "convert_no_eval_fens_as_score_zero") is >> convert_no_eval_fens_as_score_zero;
-            else if (option == "output_file_name") is >> output_file_name;
-            else
-            {
-                cout << "Unknown option: " << option << ". Ignoring.\n";
-            }
-        }
-
-        if (!target_dir.empty())
-        {
-            append_files_from_dir(filenames, base_dir, target_dir);
-        }
-        rebase_files(filenames, base_dir);
-
-        Eval::NNUE::init();
-
-        cout << "convert_bin_from_pgn-extract.." << endl;
-        convert_bin_from_pgn_extract(
-            filenames,
-            output_file_name,
-            pgn_eval_side_to_move,
-            convert_no_eval_fens_as_score_zero);
-    }
-
-    void convert_bin(std::istringstream& is)
-    {
-        std::vector<std::string> filenames;
-
-        string base_dir;
-        string target_dir;
-
-        int ply_minimum = 0;
-        int ply_maximum = 114514;
-        bool interpolate_eval = 0;
-        bool check_invalid_fen = false;
-        bool check_illegal_move = false;
-
-        bool pgn_eval_side_to_move = false;
-        bool convert_no_eval_fens_as_score_zero = false;
-
-        double src_score_min_value = 0.0;
-        double src_score_max_value = 1.0;
-        double dest_score_min_value = 0.0;
-        double dest_score_max_value = 1.0;
-
-        string output_file_name = "shuffled_sfen.bin";
-
-        while (true)
-        {
-            string option;
-            is >> option;
-
-            if (option == "")
-                break;
-
-            if (option == "targetdir") is >> target_dir;
-            else if (option == "targetfile")
-            {
-                std::string filename;
-                is >> filename;
-                filenames.push_back(filename);
-            }
-
-            else if (option == "basedir")   is >> base_dir;
-
-            else if (option == "ply_minimum") is >> ply_minimum;
-            else if (option == "ply_maximum") is >> ply_maximum;
-            else if (option == "interpolate_eval") is >> interpolate_eval;
-            else if (option == "check_invalid_fen") is >> check_invalid_fen;
-            else if (option == "check_illegal_move") is >> check_illegal_move;
-            else if (option == "pgn_eval_side_to_move") is >> pgn_eval_side_to_move;
-            else if (option == "convert_no_eval_fens_as_score_zero") is >> convert_no_eval_fens_as_score_zero;
-            else if (option == "src_score_min_value") is >> src_score_min_value;
-            else if (option == "src_score_max_value") is >> src_score_max_value;
-            else if (option == "dest_score_min_value") is >> dest_score_min_value;
-            else if (option == "dest_score_max_value") is >> dest_score_max_value;
-            else if (option == "output_file_name") is >> output_file_name;
-            else
-            {
-                cout << "Unknown option: " << option << ". Ignoring.\n";
-            }
-        }
-
-        if (!target_dir.empty())
-        {
-            append_files_from_dir(filenames, base_dir, target_dir);
-        }
-        rebase_files(filenames, base_dir);
-
-        Eval::NNUE::init();
-
-        cout << "convert_bin.." << endl;
-            convert_bin(
-                filenames,
-                output_file_name,
-                ply_minimum,
-                ply_maximum,
-                interpolate_eval,
-                src_score_min_value,
-                src_score_max_value,
-                dest_score_min_value,
-                dest_score_max_value,
-                check_invalid_fen,
-                check_illegal_move
-            );
-    }
-
-    void convert_plain(std::istringstream& is)
-    {
-        std::vector<std::string> filenames;
-
-        string base_dir;
-        string target_dir;
-
-        string output_file_name = "shuffled_sfen.bin";
-
-        while (true)
-        {
-            string option;
-            is >> option;
-
-            if (option == "")
-                break;
-
-            if (option == "targetdir") is >> target_dir;
-            else if (option == "targetfile")
-            {
-                std::string filename;
-                is >> filename;
-                filenames.push_back(filename);
-            }
-
-            else if (option == "basedir")   is >> base_dir;
-
-            else if (option == "output_file_name") is >> output_file_name;
-            else
-            {
-                cout << "Unknown option: " << option << ". Ignoring.\n";
-            }
-        }
-
-        if (!target_dir.empty())
-        {
-            append_files_from_dir(filenames, base_dir, target_dir);
-        }
-        rebase_files(filenames, base_dir);
-
-        Eval::NNUE::init();
-
-        cout << "convert_plain.." << endl;
-        convert_plain(filenames, output_file_name);
-    }
-}

src/tools/convert.h

@@ -1,18 +0,0 @@
-#ifndef _CONVERT_H_
-#define _CONVERT_H_
-
-#include <vector>
-#include <string>
-#include <sstream>
-
-namespace Stockfish::Tools {
-    void convert(std::istringstream& is);
-
-    void convert_bin_from_pgn_extract(std::istringstream& is);
-
-    void convert_bin(std::istringstream& is);
-
-    void convert_plain(std::istringstream& is);
-}
-
-#endif

src/tools/opening_book.cpp

@@ -1,43 +0,0 @@
-#include "opening_book.h"
-
-#include <fstream>
-
-namespace Stockfish::Tools {
-
-    EpdOpeningBook::EpdOpeningBook(const std::string& file, PRNG& prng) :
-        OpeningBook(file)
-    {
-        std::ifstream in(file);
-        if (!in)
-        {
-            return;
-        }
-
-        std::string line;
-        while (std::getline(in, line))
-        {
-            if (line.empty())
-                continue;
-
-            fens.emplace_back(line);
-        }
-
-        Algo::shuffle(fens, prng);
-    }
-
-    static bool ends_with(const std::string& lhs, const std::string& end)
-    {
-        if (end.size() > lhs.size()) return false;
-
-        return std::equal(end.rbegin(), end.rend(), lhs.rbegin());
-    }
-
-    std::unique_ptr<OpeningBook> open_opening_book(const std::string& filename, PRNG& prng)
-    {
-        if (ends_with(filename, ".epd"))
-            return std::make_unique<EpdOpeningBook>(filename, prng);
-
-        return nullptr;
-    }
-
-}

src/tools/opening_book.h

@@ -1,60 +0,0 @@
-#ifndef LEARN_OPENING_BOOK_H
-#define LEARN_OPENING_BOOK_H
-
-#include "misc.h"
-#include "position.h"
-#include "thread.h"
-
-#include <vector>
-#include <random>
-#include <optional>
-#include <string>
-#include <cstdint>
-#include <memory>
-#include <mutex>
-
-namespace Stockfish::Tools {
-
-    struct OpeningBook {
-
-        const std::string& next_fen()
-        {
-            assert(fens.size() > 0);
-
-            std::unique_lock lock(mutex);
-
-            auto& fen = fens[current_index++];
-            if (current_index >= fens.size())
-                current_index = 0;
-
-            return fen;
-        }
-
-        std::size_t size() const { return fens.size(); }
-
-        const std::string& get_filename() const { return filename; }
-
-    protected:
-        OpeningBook(const std::string& file) :
-            filename(file),
-            current_index(0)
-        {
-        }
-
-
-        std::mutex mutex;
-        std::string filename;
-        std::vector<std::string> fens;
-        std::size_t current_index;
-    };
-
-    struct EpdOpeningBook : OpeningBook {
-
-        EpdOpeningBook(const std::string& file, PRNG& prng);
-    };
-
-    std::unique_ptr<OpeningBook> open_opening_book(const std::string& filename, PRNG& prng);
-
-}
-
-#endif

src/tools/packed_sfen.h

@@ -1,46 +0,0 @@
-#ifndef _PACKED_SFEN_H_
-#define _PACKED_SFEN_H_
-
-#include <vector>
-#include <cstdint>
-
-namespace Stockfish::Tools {
-
-    // packed sfen
-    struct PackedSfen { std::uint8_t data[32]; };
-
-    // Structure in which PackedSfen and evaluation value are integrated
-    // If you write different contents for each option, it will be a problem when reusing the teacher game
-    // For the time being, write all the following members regardless of the options.
-    struct PackedSfenValue
-    {
-        // phase
-        PackedSfen sfen;
-
-        // Evaluation value returned from Tools::search()
-        std::int16_t score;
-
-        // PV first move
-        // Used when finding the match rate with the teacher
-        std::uint16_t move;
-
-        // Trouble of the phase from the initial phase.
-        std::uint16_t gamePly;
-
-        // 1 if the player on this side ultimately wins the game. -1 if you are losing.
-        // 0 if a draw is reached.
-        // The draw is in the teacher position generation command gensfen,
-        // Only write if LEARN_GENSFEN_DRAW_RESULT is enabled.
-        std::int8_t game_result;
-
-        // When exchanging the file that wrote the teacher aspect with other people
-        //Because this structure size is not fixed, pad it so that it is 40 bytes in any environment.
-        std::uint8_t padding;
-
-        // 32 + 2 + 2 + 2 + 1 + 1 = 40bytes
-    };
-
-    // Phase array: PSVector stands for packed sfen vector.
-    using PSVector = std::vector<PackedSfenValue>;
-}
-#endif

src/tools/sfen_packer.cpp

@@ -1,389 +0,0 @@
-#include "sfen_packer.h"
-
-#include "packed_sfen.h"
-
-#include "misc.h"
-#include "position.h"
-
-#include <sstream>
-#include <fstream>
-#include <cstring> // std::memset()
-
-using namespace std;
-
-namespace Stockfish::Tools {
-
-    // Class that handles bitstream
-    // useful when doing aspect encoding
-    struct BitStream
-    {
-        // Set the memory to store the data in advance.
-        // Assume that memory is cleared to 0.
-        void set_data(std::uint8_t* data_) { data = data_; reset(); }
-
-        // Get the pointer passed in set_data().
-        uint8_t* get_data() const { return data; }
-
-        // Get the cursor.
-        int get_cursor() const { return bit_cursor; }
-
-        // reset the cursor
-        void reset() { bit_cursor = 0; }
-
-        // Write 1bit to the stream.
-        // If b is non-zero, write out 1. If 0, write 0.
-        void write_one_bit(int b)
-        {
-            if (b)
-                data[bit_cursor / 8] |= 1 << (bit_cursor & 7);
-
-            ++bit_cursor;
-        }
-
-        // Get 1 bit from the stream.
-        int read_one_bit()
-        {
-            int b = (data[bit_cursor / 8] >> (bit_cursor & 7)) & 1;
-            ++bit_cursor;
-
-            return b;
-        }
-
-        // write n bits of data
-        // Data shall be written out from the lower order of d.
-        void write_n_bit(int d, int n)
-        {
-            for (int i = 0; i <n; ++i)
-                write_one_bit(d & (1 << i));
-        }
-
-        // read n bits of data
-        // Reverse conversion of write_n_bit().
-        int read_n_bit(int n)
-        {
-            int result = 0;
-            for (int i = 0; i < n; ++i)
-                result |= read_one_bit() ? (1 << i) : 0;
-
-            return result;
-        }
-
-    private:
-        // Next bit position to read/write.
-        int bit_cursor;
-
-        // data entity
-        std::uint8_t* data;
-    };
-
-    // Class for compressing/decompressing sfen
-    // sfen can be packed to 256bit (32bytes) by Huffman coding.
-    // This is proven by mini. The above is Huffman coding.
-    //
-    // Internal format = 1-bit turn + 7-bit king position *2 + piece on board (Huffman coding) + hand piece (Huffman coding)
-    // Side to move (White = 0, Black = 1) (1bit)
-    // White King Position (6 bits)
-    // Black King Position (6 bits)
-    // Huffman Encoding of the board
-    // Castling availability (1 bit x 4)
-    // En passant square (1 or 1 + 6 bits)
-    // Rule 50 (6 bits)
-    // Game play (8 bits)
-    //
-    // TODO(someone): Rename SFEN to FEN.
-    //
-    struct SfenPacker
-    {
-        void pack(const Position& pos, bool resetCastlingRights);
-
-        // sfen packed by pack() (256bit = 32bytes)
-        // Or sfen to decode with unpack()
-        uint8_t *data; // uint8_t[32];
-
-        BitStream stream;
-
-        // Output the board pieces to stream.
-        void write_board_piece_to_stream(Piece pc);
-
-        // Read one board piece from stream
-        Piece read_board_piece_from_stream();
-    };
-
-
-    // Huffman coding
-    // * is simplified from mini encoding to make conversion easier.
-    //
-    // Huffman Encoding
-    //
-    // Empty  xxxxxxx0
-    // Pawn   xxxxx001 + 1 bit (Color)
-    // Knight xxxxx011 + 1 bit (Color)
-    // Bishop xxxxx101 + 1 bit (Color)
-    // Rook   xxxxx111 + 1 bit (Color)
-    // Queen   xxxx1001 + 1 bit (Color)
-    //
-    // Worst case:
-    // - 32 empty squares    32 bits
-    // - 30 pieces           150 bits
-    // - 2 kings             12 bits
-    // - castling rights     4 bits
-    // - ep square           7 bits
-    // - rule50              7 bits
-    // - game ply            16 bits
-    // - TOTAL               228 bits < 256 bits
-
-    struct HuffmanedPiece
-    {
-        int code; // how it will be coded
-        int bits; // How many bits do you have
-    };
-
-    constexpr HuffmanedPiece huffman_table[] =
-    {
-        {0b0000,1}, // NO_PIECE
-        {0b0001,4}, // PAWN
-        {0b0011,4}, // KNIGHT
-        {0b0101,4}, // BISHOP
-        {0b0111,4}, // ROOK
-        {0b1001,4}, // QUEEN
-    };
-
-    // Pack sfen and store in data[32].
-    void SfenPacker::pack(const Position& pos, bool resetCastlingRights)
-    {
-        memset(data, 0, 32 /* 256bit */);
-        stream.set_data(data);
-
-        // turn
-        // Side to move.
-        stream.write_one_bit((int)(pos.side_to_move()));
-
-        // 7-bit positions for leading and trailing balls
-        // White king and black king, 6 bits for each.
-        for(auto c: Colors)
-            stream.write_n_bit(pos.king_square(c), 6);
-
-        // Write the pieces on the board other than the kings.
-        for (Rank r = RANK_8; r >= RANK_1; --r)
-        {
-            for (File f = FILE_A; f <= FILE_H; ++f)
-            {
-                Piece pc = pos.piece_on(make_square(f, r));
-                if (type_of(pc) == KING)
-                    continue;
-                write_board_piece_to_stream(pc);
-            }
-        }
-
-        if (resetCastlingRights)
-        {
-            stream.write_n_bit(0, 4);
-        }
-        else
-        {
-            stream.write_one_bit(pos.can_castle(WHITE_OO));
-            stream.write_one_bit(pos.can_castle(WHITE_OOO));
-            stream.write_one_bit(pos.can_castle(BLACK_OO));
-            stream.write_one_bit(pos.can_castle(BLACK_OOO));
-        }
-
-        if (pos.ep_square() == SQ_NONE) {
-            stream.write_one_bit(0);
-        }
-        else {
-            stream.write_one_bit(1);
-            stream.write_n_bit(static_cast<int>(pos.ep_square()), 6);
-        }
-
-        stream.write_n_bit(pos.state()->rule50, 6);
-
-        const int fm = 1 + (pos.game_ply()-(pos.side_to_move() == BLACK)) / 2;
-        stream.write_n_bit(fm, 8);
-
-        // Write high bits of half move. This is a fix for the
-        // limited range of half move counter.
-        // This is backwards compatibile.
-        stream.write_n_bit(fm >> 8, 8);
-
-        // Write the highest bit of rule50 at the end. This is a backwards
-        // compatibile fix for rule50 having only 6 bits stored.
-        // This bit is just ignored by the old parsers.
-        stream.write_n_bit(pos.state()->rule50 >> 6, 1);
-
-        assert(stream.get_cursor() <= 256);
-    }
-
-    // Output the board pieces to stream.
-    void SfenPacker::write_board_piece_to_stream(Piece pc)
-    {
-        // piece type
-        PieceType pr = type_of(pc);
-        auto c = huffman_table[pr];
-        stream.write_n_bit(c.code, c.bits);
-
-        if (pc == NO_PIECE)
-            return;
-
-        // first and second flag
-        stream.write_one_bit(color_of(pc));
-    }
-
-    // Read one board piece from stream
-    Piece SfenPacker::read_board_piece_from_stream()
-    {
-        PieceType pr = NO_PIECE_TYPE;
-        int code = 0, bits = 0;
-        while (true)
-        {
-            code |= stream.read_one_bit() << bits;
-            ++bits;
-
-            assert(bits <= 6);
-
-            for (pr = NO_PIECE_TYPE; pr <KING; ++pr)
-                if (huffman_table[pr].code == code
-                    && huffman_table[pr].bits == bits)
-                    goto Found;
-        }
-    Found:;
-        if (pr == NO_PIECE_TYPE)
-            return NO_PIECE;
-
-        // first and second flag
-        Color c = (Color)stream.read_one_bit();
-
-        return make_piece(c, pr);
-    }
-
-    int set_from_packed_sfen(Position& pos, const PackedSfen& sfen, StateInfo* si, Thread* th, bool frc)
-    {
-        SfenPacker packer;
-        auto& stream = packer.stream;
-
-        // TODO: separate streams for writing and reading. Here we actually have to
-        // const_cast which is not safe in the long run.
-        stream.set_data(const_cast<uint8_t*>(reinterpret_cast<const uint8_t*>(&sfen)));
-
-        pos.clear();
-        std::memset(si, 0, sizeof(StateInfo));
-        si->accumulator.computed[WHITE] = false;
-        si->accumulator.computed[BLACK] = false;
-        pos.st = si;
-
-        // Active color
-        pos.sideToMove = (Color)stream.read_one_bit();
-
-        // First the position of the ball
-        for (auto c : Colors)
-            pos.board[stream.read_n_bit(6)] = make_piece(c, KING);
-
-        // Piece placement
-        for (Rank r = RANK_8; r >= RANK_1; --r)
-        {
-            for (File f = FILE_A; f <= FILE_H; ++f)
-            {
-                auto sq = make_square(f, r);
-
-                // it seems there are already balls
-                Piece pc;
-                if (type_of(pos.board[sq]) != KING)
-                {
-                    assert(pos.board[sq] == NO_PIECE);
-                    pc = packer.read_board_piece_from_stream();
-                }
-                else
-                {
-                    pc = pos.board[sq];
-                    // put_piece() will catch ASSERT unless you remove it all.
-                    pos.board[sq] = NO_PIECE;
-                }
-
-                // There may be no pieces, so skip in that case.
-                if (pc == NO_PIECE)
-                    continue;
-
-                pos.put_piece(Piece(pc), sq);
-
-                if (stream.get_cursor()> 256)
-                    return 1;
-            }
-        }
-
-        // Castling availability.
-        pos.st->castlingRights = 0;
-        if (stream.read_one_bit()) {
-            Square rsq;
-            for (rsq = relative_square(WHITE, SQ_H1); pos.piece_on(rsq) != W_ROOK; --rsq) {}
-            pos.set_castling_right(WHITE, rsq);
-        }
-        if (stream.read_one_bit()) {
-            Square rsq;
-            for (rsq = relative_square(WHITE, SQ_A1); pos.piece_on(rsq) != W_ROOK; ++rsq) {}
-            pos.set_castling_right(WHITE, rsq);
-        }
-        if (stream.read_one_bit()) {
-            Square rsq;
-            for (rsq = relative_square(BLACK, SQ_H1); pos.piece_on(rsq) != B_ROOK; --rsq) {}
-            pos.set_castling_right(BLACK, rsq);
-        }
-        if (stream.read_one_bit()) {
-            Square rsq;
-            for (rsq = relative_square(BLACK, SQ_A1); pos.piece_on(rsq) != B_ROOK; ++rsq) {}
-            pos.set_castling_right(BLACK, rsq);
-        }
-
-        // En passant square. Ignore if no pawn capture is possible
-        if (stream.read_one_bit()) {
-            Square ep_square = static_cast<Square>(stream.read_n_bit(6));
-            pos.st->epSquare = ep_square;
-
-            if (!(pos.attackers_to(pos.st->epSquare) & pos.pieces(pos.sideToMove, PAWN))
-                || !(pos.pieces(~pos.sideToMove, PAWN) & (pos.st->epSquare + pawn_push(~pos.sideToMove))))
-                pos.st->epSquare = SQ_NONE;
-        }
-        else {
-            pos.st->epSquare = SQ_NONE;
-        }
-
-        // Halfmove clock
-        pos.st->rule50 = stream.read_n_bit(6);
-
-        // Fullmove number
-        pos.gamePly = stream.read_n_bit(8);
-
-        // Read the highest bit of rule50. This was added as a fix for rule50
-        // counter having only 6 bits stored.
-        // In older entries this will just be a zero bit.
-        pos.gamePly |= stream.read_n_bit(8) << 8;
-
-        // Read the highest bit of rule50. This was added as a fix for rule50
-        // counter having only 6 bits stored.
-        // In older entries this will just be a zero bit.
-        pos.st->rule50 |= stream.read_n_bit(1) << 6;
-
-        // Convert from fullmove starting from 1 to gamePly starting from 0,
-        // handle also common incorrect FEN with fullmove = 0.
-        pos.gamePly = std::max(2 * (pos.gamePly - 1), 0) + (pos.sideToMove == BLACK);
-
-        assert(stream.get_cursor() <= 256);
-
-        pos.chess960 = frc;
-        pos.thisThread = th;
-        pos.set_state(pos.st);
-
-        assert(pos.pos_is_ok());
-
-        return 0;
-    }
-
-    PackedSfen sfen_pack(Position& pos, bool resetCastlingRights)
-    {
-        PackedSfen sfen;
-
-        SfenPacker sp;
-        sp.data = (uint8_t*)&sfen;
-        sp.pack(pos, resetCastlingRights);
-
-        return sfen;
-    }
-}

src/tools/sfen_packer.h

@@ -1,22 +0,0 @@
-#ifndef _SFEN_PACKER_H_
-#define _SFEN_PACKER_H_
-
-#include "types.h"
-
-#include "packed_sfen.h"
-
-#include <cstdint>
-
-namespace Stockfish {
-    class Position;
-    struct StateInfo;
-    class Thread;
-}
-
-namespace Stockfish::Tools {
-
-    int set_from_packed_sfen(Position& pos, const PackedSfen& sfen, StateInfo* si, Thread* th, bool frc);
-    PackedSfen sfen_pack(Position& pos, bool resetCastlingRights);
-}
-
-#endif

src/tools/sfen_reader.h

@@ -1,352 +0,0 @@
-#include "sfen_stream.h"
-
-#include "packed_sfen.h"
-
-#include "misc.h"
-
-#include <string>
-#include <vector>
-#include <deque>
-#include <memory>
-#include <mutex>
-#include <list>
-#include <atomic>
-#include <optional>
-#include <iostream>
-#include <cstdint>
-#include <thread>
-#include <functional>
-
-namespace Stockfish::Tools{
-
-    enum struct SfenReaderMode
-    {
-        Sequential,
-        Cyclic
-    };
-
-    // Sfen reader
-    struct SfenReader
-    {
-        // Number of phases buffered by each thread 0.1M phases. 4M phase at 40HT
-        static constexpr size_t DEFAULT_THREAD_BUFFER_SIZE = 10 * 1000;
-
-        // Buffer for reading files (If this is made larger,
-        // the shuffle becomes larger and the phases may vary.
-        // If it is too large, the memory consumption will increase.
-        // SFEN_READ_SIZE is a multiple of THREAD_BUFFER_SIZE.
-        static constexpr const size_t DEFAULT_SFEN_READ_SIZE = 1000 * 1000 * 10;
-
-        // Do not use std::random_device().
-        // Because it always the same integers on MinGW.
-        SfenReader(
-            const std::vector<std::string>& filenames_,
-            bool do_shuffle,
-            SfenReaderMode mode_,
-            int thread_num,
-            const std::string& seed,
-            size_t read_size = DEFAULT_SFEN_READ_SIZE,
-            size_t buffer_size = DEFAULT_THREAD_BUFFER_SIZE
-        ) :
-            filenames(filenames_.begin(), filenames_.end()),
-            mode(mode_),
-            // Due to the implementation of waiting for buffer empty a bit
-            // the read size must be at least twice the buffer size.
-            sfen_read_size(std::max(read_size, buffer_size * 2)),
-            thread_buffer_size(buffer_size),
-            prng(seed)
-        {
-            packed_sfens.resize(thread_num);
-            total_read = 0;
-            end_of_files = false;
-            shuffle = do_shuffle;
-            stop_flag = false;
-            num_buffers_in_pool.store(0);
-
-            file_worker_thread = std::thread([&] {
-                this->file_read_worker();
-            });
-        }
-
-        ~SfenReader()
-        {
-            stop_flag = true;
-
-            if (file_worker_thread.joinable())
-                file_worker_thread.join();
-        }
-
-        // Load the phase for calculation such as mse.
-        PSVector read_some(uint64_t count, uint64_t count_tries, std::function<bool(const PackedSfenValue&)> do_take)
-        {
-            PSVector psv;
-            psv.reserve(count);
-
-            for (uint64_t i = 0; i < count_tries; ++i)
-            {
-                PackedSfenValue ps;
-                if (!read_to_thread_buffer(0, ps))
-                {
-                    std::cout << "ERROR (sfen_reader): Reading failed." << std::endl;
-                    return psv;
-                }
-
-                if (do_take(ps))
-                {
-                    psv.push_back(ps);
-
-                    if (psv.size() >= count)
-                        break;
-                }
-            }
-
-            return psv;
-        }
-
-        // [ASYNC] Thread returns one aspect. Otherwise returns false.
-        bool read_to_thread_buffer(size_t thread_id, PackedSfenValue& ps)
-        {
-            // If there are any positions left in the thread buffer
-            // then retrieve one and return it.
-            auto& thread_ps = packed_sfens[thread_id];
-
-            // Fill the read buffer if there is no remaining buffer,
-            // but if it doesn't even exist, finish.
-            // If the buffer is empty, fill it.
-            if ((thread_ps == nullptr || thread_ps->empty())
-                && !read_to_thread_buffer_impl(thread_id))
-                return false;
-
-            // read_to_thread_buffer_impl() returned true,
-            // Since the filling of the thread buffer with the
-            // phase has been completed successfully
-            // thread_ps->rbegin() is alive.
-
-            ps = thread_ps->back();
-            thread_ps->pop_back();
-
-            // If you've run out of buffers, call delete yourself to free this buffer.
-            if (thread_ps->empty())
-            {
-                thread_ps.reset();
-            }
-
-            return true;
-        }
-
-        // [ASYNC] Read some aspects into thread buffer.
-        bool read_to_thread_buffer_impl(size_t thread_id)
-        {
-            while (true)
-            {
-                {
-                    std::unique_lock<std::mutex> lk(mutex);
-                    // If you can fill from the file buffer, that's fine.
-                    if (packed_sfens_pool.size() != 0)
-                    {
-                        // It seems that filling is possible, so fill and finish.
-
-                        packed_sfens[thread_id] = std::move(packed_sfens_pool.front());
-                        packed_sfens_pool.pop_front();
-                        num_buffers_in_pool.fetch_sub(1);
-
-                        total_read += thread_buffer_size;
-
-                        return true;
-                    }
-                }
-
-                // The file to read is already gone. No more use.
-                if (end_of_files)
-                    return false;
-
-                // Waiting for file worker to fill packed_sfens_pool.
-                // The mutex isn't locked, so it should fill up soon.
-                // Poor man's condition variable.
-                sleep(1);
-            }
-
-        }
-
-        void file_read_worker()
-        {
-            std::string currentFilename;
-            uint64_t numEntriesReadFromCurrentFile = 0;
-
-            auto open_next_file = [&]() {
-                // no more
-                for(;;)
-                {
-                    sfen_input_stream.reset();
-
-                    if (filenames.empty())
-                        return false;
-
-                    // Get the next file name.
-                    currentFilename = filenames.front();
-                    filenames.pop_front();
-
-                    numEntriesReadFromCurrentFile = 0;
-
-                    sfen_input_stream = open_sfen_input_file(currentFilename);
-
-                    auto out = sync_region_cout.new_region();
-                    if (sfen_input_stream == nullptr)
-                    {
-                        out << "INFO (sfen_reader): File does not exist: " << currentFilename << '\n';
-                    }
-                    else
-                    {
-                        out << "INFO (sfen_reader): Opened file for reading: " << currentFilename << '\n';
-
-                        // in case the file is empty or was deleted.
-                        if (sfen_input_stream->eof())
-                        {
-                            out << "  - File empty, nothing to read.\n";
-                        }
-                        else
-                        {
-                            return true;
-                        }
-                    }
-                }
-            };
-
-            if (sfen_input_stream == nullptr && !open_next_file())
-            {
-                auto out = sync_region_cout.new_region();
-                out << "INFO (sfen_reader): End of files." << std::endl;
-                end_of_files = true;
-                return;
-            }
-
-            // We want to set the `end_of_files` only after we read everything AND copy to the buffer pool.
-            bool local_end_of_files = false;
-            while (!local_end_of_files)
-            {
-                // Wait for the buffer to run out.
-                // This size() is read only, so you don't need to lock it.
-                while (!stop_flag && num_buffers_in_pool.load() >= sfen_read_size / thread_buffer_size)
-                    sleep(100);
-
-                if (stop_flag)
-                    return;
-
-                PSVector sfens;
-                sfens.reserve(sfen_read_size);
-
-                // Read from the file into the file buffer.
-                while (sfens.size() < sfen_read_size)
-                {
-                    std::optional<PackedSfenValue> p = sfen_input_stream->next();
-                    if (p.has_value())
-                    {
-                        sfens.push_back(*p);
-                        ++numEntriesReadFromCurrentFile;
-                    }
-                    else
-                    {
-                        if (mode == SfenReaderMode::Cyclic
-                            && numEntriesReadFromCurrentFile > 0)
-                        {
-                            // The file contained data so we add it again to the end of the queue.
-                            filenames.emplace_back(currentFilename);
-                        }
-
-                        if(!open_next_file())
-                        {
-                            // There was no next file. Abort.
-                            auto out = sync_region_cout.new_region();
-                            out << "INFO (sfen_reader): End of files." << std::endl;
-                            local_end_of_files = true;
-                            break;
-                        }
-                    }
-                }
-
-                // Shuffle the read phase data.
-                if (shuffle)
-                {
-                    Algo::shuffle(sfens, prng);
-                }
-
-                std::vector<std::unique_ptr<PSVector>> buffers;
-                for (size_t offset = 0; offset < sfens.size(); offset += thread_buffer_size)
-                {
-                    const size_t count =
-                        offset + thread_buffer_size > sfens.size()
-                        ? sfens.size() - offset
-                        : thread_buffer_size;
-
-                    // Delete this pointer on the receiving side.
-                    auto buf = std::make_unique<PSVector>();
-                    buf->resize(count);
-                    memcpy(
-                        buf->data(),
-                        &sfens[offset],
-                        sizeof(PackedSfenValue) * count);
-
-                    buffers.emplace_back(std::move(buf));
-                }
-
-                {
-                    std::unique_lock<std::mutex> lk(mutex);
-
-                    // The mutex lock is required because the%
-                    // contents of packed_sfens_pool are changed.
-
-                    for (auto& buf : buffers)
-                    {
-                        num_buffers_in_pool.fetch_add(1);
-                        packed_sfens_pool.emplace_back(std::move(buf));
-                    }
-                }
-            }
-
-            end_of_files = true;
-        }
-
-    protected:
-
-        // worker thread reading file in background
-        std::thread file_worker_thread;
-
-        // sfen files
-        std::deque<std::string> filenames;
-
-        std::atomic<bool> stop_flag;
-
-        // number of phases read (file to memory buffer)
-        std::atomic<uint64_t> total_read;
-
-        // Do not shuffle when reading the phase.
-        bool shuffle;
-
-        SfenReaderMode mode;
-
-        size_t sfen_read_size;
-        size_t thread_buffer_size;
-
-        // Random number to shuffle when reading the phase
-        PRNG prng;
-
-        // Did you read the files and reached the end?
-        std::atomic<bool> end_of_files;
-
-        // handle of sfen file
-        std::unique_ptr<BasicSfenInputStream> sfen_input_stream;
-
-        // sfen for each thread
-        // (When the thread is used up, the thread should call delete to release it.)
-        std::vector<std::unique_ptr<PSVector>> packed_sfens;
-
-        // Mutex when accessing packed_sfens_pool
-        std::mutex mutex;
-
-        // pool of sfen. The worker thread read from the file is added here.
-        // Each worker thread fills its own packed_sfens[thread_id] from here.
-        // * Lock and access the mutex.
-        std::list<std::unique_ptr<PSVector>> packed_sfens_pool;
-        std::atomic<size_t> num_buffers_in_pool;
-    };
-}

src/tools/sfen_stream.h

@@ -1,222 +0,0 @@
-#ifndef _SFEN_STREAM_H_
-#define _SFEN_STREAM_H_
-
-#include "packed_sfen.h"
-
-#include "extra/nnue_data_binpack_format.h"
-
-#include <optional>
-#include <fstream>
-#include <string>
-#include <memory>
-
-namespace Stockfish::Tools {
-
-    enum struct SfenOutputType
-    {
-        Bin,
-        Binpack
-    };
-
-    static bool ends_with(const std::string& lhs, const std::string& end)
-    {
-        if (end.size() > lhs.size()) return false;
-
-        return std::equal(end.rbegin(), end.rend(), lhs.rbegin());
-    }
-
-    static bool has_extension(const std::string& filename, const std::string& extension)
-    {
-        return ends_with(filename, "." + extension);
-    }
-
-    static std::string filename_with_extension(const std::string& filename, const std::string& ext)
-    {
-        if (ends_with(filename, ext))
-        {
-            return filename;
-        }
-        else
-        {
-            return filename + "." + ext;
-        }
-    }
-
-    struct BasicSfenInputStream
-    {
-        virtual std::optional<PackedSfenValue> next() = 0;
-        virtual bool eof() const = 0;
-        virtual ~BasicSfenInputStream() {}
-    };
-
-    struct BinSfenInputStream : BasicSfenInputStream
-    {
-        static constexpr auto openmode = std::ios::in | std::ios::binary;
-        static inline const std::string extension = "bin";
-
-        BinSfenInputStream(std::string filename) :
-            m_stream(filename, openmode),
-            m_eof(!m_stream)
-        {
-        }
-
-        std::optional<PackedSfenValue> next() override
-        {
-            PackedSfenValue e;
-            if(m_stream.read(reinterpret_cast<char*>(&e), sizeof(PackedSfenValue)))
-            {
-                return e;
-            }
-            else
-            {
-                m_eof = true;
-                return std::nullopt;
-            }
-        }
-
-        bool eof() const override
-        {
-            return m_eof;
-        }
-
-        ~BinSfenInputStream() override {}
-
-    private:
-        std::fstream m_stream;
-        bool m_eof;
-    };
-
-    struct BinpackSfenInputStream : BasicSfenInputStream
-    {
-        static constexpr auto openmode = std::ios::in | std::ios::binary;
-        static inline const std::string extension = "binpack";
-
-        BinpackSfenInputStream(std::string filename) :
-            m_stream(filename, openmode),
-            m_eof(!m_stream.hasNext())
-        {
-        }
-
-        std::optional<PackedSfenValue> next() override
-        {
-            static_assert(sizeof(binpack::nodchip::PackedSfenValue) == sizeof(PackedSfenValue));
-
-            if (!m_stream.hasNext())
-            {
-                m_eof = true;
-                return std::nullopt;
-            }
-
-            auto training_data_entry = m_stream.next();
-            auto v = binpack::trainingDataEntryToPackedSfenValue(training_data_entry);
-            PackedSfenValue psv;
-            // same layout, different types. One is from generic library.
-            std::memcpy(&psv, &v, sizeof(PackedSfenValue));
-
-            return psv;
-        }
-
-        bool eof() const override
-        {
-            return m_eof;
-        }
-
-        ~BinpackSfenInputStream() override {}
-
-    private:
-        binpack::CompressedTrainingDataEntryReader m_stream;
-        bool m_eof;
-    };
-
-    struct BasicSfenOutputStream
-    {
-        virtual void write(const PSVector& sfens) = 0;
-        virtual ~BasicSfenOutputStream() {}
-    };
-
-    struct BinSfenOutputStream : BasicSfenOutputStream
-    {
-        static constexpr auto openmode = std::ios::out | std::ios::binary | std::ios::app;
-        static inline const std::string extension = "bin";
-
-        BinSfenOutputStream(std::string filename) :
-            m_stream(filename_with_extension(filename, extension), openmode)
-        {
-        }
-
-        void write(const PSVector& sfens) override
-        {
-            m_stream.write(reinterpret_cast<const char*>(sfens.data()), sizeof(PackedSfenValue) * sfens.size());
-        }
-
-        ~BinSfenOutputStream() override {}
-
-    private:
-        std::fstream m_stream;
-    };
-
-    struct BinpackSfenOutputStream : BasicSfenOutputStream
-    {
-        static constexpr auto openmode = std::ios::out | std::ios::binary | std::ios::app;
-        static inline const std::string extension = "binpack";
-
-        BinpackSfenOutputStream(std::string filename) :
-            m_stream(filename_with_extension(filename, extension), openmode)
-        {
-        }
-
-        void write(const PSVector& sfens) override
-        {
-            static_assert(sizeof(binpack::nodchip::PackedSfenValue) == sizeof(PackedSfenValue));
-
-            for(auto& sfen : sfens)
-            {
-                // The library uses a type that's different but layout-compatibile.
-                binpack::nodchip::PackedSfenValue e;
-                std::memcpy(&e, &sfen, sizeof(binpack::nodchip::PackedSfenValue));
-                m_stream.addTrainingDataEntry(binpack::packedSfenValueToTrainingDataEntry(e));
-            }
-        }
-
-        ~BinpackSfenOutputStream() override {}
-
-    private:
-        binpack::CompressedTrainingDataEntryWriter m_stream;
-    };
-
-    inline std::unique_ptr<BasicSfenInputStream> open_sfen_input_file(const std::string& filename)
-    {
-        if (has_extension(filename, BinSfenInputStream::extension))
-            return std::make_unique<BinSfenInputStream>(filename);
-        else if (has_extension(filename, BinpackSfenInputStream::extension))
-            return std::make_unique<BinpackSfenInputStream>(filename);
-
-        return nullptr;
-    }
-
-    inline std::unique_ptr<BasicSfenOutputStream> create_new_sfen_output(const std::string& filename, SfenOutputType sfen_output_type)
-    {
-        switch(sfen_output_type)
-        {
-            case SfenOutputType::Bin:
-                return std::make_unique<BinSfenOutputStream>(filename);
-            case SfenOutputType::Binpack:
-                return std::make_unique<BinpackSfenOutputStream>(filename);
-        }
-
-        assert(false);
-        return nullptr;
-    }
-
-    inline std::unique_ptr<BasicSfenOutputStream> create_new_sfen_output(const std::string& filename)
-    {
-        if (has_extension(filename, BinSfenOutputStream::extension))
-            return std::make_unique<BinSfenOutputStream>(filename);
-        else if (has_extension(filename, BinpackSfenOutputStream::extension))
-            return std::make_unique<BinpackSfenOutputStream>(filename);
-
-        return nullptr;
-    }
-}
-
-#endif

src/tools/sfen_writer.h

@@ -1,203 +0,0 @@
-#include "packed_sfen.h"
-#include "sfen_stream.h"
-
-#include "misc.h"
-
-#include "extra/nnue_data_binpack_format.h"
-
-#include "syzygy/tbprobe.h"
-
-#include <cstring>
-#include <fstream>
-#include <limits>
-#include <list>
-#include <memory>
-#include <optional>
-#include <shared_mutex>
-#include <thread>
-#include <atomic>
-
-namespace Stockfish::Tools {
-
-    // Helper class for exporting Sfen
-    struct SfenWriter
-    {
-        // Amount of sfens required to flush the buffer.
-        static constexpr size_t SFEN_WRITE_SIZE = 5000;
-
-        // File name to write and number of threads to create
-        SfenWriter(std::string filename_, int thread_num, uint64_t save_count, SfenOutputType sfen_output_type)
-        {
-            sfen_buffers_pool.reserve((size_t)thread_num * 10);
-            sfen_buffers.resize(thread_num);
-
-            auto out = sync_region_cout.new_region();
-            out << "INFO (sfen_writer): Creating new data file at " << filename_ << std::endl;
-
-            sfen_format = sfen_output_type;
-            output_file_stream = create_new_sfen_output(filename_, sfen_format);
-            filename = filename_;
-            save_every = save_count;
-
-            finished = false;
-
-            file_worker_thread = std::thread([&] { this->file_write_worker(); });
-        }
-
-        ~SfenWriter()
-        {
-            flush();
-
-            finished = true;
-            file_worker_thread.join();
-            output_file_stream.reset();
-
-#if !defined(NDEBUG)
-            {
-                // All buffers should be empty since file_worker_thread
-                // should have written everything before exiting.
-                for (const auto& p : sfen_buffers) { assert(p == nullptr); (void)p ; }
-                assert(sfen_buffers_pool.empty());
-            }
-#endif
-        }
-
-        void write(size_t thread_id, const PackedSfenValue& psv)
-        {
-            // We have a buffer for each thread and add it there.
-            // If the buffer overflows, write it to a file.
-
-            // This buffer is prepared for each thread.
-            auto& buf = sfen_buffers[thread_id];
-
-            // Secure since there is no buf at the first time
-            // and immediately after writing the thread buffer.
-            if (!buf)
-            {
-                buf = std::make_unique<PSVector>();
-                buf->reserve(SFEN_WRITE_SIZE);
-            }
-
-            // Buffer is exclusive to this thread.
-            // There is no need for a critical section.
-            buf->push_back(psv);
-
-            if (buf->size() >= SFEN_WRITE_SIZE)
-            {
-                // If you load it in sfen_buffers_pool, the worker will do the rest.
-
-                // Critical section since sfen_buffers_pool is shared among threads.
-                std::unique_lock<std::mutex> lk(mutex);
-                sfen_buffers_pool.emplace_back(std::move(buf));
-            }
-        }
-
-        void flush()
-        {
-            for (size_t i = 0; i < sfen_buffers.size(); ++i)
-            {
-                flush(i);
-            }
-        }
-
-        // Move what remains in the buffer for your thread to a buffer for writing to a file.
-        void flush(size_t thread_id)
-        {
-            std::unique_lock<std::mutex> lk(mutex);
-
-            auto& buf = sfen_buffers[thread_id];
-
-            // There is a case that buf==nullptr, so that check is necessary.
-            if (buf && buf->size() != 0)
-            {
-                sfen_buffers_pool.emplace_back(std::move(buf));
-            }
-        }
-
-        // Dedicated thread to write to file
-        void file_write_worker()
-        {
-            while (!finished || sfen_buffers_pool.size())
-            {
-                std::vector<std::unique_ptr<PSVector>> buffers;
-                {
-                    std::unique_lock<std::mutex> lk(mutex);
-
-                    // Atomically swap take the filled buffers and
-                    // create a new buffer pool for threads to fill.
-                    buffers = std::move(sfen_buffers_pool);
-                    sfen_buffers_pool = std::vector<std::unique_ptr<PSVector>>();
-                }
-
-                if (!buffers.size())
-                {
-                    // Poor man's condition variable.
-                    sleep(100);
-                }
-                else
-                {
-                    for (auto& buf : buffers)
-                    {
-                        output_file_stream->write(*buf);
-
-                        sfen_write_count += buf->size();
-
-                        // Add the processed number here, and if it exceeds save_every,
-                        // change the file name and reset this counter.
-                        sfen_write_count_current_file += buf->size();
-                        if (sfen_write_count_current_file >= save_every)
-                        {
-                            sfen_write_count_current_file = 0;
-
-                            // Sequential number attached to the file
-                            int n = (int)(sfen_write_count / save_every);
-
-                            // Rename the file and open it again.
-                            // Add ios::app in consideration of overwriting.
-                            // (Depending on the operation, it may not be necessary.)
-                            std::string new_filename = filename + "_" + std::to_string(n);
-                            output_file_stream = create_new_sfen_output(new_filename, sfen_format);
-
-                            auto out = sync_region_cout.new_region();
-                            out << "INFO (sfen_writer): Creating new data file at " << new_filename << std::endl;
-                        }
-                    }
-                }
-            }
-        }
-
-    private:
-
-        std::unique_ptr<BasicSfenOutputStream> output_file_stream;
-
-        // A new net is saved after every save_every sfens are processed.
-        uint64_t save_every = std::numeric_limits<uint64_t>::max();
-
-        // File name passed in the constructor
-        std::string filename;
-
-        // Thread to write to the file
-        std::thread file_worker_thread;
-
-        // Flag that all threads have finished
-        std::atomic<bool> finished;
-
-        SfenOutputType sfen_format;
-
-        // buffer before writing to file
-        // sfen_buffers is the buffer for each thread
-        // sfen_buffers_pool is a buffer for writing.
-        // After loading the phase in the former buffer by SFEN_WRITE_SIZE,
-        // transfer it to the latter.
-        std::vector<std::unique_ptr<PSVector>> sfen_buffers;
-        std::vector<std::unique_ptr<PSVector>> sfen_buffers_pool;
-
-        // Mutex required to access sfen_buffers_pool
-        std::mutex mutex;
-
-        // Number of sfens written in total, and the
-        // number of sfens written in the current file.
-        uint64_t sfen_write_count = 0;
-        uint64_t sfen_write_count_current_file = 0;
-    };
-}

src/tools/stats.h

@@ -1,12 +0,0 @@
-#ifndef _STATS_H_
-#define _STATS_H_
-
-#include <sstream>
-
-namespace Stockfish::Tools::Stats {
-
-    void gather_statistics(std::istringstream& is);
-
-}
-
-#endif

src/tools/training_data_generator.cpp

@@ -1,952 +0,0 @@
-#include "training_data_generator.h"
-
-#include "sfen_writer.h"
-#include "packed_sfen.h"
-#include "opening_book.h"
-
-#include "misc.h"
-#include "position.h"
-#include "thread.h"
-#include "tt.h"
-#include "uci.h"
-
-#include "extra/nnue_data_binpack_format.h"
-
-#include "nnue/evaluate_nnue.h"
-
-#include "syzygy/tbprobe.h"
-
-#include <atomic>
-#include <chrono>
-#include <climits>
-#include <cmath>
-#include <cstring>
-#include <fstream>
-#include <iomanip>
-#include <limits>
-#include <list>
-#include <memory>
-#include <optional>
-#include <random>
-#include <shared_mutex>
-#include <sstream>
-#include <unordered_set>
-
-using namespace std;
-
-namespace Stockfish::Tools
-{
-    // Class to generate sfen with multiple threads
-    struct TrainingDataGenerator
-    {
-        struct Params
-        {
-            // Min and max depths for search during gensfen
-            int search_depth_min = 3;
-            int search_depth_max = -1;
-
-            // Number of the nodes to be searched.
-            // 0 represents no limits.
-            uint64_t nodes = 0;
-
-            // Upper limit of evaluation value of generated situation
-            int eval_limit = 3000;
-
-            // minimum ply with random move
-            // maximum ply with random move
-            // Number of random moves in one station
-            int random_move_minply = 1;
-            int random_move_maxply = 24;
-            int random_move_count = 5;
-
-            // Move kings with a probability of 1/N when randomly moving like Apery software.
-            // When you move the king again, there is a 1/N chance that it will randomly moved
-            // once in the opponent's turn.
-            // Apery has N=2. Specifying 0 here disables this function.
-            int random_move_like_apery = 0;
-
-            // For when using multi pv instead of random move.
-            // random_multi_pv is the number of candidates for MultiPV.
-            // When adopting the move of the candidate move, the difference
-            // between the evaluation value of the move of the 1st place
-            // and the evaluation value of the move of the Nth place is.
-            // Must be in the range random_multi_pv_diff.
-            // random_multi_pv_depth is the search depth for MultiPV.
-            int random_multi_pv = 0;
-            int random_multi_pv_diff = 32000;
-            int random_multi_pv_depth = -1;
-            uint64_t random_multi_pv_nodes = 0;
-
-            // The minimum and maximum ply (number of steps from
-            // the initial phase) of the sfens to write out.
-            int write_minply = 16;
-            int write_maxply = 400;
-
-            uint64_t save_every = std::numeric_limits<uint64_t>::max();
-
-            std::string output_file_name = "training_data";
-
-            SfenOutputType sfen_format = SfenOutputType::Binpack;
-
-            std::string seed;
-
-            bool write_out_draw_game_in_training_data_generation = true;
-            bool detect_draw_by_consecutive_low_score = true;
-            bool detect_draw_by_insufficient_mating_material = true;
-
-            uint64_t num_threads;
-
-            std::string book;
-
-            void enforce_constraints()
-            {
-                search_depth_max = std::max(search_depth_min, search_depth_max);
-
-                // Limit the maximum to a one-stop score. (Otherwise you might not end the loop)
-                eval_limit = std::min(eval_limit, (int)mate_in(2));
-
-                save_every = std::max(save_every, REPORT_STATS_EVERY);
-
-                num_threads = Options["Threads"];
-
-                if (random_multi_pv_depth == -1)
-                    random_multi_pv_depth = search_depth_max;
-
-                if (random_multi_pv_nodes == 0)
-                    random_multi_pv_nodes = nodes;
-            }
-        };
-
-        // Hash to limit the export of identical sfens
-        static constexpr uint64_t GENSFEN_HASH_SIZE = 64 * 1024 * 1024;
-        // It must be 2**N because it will be used as the mask to calculate hash_index.
-        static_assert((GENSFEN_HASH_SIZE& (GENSFEN_HASH_SIZE - 1)) == 0);
-
-        static constexpr uint64_t REPORT_DOT_EVERY = 5000;
-        static constexpr uint64_t REPORT_STATS_EVERY = 200000;
-        static_assert(REPORT_STATS_EVERY % REPORT_DOT_EVERY == 0);
-
-        TrainingDataGenerator(
-            const Params& prm
-        ) :
-            params(prm),
-            sfen_writer(prm.output_file_name, prm.num_threads, prm.save_every, prm.sfen_format)
-        {
-            hash.resize(GENSFEN_HASH_SIZE);
-            prngs.reserve(prm.num_threads);
-            auto seed = prm.seed;
-            for (uint64_t i = 0; i < prm.num_threads; ++i)
-            {
-                prngs.emplace_back(seed);
-                seed = prngs.back().next_random_seed();
-            }
-
-            if (!prm.book.empty())
-            {
-                opening_book = open_opening_book(prm.book, prngs[0]);
-                if (opening_book == nullptr)
-                {
-                    std::cout << "WARNING: Failed to open opening book " << prm.book << ". Falling back to startpos.\n";
-                }
-            }
-
-            // Output seed to veryfy by the user if it's not identical by chance.
-            std::cout << prngs[0] << std::endl;
-        }
-
-        void generate(uint64_t limit, uint64_t limit_seconds);
-
-    private:
-        Params params;
-
-        std::vector<PRNG> prngs;
-
-        std::mutex stats_mutex;
-        TimePoint last_stats_report_time;
-
-        // sfen exporter
-        SfenWriter sfen_writer;
-
-        SynchronizedRegionLogger::Region out;
-
-        vector<Key> hash; // 64MB*sizeof(HASH_KEY) = 512MB
-
-        std::unique_ptr<OpeningBook> opening_book;
-
-        static void set_gensfen_search_limits();
-
-        void generate_worker(
-            Thread& th,
-            std::atomic<uint64_t>& counter,
-            uint64_t limit,
-            uint64_t limit_seconds);
-
-        bool was_seen_before(const Position& pos);
-
-        optional<int8_t> get_current_game_result(
-            Position& pos,
-            const vector<int>& move_hist_scores) const;
-
-        vector<uint8_t> generate_random_move_flags(PRNG& prng);
-
-        optional<Move> choose_random_move(
-            PRNG& prng,
-            Position& pos,
-            std::vector<uint8_t>& random_move_flag,
-            int ply,
-            int& random_move_c);
-
-        bool commit_psv(
-            Thread& th,
-            PSVector& sfens,
-            int8_t lastTurnIsWin,
-            std::atomic<uint64_t>& counter,
-            uint64_t limit,
-            Color result_color);
-
-        void initial_report();
-        void report(uint64_t done, uint64_t new_done);
-
-        void maybe_report(uint64_t done);
-    };
-
-    void TrainingDataGenerator::set_gensfen_search_limits()
-    {
-        // About Search::Limits
-        // Be careful because this member variable is global and affects other threads.
-        auto& limits = Search::Limits;
-
-        // Make the search equivalent to the "go infinite" command. (Because it is troublesome if time management is done)
-        limits.infinite = true;
-
-        // Since PV is an obstacle when displayed, erase it.
-        limits.silent = true;
-
-        // If you use this, it will be compared with the accumulated nodes of each thread. Therefore, do not use it.
-        limits.nodes = 0;
-
-        // depth is also processed by the one passed as an argument of Tools::search().
-        limits.depth = 0;
-    }
-
-    void TrainingDataGenerator::generate(uint64_t limit, uint64_t limit_seconds)
-    {
-        last_stats_report_time = 0;
-
-        initial_report();
-
-        set_gensfen_search_limits();
-
-        std::atomic<uint64_t> counter{0};
-        Threads.execute_with_workers([&counter, limit, limit_seconds, this](Thread& th) {
-            generate_worker(th, counter, limit, limit_seconds);
-        });
-        Threads.wait_for_workers_finished();
-
-        sfen_writer.flush();
-
-        report(counter.load(), counter.load() % REPORT_STATS_EVERY + 1);
-
-        std::cout << std::endl;
-    }
-
-    void TrainingDataGenerator::generate_worker(
-        Thread& th,
-        std::atomic<uint64_t>& counter,
-        uint64_t limit,
-        uint64_t limit_seconds)
-    {
-        // For the time being, it will be treated as a draw
-        // at the maximum number of steps to write.
-        // Maximum StateInfo + Search PV to advance to leaf buffer
-        std::vector<StateInfo, AlignedAllocator<StateInfo>> states(
-            params.write_maxply + MAX_PLY /* == search_depth_min + α */);
-
-        StateInfo si;
-
-        auto& prng = prngs[th.id()];
-
-        // end flag
-        bool quit = false;
-
-        const auto start_time = now();
-
-        const bool frc = Options["UCI_Chess960"];
-        // repeat until the specified number of times
-        while (!quit)
-        {
-            if (static_cast<uint64_t>(now() - start_time) / 1000 >= limit_seconds)
-            {
-                break;
-            }
-
-            // It is necessary to set a dependent thread for Position.
-            // When parallelizing, Threads (since this is a vector<Thread*>,
-            // Do the same for up to Threads[0]...Threads[thread_num-1].
-            auto& pos = th.rootPos;
-            if (opening_book != nullptr)
-            {
-                auto& fen = opening_book->next_fen();
-                pos.set(fen, frc, &si, &th);
-            }
-            else
-            {
-                pos.set(StartFEN, frc, &si, &th);
-            }
-
-            int resign_counter = 0;
-            bool should_resign = prng.rand(10) > 1;
-            // Vector for holding the sfens in the current simulated game.
-            PSVector packed_sfens;
-            packed_sfens.reserve(params.write_maxply + MAX_PLY);
-
-            // Precomputed flags. Used internally by choose_random_move.
-            vector<uint8_t> random_move_flag = generate_random_move_flags(prng);
-
-            // A counter that keeps track of the number of random moves
-            // When random_move_minply == -1, random moves are
-            // performed continuously, so use it at this time.
-            // Used internally by choose_random_move.
-            int actual_random_move_count = 0;
-
-            // Save history of move scores for adjudication
-            vector<int> move_hist_scores;
-
-            auto flush_psv = [&](int8_t result) {
-                quit = commit_psv(th, packed_sfens, result, counter, limit, pos.side_to_move());
-            };
-
-            for (int ply = 0; ; ++ply)
-            {
-                // Current search depth
-                const int depth = params.search_depth_min + (int)prng.rand(params.search_depth_max - params.search_depth_min + 1);
-
-                // Starting search calls init_for_search
-                auto [search_value, search_pv] = Search::search(pos, depth, 1, params.nodes);
-
-                // This has to be performed after search because it needs to know
-                // rootMoves which are filled in init_for_search.
-                const auto result = get_current_game_result(pos, move_hist_scores);
-                if (result.has_value())
-                {
-                    flush_psv(result.value());
-                    break;
-                }
-
-                // Always adjudivate by eval limit.
-                // Also because of this we don't have to check for TB/MATE scores
-                if (abs(search_value) >= params.eval_limit)
-                {
-                    resign_counter++;
-                    if ((should_resign && resign_counter >= 4) || abs(search_value) >= VALUE_KNOWN_WIN) {
-                        flush_psv((search_value >= params.eval_limit) ? 1 : -1);
-                        break;
-                    }
-                }
-                else
-                {
-                    resign_counter = 0;
-                }
-
-                // In case there is no PV and the game was not ended here
-                // there is nothing we can do, we can't continue the game,
-                // we don't know the result, so discard this game.
-                if (search_pv.empty())
-                {
-                    break;
-                }
-
-                // Save the move score for adjudication.
-                move_hist_scores.push_back(search_value);
-
-                // Discard stuff before write_minply is reached
-                // because it can harm training due to overfitting.
-                // Initial positions would be too common.
-                if (ply >= params.write_minply && !was_seen_before(pos))
-                {
-                    auto& psv = packed_sfens.emplace_back();
-
-                    // Here we only write the position data.
-                    // Result is added after the whole game is done.
-                    pos.sfen_pack(psv.sfen, pos.is_chess960());
-
-                    psv.score = search_value;
-                    psv.move = search_pv[0];
-                    psv.gamePly = ply;
-                }
-
-                // Update the next move according to best search result or random move.
-                auto random_move = choose_random_move(prng, pos, random_move_flag, ply, actual_random_move_count);
-                const Move next_move = random_move.has_value() ? *random_move : search_pv[0];
-
-                // We don't have the whole game yet, but it ended,
-                // so the writing process ends and the next game starts.
-                // This shouldn't really happen.
-                if (!is_ok(next_move))
-                {
-                    break;
-                }
-
-                // Do move.
-                pos.do_move(next_move, states[ply]);
-            }
-        }
-    }
-
-    bool TrainingDataGenerator::was_seen_before(const Position& pos)
-    {
-        // Look into the position hashtable to see if the same
-        // position was seen before.
-        // This is a good heuristic to exlude already seen
-        // positions without many false positives.
-        auto key = pos.key();
-        auto hash_index = (size_t)(key & (GENSFEN_HASH_SIZE - 1));
-        auto old_key = hash[hash_index];
-        if (key == old_key)
-        {
-            return true;
-        }
-        else
-        {
-            // Replace with the current key.
-            hash[hash_index] = key;
-            return false;
-        }
-    }
-
-    optional<int8_t> TrainingDataGenerator::get_current_game_result(
-        Position& pos,
-        const vector<int>& move_hist_scores) const
-    {
-        // Variables for draw adjudication.
-        // Todo: Make this as an option.
-
-        // start the adjudication when ply reaches this value
-        constexpr int adj_draw_ply = 80;
-
-        // 4 move scores for each side have to be checked
-        constexpr int adj_draw_cnt = 8;
-
-        // move score in CP
-        constexpr int adj_draw_score = 0;
-
-        // For the time being, it will be treated as a
-        // draw at the maximum number of steps to write.
-        const int ply = move_hist_scores.size();
-
-        // has it reached the max length or is a draw by fifty-move rule
-        // or by 3-fold repetition
-        if (ply >= params.write_maxply
-            || pos.is_fifty_move_draw()
-            || pos.is_three_fold_repetition())
-        {
-            return 0;
-        }
-
-        if(pos.this_thread()->rootMoves.empty())
-        {
-            // If there is no legal move
-            return pos.checkers()
-                ? -1 /* mate */
-                : 0 /* stalemate */;
-        }
-
-        // Adjudicate game to a draw if the last 4 scores of each engine is 0.
-        if (params.detect_draw_by_consecutive_low_score)
-        {
-            if (ply >= adj_draw_ply)
-            {
-                int num_cons_plies_within_draw_score = 0;
-                bool is_adj_draw = false;
-
-                for (auto it = move_hist_scores.rbegin();
-                    it != move_hist_scores.rend(); ++it)
-                {
-                    if (abs(*it) <= adj_draw_score)
-                    {
-                        num_cons_plies_within_draw_score++;
-                    }
-                    else
-                    {
-                        // Draw scores must happen on consecutive plies
-                        break;
-                    }
-
-                    if (num_cons_plies_within_draw_score >= adj_draw_cnt)
-                    {
-                        is_adj_draw = true;
-                        break;
-                    }
-                }
-
-                if (is_adj_draw)
-                {
-                    return 0;
-                }
-            }
-        }
-
-        // Draw by insufficient mating material
-        if (params.detect_draw_by_insufficient_mating_material)
-        {
-            if (pos.count<ALL_PIECES>() <= 4)
-            {
-                int num_pieces = pos.count<ALL_PIECES>();
-
-                // (1) KvK
-                if (num_pieces == 2)
-                {
-                    return 0;
-                }
-
-                // (2) KvK + 1 minor piece
-                if (num_pieces == 3)
-                {
-                    int minor_pc = pos.count<BISHOP>(WHITE) + pos.count<KNIGHT>(WHITE) +
-                        pos.count<BISHOP>(BLACK) + pos.count<KNIGHT>(BLACK);
-                    if (minor_pc == 1)
-                    {
-                        return 0;
-                    }
-                }
-
-                // (3) KBvKB, bishops of the same color
-                else if (num_pieces == 4)
-                {
-                    if (pos.count<BISHOP>(WHITE) == 1 && pos.count<BISHOP>(BLACK) == 1)
-                    {
-                        // Color of bishops is black.
-                        if ((pos.pieces(WHITE, BISHOP) & DarkSquares)
-                            && (pos.pieces(BLACK, BISHOP) & DarkSquares))
-                        {
-                            return 0;
-                        }
-                        // Color of bishops is white.
-                        if ((pos.pieces(WHITE, BISHOP) & ~DarkSquares)
-                            && (pos.pieces(BLACK, BISHOP) & ~DarkSquares))
-                        {
-                            return 0;
-                        }
-                    }
-                }
-            }
-        }
-
-        return nullopt;
-    }
-
-    vector<uint8_t> TrainingDataGenerator::generate_random_move_flags(PRNG& prng)
-    {
-        vector<uint8_t> random_move_flag;
-
-        // Depending on random move selection parameters setup
-        // the array of flags that indicates whether a random move
-        // be taken at a given ply.
-
-        // Make an array like a[0] = 0 ,a[1] = 1, ...
-        // Fisher-Yates shuffle and take out the first N items.
-        // Actually, I only want N pieces, so I only need
-        // to shuffle the first N pieces with Fisher-Yates.
-
-        vector<int> a;
-        a.reserve((size_t)params.random_move_maxply);
-
-        // random_move_minply ,random_move_maxply is specified by 1 origin,
-        // Note that we are handling 0 origin here.
-        for (int i = std::max(params.random_move_minply - 1, 0); i < params.random_move_maxply; ++i)
-        {
-            a.push_back(i);
-        }
-
-        // In case of Apery random move, insert() may be called random_move_count times.
-        // Reserve only the size considering it.
-        random_move_flag.resize((size_t)params.random_move_maxply + params.random_move_count);
-
-        // A random move that exceeds the size() of a[] cannot be applied, so limit it.
-        for (int i = 0; i < std::min(params.random_move_count, (int)a.size()); ++i)
-        {
-            swap(a[i], a[prng.rand((uint64_t)a.size() - i) + i]);
-            random_move_flag[a[i]] = true;
-        }
-
-        return random_move_flag;
-    }
-
-    optional<Move> TrainingDataGenerator::choose_random_move(
-        PRNG& prng,
-        Position& pos,
-        std::vector<uint8_t>& random_move_flag,
-        int ply,
-        int& random_move_c)
-    {
-        optional<Move> random_move;
-
-        // Randomly choose one from legal move
-        if (
-            // 1. Random move of random_move_count times from random_move_minply to random_move_maxply
-            (params.random_move_minply != -1 && ply < (int)random_move_flag.size() && random_move_flag[ply]) ||
-            // 2. A mode to perform random move of random_move_count times after leaving the startpos
-            (params.random_move_minply == -1 && random_move_c < params.random_move_count))
-        {
-            ++random_move_c;
-
-            // It's not a mate, so there should be one legal move...
-            if (params.random_multi_pv == 0)
-            {
-                // Normal random move
-                MoveList<LEGAL> list(pos);
-
-                // I don't really know the goodness and badness of making this the Apery method.
-                if (params.random_move_like_apery == 0
-                    || prng.rand(params.random_move_like_apery) != 0)
-                {
-                    // Normally one move from legal move
-                    random_move = list.at((size_t)prng.rand((uint64_t)list.size()));
-                }
-                else
-                {
-                    // if you can move the king, move the king
-                    Move moves[8]; // Near 8
-                    Move* p = &moves[0];
-                    for (auto& m : list)
-                    {
-                        if (type_of(pos.moved_piece(m)) == KING)
-                        {
-                            *(p++) = m;
-                        }
-                    }
-
-                    size_t n = p - &moves[0];
-                    if (n != 0)
-                    {
-                        // move to move the king
-                        random_move = moves[prng.rand(n)];
-
-                        // In Apery method, at this time there is a 1/2 chance
-                        // that the opponent will also move randomly
-                        if (prng.rand(2) == 0)
-                        {
-                            // Is it a simple hack to add a "1" next to random_move_flag[ply]?
-                            random_move_flag.insert(random_move_flag.begin() + ply + 1, 1, true);
-                        }
-                    }
-                    else
-                    {
-                        // Normally one move from legal move
-                        random_move = list.at((size_t)prng.rand((uint64_t)list.size()));
-                    }
-                }
-            }
-            else
-            {
-                Search::search(pos, params.random_multi_pv_depth, params.random_multi_pv, params.random_multi_pv_nodes);
-
-                // Select one from the top N hands of root Moves
-                auto& rm = pos.this_thread()->rootMoves;
-
-                uint64_t s = min((uint64_t)rm.size(), (uint64_t)params.random_multi_pv);
-                for (uint64_t i = 1; i < s; ++i)
-                {
-                    // The difference from the evaluation value of rm[0] must
-                    // be within the range of random_multi_pv_diff.
-                    // It can be assumed that rm[x].score is arranged in descending order.
-                    if (rm[0].score > rm[i].score + params.random_multi_pv_diff)
-                    {
-                        s = i;
-                        break;
-                    }
-                }
-
-                random_move = rm[prng.rand(s)].pv[0];
-            }
-        }
-
-        return random_move;
-    }
-
-    // Write out the phases loaded in sfens to a file.
-    // result: win/loss in the next phase after the final phase in sfens
-    // 1 when winning. -1 when losing. Pass 0 for a draw.
-    // Return value: true if the specified number of
-    // sfens has already been reached and the process ends.
-    bool TrainingDataGenerator::commit_psv(
-        Thread& th,
-        PSVector& sfens,
-        int8_t result,
-        std::atomic<uint64_t>& counter,
-        uint64_t limit,
-        Color result_color)
-    {
-        if (!params.write_out_draw_game_in_training_data_generation && result == 0)
-        {
-            // We didn't write anything so why quit.
-            return false;
-        }
-
-        auto side_to_move_from_sfen = [](auto& sfen){
-            return (Color)(sfen.sfen.data[0] & 1);
-        };
-
-        // From the final stage (one step before) to the first stage, give information on the outcome of the game for each stage.
-        // The phases stored in sfens are assumed to be continuous (in order).
-        for (auto it = sfens.rbegin(); it != sfens.rend(); ++it)
-        {
-            // The side to move is packed as the lowest bit of the first byte
-            const Color side_to_move = side_to_move_from_sfen(*it);
-            it->game_result = side_to_move == result_color ? result : -result;
-        }
-
-        const bool frc = th.rootPos.is_chess960();
-        // Write sfens in move order to make potential compression easier
-        for (auto& sfen : sfens)
-        {
-            // Skip positions with castling bestmove in FRC so that we don't
-            // need to support it in the trainer.
-            if (frc && type_of((Move)sfen.move) == CASTLING)
-            {
-                continue;
-            }
-
-            // Return true if there is already enough data generated.
-            const auto iter = counter.fetch_add(1);
-            if (iter >= limit)
-                return true;
-
-            // because `iter` was done, now we do one more
-            maybe_report(iter + 1);
-
-            // Write out one sfen.
-            sfen_writer.write(th.id(), sfen);
-        }
-
-        return false;
-    }
-
-    void TrainingDataGenerator::initial_report()
-    {
-        out = sync_region_cout.new_region();
-
-        const auto now_time = now();
-
-        out
-            << '\n'
-            << 0 << " sfens, "
-            << "at " << now_string() << endl;
-
-        last_stats_report_time = now_time;
-
-        out = sync_region_cout.new_region();
-    }
-
-    void TrainingDataGenerator::report(uint64_t done, uint64_t new_done)
-    {
-        const auto now_time = now();
-        const TimePoint elapsed = now_time - last_stats_report_time + 1;
-
-        out
-            << '\n'
-            << done << " sfens, "
-            << new_done * 1000 / elapsed << " sfens/second, "
-            << "at " << now_string() << endl;
-
-        last_stats_report_time = now_time;
-
-        out = sync_region_cout.new_region();
-    }
-
-    void TrainingDataGenerator::maybe_report(uint64_t done)
-    {
-        if (done % REPORT_DOT_EVERY == 0)
-        {
-            std::lock_guard lock(stats_mutex);
-
-            if (last_stats_report_time == 0)
-            {
-                last_stats_report_time = now();
-                out = sync_region_cout.new_region();
-            }
-
-            if (done != 0)
-            {
-                out << '.';
-
-                if (done % REPORT_STATS_EVERY == 0)
-                {
-                    report(done, REPORT_STATS_EVERY);
-                }
-            }
-        }
-    }
-
-    // Command to generate a game record
-    void generate_training_data(istringstream& is)
-    {
-        // Number of generated game records default = 8 billion phases (Ponanza specification)
-        uint64_t loop_max = 8000000000UL;
-        uint64_t time_max = 8000000000UL;
-
-        TrainingDataGenerator::Params params;
-
-        // Add a random number to the end of the file name.
-        bool random_file_name = false;
-        std::string sfen_format = "binpack";
-
-        string token;
-        while (true)
-        {
-            token = "";
-            is >> token;
-            if (token == "")
-                break;
-
-            if (token == "depth")
-            {
-                is >> params.search_depth_min;
-                params.search_depth_max = params.search_depth_min;
-            }
-            else if (token == "min_depth")
-                is >> params.search_depth_min;
-            else if (token == "max_depth")
-                is >> params.search_depth_max;
-            else if (token == "nodes")
-                is >> params.nodes;
-            else if (token == "count")
-                is >> loop_max;
-            else if (token == "max_time_seconds")
-                is >> time_max;
-            else if (token == "max_time_minutes")
-            {
-                is >> time_max;
-                time_max *= 60;
-            }
-            else if (token == "max_time_hours")
-            {
-                is >> time_max;
-                time_max *= 3600;
-            }
-            else if (token == "output_file_name")
-                is >> params.output_file_name;
-            else if (token == "eval_limit")
-                is >> params.eval_limit;
-            else if (token == "random_move_min_ply")
-                is >> params.random_move_minply;
-            else if (token == "random_move_max_ply")
-                is >> params.random_move_maxply;
-            else if (token == "random_move_count")
-                is >> params.random_move_count;
-            else if (token == "random_move_like_apery")
-                is >> params.random_move_like_apery;
-            else if (token == "random_multi_pv")
-                is >> params.random_multi_pv;
-            else if (token == "random_multi_pv_diff")
-                is >> params.random_multi_pv_diff;
-            else if (token == "random_multi_pv_depth")
-                is >> params.random_multi_pv_depth;
-            else if (token == "random_multi_pv_nodes")
-                is >> params.random_multi_pv_nodes;
-            else if (token == "write_min_ply")
-                is >> params.write_minply;
-            else if (token == "write_max_ply")
-                is >> params.write_maxply;
-            else if (token == "save_every")
-                is >> params.save_every;
-            else if (token == "book")
-                is >> params.book;
-            else if (token == "random_file_name")
-                is >> random_file_name;
-            else if (token == "keep_draws")
-                is >> params.write_out_draw_game_in_training_data_generation;
-            else if (token == "adjudicate_draws_by_score")
-                is >> params.detect_draw_by_consecutive_low_score;
-            else if (token == "adjudicate_draws_by_insufficient_material")
-                is >> params.detect_draw_by_insufficient_mating_material;
-            else if (token == "data_format")
-                is >> sfen_format;
-            else if (token == "seed")
-                is >> params.seed;
-            else if (token == "set_recommended_uci_options")
-            {
-                UCI::setoption("Skill Level", "20");
-                UCI::setoption("UCI_LimitStrength", "false");
-                UCI::setoption("PruneAtShallowDepth", "false");
-                UCI::setoption("EnableTranspositionTable", "true");
-            }
-            else
-            {
-                cout << "ERROR: Unknown option " << token << ". Exiting...\n";
-                return;
-            }
-        }
-
-        if (!sfen_format.empty())
-        {
-            if (sfen_format == "bin")
-                params.sfen_format = SfenOutputType::Bin;
-            else if (sfen_format == "binpack")
-                params.sfen_format = SfenOutputType::Binpack;
-            else
-                cout << "WARNING: Unknown sfen format `" << sfen_format << "`. Using bin\n";
-        }
-
-        if (random_file_name)
-        {
-            // Give a random number to output_file_name at this point.
-            // Do not use std::random_device().  Because it always the same integers on MinGW.
-            PRNG r(params.seed);
-
-            // Just in case, reassign the random numbers.
-            for (int i = 0; i < 10; ++i)
-                r.rand(1);
-
-            auto to_hex = [](uint64_t u) {
-                std::stringstream ss;
-                ss << std::hex << u;
-                return ss.str();
-            };
-
-            // I don't want to wear 64bit numbers by accident, so I'next_move going to make a 64bit number 2 just in case.
-            params.output_file_name += "_" + to_hex(r.rand<uint64_t>()) + to_hex(r.rand<uint64_t>());
-        }
-
-        params.enforce_constraints();
-
-        std::cout << "INFO: Executing generate_training_data command\n";
-
-        std::cout << "INFO: Parameters:\n";
-        std::cout
-            << "  - search_depth_min       = " << params.search_depth_min << endl
-            << "  - search_depth_max       = " << params.search_depth_max << endl
-            << "  - nodes                  = " << params.nodes << endl
-            << "  - count                  = " << loop_max << endl
-            << "  - max_time_seconds       = " << time_max << endl
-            << "  - eval_limit             = " << params.eval_limit << endl
-            << "  - num threads (UCI)      = " << params.num_threads << endl
-            << "  - random_move_min_ply    = " << params.random_move_minply << endl
-            << "  - random_move_max_ply    = " << params.random_move_maxply << endl
-            << "  - random_move_count      = " << params.random_move_count << endl
-            << "  - random_move_like_apery = " << params.random_move_like_apery << endl
-            << "  - random_multi_pv        = " << params.random_multi_pv << endl
-            << "  - random_multi_pv_diff   = " << params.random_multi_pv_diff << endl
-            << "  - random_multi_pv_depth  = " << params.random_multi_pv_depth << endl
-            << "  - random_multi_pv_nodes  = " << params.random_multi_pv_nodes << endl
-            << "  - write_min_ply          = " << params.write_minply << endl
-            << "  - write_max_ply          = " << params.write_maxply << endl
-            << "  - book                   = " << params.book << endl
-            << "  - output_file_name       = " << params.output_file_name << endl
-            << "  - save_every             = " << params.save_every << endl
-            << "  - random_file_name       = " << random_file_name << endl
-            << "  - write_drawn_games      = " << params.write_out_draw_game_in_training_data_generation << endl
-            << "  - draw by low score      = " << params.detect_draw_by_consecutive_low_score << endl
-            << "  - draw by insuff. mat.   = " << params.detect_draw_by_insufficient_mating_material << endl;
-
-        // Show if the training data generator uses NNUE.
-        Eval::NNUE::verify();
-
-        Threads.main()->ponder = false;
-
-        TrainingDataGenerator gensfen(params);
-        gensfen.generate(loop_max, time_max);
-
-        std::cout << "INFO: generate_training_data finished." << endl;
-    }
-}

src/tools/training_data_generator.h

@@ -1,14 +0,0 @@
-#ifndef _GENSFEN_H_
-#define _GENSFEN_H_
-
-#include "position.h"
-
-#include <sstream>
-
-namespace Stockfish::Tools {
-
-    // Automatic generation of teacher position
-    void generate_training_data(std::istringstream& is);
-}
-
-#endif