2021 May 29 Breaking Changes Update (#13034)

* Add Per Key functionality for AutoShift (#11536)

* LED Matrix: Reactive effect buffers & advanced indicators (#12588)

* [Keyboard] kint36: switch to sym_eager_pk debouncing (#12626)

* [Keyboard] kint2pp: reduce input latency by ≈10ms (#12625)

* LED Matrix: Split (#12633)

* [CI] Format code according to conventions (#12650)

* feat: infinite timeout for leader key (#6580)

* feat: implement leader_no_timeout logic

* docs(leader_key): infinite leader timeout docs

* Format code according to conventions (#12680)

* Update ADC driver for STM32F1xx, STM32F3xx, STM32F4xx (#12403)

* Fix default ADC_RESOLUTION for ADCv3 (and ADCv4)

Recent ChibiOS update removed ADC_CFGR1_RES_10BIT from the ADCv3 headers
(that macro should not have been there, because ADCv3 has CFGR instead of
CFGR1).  Fix the default value for ADC_RESOLUTION to use ADC_CFGR_RES_10BITS
if it is defined (that name is used for ADCv3 and ADCv4).

* Update ADC docs to match the actually used resolution

ADC driver for ChibiOS actually uses the 10-bit resolution by default
(probably to match AVR); fix the documentation accordingly.  Also add
both ADC_CFGR_RES_10BITS and ADC_CFGR1_RES_10BIT constants (these names
differ according to the ADC implementation in the particular MCU).

* Fix pinToMux() for B12 and B13 on STM32F3xx

Testing on STM32F303CCT6 revealed that the ADC mux values for B12 and
B13 pins were wrong.

* Add support for all possible analog pins on STM32F1xx

Added ADC mux values for pins A0...A7, B0, B1, C0...C5 on STM32F1xx
(they are the same at least for STM32F103x8 and larger F103 devices, and
also F102, F105, F107 families).  Actually tested on STM32F103C8T6
(therefore pins C0...C5 were not tested).

Pins F6...F10, which are present on STM32F103x[C-G] in 144-pin packages,
cannot be supported at the moment, because those pins are connected only
to ADC3, but the ChibiOS ADC driver for STM32F1xx supports only ADC1.

* Add support for all possible analog pins on STM32F4xx

Added ADC mux values for pins A0...A7, B0, B1, C0...C5 and optionally
F3...F10 (if STM32_ADC_USE_ADC3 is enabled).  These mux values are
apparently the same for all F4xx devices, except some smaller devices may
not have ADC3.

Actually tested on STM32F401CCU6, STM32F401CEU6, STM32F411CEU6 (using
various WeAct “Blackpill” boards); only pins A0...A7, B0, B1 were tested.

Pins F3...F10 are inside `#if STM32_ADC_USE_ADC3` because some devices
which don't have ADC3 also don't have the GPIOF port, therefore the code
which refers to Fx pins does not compile.

* Fix STM32F3xx ADC mux table in documentation

The ADC driver documentation had some errors in the mux table for STM32F3xx.
Fix this table to match the datasheet and the actual code (mux settings for
B12 and B13 were also tested on a real STM32F303CCT6 chip).

* Add STM32F1xx ADC pins to the documentation

* Add STM32F4xx ADC pins to the documentation

* Add initial support for tinyuf2 bootloader (when hosted on F411 blackpill) (#12600)

* Add support for jumping to tinyuf2 bootloader. Adds blackpill UF2 example.

* Update flashing.md

* Update chconf.h

* Update config.h

* Update halconf.h

* Update mcuconf.h

* eeprom driver: Refactor where eeprom driver initialisation (and EEPROM emulation initialisation) occurs to make it non-target-specific. (#12671)

* Add support for MCU = STM32F446 (#12619)

* Add support for MCU = STM32F446

* Update platforms/chibios/GENERIC_STM32_F446XE/configs/config.h

* Restore mcuconf.h to the one used by RT-STM32F446RE-NUCLEO64

* stm32f446: update mcuconf.h and board.h for 16MHz operation, with USB enabled, and other peripherals disabled.

* Format code according to conventions (#12682)

* Format code according to conventions (#12687)

* Add STM32L433 and L443 support (#12063)

* initial L433 commit

* change to XC

* fix L433

* disable all peripherals

* update system and peripheral clocks

* 433 change

* use its own board  files

* revert its own board files

* l433 specific change

* fix stm32l432xx define

* remove duplicate #define

* fix bootloader jump

* move to L443xx and add i2c2, spi2, usart3 to mcuconf.h

* move to L443

* move to L443

* fix sdmmc in mcuconf.h

* include STM32L443

* add L443

* Include L443 in compatible microcontrollers

* Include L443 in compatible microcontrollers

* Update config bootloader jump description

* Update ChibiOS define reasoning

* Update quantum/mcu_selection.mk

* fix git conflict

* Updated Function96 with V2 files and removed chconf.h and halconf.h (#12613)

* Fix bad PR merge for #6580. (#12721)

* Change RGB/LED Matrix to use a simple define for USB suspend (#12697)

* [CI] Format code according to conventions (#12731)

* Fixing transport's led/rgb matrix suspend state logic (#12770)

* [CI] Format code according to conventions (#12772)

* Fix comment parsing (#12750)

* Added OLED fade out support (#12086)

* fix some references to bin/qmk that slipped in (#12832)

* Resolve a number of warnings in `qmk generate-api` (#12833)

* New command: qmk console (#12828)

* stash poc

* stash

* tidy up implementation

* Tidy up slightly for review

* Tidy up slightly for review

* Bodge environment to make tests pass

* Refactor away from asyncio due to windows issues

* Filter devices

* align vid/pid printing

* Add hidapi to the installers

* start preparing for multiple hid_listeners

* udev rules for hid_listen

* refactor to move closer to end state

* very basic implementation of the threaded model

* refactor how vid/pid/index are supplied and parsed

* windows improvements

* read the report directly when usage page isn't available

* add per-device colors, the choice to show names or numbers, and refactor

* add timestamps

* Add support for showing bootloaders

* tweak the color for bootloaders

* Align bootloader disconnect with connect color

* add support for showing all bootloaders

* fix the pyusb check

* tweaks

* fix exception

* hide a stack trace behind -v

* add --no-bootloaders option

* add documentation for qmk console

* Apply suggestions from code review

* pyformat

* clean up and flesh out KNOWN_BOOTLOADERS

* Remove pointless SERIAL_LINK_ENABLE rules (#12846)

* Make Swap Hands use PROGMEM (#12284)

This converts the array that the Swap Hands feature uses to use PROGMEM,
and to read from that array, as such. Since this array never changes at
runtime, there is no reason to keep it in memory. Especially for AVR
boards, as memory is a precious resource.

* Fix another bin/qmk reference (#12856)

* [Keymap] Turn OLED off on suspend in soundmonster keymap (#10419)

* Fixup build errors on `develop` branch. (#12723)

* LED Matrix: Effects! (#12651)

* Fix syntax error when compiling for ARM (#12866)

* Remove KEYMAP and LAYOUT_kc (#12160)

* alias KEYMAP to LAYOUT

* remove KEYMAP and LAYOUT_kc

* Add setup, clone, and env to the list of commands we allow even with broken modules (#12868)

* Rename `point_t` -> `led_point_t` (#12864)

* [Keyboard] updated a vendor name / fixed minor keymap issues (#12881)

* Add missing LED Matrix suspend code to suspend.c (#12878)

* LED Matrix: Documentation (#12685)

* Deprecate `send_unicode_hex_string()` (#12602)

* Fix spelling mistake regarding LED Matrix in split_common. (#12888)

* [Keymap] Fix QWERTY/DVORAK status output for kzar keymap (#12895)

* Use milc.subcommand.config instead of qmk.cli.config (#12915)

* Use milc.subcommand.config instead

* pyformat

* remove the config test

* Add function to allow repeated blinking of one layer (#12237)

* Implement function rgblight_blink_layer_repeat to allow repeated blinking of one layer at a time

* Update doc

* Rework rgblight blinking according to requested change

* optimize storage

* Fixup housekeeping from being invoked twice per loop. (#12933)

* matrix: wait for row signal to go HIGH for every row (#12945)

I noticed this discrepancy (last row of the matrix treated differently than the
others) when optimizing the input latency of my keyboard controller, see also
https://michael.stapelberg.ch/posts/2021-05-08-keyboard-input-latency-qmk-kinesis/

Before this commit, when tuning the delays I noticed ghost key presses when
pressing the F2 key, which is on the last row of the keyboard matrix: the
dead_grave key, which is on the first row of the keyboard matrix, would be
incorrectly detected as pressed.

After this commit, all keyboard matrix rows are interpreted correctly.

I suspect that my setup is more susceptible to this nuance than others because I
use GPIO_INPUT_PIN_DELAY=0 and hence don’t have another delay that might mask
the problem.

* ensure we do not conflict with existing keymap aliases (#12976)

* Add support for up to 4 IS31FL3733 drivers (#12342)

* Convert Encoder callbacks to be boolean functions (#12805)

* [Keyboard] Fix Terrazzo build failure (#12977)

* Do not hard set config in CPTC files (#11864)

* [Keyboard] Corne - Remove legacy revision support (#12226)

* [Keymap] Update to Drashna keymap and user code (based on develop) (#12936)

* Add Full-duplex serial driver for ARM boards (#9842)

* Document LED_MATRIX_FRAMEBUFFER_EFFECTS (#12987)

* Backlight: add defines for default level and breathing state (#12560)

* Add dire message about LUFA mass storage bootloader (#13014)

* [Keyboard] Remove redundant legacy and common headers for crkbd (#13023)

Was causing compiler errors on some systems.

* Fix keyboards/keymaps for boolean encoder callback changes (#12985)

* `backlight.c`: include `eeprom.h` (#13024)

* Add changelog for 2021-05-29 Breaking Changes merge (#12939)

* Add ChangeLog for 2021-05-29 Breaking Changes Merge: initial version

* Add recent develop changes

* Sort recent develop changes

* Remove sections for ChibiOS changes per tzarc

No ChibiOS changes this round.

* Add and sort recent develop changes

* add notes about keyboard moves/deletions

* import changelog for PR 12172

Documents the change to BOOTMAGIC_ENABLE.

* update section headings

* re-sort changelog

* add additional note regarding Bootmagic changes

* remove changelog timestamp

* update dates in main Breaking Changes docs

* fix broken section anchors in previous changelogs

* add link to backlight/eeprom patch to changelog

* highlight some more changes

* link PRs from section headers

* Restore standard readme

* run: qmk cformat --core-only

1 files changed, 319 insertions, 0 deletions

diff --git a/util/uf2conv.py b/util/uf2conv.py
new file mode 100755
index 000000000..044a7f231
--- /dev/null
+++ b/util/uf2conv.py
@@ -0,0 +1,319 @@
+#!/usr/bin/env python3
+import sys
+import struct
+import subprocess
+import re
+import os
+import os.path
+import argparse
+
+
+UF2_MAGIC_START0 = 0x0A324655 # "UF2\n"
+UF2_MAGIC_START1 = 0x9E5D5157 # Randomly selected
+UF2_MAGIC_END    = 0x0AB16F30 # Ditto
+
+families = {
+    'SAMD21': 0x68ed2b88,
+    'SAML21': 0x1851780a,
+    'SAMD51': 0x55114460,
+    'NRF52': 0x1b57745f,
+    'STM32F0': 0x647824b6,
+    'STM32F1': 0x5ee21072,
+    'STM32F2': 0x5d1a0a2e,
+    'STM32F3': 0x6b846188,
+    'STM32F4': 0x57755a57,
+    'STM32F7': 0x53b80f00,
+    'STM32G0': 0x300f5633,
+    'STM32G4': 0x4c71240a,
+    'STM32H7': 0x6db66082,
+    'STM32L0': 0x202e3a91,
+    'STM32L1': 0x1e1f432d,
+    'STM32L4': 0x00ff6919,
+    'STM32L5': 0x04240bdf,
+    'STM32WB': 0x70d16653,
+    'STM32WL': 0x21460ff0,
+    'ATMEGA32': 0x16573617,
+    'MIMXRT10XX': 0x4FB2D5BD,
+    'LPC55': 0x2abc77ec,
+    'GD32F350': 0x31D228C6,
+    'ESP32S2': 0xbfdd4eee,
+    'RP2040': 0xe48bff56
+}
+
+INFO_FILE = "/INFO_UF2.TXT"
+
+appstartaddr = 0x2000
+familyid = 0x0
+
+
+def is_uf2(buf):
+    w = struct.unpack("<II", buf[0:8])
+    return w[0] == UF2_MAGIC_START0 and w[1] == UF2_MAGIC_START1
+
+def is_hex(buf):
+    try:
+        w = buf[0:30].decode("utf-8")
+    except UnicodeDecodeError:
+        return False
+    if w[0] == ':' and re.match(b"^[:0-9a-fA-F\r\n]+$", buf):
+        return True
+    return False
+
+def convert_from_uf2(buf):
+    global appstartaddr
+    numblocks = len(buf) // 512
+    curraddr = None
+    outp = []
+    for blockno in range(numblocks):
+        ptr = blockno * 512
+        block = buf[ptr:ptr + 512]
+        hd = struct.unpack(b"<IIIIIIII", block[0:32])
+        if hd[0] != UF2_MAGIC_START0 or hd[1] != UF2_MAGIC_START1:
+            print("Skipping block at " + ptr + "; bad magic")
+            continue
+        if hd[2] & 1:
+            # NO-flash flag set; skip block
+            continue
+        datalen = hd[4]
+        if datalen > 476:
+            assert False, "Invalid UF2 data size at " + ptr
+        newaddr = hd[3]
+        if curraddr == None:
+            appstartaddr = newaddr
+            curraddr = newaddr
+        padding = newaddr - curraddr
+        if padding < 0:
+            assert False, "Block out of order at " + ptr
+        if padding > 10*1024*1024:
+            assert False, "More than 10M of padding needed at " + ptr
+        if padding % 4 != 0:
+            assert False, "Non-word padding size at " + ptr
+        while padding > 0:
+            padding -= 4
+            outp += b"\x00\x00\x00\x00"
+        outp.append(block[32 : 32 + datalen])
+        curraddr = newaddr + datalen
+    return b"".join(outp)
+
+def convert_to_carray(file_content):
+    outp = "const unsigned long bindata_len = %d;\n" % len(file_content)
+    outp += "const unsigned char bindata[] __attribute__((aligned(16))) = {"
+    for i in range(len(file_content)):
+        if i % 16 == 0:
+            outp += "\n"
+        outp += "0x%02x, " % file_content[i]
+    outp += "\n};\n"
+    return bytes(outp, "utf-8")
+
+def convert_to_uf2(file_content):
+    global familyid
+    datapadding = b""
+    while len(datapadding) < 512 - 256 - 32 - 4:
+        datapadding += b"\x00\x00\x00\x00"
+    numblocks = (len(file_content) + 255) // 256
+    outp = []
+    for blockno in range(numblocks):
+        ptr = 256 * blockno
+        chunk = file_content[ptr:ptr + 256]
+        flags = 0x0
+        if familyid:
+            flags |= 0x2000
+        hd = struct.pack(b"<IIIIIIII",
+            UF2_MAGIC_START0, UF2_MAGIC_START1,
+            flags, ptr + appstartaddr, 256, blockno, numblocks, familyid)
+        while len(chunk) < 256:
+            chunk += b"\x00"
+        block = hd + chunk + datapadding + struct.pack(b"<I", UF2_MAGIC_END)
+        assert len(block) == 512
+        outp.append(block)
+    return b"".join(outp)
+
+class Block:
+    def __init__(self, addr):
+        self.addr = addr
+        self.bytes = bytearray(256)
+
+    def encode(self, blockno, numblocks):
+        global familyid
+        flags = 0x0
+        if familyid:
+            flags |= 0x2000
+        hd = struct.pack("<IIIIIIII",
+            UF2_MAGIC_START0, UF2_MAGIC_START1,
+            flags, self.addr, 256, blockno, numblocks, familyid)
+        hd += self.bytes[0:256]
+        while len(hd) < 512 - 4:
+            hd += b"\x00"
+        hd += struct.pack("<I", UF2_MAGIC_END)
+        return hd
+
+def convert_from_hex_to_uf2(buf):
+    global appstartaddr
+    appstartaddr = None
+    upper = 0
+    currblock = None
+    blocks = []
+    for line in buf.split('\n'):
+        if line[0] != ":":
+            continue
+        i = 1
+        rec = []
+        while i < len(line) - 1:
+            rec.append(int(line[i:i+2], 16))
+            i += 2
+        tp = rec[3]
+        if tp == 4:
+            upper = ((rec[4] << 8) | rec[5]) << 16
+        elif tp == 2:
+            upper = ((rec[4] << 8) | rec[5]) << 4
+            assert (upper & 0xffff) == 0
+        elif tp == 1:
+            break
+        elif tp == 0:
+            addr = upper | (rec[1] << 8) | rec[2]
+            if appstartaddr == None:
+                appstartaddr = addr
+            i = 4
+            while i < len(rec) - 1:
+                if not currblock or currblock.addr & ~0xff != addr & ~0xff:
+                    currblock = Block(addr & ~0xff)
+                    blocks.append(currblock)
+                currblock.bytes[addr & 0xff] = rec[i]
+                addr += 1
+                i += 1
+    numblocks = len(blocks)
+    resfile = b""
+    for i in range(0, numblocks):
+        resfile += blocks[i].encode(i, numblocks)
+    return resfile
+
+def to_str(b):
+    return b.decode("utf-8")
+
+def get_drives():
+    drives = []
+    if sys.platform == "win32":
+        r = subprocess.check_output(["wmic", "PATH", "Win32_LogicalDisk",
+                                     "get", "DeviceID,", "VolumeName,",
+                                     "FileSystem,", "DriveType"])
+        for line in to_str(r).split('\n'):
+            words = re.split('\s+', line)
+            if len(words) >= 3 and words[1] == "2" and words[2] == "FAT":
+                drives.append(words[0])
+    else:
+        rootpath = "/media"
+        if sys.platform == "darwin":
+            rootpath = "/Volumes"
+        elif sys.platform == "linux":
+            tmp = rootpath + "/" + os.environ["USER"]
+            if os.path.isdir(tmp):
+                rootpath = tmp
+        for d in os.listdir(rootpath):
+            drives.append(os.path.join(rootpath, d))
+
+
+    def has_info(d):
+        try:
+            return os.path.isfile(d + INFO_FILE)
+        except:
+            return False
+
+    return list(filter(has_info, drives))
+
+
+def board_id(path):
+    with open(path + INFO_FILE, mode='r') as file:
+        file_content = file.read()
+    return re.search("Board-ID: ([^\r\n]*)", file_content).group(1)
+
+
+def list_drives():
+    for d in get_drives():
+        print(d, board_id(d))
+
+
+def write_file(name, buf):
+    with open(name, "wb") as f:
+        f.write(buf)
+    print("Wrote %d bytes to %s" % (len(buf), name))
+
+
+def main():
+    global appstartaddr, familyid
+    def error(msg):
+        print(msg)
+        sys.exit(1)
+    parser = argparse.ArgumentParser(description='Convert to UF2 or flash directly.')
+    parser.add_argument('input', metavar='INPUT', type=str, nargs='?',
+                        help='input file (HEX, BIN or UF2)')
+    parser.add_argument('-b' , '--base', dest='base', type=str,
+                        default="0x2000",
+                        help='set base address of application for BIN format (default: 0x2000)')
+    parser.add_argument('-o' , '--output', metavar="FILE", dest='output', type=str,
+                        help='write output to named file; defaults to "flash.uf2" or "flash.bin" where sensible')
+    parser.add_argument('-d' , '--device', dest="device_path",
+                        help='select a device path to flash')
+    parser.add_argument('-l' , '--list', action='store_true',
+                        help='list connected devices')
+    parser.add_argument('-c' , '--convert', action='store_true',
+                        help='do not flash, just convert')
+    parser.add_argument('-D' , '--deploy', action='store_true',
+                        help='just flash, do not convert')
+    parser.add_argument('-f' , '--family', dest='family', type=str,
+                        default="0x0",
+                        help='specify familyID - number or name (default: 0x0)')
+    parser.add_argument('-C' , '--carray', action='store_true',
+                        help='convert binary file to a C array, not UF2')
+    args = parser.parse_args()
+    appstartaddr = int(args.base, 0)
+
+    if args.family.upper() in families:
+        familyid = families[args.family.upper()]
+    else:
+        try:
+            familyid = int(args.family, 0)
+        except ValueError:
+            error("Family ID needs to be a number or one of: " + ", ".join(families.keys()))
+
+    if args.list:
+        list_drives()
+    else:
+        if not args.input:
+            error("Need input file")
+        with open(args.input, mode='rb') as f:
+            inpbuf = f.read()
+        from_uf2 = is_uf2(inpbuf)
+        ext = "uf2"
+        if args.deploy:
+            outbuf = inpbuf
+        elif from_uf2:
+            outbuf = convert_from_uf2(inpbuf)
+            ext = "bin"
+        elif is_hex(inpbuf):
+            outbuf = convert_from_hex_to_uf2(inpbuf.decode("utf-8"))
+        elif args.carray:
+            outbuf = convert_to_carray(inpbuf)
+            ext = "h"
+        else:
+            outbuf = convert_to_uf2(inpbuf)
+        print("Converting to %s, output size: %d, start address: 0x%x" %
+              (ext, len(outbuf), appstartaddr))
+        if args.convert or ext != "uf2":
+            drives = []
+            if args.output == None:
+                args.output = "flash." + ext
+        else:
+            drives = get_drives()
+
+        if args.output:
+            write_file(args.output, outbuf)
+        else:
+            if len(drives) == 0:
+                error("No drive to deploy.")
+        for d in drives:
+            print("Flashing %s (%s)" % (d, board_id(d)))
+            write_file(d + "/NEW.UF2", outbuf)
+
+
+if __name__ == "__main__":
+    main()