summaryrefslogtreecommitdiff
path: root/ironic_python_agent/extensions
Commit message (Collapse)AuthorAgeFilesLines
* Use utf-16-le if BOM not presentRiccardo Pittau2022-10-131-2/+8
| | | | | | | | | | | | | | In case no BOM is present in the CSV file the utf-16 codec won't work. We fail over to utf-16-le as Little Endian is commonly used. NOTE: The original change landed this fix in efi_utils.py, however that was introduced after the Xena development cycle, so this backport moves the original change to where the code originally came from to populate the efi_utils.py file. Change-Id: I3e25ce4997f5dd3df87caba753daced65838f85a (cherry picked from commit 697fa6f3b6db10408eaadd57450456de87f13519) (cherry picked from commit 99b9d1403cacd3bbd489a6cf6913f32746ef6083)
* Do not try to guess EFI partition path by its numberDmitry Tantsur2022-05-061-18/+52
| | | | | | | | | | | | | The logic of adding a partition number to the device path does not work for devicemapper devices (e.g. a multipath storage device). Conflicts: ironic_python_agent/efi_utils.py ironic_python_agent/tests/unit/extensions/test_image.py ironic_python_agent/tests/unit/test_efi_utils.py Change-Id: I9a445e847d282c50adfa4bad5e7136776861005d (cherry picked from commit f09f6c9f1a09c7062d0450b3e0a4d3164fd53f7f)
* SoftwareRAID: Use efibootmgr (and drop grub2-install)Arne Wiebalck2022-03-161-17/+71
| | | | | | | | | | | | | | | | | | | | | | | Move the software RAID code path from grub2-install to efibootmgr: - remove the UEFI efibootmgr exception for software RAID - create and populate the ESPs on the holder disks - update the NVRAM with all ESPs (the component devices of the ESP mirror, use unique labels to avoid unintentional deduplication of entries in the NVRAM) In addition, use canonical device names for the RAID device for the ESP: it seems like tinyIPA silently replaces /dev/md/esp with /dev/md127. Find the next free /dev/md device and use it instead. Also rescan the resulting device before copying files. Co-authored-by: Dmitry Tantsur <dtantsur@protonmail.com> Story: #2009794 Change-Id: I7ed34e595215194a589c2f1cd0b39ff0336da8f1 (cherry picked from commit 62c5674a600baeeef0af3b12baeab486870eb103)
* Create fstab entry with appropriate labelJulia Kreger2022-03-111-5/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Depending on the how the stars align with partition images being written to a remote system, we *may* end up with *either* a Partition UUID value, or a Partition's UUID value. Which are distinctly different. This is becasue the value, when collected as a result of writing an image to disk *falls* back and passes the value to enable partition discovery and matching. Later on, when we realized we ought to create an fstab entry, we blindly re-used the value thinking it was, indeed, always a Partition's UUID and not the Partition UUID. Obviously, the label type is quite explicit, either UUID or PARTUUID respectively, when initial ramdisk utilities such as dracut are searching and mounting filesystems. Adds capability to identify the correct label to utilize based upon the current state of the block devices on disk. Granted, we are likely only exposed to this because of IO race conditions under high concurrecy load operations. Normally this would only be seen on test VMs, but systems being backed by a Storage Area Network *can* exibit the same IO race conditions as virtual machines. Change-Id: I953c936cbf8fad889108cbf4e50b1a15f511b38c Resolves: rhbz#2058717 Story: #2009881 Task: 44623 (cherry picked from commit 99ca1086dbfc7b6e41cf800b0bd899565e2e8922)
* Re-read the partition table with partx -aArne Wiebalck2021-11-091-1/+1
| | | | | | | | | | | | Re-read the partition table with 'partx -a', rather than 'partx -u'. This should fix an timing issue where the bootloader installation fails to mount the EFI partition from a whole disk image since it is not yet aware of the new partitions (observed with both, the iscsi and the direct deploy interface). Change-Id: If5da3075e813ae01df3decf8f0647aba111b0515 (cherry picked from commit dc8c1f16f9a00e2bff21612d1a9cf0ea0f3addf0)
* Fix UEFI record regexJulia Kreger2021-11-081-1/+1
| | | | | | | | | | | | | I accidently put colons on the test data and remembered taking the colon character out of the regex I was working on, but apparently left it in, and accounted for the active entry indicator flag which appears to have inconsistent support across vendors. The regex has been fixed, and a test added from a Lenovo SR650 which has some additional string entry data in the UEFI output which may separate entries. Change-Id: I1f67b0fb1f645fa82e98bd7c7bba3ffc7755cc74
* Delete EFI boot entry duplicate labels firstJulia Kreger2021-11-041-15/+19
| | | | | | | | | | | | | | | | | | | | | | | | Some firmware seems to take an objection with EFI nvram entries being deleted after one is added, resulting in the entire entry table being reset to the last known good state. This is problematic, as ultimately deployments can time out if we previously booted with Networking, and the machine, while commanded to do other wise, reboots back to networking regardless. We will now delete entries first, before proceeding. Additionally, for general use, this pattern may serve the community better by avoiding cases where we would have previously just relied upon efibootmgr[0] to warn us of duplicate entries. [0]: https://github.com/rhboot/efibootmgr/blob/103aa22ece98f09fe3ea2a0c83988f0ee2d0e5a8/src/efibootmgr.c#L228 Change-Id: Ib61a7100a059e79a8b0901fd8f46b9bc41d657dc Story: 2009649 Task: 43808 (cherry picked from commit 67eddfa7e3fedbb530045f5b43a2c89db832fa2a)
* Assert EFI part UUID is not None before editing fstabArne Wiebalck2021-10-211-1/+1
| | | | | | | | | | | The EFI partition UUID may be None and this will break the fstab editing. While this is not necessarily fatal when instantiating a node, it creates an exception at the end of bootloader installation, so only attempt to add a line to fstab when the UUID is not None. Change-Id: I68799980e67c05afe4ca68ca9733605dd166d54d (cherry picked from commit 333ed70c94e366f16d8f2633f74a5ef05aa5fadb)
* Trivial: minor fixes in error messagesDmitry Tantsur2021-09-073-3/+3
| | | | Change-Id: I06b32c2eb576520cddff88074e4619070731017d
* Merge "Output verbose info from efibootmgr"Zuul2021-08-111-2/+2
|\
| * Output verbose info from efibootmgrDerek Higgins2021-08-031-2/+2
| | | | | | | | | | | | | | When debugging boot manager problems it can be advantageous to see all the full entries rather then just their labels. Change-Id: I6a1bb78acaf5a4284727bdf533d4be6db2099f50
* | Increase version of hacking and pycodestyleRiccardo Pittau2021-07-302-15/+24
|/ | | | | | Fix H904 "Delay string interpolations at logging calls" errors Change-Id: I331808d0132094faf739998a6984440787d3ebf8
* Catch ismount not being handledJulia Kreger2021-06-291-9/+32
| | | | | | | | | | | | | | | While investigating another grub issue, I was confused by the path taken in the logs reported, and noticed that on a ramdisk, we might not actually have a valid response to os.path.ismount, I'm guessing depending on what in memory filesystem is in use while also coupled with attempting to check a filesystem. Adds a test to validate that exceptions raised on these commands where this issue can be encountered, are properly bypassed, and also adds additional logging to make it easier to figure out what is going on in the entire bootloader setup sequence. Change-Id: Ibd3060bef2e56468ada6b1a5c1cc1632a42803c3
* Only mount the ESP if not yet mountedArne Wiebalck2021-06-211-1/+3
| | | | | | | Check if the ESP is already mounted before attempting to mount it for the bootloader installation. Change-Id: Ifd738b2c5663f1a211d7e13b5ba386be631d8db1
* Utilize CSV file for EFI loader selectionJulia Kreger2021-06-101-10/+43
| | | | | | | | | | | | | | | | | | | | | | Adds support to identify and utilize a CSV file to signal which bootloader to utilize, and set it when the OS is running as opposed to when EFI is running. This works around EFI loader potentially crashing some vendors hardware types when entry stored in the image does not match the EFI loader record which was utilzied to boot. Grub2+shim specifically specifically needs the CSV file name and entry label to match what the system was booted with in order to prevent the machine from potentially crashing. See https://storyboard.openstack.org/#!/story/2008962 and https://bugzilla.redhat.com/show_bug.cgi?id=1966129#c37 for more information. Change-Id: Ibf1ef4fe0764c0a6f1a39cb7eebc23ecc0ee177d Story: 2008962 Task: 42598 Co-Authored-By: Bob Fournier <bfournie@redhat.com>
* Merge "Ignore efi grub2-install failure"Zuul2021-06-071-28/+32
|\
| * Ignore efi grub2-install failureSteve Baker2021-06-041-28/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Recent releases of redhat grub2 will always fail when installing to EFI paths, to encourage a transition to the signed shim bootloader. Partition image deploys avoid calling grub2-install with the preserve-efi-assets functions. Deploying whole disk images doesn't require grub2-install. This leaves whole disk images installed onto softraid devices, which still attempts to call grub2-install. This change will still attempt to run grub2-install in this one remaining case, but will ignore any failure. A future enhancement can avoid calling grub2-install entirely so that non-redhat secure-boot capable images can keep their signed bootloaders. Story: 2008923 Task: 42521 Change-Id: If432ef795d64d76442d739eb4f7d155ff847041e
* | Merge "Refactor: use convert_image from ironic_lib"Zuul2021-06-041-15/+2
|\ \
| * | Refactor: use convert_image from ironic_libDmitry Tantsur2021-06-011-15/+2
| |/ | | | | | | Change-Id: If890baf3545cff6cef7c645c42e7f9d9038c9aa7
* | Merge "Stop accepting duplicated configdrive"Zuul2021-06-021-15/+33
|\ \ | |/ |/|
| * Stop accepting duplicated configdriveDmitry Tantsur2021-06-021-15/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We're currently requiring it twice: in image_info and in a separate configdrive argument. I think we should eventually settle on separate arguments for separate entities, so this change makes the value in image_info optional with a goal to stop accepting it. We could probably just remove the handling in image_info, but a deprecation is safer. The (unused in ironic) cache_image call is updated with an optional configdrive arguments. Story: #2008904 Task: #42480 Change-Id: I1e2efa28efa3ea7e389774cb7633d916757bc6ed
* | Merge "Make _get_efi_bootloaders return relative paths"Zuul2021-05-271-6/+16
|\ \
| * | Make _get_efi_bootloaders return relative pathsSteve Baker2021-05-261-6/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make this function useful for purposes other than efibootmgr entries, this change moves the path manipulation to _run_efibootmgr. This change also adds boot*.efi entries to BOOTLOADERS_EFI so that it includes every entry in the UEFI Spec 2.9[1] Table 3-2 UEFI Image Types. [1] https://uefi.org/sites/default/files/resources/UEFI_Spec_2_9_2021_03_18.pdf Story: 2008923 Task: 42521 Change-Id: Ibe02786609aa0de65115897d8f4a9b4f36c8aed2
* | | Limit qemu-img execution arenasJulia Kreger2021-05-261-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | qemu-img attempts to launch multiple threads by default *and* attempts to have multiple memory allocation arenas to operate from. While multithreading can be good for performance, this pattern and the memory footprint for process launch and dependencies can turn the memory footprint for a cirros image conversion (16MB) into 1.2GB of memory being asked for by the qemu-img tool. In order to limit this impact, as the default number of arenas is governed by the number of CPUs times the number 8, it seems reasonable to lower this to a more reasonable number which also helps keep our possible memory footprint from being exceeded. Change-Id: I71a28ec59ec31c691205eb34d9fcab63a2ccb682 Story: 2008928 Task: 42528
* | | Merge "Rewrite write_image.sh in Python"Zuul2021-05-261-13/+14
|\ \ \ | |/ / |/| |
| * | Rewrite write_image.sh in PythonDmitry Tantsur2021-05-181-13/+14
| | | | | | | | | | | | Change-Id: I0caa65561948f4e0934943a7a0d3a209701b5a59
* | | Merge "Do not serialize command_params"Zuul2021-05-181-1/+1
|\ \ \ | |/ / |/| |
| * | Do not serialize command_paramsDmitry Tantsur2021-05-181-1/+1
| |/ | | | | | | | | | | | | | | | | The command params can be huge when configdrive is used. There is no point in sending them back, Ironic does not use them anyhow. Story: #2008904 Task: #42479 Change-Id: I6e3db5db2042ca3fb5dafacfacf036fd7fc2fc4c
* | Merge "Remove the iscsi extension"Zuul2021-05-122-226/+1
|\ \
| * | Remove the iscsi extensionDmitry Tantsur2021-05-102-226/+1
| |/ | | | | | | Change-Id: I2f0e581575112d6c7ba0d211661cab3e0b6caca6
* | Merge "Software RAID: RAID the ESPs"Zuul2021-05-111-90/+87
|\ \
| * | Software RAID: RAID the ESPsArne Wiebalck2021-04-161-90/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For software RAID in UEFI mode, we create ESPs on all holder disks and copy the bootloader there. Since there is no mechanism to keep the ESPs in sync, e.g. on kernel upgrades or when kernel parameters are updated, the ESPs will get out of sync eventually. This may lead to a situation where a node boots with outdated parameters or does not have any of the installed kernels in the boot menu anymore. This change proposes to RAID the ESPs. While the UEFI firmware will find an ESP partition (one leg of the mirror), the node will see an md device and all subsequent updates will go to all member disks. Also, remove the source ESP after copying in order to avoid mount confusion (same UUID!). Story: #2008745 Task: #42103 Change-Id: I9078ef37f1e94382c645ae98ce724ac9ed87c287
* | | Merge "Fix NVMe Partition image on UEFI"Zuul2021-05-101-1/+6
|\ \ \ | |_|/ |/| |
| * | Fix NVMe Partition image on UEFIJulia Kreger2021-05-041-1/+6
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The _manage_uefi code has a check where it attempts to just identify the precise partition number of the device, in order for configuration to be parsed and passed. However, the same code did not handle the existence of a `p1` partition instead of just a partition #1. This is because the device naming format is different with NVMe and Software RAID. Likely, this wasn't an issue with software raid due to how complex the code interaction is, but the docs also indicate to use only whole disk images in that case. This patch was pulled down my one RH's professional services folks who has confirmed it does indeed fix the issue at hand. This is noted as a public comment on the Red Hat bugzilla. https://bugzilla.redhat.com/show_bug.cgi?id=1954096 Story: 2008881 Task: 42426 Related: rhbz#1954096 Change-Id: Ie3bd49add9a57fabbcdcbae4b73309066b620d02
* | Import deployment logic from ironic-libDmitry Tantsur2021-05-031-14/+15
|/ | | | | | | | | | | | The two functions work_on_disk and create_config_drive_partition contain a substantial part of the deployment logic. Previously we placed them in ironic-lib for re-using on the conductor side in the iSCSI deploy interface. Since the iSCSI deploy is going away, we can move this code to ironic-python-agent to simplify maintenance. Imports code from ironic_lib commit 9fb5be348202f4854a455cd08f400ae12b99e1f2. Change-Id: I6cbcd81533f135208b57746cb0e33ffdfaf94eee
* Always fall back to sysrq when power off failsDmitry Tantsur2021-04-131-7/+16
| | | | | | | The line we're looking for is not there when IPA is in a container, at least for CentOS based containers. Just fall back to sysrq on errors. Change-Id: Ie4ee605ad9c6cda58808512a563247175859c71e
* Fix root UUID for streamed partition imagesSteve Baker2021-03-191-0/+4
| | | | | | | | | | | | | | The root UUID changes after a streamed partition image is written to the block device, causing later deployment failure when assuming the old UUID. This change updates the root UUID after streaming the partition image is complete. This issue may have been missed in local testing because deploying the same image repeatedly will result in stable root UUID across runs. Change-Id: Ice4630c16fc216980488d1427f3b02e1b8a417fa
* Remove default parameter from executeRiccardo Pittau2021-03-021-3/+2
| | | | | | | | | The param check_exit_code from the processutils extension execute has default already at [0] See: https://opendev.org/openstack/oslo.concurrency/src/branch/master/oslo_concurrency/processutils.py#L214 Change-Id: Iedff5325e0737556d5eb3da601c984ddfc633873
* Fixes local boot for partition imageskartikeyaj02021-02-171-1/+1
| | | | | | | | | | IPA is not properly checking if the root partition is already mounted. Device is being passed to os.path.ismount() instead of the mount point. Story: 2008631 Task: 41839 Change-Id: I37a6e7e6bbe0bbbb0317c6e55bb822dafe7cce20
* Fix error message with UEFI-incompatible imagesDmitry Tantsur2021-02-101-21/+29
| | | | | | | | It's somewhat confusing at the moment, since we're trying to find a UEFI partition by UUID "None". Don't search for partition if we don't know its UUID, and provide a better error message. Change-Id: Ief874084132797a445ddae8009264712a05facfd
* Fix UEFI boot entry creation for aarch64Xinliang Liu2021-01-271-1/+2
| | | | | | | | | | | | | | | | | | Diskimage-builder installs grub with option '--removable'[1], thus for aarch64 no 'grubaa64.efi' file in efi directory only got 'BOOTAA64.EFI': linaro@bm-ubuntu:~$ tree /boot/efi /boot/efi └── EFI └── BOOT └── BOOTAA64.EFI 2 directories, 1 file [1]: https://github.com/openstack/diskimage-builder/blob/8f12d9530ed79359fb988688caadfa6dc318f7a5/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L158 Task: #41698 Story: #2008560 Change-Id: I9fc55c068ea980beae273411db9d3568eec25eb8
* Fix boot mode detection for partition imagesJulia Kreger2020-12-191-1/+1
| | | | | | | | | | | | Previously, partition images were hard coded to be bios based as opposed to consulting all of the values AND the node itself before making the most appropriate determination. Now the agent utilises the internal helper to properly determine the boot mode when calling ironic-lib. Story: 2008070 Task: 41265 Change-Id: Id5eeda69d5b9de2b393af414472d57b0d4380c43
* Change default ironic_lib invocation to flag local bootingJulia Kreger2020-12-191-1/+1
| | | | | | | | | | The partition image support has been telling ironic-lib that the machine will be local booted. While this is likely harmless, and doesn't seem to break anythign, we should have it match moving forward just to be on the safe side so we don't accidently break things down the road. Change-Id: I33e5d583964ef8c21aa04d7427bcd3957b89d449
* Add fstab pointer to EFI partitionJulia Kreger2020-12-171-0/+25
| | | | | | | | | | | | | | Adds support for the EFI partition to be appended to fstab so the filesystem can be automounted and EFI loader updated should the deployed operating system need to do so. This should enable bootloaders to be upgraded by linux based operating systems after the instance has been deployed when a partition image was utilized for the initial deployment. Change-Id: Iec28a8841cc01ec8b01a3f5cca070c934c7a2531 Story: 2008070 Task: 40754
* Prevent broken partition image UEFI deploysJulia Kreger2020-12-141-63/+283
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Partition images can sometimes contain a /boot folder structure event he assets for EFI booting on that filesystem. Which is a good thing. The conundrum is that Ironic does not handle this properly and potentially replaces the bootloader in this sequence such that grub2-install is used instead of signed bootloader assets. As such, we should be preserving the assets and using them from a partition image much like we do when we have a wholedisk image and can identify the assets. Now we will preserve the EFI boot assets, copy them to the new EFI boot partition, and call the EFI setup methods to manage the EFI nvram. Note, this change also splits the logic path out that performs the end call of the EFI boot manager into a reusable method but does not retool all of the testing as it is intertwined in the install_grub2 testing. Also adds some additional debug logging, as much of the bootloader installation code has multiple fallback/cleanup points which makes it difficult to debug from logs. Story: 2008070 Task: 40753 Change-Id: If17d4b4c06df5504987e61a1fde6662e9acd6989
* Fix default disk label with partition imagesJulia Kreger2020-12-141-1/+6
| | | | | | | | | | | | Partition images through the agent have the unfortunate side effect of being executed without full node context by default. Luckilly we've had a similar problem and cache the node. This patch changes the lookup from a default of msdos partitions to use the cached node object. Change-Id: I002816c9372fdf1cc32f3c67f420073551479fd9
* Option to enable bootloader config failure bypassJulia Kreger2020-12-101-9/+30
| | | | | | | | | | | | | | | | | | | Some hardware is very well intentioned. However this intention can result in the UEFI NVRAM table being full which prevents us from adding new records to the table. We can't be sure what to delete, so in this case some operators just need the ability to tell ironic "it is okay if this fails, it will still work." The added ``ignore_bootloader_failure`` option adds this capability which can be set per-node either in the agent configuation via the ramdisk image, or in the pxe_append_params configuration parameter for the node itself with a ``ipa-ignore-bootloader-failure`` option in order to prevent the failure from being raised. Change-Id: If3c83fb2ea2025fce092d495a64f32077c70d2d6 Story: 2008386 Task: 41309
* Support using LABEL as identifier for rootfsFedor Tarasenko2020-11-031-1/+6
| | | | | | | | | Add possibility to use disk LABEL to identify rootfs uuid for Software RAID deployment Change-Id: I77f36e70ddc539af0190db1c1abe0fb2c66f34b4 Story: 2008303 Task: 41188
* Don't run os-prober from grub2-mkconfigJulia Kreger2020-10-221-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default, grub2-mkconfig scans everything to look for other environments and then load those into the grub configuration. It makes sense, but on newer versions of grub2 in distribution images, os-prober is taking an exceptionally long time in some cases where more than one storage device exists with other filesystems. As a result, of the os-prober execution by grub2-mkconfig, the bootloader installation can completely time out and fail the deployment. This is presently experienced with metalsmith on centos8. There are numerous sporatic reports of issues like this issue where grub2-mkconfig hangs for some period of time, and this is observable on Centos8.2 in our CI. While one report[0] mentions this issue, Another bug [1] has the dialog that actually helps us frame the context as to what we likely should do. Also, fixes the unit testing so we actually test if we're running with grub2. :\ [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1744693 [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1709682 Depends-On: https://review.opendev.org/#/c/748315 Change-Id: I14bf299afef3a1ddb2006fe5f182d7f0d249e734
* Do not silently swallow errors in the write_image deploy stepDmitry Tantsur2020-10-091-0/+11
| | | | | | | | Calling join() does not raise, we need to explicitly check the result. Change-Id: I81d3d727af220c2b50358edab8139f07874611f0 Story: #2008240 Task: #41083
View Lorry Controller Status