LVM metadata inconsistency trouble on boot

This is what “Murphy’s Law” is all about: You change a minor issue and spend the day fixing big trouble.

In this specific case, I had to reboot one of our more important servers because of problems with persistent network names. But after the reboot, the machine didn’t come up as expected (or should I rather say “hoped for”?), but instead went into Dracut rescue mode, AKA “initrd shell”.

We quickly noticed that only some of the LVM volume groups were activated and noticed that the volume group carrying the system LVs was unavailable. “lvm pvscan” not only gave a too short list of PVs (the two disks for system VG missing), but also spat out a number of error messages: That it wouldn’t activate our system volume group, that it could not work with a standalone PV, and “read-only locking type set. write locks are prohibited”. Now what’s that about? Continue reading

Posted in OpenSUSE, SLES (SUSE Linux Enterprise Server) | Leave a comment

bad device, bad memory – how to fix a case of “bad expectations”

This is the second case in twelve months, so I decided to dump some self-experience in this article. Hopefully it will help both me and others to circumvent this type of errors, or at least to not waste too much time on them. The actual error is on my behalf, not the technical components involved. This time it’s a case of “bad memory”, user-wise, as I ran across a similar wrong approach only some months ago.

Every once in a while, I’m running into a problem, see symptoms, and assume a certain cause much too quickly. And only notice later that everything is actually working as expected or I was barking up the wrong tree. A case of “bad expectations”, and that’s something that I really need to fight.

So what went wrong in the first place? I had replaced an SATA disk of a laptop, this time not because of a failing original disk, but simply because I ran out of space. In order to be able to migrate data from the original device to the new one, I purchased an external, USB3-connected M.2 SATA enclosure, as this was my first case of swapping such a device. And when doing the actual data migration, I ran into strange, persisting error indicator symptoms on the new device (after already having moved the better of 128 GB) while it was connected via that new enclosure. But once I swapped the setup (moving the original M.2 device to the enclosure and mounting the new M.2 SSD to the laptop’s mainboard directly), the new SSD reset it’s error LED and both transfer and later operations went like a charm.

But some weeks later I needed to access some of the old data on the original device, so I plugged in the external enclosure and was again facing problems. Any access to the device resulted in ugly kernel messages:

kernel: [ 3488.149220] usb 2-3: new SuperSpeed USB device number 8 using xhci_hcd
kernel: [ 3488.169802] usb 2-3: New USB device found, idVendor=174c, idProduct=55aa
kernel: [ 3488.169807] usb 2-3: New USB device strings: Mfr=2, Product=3, SerialNumber=1
kernel: [ 3488.169810] usb 2-3: Product: ASM105x 
kernel: [ 3488.169813] usb 2-3: Manufacturer: ASMT
kernel: [ 3488.169816] usb 2-3: SerialNumber: 00000000000000000000
kernel: [ 3488.171858] scsi host2: uas 
kernel: [ 3488.172374] scsi 2:0:0:0: Direct-Access     ASMT     2115             0    PQ: 0 ANSI: 6    
kernel: [ 3488.173257] sd 2:0:0:0: Attached scsi generic sg1 type 0
kernel: [ 3488.173834] sd 2:0:0:0: [sdb] 250069680 512-byte logical blocks: (128 GB/119 GiB)
kernel: [ 3488.173840] sd 2:0:0:0: [sdb] 4096-byte physical blocks
kernel: [ 3488.173947] sd 2:0:0:0: [sdb] Write Protect is off 
kernel: [ 3488.173951] sd 2:0:0:0: [sdb] Mode Sense: 43 00 00 00 
kernel: [ 3488.174116] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
mtp-probe: checking bus 2, device 8: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3"
mtp-probe: bus: 2, device: 8 was not an MTP device
kernel: [ 3488.181505] scsi_io_completion: 3 callbacks suppressed
kernel: [ 3488.181512] sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
kernel: [ 3488.181516] sd 2:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current]
kernel: [ 3488.181521] sd 2:0:0:0: [sdb] tag#0 Add. Sense: Invalid field in cdb 
kernel: [ 3488.181525] sd 2:0:0:0: [sdb] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00 
kernel: [ 3488.181527] print_req_error: 3 callbacks suppressed
kernel: [ 3488.181529] print_req_error: critical target error, dev sdb, sector 0
kernel: [ 3488.181532] buffer_io_error: 2 callbacks suppressed
kernel: [ 3488.181533] Buffer I/O error on dev sdb, logical block 0, async page read

Access to the original M.2 SSD seemed impossible.I spend quite some time on problems with USB3-attached devices and tried to locate experiences with and advice on problems with the chipsets involved. Nothing helped, all paths I chose seemed to be dead ends. So the enclosure was bad, after all. Right? Wrong. Continue reading

Posted in Linux | Leave a comment

PXE-booting UEFI systems via SUSE Manager’s Cobbler

Booting BIOS-based PXE clients via SUSE Manager and its integrated “Cobbler” component is very easy and well-documented. But as more and more hardware vendors decide to switch to UEFI, how about PXE-booting these? Continue reading

Posted in Linux, SUSE Manager | Tagged , , | Leave a comment