Forums

Resolved
3 votes
I'm pleased to be accepted in the CLearOS Community.
I'll like to report what I believe is a bug, attempting to install the kernel module kmod-hpdsa for a b140i Smart Array, default RAID solution in many HP Servers, (Mine is a ML110 Gen9), I found it was not possible to see the RAID in the GUI.
A little research shows the following (look at dmesg)
hpdsa: disagrees about version of symbol wake_up_process.
hpdsa: Unknown symbol wake_up_process

Apparently the kernel module hpdsa is looking for symbol 0xe65cdceb wake_up_process,
however the ClearOS kernel reports 0xfa87ae45 for the above wake_up_process, and there comes the mismatch.

How to reproduce:
Controller HP b140i, array assembled from the HP SmartArray Manager
ClearOS-DVD-x86_64.iso on USB, installing EFI mode.
hpdsa-1.2.4-140.rhel7u1.x86_64.dd.gz downloaded from HP Enterprise SDR (http://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p286329307/v108824/hpdsa-1.2.6-115.rhel7u1.x86_64.dd.gz) I also tried more recent ones.
Blacklisted AHCI and added inst.dd at grub boot prompt
The RAID volume does not appear at the install GUI.

However, using CentOS-7.1-1503 (kernel 3.10.0-229.el7.x86_64) or CentOS-7.2-1511 (kernel 3.10.0-327.el7.x86_64) the RAID is seen perfectly and the installation runs flawless.
Tuesday, February 02 2016, 01:07 AM
Share this post:
Responses (10)
  • Accepted Answer

    Tuesday, February 02 2016, 10:11 PM - #Permalink
    Resolved
    0 votes
    Confirmed.

    Once extracting the software and then mounting the dd file as a loop device the rpm is exposed. When running the install I get:


    [root@localhost x86_64]# yum localinstall kmod-hpdsa-1.2.6-115.rhel7u1.x86_64.rpm
    Loaded plugins: clearcenter-marketplace, fastestmirror
    ClearCenter Marketplace: fetching repositories...
    ClearCenter Marketplace: System not registered. Code: 3
    Examining kmod-hpdsa-1.2.6-115.rhel7u1.x86_64.rpm: kmod-hpdsa-1.2.6-115.rhel7u1.x86_64
    Marking kmod-hpdsa-1.2.6-115.rhel7u1.x86_64.rpm to be installed
    Resolving Dependencies
    --> Running transaction check
    ---> Package kmod-hpdsa.x86_64 0:1.2.6-115.rhel7u1 will be installed
    --> Processing Dependency: kernel(wake_up_process) = 0xe65cdceb for package: kmod-hpdsa-1.2.6-115.rhel7u1.x86_64
    clearos | 3.6 kB 00:00
    clearos-contribs | 3.0 kB 00:00
    clearos-fast-updates | 1.9 kB 00:00
    clearos-infra | 3.0 kB 00:00
    clearos-updates | 3.0 kB 00:00
    (1/5): clearos/7/group_gz | 1.5 kB 00:00
    (2/5): clearos-infra/7/primary_db | 11 kB 00:00
    (3/5): clearos-contribs/7/primary_db | 28 kB 00:00
    (4/5): clearos-updates/7/primary_db | 689 kB 00:01
    (5/5): clearos/7/primary_db | 858 kB 00:02
    clearos-fast-updates/7/x86_64/primary_db | 17 kB 00:00
    Determining fastest mirrors
    * clearos: clearos.mirrors.ovh.net
    * clearos-contribs: clearos.mirrors.ovh.net
    * clearos-fast-updates: download1.clearsdn.com
    * clearos-infra: clearos.mirrors.ovh.net
    * clearos-updates: clearos.mirrors.ovh.net
    --> Finished Dependency Resolution
    Error: Package: kmod-hpdsa-1.2.6-115.rhel7u1.x86_64 (/kmod-hpdsa-1.2.6-115.rhel7u1.x86_64)
    Requires: kernel(wake_up_process) = 0xe65cdceb
    Installed: kernel-3.10.0-229.7.2.v7.x86_64 (@anaconda/7.1.0)
    kernel(wake_up_process) = 0xfa87ae45
    Available: kernel-debug-3.10.0-229.7.2.v7.x86_64 (clearos)
    kernel(wake_up_process) = 0xec22466b
    You could try using --skip-broken to work around the problem
    You could try running: rpm -Va --nofiles --nodigest
    Like
    1
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, February 02 2016, 11:08 PM - #Permalink
    Resolved
    0 votes
    I've asked some team members who are more familiar with the kernel modifications to take a look at it. Thanks for your patience.
    Like
    1
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, February 04 2016, 04:58 PM - #Permalink
    Resolved
    0 votes
    Ok, a little more followup on what you are seeing here. The 'wake_up_process' was replaced in order to facilitate some functionality with IMQ.

    https://github.com/clearos/kernel/commit/d58b5231ad9d11b0815b178702ffe99045bb30b3#diff-b5a901bd97b1fb1bf7ab9cb1499b82a4

    The recommendation from our kernel developers is "dump imq, or use the weak update (kmod) system for kernel modules so their module isn't so unessarily picky about subtle symbol changes. or recompile their stuff against our kernel (which is what they do to support redhat kernels)."

    Effectively, your solution dumps IMQ. That may leave you with update issues in the future. The second suggestion of using weak updates would require a rewrite of your driver (not likely to happen unless you have some pull with HP). The third suggestion is to recompile the driver against the ClearOS kernel. This would be trivial if we had the source code for the driver but that is also unlikely since it is not an open source stack.

    IMQ is not being implemented in the kernel by upstream which is why it is causing the issue. In ClearOS, IMQ is necessary for some packet inspection services that are the basis of apps in the marketplace that are coming.
    Like
    1
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, February 03 2016, 02:44 AM - #Permalink
    Resolved
    0 votes
    Now I have proved that with the kernel from upstream CentOS the driver works. Could it be possible to compile a ClearOS kernel with the right symbol?
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, February 03 2016, 02:55 AM - #Permalink
    Resolved
    0 votes
    Thank-you Dave Loper very much for your interest and good disposition.

    As I need to crank the project up, and no other server made available, I decided to make a frankenbuild, installing to a pen drive, with the AHCI controller blacklisted, and no boot record on it as per the advice of Fredrik Forrnstad, in his post https://www.clearos.com/clearfoundation/social/community/how-to-install-raid-on-big-gpt-disks-while-using-uefi-booting [Thank-you very much for the guidelines] , then booted CentOS 7.1 build 1503 rescue mode with the hpdsa driver loaded, make the partitions in the HP Logic volume, copied all the files in the pen drive into the Logical volume, remounted the array in mnt/sysimage, installed the CentOS kernel and hpdsa module, and bingo! , got a working system.

    Here are the detailed steps I took:

    1. Installed ClearOS 7.1 in a 4GB pen drive, booting EFI mode and blacklisting AHCI kernel module editing the grub prompt.
    2. At partitioning, I unticked boot according to the above Frederik Fornstad instructions, select LVM and renamed the volume group clearos1
    3. Set Network, Date/Time, base system
    3. Let installation finish, but before rebooting, I followed the procedure again to update boot and boot/efi according to Frederik's post and taking note of the devices mounted in /mnt/sysimage.
    4. Rebooted the system,this time with the CentOS 7.1 (1503) USB install disk, added the already prepared USB key with the hpdsa driver taken from https:/downloads.linux.hpe.com/SDR/repo/spp/2015.10.0_supspp_rhel7.2_x86_64/hpdsa-1.2.8-107.rhel7u1.x86_64.dd.gz
    5. Selected Install, e for grub editing and at the boot line, after the "quiet" option I appended: modprobe.blacklist=ahci inst.dd
    6. Let the system boot, went to partitioning, this time I found the HP Logical Volume, select it, let the install partition it, with LVM and renamed the volume group just clearos, then letting the system begin the installation, I quit install immediately after started to copy files, just wanted a quick & dirty way for partitioning the drive, it could be done manually, however was short of time.
    7. Rebooted the CentOS USB install disk, this time in rescue mode.
    8. Again, press e, and at the boot line, after the "quiet" option I appended: modprobe.blacklist=ahci inst.dd and booted
    9. I let the system continue and not let mount the Linux install, just hit the option to get a shell prompt.
    10. Find the volume groups:
    lvscan

    INACTIVE '/dev/clearos1/root' [2.77 GiB] inherit
    INACTIVE '/dev/clearos1/swap' [528 MiB] inherit
    INACTIVE '/dev/clearos/root' [457.38 GiB] inherit
    INACTIVE '/dev/clearos/swap' [7.62 GiB] inherit

    11. Made the volume groups active:

    vgchange -a y clearos1
    vgchange -a y clearos # I named the volume groups different otherwise the OS would not allow to proceed

    12 Rescan the volume groups

    lvscan
    ACTIVE '/dev/clearos1/root' [2.77 GiB] inherit
    ACTIVE '/dev/clearos1/swap' [528 MiB] inherit
    ACTIVE '/dev/clearos/root' [457.38 GiB] inherit
    ACTIVE '/dev/clearos/swap' [7.62 GiB] inherit

    13.Type blkid to find the location of my devices and partitions, in my case were:

    /dev/sda PARTLABEL OEMDRV # This is the hpdsa kernel module disk Also get the UUID's
    /dev/sdb1 PARTLABEL = "EFI System Partition" # EFI partition of the ClearOS pen drive
    /dev/sdb2 TYPE=xfs # boot partition of the ClearOS pen drive
    /dev/mapper/clearos1-root # root partition of the ClearOS pen drive
    /dev/mapper/clearos1-swap # swap partition of the ClearOS pen drive
    /dev/sdc1 PARTLABEL = "EFI System Partition" # EFI partition of the RAID Volume
    /dev/sdc2 TYPE=xfs # boot partition of the RAID Volume
    /dev/mapper/clearos-root # root partition of the RAID Volume
    /dev/mapper/clearos-swap # swap partition of the RAID Volume

    14. copied the partitions from the pen drive into the RAID volume using dd

    dd if=/dev/sdb1 of=/dev/sdc1
    dd if=/dev/sdb2 of=/dev/sdc2
    dd if=/dev/mapper/clearos1-root of=/dev/mapper/clearos-root
    It is not necessary to copy the swap partition

    15. With the installed files now in the RAID Volume, proceed to mount it in /mnt/sysimage, but before, create the mount points

    mkdir /mnt/sysimage/boot
    mkdir /mnt/sysimage/boot/efi
    mkdir /mnt/sysimage/dev
    mkdir /mnt/sysimage/dev/shm

    16. Mount the RAID volume in /mnt/sysimage and special device tmpfs (as per the notes taken in point 3)

    mount /dev/sdc1 /mnt/sysimage/boot/efi
    mount /dev/sdc2 /mnt/sysimage/boot
    mount /dev/mapper/clearos-root /mnt/sysimage
    mount -t tmpfs -o size=50m tmpfs /mnt/dev/shm

    17 Chroot into the RAID Volume

    chroot /mnt/sysimage

    18. Verify that the partitions are mounted issuing df

    19. Activate the network, we'll need it in the next steps

    ifconfig eno1 192.168.x.y netmask 255.255.255.0
    route add -net 0.0.0.0 netmask 0.0.0.0 gw 192.168.x.1
    echo "nameserver 8.8.8.8" >> /etc/resolv.conf

    20. cd into /root, Get the hpdsa rpm and the kernel rpm

    wget http://vault.centos.org/7.1.1503/os/x86_64/Packages/kernel-3.10.0-229.el7.x86_64.rpm << from vault.centos.org as 7.1 is now deprecated
    wget https://downloads.linux.hpe.com/SDR/repo/spp/2015.10.0_supspp_rhel7.2_x86_64/kmod-hpdsa-1.2.8-107.rhel7u1.x86_64.rpm

    21. Install both rpm's

    yum localinstall kernel-3.10.0-229.el7.x86_64.rpm kmod-hpdsa-1.2.8-107.rhel7u1.x86_64.rpm

    22. Verify in /etc/fstab the partitions and/or UUIDs match the ones in the RAID Volume

    23. Create /etc/default/grub, mine looks like:

    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR="ClearOS"
    GRUB_DEFAULT=saved
    GRUB_DISABLE_SUBMENU=false
    GRUB_TERMINAL_OUTPUT="console"
    GRUB_CMDLINE_LINUX="nomodeset quiet" # I'm using nomodeset as my monitor is a bit old

    24. Enroll the HP GPG public Keys to validate the modules, otherwise the kernel may complaint for the hpdsa

    rpm --import http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub
    rpm --import http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub
    rpm --import http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub

    25. You may then verify the kmod-hpdsa rpm package signature

    rpm --checksig kmod-hpdsa-1.2.8-107.rhel7u1.x86_64.rpm
    kmod-hpdsa-1.2.8-107.rhel7u1.x86_64.rpm: rsa sha1 (md5) pgp md5 OK

    26. In preparation for installing the vmlinuz kernel and initrd, then execute:

    dracut --print-cmdline >> /etc/dracut.conf

    27. Edit Dracut configuration file

    nano /etc/dracut.conf
    Look at the end of the file, you should see these three lines:

    rd.lvm.lv=clearos/swap
    rd.lvm.lv=clearos/root
    root=/dev/mapper/clearos-root rootflags=rw,relatime,seclabel,attr2,inode64,noquota rootfstype=xfs

    Change the first two into:
    rd_LVM_LV=clearos/swap
    rd_LVM_LV=clearos/root # otherwise dracut will complait at forcing the generation of kernel image and initramdisk

    Encapsulate the whole last (very long...) line with quotes and add kernel_cmdline+= so it looks like this:

    kernel_cmdline+=" root=/dev/mapper/clearos-root rootflags=rw,relatime,seclabel,attr2,inode64,noquota rootfstype=xfs "

    Instead of devices, you may see UUID's, verify are the correct ones

    Save and exit dracut.conf editing.

    28. Force the generation of kernel image and initrd

    dracut --force

    Wait a little while....

    29. Generate grub2 Configuration file

    grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

    30. Create the boot entry in the motherboard's nvram

    efibootmgr --create --disk /dev/sdc --label "ClearOS 7 sda" --load "\\EFI\\centos\\grubx64.efi"

    Your device may be sdx

    31. Reboot, removing all the pen drives (installation USB disk, installed files USB disk and device driver USB as well). If all went smooth, you should have now ClearOS 7.1 with CentOS 7.1 kernel and the hpdsa kernel object loaded, making possible to boot from the RAID Volume.


    Results:

    Apparently it works fine, no errors, however there is a caveat, if attempting to do yum update, it will update the kernel, as the ClearOS one 3.10.0-229.7.2.v2.el7 supersedes 3.10.0-229.el7. Other than that, It runs very well, of course, I loose the kernel optimizations of the ClearOS one, but I got my ML110 with the controller b140i up and running with ClearOS. 7.1.
    Sorry for the long post.

    Update Feb 03 2016: To avoid the kernel to be updated from the ClearOS repositories, I installed most recent kernel from CentOS Vault repository. In this way, the above kernel supersedes the one at ClearOS for now, and after that you may safely run yum update

    yum localinstall http://vault.centos.org/7.1.1503/updates/x86_64/Packages/kernel-3.10.0-229.20.1.el7.x86_64.rpm
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, February 04 2016, 08:05 PM - #Permalink
    Resolved
    0 votes
    Dave Loper wrote:

    Ok, a little more followup on what you are seeing here. The 'wake_up_process' was replaced in order to facilitate some functionality with IMQ.

    Effectively, your solution dumps IMQ. That may leave you with update issues in the future. The second suggestion of using weak updates would require a rewrite of your driver (not likely to happen unless you have some pull with HP). The third suggestion is to recompile the driver against the ClearOS kernel. This would be trivial if we had the source code for the driver but that is also unlikely since it is not an open source stack.

    IMQ is not being implemented in the kernel by upstream which is why it is causing the issue. In ClearOS, IMQ is necessary for some packet inspection services that are the basis of apps in the marketplace that are coming.


    WIth all due respect, I have another suggestion for ClearOS Developers... Why not implementing the intermediate queuing device (IMQ) as a kernel object, like imq.ko, leavinq the kernel matching the upstream Redhat Kernel, therefore taking away the mismatch with proprietary kernel objects as the one subject of this post? Openwrt, a linux distribution made for routers, using IMQ, makes the kmod-imq.control as a kernel object to give kernel support for the Intermediate Queueing device https://dev.openwrt.org/browser/trunk/openwrt/target/linux/control/kmod-imq.control?rev=2430 .

    On a second thought, why the kernel object hpsa.ko (kernel object to support HP b140i as a SATA controller) is included in ClearOS kernel module tree, (/lib/modules/3.10.0-229.7.2.v2.x86_64/kernel/drivers/scsi/hpsa.ko) and not includes the hpdsa.ko that provides RAID functionality?

    Anyhow, I placed the issue at Hewlett-Packard Enterprise Community Forum as well. My inquiry may be as small as a mosquito itch, but is a try ... Here is a copy of it


    SmartArray controller b140i for RedHat Linux 7.x with IMQ device not loading

    Hi.

    Just want to bring your attention to the fact that when Red Hat kernel is built supporting Intermediate queuing device (IMQ), the symbol table changes to:

    kernel(wake_up_process) = 0xfa87ae45

    from your module's kmod-hpdsa compiled

    kernel(wake_up_process) = 0xe65cdceb


    The kernel complaints about the mismatch not loading the module

    Error: Package: kmod-hpdsa-1.2.6-115.rhel7u1.x86_64 (/kmod-hpdsa-1.2.6-115.rhel7u1.x86_64)
    Requires: kernel(wake_up_process) = 0xe65cdceb
    Installed: kernel-3.10.0-229.7.2.v7.x86_64 (@anaconda/7.1.0)
    kernel(wake_up_process) = 0xfa87ae45


    One solution may be to compile the RH kernel without the IMQ device, and compile the device as a IMQ kernel object, another solution may be you may compile hpdsa.ko to include the above symbol.

    Thank-you, regards,
    The reply is currently minimized Show
  • Accepted Answer

    Friday, February 05 2016, 01:48 AM - #Permalink
    Resolved
    0 votes
    I'm being told that if that was an option that they would have done that. IMQ, unfortunately, is no longer kmod-able. It used to be as is noted in your 10 year old source checkin. But not anymore. That is because it touches things like skbuff.h.

    You can see that and the other things that it hits here:

    https://github.com/imq/linuximq/tree/master/kernel/v3.x

    I suppose if the HP software was open source we would not be having this conversation because the solution would be so easy. As such, one thing that people might want to do to install the software is to install the CentOS kernel and use it temporarily to install the software. The question then remains, does the HP software actually require that symbol to work? There is a possibility that it doesn't utilize it at all or if it does, the changes by IMQ don't affect it. Whatever does that software have to do with 'wake_up_process'. Perhaps nothing at all. So the question is, if installed, does the software work on the ClearOS kernel?
    The reply is currently minimized Show
  • Accepted Answer

    Friday, February 05 2016, 05:37 PM - #Permalink
    Resolved
    1 votes
    As to the question, "why the kernel object hpsa.ko ... included in ClearOS kernel module tree ... and not ... hpdsa.ko"

    The reason for this is because hpsa.ko is open source and licensed under GPLv2. Not only does HP contribute to this package but so do many others. In short, this isn't HP's driver at all, it is the community driver that happens to enable HP equipment.

    The rpm kmod-hpdsa and all the code under it is licensed under this proprietary license. If we wanted to implement this code into our kernel we most certainly could do that for our own purposes (provided we had all of the proper sources...and the permission from HP to do so). However, we would not be able to distribute it to the larger community because it would have tainted the linux kernel and the prohibitions on the linux kernel's license would not allow us to do so. Moreover, the HP software ALSO does not allow this or even the ability to include the drivers on our media:

    "HP grants you a non-exclusive non-transferable license to use one copy of the version or release of the accompanying software for your internal purposes only"

    In addition that license states:

    - You may not use software to provide services to third parties. (We'd be doing this)
    - You may not make copies and distribute, resell or sublicense software to third parties. (...and this)
    - You may not download and use patches, enhancements, bug fixes, or similar updates unless you have a license to the underlying software. However, such license doesn't automatically give you a right to receive such updates and HP reserves the right to make such updates only available to customers with support contracts.
    - You may not copy software or make it available on a public or external distributed network. (...and this)
    - You may not allow access on an intranet unless it is restricted to authorized users.
    - You may make one copy of the software for archival purposes or when it is an essential step in authorized use. (...we'd violate this)
    - You may not modify, reverse engineer, disassemble, decrypt, decompile or make derivative works of software. If you have a mandatory right to do so under statute, you must inform HP in writing about such modifications. (this is what you've asked us to do)

    If we were to get some special dispensation from HP, we would certainly make running ClearOS easier on their equipment and we'd do so through agreement. It is in our interest to pursue this sort of arrangement and we will do what we can to make the lives of admins who use HP gear with ClearOS on it that much more turn-key and efficient. So we will see. I'm glad to see that there is interest in the community about furthering compatibility between HP and other vendors. It actually does help to ask for these things. We will talk to them about this and see what they can do.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, March 18 2016, 12:39 PM - #Permalink
    Resolved
    0 votes
    Any news in relation to this problem? I'm in the same case here with a new server stopped waiting to install ClearOS.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, March 28 2016, 11:03 PM - #Permalink
    Resolved
    1 votes
    There is a test ISO that has some UEFI updates to it. Please try it and let me know how it works:

    http://mirror.clearos.com/clearos/testing/daily_iso/ClearOS-DVD-x86_64-7.iso
    The reply is currently minimized Show
Your Reply