Linux

Systemctl: Assignment outside of section. Ignoring.

I wanted to throw together a quick post for a recent issue I have seen on Redhat 7/CentOS 7 boxes. A recent OS update has brought a small but important change to SystemD. In the past if you wanted to add environment variables to a SystemD service, you could enter # systemctl edit postgresql-14 (note I will be using postgresql-14 as the example service in this post), then add a line such as:

Environment=PGDATA=/opt/postgres/14/data/

After saving the file, and starting the service you are good to go. Recently after a minor update, I started getting the error “[/etc/systemd/system/postgresql-14.service.d/override.conf:1] Assignment outside of section. Ignoring.”, then the service would not start. It turns out, you can no longer drop Environment lines directly into the SystemD overrides, you need to mark which section of the SystemD file you are overriding. Below is the new proper way to go about this:

[Service]
Environment=PGDATA=/opt/postgres/14/data/

Quick fix, but can take a bit of digging. Also for SystemD and Postgres 14, this is the current way to easily redirect the data folder. Hope this helps someone!

CentOS 8 Migration

I have a pipeline which creates live images to network boot different systems. Historically this has been based on CentOS. A little while ago I moved it to CentOS 8 because I had some newer hardware that was not supported on the older kernel of 7. Everything was working well until recently when CentOS 8 went end of life, and I could no longer rely on the CentOS 8 Docker containers.

The journey began for a new EL8 system. I wanted to keep on EL8 instead of switching to Streams because all the other systems I had running were EL8 (CentOS 8 or RHEL8), and I wanted to keep compatibility. At the same time, I didn’t want to do a new build of the image, have things break, and not realize it was because of a CentOS Streams change upstream. I also used the CentOS 8 docker container which seems to have been pulled, so that forced me to do this change now.

My first thought was Oracle Linux. It has been around for a while, is ALMOST drop in compatible, and can be used without going and getting licenses (RHEL). (There are some small silly things like instead of “epel-release” the package is “oracle-epel-release-el8”) This lead to nothing but issues. I replaced all the repos I had in the image creation stage with Oracle Linux ones, then every build I got a ton of “nothing provides module(platform:el8)” lines for any package that used yum/dnf modules. I spent a chunk of time on this, finding no real answers, and one Oracle support page that looked like it could help saying I needed to buy a support contract. Classic Oracle. At one point I thought it had something to do with Commit – rpms/centos-release – 89457ca3bf36c7c29d47c5d573a819dd7ee054fe – CentOS Git server where a line in os-release confuses dnf, but then that line was there. Also Oracle doesn’t seem to have a kickstart url repo, which is needed to do this sort of network boot. They wanted the end user to set that repo up, which may be the source of my issues. This also touched on the issue Disable Modular Filtering in Kickstart Repos – Red Hat Customer Portal, but I wasn’t even getting to a base OS setup, then I could make changes to the os and dnf for how it processes modules.

In my searches I did find this nice script to get bash variables for OS and version. https://unix.stackexchange.com/a/6348

Then I figured I would try either AlmaLinux or Rocky Linux. They both came out around when Redhat said Cent 8 was going away. Looking into both projects, they both are backed by AWS and Equinix who are big players, which made me feel a bit better about it. I had heard a bit more about Rocky and its support, so I tried that. I dropped in the new repos, and kickstart location, and everything just worked… Even things that were a issue when playing with Oracle Linux went away. For example, epel-release was once again called what it should be.

In the end so far it seems to be happy! We will see if any other small differences pop up and bite me…

Below is an example of the top of the kickstart I am using, if anyone is interested in more of how I create live images, leave a comment and I can do a post on it:

lang en_US.UTF-8
keyboard us
timezone Europe/Brussels --isUtc
auth --useshadow --enablemd5
selinux --disable
network --device=eno1 --bootproto=dhcp
skipx
part / --size 4096 --fstype ext4
part /opt --size 4096 --fstype ext4
firewall --disabled

url --url=https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/kickstart/

# Root password
rootpw --iscrypted <Insert encrypted password here>

repo --name=baseos --baseurl=https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/ --install
repo --name=extras --baseurl=https://download.rockylinux.org/pub/rocky/8/extras/x86_64/os/ --install
repo --name=appstream --baseurl=https://download.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/ --install

SSSD with Active Directory Only Showing Primary Group

I was domain joining some Redhat Enterprise Linux 7 boxes to a Windows domain. Everything went smoothly except many of my users could only see their Primary groups. Some users whom had more permissions on the domain could see all their groups, just not some particular users. This seems to be a common failure scenario for SSSD with AD, and many people have opened bugs or chimed in with different fixes online. I found the solution on one forum post, and it saved me, and I wanted to amplify it.

As long as some of your users can see all their groups, you know its not exactly a problem with RHEL connecting to AD, or a protocol like LDAP being blocked. A odd side effect of this setup was periodically the groups could be scanned and then it would show the users in that group. If I ran “sss_cache -E“, then “getent group SecondaryGroup“, some of the time it would show the users inside the group. Then once the user logged in, the user would be removed via that command, as well as when I ran “groups” under the user.

The SSSD log didnt have a ton of help other than it couldn’t read all the groups. I tried a TON of the recommended settings, like enabling enumerate = True, enumerate = false, ldap_use_tokengroups = true, ldap_use_tokengroups = false; none of these changed anything. Then https://serverfault.com/a/938893 mentioned it may be a permissions problem between the computer object in AD and the user object. I looked and sure enough, my system had NO permissions on the users that were failing. I attempted to add the tokenGroups permission mentioned in this article and that still didnt help, but we were on the right track!

The answer came from https://serverfault.com/a/796005, there is a permission needed called “Read Remote Access Information”, once that is granted to your computer object onto the user, then secondary groups will start populating. I gave “Domain Computers” that permission, since it seemed to only be effecting some of the Linux systems and Windows was happy to have it as well.

Some random commands that can help you debugging SSSD:

SSSD likes to cache a lot, making it hard to troubleshoot, using the following clears all caches and restarts SSSD:

systemctl stop sssd && rm -rf /var/lib/sss/db/* && rm -rf /var/lib/sss/mc/* && systemctl start sssd

Installing HipChat 4 on Fedora/Rhel/CentOS/el 7

HipChat 4 has recently come out, and then shortly after it was released to my companies internal HipChat server. Being a Linux user I hoped that the aged HipChat 2 client was finally updated for Fedora or Red Hat or CentOS 7 so I could just use yum to install it. When I went to the download page the old yum instructions were replaced by only Ubuntu/Debian instructions! After playing around with the Debian package and getting it to load, I thought I would look at the repo a little more. Low and behold, Atlassian is making a yum repo! Just not publishing instructions on how to use it! The downside is they seem to not be signing the repo, but the code below works with yum to download the latest version.

 

sudo bash -c ‘cat > /etc/yum.repos.d/hipchat.repo << EOF_hipchat
[atlassian-hipchat]
name=Atlassian Hipchat
baseurl=https://atlassian.artifactoryonline.com/atlassian/hipchat-yum-client/
enabled=1
gpgcheck=0
EOF_hipchat’

sudo yum update

sudo yum install hipchat4

Fixing CentOS 6.6 Kickstart Issues

I recently have been working on a system automating CentOS 6 installs for servers. When upgrading to 6.6 my test environment (VMWare Fusion) stopped working. I got a hard kernel panic and halt on loading. Now VMware forums and CentOS site, have posts about work arounds for this. A bunch of them are complex and involve changing modules around, and other files. There is a very easy fix for this, and its detailed below.

NOTE: I am running VMware Fusion, so I will open a package, in Windows and Linux you dont have to do this, just go to the folder.

  1. Stop the VM
  2. Find the VM files
    1. For Fusion there will be a %Your VM%.vmwarevm file, you have to right click that and “Show package contents”
  3. There should be a %Your VM%.vmx file, open that with a text editor
    1. If you are on a Mac, or other machine that likes to do smart quotes, make sure to use a program like vim or Sublime Text that doest add “smart quotes”
  4. A line will read: ethernet0.virtualDev = “e1000e”, change that to ethernet0.virtualDev = “e1000”, just remove the last e. This changes the card from a E1000 in enhanced mode to a normal one. Now CentOS 6.6 will boot.

Here are some place people have discussed issues:

https://communities.vmware.com/message/2443777

VM Experimentation

I am the type of programmer/IT person who enjoys having all my experimentation of systems done inside a virtual machine. That way if I break something, I can easily role back the virtual machine or just delete it. As seen in my last post, I recently built a new NAS. The original plan was to turn my old server into a Proxmox or ESXi box, the downside to that plan I found out quickly; the old box used DDR2, and at this point to get DDR2 memory it is quite expensive. That, along with my worry of power usage on the old box, I decided to give another solution a try.

After researching around I found my local Fry’s Electronics had the Intel NUC in stock. This is a tiny tiny PC that can take up to 16GB of RAM, has an Intel Core i5, and only uses 17 Watts. The box also has Intel vPro; what is vPro you ask? vPro allows you to remotely manage the system, so I can remote into it without buying a fancy management card, I can also remote power the box on and off, or mount a virtual CD. not bad for a ~$300 box. The model I got, DC53427, is a last gen i5, so it was a little cheaper, at the cost of having only 1 USB 3.0 port. It came with a VESA mount, so the NUC could be attached to the back of a monitor, that was a nice feature. I got USB 3.0 enclosure for 2 older 500GB hard drives, and used those as my storage. I installed Proxmox  on the system since my work has been starting to use that software more and more, and this was a chance for me to learn it.

A quick note about Proxmox to those who have not used it, I had come from a VMWare background so my work was my first experience with Proxmox. It is a free system, the company offers paid subscriptions for patches and such, without that the web page bothers you one time when you login, and you just dismiss the message. The software is a wrapper around KVM and some other Linux virtualization technologies. It can handle Windows and Linux systems without a problem. The interface is completely web based, with a Java virtual console; if you don’t update to the latest patches the java console can break with Java 7 Update 51. The software works well enough. There are still some areas that is needs improvements; in VMWare if you want to make a separate virtual network you can use their interface, on Proxmox that’s when you go to the Linux console and start creating virtual bridges. But once I got everything working, it seemed to work well. I don’t know how long I will keep it without trying another system, but for now it is nice. Since the system relies on KVM, it can do feautres like Dynamic memory allocation, if a VM is only using 1 GB of ram but is allocated 6, it will only take 1GB at that time. Also KVM can do deduplication of memory, so if two VMs are running the same OS, it only stores those files in memory once, freeing up more memory space.

I ran into one problem during install of Proxmox, the NUC is so fast, that it would start to boot before the USB 3.0 hard drives had been mounted. After searching around everywhere I found a fix on http://forum.proxmox.com/threads/12922-Proxmox-Install-on-USB-Device; adding a delay in the GRUB boot loader allows enough time for the system to mount the LVM disks correctly and then start. At first I just went to the Grub boot menu, hit “e” then added “rootdelay=10”, to the “linux /vmlinuz-2.6.32-17-pve root=/dev/mapper/pve-root ro rootdelay=10 quiet” line. After the system loaded I went into /Boot and added the same entry to the real Grub menu. Now I had a Intel NUC with 1TB of storage and 16GB of RAM. I could have used the NAS with iSCSI, but that was a lot of config I didn’t want to do; along with, I was setting up some Databases on the system and didn’t want the overhead of using the NASs RAIDZ2 at this time.

I have been using it for a few weeks, and its a nice little box. It never makes a audible level of noise (although it does sit next to its louder brother the NAS). Down the road if I want more power I can always get another NUC and put Proxmox into a clustered mode. These boxes keep going down in price and up in power, so this can grow with my needs.

LDAP Authentication RPI Tutorial (Part 2)

Last time I spoke of how to setup ldap with PHP and briefly touched on using the “ldapsearch” command. I would like to go more in-depth on “ldapsearch”, and show you how you can use it to craft searches for your PHP application. Specifically for RPI, if the user has a RCS account, they can ssh into “rcs-ibm.rpi.edu” and run the following commands. (RCS-IBM puts you on either clark.server.rpi.edu or lewis.server.rpi.edu, these two have the commands you need on them and run AIX) To briefly review the command:

  • First we add the command, then enter the host you are searching, tell the server to try simple anonymous authentication. Next give the server a base to start the search (I am using RPI specific domain components), finally we have to give the heart of our search. I am looking for any Unique ID (username) that starts with “berk”, and ends with anything “*”.
  • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’ ‘uid=berk*’”

The main part of the search we will be editing is the ending. Here we specify a filter to find the information we are attempting to access. Each LDAP server has different attributes it can give about each object. For example, the ldap.rpi.edu server gives out “givenName, objectClass, cn(full concatenated name, or common name), sn (surname), loginShell,” and many others; while at the same time “ldap1.server.rpi.edu” returns a much different lists of results.

Finding Which Attributes Will be Returned

The best way to find which fields are available is by doing a search without a filter. Just running the search below will return an unfiltered list of everything in the directory, up till you hit the individual servers limit. I am purposefully not publishing results from these searches for privacy reasons; here is some results for me with some data omitted.

  • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’”
  • # berkod2, accounts, rpi, edu
    dn: uid=berkod2,ou=accounts,dc=rpi,dc=edu
    sn: Berkowitz
    cn: Berkowitz, Daniel
    objectClass: top
    objectClass: posixAccount
    objectClass: inetOrgPerson
    objectClass: eduPerson
    objectClass: rpiDirent
    objectClass: mailRecipient
    objectClass: organizationalPerson
    objectClass: person
    uid: berkod2
    loginShell: /bin/bash
    uidNumber: #####
    mailAlternateAddress: berkod2@rpi.edu
    givenName: Daniel
    gecos: Daniel  Berkowitz
    rpiclusterhomedir: /home/berkod2
    description: PRIMARY-STU
    homeDirectory: /home/06/berkod2
    gidNumber: ###

Now that we have an idea about the data structure and what this server has on it we can reverse the lookup and tweak it. I know ‘uid’ will be the username, and I can get the users name from that! So using CAS I can log a user in and get their username, then I can lookup there LDAP information. (EXAMPLE 1) If a user enters a name, then a user can search for their UID doing the reverse. (EXAMPLE 2) The wild card can also be used if the full name is not known. (EXAMPLE 3) Last we can use multiple fields, combining these ideas to narrow down the result. (Example 4)

  • Example 1
    • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’ ‘uid=berkod2’”
  • Example 2
    • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’ ‘sn=Berkowitz’”
  • Example 3
    • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’ ‘sn=Berko*’”
  • Example 4
    • “ldapsearch -h ‘ldap.rpi.edu’ -x -b ‘dc=rpi, dc=edu’ ‘sn=Berko*’ ‘uid=berkod*'”

LDAP Authentication RPI Tutorial (Part 1)

After writing about how to use CAS with PHP, I thought I would write a post about how to use LDAP(Lightweight Directory Access Protocol) at RPI but these methods can be used anywhere. LDAP is a protocol to query user databases, this is a protocol that can be sed along with Active Directory, or another directory system for computers and user accounts. This protocol is widely used to allow different applications to interact with your user database. Here I will be showing how to implement search with LDAP to a web application. This guide will be using LDAP with PHP, this requires the LDAP module to be enabled within PHP; that will be the purpose of this article, then the next one will discuss how to actually query LDAP.

LDAP Linux (Debian/Ubuntu) Install

Linux is easy to get LDAP working with PHP, as long as you have a standard installation of Apache, with PHP 5 working.

  1. Install the LDAP module onto the machine, using either aptitude or apt-get
    • “sudo aptitude install php5-ldap”
    • OR “sudo apt-get install php5-ldap”
  2. PHP should now be able to use LDAP, if it is not working yet, you will need to restart Apache.
    • “sudo service apache2 restart”

LDAP Windows (XAMPP) Install

Xampp for Windows comes with LDAP, but there is a bug in their implementation and a file needs to be copied before LDAP will work. I am going to use “C:\xampp”, the default directory for Xampp in this example.

  1. Go into the PHP folder, C:\xampp\php\
  2. Edit the file “php.ini” with any text editor
  3. Find the line “;extension=php_ldap.dll”, and remove the semi-colon. “extension=php_ldap.dll”
  4. Now if you were to reboot Apache it should be working, but its not! Why not? There is a missing DLL. You need to
    copy libsasl.dll from c:\xampp\php\libsasl.dll to C:\xampp\apache\bin\.
  5. Now restart Apache

LDAP Search

Now that PHP can search LDAP we are going to want to start creating queries in PHP; but it is much easier to tweak the search in the command line, and then put that query into PHP. The following are steps that can be taken on a Linux computer (again Ubuntu/Debian) to install and use a ldap command line search.

  1. First we need to install the OpenLDAP utilities that will give us the “ldapsearch” command
    • “sudo aptitude install openldap-utils”
    • OR “sudo apt-get install openldap-utils”
  2. Now we are making our query
    • First we add the command, then enter the host you are searching, tell the server to try simple anonymous authentication. Next give the server a base to start the search (I am using RPI specific domain components), finally we have to give the heart of our search. I am looking for any Unique ID (username) that starts with “berk”, and ends with anything “*”.
    • “ldapsearch -h “ldap.rpi.edu” -x -b “dc=rpi, dc=edu” “uid=berk*”
    • Now this gives one result, and this can be used to see what data will be returned from this server. You can also try “ldap1.server.rpi.edu” this returns a entirely different list of variables, and sometimes more users.
    • If you are interested in researching this command more, die.net has a great resource. http://linux.die.net/man/1/ldapsearch
    • Troubleshooting: For those of you here at RPI trying to follow this guide specifically, if you do not get any results or a error connecting, RPI firewalls the LDAP servers heavily. I have found a lot of the time I have to be in the VCC to make this work, you can also VPN in, then your network connection is within the VCC and it will work. I have VPNed in, while on campus in the Union to get LDAP to work.

UPDATE: I added a little about what LDAP is

Adding Strict Standards, (Or Removing)

I have been developing on a Debian Apache system for a long time, for one of the projects I have been working on I had to run on a Windows Server. After installing XAMPP I noticed that by default, XAMPP sets the developer settings of Strict Standards. Meanwhile the Linux system didn’t have that enabled. I went out wanting to set the Linux server to have the Strict settings to force me to code properly. I found many places that would tell me how to disable strict standards (because users found it bothersome), but after a little searching I didn’t find a clear guide so I thought I would write one for PHP5.

  1. Find “php.ini”
    • Windows (XAMPP): The file is kept under C:/xampp/php/php.ini
    • Linux: For php5 (at least in debian) /etc/php5/apache2/php.ini
  2. Open the file and scroll to;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ; Error handling and logging ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  3. The line you want to edit iserror_reporting =…
  4. Copying the line below will enable strict settings, while other combinations listed below will enable different values.
    • Development Value: E_ALL | E_STRICT

My Linux box configured itself as  “error_reporting = E_ALL & ~E_DEPRECATED”, which is the standard for production.

Here is the description that comes with php.ini,

;Default Value: E_ALL & ~E_NOTICE
; Development Value: E_ALL | E_STRICT
; Production Value: E_ALL & ~E_DEPRECATED
; http://php.net/error-reporting