SUSE Linux Enterprise servers (SLES) need subscription licenses to receive updates (and technical support by SUSE staff, if running on a “standard” or “priority” subscription), and you have the choice of one-year or three-year licenses. Those licenses come in the form of “activation codes” that are used to register each server with the SUSE registration service, still known as NCC (“Novell Customer Center”).
SLES-equipped servers can run Xen, a hypervisor shipped for free with each SLES version, and you can run as many SLES virtual machines on top of that hypervisor as you like (license-wise), as long as the physical machine is a properly licensed SLES server. (There are other options for licensing, if you’d rather employ a different hypervisor, but that’s outside the scope of this article.)
A nice feature is that you need not provide activation codes with these virtual machines (VMs), so once you’ve set up the base system (“Dom0”), you’re ready to go. But how does that work, what does that mean for typical (and not so typical) scenarios, and what can you do if things don’t work as expected?
As many times, this article is the result of some practical problem that we encountered with our own or with customer systems. This time, it was during a license renewal. Updating the base system worked like a charm, but the VMs associated with the original license did not pick up the new code. So how is this all supposed to work, and why didn’t it work as expected?
Quick overview
Looking at a typical setup, four major components enter the picture:
- The physical Xen server(s), for the sake of discussion configured to create a cluster where each virtual machine can run on any of the Xen servers
- The virtual machines (I’ll focus on SLES machines, as we’re talking about SLES licensing issues)
- A “SUSE Subscription Management Tool” server, “SMT” in short. It’s a free (as in “costs no extra bucks”) tool for SLES, allowing you to centralize all updates and registrations of SLES systems within your environment. Instead of individually contacting SUSE’s update and registration servers via the Internet, your servers will register with your SMT server and receive updates via that server. It is SMT’s job to keep an up-to-date copy of all required update repositories on the SMT machine and to “discuss” all registration issues with NCC, easing conformance to security policies (only the SMT server needs access to the Internet) and minimizing downloads (only that single SMT server downloads updates from SUSE’s servers, and only once).
- “SUSE Customer Center”, still known as NCC to long-time customers.
Having an SMT server does not change the registration mechanisms that I’ll describe in this article, but it gives you a local interface to your licensing data and will be referenced in my descriptions.
During installation, you’ll register your SLES server to receive updates. When it’s a physical server, this will require you to specify your NCC account’s email address and the license’s “activation code”. A tool called “suse_register” runs on your freshly installed server and will generate a unique “GUID” for your system, if not already present, and then tries to contact either NCC or SMT, if you happen to have specified the SMT URL under the advanced options. During this registration run, the validity of the license associated with the activation code is verified by the registration server (especially checking that it’s not expired yet) and some system details, especially information about SUSE products installed, is transferred. As our physical servers are Xen hosts, “suse_register” will also enter the system’s GUID in the Xen store, in a “well-known location”.
When you’re installing a virtual machine on one of the Xen hosts, “suse_register” will detect this fact and will not even bother to ask for your email address nor activation code. Instead, it takes the GUID from the Xen store (which is accessible to the VMs as well as to the Dom0) and will tell the registration server both about being a Xen VM and the GUID of the Dom0, in addition to its own GUID. The registration server then is able to match that reported Dom0 GUID with the list of already registered systems and links your VM to the same license as used by Dom0.
When it comes to NCC, I advise to be patient: While it may appear to be a live access to the database, the results of “suse_register” and SMT actions will not appear immediately via the NCC web interface. So if you register a system and don’t see it in NCC right away, or the data seems to be incomplete, check again a few (or more) coffees later – I typically give it one hour before I start getting nervous.
To see how registration worked from the client side, you can have a look at the file “.suse_register.log” in the home directory of the root user – look for XML nodes named “<register />”, with sub node “<guid />” (and “<host />”, if you’re inside a VM). You’ll recognize the GUIDs used there as being those named “System ID” in NCC or listed as “Unique ID” in the various outputs of SMT. The GUID of the SMT server reflects in the NCC CSV-formatted exports of your registered systems – there’s a column named “SMTGUID”, so if you have multiple SMT servers, you can see which one feels responsible for which system.
On your server systems, the GUID is stored in “/etc/zmd/deviceid” for SLES10, or in “/etc/zypp/credentials.d/NCCcredentials” (the “username” field) if you happen to run SLES11.
Possible (and likely) problems
Whatever can go wrong, will go wrong – eventually.
Reading the description above, one possible breaking point pops up immediately: What if the Xen subsystem has not been available, when the Dom0 GUID was to be stored there? Well, it depends. In all cases, there will be no GUID in the Xen store. The conclusions drawn from that, are differing by SLES version: When registering a SLES10 DomU, a missing Dom0 GUID will make the VM count as a physical system – and you might exceed the limits of your licensing, at least according to NCC.
SLES11, on the other hand, will still detect that you’re inside a Xen VM, but will register with a Dom0 GUID of “Y”. That’s hard-coded, at least when I look at the SLES11 SP3 code, and of course makes matching your DomU to some valid license impossible. But it still is a VM, so probably the licensing server will not bother too much.
Now that situation (no Dom0 GUID in xen-store) wasn’t too common with SLES10, but obvious it happened sufficiently enough to justify creating a TID for this issue. SLES11, on the other hand, seems to have quite an issue with this: Unlike SLES10, “suse_register” is no longer run during system startup in SLES11, so there’s no-one to ever enter the Dom0 GUID after starting Xen on the physical server (the xen-store entry is non-persisting, thus gone after a reboot). This breaks the automatism and, in my eyes, is a pain in the back, but when I have been talking to people about this, the answers ranged from “you have to specify the activation code even when registering VMs” to “modify ‘/etc/suseRegister.conf’ as per the TID”. But what nobody ever mentioned was a way to fix the root cause: Enter the GUID into xen-store!
I especially object registering VMs while specifying activation codes, because at least every three years (or even annually, depending on your license) you’ll have to renew, which results in re-registering the systems with a new activation code. Even hard-coding Dom0’s GUID in the DomU’s configuration file (as suggested by the TID) can be a night mare, since you’ll have to do that for every virtual machine… and if for whatever reason you need to change the GUID of your server, you’ll have to update the configuration files. All of them. I’d rather do that in a single place – the Dom0. And not to have to bother again, even when updating the GUID of the Dom0.
How can we achieve this? Before starting the first VM (or more precisely, before running “suse_register” in the VM), we have to make sure that the GUID is entered into the xen store. There’s an official way to do this – call “suse_register” on Dom0. You need not specify any codes nor email addresses – just call it, and if you want it to be quicker, give a “-n” to suppress optional data (that would have to be gathered from your system first).
If there’s an official way, there’s another one, right? Right. As mentioned above, the GUID is stored in /etc/zmd/deviceid (SLES10) or /etc/zypp/credentials.d/NCCcredentials (SLES11). You can grab that value and use a simple and quick one-liner to update the xen store:
xenstore-write /tool/SR/HostDeviceID the_guid_of_the_server
You needn’t even check for a previous value, which is simply overwritten.
Once you have the Dom0 GUID set, invoking “suse_register” inside the VMs will update your registration with the proper association to the license of the physical machine. And for what I can tell, this will even work when you renew the license of the physical machine: Once the next “suse_register” is run successfully, the association with the new license is stored in NCC. Such a call to “suse_register” is made for you automatically once per month by a cron script (see “/etc/cron.d/novell.com-suse_register”), which probably will be the root cause if you see VM registrations shifting from one license to another, from time to time. It simply depends on which GUID is retrieved from the xen store by “suse_register”, thus on which physical server the VM is active at that moment.
We’ve noticed a problem once the old license had actually expired… calling “suse_register” from within a VM did not move the registration, but rather complained about an expired license. That probably is a bug in the registration server, as it has to check against the “host license”, rather than the VM license, if the Dom0 GUID is given during registration. But it looks as if that code simply checks the license associated with the system’s GUID (which happens to be the expired one) and as no new activation code is given, refuses to continue. I’ve reported that problem to SUSE already, we’ll see if and when this might get fixed. But do not despair, two easy work-arounds are available: Either run “suse_register” in your VMs before the old license expires, or delete all registrations via your SMT server: “smt-list-registrations -v” lets you “grep” for the (old) registration code and “grep” / “cut” to select the associated systems’ “Unique code”, which you can then feed to “smt-delete-registration -g <guid>” calls. After the next “smt-register; smt-ncc-sync” cycle you’ll be able to register the DomUs without any problem.
Beautifying you data
When you run a Xen cluster, it obviously isn’t optimum to see your VM registrations spread across all your physical cluster servers, or rather their licenses. But now that you know how the mechanics of this work, you can simply store one of the cluster servers’ GUID in all xen stores, on every Xen host (you’ll obviously have to hard-coded that) and voila, every DomU will register against the license associated with that “master server”. Only thing to look out for is that periodic “suse_register” call I mentioned above… but when you’re good enough to fine-tune your registration details, you should be able to update that cron job so that the “master” GUID is put back into the xen store right after.
This made for a somewhat lengthy article, thank you for staying with me this long!