Phil's Occasional Blog : Software

Converting CentOS 8 Stream to AlmaLinux 8.4

There's no better time to migrate CentOS 8 Stream to AlmaLinux 8.4. Because RedHat Enterprise Linux 8.4 has only just been released, the differences between Stream and 8.4 are minor.

First off, make sure your box is up to date:

dnf distro-sync

Then back it up or snapshot it. Some packages may revert to older versions and break things.

Then download the AlamLinux conversion script

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh

edit almalinux-deploy.sh

MINIMAL_SUPPORTED_VERSION='8'

In REMOVE_PKGS add "centos-stream-release" "centos-stream-repos"

Amend the grep command in get_os_version:

        if ! os_version="$(grep -oP 'CentOS\s+(Linux|Stream)\s+release\s+\K(\d+(\.\d+)?)' \

                                    "${REDHAT_RELEASE_PATH}" 2>/dev/null)"; then

Or alternatively, download my modified install script from my repo:

curl -O https://raw.githubusercontent.com/philrandal/almalinux-deploy/master/almalinux-deploy.sh

Then you're set to go. The script works with CentOS 8.3 and CentOS 8 Stream.

bash almalinux-deploy.sh

and if it succeeds

reboot

After the reboot, check for orphaned packages: I got this from migrating a full CentOS 8 Stream install to AlmaLinux 8.4

dnf list extras
Extra Packages

llvm-compat-libs.x86_64 10.0.1-1.module_el8.4.0+533+50191577

They can safely be removed.

When you reboot, you'll see that the original CentOS rescue kernel is still there.

Fix that with:

rm -f /boot/vmlinuz-0-rescue-* /boot/initramfs-0-rescue-*.img
/usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) "" /lib/modules/$(uname -r)/vmlinuz

To remove old CentOS kernels,

rpm -qa | grep kernel-core

Then use yum remove to get rid of any superfluous ones.

Test, test, and test again.

Remove any snapshots once you're happy.

Postscript:

So you just want to convert back to CentOS 8.x from CentOS Stream?

No worries (except for the caveat that downgrades may break things).

curl -O http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/Packages/centos-linux-release-8.3-1.2011.el8.noarch.rpm
curl -O http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/Packages/centos-linux-repos-8-2.el8.noarch.rpm

rpm -e --nodeps centos-stream-repos centos-stream-release

rpm -ivh centos-linux-release-8.3-1.2011.el8.noarch.rpm centos-linux-repos-8-2.el8.noarch.rpm

dnf distro-sync

reboot

Posted by Phil at 5:28 PM
Edited on: Wednesday, June 09, 2021 12:55 PM
Categories: IT, Software

About to update to RedHat Enterprise Linux or CentOS 7.7? Beware!

There's a couple of caveats before upgrading to RHEL / CentOS 7.7.

1: Upgrading to latest RHEL 7 PCP package pcp-4.3.2-2.el7 gives several issues.

During installation/upgrade of pcp-selinux:

    Updating / installing...
    1:pcp-selinux-4.3.2-2.el7 ################################# [100%]
    Failed to resolve allow statement at /etc/selinux/targeted/tmp/modules/400/pcpupstream/cil:83
    semodule: Failed!

Followed by lots of selinux errors logged.

Workaround is to install 7.7's selinux-policy first

    yum update selinux-policy

or, if you've already updated,

    yum reinstall pcp-selinux

Documented in RedHat Bugzilla entry 1714101

2: bind is updated from version 9.9 to 9.11. This breaks configurations in which the same zone file is defined in several views.

The solution is to use bind 9.11's "in-view xxx" in subsequent definitions of the same zone in different views.

The fix is described in detail here.

Posted by Phil at 7:32 PM
Edited on: Saturday, September 14, 2019 3:47 PM
Categories: IT, Software

A Raspberry Pi Stratum 1 NTP Server

Overview

Updated for the Pi 5 and Raspberry Pi OS Bookworm

The following instructions are how to make a cheap Pulse Per Second (PPS) disciplined Stratum 1 NTP Time server using one of the Raspberry Pi U-blox M8Q based GPS boards sold by Uputronics.

Our basic requirement is for an NTP server which will work standalone without connectivity to other NTP servers, so if a datacentre loses connectivity its servers will still have a stable time source.

As we’re only interested in time and not height, motion, or location, all we’re interested in is the NMEA xxZDA and xxRMC sentences and the highly accurate PPS signal from the GPS module, both of which we feed straight into the ntp daemon. The xxZDA sentences (which give us full 4-digit years) are not output by default by the M8Q so we have to enable them at boot time.

Many guides on the net use gpsd to feed ntp via shared memory. That’s an additional overhead and complexity, especially if we want to enable Galileo reception (gpsd initialises the U-blox the way it sees fit). I prefer to let the NTP daemon read the NMEA stream from the GPS receiver itself, avoiding the middle man.

In this guide we’ve chosen to use NMEA output from the GPS module via the ntpsec Generic GPS Receiver driver 20 ( https://docs.ntpsec.org/latest/driver_nmea.html ) and the PPS driver ( https://docs.ntpsec.org/latest/driver_pps.html ).

You will need a Raspberry Pi 4B, the Uputronics Raspberry Pi+ GPS Expansion Board and a suitable GPS antenna.

This guide assumes that you’re using Raspberry Pi OS Bookworm, 64-bit Lite edition. Download and write this to an SD card.

See https://www.raspberrypi.org/documentation/installation/installing-images/windows.md

You'll need a keyboard and display attached for first boot configuration.

Create a user (mine was ntpadmin, use whatever you like) and set its password.

Attach the Uputronics Raspberry Pi+ GPS Expansion Board to the Pi, insert the SD card, connect the antenna and network cable and boot the Pi up. Either connect locally or via SSH to the Pi. If you can’t SSH in and don’t have a monitor see this

https://www.raspberrypi.org/documentation/configuration/wireless/headless.md

Follow the instructions carefully if you miss steps things won’t work.

The Uputronics board has u-blox firmware 3.01 on it, dated 2016.

The week number rollover is set to 1867 (October 2015). All transmitted week numbers are mapped to the ~19.5 year period between week 1867 and week 2990 (April 2035).

A Note About Accuracy

In theory, GPS-based time receivers can give a very high accuracy, with the PPS (Pulse per Second) signal being accurate to within 10ns.

However, fix data from gpsctl shows:

pi@ntp2:~ $ /usr/local/bin/gpsctl -Q fix

Time (UTC): 2019-04-18 15:31:27 (yyyy-mm-dd hh:mm:ss)

Latitude: 52.05994980 N

Longitude: 2.72698960 W

Altitude: 198.789 feet

Motion: 0.338 mph at 53.114 degrees heading

Satellites: 5 used for computing this fix

Accuracy: time (39 ns), height (+/-18.199 feet), position (+/-103.911 feet), heading(+/-8.230 degrees), speed(+/-0.132 mph)

Note the 39ns time accuracy from that fix. The more satellites, the better.

Errors are also caused by signal delay (4ns per foot) in the cable from the aerial to the GPS receiver, etc.

So we’d be lucky to get 100ns accuracy from the PPS pulse.

Add to that the processing overheads of the PPS interrupt, and processor clock jitter in the Raspberry Pi, and overheads transferring and decoding the NMEA sentences. The jitter on the PPS signal is less than 5 microseconds.

The output from the ntp NMEA driver without PPS correction can have a jitter of a few milliseconds, and an offset from real time which has to be tweaked manually to get within 5ms of the correct time.

NTP sources on the internet can show offsets / jitter of over 5ms, local LAN 50us or more.

On the Pi 4B we can expect an average jitter of less than 1 microsecond.

Required Components

I sourced my components from the Pi Hut unless noted elsewhere:

1 Raspberry Pi Model 4B 1GB RAM

1 8GB or larger micro SD card

(a Transcend High Endurance 32GB micro SD card would be better choice than a generic one for longevity)

1 Uputronics GPS Hat

1 Pi Hut Pi 4 GPS case

1 Raspberry Pi 4 power supply

1 GPS SMA Antenna

(optional) 1 SMA-male to TNC-female (or BNC, as needed) adaptor to connect to existing GPS aerial (from Amazon or eBay)

Prerequisite Settings

Login as pi / raspberry, and immediately change the password from the default

passwd

sudo raspi-config

Advanced Options

Expand filesystem (this is automatically done on first boot)

Interfacing Options

SSH -> Would you like the SSH server to be enabled – Yes (Recommended)

I2C -> Would you like the ARM I2C interface to be enabled? - Yes

Serial -> Login Shell (no) Hardware (yes) (Optional)

Quit but no need to reboot at this point.

for i in systemd-timesyncd avahi-daemon alsa-state bluetooth triggerhappy hciuart rng-tools autologin getty;

do

sudo systemctl disable $i.service; sudo systemctl stop $i.service;

done

sudo systemctl disable serial-getty@ttyAMA0.service

sudo systemctl mask serial-getty@ttyAMA0.service

sudo apt install pps-tools ntpsec ntpsec-ntpviz i2c-tools linux-cpupower

sudo systemctl enable ntpsec

sudo apt purge bluez bluez-firmware wpasupplicant (assuming no wifi required)

sudo apt update

sudo apt full-upgrade

If you want your Pi to run at full clock speed (active cooling is recommended),

sudo nano /etc/systemd/system/cpu-governor.service

[Unit]

Description=CPU Governor Service

After=network-online.target

[Service]

Type=oneshot

ExecStart=/usr/bin/cpupower frequency-set -g performance

[Install]

WantedBy=multi-user.target

Then

sudo systemctl enable cpu-governor.service --now

sudo nano /boot/config.txt and add at the bottom :

dtoverlay=disable-bt
dtoverlay=disable-wifi
dtoverlay=pps-gpio,gpiopin=18

On a Pi 4 or below, make sure that enable_uart=1 is set in /boot/config.txt

On a Pi 5, make sure that dtparam=uart0=on is set in boot/config.txt

sudo nano /boot/cmdline.txt and add at the end

nohz=off

sudo nano /etc/modules and add at the bottom :

pps-gpio

Save and Quit Nano.

and then sudo reboot

Enabling Galileo Satellites and Setting Stationary Mode

Stationary mode gives a faster GPS fix.

See Tom Dilatush’s gpsctl:

http://www.jamulblog.com/2017/11/paradise-ponders-gpsctl-functionally.html

I’ve customised it further to enable the use of a config file and put it in my GitHub repo:

https://github.com/philrandal/gpsctl

Download and build gpsctl.

cd ~
wget https://github.com/philrandal/gpsctl/archive/master.zip
unzip master.zip
mv gpsctl-master gpsctl
cd gpsctl
./build.sh
sudo cp ~/gpsctl/gpsctl /usr/local/bin
sudo cp ~/gpsctl/etc/gpsctl.conf /etc/gpsctl.conf

Individual settings can be tweaked in /etc/gpsctl.conf.

To set port speed to 115200 baud, enable Galileo satellites, set stationary mode, tweak antenna delays, PPS timing, etc:

/usr/local/bin/gpsctl -a -B 115200 --configure_for_timing -vv

To reset the device to its defaults

/usr/local/bin/gpsctl -a --reset -vv

To view info:

/usr/local/bin/gpsctl -a -Q satellites

/usr/local/bin/gpsctl -a -Q config

/usr/local/bin/gpsctl -a -Q fix

Note that these commands can only be run when gpsd / ntp are not using /dev/ttyAMA0

Example /etc/gpsctl.conf which configures the U-blox M8Q for this environment:

    
#
# example gpsctl.conf which enables Galileo as in --galileo parameter
#
[gpsctl]
port = /dev/ttyAMA0
# sync method: ASCII = 1, NMEA = 2, UBX = 3
sync method = 3
verbosity = 0
[NMEA]
enabled = true
version = 41
GGA = off
GLL = off
GSA = off
GSV = off
RMC = on
VTG = off
GRS = off
GST = off
ZDA = on
[GPS]
enabled = yes
minimum channels=8
maximum channels=16
[SBAS]
enabled = no
minimum channels=1
maximum channels=3
[Galileo]
enabled = yes
minimum channels=4
maximum channels=8
[Beidou]enabled = no
minimum channels=8
maximum channels=16
[IMES]
enabled = no
minimum channels=0
maximum channels=8
[QZSS]
enabled = no
minimum channels=0
maximum channels=3
[GLONASS]
enabled = yes
minimum channels=8
maximum channels=14
[Navigation Engine]
# Dynamic model: Portable = 0, Stationary = 2, Pedestrian = 3, Automotive = 4,
# Sea = 5, Air1G = 6, Air2G = 7, Air4G = 8, Watch = 9
Dynamic model = 2
# Fix mode: 2D only = 1, 3D only = 2, auto 2D/3D = 3
Fix mode = 3
Fixed altitude (2D) = 0.00 meters
Fixed altitude variance (2D) = 1.0000 meters^2
Minimum elevation = 5 degrees
Position DoP mask = 10.0
Time DoP mask = 10.0
Position accuracy mask = 100 meters
Time accuracy mask = 300 meters
Static hold threshold = 0 cm/s
Dynamic GNSS timeout = 60 seconds
Threshold above C/No = 0 satellites
C/No threshold = 0 dBHz
Static hold max distance = 0 meters
# UTC Standard: AutoUTC = 0, USNO_UTC = 3, GLONASS_UTC = 6, BeiDou_UTC = 7
UTC standard = 3
[Time Pulse]
# the nanoseconds / microseconds after the numbers are just reminders,
# they don't mean anything to the config parserAntenna cable delay = 56 nanoseconds
RF group delay = 20 ns
Unlocked pulse period = 1000000 microseconds
Unlocked pulse length = 0
Locked pulse period = 1000000 microseconds
Locked pulse length = 500000 microseconds
User configurable delay = 0

To run gpsctl at system startup,

sudo cp ~/gpsctl/etc/systemd/system/ublox-init.service /etc/systemd/system/ublox-init.service

or

nano /etc/systemd/system/ublox-init.service

and add the following contents

[Unit]
Description=u-blox initialisation
Before=gpsd.service
Before=ntp.service
[Service]
Type=oneshot
Environment="PARAMS=-q -a -B 115200 --configure_for_timing"
EnvironmentFile=-/etc/default/gpsctl
ExecStart=/bin/bash '/usr/local/bin/gpsctl ${PARAMS}'
[Install]
WantedBy=multi-user.target

This will enable Galileo satellites, set stationary mode, configure comms at 115200 baud, and restrict NMEA output to RMC and ZDA records.

To use the default baud rate of 9600, create /etc/default/gpsctl with the contents

PARAMS=-q --configure-for-timing

Then

sudo systemctl enable ublox-init.service

sudo systemctl daemon-reload

Verifying that PPS Is Working

Ensure the GPS has a lock and the Green PPS LED on the Uputronics Pi+ GPS Expansion Board is blinking once a second.

lsmod | grep pps

Output should be similar to :

pps_gpio                3089  1
pps_core                8606  4 pps_gpio

dmesg | grep pps

Output should be similar to :

[    2.735586] pps_core: LinuxPPS API ver. 1 registered
[ 2.738121] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 2.763842] pps pps0: new PPS source pps@12.-1
[ 2.766361] pps pps0: Registered IRQ 169 as PPS source

This indicates that the PPS Module is loaded.

sudo ppstest /dev/pps0

Output should be similar to:

trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1418933982.998042450, sequence: 970 - clear  0.000000000, sequence: 0
source 0 - assert 1418933983.998045441, sequence: 971 - clear  0.000000000, sequence: 0

(Press CTRL+C to quit). This indicates that the PPS Module is working.

You need to pick a few local NTP servers to use. The easiest way to do this is pick your region:

https://support.ntp.org/bin/view/Servers/NTPPoolServers

Select your region then you get a list of the country servers. E.g for the UK its uk.pool.ntp.org.

Type:

dig uk.pool.ntp.org

You will get four IP’s back:

;; &lt;&lt;&gt;&gt; DiG 9.10.3-P4-Raspbian &lt;&lt;&gt;&gt; +answer uk.pool.ntp.org
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 51647
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
;; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;uk.pool.ntp.org. IN A
;; ANSWER SECTION:
uk.pool.ntp.org. 24 IN A 193.150.34.2
uk.pool.ntp.org. 24 IN A 176.58.109.199
uk.pool.ntp.org. 24 IN A 195.195.221.100
uk.pool.ntp.org. 24 IN A 185.53.93.157

sudo nano /etc/ntpsec/ntp.conf

Amend the start of ntp.conf to look like this

## nts enable

# You must create /var/log/ntpsec (owned by ntpsec:ntpsec) to enable logging.
#statsdir /var/log/ntpsec/
#statistics loopstats peerstats clockstats
#filegen loopstats file loopstats type day enable
#filegen peerstats file peerstats type day enable
#filegen clockstats file clockstats type day enable

# This should be maxclock 7, but the pool entries count towards maxclock.
#tos maxclock 11

# Comment this out if you have a refclock and want it to be able to discipline
# the clock by itself (e.g. if the system is not connected to the network).
#tos minclock 4 minsane 3

# Specify one or more NTP servers.

# Public NTP servers supporting Network Time Security:
# server time.cloudflare.com nts

#--------------------------
# PPS Driver
refclock pps unit 0 minpoll 4 maxpoll 4 ppspath /dev/pps0
#flag3 Controls the kernel PPS discipline: 0 for disable (default), 1 for enable.
#time1 PPS time offset
# use local clock at stratum 10 if no others available
tos orphan 10
tos mindist 0.002
# NMEA driver (/dev/gps0 and /dev/gpspps0)
# process $GPZDA only (mode 8)
refclock nmea prefer minpoll 4 maxpoll 4 mode 8 stratum 2 baud 115200 time2 0.060 path /dev/ttyAMA0
#time2 NMEA time offset
#mode bit 0 - process $GPMRC (value = 1)
# bit 1 - process $GPGGA (value = 2)
# bit 2 - process $GPGLL (value = 4)
# bit 4 - process $GPZDA or $GPZDG (value = 8)
# mode 9 = process only xxRMC and xxZDA NMEA records
#---------------------------
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <https://www.pool.ntp.org/join.html>
server 0.uk.pool.ntp.org iburst
server 1.uk.pool.ntp.org iburst
server 2.uk.pool.ntp.org iburst
#pool 0.debian.pool.ntp.org iburst
#pool 1.debian.pool.ntp.org iburst
#pool 2.debian.pool.ntp.org iburst
#pool 3.debian.pool.ntp.org iburst

(Note that I used “time2 0.060 …” to adjust the GPS time according to the other NTP servers. You can try some higher/lower values to have the offset of your NMEA driver compared to the offsets of those other NTP servers very small. We’ll cover that later.)

We want to make sure that the second reported by the NMEA driver is the second that the last PPS pulse referred to, so it needs to be within a few hundred milliseconds of the correct time so that NTP does the right thing. This is why it’s desirable to mark another NTP server as “prefer” to help the NTP daemon get it right.

Note that we use NTP PPS discipline, not kernel PPS (which in my testing results in warnings from ntp that kernel PPS discipline isn’t supported).

Comment out all the pool lines.

Add the servers from the dig command, or use servers of your choice) with the top one saying prefer on it (example only, don't all use these IP addresses):

server 176.58.109.199 iburst prefer
server 195.195.221.100 iburst
server 185.53.193.157 iburst
# By default, exchange time with everybody, but don't allow configuration.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict -6 ::1
restrict 10.0.0.0 mask 255.0.0.0
restrict 172.16.0.0 mask 255.240.0.0
restrict 192.168.0.0 mask 255.255.0.0
# Drift file etc.
driftfile /var/lib/ntpsec/ntp.drift

Note You MUST add a preferred server or PPS doesn’t work.

Save and close nano.

sudo systemctl restart ntpsec.service

After a few minutes run

ntpq –p

If you get oPPS(0) this indicates source selected, Pulse Per Second (PPS) used and everything is working.

pi@ntp2:~ $ ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
o127.127.22.0 .PPS. 0 l 6 16 377 0.000 -0.002 0.001
*127.127.20.0 .GPS. 1 l 5 16 377 0.000 -0.323 1.052
+195.195.221.100 .GPS. 1 u 49 64 377 18.307 0.190 0.127
+193.150.34.2 85.199.214.101 2 u 56 64 377 7.953 0.067 0.114

And then

pi@ntp2:~ $ ntpq -csysinfo
associd=0 status=0115 leap_none, sync_pps, 1 event, clock_sync,
system peer: PPS(0)
system peer mode: client
leap indicator: 00
stratum: 1
log2 precision: -21
root delay: 0.000
root dispersion: 2.015
reference ID: PPS
reference time: e0601d81.248b4b37 Tue, Apr 16 2019 10:23:13.142
system jitter: 0.000477
clock jitter: 0.003
clock wander: 0.001
broadcast delay: -50.000
symm. auth. delay: 0.000

If you aren’t seeing the settings its possible the NTP server is picking up the NTP information via DHCP which is overriding your settings above. Do this :

rm /etc/dhcp/dhclient-exit-hooks.d/ntp

rm /var/lib/ntp/ntp.conf.dhcp

At this point you have a NTP server which will use an external time source and use your local PPS to discipline it.

Note: Once you've configured ntp to use your GPS source, you'll need to stop the ntp service before running gpsctl, otherwise they'll both be trying to access ttyAMA0 at the same time.

GPS Offset Tuning

Your PPS time is going to be more accurate than NTP pool servers. Unless you have specialised equipment and local LAN GPS / DCF77 / MSF PPS sources to calibrate against, it’s not really possible to determine the appropriate PPS offset. It’s likely to be in the order of a few microseconds at most ( htt://lists.ntp.org/pipermail/questions/2011-September/030338.html ).

The PPS is a precise signal with around 10 ns of jitter. On the other hand, the offset of the GPS serial data output (GPSD) generally will have much more variation than the PPS because of the variables involved in sending data over an asynchronous serial port. However, the GPSD offset can be reduced somewhat by adjusting the GPSD reference clock fudge parameter time2 in the ntp.conf file.

The initial value used in our ntp.conf file is 60ms (0.060s). This means that the GPS ZDA packet arrives in the NTP daemon approximately 60ms after the PPS pulse corresponding to it. On our test device, we see a jitter of under 3ms on the GPS_NMEA clock.

We might need to adjust the GPS ntp.conf time2 (offset) value to get the NMEA time offset low enough that the offset isn’t great enough to confuse NTP as to which second the PPS pulse referred to.

This option can be used to compensate for a constant error. The specified offset (in seconds) is applied to all samples produced by the reference clock. The default is 0.000s.

Start with these ntp.conf settings, they should get you close enough to get everything working properly:

sudo nano /etc/ntpsec/ntp.conf

# PPS Driver
refclock pps unit 0 minpoll 4 maxpoll 4 ppspath /dev/pps0
#flag3 Controls the kernel PPS discipline: 0 for disable (default), 1 for enable.
#time1 PPS time offset
# use local clock at stratum 10 if no others available
tos orphan 10
tos mindist 0.002
# NMEA driver (/dev/gps0 and /dev/gpspps0)
refclock nmea prefer minpoll 4 maxpoll 4 mode 8 stratum 2 baud 115200 time2 0.060 path /dev/ttyAMA0
#flag1 Disable PPS signal processing if 0 (default); enable PPS signal processing if 1.
#flag2 If PPS signal processing is enabled, capture the pulse on the rising edge if 0 (default); capture on the falling edge if 1.
#flag3 If PPS signal processing is enabled, use the ntpd clock discipline if 0 (default); use the kernel discipline if 1.
#time1 PPS time offset
#time2 NMEA time offset
#mode bit 0 - process $GPMRC (value = 1)
# bit 1 - process $GPGGA (value = 2)
# bit 2 - process $GPGLL (value = 4)
# bit 4 - process $GPZDA or $GPZDG (value = 8)
# mode 9 = process only xxRMC and xxZDA NMEA records at 115200 baud
#---------------------------
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <https://www.pool.ntp.org/join.html>
server 0.uk.pool.ntp.org iburst
server 1.uk.pool.ntp.org iburst
server 2.uk.pool.ntp.org iburst

To calculate the GPS offset we must disable GPS by placing a noselect in the ntp.conf "refclock nmea" line. We'll run the time server for a few hours and then compare the ntpq -p GPS offset to the average public time server offset. For accurate tuning use a bunch of known-good Stratum 1 servers in ntp.conf. Remember to remove the "noselect" after tweaking.

sudo systemctl stop ntpsec.service

sudo rm –f /var/log/ntpsec/*

sudo systemctl start ntpsec.service

Start ntpd and let it run for at least four hours. Periodically check progress with "ntpq -p" and wait until change has settled out.

Look for the row with GPS_NMEA(0) and refid GPSD. The offset values probably will be different for each

query. Note that the ntpq offsets are in milliseconds, but the peerstats file offsets and NTP’s time2 parameter are in seconds.

Calculate the average GPS offset (in seconds) using this script:

sudo nano ~/nmeaoffset

#!/bin/sh
#
# Generate an estimate of your GPS's offset from a peerstats file
#

awk '
/NMEA/ { sum -= $5 ; cnt++; }
END { printf("%.6f\n", sum / cnt); }
' </var/log/ntpsec/peerstats

Then sudo chmod +x ./nmeaoffset

Run by typing

pi@ntp2:~ $ ./nmeaoffset

-0.001212

That’s within a few milliseconds, close enough for anyone as PPS is going to do its magic to give us accuracy within a few microseconds.

Adjust the "time2" value for the GPS source of your ntp.conf by adding the average offset from above.

sudo systemctl stop ntpsec.service

sudo rm –f /var/log/ntpsec/*

sudo systemctl start ntpsec.service

Repeat the procedure above until -5ms < offset < +5ms (or under 10ms, if that’s OK with you). When you’re done, remove the noselect from the "refclock nmea" line in ntp.conf.

If you decide to recalculate the average offset using the above procedures, wait at least another day or two.

Avoid unnecessarily changing the time2 value. A typical value for the Adafruit GPSD driver is +0.534 s when using its default 4800 baud interface and in the HAB Supplies/Uputronics GPSD it’s +0.060 s when using 115200 baud and the gpsctl --config_for_timing tweaks.

Automatically updating GPS leap seconds

These semi-annual changes will be made no later than 1 June and 1 December of each year to indicate what action (if any) is to be taken on 30th June and 31st December, respectively.

In Buster, the default ntp.conf points LeapFile at the tzdata-provided leap-seconds.list. That's fine, as long as you auto apply system updates via cron or manually on a regular basis.

Automatically updating Raspbian

To automatically update Raspbian on a schedule, follow these steps:

sudo apt install unattended-upgrades mailutils

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Edit to include these sources (remove all others)

    
"origin=Raspbian,codename=${distro_codename},label=Raspbian";
"origin=Raspberry Pi Foundation,codename=${distro_codename},label=Raspberry Pi Foundation";

and add

Unattended-Upgrade::Automatic-Reboot "true";

sudo nano /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Verbose "1";
APT::Periodic::AutocleanInterval "7";

sudo dpkg-reconfigure --priority=low unattended-upgrades

Set the systemctl apt timer to fire at 2AM daily

sudo systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
# OnCalendar=DayOfWeek Year-Month-Day Hour:Minute:Second
# see https://wiki.archlinux.org/index.php/Systemd/Timers
OnCalendar=02:00
RandomizedDelaySec=0

To test,

sudo unattended-upgrade -d -v --dry-run

Static IP and Hostname

If you want to fix your LAN IP you do it by running nmtui to configure your network.

Amend your hostname via raspi-config.

Migrating Nagios Configuration from Nagmin to Check_MK's WATO

When I first set up a Nagios server, many years ago, in the days of Nagios 1.x, the best configuration tool I could find was Fred Reimers' Nagmin. That has since turned into abandonware, but there is a fork, NagminV, under development.

I'd patched Nagmin to support Nagios 2.x and 3.x, and added a few fields to its database, but it was still buggy and quirky.

So, after I'd installed Mathias Kettner's Check_MK for its livestatus broker module for use with PNP4Nagios, I started investigating Check_MK's broader features.

What soon caught my eye was Check_MK's use of rulesets based on host tags. The temptation of editing text files (python scripts in disguise) was too great for me, so I started converting my Nagios service checks into Check_MK format.

These are very rough notes, proceed with caution. Back up everything first!!!

First thing to do was get Check_MK's agent on all our hosts.

Then, to list all of them in /etc/check_mk/main.mk:

# don't generate host config yet

# comment this out when Nagmin is decommissioned

generate_hostconf = False

all_hosts = [

host1,

host2,

]

Then I added the obvious tags, win, linux, etc, and created config files for legacy checks in /etc/check_mk/conf.d, until eventually all service checks were defined in Check_MK and not in Nagmin.

After each check was migrated to Check_MK, I'd run

check_mk -U

nagios -v /etc/nagios/nagios.cfg

The first command generates a new Nagios config in /etc/nagios/check_mk.d/check_mk_objects.cfg

The second validates the resulting config. Here I'd find duplicate definitions, reminding me to delete them from Nagmin.

So, after a while plodding away at this - in my case this meant over a year of coexistence - all that was left in Nagmin was hosts, host groups, contacts, contact groups, and timeperiods.

Time to abandon Nagmin and get serious.

Do not, under any circumstances, try to generate a Nagios config using Nagmin again. It will always fail!

Comment out the Command.cfg include in nagios.cfg

#cfg_file=/etc/nagios/Command.cfg

Try validating the nagios config again, with nagios -v. If it fails, you've forgotten to define some commands as legacy checks

Rinse, repeat until you've got it right.

Check the contents of /etc/nagios/Services.cfg - if it contains any service definitions, you've forgotten something.

Do the same with all the service-related .cfg files; if things fail, change your config to use Check_MK's default service templates.

Get your custom Time Periods into WATO - check_mk has a default, hidden timeperiod 24X7 (Nagmin's is 24x7, case matters), and comment out TimePeriods.cfg in nagios.cfg. Revalidate.

And so on till you're left only with hosts, host groups, contacts, and contact groups from the original Nagmin.

So now was the time to take the leap and use Check_MK's WATO to configure nagios.

I created contacts and contact groups manually in WATO. There's no conflict with your existing Nagios config until you associate a contact group with hosts and/or services in WATO. That was the last thing I did.

Now comes the fun bit.

We need to import our hosts into WATO.

Run the attached hosts.py which will generate a list of hosts in the format hostname;alias;parents;ipaddress

python hosts.py >wato.csv

Our import script requires a file containing

wato folder;hostname;alias;parents;ipaddress;tags

where tags is a list of check_mk tags separated by |

So, we'll have to manually edit it. I didn't use WATO folders so just preceded each line with a ;

Tags are the fun bits. By now you may have been using them in your check_mk config. WATO comes with some predefined ones, which we need to list in our import file (wato.csv).

You'll need one each of the 'agent', 'criticality', and 'networking' tags. As well as your custom tags, which should also be entered into WATO.

Download and edit my_wato_import.py adding your tag definitions into tagz, run:

python my_wato_import.py wato.csv

and watch the output scroll by. If there are any errors reported, then you've forgotten to add a tag definition to tagz, or misspelt a tag in wato.csv. Rinse and repeat until all's well.

Set testing = False near the top of the script, and run again.

Comment out generate_hostconf = False in your main.mk, and check_mk -U

Comment out Host.cfg in /etc/nagios/nagios.cfg

Validate your nagios configuration. It should be OK.

A look in WATO will show all your hosts, with appropriate tags.

Hidden away in /usr/share/doc/check_mk/treasures is a script wato_host_svc_groups.py

I ran this against my original Nagmin Hosts.cfg file, which produced output which was easily massaged into the form needed for WATO. I could have amended the script to produce output of the following form, but a few regular-expression search and replaces got me there quickly enough.

host_groups = [

( 'group1', []. ['server1', 'server2']),

( 'group2', []. ['server3', 'server3']),

]

Place that code into /etc/check_mk/conf.d/wato/rules.mk and create the host groups (with descriptions) in WATO. Before applying changes in WATO, run check_mk -U

Comment out the Hostgroup.cfg include in nagios.cfg, and validate config once more.

#cfg_file=/etc/nagios/HostGroup.cfg

Now, if you've done everything properly, the Nagios config validation will succeed, and on a restart of Nagios your host groups will be there as before.

That just leaves Contacts, Contact Groups and notifications.

I'll leave that as an exercise for the reader. Hint: don't try any of the above until you've figured out how to apply contact groups to hosts and services.

And you'll also need to adjust host and service check intervals and retries in WATO too, otherwise everything gets polled every minute, which probably isn't what you want.

Posted by Phil at 9:59 PM
Edited on: Wednesday, April 01, 2015 9:24 PM
Categories: IT, Software

Installing Exchange 2010 Service Pack 1

I had a few hassles installing Microsoft Exchange 2010 SP1 today.

I guess if things ran smoothly all the time, I wouldn't have a job, but....

All worked perfectly yesterday on our virtualised test setup, but in real life, it is never that simple.

The Exchange Team have provided a handy list of prerequisites and known problems with SP1 in their "Exchange 2010 SP1 FAQ and Known Issues" blog post. Read it carefully.

On one of our CAS servers, I got an error warning me of a failed installation. Following the instructions in "How does Exchange 2007 setup know to resume a failed setup?", I determined that there had been a previous error installing the Admin Tools. Clobbering the registry entry (the one with the install value) enabled the install to proceed.

After the install, an attempt to start the Management Console failed. Turning again to the MS Exchange Team's blog provided the solution, in "Troubleshooting Exchange 2010 Management Tools startup issues". In our case, the Service Pack install had stripped the WSMan module entry from the global modules section of C:\Windows\System32\Inetsrv\config\ApplicationHost.config. Adding it back in after the SP1 install and restarting the default web site, that issue was solved. We had to do this on all our Exchange 2010 boxes.

Next error came moving the active database from one DAG member server to another. Corrupt index. After a quick Google I discovered that restarting the Windows Search (Exchange) service on the Mailbox servers was enough to fix this issue. I guess a reboot after the Service Pack install would have had the same effect, even though the Service Pack install didn't request one.

It is all working fine now (fingers crossed).

Posted by Phil at 9:07 PM
Edited on: Thursday, September 02, 2010 9:50 PM
Categories: Comment, Software

Active Virus Shield

AOL is widely regarded as an evil empire, second only to Microsoft, but this week they redeemed themselves somewhat by releasing a free Windows antivirus program, Active Virus Shield, based on Kapersky Labs' Personal Antivirus.

I've given it a whirl here and so far I'm impressed. Kapersky's Virus scanning engine is one of the best, and they are renowned for the speed of their responses to new malware. So far I'm seeing several pattern updates a day, which is what I'd expect from any proper antivirus vendor.

It's staying on my PC for a while. It may yet replace Alwil's Avast! here, but I have a certain irrational fondness for products whose support forums give the users direct contact with the developers.

Postscript, September 5th

Independent research confirms Active Virus Shield as being one of the best antivirus products. There's commentary and discussion over at cybernetnews.com.

Posted by Phil at 11:00 PM
Edited on: Tuesday, September 05, 2006 11:56 AM
Categories: Computer Security, Software

Mentioned in Dispatches

Thanks Blake, for your kind mention in your weblog. I'm honoured!

Posted by Phil at 9:50 PM
Edited on: Thursday, June 08, 2006 9:52 PM
Categories: Software, Waffle

50,000,000

Congratulations to the Mozilla Firefox team on their 50 millionth download.

I was a bit slow with the screenshot there :-)

Posted by Phil at 12:20 PM
Edited on: Wednesday, April 01, 2015 9:59 PM
Categories: Software

I'm a Firefoxer

My humblest apologies to Prodigy...

If you're bored you might want to check out my other blog at SpreadFirefox.

Posted by Phil at 9:36 PM
Edited on: Saturday, April 23, 2005 10:20 AM
Categories: Software

Favourite Software

Just because you asked (or didn't), here's my favourite software, in no particular order:

Mozilla, the most standards-compliant web browser out there.

Audacity, for sound file editing.

Gnucleus, a peer-to-peer filesharing client completely free of spyware.

The GIMP, the graphics program which doesn't cost an arm and a leg. Installing it on windows is a bit involved, however.

OpenOffice, who needs Microsoft's bloatware when this will do nicely for most people?

All the above are open source and free to use wherever you like.

ConText, a lean and mean text editor. Free, no source included.

IrfanView, a great image viewer. Free for personal use, no source included.

Spybot Search and Destroy, the best SpyWare/AdWare cleaner.

So tell me, who needs to pay for software?

Posted by Phil at 9:03 PM
Edited on: Saturday, April 23, 2005 10:22 AM
Categories: Software

Phil's Occasional Blog

Thursday, May 27, 2021

Converting CentOS 8 Stream to AlmaLinux 8.4

Saturday, August 17, 2019

About to update to RedHat Enterprise Linux or CentOS 7.7? Beware!

Saturday, April 27, 2019

A Raspberry Pi Stratum 1 NTP Server

Overview

A Note About Accuracy

Required Components

Prerequisite Settings

Enabling Galileo Satellites and Setting Stationary Mode

Verifying that PPS Is Working

GPS Offset Tuning

Automatically updating GPS leap seconds

Automatically updating Raspbian

Static IP and Hostname

Further Reading

Monday, April 29, 2013

Migrating Nagios Configuration from Nagmin to Check_MK's WATO

Thursday, September 02, 2010

Installing Exchange 2010 Service Pack 1

Friday, August 11, 2006

Active Virus Shield

Thursday, June 08, 2006

Mentioned in Dispatches

Saturday, April 30, 2005

50,000,000

Monday, September 20, 2004

I'm a Firefoxer

Tuesday, January 14, 2003

Favourite Software