MidnightBSD Developer Journal

MidnightBSD 3.1 includes mport 2.4.3

mport clean now removes temporary files that might get left behind by other operations

mport clean now removes left over /var/db/mport/infrastructure/* folders that might get left behind prior to a fix for mtree files last year. (mostly for older systems)

mport's internal rmtree functionality has been modified to use native C routines rather than executing rm -r as a system command. (Please report any issues with removing files in packages on delete with this) This is slightly faster with very large packages. (0.03 seconds or so)

mport list updates will now give you better information about why a package is not found in the index. If the package is listed in the MOVED file in mports repository, it will tell you if it's removed/expired or moved to another location.

Now that MOVED file contents are part of the index, we can start doing smarter updates in the future. The first package build to include this data is the latest amd64 3.1 build that is likely going to be used for the upcoming midnightbsd 3.1 release.

We've tagged 2.4.4 in git with additional improvements

• Adds mport config list to display all configured values

• Fixes a bug with non-interactive console messages when downloading a file. (Uses percent and cuts down on output)

MidnightBSD 3.1.0 has been released.  You can learn more about it in the release notes on github https://github.com/MidnightBSD/src/releases/tag/3.1.0 or the MidnightBSD website at https://www.midnightbsd.org/notes/

I started working on some changes for Magus to parse the MOVED file. In MidnightBSD, a lot of tools don't do much with that file so it's never been consistently managed. 

The idea is that we can index it with each magus run and then provide the data in the index file generated for packages.  This would provide hints about packages that were really removed vs renamed.  We could then install the renamed/moved package on upgrades and when using the mport list command, it could more accurately guess if packages were removed or just unavailable. 

This solves a problem that's bugged me for awhile with mport.  Of course, we're only working on the first step now.  

The steps for this are:

1. add it to the magus indexer.  (needs testing)

2. Export the data during a magus bless from the new MOVED table in the postgresql database to a table in the generated sqlite index files for mport to use.  (need to update magus bless utility)

3. Modify mport package manager to read the data from the table in the index 

4. Update the mport list command to display more accurate data about packages that have expired, will expire, or are deleted/removed

5. Modify the mport info command to display information about package expiry etc 

6. Modify the upgrade command (and possibly update) to make more intelligent decisions about package renames and possibly ask a question on upgrade/update/install paths about packages that are deprecated.

The Ravenports folks are looking into supporting MidnightBSD so there'd be an additional source of packages for the OS available.

We've done some work on the firstboot script in Current to support this experimental bootstrap/integration.

https://ravenports.com

This is certainly not final yet, but looking promising. There are a number of packages available via Ravenports that aren't available in mports.

For example: recent firefox releases!

We won't be discontinuing mports, and consider this an additional package manager / repository. However, we will try to make it easy to use on a fresh install and support it as much as we can.

Midori 9.x, the default browser in MidnightBSD, went EOL.? The 10.x version of the browser is electron based and doesn't support many operating systems.? That put us in a pickle.? We need a default browser and webkit browsers have been the most consistent with working on MidnightBSD.? Therefore, we forked Midori 9 as Raphael.? It's on github at https://github.com/midnightbsd/raphael/

All browsers don't have to be chromium based.? That's a bad thing!?

We've also made progress on the firefox-esr port and getting that to build.? It's close to working but still some hurdles.? If we can get that fixed, we have two?decent options.

If you're interested in helping with the Raphael browser or fixing the firefox ports, we are open to pull requests!

Here's a list of some short term goals we're hoping to get done in the next few months:

• mports: fix firefox-esr port

• Add llvm12 or 13 to mports and update mesa libraries using it

• Update perl in current?

• Look into updating CPAN included in stable/3.0 perl to fix a CVE

• Import bug fixes / improvements from FreeBSD 12-stable branch from September-December 2022 into midnightbsd current.?

• Look into updating the mport manager to work with newer libmport versions

First, we've had several contributions over the last few weeks.? Thanks, new contributors!??

In current, we imported 2?months' worth of bug fixes from FreeBSD 12-stable.? ?We've also made several changes to the firstboot script.? For folks running current, please test the new firstboot script changes.? If you want to run it, just delete /etc/fbreceipt and run /etc/rc.d/firstboot start (after installing the latest version)?

We've also had a few server outages lately. Here's what's going on:?
1. We had a botched upgrade to our primary virtualization server.? This was running VMware ESXi.? A CPU upgrade killed a motherboard as the CPU wasn't actually supported, as I misread the support list.? The motherboard was replaced.? Additionally, some memory modules had failed in that.? Those were replaced.? We are in the process of migrating VMs to a bhyve environment.??

2. We upgraded our primary network switch.? The new switch is an aruba instant on 1960?12XGT and replaced an aging buffalo bs208 8 port nbase-t switch.? Since the new switch doesn't have 2.5G support, we had to upgrade the network card in our file server to an intel x540.? (it was using a realtek dragon 2.5G)?

3. While upgrading the network switch, we uncovered a thermal issue with another server.? We'll have to bring that down to replace some fans soon.? It's our other virtualization server used primarily for package builds.

project update:

We're still restoring VMs from our recent server issue. Some services are back up, but we still have a few like Jenkins, OpenGrok, and build nodes for our package cluster to restore.

We've been running package builds on a single server lately. The latest amd64 run has had a few issues that we're working through.

mport package manager has received several updates in recent weeks. It now supports an audit command that lets you check for CVEs against a copy of the NVD data.
Usage:
mport audit
mport audit -r
mport -q audit

The first version prints a list of all CVEs with descriptions for each package.

The second includes a list of packages that depend on this vulnerable port so you can also update those.

The third doesn't give details about the vulnerabilities and just prints a list of vulnerable packages with package name and package number using the "global" -q aka quiet flag.

This isn't included in MidnightBSD src git yet as we're working through a few bugs. You can check it out and try it now though git clone https://github.com/midnightbsd/mport.git

Check out the episode here https://www.bsdnow.tv/504

First, identify the failed disk. You can run zpool list to identify the failed drive.

zpool status

If you were clever, you named your disks with meaningful labels. If not, it might be trial and error. If it's ada0, you know it's in the first SATA port, for instance.

Remove the old disk. Confirm the disk is gone in the BIOS after booting the system.

Now put in the new disk.

It's recommended you boot into single-user mode. (see the option on the midnightbsd boot menu)

Now verify the OS detected the new drive.

camcontrol devlist

You should see the new drive. If you placed it in the same sata port, it is likely the same name. For our example, ada0

gpart create -s gpt ada0

gpart show

gpart add -a 4k -t mnbsd-zfs -l mydisklabel ada0

gpart show

Now if you run zpool status again you should see one drive marked as offline with a long number identifier rather than a label or device name (ada0)

zpool replace mypoolname longnumberfromzpoolstatus gpt/mydisklabel

If you didn't use the label flag above, replace that last part with ada0

This will start the resilvering process, which can take several hours. You can run zpool status to check the status of this process.

While you can boot the system at this point, and it will work, it will run slowly. If possible, let it resilver in single-user mode. It will go a lot faster.

Note there is a better guide on this here:?https://dan.langille.org/2015/08/03/replacing-a-failed-drive-in-zfs-on-freebsd/?The big difference is the partition type above (mnbsd-zfs vs freebsd-zfs)

If you're an end user and you want to know what software is available for MidnightBSD, the best place to go is the?app store. You can also use the mport search command to look for available packages, or in mports use make search key=

For software developers, there are several ways to get the data for consumption. Our package build cluster software has several endpoints available including:

https://www.midnightbsd.org/magus/api/latest

https://www.midnightbsd.org/magus/api/run-ports-list?run=328&status=pass

The latest endpoint gives you a list of packages available in the most recent magus runs that have been blessed (published). The other one lets you look at a specific run and filter on status.

There are multiple APIs available in the app store as well. Consult the midnightbsd-app-store repository on GitHub for details on those.

Finally, there are the actual sqlite3 databases generated by blessed package builds per arch and os-release.
http://index.mport.midnightbsd.org/amd64/3.0/index.db.bz2

We've updated the security advisory api recently and it now has a new endpoint that lets you lookup by CPE.?

https://sec.midnightbsd.org/api/cpe/partial-match?cpe=cpe:2.3:a:openssl:openssl:1.0.1d:::::::*

This URL will lookup CVEs that may impact openssl 1.0.1d?

We're considering possibly using this with mport to provide security notifications via a new command.? Right now we have the security-advisory-client in mports that hits different endpoints within this API to provide auditing data.

We've had a bad hardware upgrade and some of our infrastructure is down.? This includes the midnightbsd app store, jenkins and portsnap updates.? We also use this system for a number of magus package builds.?

A replacement motherboard should arrive soon to fix this.

3.0.1 was tagged in our git repository and we've started building ISOs for it.? It includes several security updates such as OpenSSL 1.1.1t, doas 6.3p9, a fix for a telnetd vulnerability.? It also includes some cleanup work on rc.d scripts, fixes for periodic scripts that were incorrect around ntpd and restoring msearch/mport db backups.? ?We also updated mport to 2.2.9 which fixed the mport mirror list command.

3.0 i386 packages haven't been built for the release yet.  There are some older ones available but without the full desktop environment.  We'll be building those this weekend. 

We're uploading some ISOs but still need to do final testing on them.

First, the 3.0 release has been blocked for some time due to a few key mports that just refused to build.  Today, we got LLVM 9/10/11 working on MidnightBSD 3.0.   This unblocks us to work on the remaining desktop ports to prepare for the release. 

We've published updated packages for 2.2 amd64, 2.2 i386, and 3.0 i386 over the weekend.  A new package build for 3.0 amd64 is in progress. 

Several bugs have been reported with the mport package manager stemming from some changes made back in December.  Users on 3.0 or 2.2 from git are impacted.  If you have 2.2.7 or older installed, it's not a problem.  We have a workaround committed in 2.2 stable branch to allow folks to install packages.  We're working on a better fix for the 3.0 release cycle. 

We fixed a recently reported security issue in the magus package cluster web interface.  Thanks for the report!

The new tentative plan is to release MidnightBSD 3.0 by the end of April.  This gives us time to get mports in shape and do additional testing.  

We've added some documentation on settings to use when creating virtual machines of MidnightBSD guests on the MidnightBSD wiki on Github (under src). 

Known issues: 
Some options don't appear to be working on some mports.  
Perl ports on some 3.0 installs aren't working properly.  
OpenJDK 8 is broken on 3.0
Avahi and boost broken on 3.0 and the biggest blockers to release at the moment.

We released a minor update to MidnightBSD which contains the following changes:

tzdata 2022g

less 551

ISOs are on the main site and starting to copy to mirrors.

The 3.0 stable branch is pretty much ready.  We may update sqlite3 yet. 

The delay in the release has been due to issues with several mports.  We're still working through those problems, but one can't ship a desktop OS without a working desktop.

We have our first snapshot available on the FTP. it has a lot of problems and we're working on squashing bugs right now.

We're waiting on packages, but the ISOs are ready.