MidnightBSD Developer Journal

I've found myself sidetracked lately with server upgrades, a new laptop I'm trying to get working with MidnightBSD, and other fun things.

Several new happenings in MidnightBSD:

xz 5.0.3 in CURRENT

improvements to mport, msearch and batt(1).

In mports, apache 2.2.21 and xz 5.0.3.    

The last magus run is looking a lot better.  Here's the results from i386 0.4-CURRENT:

The mport command now supports a new "download" command.  This allows a user to download a package independant of installation.  It's saved to the normal package storage path inside of /var. 

This is useful when you know you're going to lose network connectivity and want to fetch several packages for installation later.  

This past weekend, I've focused on updating ports.  www/linux-firefox-devel (3.6.21), most multimedia ports, and gnustep related ports have been updated. 

I've also currently running package builds for i386 and amd64. 

I just blessed the magus run for i386.  That means there are a new set of packages on the FTP server for i386.  I also put out a new ports index for this release tied to the mport tools sqlite3 database.

I've got a dedicated i386 system for package builds again.  It takes a few days to do a run, but it's better than nothing.  It's a Pentium D 805 w/ 2GB RAM. 

MidnightBSD mports now includes the latest Apache release.  This fixes a DOS attack using range headers.  Users are encouraged to upgrade. 

I've just updated the opera port to 11.51.  This version includes some security fixes.  

MidnightBSD has been lagging some other systems with respect to web browsers.  Tonight, I updated Opera, Flash 10 (linux emulation), Firefox 6 and Thunderbird 6 (linux) ports and the nspluginwrapper port.  It's now possible to play flash video on several browsers in current (again) and access youtube.

You must be running 0.4-CURRENT to use some of these.  

mport perl updated

Perl 5.14.1 is now available in mports.  While I've considered updating base, 5.14.2 is scheduled for release next month.  If 0.4 is not ready by the time the release occurs, I'll update it in base, otherwise it will be included in 0.5.

Base Perl issues 

Users having difficulty upgrading from MidnightBSD 0.3 to 0.4 with failures in perl should try freshly checking out perl and building it outside of the full buildworld cycle (cd /usr/src/usr.bin/perl && make && make install) first. 

There's an apache exploit that affects everyone running 2.x.  On MidnightBSD, we only ship 2.2.x now.  A security patch was supposed to be out for this yesterday, but it hasn't happened.  As such, I'm providing a link to the discussion. There has been active discussion on the Apache mailing lists starting on the 24th.  

I've been working hard on getting the port upgrades in place for the next MidnightBSD release.  The following ports have been updated recently:

QT 4.7.3

GNUstep (make, base, back, gui, ...) .  This uses the system compiler now as there are some issues with the GCC 4.4 snap Objective-C support with it.

GCC 4.4.7 snap 6/28

 One of the nice features of Mac OS X is Apple's spotlight.  It makes it easy to find documents because it supports full text search and is aware of different file types.  In the open source world, there are many search tools for Linux, but they all fail in different ways.  Some of them are slow.  Others don't support full text search and rely on inotify.

Linux solutions 

With inotify, the Linux kernel can notify a program that a file has changed by path name.  In the BSD community, we have kqueue that will report changes via fd.  Ideally, one would create a system daemon that can monitor changes in files and update the index on the fly.  This is planned for a future version of msearch(1).  A flaw with most BSD approaches is that it's easy to hit the kern.maxfiles limit as one has to have many directories and files open to detect changes.  kqueue approaches tend to work with UFS and UFS2 file systems only.  Someone using ZFS or fat32 would not get changes unless polling was used. Most modern Linux systems use gamin or FAM to monitor file changes.  

Many of the Linux solutions are under the GPL license. They were not designed for BSD.  I've started down the path of solving this problem.  The first iteration of my work is called msearch.  msearch(1) is a command line tool to search for files on the computer either matching elements of the path or by using the full text search feature.  

Indexing

All text files on the computer can be indexed by msearch.  It uses libmagic to determine the mime type of the file.  This allows it to skip files that are empty, binary, or otherwise useless to the search tool.  

msearch(1) uses two index files generated by a program called msearch.index.  /var/db/msearch.db is a sqlite database containing path information, owner, group, and file size at the time of indexing. /var/db/msearch_full.db contains a sqlite 3 FTS4 full text index of the text files on the computer.  It makes use of zlib to compress the text data.  On my computer, approximately 350,000 files were indexed and 84,000 were considered text files indexable by the full text engine.  Prior to adding compression, the database used 850MB of space. After compression, the file uses 413MB.  Another compression algorithm might cut off additional space at the expense of indexing performance. 

The current version of msearch relies on a periodic script similar to locate(1).  It is run weekly and most be turned on with weekly_msearch_enable="YES" in periodic.conf.  I would like to replace this process with a daemon that handles search requests and indexing.  Apple's search features work in this manner.

Graphical Search

Most of the logic for msearch(1) was placed in a shared library, libmsearch, which can be used to create a graphical search tool.   I envision a sherlock like search tool for the initial release and possibly an integrated solution if MidnightBSD ever gets it's own window manager.  

Security

There are several possible issues with generating an index of all files.  If the index is readable by any user, it could allow one to open the sqlite file and read the contents of sensitive files.  For this reason, I've limited the indexer so that it cannot run as the root user.  Files most be readable by nobody (if using the periodic script) to become part of the index.  

There is also the possibility of sql injection.  The database files aren't writable by normal users and the indexer uses prepare statements.  As the searching functionality is currently using a custom built search string, this could result in undesired behavior.  It's also not recommended to do a search as the root user.  sqlite does have the ability to load extensions, and this feature is used to compress and rank full text data.  The extension loading is turned off right after the database is created to avoid problems form uesrs. 

Future directions

I have a large list of features to add to mserach(1).  I plan to add filtering based on file size, user id, group id, created and modified times. I've considered adding a network search feature in combination with the plans for the search daemon and indexing in near 'real time" with file monitoring.  In order for this to work efficiently, a new kernel interface would need to be created or kqueue would need to be modified.  

I don't intend for this tool to replace locate(1), find(1) or similar search functions, but merely allow users to have an additional option with full text.  

Performance

Full text searches are quire fast.  Simple queries such as searching for Linux are done in seconds.  A search against path names takes longer than locate(1), but is still respectable. locate(1) uses a path compression technique to keep the database small and was optimized for low resources.  msearch(1) takes advantage of the convenience of sqlite 3 and the modern performance of PCs. 

I've finally completed the libxul port (mozilla gecko).  This allows us to build yelp again for Gnome.  I've also created a linux-firefox5 port and a linux-thunderbird5 port.

Opera, curl, tnftp, and tnftpd mports were updated tonight.  I also removed ftp/mirror as the upstream disappeared and the license limited what we could do with the software.

A vulnerability exists in the base system version of BIND as well as BIND 9.7 and 9.8 in mports.  The mports were updated last night, but the base system has not been patched yet.  It is recommended that users track the ports versions of BIND at this point.  

I will work on getting BIND updated in src.  I'm seriously considering dropping BIND from base as there have been a lot of security holes and changes at a fast pace over the last year.  It's becoming a lot of work and updating ports is much quicker. If I choose to do this, the first change will be switching to the separate libbind library and updating the few tools from BIND such as dig.  

I've added the Google +1 button to the MidnightBSD website as an experiment.  I'm hoping users will use it on pages they find useful.  It will help MidnightBSD in search results and let me know what pages I'm succeeding on.

I've also realized I've ignored the website too long.  I plan on adding more content to it. 

mport tools

Progress has been made on the MidnightBSD package tools.  The mport tool can now download an index generated from the Magus build cluster using a bootstrap site and then fetch from mirrors defined in that file.  It checks for updates weekly for index files.  Several bugs have been fixed in fetching files.  Delete and Update commands are working and progress has been made on the upgrade command, although it's not quite production ready.  Upgrade allows you to update all packages on your system.  There are some ordering issues left to fix and we have some minor bugs with ports that contain pkgnamesuffix.

mport has a new command deleteall, which will remove all packages for a system.  This is good when you want to start over with an update or you want to test different package configurations. 

The remaining steps for mport tools include integration with the installer and squashing a few more bugs before the 0.4-RELEASE.

 mports

Browsers

The GECKO engine makefiles have been tweaked to work better with newer versions of Mozilla products. Unbranded versions of Firefox are building again and work has begun on porting libxul for use with GNOME ports. 

Opera 11 has been added to the ports collection, but requires a very recent version of 0.4-CURRENT. . 

Chromium 5 has been in the tree for a few months and we're working on getting a newer version ported. 

Gnome

Gnome 2.32 is slowly shaping up in the tree. We have 90% of gnome2-lite working excluding yelp and epiphany which require libxul.  Gtk, and many Gnome ports have been updated in the process

Databases

MySQL, Postgresql, and sqlite3 have been updated to recent builds.  This includes several performance and security enhancements. 

X.org

X.org 7.5 has been imported into mports.  This combined with the recent drm update, allow newer video cards to work with acceleration.  This should improve support for all major graphics card vendors.

src

0.4-CURRENT has many improvements including mksh R40, Perl 5.14, security updates for BIND, drm updates, fixes for cam(3), and the default blocksize for UFS/FFS was increased to 32K with 4K fragments. The latter change will work better with advanced format hard drives like the WD Green drives (EARS).  

Sendmail 8.14.5 was imported.

Finally, xz and liblzma have been added to the tree.

X.org 7.5 is working on MidnightBSD now.  There are a few tweaks I want to make to ports, but it's running on i386 CURRENT.  The nv driver didn't get installed automatically.  

I've also started the migration to gnome 2.32.  

Totally unrelated, GNU grep was bumped to 2.7 tonight. 

For those of you using sendmail on MidnightBSD, the port has just been updated to 8.14.5.  i plan on updating the base system sendmail in current when I get time. 

These results aren't too good.  Part of this is some cleanup happening to make the new X.org happen.  

I'm working on updating several key ports I've let slide for far too long.  Getting ready for the next release.