MidnightBSD Developer Journal

MidnightBSD mports now includes the latest Apache release.  This fixes a DOS attack using range headers.  Users are encouraged to upgrade. 

I've just updated the opera port to 11.51.  This version includes some security fixes.  

MidnightBSD has been lagging some other systems with respect to web browsers.  Tonight, I updated Opera, Flash 10 (linux emulation), Firefox 6 and Thunderbird 6 (linux) ports and the nspluginwrapper port.  It's now possible to play flash video on several browsers in current (again) and access youtube.

You must be running 0.4-CURRENT to use some of these.  

mport perl updated

Perl 5.14.1 is now available in mports.  While I've considered updating base, 5.14.2 is scheduled for release next month.  If 0.4 is not ready by the time the release occurs, I'll update it in base, otherwise it will be included in 0.5.

Base Perl issues 

Users having difficulty upgrading from MidnightBSD 0.3 to 0.4 with failures in perl should try freshly checking out perl and building it outside of the full buildworld cycle (cd /usr/src/usr.bin/perl && make && make install) first. 

There's an apache exploit that affects everyone running 2.x.  On MidnightBSD, we only ship 2.2.x now.  A security patch was supposed to be out for this yesterday, but it hasn't happened.  As such, I'm providing a link to the discussion. There has been active discussion on the Apache mailing lists starting on the 24th.  

I've been working hard on getting the port upgrades in place for the next MidnightBSD release.  The following ports have been updated recently:

QT 4.7.3

GNUstep (make, base, back, gui, ...) .  This uses the system compiler now as there are some issues with the GCC 4.4 snap Objective-C support with it.

GCC 4.4.7 snap 6/28

 One of the nice features of Mac OS X is Apple's spotlight.  It makes it easy to find documents because it supports full text search and is aware of different file types.  In the open source world, there are many search tools for Linux, but they all fail in different ways.  Some of them are slow.  Others don't support full text search and rely on inotify.

Linux solutions 

With inotify, the Linux kernel can notify a program that a file has changed by path name.  In the BSD community, we have kqueue that will report changes via fd.  Ideally, one would create a system daemon that can monitor changes in files and update the index on the fly.  This is planned for a future version of msearch(1).  A flaw with most BSD approaches is that it's easy to hit the kern.maxfiles limit as one has to have many directories and files open to detect changes.  kqueue approaches tend to work with UFS and UFS2 file systems only.  Someone using ZFS or fat32 would not get changes unless polling was used. Most modern Linux systems use gamin or FAM to monitor file changes.  

Many of the Linux solutions are under the GPL license. They were not designed for BSD.  I've started down the path of solving this problem.  The first iteration of my work is called msearch.  msearch(1) is a command line tool to search for files on the computer either matching elements of the path or by using the full text search feature.  

Indexing

All text files on the computer can be indexed by msearch.  It uses libmagic to determine the mime type of the file.  This allows it to skip files that are empty, binary, or otherwise useless to the search tool.  

msearch(1) uses two index files generated by a program called msearch.index.  /var/db/msearch.db is a sqlite database containing path information, owner, group, and file size at the time of indexing. /var/db/msearch_full.db contains a sqlite 3 FTS4 full text index of the text files on the computer.  It makes use of zlib to compress the text data.  On my computer, approximately 350,000 files were indexed and 84,000 were considered text files indexable by the full text engine.  Prior to adding compression, the database used 850MB of space. After compression, the file uses 413MB.  Another compression algorithm might cut off additional space at the expense of indexing performance. 

The current version of msearch relies on a periodic script similar to locate(1).  It is run weekly and most be turned on with weekly_msearch_enable="YES" in periodic.conf.  I would like to replace this process with a daemon that handles search requests and indexing.  Apple's search features work in this manner.

Graphical Search

Most of the logic for msearch(1) was placed in a shared library, libmsearch, which can be used to create a graphical search tool.   I envision a sherlock like search tool for the initial release and possibly an integrated solution if MidnightBSD ever gets it's own window manager.  

Security

There are several possible issues with generating an index of all files.  If the index is readable by any user, it could allow one to open the sqlite file and read the contents of sensitive files.  For this reason, I've limited the indexer so that it cannot run as the root user.  Files most be readable by nobody (if using the periodic script) to become part of the index.  

There is also the possibility of sql injection.  The database files aren't writable by normal users and the indexer uses prepare statements.  As the searching functionality is currently using a custom built search string, this could result in undesired behavior.  It's also not recommended to do a search as the root user.  sqlite does have the ability to load extensions, and this feature is used to compress and rank full text data.  The extension loading is turned off right after the database is created to avoid problems form uesrs. 

Future directions

I have a large list of features to add to mserach(1).  I plan to add filtering based on file size, user id, group id, created and modified times. I've considered adding a network search feature in combination with the plans for the search daemon and indexing in near 'real time" with file monitoring.  In order for this to work efficiently, a new kernel interface would need to be created or kqueue would need to be modified.  

I don't intend for this tool to replace locate(1), find(1) or similar search functions, but merely allow users to have an additional option with full text.  

Performance

Full text searches are quire fast.  Simple queries such as searching for Linux are done in seconds.  A search against path names takes longer than locate(1), but is still respectable. locate(1) uses a path compression technique to keep the database small and was optimized for low resources.  msearch(1) takes advantage of the convenience of sqlite 3 and the modern performance of PCs. 

I've finally completed the libxul port (mozilla gecko).  This allows us to build yelp again for Gnome.  I've also created a linux-firefox5 port and a linux-thunderbird5 port.

Opera, curl, tnftp, and tnftpd mports were updated tonight.  I also removed ftp/mirror as the upstream disappeared and the license limited what we could do with the software.

A vulnerability exists in the base system version of BIND as well as BIND 9.7 and 9.8 in mports.  The mports were updated last night, but the base system has not been patched yet.  It is recommended that users track the ports versions of BIND at this point.  

I will work on getting BIND updated in src.  I'm seriously considering dropping BIND from base as there have been a lot of security holes and changes at a fast pace over the last year.  It's becoming a lot of work and updating ports is much quicker. If I choose to do this, the first change will be switching to the separate libbind library and updating the few tools from BIND such as dig.  

I've added the Google +1 button to the MidnightBSD website as an experiment.  I'm hoping users will use it on pages they find useful.  It will help MidnightBSD in search results and let me know what pages I'm succeeding on.

I've also realized I've ignored the website too long.  I plan on adding more content to it. 

mport tools

Progress has been made on the MidnightBSD package tools.  The mport tool can now download an index generated from the Magus build cluster using a bootstrap site and then fetch from mirrors defined in that file.  It checks for updates weekly for index files.  Several bugs have been fixed in fetching files.  Delete and Update commands are working and progress has been made on the upgrade command, although it's not quite production ready.  Upgrade allows you to update all packages on your system.  There are some ordering issues left to fix and we have some minor bugs with ports that contain pkgnamesuffix.

mport has a new command deleteall, which will remove all packages for a system.  This is good when you want to start over with an update or you want to test different package configurations. 

The remaining steps for mport tools include integration with the installer and squashing a few more bugs before the 0.4-RELEASE.

 mports

Browsers

The GECKO engine makefiles have been tweaked to work better with newer versions of Mozilla products. Unbranded versions of Firefox are building again and work has begun on porting libxul for use with GNOME ports. 

Opera 11 has been added to the ports collection, but requires a very recent version of 0.4-CURRENT. . 

Chromium 5 has been in the tree for a few months and we're working on getting a newer version ported. 

Gnome

Gnome 2.32 is slowly shaping up in the tree. We have 90% of gnome2-lite working excluding yelp and epiphany which require libxul.  Gtk, and many Gnome ports have been updated in the process

Databases

MySQL, Postgresql, and sqlite3 have been updated to recent builds.  This includes several performance and security enhancements. 

X.org

X.org 7.5 has been imported into mports.  This combined with the recent drm update, allow newer video cards to work with acceleration.  This should improve support for all major graphics card vendors.

src

0.4-CURRENT has many improvements including mksh R40, Perl 5.14, security updates for BIND, drm updates, fixes for cam(3), and the default blocksize for UFS/FFS was increased to 32K with 4K fragments. The latter change will work better with advanced format hard drives like the WD Green drives (EARS).  

Sendmail 8.14.5 was imported.

Finally, xz and liblzma have been added to the tree.

X.org 7.5 is working on MidnightBSD now.  There are a few tweaks I want to make to ports, but it's running on i386 CURRENT.  The nv driver didn't get installed automatically.  

I've also started the migration to gnome 2.32.  

Totally unrelated, GNU grep was bumped to 2.7 tonight. 

For those of you using sendmail on MidnightBSD, the port has just been updated to 8.14.5.  i plan on updating the base system sendmail in current when I get time. 

These results aren't too good.  Part of this is some cleanup happening to make the new X.org happen.  

I'm working on updating several key ports I've let slide for far too long.  Getting ready for the next release. 

 Currently uploading packages for amd64 current from the last magus run to the ftp server.  These are located in the 0.4 directory rather than the historical use of current.  They are mport based packages.  An index will be generated from them for further testing the new mport tools.

Reverse DNS for our IPV6 addresses was finally completed today.  

This month has been crazy for me.  As I had a business trip and then got sick, I haven't had as much time as I'd like with MidnightBSD.  I managed to get a few ports updated today and fixed a bug in src showing up in tinderbox.  

The MidnightBSD forums have been taken offline until I can find a solution to the terrible spam problem.  Please use the mailing list instead.   The BSD licensed forum software I was using, jforum, has a few antispam features but it's rather clunky to use.  It would be nice if I could block ip quickly by user's last ip or even a subnet.  The spam is coming from China.  I suspect there are a few bugs with input parsing and a script is able to create accounts and spam from it even with activation emails and other measures. It was using a large amount of my bandwidth as well as slowing down several of the websites and causing large companies to contact me regarding trademark use.  :) 

My original goal was to get the 0.4-RELEASE out in May.  As I've lost most of April, I'm not shooting for June.  This is a smaller release focusing on the transition to the mport package system.  I'm also trying to get as many ports as possible updated for the release.  

The current focus of the project is to get the mport tools ready for the 0.4 release.  mport tools are now the default on CURRENT.  They work for basic operations and while we're not actively using it yet, they have a downloadable index with current packages available.  This index is created on the server with data from our Magus package cluster database and sqlite3.  It includes information like package name, license, mirrors to download the file, version, etc.  

I'm currently working on adding additional information to the index about categories mports belong to.  This would be nice for display in the installer as well as for searches.  

Speaking of the installer, I've been working on modifications to use the mport tools instead of the legacy pkg_tools.  sysinstall will eventually be replaced, although I doubt we'll use the pc-sysinstall or bsdinstaller backends from other systems.  Most likely it will not be a shell or lua based environment.  The installer is our next big project after mport tools.  

sysinstall was simplified recently.  Several menu options that are confusing to our userbase were removed.

In addition to this work, we've been updating and adding new features to the base system.  DRM/DRI code was imported from FreeBSD 7.1 as well as cdevpriv wrappers.  nss_mdns was added to the base system; it allows one to find other systems on the network using multicast DNS.  Systems running mDNSResponder or Avahi can now be seen on the network.  I'm now able to ssh between my iMac and MidnightBSD machines by name (.local).  

The time zone update file, tzdata2011c, was imported. An OpenSSL security issue was patched that only affects CURRENT.   

Finally, xz 5.0.1 and liblzma were imported into the base system.  mports was modified to use the system version when possible. 

I read a recent post on osnews.com about their recent traffic and wondered what type of users are visiting the MidnightBSD website.  We don't get a lot of traffic, but still it was an interesting question who visits the site.  Here is a breakdown by OS:

 1 3677372 485071 OS unknown

2 1519710 228260 Known robots

3 767452 108338 Windows

  437089 78553  Windows XP

  306080 24350  Unknown Windows

  13331 2691  Windows Server 2003

  8845 1763  Windows 2000

  1149 403  Windows 98

  471 282  Windows NT

  143 126  Windows ME

  140 79  Windows 95

  183 72  Windows CE

  21 19  Windows 3.1

4 488384 35717 Unix

  449753 32489  Linux

  29375 2302  FreeBSD

  3039 291  OpenBSD

  2124 172  NetBSD

  903 151  Unknown BSD

  1668 150  SunOS

  964 100  MidnightBSD

  101 25  MirBSD

  330 20  DragonFly

  126 16  Other Unix

  1 1  AIX

5 119959 9461 Macintosh

6 189 21 Symbian OS

7 137 12 Sony PS3

8 31 5 BeOS

9 26 4 OS/2

10 81 4 Amiga

11 9 2 Sony PSP

12 1 0 Nintendo Wii