Table of Contents
When targeting a product for an embedded platform, it's not feasible to have all the development tools available on that same platform. Instead, some method of crosscompiling is usually used today. NetBSD 1.6 and forward comes with a framework to build both the operating system's kernel and the whole userland for either the same platform that the compiler runs on, or for a different platform, using crosscompiling. Crosscompiling requires assembler, linker, compiler etc. to be available and built for the target platform. The new build scheme will take care of creating these tools for a given platform, and make them available ready to use to do development work.
In this chapter, we will show how to use
build.sh to first create a crosscompiling
toolchain, including cross-compiler, cross-assembler, cross-linker
and so on.
While native kernel builds are covered in Chapter 31, Compiling the kernel, these tools are then used to
manually configure and crosscompile a kernel for a different
platform, and then show how to use build.sh
as a convenient alternative. After that works,
the whole NetBSD userland will be compiled and packed up in the
format of a NetBSD release. In the examples, we will use the Sun
UltraSPARC ("sparc64") 64-bit platform as target
platform, any other platform supported by NetBSD can be targetted as
well specifying its name (see
/usr/src/sys/arch).
Before starting, take note that it is assumed that the NetBSD
sources from the "netbsd-4-0" branch are available in
/usr/src as described in
Chapter 29, Obtaining the sources.
A more detailed description of the build.sh
framework can be found in Luke Mewburn and Matthew Green's
paper
and their
presentation
from BSDCon 2003 as well as in
/usr/src/BUILDING.
The first step to do cross-development is to get all the necessary tools available. In NetBSD terminology, this is called the "toolchain", and it includes BSD-compatible make(1), C/C++ compilers, linker, assembler, config(8), as well as a fair number of tools that are only required when crosscompiling a full NetBSD release, which we won't cover here.
The command to create the crosscompiler is quite simple, using
NetBSD's new src/build.sh script. Please note
that all the commands here can be run as normal (non-root) user:
$cd /usr/src$./build.sh -m sparc64 tools
Make sure that the directory /usr/obj
does exist, or add a "-O" option to the build.sh call, redirecting
the object directory someplace else.
If the tools have been built previously and they only need updated, then the update option "-u" can be used to only rebuild tools that have changed:
$./build.sh -u -m sparc64 tools
When the tools are built, information about them and several environment variables is printed out:
...
===> build.sh started: Thu Dec 2 22:18:11 CET 2007
===> build.sh ended: Thu Dec 2 22:28:22 CET 2007
===> Summary of results:
build.sh command: ./build.sh -m sparc64 tools
build.sh started: Thu Dec 2 22:18:11 CET 2007
No nonexistent/bin/nbmake, needs building.
Bootstrapping nbmake
MACHINE: sparc64
MACHINE_ARCH: sparc64
TOOLDIR path: /usr/src/tooldir.NetBSD-4.0-i386
DESTDIR path: /usr/src/destdir.sparc64
RELEASEDIR path: /usr/src/releasedir
Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
makewrapper: /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
Tools built to /usr/src/tooldir.NetBSD-4.0-i386
build.sh started: Thu Dec 2 22:18:11 CET 2007
build.sh ended: Thu Dec 2 22:28:22 CET 2007
===> .
During the build, object directories are used consistently, i.e. special directories are kept that keep the platform-specific object files and compile results. In our example, they will be kept in directories named "obj.sparc64" as we build for UltraSPARC as target platform.
The toolchain itself is part of this, but as it's hosted and compiled for a i386 system, it will get placed in its own directory indicating where to cross-build from. Here's where our crosscompiler tools are located:
$pwd/usr/src$ls -d tooldir.*tooldir.NetBSD-4.0-i386
So the general rule of thumb is for a given "host" and "target" system combination, the crosscompiler will be placed in the "src/tooldir.host" directory by default. A full list of all tools created for crosscompiling the whole NetBSD operating system includes:
$ls tooldir.NetBSD-4.0-i386/bin/nbasn1_compile nbmakefs nbzic nbcap_mkdb nbmakeinfo sparc64--netbsd-addr2li nbcat nbmakewhatis sparc64--netbsd-ar nbcksum nbmenuc sparc64--netbsd-as nbcompile_et nbmkcsmapper sparc64--netbsd-c++ nbconfig nbmkdep sparc64--netbsd-c++filt nbcrunchgen nbmkesdb sparc64--netbsd-cpp nbctags nbmklocale sparc64--netbsd-dbsym nbdb nbmknod sparc64--netbsd-g++ nbeqn nbmktemp sparc64--netbsd-g77 nbfgen nbmsgc sparc64--netbsd-gcc nbfile nbmtree sparc64--netbsd-gcc-3.3 nbgencat nbnroff sparc64--netbsd-gccbug nbgroff nbpax sparc64--netbsd-gcov nbhexdump nbpic sparc64--netbsd-ld nbhost-mkdep nbpwd_mkdb sparc64--netbsd-lint nbindxbib nbrefer sparc64--netbsd-mdsetim nbinfo nbrpcgen sparc64--netbsd-nm nbinfokey nbsoelim sparc64--netbsd-objcopy nbinstall nbstat sparc64--netbsd-objdump nbinstall-info nbsunlabel sparc64--netbsd-ranlib nbinstallboot nbtbl sparc64--netbsd-readelf nblex nbtexi2dvi sparc64--netbsd-size nblorder nbtexindex sparc64--netbsd-strings nbm4 nbtsort sparc64--netbsd-strip nbmake nbuudecode nbmake-sparc64 nbyacc
As you can see, most of the tools that are available native on NetBSD are present with some program prefix to identify the target platform for tools that are specific to a certain target platform.
One important tool that should be pointed out here is "nbmake-sparc64". This is a shell wrapper for a BSD compatible make(1) command that's setup to use all the right commands from the crosscompiler toolchain. Using this wrapper instead of /usr/bin/make allows crosscompiling programs that were written using the NetBSD Makefile infrastructure (see src/share/mk). We will use this make(1) wrapper in a second to cross compile the kernel!
Now that we have a working crosscompiler available, the "usual" steps for building a kernel are needed - create a kernel config file, run config(8), then build. As the config(8) program used to create header files and Makefile for a kernel build is platform specific, we need to use the "nbconfig" program that's part of our new toolchain. That aside, the procedure is just as like compiling a "native" NetBSD kernel. Commands involved here are:
$cd /usr/src/sys/arch/sparc64/conf$cp GENERICMYKERNEL$viMYKERNEL$/usr/src/tooldir.NetBSD-4.0-i386/bin/nbconfigMYKERNEL
That's all. This command has created a directory
../compile/
with a number of header files defining information about devices
to compile into the kernel, a Makefile that is setup to build
all the needed files for the kernel, and link them together.MYKERNEL
We have all the files and tools available to crosscompile our
UltraSPARC-based kernel from our Intel-based host system, so
let's get to it! After changing in the directory created in the
previous step, we need to use the crosscompiler toolchain's
nbmake-sparc64 shell wrapper, which just
calls make(1) with all the necessary settings for crosscompiling
for a sparc64 platform:
$cd ../compile/MYKERNEL/$/usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64 depend$/usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
This will churn away a bit, then spit out a kernel:
... text data bss dec hex filename 5016899 163728 628752 5809379 58a4e3 netbsd$ls -l netbsd-rwxr-xr-x 1 feyrer 666 5874663 Dec 2 23:17 netbsd$file netbsdnetbsd: ELF 64-bit MSB executable, SPARC V9, version 1 (SYSV), statically linked, not stripped
Now the kernel in the file netbsd can
either be transferred to a UltraSPARC machine (via NFS, FTP,
scp, etc.) and booted from a possible harddisk, or directly from
our cross-development machine using NFS.
After configuring and crosscompiling the kernel, the next
logical step is to crosscompile the whole system, and bring it
into a distribution-ready format. Before doing so, an
alternative approach to crosscompiling a kernel will be shown
in the next section, using the build.sh
script to do configuration and crosscompilation of the kernel
in one step.
A cross compiled kernel can be done manually as described in
the previous sections, or by the easier method of using
build.sh, which will be shown here.
Preparation of the kernel config file is the same as described above:
$cd /usr/src/sys/arch/sparc64/conf$cp GENERICMYKERNEL$viMYKERNEL
Then edit MYKERNEL and once finished,
all that needs to be done is
to use build.sh to build the kernel
(it will also configure it, running the steps shown above):
$cd /usr/src$./build.sh -u -m sparc64 kernel=MYKERNEL
Notice that update ("-u") was specified, the tools are
already built, there
is no reason to rebuild all of the tools. Once the kernel is built,
build.sh will print out the location
of it along with other information:
...
===> Summary of results:
build.sh command: ./build.sh -u -m sparc64 kernel=MYKERNEL
build.sh started: Thu Dec 2 23:30:02 CET 2007
No nonexistent/bin/nbmake, needs building.
Bootstrapping nbmake
MACHINE: sparc64
MACHINE_ARCH: sparc64
TOOLDIR path: /usr/src/tooldir.NetBSD-4.0-i386
DESTDIR path: /usr/src/destdir.sparc64
RELEASEDIR path: /usr/src/releasedir
Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
makewrapper: /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
Building kernel without building new tools
Building kernel: MYKERNEL
Build directory: /usr/src/sys/arch/sparc64/compile/obj.sparc64/GENERIC
Kernels built from MYKERNEL:
/usr/src/sys/arch/sparc64/compile/obj.sparc64/MYKERNEL/netbsd
build.sh started: Thu Dec 2 23:30:02 CET 2007
build.sh ended: Thu Dec 2 23:38:22 CET 2007
===> .
The path to the kernel built is of interest here:
/usr/src/sys/arch/sparc64/compile/obj.sparc64/,
it can be used the same way as described above.MYKERNEL/netbsd
By now it is probably becoming clear that the toolchain actually
works in stages. First the crosscompiler is built, then a
kernel. Since build.sh will attempt to
rebuild the tools at every invocation, using “update”
saves time. It is probably also clear that outside of a few options,
the build.sh semantics are basically
build.sh command.
So, it stands to reason that
building the whole userland and/or a release is a matter of
using the right commands.
It should be no surprise that building and creating a release would look like the following:
$./build.sh -U -u -m sparc64 release
These commands will compile the full NetBSD userland and put it
into a destination directory, and then build a release from it
in a release directory. The -U switch is added here for an
unprivileged build, i.e. one that's running as normal
user and not as root. As no further switches to
build.sh were given nor any environment
variables were set, the defaults of
DESTDIR=/usr/src/destdir.sparc64 and
RELEASEDIR=/usr/src/releasedir are used, as
shown in the build.sh-output above.
The NetBSD project has its own copy of the X Window System's
source which is currently based on XFree86 version 4, and
which contains changes to make X going on as many of the
platforms supported by NetBSD as possible. Due to this, it is
desirable to use the X Window System version available from
and for NetBSD, which can also be crosscompiled much like the
kernel and base system. To do so, the "xsrc" sources
must be checked out from CVS into
/usr/xsrc just as "src" and "pkgsrc" were
as described in Chapter 29, Obtaining the sources.
After this, X can be crosscompiled for the target platform by
adding the -x switch to build.sh, e.g. when creating
a full release:
$./build.sh -U -x -u -m sparc64 release
The -U
flag for doing unprivileged (non-root) builds and the
-u flag for not removing old files before
building as well
as the -m
option to define
the target architecture have already been introduced, and the
arch-x option to also (cross)compile
"xsrc" is another option.
Similar to the old, manual building method, the new
toolchain has a lot of variables that can be used to direct
things like where certain files go, what (if any) tools are used
and so on. A look in src/BUILDING covers
most of them. In this section some examples of changing default
settings are given, each following its own ways.
Many people like to track NetBSD-current and perform cross compiles of architectures that they use. The logic for this is simple, sometimes a new feature or device becomes available and someone may wish to use it. By keeping track of changes and building every now and again, one can be assured that these architectures can build their own release.
It is reasonable to assume that if one is tracking and building for more than one architecture, they might want to keep the builds in a different location than the default. There are two ways to go about this, either use a script to set the new DESTDIR, or simply do so interactively. In any case, it can be set the same way as any other variable (depending on your shell of course).
For bash, the Bourne or Korn shell, this is:
$export DESTDIR=/usr/builds/sparc64
For tcsh and the C shell, the command is:
$setenv DESTDIR /usr/builds/sparc64
Simple enough. When the build is run, the binaries and
files will be sent to /usr/builds.
The NetBSD toolchain builds and links against shared libraries
by default. Many users still prefer to be able to link
statically. Sometimes a small system can be created without
having shared libraries, which is a good example of doing a
full static build. If a particular build machine will always
need one environment variable set in a particular way, then it
is easiest to simply add the changed setting to
/etc/mk.conf.
To make sure a build box always builds statically, simply
add the following line to
/etc/mk.conf:
LDSTATIC=-static
Besides variables in environment and
/etc/mk.conf, the build process can be
influenced by a number of switches to the
build.sh script itself, as we have
already seen when forcing unprivileged (non-root) builds,
selecting the target architecture or preventing deletion of
old files before the build. All these options can be
listed by running build.sh -h:
$cd /usr/src$build.sh -hUsage: build.sh [-EnorUux] [-a arch] [-B buildid] [-D dest] [-j njob] [-M obj] [-m mach] [-N noisy] [-O obj] [-R release] [-T tools] [-V var=[value]] [-w wrapper] [-X x11src] [-Z var] operation [...] Build operations (all imply "obj" and "tools"): build Run "make build". distribution Run "make distribution" (includes DESTDIR/etc/ files). release Run "make release" (includes kernels and distrib media). Other operations: help Show this message and exit. makewrapper Create nbmake-${MACHINE} wrapper and nbmake. Always performed. obj Run "make obj". [Default unless -o is used] tools Build and install tools. install=idir Run "make installworld" to `idir' to install all sets except `etc'. Useful after "distribution" or "release" kernel=conf Build kernel with config file `conf' releasekernel=conf Install kernel built by kernel=conf to RELEASEDIR. sets Create binary sets in RELEASEDIR/MACHINE/binary/sets. DESTDIR should be populated beforehand. sourcesets Create source sets in RELEASEDIR/source/sets. params Display various make(1) parameters. Options: -a arch Set MACHINE_ARCH to arch. [Default: deduced from MACHINE] -B buildId Set BUILDID to buildId. -D dest Set DESTDIR to dest. [Default: destdir.MACHINE] -E Set "expert" mode; disables various safety checks. Should not be used without expert knowledge of the build system. -j njob Run up to njob jobs in parallel; see make(1) -j. -M obj Set obj root directory to obj; sets MAKEOBJDIRPREFIX. Unsets MAKEOBJDIR. -m mach Set MACHINE to mach; not required if NetBSD native. -N noisy Set the noisyness (MAKEVERBOSE) level of the build: 0 Quiet 1 Operations are described, commands are suppressed 2 Full output [Default: 2] -n Show commands that would be executed, but do not execute them. -O obj Set obj root directory to obj; sets a MAKEOBJDIR pattern. Unsets MAKEOBJDIRPREFIX. -o Set MKOBJDIRS=no; do not create objdirs at start of build. -R release Set RELEASEDIR to release. [Default: releasedir] -r Remove contents of TOOLDIR and DESTDIR before building. -T tools Set TOOLDIR to tools. If unset, and TOOLDIR is not set in the environment, nbmake will be (re)built unconditionally. -U Set MKUNPRIVED=yes; build without requiring root privileges, install from an UNPRIVED build with proper file permissions. -u Set MKUPDATE=yes; do not run "make clean" first. Without this, everything is rebuilt, including the tools. -V v=[val] Set variable `v' to `val'. -w wrapper Create nbmake script as wrapper. [Default: ${TOOLDIR}/bin/nbmake-${MACHINE}] -X x11src Set X11SRCDIR to x11src. [Default: /usr/xsrc] -x Set MKX11=yes; build X11R6 from X11SRCDIR -Z v Unset ("zap") variable `v'.
As can be seen, a number of switches can be set to change the standard build behaviour. A number of them has already been introduced, others can be set as appropriate.
Several variables control the behaviour of NetBSD builds.
Unless otherwise specified, these variables may be set in
either the process environment or in the make(1)
configuration file specified by MAKECONF.
For a definitive list of these options, see
BUILDING and
share/mk/bsd.README files in the
toplevel source directory.
Identifier for the build. The identifier will be appended to object directory names, and can be consulted in the make(1) configuration file in order to set additional build parameters, such as compiler flags.
Directory to contain the built NetBSD system. If set, special
options are passed to the compilation tools to prevent their
default use of the host system's
/usr/include,
/usr/lib, and so forth. This pathname
should not end with a slash (/) character (For installation
into the system's root directory, set DESTDIR
to an empty string). The directory must reside on a filesystem
which supports long filenames and hard links.
Defaults to an empty string if USETOOLS is
“yes”; unset otherwise. Note:
build.sh will provide a default
(destdir.MACHINE in the top-level .OBJDIR)
unless run in “expert” mode.
If defined by the user, points to the root of an
external toolchain (e.g.
/usr/local/gnu). This enables
the cross-build framework even when default toolchain
is not available (see TOOLCHAIN_MISSING
below).
Default: Unset
The verbosity of build messages. Supported values:
| 0 | No descriptive messages are shown. |
| 1 | Descriptive messages are shown. |
| 2 | Descriptive messages are shown (prefixed with a '#') and command output is not suppressed. |
Default: 2
Can be set to “yes” or “no”. Indicates whether preformatted plaintext manual pages will be created during a build.
Default: “yes”
Can be set to “yes” or “no”. Indicates whether cryptographic code will be included in a build; provided for the benefit of countries that do not allow strong cryptography. Will not affect the standard low-security password encryption system, crypt(3).
Default: “yes”
Can be set to “yes” or “no”. Indicates
whether system documentation destined for
DESTDIR/usr/share/doc will
be installed during a build.
Default: “yes”
Can be set to “yes” or “no”. If set to “yes”, then for programs intended to be run on the compile host, the name, release and architecture of the host operating system will be suffixed to the name of the object directory created by “make obj”. This allows for multiple host systems to compile NetBSD for a single target. If set to “no”, then programs built to be run on the compile host will use the same object directory names as programs built to be run on the target.
Default: “no”
Can be set to “yes” or “no”. Indicates whether GNU info files, used for the documentation of most of the compilation tools, will be created and installed during a build.
Default: “yes”
Can be set to “yes” or “no”. Indicates
whether lint(1) will be run against portions of the NetBSD
source code during the build, and whether lint libraries will
be installed into
DESTDIR/usr/libdata/lint
Default: “yes”
Can be set to “yes” or “no”. Indicates whether manual pages will be installed during a build.
Default: “yes”
Can be set to “yes” or “no”. Indicates whether Native Language System locale zone files will be compiled and installed during a build.
Default: “yes”
Can be set to “yes” or “no”. Indicates whether object directories will be created when running “make obj”. If set to “no”, then all built files will be located inside the regular source tree.
Default: “yes”
Can be set to “yes” or “no”. Indicates whether shared objects and libraries will be created and installed during a build. If set to “no”, the entire build will be statically linked.
Default: Platform dependent. As of this writing, all platforms except sh3 default to “yes”
Can be set to “yes” or “no”. Indicates
whether the ar(1) format libraries
(lib*_pic.a), used to generate shared
libraries, are installed during a build.
Default: “yes”
Can be set to “yes” or “no”. Indicates
whether profiled libraries (lib*_p.a)
will be built and installed during a build.
Default: “yes”; however, some platforms turn off
MKPROFILE by default at times due to toolchain
problems with profiled code.
Can be set to “yes” or “no”. Indicates
whether files destined to reside in
DESTDIR/usr/share will be
built and installed during a build. If set to “no”,
then all of MKCATPAGES,
MKDOC, MKINFO,
MKMAN and MKNLS will be set
to “no” unconditionally.
Default: “yes”
Can be set to “yes” or “no”. For X builds, decides if the TrueType bytecode interpreter is turned on. See freetype.org for details.
Default: “no”
Can be set to “yes” or “no”. Indicates
whether an unprivileged install will occur. The user, group,
permissions and file flags will not be set on the installed items;
instead the information will be appended to a file called
METALOG in DESTDIR. The
contents of METALOG are used during the
generation of the distribution tar files to ensure that the
appropriate file ownership is stored.
Default: “no”
Can be set to “yes” or “no”. Indicates
whether all install operations intended to write to
DESTDIR will compare file timestamps before
installing, and skip the install phase if the destination files
are up-to-date. This also has implications on full builds (See
below).
Default: “no”
Can be set to “yes” or “no”. Indicates
whether X11R6 is built from X11SRCDIR.
Default: “yes”
Directory to hold the host tools, once built. This directory
should be unique to a given host system and NetBSD source tree.
(However, multiple targets may share the same
TOOLDIR; the target-dependent files have unique
names). If unset, a default based on the uname(1) information
of the host platform will be created in the
.OBJDIR of src.
Default: Unset.
Indicates whether the tools specified by
TOOLDIR should be used as part of a build in
progress. Must be set to “yes” if cross-compiling.
| yes | Use the tools from TOOLDIR.
|
| no | Do not use the tools from
TOOLNAME, but refuse to build native
compilation tool components that are version-specific for
that tool. |
| never | Do not use the tools from
TOOLNAME, even when building native
tool components. This is similar to the traditional NetBSD
build method, but does not verify that the compilation
tools in use are up-to-date enough in order to build the
tree successfully. This may cause build or runtime problems
when building the whole NetBSD source tree. |
Default: “yes” if building all or part of a whole
NetBSD source tree (detected automatically); “no”
otherwise (to preserve traditional semantics of the
bsd.*.mk make(1) include files).
Directory containing the X11R6 source. The main X11R6 source is
found in
X11SRCDIR/xfree/xc.
Default: “usr/xsrc”
The following variables only affect the top level
Makefile and do not affect manually building
subtrees of the NetBSD source code.
Location for the “make installworld” target to install to.
Default: “/”
Can be set to “yes” or “no”. Indicates whether object directories will be created automatically (via a “make obj” pass) at the start of a build.
Default: “no”
Can be set to “yes” or “no”. If set,
then addition to the effects described for
MKUPDATE=yes above, this implies the effect of
NOCLEANDIR (i.e., “make cleandir”
is avoided).
Default: “no”
If set, avoids the “make cleandir” phase of a full build. This has the effect of allowing only changed files in a source tree to recompiled. This can speed up builds when updating only a few files in the tree.
Default: Unset
If set, avoids the “make distrib-dirs” of a full
build. This skips running mtree(8) on
DESTDIR, useful on systems where building as an
unprivileged user, or where it is known that the system wide mtree
files have not changed.
Default: Unset
If set, avoids the “make includes” phase of a full
build. This has the effect of preventing make(1) from
thinking that some programs are out-of-date simply because system
include files have changed. However, this option should not be
trusted when updating the entire NetBSD source tree arbitrarily; it
is suggested to use MKUPDATE=yes in that case.
Default: Unset
If set, specifies the directory to which a release(7) layout will be written at the end of a “make release”.
Default: Unset
Set to “yes” on platforms for which there is no working in-tree toolchain, or if you need/wish using native system toolchain (i.e. non-cross tools available via your shell search path).
Default: depends on target platform; on platforms with in-tree toolchain is set to “no”.