FreeBSD Offline Package Downloader: Automate Fetching, Transfer, and Local ReposKeeping FreeBSD systems up to date in environments with limited or no Internet connectivity (air-gapped, isolated networks, high-security facilities, or remote locations with intermittent links) requires a reliable offline package workflow. This article shows practical, automated methods to fetch packages on a connected machine, prepare them for transfer, install them on offline FreeBSD systems, and maintain local repositories. You’ll find concrete commands, scripts, directory layouts, checksum handling, and tips for scaling the process.
Why offline package management for FreeBSD?
- Security: Air-gapped systems often cannot be exposed to external repositories.
- Stability: Controlled package versions prevent unexpected changes.
- Bandwidth: Download large packages once and reuse them across many systems.
- Compliance: Regulated environments may require verified, auditable updates.
Overview of the workflow
- Prepare a connected host (the “download host”) with FreeBSD or compatible tools.
- Fetch required packages (pkg and binary packages for ports, plus dependencies).
- Package and verify the downloads (checksums, signatures).
- Transfer files to the offline host(s) via secure removable media or internal network.
- Serve or configure a local repository on the offline host for pkg to use.
- Automate recurring fetches and repository rotation.
Prerequisites and assumptions
- Download host has Internet access and runs FreeBSD (or a compatible environment with fetch/curl/rsync).
- Offline host(s) run FreeBSD and use the pkg package manager.
- You have sufficient removable media (external disk) or a trusted internal transfer mechanism.
- You have root or sudo access on both ends to run package and repository commands.
Directory layout and naming conventions
Keep a consistent layout to simplify automation and verification:
/srv/offline-pkgs/ — root for the current mirror snapshot
/srv/offline-pkgs/packages/ — pkg binary packages (pkg.txz files)
/srv/offline-pkgs/sets/ — optional for FreeBSD base/sets if needed
/srv/offline-pkgs/repo/ — repository metadata (repo.txz, meta files)
/srv/offline-pkgs/checksums/— checksums and signature files
/srv/offline-pkgs/logs/ — logs for fetch jobs
Use date-stamped snapshots when producing regular updates, e.g. /srv/offline-pkgs/2025-08-30/.
Fetching packages: methods
Two common methods:
- 1) Mirror a portion of the official FreeBSD package repository (rsync or ftp).
- 2) Use pkg to fetch specific packages and their dependencies into a local directory.
Method A — rsync (recommended for full or partial mirrors)
- rsync is efficient and preserves repo structure. Official FreeBSD package mirrors allow rsync.
- Example rsync command to mirror amd64 quarterly repo (adjust paths and mirror):
rsync -avz --delete rsync://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages/quarterly/ /srv/offline-pkgs/packages/
Notes:
- Use the correct architecture (amd64, arm64, etc.) and branch (latest, quarterly, quarterly/12 for branches).
- Add –partial and –progress for large transfers, and –bwlimit if you must throttle.
Method B — pkg fetch (recommended for selective packages)
-
Use a connected FreeBSD host to resolve dependencies and download only needed pkgs.
-
Example:
# create workdir mkdir -p /srv/offline-pkgs/packages # download package and deps into directory without installing export PACKAGESITE="pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest" pkg fetch -o /srv/offline-pkgs/packages -y -d nginx
-
pkg fetch resolves dependencies and stores .txz packages. Use -o to set output directory and -d to fetch dependencies.
Tip: To fetch a list of packages, script through pkg install -n or pkg-query tools to generate the list first.
Creating repository metadata for offline use
pkg repos need metadata. After downloading .txz packages into packages/ run pkg repo to generate repo metadata.
Example:
cd /srv/offline-pkgs/packages pkg repo . # produces meta.txz, packagesite.yml, digests.sha256, etc.
If you mirrored an existing repo via rsync, metadata will already be present; running pkg repo on a snapshot ensures consistency and updates digests.
Signing repository metadata (optional but recommended):
- Use signify (or OpenPGP) to sign metadata for tamper-evidence.
- Example with signify (replace KEY and SECRET):
signify -S -s /path/to/private.sec -m /srv/offline-pkgs/packages/repo.meta
FreeBSD uses signify for base system releases; for pkg repositories you can adopt similar signing practices and record signatures in checksums/.
Packaging and transfer
Prepare a portable snapshot:
- Create a tarball or disk image of /srv/offline-pkgs/
. - Include checksums and a manifest file listing packages and their SHA256.
Example manifest and tar:
cd /srv/offline-pkgs sha256sum packages/*.txz > checksums/sha256-2025-08-30.txt tar -C /srv/offline-pkgs -cJf offline-pkgs-2025-08-30.tar.xz packages repo checksums
For large repos, format an external USB drive with UFS/Ext4/NTFS as appropriate, rsync files to it, then unmount and transfer.
Security: verify the destination’s mount options and filesystem support large files.
Installing on the offline host
Option A — Local directory as repository
- Copy packages/ and repo metadata to the offline host under /var/db/pkg-repo or /usr/local/packages.
- Configure pkg to use the local repository by editing /usr/local/etc/pkg/repos/local.conf (or /etc/pkg/FreeBSD.conf) and pointing to file:// URL.
Example repo config (/etc/pkg/FreeBSD-local.conf):
FreeBSD-local: { url: "file:///usr/local/offline-pkgs/packages", enabled: yes }
Then run:
pkg update -f pkg install nginx
Option B — Serve via simple HTTP on an isolated host
- On the offline network you can run a tiny web server to serve the packages directory (python -m http.server or an httpd). Configure pkg config to point to http://LOCAL-SERVER/packages.
Option C — Install direct from .txz files
- Use pkg add for single packages:
pkg add /path/to/package.txz
This bypasses dependency resolution, so ensure dependencies are present.
Automating recurring fetch and transfer
Create a script run by cron or CI to produce snapshots. Core steps:
- Update package list from remote repo (rsync or pkg fetch loop).
- Run pkg repo to rebuild metadata.
- Generate checksums/manifests.
- Create an atomic snapshot directory (use temp dir + mv).
- Optionally compress or rsync to a staging server or attached drive.
Example fetch script (selective fetch with pkg fetch)
#!/bin/sh set -eu SNAP=/srv/offline-pkgs/$(date +%F) mkdir -p "$SNAP/packages" "$SNAP/checksums" PACKAGES="nginx python39 bash" for p in $PACKAGES; do pkg fetch -o "$SNAP/packages" -d -y "$p" done cd "$SNAP/packages" sha256sum *.txz > ../checksums/sha256.txt pkg repo . mv "$SNAP" /srv/offline-pkgs/ready/$(date +%F)
Add logging and error handling as needed. Test the script on a sandbox before production.
Verifying integrity on the offline host
Always verify checksums after transfer:
sha256 -c /path/to/checksums/sha256-2025-08-30.txt
Verify repository metadata signature if used. Test installing in a VM or staging host prior to pushing to production.
Tips for scaling and maintenance
- Use snapshots and retain multiple dates to allow rollback.
- Prune old snapshots to save space; keep a retention policy (e.g., last 3 monthly snapshots).
- For many offline hosts, set up a local mirror server on the isolated LAN rather than copying to each host.
- Automate package lists per host-role so each offline host gets only what it needs.
- Track installed package versions with pkg version -v and compare against snapshot manifests for auditing.
- Consider binary package caches for ports-built software: build on a builder host and include resulting packages in the offline repo.
Example: Full end-to-end scenario
-
On download host:
- Use a scripted list of required software for web servers, monitoring, and tooling.
- Run pkg fetch loop to collect all packages and deps into /srv/offline-pkgs/2025-08-30/packages.
- Run pkg repo . and generate checksums.
- Tar the snapshot and copy to encrypted USB.
-
On offline network:
- Mount USB on internal staging server.
- rsync to local mirror server: /var/www/packages/.
- Configure offline hosts’ /etc/pkg/FreeBSD-local.conf to point to http://mirror.internal/packages.
- Run pkg update -f and install or upgrade packages.
Troubleshooting common issues
- Missing dependencies when using pkg add: ensure you fetched dependency packages or use a repo and pkg install.
- Architecture mismatch: verify pkg packages are built for the offline host’s architecture (amd64 vs arm64).
- Corrupt transfers: use sha256 checks after copying and consider verifying filesystem journaling.
- pkg repo errors: run pkg repo -v . to see verbose diagnostics; ensure repository root contains packages and metadata files.
Security considerations
- Sign metadata and keep private keys secure.
- Use encrypted removable media for transfers if the physical chain-of-custody is not trusted.
- Limit which systems can write to the local mirror; prefer read-only mounts where possible.
- Audit manifests and keep logs of snapshot creation and transfer.
Conclusion
Automating FreeBSD offline package downloading and repository creation reduces manual work, improves security and reproducibility, and scales from single servers to entire isolated fleets. Key elements: choose the right fetch method (rsync for large mirrors, pkg fetch for selective needs), generate repository metadata with pkg repo, verify integrity with checksums/signatures, and automate snapshot creation and transfer. With a robust workflow you can keep air-gapped FreeBSD systems secure, up to date, and auditable.