Radish alpha
H
HardenedBSD-pkg
rad:z3QDZAW2FAfuLvihrhiyDC9fAD8G9
HardenedBSD Package Manager
Radicle
Git
Commits › 2f108a1
pkg-repository(5): large update of pkg-repository(5) documentation
Baptiste Daroussin committed 2 months ago
commit 2f108a168d29a7fe3454b6ce89651a5fb6214821
parent ba68da9
1 file changed +182 -135
modified docs/pkg-repository.5
@@ -26,10 +26,8 @@
.Nm "Package repositories"

used by the

.Xr pkg 8
- 
program consist of one or more collections of
- 
.Dq package tarballs
- 
together with package catalogues and optionally various other
- 
collected package metadata.
+ 
program consist of one or more collections of packages together with
+ 
package catalogues and various other collected package metadata.

.Pp

Each collection consists of packages suitable for installation on a

specific system
@@ -38,7 +36,7 @@ a combination of operating system, CPU architecture, OS version, word
size and, for certain processors, endianness or similar attributes.

.Pp

The package collections are typically made available to users for
- 
download via a web or FTP server although various other means of access
+ 
download via a web server although various other means of access

may be employed.

Encoding the

.Sy ABI
@@ -54,113 +52,107 @@ Repositories may be mirrored over several sites:
.Nm pkg

has built-in support for discovering available mirrors dynamically

given a common URL by several mechanisms.
- 
.Sh FILESYSTEM ORGANIZATION
- 
Only very minimal constraints on repository layout are prescribed by
- 
.Nm pkg .
- 
The following constraints are all that must be met:
- 
.Bl -bullet
- 
.It
- 
A repository may contain several package collections with parallel
- 
.Cm REPOSITORY_ROOTs
- 
in order to support diverse system
- 
.Cm ABIs .
- 
.It
- 
All of the content for one
- 
.Sy ABI
- 
should be accessible in a filesystem or URL hierarchy beneath the
- 
.Cm REPOSITORY_ROOT .
- 
.It
- 
All packages available beneath one
- 
.Cm REPOSITORY_ROOT
- 
should be binary compatible with a specific system
- 
.Cm ABI .
- 
.It
- 
The repository catalogue is located at the apex of the
- 
repository, at a specific location relative to the
- 
.Cm REPOSITORY_ROOT .
- 
.El
+ 
.Sh REPOSITORY FORMAT
+ 
The current repository format is version 2.
+ 
It is generated by
+ 
.Xr pkg-repo 8 .

.Pp
- 
Package catalogues contain the paths relative to the
+ 
A repository consists of package files
+ 
.Pq Pa .pkg
+ 
and a set of catalogue files at the root of the repository.
+ 
.Ss Catalogue Files
+ 
The following files are generated at the

.Cm REPOSITORY_ROOT
- 
for each package, allowing the full URL for downloading the
- 
package to be constructed.
- 
.Pp
- 
Where a package may be applicable to more than one
- 
.Sy ABI
- 
(e.g., it contains only text files) symbolic or hard links, URL mappings
- 
or other techniques may be utilised to avoid duplication of storage.
- 
.Pp
- 
Although no specific filesystem organization is required, the usual
- 
convention (inherited from
- 
.Xr pkg-install 8 )
- 
is to create a filesystem hierarchy thus:
- 
.Bl -tag -width "REPOSITORY"
- 
.It Pa $REPOSITORY_ROOT/All
- 
One directory that contains every package available from the
- 
repository for that
- 
.Sy ABI .
- 
Packages are stored as package tarballs identified by name and
- 
version.
- 
This directory may contain several different versions of each package
- 
accumulated over time, but the repository catalogue will only record
- 
the latest version for each distinct package name.
- 
.It Pa $REPOSITORY_ROOT/Latest/
- 
May contain symbolic links to the latest versions of packages in the
- 
.Pa All
- 
directory.
- 
Symbolic links contain a
- 
.Sq latest link
- 
style name only, without version.
- 
As the whole
- 
.Sq latest link
- 
concept is rendered obsolete by
- 
.Nm pkg ,
- 
this will usually contain only the
- 
.Pa pkg.txz
- 
link, used for bootstrapping
- 
.Nm pkg
- 
itself on a new system.
- 
.It Pa $REPOSITORY_ROOT/packagesite.txz
- 
Contains a single file, usually named
+ 
by
+ 
.Xr pkg-repo 8 :
+ 
.Bl -tag -width "packagesite.pkg"
+ 
.It Pa meta.conf
+ 
A UCL file describing the repository metadata.
+ 
It contains the following fields:
+ 
.Bl -tag -width "packing_format" -compact
+ 
.It Cm version
+ 
The repository format version (currently 2).
+ 
.It Cm packing_format
+ 
The compression format used for catalogue archives
+ 
.Pq tzst, txz, tbz, tgz, or tar .
+ 
.It Cm manifests
+ 
The name of the uncompressed manifests file
+ 
.Pq default: Pa packagesite.yaml .
+ 
.It Cm manifests_archive
+ 
The base name of the compressed manifests archive
+ 
.Pq default: Pa packagesite .
+ 
.It Cm data
+ 
The name of the uncompressed data file
+ 
.Pq default: Pa data .
+ 
.It Cm data_archive
+ 
The base name of the compressed data archive
+ 
.Pq default: Pa data .
+ 
.It Cm filesite
+ 
The name of the uncompressed file listing
+ 
.Pq default: Pa filesite.yaml .
+ 
.It Cm filesite_archive
+ 
The base name of the compressed file listing archive
+ 
.Pq default: Pa filesite .
+ 
.It Cm maintainer
+ 
Optional maintainer string.
+ 
.It Cm source
+ 
Optional source string.
+ 
.It Cm source_identifier
+ 
Optional source identifier.
+ 
.It Cm revision
+ 
Optional integer revision number.
+ 
.It Cm eol
+ 
Optional end-of-life timestamp (Unix epoch).
+ 
.El
+ 
.It Pa packagesite.pkg
+ 
A compressed archive containing

.Pa packagesite.yaml ,
- 
a concatenation of the
- 
.Pa +MANIFEST
- 
files from the packages in the repository.
- 
Each manifest is represented as a single-line
- 
.Cm JSON
- 
text (no carriage returns or line feeds are used as
- 
whitespace within the
- 
.Cm JSON
- 
text),
+ 
a concatenation of the manifests from all packages in the repository.
+ 
Each manifest is represented as a single-line compact JSON text,

and the manifests are separated by newlines.
- 
The complete file is not a valid
- 
.Cm JSON
- 
text.
- 
This is used by
- 
.Nm pkg-1.1
- 
or later.
- 
.It Pa $REPOSITORY_ROOT/filesite.txz
- 
(Optional).
- 
Contains a single file, usually named
+ 
.It Pa data.pkg
+ 
A compressed archive containing the
+ 
.Pa data
+ 
file in JSON format with the following top-level keys:
+ 
.Bl -tag -width "expired_packages" -compact
+ 
.It Cm groups
+ 
An array of package group definitions (if configured).
+ 
.It Cm expired_packages
+ 
An array of expired package entries (if configured).
+ 
.It Cm packages
+ 
An array of all package manifests.
+ 
.El
+ 
.It Pa filesite.pkg
+ 
(Optional, generated with
+ 
.Fl l
+ 
flag to
+ 
.Xr pkg-repo 8 ) .
+ 
A compressed archive containing

.Pa filesite.yaml ,
- 
a concatenation of the file lists
- 
from the packages in the repository.
- 
Each file list is represented as a single-line
- 
.Cm JSON
- 
text (no carriage returns or line feeds are used as
- 
whitespace within the
- 
.Cm JSON
- 
text),
- 
and the file lists are concatenated with no delimiters.
- 
The complete file is not a valid
- 
.Cm JSON
- 
text.
+ 
a concatenation of the file lists from all packages in the repository.

.El

.Pp
- 
The repository may optionally contain sub-directories corresponding to
- 
the package origins within the
- 
.Os
- 
ports tree.
+ 
Compressed archives use the
+ 
.Pa .pkg
+ 
extension and default to Zstandard
+ 
.Pq tzst
+ 
compression.
+ 
When a repository uses a different compression format, the
+ 
.Pa meta.conf
+ 
file indicates which format is used.
+ 
.Pp
+ 
Repositories may optionally be signed.
+ 
See
+ 
.Xr pkg-repo 8
+ 
for details on signing repositories.
+ 
.Ss Package Layout
+ 
Packages are stored as
+ 
.Pa .pkg
+ 
files within the repository hierarchy.
+ 
The repository catalogue records the relative path from the
+ 
.Cm REPOSITORY_ROOT
+ 
to each package, allowing the full URL for downloading to be
+ 
constructed.

.Pp

Each of the packages listed in the repository catalogue must have a

unique
@@ -168,37 +160,93 @@ unique
There are no other constraints: package sets are not required to be

either complete (i.e., with all dependencies satisfied) or

self-consistent within a single repository.
+ 
.Ss Hash Mode
+ 
When
+ 
.Xr pkg-repo 8
+ 
is invoked with the
+ 
.Fl h
+ 
flag, packages are reorganized into a
+ 
.Pa Hashed
+ 
subdirectory with a SHA256 hash appended to the filename, separated
+ 
by a tilde
+ 
.Pq Sq ~ .
+ 
With the additional
+ 
.Fl s
+ 
flag, symbolic links are created in the original location pointing
+ 
to the hashed file.
+ 
.Ss Groups
+ 
Package groups allow organizing packages into logical collections.
+ 
Groups are defined in UCL files within a directory passed to
+ 
.Xr pkg-repo 8
+ 
via the
+ 
.Fl g
+ 
flag.
+ 
Each group file contains:
+ 
.Bl -tag -width "requires" -compact
+ 
.It Cm name
+ 
The group name (required).
+ 
.It Cm comment
+ 
A description of the group (required).
+ 
.It Cm requires
+ 
An array of required group names.
+ 
.It Cm depends
+ 
An array of dependent group names.
+ 
.El
+ 
.Ss Expired Packages
+ 
Repositories can declare packages as expired.
+ 
Expired packages are defined in UCL files and contain:
+ 
.Bl -tag -width "replaced_by" -compact
+ 
.It Cm name
+ 
The package name (required).
+ 
.It Cm reason
+ 
Why the package was expired.
+ 
.It Cm replaced_by
+ 
The replacement package, if any.
+ 
.El

.Sh REPOSITORY ACCESS METHODS

.Nm pkg
- 
uses standard network protocols for repository access.
- 
Any URL scheme understood by the
- 
.Xr fetch 3
- 
library may be used
- 
.Cm ( HTTP ,
- 
.Cm HTTPS ,
- 
.Cm FTP
- 
or
- 
.Cm FILE )
- 
as well as remote access over
- 
.Cm SSH .
- 
See
- 
.Xr fetch 3
- 
for a description of additional environment variables, including
- 
.Ev FETCH_BIND_ADDRESS ,
- 
.Ev FTP_LOGIN ,
- 
.Ev FTP_PASSIVE_MODE ,
- 
.Ev FTP_PASSWORD ,
- 
.Ev FTP_PROXY ,
- 
.Ev ftp_proxy ,
- 
.Ev HTTP_AUTH ,
+ 
uses
+ 
.Xr libcurl 3
+ 
for HTTP and HTTPS access, and has built-in support for
+ 
.Cm SSH
+ 
and local
+ 
.Cm file://
+ 
access.
+ 
.Pp
+ 
The following URL schemes are supported:
+ 
.Cm pkg+http:// ,
+ 
.Cm pkg+https:// ,
+ 
.Cm https:// ,
+ 
.Cm http:// ,
+ 
.Cm file:// ,
+ 
.Cm ssh://
+ 
and
+ 
.Cm tcp:// .
+ 
.Pp
+ 
The
+ 
.Cm pkg+http://
+ 
and
+ 
.Cm pkg+https://
+ 
schemes are used with
+ 
.Cm SRV
+ 
mirror type to enable DNS-based mirror discovery.
+ 
.Pp
+ 
Additional URL schemes can be configured via the
+ 
.Cm VALID_URL_SCHEME
+ 
option in
+ 
.Xr pkg.conf 5 .
+ 
.Pp
+ 
The following environment variables affect HTTP and HTTPS access:

.Ev HTTP_PROXY ,

.Ev http_proxy ,
- 
.Ev HTTP_PROXY_AUTH ,
- 
.Ev HTTP_REFERER ,
- 
.Ev HTTP_USER_AGENT ,
- 
.Ev NETRC ,
- 
.Ev NO_PROXY No and
+ 
.Ev HTTPS_PROXY ,
+ 
.Ev NO_PROXY
+ 
and

.Ev no_proxy .
+ 
The
+ 
.Cm HTTP_USER_AGENT
+ 
can be configured in
+ 
.Xr pkg.conf 5 .

.Sh REPOSITORY MIRRORING

Multiple copies of a repository can be provided for resilience or

to scale up site capacity.
@@ -206,8 +254,8 @@ Two schemes are provided to auto-discover sets of mirrors given a
single repository URL.

.Bl -tag -width "HTTP"

.It Cm HTTP
- 
The repository URL should download a text document containing a sequence
- 
of lines beginning with
+ 
The repository URL should download a text document containing a
+ 
sequence of lines beginning with

.Sq URL:

followed by any amount of white space and one URL for a repository

mirror.
@@ -215,7 +263,7 @@ Any lines not matching this pattern are ignored.
Mirrors are tried in the order listed until a download succeeds.

.It Cm SRV

For an SRV mirrored repository where the URL is specified as
- 
.Pa http://pkgrepo.example.org/
+ 
.Pa pkg+http://pkgrepo.example.org/

.Cm SRV

records should be set up in the DNS:

.Bd -literal -offset indent
@@ -283,7 +331,6 @@ to
.Sy true

in

.Pa pkg.conf .
- 
.Pp

.Sh SEE ALSO

.Xr pkg_create 3 ,

.Xr pkg_printf 3 ,