Advanced configuration and troubleshooting of APT package manager

APT is short for Advanced Package Tool and is the reason why Debian Linux became so popular in the first place. Thanks to APT, software could be installed with automatic dependency installation. Most users know how to install and upgrade the packages using the apt commands, but only few know that APT is highly configurable.

This tutorial was made for both beginners and advanced users of APT and contains basic information but also advanced configuration examples and problem solutions.

The default (distribution) repositories

The official repositories are (by default) configured in the /etc/apt/sources.list file. They contain the type of package, the URLs of the repositories, the version of the distribution and the repository "name".

A typical Debian repository list (here for Debian 9/Stretch) looks like this:

ck@debian:~$ cat /etc/apt/sources.list
deb http://ftp.ch.debian.org/debian/ stretch main contrib non-free 
deb-src http://ftp.ch.debian.org/debian/ stretch main contrib non-free 

deb http://security.debian.org/debian-security stretch/updates main contrib non-free
deb-src http://security.debian.org/debian-security stretch/updates main contrib non-free

# stretch-updates, previously known as 'volatile'
deb http://ftp.ch.debian.org/debian/ stretch-updates main contrib non-free
deb-src http://ftp.ch.debian.org/debian/ stretch-updates main contrib non-free

And here's a typical Ubuntu repository list on an Ubuntu 20.04 (Focal):

ck@ubuntu:~$ cat /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://ch.archive.ubuntu.com/ubuntu/ focal main restricted
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://ch.archive.ubuntu.com/ubuntu/ focal-updates main restricted
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://ch.archive.ubuntu.com/ubuntu/ focal universe
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal universe
deb http://ch.archive.ubuntu.com/ubuntu/ focal-updates universe
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://ch.archive.ubuntu.com/ubuntu/ focal multiverse
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal multiverse
deb http://ch.archive.ubuntu.com/ubuntu/ focal-updates multiverse
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
#deb http://ch.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
# deb-src http://ch.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu focal partner
# deb-src http://archive.canonical.com/ubuntu focal partner

deb http://security.ubuntu.com/ubuntu focal-security main restricted
# deb-src http://security.ubuntu.com/ubuntu focal-security main restricted
deb http://security.ubuntu.com/ubuntu focal-security universe
# deb-src http://security.ubuntu.com/ubuntu focal-security universe
deb http://security.ubuntu.com/ubuntu focal-security multiverse
# deb-src http://security.ubuntu.com/ubuntu focal-security multiverse

Difference between deb and deb-src

In the Debian distribution, the deb-src repositories are by default enabled, yet on Ubuntu machines they are by default disabled (commented). By using a deb-src repository allows a user to install the source package(s) of a software, including the changes made specifically for the distribution.

To install a source package (here: curl), enable the deb-src repositories and then install using apt-get source (instead of install). This will then show the curl source code and the distribution adjustments in the debian.tar.xz file:

ubuntu@focal:~$ apt-get source curl
ubuntu@focal:~$ ls -l
total 4044
drwxrwxr-x 16 ubuntu ubuntu    4096 Jun  4 11:52 curl-7.68.0
-rw-r--r--  1 ubuntu ubuntu   31648 Apr 15  2020 curl_7.68.0-1ubuntu2.debian.tar.xz
-rw-r--r--  1 ubuntu ubuntu    2071 Apr 15  2020 curl_7.68.0-1ubuntu2.dsc
-rw-r--r--  1 ubuntu ubuntu 4096350 Feb 23  2020 curl_7.68.0.orig.tar.gz

Debian's main, contrib and non-free

From the output of /etc/apt/sources.list above, there are a couple of repository names at the end of each repository line. But what is the meaning behind these repository names?

• main: This repository contains packages which follow the Debian Free Software Guidelines (DFSG) which only allows pure Open Source Software.
• contrib: This repository contains packages which basically follow the DFSG but have a dependency to another package which is not listed in the "main" repository.
• non-free: This repository contains packages which are not following or are not compatible with the DFSG.

Ubuntu's main, restricted, universe and multiverse

Ubuntu does not follow a strict DFSG and uses a different terminology on the repository names:

• main: Free and Open Source Software, maintained and supported by Canonical (the company behind Ubuntu)
• restricted: Proprietary software, usually drivers for hardware devices (comparable to Debian's non-free)
• universe: Free and Open Source Software, maintained by the Ubuntu community
• multiverse: Non Open Source Software, could be restricted by copyright

Adding custom repositories

APT supports custom repositories. However to keep the official system repositories and custom added repositories apart, additional repositories should not be added into /etc/apt/sources.list. Instead they should be added into the /etc/apt/sources.list.d/ directory (not file!).

To add a custom repository simply create a new file in the /etc/apt/sources.list.d/ directory with a file extension .list. In the following example, the Grafana APT repository is added:

ubuntu@focal:~$ echo "deb http://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Note: Ubuntu offers the command add-apt-repository to add custom repositories. But this does basically the same.

Depending on the custom repository, a specific distribution version needs to be defined (examples: Icinga2, InfluxDB). Others are more generic and simply require a version related to the custom repository (examples: Elastic, Grafana).

HTTPS repositories

APT also supports repositories using encrypted HTTPS. In order to use these, the package apt-transport-https needs to be installed. The way to add the repository is the same as a HTTP repo, just the URL contains https. Here an example with the InfluxDB repository:

ubuntu@focal:~$ echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee -a /etc/apt/sources.list.d/influxdb.list

Adding APT Keys

On most (professional) APT repositories, the packages are signed with a GPG key. This helps to ensure that only signed packages are installed. Only if APT "knows" the package signer, it will allow a package to be installed.

By default only the keys of the official distribution repositories are installed. When custom repositories are added to a system, the key(s) of this custom repository needs to be installed.

A common way to do this is to download the key directly from the repository and then import it to APT. In the following example, the public GPG key is downloaded using wget from the InfluxDB repositories:

ubuntu@focal:~$ wget -O - https://repos.influxdata.com/influxdb.key | sudo apt-key add -
OK

If you hit a certificate error (e.g. unable to get local issuer certificate), you can also run wget –no-check-certificate to skip the certificate check.

If the key was successfully added to APT, the apt-key command returns OK in the output.

Note: The name of the public GPG key can vary and heavily depends on the repository maintainer. Many variations exists, for example: reponame.key, gnupg.key, public.gpg and many more. The GPG key can usually be found in the top level directory of a custom repository.

Another way to install the repository key is to use apt-key with the adv sub-command. This uses HKP (HTTP Keyserver Protocol) instead of HTTP in the background to fetch the keys from central key servers. The following example installs the MariaDB repository key from the Ubuntu keyserver:

ubuntu@focal:~$ sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
Executing: /tmp/apt-key-gpghome.uNqEenpES4/gpg.1.sh --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
gpg: key F1656F24C74CD1D8: public key "MariaDB Signing Key <signing-key@mariadb.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1

This method however requires clear instructions from the repository documentation.

Advanced APT configuration

The default APT settings are usually enough to use the package manager. But depending on your environment you might have to adjust the base configuration.

Note: Always run apt-get update or apt update after a configuration change.

Using an outgoing HTTP Proxy

APT supports outgoing HTTP/HTTPS proxies with the internal "Acquire" setting. If you are using both http and https repositories, make sure both lines are defined in either /etc/apt/apt.conf or in a file inside the directory /etc/apt/apt.conf.d/ (without file extension).

A proxy exception for a specific repository (here packages.icinga.com) can also be defined by using the DIRECT option.

ck@focal:~$ cat /etc/apt/apt.conf.d/10proxy
Acquire::http::Proxy "http://proxy.example.com:8080";
Acquire::https::Proxy "http://proxy.example.com:8080";
Acquire::https::Proxy::packages.icinga.com DIRECT;

If your outgoing proxy requires authentication, APT also supports proxy authentication:

ck@focal:~$ cat /etc/apt/apt.conf.d/10proxy
Acquire::http::proxy "http://proxyuser:password@proxyserver:8080/";
Acquire::https::proxy "http://proxyuser:password@proxyserver:8080/";
Acquire::ftp::proxy "http://proxyuser:password@proxyserver:8080/"; 

Skipping TLS certificate check

When using repositories using HTTPS, there could be problems when APT is unable to verify the server certificate of the repository server. An error "server certificate verification failed" could be showing up and packages and updates cannot be installed from the affected repository.

In this situation, an additional APT configuration to skip the TLS verification can be added. This setting works on a per-repository basis:

ck@focal:~$ cat /etc/apt/apt.conf.d/99influxdata-cert
Acquire::https::repos.influxdata.com::Verify-Peer "false";

In this case, the remote server certificate of the InfluxDB repository (repos.influxdata.com) will be skipped.

Another reason for this error to be showing up is that the certificate chain verification results in an error. When a (locally installed) Root CA expires, this can be the case. This was recently seen in September/October 2021 when the Let's Encrypt Root CA X3 expired.

Setting a different repository priority (pinning)

Some custom repositories might contain the same software packages as the official repositories from the distribution. But the packages from the custom repositories are newer and you prefer the custom repositories over the distribution repos for certain packages. This is where "APT Pinning" comes into play.

Basically APT is being told which repository has a higher priority using a higher "pin" priority. The following example shows that all packages (*) from the Proxmox repositories (enterprise.proxmox.com and download.proxmox.com) have a higher priority than the official Debian repos (verify the used distribution repos in /etc/apt/sources.list):

ck@buster:~$ cat /etc/apt/preferences.d/proxmox
Package: *
Pin: origin enterprise.proxmox.com
Pin-Priority: 900

Package: *
Pin: origin download.proxmox.com
Pin-Priority: 850

Package: *
Pin: origin security.debian.org
Pin-Priority: 800

Package: *
Pin: origin ftp.ch.debian.org
Pin-Priority: 700 

Note that APT pinning should happen in the /etc/apt/preferences.d/ directory.

Ignoring package installation from a certain repository

By using a negative pin-priority, APT can be told to not install a package from a specific repository. In the following pinning example, the MySQL repositories are defined with a higher pin-priority (default pin-priority is 500) and the mysql packages should be ignored from the official Ubuntu repositories:

ck@focal:~$ cat /etc/apt/preferences.d/mysql
Package: *
Pin: origin repo.mysql.com
Pin-Priority: 900

Package: libmysqlclient* mysql*
Pin: origin ch.archive.ubuntu.com
Pin-Priority: -1

Package: libmysqclient* mysql*
Pin: origin security.ubuntu.com
Pin-Priority: -1

Without this setting, APT could install certain MySQL packages from the Ubuntu, others from the MySQL repositories and cause dependency conflicts.

Enabling or disabling automatic package updates

APT can be configured to automatically install package updates. This method is called "Unattended Upgrades" and requires the unattended-upgrades package to be installed:

ck@focal:~$ dpkg -l|grep unattended
ii  unattended-upgrades   2.3ubuntu0.1  all  automatic installation of security upgrades

The Unattended Upgrades configuration actually consists of two parts:

1. Unattended Upgrades itself, including the configuration of which packages should be installed (or excluded from automatic installation), from which sources.
2. The automatic "periodic launch" of the unattended upgrades, called Auto Upgrades.

The unattended upgrades are highly configurable with the config file /etc/apt/apt.conf.d/50unattended-upgrades. For example if you want to only use automatic security upgrades but avoid automatic upgrades of the Linux Kernel, use the following example configuration:

ck@stretch:~$ cat /etc/apt/apt.conf.d/50unattended-upgrades 
Unattended-Upgrade::Origins-Pattern {
	"origin=Debian,a=stretch,label=Debian-Security";
};
Unattended-Upgrade::Package-Blacklist {
	"linux-image-amd64";
};
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";

Now that the Unattended Upgrades are configured, they need to be enabled by the Auto Upgrades. The configuration file /etc/apt/apt.conf.d/20auto-upgrades with the following content is already enough:

ck@stretch:~$ cat /etc/apt/apt.conf.d/20auto-upgrades 
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

There is no cron job or similar necessary to enable the automatic package upgrades. This is handled by Systemd's apt-daily-upgrade.timer, which checks for the existence and configuration of /etc/apt/apt.conf.d/50unattended-upgrades and /etc/apt/apt.conf.d/20auto-upgrades.

To disable the automatic upgrades, simply change /etc/apt/apt.conf.d/20auto-upgrades and set the value from 1 to 0:

ck@stretch:~$ cat /etc/apt/apt.conf.d/20auto-upgrades 
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";

Troubleshooting APT

As with every software, APT can also run into problems in certain situations. However the APT error messages are usually very useful and help to quickly spot the source of the problem. Here are a couple of troubleshooting tips.

Error: files list file for package contains empty filename

When installing or upgrading packages, APT also checks the currently installed packages by using the information from dpkg, located in /var/lib/dpkg/info/. If one of these list files is corrupt, APT fails with an error:

Preconfiguring packages ...
dpkg: unrecoverable fatal error, aborting:
 files list file for package `psmisc' contains empty filename
E: Sub-process /usr/bin/dpkg returned an error code (2) 

A situation like this could happen, when /var runs out of available disk space.

Error: Could not get lock /var/lib/apt/lists/lock

This error is shown by APT when another process is already using apt. Whenever APT is running, it creates a lock file – if a lock file already exists, APT exits with an error could not get lock:

ck@focal:~$ sudo apt-get update
Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/ 

Use the ps command to find out which other process is currently locking apt.

ck@focal:~$ ps auxf| grep -i apt
root     28044  0.0  0.0   4508  1700 ?        Ss   03:34   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily
root     28077  0.0  0.1  44628  7344 ?        S    03:34   0:03  \_ apt-get -qq -y update
_apt     28081  0.2  1.4 237124 57924 ?        S    03:34   1:44      \_ /usr/lib/apt/methods/https
_apt     28082  0.0  0.1  43212  5844 ?        S    03:34   0:00      \_ /usr/lib/apt/methods/http
_apt     28083  0.0  0.1  43276  5584 ?        S    03:34   0:00      \_ /usr/lib/apt/methods/http
_apt     28588  0.0  0.1  41036  5432 ?        S    03:36   0:00      \_ /usr/lib/apt/methods/gpgv 

Error: Repository changed its 'Suite' value

When an APT repository changes its own "Suite" description, apt will return an alert and a notice:

root@buster:~# apt-get update
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Hit:3 http://httpredir.debian.org/debian buster InRelease
Get:4 https://packages.sury.org/php buster InRelease [6,837 B]
Hit:5 http://repo.mysql.com/apt/debian buster InRelease
Hit:6 https://packagecloud.io/varnishcache/varnish60lts/debian buster InRelease
Reading package lists... Done
E: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

As long as this Suite change is not accepted, no further packages will be installed from the repository. To handle this, launch apt with the parameter –allow-releaseinfo-change:

root@buster:~# apt-get update --allow-releaseinfo-change
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Get:3 https://packages.sury.org/php buster InRelease [6,837 B]
Hit:4 http://repo.mysql.com/apt/debian buster InRelease
Hit:5 http://httpredir.debian.org/debian buster InRelease
Get:7 https://packages.sury.org/php buster/main amd64 Packages [316 kB]
Hit:6 https://packagecloud.io/varnishcache/varnish60lts/debian buster InRelease
Fetched 323 kB in 2s (199 kB/s)
Reading package lists... Done
N: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'

The error message is now gone and only a notice/informational message is showing up. Packages and updates can now be installed from this repository again. The notice will disappear after the next apt-get update, too.

Error: Unable to connect (to IPv6 resolved repository servers)

When both IPv4 and IPv6 addresses are configured on the machine, the preferred DNS resolving happens on IPv6. This means that the repository domains (e.g. archive.ubuntu.com) is resolved to IPv6 addresses:

;; ANSWER SECTION:
archive.ubuntu.com.	16	IN	AAAA	2001:67c:1360:8001::23
archive.ubuntu.com.	16	IN	AAAA	2001:67c:1360:8001::24

But this of course only works, if the whole network (including gateway) is able to work with IPv6 addresses. If the network is not IPv6 ready, a connection refused or unable to connect error or similar is shown after apt-get update.

To solve this, apt can be told to force communication using IPv4:

ck@focal:~$ echo 'Acquire::ForceIPv4 "true";' | sudo tee -a /etc/apt/apt.conf.d/99ipv4

After yet another apt update, the setting is read and communication should work on IPv4.