Replacement for httpredir in sources.list?

johnraff · 2017-04-07 01:13:50

httpredir urls in apt sources.list files still work, but they are just being redirected to deb.debian.org

We need to think about:
1) What to recommend to our current users.
2) What to use in Helium.

(Also posted in "News".)

Sector11 · 2017-04-07 02:30:56

netselect-apt to the rescue.

I have never used httpredir ... I have always used netselect-apt since my early #! days.

 06 Apr 17 @ 23:13:44 ~
  $ aptitude show netselect-apt
Package: netselect-apt                   
State: installed
Automatically installed: no
Version: 0.3.ds1-26
Priority: optional
Section: net
Maintainer: Javier Fernández-Sanguino Peña <jfs@debian.org>
Architecture: all
Uncompressed Size: 63.5 k
Depends: wget, netselect (>= 0.3.ds1-17)
Recommends: curl
Suggests: dpkg-dev
Enhances: apt
Description: speed tester for choosing a fast Debian mirror
 This package provides a utility that can choose the best Debian mirror by downloading the full mirror list and using netselect to find the
 fastest/closest one. 
 
 It can output a sources.list(5) file that can be used with package management tools such as apt or aptitude.
Homepage: http://github.com/apenwarr/netselect
 
 06 Apr 17 @ 23:14:01 ~
  $

It will create a "sources.list" in whatever directory you run it from:

Like this:

 06 Apr 17 @ 23:18:17 ~
  $ mkdir netselect
 
 06 Apr 17 @ 23:18:29 ~
  $ cd netselect
 
 06 Apr 17 @ 23:18:37 ~/netselect
  $ netselect-apt
Sorry, you need to be root to run /usr/bin/netselect-apt since the netselect
binary we will use (/usr/bin/netselect) is not setuid.
 
 06 Apr 17 @ 23:18:48 ~/netselect
  $ sudo netselect-apt
[sudo] password for sector11: 
Using distribution stable.
Retrieving the list of mirrors from www.debian.org...

--2017-04-06 23:19:02--  http://www.debian.org/mirror/mirrors_full
Resolving www.debian.org (www.debian.org)... 200.17.202.197, 2801:82:80ff:8009:e61f:13ff:fe63:8e88
Connecting to www.debian.org (www.debian.org)|200.17.202.197|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 210089 (205K) [text/html]
Saving to: ‘/tmp/netselect-apt.6QL4zL’

/tmp/netselect-apt.6QL4zL           100%[===================================================================>] 205.17K   238KB/s   in 0.9s   

2017-04-06 23:19:03 (238 KB/s) - ‘/tmp/netselect-apt.6QL4zL’ saved [210089/210089]

Choosing a main Debian mirror using netselect.
netselect: 403 (23 active) nameserver request(s)...       
Duplicate address 218.100.43.30 (http://ftp.au.debian.org/debian/, http://mirror.waia.asn.au/debian/); keeping only under first name.
netselect: 388 (23 active) nameserver request(s)...       
Duplicate address 213.129.232.18 (http://ftp.at.debian.org/debian/, http://debian.sil.at/debian/); keeping only under first name.
netselect: 387 (23 active) nameserver request(s)...
Duplicate address 200.236.31.3 (http://debian.c3sl.ufpr.br/debian/, http://ftp.br.debian.org/debian/); keeping only under first name.
netselect: 380 (22 active) nameserver request(s)...       
Duplicate address 195.234.45.114 (http://ftp.be.debian.org/debian/, http://mirror.as35701.net/debian/); keeping only under first name.
netselect: 375 (23 active) nameserver request(s)...       
Duplicate address 192.175.120.168 (http://192.175.120.168/debian/, http://debian.mirror.iweb.ca/debian/); keeping only under first name.
netselect: 370 (23 active) nameserver request(s)...       
Duplicate address 62.44.96.11 (http://ftp.uni-sofia.bg/debian/, http://ftp.bg.debian.org/debian/); keeping only under first name.
netselect: 368 (23 active) nameserver request(s)...       
Duplicate address 207.210.46.249 (http://207.210.46.249/debian/, http://debian.mirror.rafal.ca/debian/); keeping only under first name.
netselect: 344 (23 active) nameserver request(s)...       
Duplicate address 195.113.161.73 (http://ftp.cz.debian.org/debian/, http://ftp.debian.cz/debian/); keeping only under first name.
netselect: 338 (23 active) nameserver request(s)...       
Duplicate address 137.226.34.46 (http://ftp.halifax.rwth-aachen.de/debian/, http://ftp2.de.debian.org/debian/); keeping only under first name.
netselect: 337 (23 active) nameserver request(s)...
Duplicate address 141.76.2.4 (http://debian.inf.tu-dresden.de/debian/, http://ftp.de.debian.org/debian/); keeping only under first name.
netselect: 312 (22 active) nameserver request(s)...       
Duplicate address 202.38.95.110 (http://mirrors.ustc.edu.cn/debian/, http://ftp.cn.debian.org/debian/); keeping only under first name.
netselect: 304 (24 active) nameserver request(s)...       
Duplicate address 130.225.254.116 (http://mirrors.dotsrc.org/debian/, http://ftp.dk.debian.org/debian/); keeping only under first name.
netselect: 297 (23 active) nameserver request(s)...       
Duplicate address 82.194.78.250 (http://ftp.es.debian.org/debian/, http://ulises.hostalia.com/debian/); keeping only under first name.
netselect: 296 (24 active) nameserver request(s)...       
Duplicate address 195.80.123.154 (http://ftp.aso.ee/debian/, http://ftp.ee.debian.org/debian/); keeping only under first name.
netselect: 283 (23 active) nameserver request(s)...       
Duplicate address 212.27.32.66 (http://debian.proxad.net/debian/, http://ftp.fr.debian.org/debian/); keeping only under first name.
netselect: 261 (23 active) nameserver request(s)...       
Duplicate address 101.6.6.178 (http://mirrors.tuna.tsinghua.edu.cn/debian/, http://ftp2.cn.debian.org/debian/); keeping only under first name.
netselect: 242 (24 active) nameserver request(s)...       
Duplicate address 78.129.164.123 (http://free.hands.com/debian/, http://ftp.uk.debian.org/debian/); keeping only under first name.
netselect: 235 (23 active) nameserver request(s)...       
Duplicate address 147.102.222.211 (http://ftp.gr.debian.org/debian/, http://debian.noc.ntua.gr/debian/); keeping only under first name.
netselect: 228 (23 active) nameserver request(s)...       
Duplicate address 161.53.160.11 (http://ftp.hr.debian.org/debian/, http://debian.carnet.hr/debian/); keeping only under first name.
netselect: 222 (23 active) nameserver request(s)...       
Duplicate address 193.1.193.65 (http://debian.heanet.ie/debian/, http://ftp.ie.debian.org/debian/); keeping only under first name.
netselect: 219 (23 active) nameserver request(s)...       
Duplicate address 195.228.252.133 (http://ftp.fsn.hu/debian/, http://ftp.hu.debian.org/debian/); keeping only under first name.
netselect: 210 (22 active) nameserver request(s)...       
Duplicate address 194.105.226.20 (http://debian.simnet.is/debian/, http://ftp.is.debian.org/debian/); keeping only under first name.
netselect: 195 (23 active) nameserver request(s)...       
Duplicate address 129.132.53.171 (http://ftp.ch.debian.org/debian/, http://debian.ethz.ch/debian/); keeping only under first name.
netselect: 192 (23 active) nameserver request(s)...       
Duplicate address 213.156.32.111 (http://ftp.it.debian.org/debian/, http://debian.fastweb.it/debian/); keeping only under first name.
netselect: 177 (23 active) nameserver request(s)...       
Duplicate address 103.22.220.133 (http://ftp.kaist.ac.kr/debian/, http://ftp.kr.debian.org/debian/); keeping only under first name.
netselect: 168 (22 active) nameserver request(s)...       
Duplicate address 132.248.182.159 (http://mmc.geofisica.unam.mx/debian/, http://ftp.mx.debian.org/debian/); keeping only under first name.
netselect: 153 (23 active) nameserver request(s)...       
Duplicate address 158.129.159.169 (http://debian.mirror.vu.lt/debian/, http://ftp.lt.debian.org/debian/); keeping only under first name.
netselect: 152 (23 active) nameserver request(s)...
Duplicate address 178.17.160.48 (http://mirror.as43289.net/debian/, http://ftp.md.debian.org/debian/); keeping only under first name.
netselect: 149 (23 active) nameserver request(s)...       
Duplicate address 150.65.7.130 (http://ftp.jaist.ac.jp/debian/, http://ftp.jp.debian.org/debian/); keeping only under first name.
netselect: 130 (23 active) nameserver request(s)...       
Duplicate address 78.96.7.8 (http://ftp.upcnet.ro/debian/, http://ftp.ro.debian.org/debian/); keeping only under first name.
netselect: 121 (23 active) nameserver request(s)...       
Duplicate address 194.71.11.165 (http://ftp.no.debian.org/debian/, http://ftp.se.debian.org/debian/); keeping only under first name.
netselect: 120 (23 active) nameserver request(s)...
Duplicate address 194.71.11.165 (http://ftp.no.debian.org/debian/, http://ftp.acc.umu.se/debian/); keeping only under first name.
netselect: 109 (23 active) nameserver request(s)...       
Duplicate address 210.23.25.77 (http://mirror.0x.sg/debian/, http://ftp.sg.debian.org/debian/); keeping only under first name.
netselect: 107 (24 active) nameserver request(s)...       
Duplicate address 115.126.160.7 (http://ftp.nc.debian.org/debian/, http://debian.nautile.nc/debian/); keeping only under first name.
netselect: 102 (23 active) nameserver request(s)...       
Duplicate address 213.157.243.77 (http://debmirror.amis.net/debian/, http://ftp.si.debian.org/debian/); keeping only under first name.
netselect: 97 (23 active) nameserver request(s)...        
Duplicate address 190.5.143.16 (http://ftp.sv.debian.org/debian/, http://debian.salud.gob.sv/debian/); keeping only under first name.
netselect: 93 (23 active) nameserver request(s)...       
Duplicate address 202.8.44.105 (http://ftp.citylink.co.nz/debian/, http://ftp.nz.debian.org/debian/); keeping only under first name.
netselect: 90 (23 active) nameserver request(s)...       
Duplicate address 62.168.95.115 (http://ftp.sk.debian.org/debian/, http://ftp.debian.sk/debian/); keeping only under first name.
netselect: 87 (23 active) nameserver request(s)...       
Duplicate address 85.143.112.112 (http://mirror.mephi.ru/debian/, http://ftp.ru.debian.org/debian/); keeping only under first name.
netselect: 84 (23 active) nameserver request(s)...       
Duplicate address 140.112.8.139 (http://debian.linux.org.tw/debian/, http://ftp.tw.debian.org/debian/); keeping only under first name.
netselect: 76 (23 active) nameserver request(s)...       
Duplicate address 128.30.2.26 (http://128.30.2.26/debian/, http://debian.csail.mit.edu/debian/); keeping only under first name.
netselect: 72 (22 active) nameserver request(s)...       
Duplicate address 64.50.236.52 (http://64.50.236.52/debian/, http://64.50.236.52/debian/); keeping only under first name.
netselect: 72 (24 active) nameserver request(s)...
Duplicate address 64.50.233.100 (http://64.50.233.100/debian/, http://64.50.233.100/debian/); keeping only under first name.
netselect: 69 (22 active) nameserver request(s)...       
Duplicate address 128.61.240.89 (http://128.61.240.89/debian/, http://debian.gtisc.gatech.edu/debian/); keeping only under first name.
netselect: 60 (22 active) nameserver request(s)...       
Duplicate address 91.200.60.59 (http://debian.org.ua/debian/, http://ftp.ua.debian.org/debian/); keeping only under first name.
netselect: 55 (23 active) nameserver request(s)...       
Duplicate address 193.140.100.100 (http://ftp.linux.org.tr/debian/, http://ftp.tr.debian.org/debian/); keeping only under first name.
netselect: 54 (23 active) nameserver request(s)...
Duplicate address 193.140.100.100 (http://ftp.linux.org.tr/debian/, http://debian.doratelekom.com/debian/); keeping only under first name.
netselect: 17 (17 active) nameserver request(s)...       
Duplicate address 208.80.154.15 (http://208.80.154.15/debian/, http://mirrors.wikimedia.org/debian/); keeping only under first name.
Running netselect to choose 10 out of 388 addresses.     
........................................................ {snip a really long line of dots}
The fastest 10 servers seem to be:

	http://debian.unnoba.edu.ar/debian/
	http://repo.cure.edu.uy/debian/
	http://ftp.ccc.uba.ar/pub/linux/debian/debian/
	http://mirrors.tecnoera.com/debian/
	http://debian.ec.as6453.net/debian/
	http://sft.if.usp.br/debian/
	http://dist1.800hosting.com/debian/
	http://debian.mirror.constant.com/debian/
	http://debian.pop-sc.rnp.br/debian/
	http://ftp.us.debian.org/debian/

Of the hosts tested we choose the fastest valid for HTTP:
        http://debian.unnoba.edu.ar/debian/

Writing sources.list.
Done.
 
 06 Apr 17 @ 23:19:32 ~/netselect
  $ ls
sources.list

Therefor I use: #10 >> http://ftp.us.debian.org/debian/

Educate new users to do the same OR have bl-welcome do this maybe giving an option from 1 - 10 for the user to pick.

While, http://debian.unnoba.edu.ar/debian/, was #1 for speed, it's not updated as often as: http://ftp.us.debian.org/debian/

http://ftp.us.debian.org/debian/ in my opinion should be the default if the bl-welcome idea is too much.

EDIT: Check the man page. Lots of options.

Last edited by Sector11 (2017-04-07 02:32:16)

Sector11 · 2017-04-07 02:41:10

Just for show (see previous post)

/home/sector11/netselect/sources.list

# Debian packages for stable
deb http://debian.unnoba.edu.ar/debian/ stable main contrib
# Uncomment the deb-src line if you want 'apt-get source'
# to work with most packages.
# deb-src http://debian.unnoba.edu.ar/debian/ stable main contrib

# Security updates for stable
deb http://security.debian.org/ stable/updates main contrib

And my actual source.list:

## DEBIAN
deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
#deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free

## SECURITY
deb http://security.debian.org/ jessie/updates main contrib non-free
# deb-src http://security.debian.org/ jessie/updates main contrib non-free

## UPDATES
deb http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free
# deb-src http://ftp.us.debian.org/debian/ jessie-updates main contrib non-free

## BACKPORTS
deb http://ftp.us.debian.org//debian jessie-backports main contrib non-free
# deb http://httpredir.debian.org/debian jessie-backports main contrib non-free

Head_on_a_Stick · 2017-04-07 06:44:41

I had noticed that the redirector was using their CDN mirror anyway so I had already changed my sources to

http://cdn-fastly.deb.debian.org/debian jessie main

Seemsed to work well for me.

Last edited by Head_on_a_Stick (2017-04-07 06:45:56)

Head_on_a_Stick · 2017-04-07 06:51:47

Sector11 wrote:

Just for show (see previous post)

/home/sector11/netselect/sources.list

# Debian packages for stable
deb http://debian.unnoba.edu.ar/debian/ stable main contrib
# Uncomment the deb-src line if you want 'apt-get source'
# to work with most packages.
# deb-src http://debian.unnoba.edu.ar/debian/ stable main contrib

# Security updates for stable
deb http://security.debian.org/ stable/updates main contrib

If that netselect-apt tool writes stable into /etc/apt/sources.list rather than the actual release name then it shouldn't be used — everybody's system would break at the end of the freeze

johnraff · 2017-04-07 08:50:04

Head_on_a_Stick wrote:

I had noticed that the redirector was using their CDN mirror anyway so I had already changed my sources to
http://cdn-fastly.deb.debian.org/debian jessie main
Seemsed to work well for me.

OK I'll try that with the CD iso build I'm just starting.

brontosaurusrex · 2017-04-07 08:58:01

Just some observations, not proposing anything:

@sector11: netselect-apt will fail in certain circumstances (company firewall for example):

Running netselect to choose 10 out of 388 addresses.    

Did not found any valid hosts (you requested 10)
netselect was unable to find a mirror, this probably means that
you are behind a firewall and it is blocking ICMP and/or
UDP traceroute. Or the servers test are actively blocking
ICMP and/or UDP traceroute probes.

Changing http in deb.debian.org to https will get you another mirror (apt-transport-https needed)

...
Get:3 https://cdn-aws.deb.debian.org/debian stretch InRelease [184 kB]
Hit:3 https://cdn-aws.deb.debian.org/debian stretch InRelease
Get:4 https://cdn-aws.deb.debian.org/debian stretch-updates InRelease [88.5 kB]
Hit:4 https://cdn-aws.deb.debian.org/debian stretch-updates InRelease
Hit:5 https://cdn-aws.deb.debian.org/debian stretch-proposed-updates InRelease
....

Not sure how to get SRV records

host -t SRV deb.debian.org
deb.debian.org is an alias for static.debian.org.

host -t SRV static.debian.org
static.debian.org has no SRV record

Last edited by brontosaurusrex (2017-04-07 09:35:05)

nobody · 2017-04-07 09:33:59

For the record:

This message.

> Does that means most mirrors of Debian (including those official mirrors,
> ftp.*.debian.org) will no longer receive requests redirected from
> httpredir.debian.org, as long as they are not related to CDN providers?
Yes.

So picking a fast mirror near you is probably the best option if you want to take advantage of a) decentralized architecture, b) do not want to give big CDN providers (AWS, Google, Cloudflare, ...) any more traffic they do not deserve, c) live in an area where those services are not available (I read Iran somewhere in that email thread).

hhh · 2017-04-07 13:50:00

My 2 cents, I've always used ftp.us.debian.org/debian/ since I'm in the US. I could use ftp.debian.org/debian/ as the default and have the user add their country code. Then we're not relying on a company's service.

Head_on_a_Stick · 2017-04-07 19:59:24

Actually, looking at the OP's link again:

The redirection service is also available on HTTPS, so with the apt-transport-https package installed, you can use:
deb https://deb.debian.org/debian stable main
deb https://deb.debian.org/debian-security stable/updates main

https://packages.debian.org/jessie/apt-transport-https

I think we should definitely use https if possible.

hhh · 2017-04-07 20:05:30

I thought ftp was faster. That may be a misunderstanding on my part, or maybe outdated info.

Head_on_a_Stick · 2017-04-07 20:10:06

^ Apparently:

ArchWiki wrote:

HTTP mirrors are faster than FTP due to persistent HTTP connection: with FTP, a new connection to server has to be established each time pacman requests a package to be downloaded, which results in a brief pause.

https://wiki.archlinux.org/index.php/mi … fic_mirror

EDIT: more here: https://daniel.haxx.se/docs/ftp-vs-http.html

However, I was referring to https — this is a more secure version of http

Last edited by Head_on_a_Stick (2017-04-07 20:10:53)

johnraff · 2017-04-08 03:34:09

As per OP, we have to separate out what advice to offer existing Jessie users from what we should do in Helium/Stretch. Apt in Stretch will handle deb.debian.org differently.

hhh · 2017-04-08 13:45:22

OK, so ftp.debian.org/debian/ should definitely not be used, as per this wiki page and accompanying links...
https://wiki.debian.org/ftp.debian.org

The Oregon State University Open Source Lab offers an https mirror that uses ftp, but not http...
https://ftp.osuosl.org/debian/

Debian mirror master-list...
https://anonscm.debian.org/viewvc/webwm … iew=markup

hhh · 2017-04-08 13:53:01

Ah, discussion about https Debian mirrors is ongoing...
https://lists.debian.org/debian-mirrors … 00000.html

Sector11 · 2017-04-08 16:54:20

https://ftp.us.debian.org/debian/

Your connection is not secure

The owner of ftp.us.debian.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

Learn more…

Interesting as all get up and go.
Clicked on advanced and get:

ftp.us.debian.org uses an invalid security certificate. The certificate is only valid for the following names: osuosl.org, *.osuosl.org Error code: SSL_ERROR_BAD_CERT_DOMAIN

but http://ftp.us.debian.org/debian/ works just fine (which, again, is what I use.)

Bearded_Blunder · 2017-08-18 09:11:52

I'm with default to something generic.. then offer netselect-apt in bl-welcome.
Seem to remember posting about using that in tips & trucks ages back.

Bearded_Blunder · 2017-08-19 07:42:35

Or here's a radical idea, how about having users pick a mirror during setup the way a netinstall does? The code to do it is right in the installer, and the licensing isn't an issue.

KrunchTime · 2017-08-19 22:29:50

My preference would be to do this in the bl-welcome script. I don't use that script and would prefer to configure my preferred repo by hand.

Sector11 · 2017-08-19 22:33:37

Bearded_Blunder wrote:

Or here's a radical idea, how about having users pick a mirror during setup the way a netinstall does? The code to do it is right in the installer, and the licensing isn't an issue.

This!

#1 2017-04-07 01:13:50

Replacement for httpredir in sources.list?

#2 2017-04-07 02:30:56

Re: Replacement for httpredir in sources.list?

#3 2017-04-07 02:41:10

Re: Replacement for httpredir in sources.list?

#4 2017-04-07 06:44:41

Re: Replacement for httpredir in sources.list?

#5 2017-04-07 06:51:47

Re: Replacement for httpredir in sources.list?

#6 2017-04-07 08:50:04

Re: Replacement for httpredir in sources.list?

#7 2017-04-07 08:58:01

Re: Replacement for httpredir in sources.list?

#8 2017-04-07 09:33:59

Re: Replacement for httpredir in sources.list?

#9 2017-04-07 13:50:00

Re: Replacement for httpredir in sources.list?

#10 2017-04-07 19:59:24

Re: Replacement for httpredir in sources.list?

#11 2017-04-07 20:05:30

Re: Replacement for httpredir in sources.list?

#12 2017-04-07 20:10:06

Re: Replacement for httpredir in sources.list?

#13 2017-04-08 03:34:09

Re: Replacement for httpredir in sources.list?

#14 2017-04-08 13:45:22

Re: Replacement for httpredir in sources.list?

#15 2017-04-08 13:53:01

Re: Replacement for httpredir in sources.list?

#16 2017-04-08 16:54:20

Re: Replacement for httpredir in sources.list?

#17 2017-08-18 09:11:52

Re: Replacement for httpredir in sources.list?

#18 2017-08-19 07:42:35

Re: Replacement for httpredir in sources.list?

#19 2017-08-19 22:29:50

Re: Replacement for httpredir in sources.list?

#20 2017-08-19 22:33:37

Re: Replacement for httpredir in sources.list?

Board footer