FTP file Fetch script

 FTP file Fetch script

===========================
#!/bin/bash
export SSHPASS=******(ftp passwd
sshpass -e sftp -oBatchMode=no -b -  test@sftp2.urserver.com << !
   cd inbound
   mget *.zip
   rename *.zip archive/*.zip
   bye
!
===========================

Connect SFTP using Private Key.

Connect SFTP using Private Key.


sftp -oIdentityFile=/home/xxxx.key user@servername

create an apt-mirror with ubuntu 12.04 

If you run several ubuntu machines on your network, you may want to mirror the ubuntu repositories locally to both save bandwidth, and increase the overall download rate at which you can update your machines, you don't want to be going out to the internet to download the same update file time and time again now do you?

The following describes how to setup an apt mirror on an ubuntu server (12.04 LTS) to mirror the main ubuntu repositories (main, multiverse, restricted and universe), security repository, as well as the puppet apt repository.

Server Setup - install and configure apt-mirror

For this setup I had an ubuntu 12.04 server with a 200GB partition mounted at /var/spool/apt-mirror. Step one is to install apt-mirror :

# sudo apt-get install apt-mirror

The default configuration file can be found at /etc/apt/mirror.list :

############# config ##################
#
# set base_path    /var/spool/apt-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

deb-src http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu

You can add extra mirrors to the list, change the default directory into which the mirror is created, change the number of threads used for downloading packages etc. For example, to download only 64 bit architecture, change the deb section for each line to read deb-amd64 (or deb-i386 for 32 bit packages).

Once edited, my configuration looked as such :

############# config ##################
#
# set base_path    /var/spool/apt-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

# Precise 64Bit Mirror
deb-amd64 http://gb.archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb-amd64 http://gb.archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb-amd64 http://gb.archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
deb-amd64 http://gb.archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
deb-amd64 http://gb.archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

# Precise 32Bit Mirror
deb-i386 http://gb.archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb-i386 http://gb.archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb-i386 http://gb.archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
deb-i386 http://gb.archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
deb-i386 http://gb.archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

#deb-src http://gb.archive.ubuntu.com/ubuntu precise main restricted universe multiverse
#deb-src http://gb.archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
#deb-src http://gb.archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
#deb-src http://gb.archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
#deb-src http://gb.archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

# Puppet mirror
deb-amd64 http://apt.puppetlabs.com/ precise main
deb-i386 http://apt.puppetlabs.com/ precise main

clean http://gb.archive.ubuntu.com/ubuntu
clean http://apt.puppetlabs.com/

You can see that I am mirroring both the 32bit and 64bit architectures for 12.04 (precise) as well as the precise mirror from puppetlabs. Since I'm not often recompiling packages from the source, I haven't mirrored the source packages.

Another change to make from the default file is to change the mirror that your are downloading your packages from in order to download from a closer, faster mirror. This should be done by adding the country code to the beginning of archive.ubuntu.com for example here, gb.archive.ubuntu.com.

At this point in time, my mirror is using 67GB of space to help calculate how much space to use.

To perform the first manual update of the mirror, run the following command (warning, depending on your connection, this could take quite a while, go get a coffee, or five!) :

# sudo apt-mirror /etc/apt/mirror.list

If you have a problem with your configuration and apt-mirror quits unexpectedly, or you break out it (with a ctrl-c for example) you may need to remove the lockfile at /var/spool/apt-mirror/var/apt-mirror.lock
In order to keep the mirror up-to-date automatically, the apt-mirror installs a cron job at /etc/cron.d/apt-mirror :

#
# Regular cron jobs for the apt-mirror package
#
#0 4 * * * apt-mirror /usr/bin/apt-mirror > /var/spool/apt-mirror/var/cron.log

Removing the comment (#) from the start of the last line will make your mirror update daily at 4am, you may wish to change this to a more convenient time, but I would advise against running it too often, and putting unnecessary strain on the remote mirror.

Every once in a while you may wish to clean up your mirror to save space, this can be done by running the following command :

# sudo /var/spool/apt-mirror/var/clean.sh

Server Setup - install and configure apache for http access

Ubuntu clients normally access their mirrors over http, we can setup our mirror for access using apache2. Firstly install apache :

# sudo apt-get install apache2

This will install apache2 and any other necessary dependancies. The quickest (but not necessarily best) way to set up apache to serve the mirror out to clients is as follows :

# sudo ln -s /var/spool/apt-mirror/mirror/gb.archive.ubuntu.com/ubuntu/ /var/www/ubuntu
# sudo ln -s /var/spool/apt-mirror/mirror/apt.puppetlabs.com /var/www/puppet

Remember to change the relevant directory names if your using a different mirror to download from!

Client Setup

Finally we have to update our clients to point to the mirror, this is a simple as editing the file /etc/apt/sources.list and changing the URLs to point to our local mirror, below is an example of an update sources.list file

# deb cdrom:[Ubuntu 11.10 _Oneiric Ocelot_ - Release i386 (20111012)]/ oneiric main restricted

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://apt-mirror.localdomain/ubuntu/ precise main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://apt-mirror.localdomain/ubuntu/ precise-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://apt-mirror.localdomain/ubuntu/ precise universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise universe
deb http://apt-mirror.localdomain/ubuntu/ precise-updates universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://apt-mirror.localdomain/ubuntu/ precise multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise multiverse
deb http://apt-mirror.localdomain/ubuntu/ precise-updates multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://apt-mirror.localdomain/ubuntu/ precise-backports main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse

deb http://apt-mirror.localdomain/ubuntu precise-security main restricted
deb-src http://security.ubuntu.com/ubuntu precise-security main restricted
deb http://apt-mirror.localdomain/ubuntu precise-security universe
deb-src http://security.ubuntu.com/ubuntu precise-security universe
deb http://apt-mirror.localdomain/ubuntu precise-security multiverse
deb-src http://security.ubuntu.com/ubuntu precise-security multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
deb http://archive.canonical.com/ubuntu oneiric partner
deb-src http://archive.canonical.com/ubuntu oneiric partner

## This software is not part of Ubuntu, but is offered by third-party
## developers who want to ship their latest software.
deb http://extras.ubuntu.com/ubuntu precise main
deb-src http://extras.ubuntu.com/ubuntu precise main

Once you've updated the sources.list file, run the following command on the client to download the catalogs from your new mirror and update your software :

# sudo apt-get update $$ sudo apt-get upgrade

                                                                thanx:unixrob