How to overcome incoming ssh firewalls through an intermediate relay ssh server

The way to forward ssh is as follows. I essentially read the openssh
cookbook here

https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Tunnels

I have machine A (thinkpad), behind a firewall that cannot accept
incoming network connections. Then we have machine B (imac) that is also
behind a firewall that cannot accept incoming network connections. But
they can both connect to my home server. A simple schematic of the
allowed connections is as follows:

thinkpad --> shirl_dell_server
imac --> shirl_dell_server

I want to make a connection from

thinkpad --> imac

The way to do this is to use shirl_dell_server as a relay to
circumvent the firewall that prevents incoming connections on the
thinkpad and imac. For example, my University does this. The way to do
this is to establish one normal ssh tunnel and one reverse ssh tunnel.

1. From thinkpad run

   ssh -fN -L 9000:localhost:9000 shirl_dell_server 
   

   This establishes a tunnel from local port 9000 (-L) to remote port
   9000 on shirl_dell_server. The -N allows you not run any command
   on the remote server -- it usually runs the login shell -- and the
   -f allows the process to drop to the background.

2. From the imac run

   ssh -fN -R 9000:localhost:22 shirl_dell_server
   

   This establishs a reverse ssh tunnel from remote port 9000 (-R) to
   the localport 22 where an sshd server is listening for incoming
   connections.

3. Then from thinkpad, run

   ssh -p 9000 localhost -l username
   

   This would get forwarded to the remote server's 9000 port through
   the ssh tunnel. The remote server would forward the incoming traffic
   on 9000 to the imac's port 22 through the reverse ssh tunnel.

This would then show you an ssh login prompt.

Note that you may have to enable to options

AllowTcpForwarding  yes
PermitTunnel        yes

Note that disabling AllowTcpForwarding does not improve security as the
sshd_config manpage says. Permitting tunnels, however, might decrease
security since it might allow you to circumvent other firewalls.

Silent failure of anacron on resume with systemd

Anacron fails silently on resume. It never runs the tasks its supposed
to run. My backups have been failing.

This is a a really annoying failure, and some features are being
implemented in systemd to fix this. See this debian
bugtracker.

My solution for this is to simply have anacron be run every hour by
cron. I have in /etc/cron.d/anacron

# /etc/cron.d/anacron: crontab entries for the anacron package

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

30 *    * * *   root    test -x /etc/init.d/anacron && /usr/sbin/invoke-rc.d anacron start >/dev/null

which simply runs it every hour now. This is a little annoying, but it's
the best I can do. The debian bugtracker states that it's been fixed in
a future release of systemd. I don't really care because I'm using
Jessie. But Jessie is so old that apparently even critical bugs like
this don't get fixed.

pass, passmenu and dmenu are pretty amazing together

I've switched to pass for managing my
passwords on my linux box, nexus phone and android tablet. It's pretty
amazing.

Passwords are encrypted using pgp, and synchronized on my desktops using
dropbox. pass also has git support. So I push all my passwords to my
home ssh server using git. I can pull down these passwords to my android
phone and tablet using git packages and a pgp manager.

Setting pass up on android is a little painful. I use

1. A pgp key management app.
2. pass for
   android

To get my private pgp key into the app, I do something fairly insecure.
I transfer my key onto a dropbox encfs partition. Then I access this
encfs partition using another app called

BoxCryptor Classic

Then I also need to generate an ssh keypair so that pass for android has
access to my ssh server at home. I again copy the public key to dropbox
and have my server copy it there into the authorized users file.

Passmenu (bundled with pass) is especially powerful since I use dmenu to
launch programs.

Hibernate issues on Thinkpad T450s

Read updates after the main body for context.

UPDATE (May 22 2017) When I forced the firmware to use ucode version
17 for the iwlwifi driver, my network has been a lot more stable even on
the enterprise networks like my university network and eduroam.
Hibernate is pretty stable except for the following problem:

Close the lid and systemd-logind triggers suspend. Using the suspend hook,
the delayed hibernate script triggers a wake after 2 minutes. Then the
computer hibernates. However, I cannot awaken the computer (using the power
switch until I plug it in again!) Once I do so, the power switch works
normally and the computer resumes from hibernate.

One option is to manually handle the powerswitch using xfce power
manager or something. Here are some ideas:

1. Disable systemd-logind handling of the lidswitch and try xfce4 power
   manager, and see if the original issues it had are resolved.
2. Disable logind and xfce power manager (that I use primarily for
   dimming and locking the screen). Then, switch to acpid for hitting
   systemctl to handle the lidswitch

UPDATE (May 13 2017) I couldn't track down the patches on the 4.4
kernel to apply it properly. So I reverted the ucode version that the
iwlwifi driver was using on the 4.10 kernel to see if this solves my
wifi instability issues.

dmesg | grep iwl

The 4.10 kernel appears to use

loaded firmware version 22.391740.0 op_mode iwlmvm

and the 4.4 kernel uses

loaded firmware version 17.352738.0 op_mode iwlmvm

I deleted the

/usr/lib/firmware/iwlwifi/*7265D-2[0-9].ucode

and now the 4.10 kernel seems to load an earlier firmware version.
Perhaps this fixes the instability on networks that use WPA2-Enterprise.

UPDATE (May 12 2017) I have wifi instability issues on the newer
kernels like 4.9 - 4.10 on the iwlwifi driver. I don't know what
changed, but I frequently need to ifconfig down/up the interface or
rmmod/insmod the iwlwifi driver.

So I went back to the 4.4-lts kernel available on the AUR with the
4.4.64 patch set. Then I installed the following two patches:

1. https://patchwork.kernel.org/patch/9208541/
2. https://patchwork.kernel.org/patch/9202321/

One of the patches didn't apply since they were meant for the 4.1.6
kernel, so I manually inserted some of the rejected patches. It appears
to work after a couple of tests. The patch appears to be merged with the
following commit number

406f992e4a372dafbe3c2cff7efbb2002a5c8ebd

It's quite annoying that the iwlwifi driver is so unstable on the newer
kernels with my Intel 7265 card. I will update it if there is a problem.
Also, the unpatched 4.4.0 kernel also appears to work quite well, even
without the patches, but I haven't tested it with more that 8GB memory
in used, which is what seems to create the problem.

UPDATE (May 09 2017) Intel rapid start (irst) craps out every once
in a while. After a few successful hibernates, it seems to go into a
boot loop of some sort, and then simply unsuccessfully reboots. So I've
disabled it, and moved to a newer kernel (4.9) that does not appear to
have hibernate issues. I'm using this delayed hibernate script on the
ArchWiki.

I've been having Hibernate issues on my thinkpad 450s. Apparently this
is quite common see this kernel bug
report and has
existed in the newer kernels since version 3.x. So my solution was to
use the Intel Rapid Start technology. It was originally suggested on the
kernel bug report. It was quite easy to do.

1. Make a partition that is greater than or equal to the amount of ram
   in your notebook.

   gdisk /dev/sda
   n
   

   Then, set partition name, start sector and size by following the
   prompts. Select 8400 (Intel Rapid Start) when prompted for a
   Hex Code or GUID. Hit w to write the partition table and q to
   quit.

2. Go into the bios, and under power you'll find the Intel Rapid Start
   technology. You can set deep sleep mode to set in after an hour of
   suspending - that's what I've set mine too.

3. Next time just suspend your thinkpad. Rapid Start works flawlessly.

The old hibernate issues have been resolved on newer kernels (4.7 and
above). But I always find hibernate a little buggy. For example, the
display frequently gets screwed up on my laptop and I have to restart X
or my display manager.

Exim with gmail as smarthost

I've been using exim with my google account to send email quite
successfully. I essentially followed the arch
wiki.
It allows me to send mail from any of my machine on my local area
network through my server. This is quite convenient because a lot of
them run cron jobs, and it's useful to a have an email relay server
running on the network. Comcast, my ISP, blocks smtp ports and so I
cannot run my own email server at home. So I setup an exim based relay
that accepts email from within my local area network and sends it using
my gmail address.

However, it stopped working without warning one day. My log
/var/log/exim/mainlog started being populated with messages like

    535-5.7.8 Username and Password not accepted. Learn more at
    535 5.7.8  https://support.google.com/mail/?p=BadCredentials v9sm11944329pfg.133 - gsmtp

This was apparently because I did not have two-factor authentication
enabled, and gmail had started flagging my application. So I enabled
two-factor authentication, and gave my it an app password. It started
working again.