2020 Macbook Pro Screen “Washed Out” on Wake / Resume

I have no idea how to explain this one, but I’ll do my best.

Infrequently when I open the lid on my 2020 Macbook Pro, the laptop wakes up, but the screen is “washed out” / has this weird very white overlay going on.

Nothing I have done seems to resolve this.

It started almost immediately after purchasing, and subsequent OS updates haven’t solved it. I have no external monitors plugged in, or in fact, anything external at all. The one thing that may be a commonality is having the power cable plugged in.

Annoyingly, when I go to shut down / restart (which fixes it, btw), the screen comes back to normal just as I’m shutting down. How frustrating.

So a cunning fix I’ve found – that solves a crappy problem with a three grand laptop – is to open an app like iTerm, then go to shutdown.

iTerm will then prompt saying “hey, do you really want to shut down?” And this allows me to stop the shutdown process and my screen goes back to normal.

Should I need a bodge workaround for a top of the line Apple product? Apparently: yes.

I should add, btw, that this is not the only problem I have with this laptop. I also suffer from:

  • intermittent key repeat / sticking;
  • headphone jack doesn’t work on wake from sleep;
  • my fat fingers keep accidentally touching the “touch bar” when typing.

Admittedly the last one is my fault… I guess.

Disappointing.

Docker Ate My Harddisk – No space left on device

I’ve recently been forced to migrate from Rancher v1.x to using docker-compose to manage my production Docker containers.

In many ways its actually a blessing in disguise. Rancher was a nice GUI, but under the hood it was a total black box. Also, fairly recently they migrated to v2, and no true transition path was provided – other than “LOL, reinstall?”

Anyway, one thing that Rancher was doing for me – at least, I think – was making sure the log files didn’t eat up all my hard disk space.

This one just completely caught me by surprise, as I am yet to get my monitoring setup back up and running on this particular box:

➜ ssh chris@1.22.33.44 # obv not real

Welcome to chris server
Documentation: https://help.ubuntu.com
Management: https://landscape.canonical.com
Support: https://ubuntu.com/advantage
System information as of Wed May 13 12:28:24 CEST 2020
System load: 3.37

Usage of /: 100.0% of 1.77TB

Memory usage: 13%
Swap usage: 0%
Processes: 321
Users logged in: 0

=> / is using 100.0% of 1.77TB

Yowser.

There’s a pretty handy command to drill down into exactly what is eating up all your disk space – this isn’t specific to Docker either:

chris@chris-server / # du -h --max-depth=1
16G ./export
607M ./lib
60M ./boot
13M ./bin
5.1M ./etc
48K ./tmp
180K ./root
8.0K ./media
2.3T ./var
4.0K ./mnt
0 ./sys
4.0K ./srv
28K ./home
0 ./dev
59G ./docker
1.2G ./usr
13M ./sbin
0 ./proc
8.0K ./snap
4.0K ./lib64
316M ./run
16K ./lost+found
16K ./opt
2.4T .

The culprit here being /var with its 2.3T used… of 1.8T file system? Yeah.. idk.

Anyway you can keep drilling down with the disk usage command until you isolate the culprit. But as this is Docker related, I’ll save you the bother:

chris@chris-server /var/lib/docker/containers # du -h --max-depth=1
270G ./b3ec5a04079d5b3060d5575e011dd5e482950a794d52b4470659823dc5d5a6be
1.9M ./4b1eff96be524bdb20e87448520a30329a5e42bb36f9ba523f458d87cb8b5dc7
44K ./fc7552b499aca543a9bc6d8ef223a09d1ed21c4d054aa150b29c1d4023c151d9
40K ./43ef97d1a135f7404ca4a0330fb0ae6310ef87316358324a05bf6a65cc8d06b0
64K ./a0155805cc59d65b5815352f97939f12a4e87e30662eafa253ae8395eb553e0c
44K ./b4fd3f35324278dc832a5e98fe5a3e5655fecaf61d4de8c5e8080f8af63c22fe
40K ./b37593cd946407872e71c57af6d30b1c85c30fc42a8e075aad542fa820fbdc97
192K ./cb1e3eac9d1c1d997471f915d42fd0f999454fbee772ddc451f28e60ff1a4d22
281G ./3dd2886e627555cd6db45f68e0d1d9e520a6ab4ad443a2e43c70498b51b81fc4
1.9M ./552a0d7a7dd13afa65050381104e86395b2510c6437f650782260664ba195deb
40K ./111da287b97d0eff6b710cd7a9f0d80b7febd03c4ee30db8e2c0519e9a506c8a
40K ./47112f1fd4b0399c506fc1318ba2c0d03502b3ad731169f43a317a09556b2a6d
40K ./afeb8b8c2fbc918caaf199ae0aaca78f68e91412f9a7bc389963c6c15d6e2832
1.9M ./cb3c71ddb7181f9bbd3803cc644f36c05eedd922d4529c62354c68dedfc3bc02
624K ./c7ba5a354cdb2ab6f31fe443c04b72fad2a2e8107bc42a875422c4d80a164144
212K ./5d2b649bdb9eca4f0e14304e4ffad22b7d50d5c099a72072862a1135488105c6
56K ./781635651e0890695e1002639b5167cebf56e78fdeed3e839fc91c0681d9edd2
28M ./65aa3252acc8cf7a7af9301a0df9c117670c602f08bdb0f256eb37382a3eb859
68K ./8c8bf1febde03cf957cae5b695baf70427f6e2dec53d07d3a495ecbf2c9a3ac0
44K ./990279783af4cfacaa2e71c18eba0e6870d472dfd11e261e40a512779783fef5
40K ./37dfd1291842da4debff4c07c58d60b4efcd9e060ebfe29f1508bf418c284ca8
304K ./34c7bfcf01a6ca8e830e6108056eb0df70d4fe88eda699f3e3aa067c1afca2da
40K ./c7be4d913f95ffc040d72374bb77ad37a31846a80f40404c8b82bd5ec725d99f
1.9M ./af173be6ea5d961be308596a28e8039a20e38bc0facf623a65ee0894f28920e2
44K ./3cb4c8b263ca4c802b6df699510374929df50784b25ab8e63802ee8f2054ffc9
328K ./574599803c9143e2f87cabcc211fa3ee47f28f1e765298141f081fa1e457ced6
281G ./6bfcad1f93a7fffa8f0e2b852a401199faf628f5ed7054ad01606f38c24fc568
108K ./e5656136a35ffb242590abf50c203f5478e09bf954a6ca682cb391c11328d251
1.1M ./0150dafffbdbb830dc6ab158913eb8bd4003bf07e280579421aaeb29a7ac5623
271G ./0b5947f687fa47a70b71869468d851e4aeb3857a599a2d7eba8cf58f3b8d6bda
40K ./d014ae83b6314bcad0a159ae3cfad03a8589d4631e7c66a68ad55f1ad722f2fe
52K ./a85b7ab40a097b1548a8dea2826277742815a5e9b0c3c803b065504a2c526bb4
588K ./b5e0a0a4900e1c6fd066f5ae8d534bb832708f0caf917839c578d7debaea3783
52K ./53dd57dcbbca5248456423ebb4f0499b5d15eb5f4be7c0821979a7c880dbaa89
1.1T .

Son of a diddly.

Basically, this wasn’t caused by Docker directly. This was caused by my shonky migration.

The underlying issue here is that some of the Docker containers I run are Workers. They are little Node apps that connect to RabbitMQ, pull a job down, and do something with it.

When the brown stuff hits the twirly thing, they log out a bit of info to help me figure out what went wrong. Fairly standard stuff, I admit.

However, in this new setup, there was no limit to what was getting logged. I guess previously Rancher had enforced some max filesize limits or was helpfully rotating logs periodically.

In this case, the first port of call was to truncate a log. This might not actually be safe, but seeing as it’s my server and it’s not mission critical, I just truncated one of the huge logs:

/var/lib/docker/containers/6bfcad1f93a7fffa8f0e2b852a401199faf628f5ed7054ad01606f38c24fc568 # ls -la
total 294559628
drwx------ 4 root root 4096 May 9 10:26 .
drwx------ 36 root root 12288 May 9 16:50 ..
-rw-r----- 1 root root 301628608512 May 13 12:44 6bfcad1f93a7fffa8f0e2b852a401199faf628f5ed7054ad01606f38c24fc568-json.log
drwx------ 2 root root 4096 May 2 10:14 checkpoints
-rw------- 1 root root 4247 May 9 10:26 config.v2.json
-rw-r--r-- 1 root root 1586 May 9 10:26 hostconfig.json
-rw-r--r-- 1 root root 34 May 9 10:25 hostname
-rw-r--r-- 1 root root 197 May 9 10:25 hosts
drwx------ 3 root root 4096 May 2 10:14 mounts
-rw-r--r-- 1 root root 38 May 9 10:25 resolv.conf
-rw-r--r-- 1 root root 71 May 9 10:25 resolv.conf.hash

truncate --size 0 6bfcad1f93a7fffa8f0e2b852a401199faf628f5ed7054ad01606f38c24fc568-json.log

That freed up about 270gb. Top lols.

Anyway, I had four of these workers running, so that’s where all my disk space had gone.

Not Out Of The Woods Just Yet

There’s two further issues to address though at this point:

Firstly, I needed to update the Docker image to set the proper path to the RabbitMQ instance. This would stop the log file spam. Incidentally, within the space of truncating and then running a further ls -la, the log was already at 70mb. That’s some aggressive connecting.

This would have been nicer as an environment variable – you shouldn’t need to do a rebuild to fix a parameter. But that’s not really the point here. Please excuse my crappy setup.

Secondly, and more importantly, I needed a way to enforce Docker never to misbehave in this way again.

Fortunately, docker-compose has a solution to this problem.

Here’s a small sample from my revised config:

version: '3.7'

x-logging:
  &default-logging
  options:
    max-size: '12m'
    max-file: '5'
  driver: json-file

services:

    db:
        image: someimage:version
        environment:
          BLAH: 'blah'
        logging: *default-logging

    worker:
        image: path.to.my/worker:version
        depends_on:
          - rabbitmq
        logging: *default-logging

OK, obviously a bit stripped down, but the gist of this is I borrow the config directly from the Docker Compose docs.

The one thing that I had to do was to put the x-logging declaration above the services declaration. Not sure why the order matters, but it didn’t seem to want to work until I made this change.

Once done, restarting all the Docker containers in this project (with the revised Docker image for the workers) not only resolved the log spam, but helpfully removed all the old containers – and associated huge log files – as part of the restart process.

Another fine disaster averted.

A Makefile To Run Makefiles

Here’s a small helper thingy that I use to help manage multiple docker-compose setups in production. But this will work with anything that uses Makefiles.

The idea is that I have a helpful Makefile per docker-compose project, as kinda mentioned in this post.

In short, using a Makefile per project allows me to mask away some long winded commands that make kick starting each environment much easier than it may otherwise have been.

The problem is that I have one Makefile per project directory, and some projects have several services. An example might be a project with:

  • www
  • api
  • management console
  • demo site

And so on.

Whilst it’s nice to have one command per service, it does still mean I have to log on to the server, cd to each dir, then run the make start command. And in some cases this needs to be done in a particular order, so that dependant services are up before workers try to connect, and so on.

A better way is to have one Makefile in the project root dir, which then calls the make start command in each sub dir. Something like this:

touch /docker/myproject.com/Makefile
vim /docker/myproject.com/Makefile

start:
    make start --directory /docker/myproject.com/api && \
        make start --directory /docker/myproject.com/admin && \
        make start --directory /docker/myproject.com/www

This way I can now just run one command on the project root dir, and it will take care of calling all the sub tasks that kick start the project.

How I Fixed: Docker-Compose Exec ERROR: No container found for…

This is a pretty specific problem to my infrastructure, but it may come to bite you, too.

There’s a very simple fix to the problem whereby you run a command like:

docker-compose exec mysql /bin/bash

And you see:

ERROR: No container found for mysql_1

But I’m going to assume you have run docker-compose up -d already 😉

In my case, this one is a little bit more subtle.

Multiple Docker Compose Projects

In my case, I’ve recently migrated various prod servers from Rancher v1.x to running docker-compose behind Traefik.

One of the early gotchas was that if I have a directory structure like this:

/docker/mysite.com/www/docker-compose.yaml
/docker/anothersite.com/www/docker-compose.yaml

And say I go into the mysite.com/www dir and run docker-compose up, all is good.

Then I go to the anothersite.com/www dir and run docker-compose up, and docker compose would first shut down the mysite.com/www containers because, by default, docker compose uses the basename of the directory where your docker-compose.yaml file lives as the project name.

To put it another way:

basename /docker/mysite.com/www
www

basename /docker/anothersite.com/www
www

So docker-compose assumes these two different projects are the same thing.

There’s a fix to this.

We can pass in a project name when running docker-compose, like so:

docker-compose up -d...

Of course, make sure your project name differs for each of your projects. And once done, your individual docker-compose projects should run in the way you would intuitively expect.

But, this creates another problem. The subtle problem I mentioned above.

Once you start docker-compose projects in this way, all subsequent docker-compose commands need to the -p my_project_name flag. Or they will do the (apparently) unintuitive thing.

cd /docker/mysite.com/www
docker-compose up -p my_project_name -d

Starting my_project_name_nginx … done

docker-compose top
# ??? - nothing shown

docker-compose exec mysql /bin/sh
ERROR: No container found for mysql_1
# ??? wtf

This confused me for a good half an hour or so, even leading me to upgrade docker-compose, try restarting docker, try rebooting the production server… the works.

Of course, none of that worked.

What did work was to include the project name with the command!

docker-compose -p my_project_name top

my_project_name_mysql
UID PID PPID C STIME TTY TIME CMD
999 9998 9973 0 10:30 ? 00:00:01 mysqld

# and 

docker-compose -p my_project_name exec mysql /bin/bash
root@3cb4c8b263ca:/#

Not that this isn’t a bit of a ball ache, but still, at least now it makes sense.

Ubuntu 20.04 Portrait Mode Monitor Problems

On Saturday dinner time I decided to bump my Ubuntu 19.04 release to Ubuntu 20.04. What could go wrong?!

Well, here I am on a completely fresh installation. So it turns out: quite a lot.

I knew I was ballsed when the upgrade process failed in the terminal. I did an apt-get update and it seemed to think I was already on 20.04. As soon as I rebooted, of course, the OS never came back up. Sad times.

Anyway, tons of other difficulties aside, the issues I hit upon when finally re-installed were not that new to me when it comes to Linux:

Monitor problems.

In particular, I had two.

Firstly, the mini-display port to display port cable just inexplicably died on me. This completely threw me as all of a sudden my main monitor – a Dell P2715Q – seemed to be working, but the screen was black. I knew something was amiss as when I turned on the screen, Ubuntu would make it my main display but of course, it was all dark so I couldn’t see the log in prompt, or stuff like that.

Long story short – after a full re-install – I realised the cable was at fault. Sad times, and more hours lost.

But that’s fixed now. All it took was a new display port to display port cable, which very fortunately, I had in the spare parts box.

Portrait Mode Problems

A new one on me for Ubuntu 20.04.

I have another 27″ Dell monitor, a U2713HMt, which isn’t 4k. And has therefore been relegated to my second monitor.

I’ve never had issues with this monitor. It is always detected as my primary during install, so I have to install with my head tilted 90 degrees… or just turn the monitor back round, but aside from that, it’s been really solid.

Incidentally, having a 27″ screen at 2560×1440 makes a really nice super big terminal window if you are a massive nerd like me, and spend a lot of time in such places.

Anyway, Ubuntu 20.04 did not like putting this monitor into Portrait Right.

The issues I hit were that it would allow me to specify the setting, but when applied, it would either revert, or kill the monitor entirely.

Fixing Ubuntu 20.04 Portrait Mode Problems

Unfortunately I have not found a fix for the “Screen Display” menu. It won’t take the setting directly.

However, there is a workaround that seems to be working about 95% of the time for me.

sudo apt install arandr

Firstly, I installed arandr. This is a GUI for the more cryptic XRandR. If you’re a whizz with XRandR you can likely do the next bit from there directly somehow. And likely you don’t need blog posts like this to get your PC working properly. Fair play to you.

For the rest of us…

When installed, run arandr but run it as you, not sudo.

Make your monitor setup look how you want it. It’s intuitive enough. The cable names and types are labelled sufficiently that you should be able to figure out what is what.

The “Outputs” section is used to select the individual monitors and then modify them as needed. I just needed to make DVI-I-1 into Orientation “Left”.

Once done, choose “Save As…” and this give your file a name. I called mine triple.sh

triple.sh

Oh, the wit.

The reason we didn’t run this as sudo is because this file will, by default, be saved to your home directory, e.g. /home/chris/.screenlayout/triple.sh

You should be able to Apply this script now (from arandr) and your monitors should be correctly displaying.

However, this won’t last between reboots.

In order to persist between reboots, I used a “startup application” entry.

To get to this, hit the super key and type “Startup”, and then I entered the following:

Giving me this:

So far, it’s worked every time except once, when it didn’t. For which I have no reason.

But hey, that’s Ubuntu baby. If you want an easy life, blow 3 grand on a Mac.

Edit: Less than 24 hours later, I have found that when resuming from sleep, I need to re-run the script. Not ideal. To make this as easy as possible, I moved the shell script to my desktop, and updated the start up script location above to point to the new location. Now, I just double click the file on my desktop whenever I resume and the monitors are out of whack.