Otdfbt

On Google Compute Engine, I have a startup-script that uses gsutil and gcloud, which are part of the google-cloud-sdk snap that is preinstalled on the ubuntu-minimal-1804-lts public image, which is the base image for the image that I am using. But when I ran the startup script today, it failed 4 out of 8 times because snapd decided to auto-refresh at the same time that the startup-script is run.

Within the startup-script, how can I wait until snap’s auto-refresh has completed?

Here are the commands I ran to determine that snapd was updating at the same time as my startup-script:

$ snap changes --abs-time
ID Status Spawn Ready Summary
3 Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Auto-refresh snaps "core", "google-cloud-sdk"

$ snap tasks 3 --abs-time
Status Spawn Ready Summary
…
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:45Z Ensure prerequisites for "google-cloud-sdk" are available
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:46Z Download snap "google-cloud-sdk" (82) from channel "stable/ubuntu-18.04"
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:47Z Fetch and check assertions for snap "google-cloud-sdk" (82)
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:48Z Mount snap "google-cloud-sdk" (82)
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:48Z Run pre-refresh hook of "google-cloud-sdk" snap if present
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:48Z Stop snap "google-cloud-sdk" services
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:48Z Remove aliases for snap "google-cloud-sdk"
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:48Z Make current revision for snap "google-cloud-sdk" unavailable
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Copy snap "google-cloud-sdk" data
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Setup snap "google-cloud-sdk" (82) security profiles
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Make snap "google-cloud-sdk" (82) available to the system
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Automatically connect eligible plugs and slots of snap "google-cloud-sdk"
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Set automatic aliases for snap "google-cloud-sdk"
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Setup snap "google-cloud-sdk" aliases
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Run post-refresh hook of "google-cloud-sdk" snap if present
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Start snap "google-cloud-sdk" (82) services
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Clean up "google-cloud-sdk" (82) install
Done 2019-05-16T18:08:34Z 2019-05-16T18:08:49Z Run configure hook of "google-cloud-sdk" snap if present

$ journalctl -u google-startup-scripts
…
May 16 18:08:39 my-instance startup-script[1469]: INFO startup-script-url: + gsutil cp gs://my-bucket/app.tar.gz /opt/my-bucket/app.tar.gz
May 16 18:08:41 my-instance startup-script[1469]: INFO startup-script-url: /snap/google-cloud-sdk/41/usr/bin/python2: relocation error: /lib/x86_64-linux-gnu/libnss_dns.so.2: symbol 
__resolv_context_get, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

On another instance, gsutil failed with a different message:

ERROR: (gsutil) /snap/google-cloud-sdk/41/usr/bin/python2: command not found

I have been trying to reproduce your issue, by creating an instance with the 'ubuntu-minimal-1804-lts' image as you state

gcloud compute instances create 
--image-family ubuntu-minimal-1804-lts 
--image-project ubuntu-os-cloud 
--machine-type n1-standard-1 
--boot-disk-size=15 
--boot-disk-type=pd-ssd 
--zone=europe-west1-c 
--tags=allow-incoming-ssh 
--metadata=startup-script-url="gs://testinggrounds/startup.sh" 
ubuntu-test

and then reboot it several times

for i in 1..7; 
do 
gcloud compute instances reset ubuntu-test --zone europe-west1-c; 
sleep 120s; 
done

but I did not manage to get this problem that you state.

Searching around for the error that you get, I found this thread where the user reporting the issue presents a highly similar problem which might be the same as you have found. The problem is supposed to be fixed, as reported by the team behind it. Can you check if the date for your Ubuntu 18.04 image is prior to the date opening of this thread I shared? Maybe by using a newer image your problem gets solved.

I saw that you said this suddenly appeared not too long ago, so probably it is not the version, but it does not hurt to check.

If this was not the case, then I would propose to just delay the execution of the startup script or try to make it run after other snapd services.

[Unit]
Description=Google Compute Engine Startup Scripts
After=local-fs.target network-online.target network.target rsyslog.service
After=google-instance-setup.service google-network-daemon.service
After=cloud-final.service multi-user.target
After=snapd.seeded.service
After=snapd.service
Wants=local-fs.target network-online.target network.target cloud-final.service
Wants=snapd.seeded.service

[Service]
ExecStartPre=/bin/sleep 20
ExecStart=/usr/bin/google_metadata_script_runner --script-type startup
KillMode=process
Type=oneshot
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

[Install]
WantedBy=multi-user.target

This file is located at /lib/systemd/system/google-startup-scripts.service. I added the After=snapd.service and ExecStartPre=/bin/sleep 20 to try and delay the execution of the service. Your file should look exactly like the one above without my additions.

If it still doesn't work, you can try to add other snap services with the After= directive, as there are a few more in the systemd services folder:

$ ls /lib/systemd/system/ | grep snap

snapd.autoimport.service
snapd.core-fixup.service
snapd.failure.service
snapd.seeded.service
snapd.service
snapd.snap-repair.service
snapd.snap-repair.timer
snapd.socket
snapd.system-shutdown.service

Hope it helps!

I ended up running snap changes until everything was Done by putting this at the top of my startup-script:

# Returns 0 if snapd is not updating google-cloud-sdk
# (nothing has happened, or all changes are Done, Hold, Error)
# Returns 1 if snapd is updating google-cloud-sdk
# (there is at least one chnage that is Doing, Abort, Undo, or Undoing)
# Echos the offending line
# https://serverfault.com/questions/967674/how-to-wait-for-snapd-auto-refresh-to-complete
function isSnapIdle [ "$state" = Undo ] 
function waitUntilSnapIdle 
 local lastSnapChange=
 local snapChange
 while ! snapChange=$(isSnapIdle); do
 if [ "$snapChange" != "$lastSnapChange" ]; then
 echo >&2 "Waiting for snapd to finish installing: $snapChange"
 fi
 lastSnapChange="$snapChange"
 sleep 1
 done
 echo >&2 "Snapd does not appear to be installing google-cloud-sdk"


waitUntilSnapIdle