Jam and Bees

I created a new k8s user, added a role and a role binding granting it access to list nodes, but when I tried running kubectl --context=bob-context get nodes I received the lovely error error: You must be logged in to the server (Unauthorized). If you search google, you get a lot of results saying “this is probably certificate expiration” and not too much else. My certificates were not expired.

I re-ran the command with higher verbosity and got

I0509 18:48:16.413290 6230 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0509 18:48:16.413417 6230 helpers.go:219] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}]
F0509 18:48:16.413430 6230 helpers.go:118] error: You must be logged in to the server (Unauthorized)

This was useful: the ‘Unauthorized’ bit is just telling me that we received a 401 back, not that I’m actually unauthorized. I may be unauthenticated.

I found my answer in the docs that deal with authorization – https://kubernetes.io/docs/reference/access-authn-authz/authentication/

Reading this, I immediately noticed this paragraph

Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster’s certificate authority (CA) is considered authenticated. In this configuration, Kubernetes determines the username from the common name field in the ‘subject’ of the cert (e.g., “/CN=bob”).

https://kubernetes.io/docs/reference/access-authn-authz/authentication/

I had been lazy when generating my cert and hadn’t specified a -subj field to openssl, so it had generated a default that didn’t match my user’s account name. Once I added that, I got past the error.

I want to follow Sander van Vugt’s CKA course on a home linux box using virtualbox for controller/worker nodes.

This repo seems to contain the right goodies to start with: https://github.com/jeromeza/k8s_cka_vagrant

tl;dr – https://github.com/gwynforthewyn/k8s_cka_vagrant and https://github.com/gwynforthewyn/cka contain the patches to get this working

What Was The Investigation?

• cloned the upstream repos to my linux box
• realise I don’t have virtualbox installed. Install virtualbox with dnf install virtualbox . Try to start virtualbox and receive a nice error:

; virtualbox                                                                                                                                                                                                                               
WARNING: The vboxdrv kernel module is not loaded. Either there is no module
         available for the current kernel (5.16.16-200.fc35.x86_64) or it failed to
         load. Please try load the kernel module by executing as root

           dnf install akmod-VirtualBox kernel-devel-5.16.16-200.fc35.x86_64
           akmods --kernels 5.16.16-200.fc35.x86_64 && systemctl restart vboxdrv.service

         You will not be able to start VMs until this problem is fixed.
Qt WARNING: could not connect to display 
Qt FATAL: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, xcb.

zsh: IOT instruction (core dumped)  virtualbox

• so I installed the kernel mod and loaded it, following the instructions for my kernel version
• wonder why networking won’t work, then read https://www.virtualbox.org/manual/ch06.html#network_hostonly and see that the valid address ranges for host-only networks have changed. Forked upstream, updated and sent in a PR.
• vagrant up fails with a yum error. Read https://cloud.google.com/compute/docs/troubleshooting/known-issues#keyexpired. Fork the upstream github repo for sandervanvugt and patched the scripts.
• Finally, patched the README for the vagrantfile

I updated fedora from 31 to 33, skipping 32 because I live on the edge.

When I tried using DNF, fedora spewed this on stderr warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.

Sweet, no worries! A swift search and you find this advice

In some circumstances [*] users may see messages like “warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.” This is a harmless indication that rpm configuration and what’s on disk disagree. It can be silenced either by running rpmdb –rebuilddb to convert the database to match configuration, or by overriding configuration to match what is on disk (see above).

https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb

Great! I ran rpmdb --rebuilddb and it returned quickly. Running more dnf commands, though, continued giving the same error.

I checked journalctl and found the following errors

AVC avc: denied { read } for pid=12602 comm="rpmdb" name="resolv.conf" dev="dm-0" ino=1966429 scontext=unconfined_u:unconfined_r:rpmdb_t:s0-s0:c0.c1023 tcontext=system_u:object_r:net_conf_t:s0 tclass=lnk_file permissive=0
AVC avc: denied { open } for pid=12906 comm="rpmdb" path="/var/lib/rpm/.rpm.lock" dev="dm-0" ino=799036 scontext=unconfined_u:unconfined_r:rpmdb_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
AVC avc: denied { lock } for pid=13224 comm="rpmdb" path="/var/lib/rpm/.rpm.lock" dev="dm-0" ino=799036 scontext=unconfined_u:unconfined_r:rpmdb_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

I noticed that the errors say the command causing the errors is rpmdb, which is exactly what I was hoping to find. Perfect! I searched for “AVC avc: denied” and found a link to this fantastic page on the gentoo wiki explaining what I was reading https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details

From there, I knew I needed to allow the rpmdb command the permissions that were being denied. I saw that audit2allow seemed to be a command I needed to allow these permissions. The man page lists the purpose of the command as

   audit2allow - generate SELinux policy allow/dontaudit rules from logs of denied operations

I copy and pasted the errors from journalctl into a temporary file I called temp.pp, using audit2allow to create a module I called rpmdb and then installed the module into selinux, reran the rpmdb command and repeated the process until I’d dealt with all of the errors.

audit2allow -M rpmdb < temp.pp
semodule -i rpmdb.pp

As an experiment, I tried repeatedly adding the rules to the same temp file and re-adding the temp file with the same audit2allow/semodule commands, worked perfectly.

Reading the man page for audit2allow and this page https://danwalsh.livejournal.com/24750.html (linked to from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-fixing_problems-allowing_access_audit2allow), I’m pretty sure I didn’t need to use the temp file.

Once all the errors were dealt with, the rpmdb rebuilt just fine to

I always keep this blog in the back of my mind, and I keep paying for it, but I’m never 100% sure why.

I went through a lot of blogging as a late-teen/early-twenties person. I may be trying to maintain the illusion of persistence of identity.

A piece of why I maintain this is so that employers have something to look at so they can know before an interview that I have some idea of what technical things actually are. “Oh, yeah, that person wrote a blog post about a terraform thingy, so they may have actually used it.”

But I sort of want to be a bit more than that sometimes. I want to document my own changing changes again. I’ve been writing a lot of that on paper, but it’d be cool to do it in text.

Honestly, I wonder if what I’m really trying to do is persuade myself to write my own blogging software, let myself get lost in building technical details to better express myself without having to go through the hassle of actually expressing myself.

Time to begin.

I got back to looking at HomeBrew today. Something I noticed last week, though, was that HomeBrew was always using the system ruby, even when I’m using rbenv to set a global ruby. I decided to take an hour or two out to figure out why that was happening.

Last week, I spent some time figuring out how HomeBrew does command loading. It’s pretty simple: there’s the brew command, which is a shell script, and mostly it sets some variables and dispatches to Library/brew.sh.

brew.sh sets up a whole bunch of the HomeBrew environment stuff, then it attempts to find a shell version of the command you’ve asked for. If a shell implementation can’t be found, it dispatches to this:

{ update-preinstall; exec "$HOMEBREW_RUBY_PATH" -W0 "$HOMEBREW_LIBRARY/brew.rb" "$@"; }

brew.rb attempt to find a ruby implementation of the command you’re looking for.

So, it looked like $HOMEBREW_RUBY_PATH is probably the thing that I need to research. How does it get set? In brew.sh, we can see the following code:

if [[ -z "$HOMEBREW_RUBY_PATH" ]]
then
 if [[ -n "$HOMEBREW_OSX" ]]
 then
 HOMEBREW_RUBY_PATH="$(which ruby)"
 else
 HOMEBREW_RUBY_PATH="/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby"
 if [[ -z "$HOMEBREW_RUBY_PATH" ]]
 then
 odie "No Ruby found, cannot proceed."
 fi
 fi
fi

What’s super cool here is that we can see that on Max OS X, there’s an explicit requirement to use the system ruby. Rbenv and the like aren’t even _asked_ about what ruby they’re providing. As an aside, any budding hackers who’re interested may want to inject support for rbenv here.

What problem is this addressing? I suspect that the homebrew devs don’t want to have to support multiple versions of ruby on OS X. In fact, the commit that implements this reads:

Replace /usr/bin/ruby with full Framework path

Rationale: some users insist on replacing the /usr/bin/ruby symlink
 to point to another ruby on their system, which may break homebrew.
 Use the full Framework path instead, which is less likely to be tampered with.

This also reorganizes the brew --config checks to reflect the different path.

Fixes Homebrew/homebrew#12009.

Closes Homebrew/homebrew#12333.

To see why this was implemented, I checked the issues noted in the git log message.

https://github.com/Homebrew/legacy-homebrew/issues/12009

That all seems reasonable enough. I had wondered if this was an issue of trying to reduce maintenance overhead, and it appears that it was.

Just through exploring the codebase a little, I can see that homebrew behaves a little differently if you have the environment variable HOMEBREW_DEVELOPER set. So that’s a neat little win.

I wanted to understand how brew dispatches all the way to gist-logs, so I could be sure I’m implementing a feature in the right place. I started out by just calling gist-logs with no arguments and received a handy error message:

usage: brew gist-logs [--new-issue|-n] <formula>

I searched for the string “–new-issue” in the brew codebase and found it’s only in the gist-logs.rb file. So that’s useful! I then got sidetracked trying to figure out why these two behaved differently:

[james@greentreeredsky brew (issue44706)]$ brew gist-logs
usage: brew gist-logs [--new-issue|-n] <formula>
b[james@greentreeredsky brew (issue44706)]$ brew gist-logs --help
Warning: No help text in: /usr/local/Library/Homebrew/cmd/gist-logs.rb
Example usage:
  brew search [TEXT|/REGEX/]
  brew (info|home|options) [FORMULA...]
  brew install FORMULA...
  brew update
  brew upgrade [FORMULA...]
  brew uninstall FORMULA...
  brew list [FORMULA...]

Troubleshooting:
  brew config
  brew doctor
  brew install -vd FORMULA

Brewing:
  brew create [URL [--no-fetch]]
  brew edit [FORMULA...]
  https://github.com/Homebrew/brew/blob/master/share/doc/homebrew/Formula-Cookbook.md

Further help:
  man brew
  brew help [COMMAND]
  brew home

It turns out homebrew has an internal help generation system! If you are writing a new tool, and you use a magic comment beginning with #:, then homebrew will automatically find the help.

Which means that the failure of the –help command is a defect! I’m an hour in and finding something to fix! Yay! I’m going to let myself get sidetracked by this, as it seems like an easy fix.

First, I added the following magic comment to the top of the file:

#: usage: brew gist-logs [--new-issue|-n] <formula>

Now both –help and no arguments behave the same, but at the cost of introducing duplication. I don’t want to introduce duplication, so I want to figure out how a different command handles both the case that someone enters the –help argument and that ARGV is empty.

Fortunately, I’m half way there. –help is handled by the magic comment and a little support in brew.rb. But the way that the empty ARGV is handled internally to gist-logs is with this little piece of logic:

 if ARGV.resolved_formulae.length != 1
  puts "usage: brew gist-logs [--new-issue|-n] <formula>"
  Homebrew.failed = true
  return
 end

So, I need to figure out how to remove that logic, and invoke the help system when there’s no arguments.

I checked the install.rb command to see how it handles this situation. It raises a FormulaUnspecifiedError exception, which is caught in brew.rb (FormulaUnspecifiedError inherits from UsageError). This dispatches to Homebrew#help, which displays the error message and the text from the FormulaUnspecifiedError exception and exits with 1. So changing the code to this seems reasonable:

raise FormulaUnspecifiedError if ARGV.resolved_formulae.length != 1

The problem here’s simple enough, though. The old implementation set Homebrew.failed = 1, but the new code does not set that. The question is, does that change matter?

From what I can tell, Homebrew.failed is used in brew.rb to simply do an “exit 1”. The help implementation also does an “exit 1”, so the command line contract is maintained, and because Homebrew.failed isn’t actually used anywhere in the new call chain, and it wasn’t _doing_ anything in the old call chain, I think the refactor is a win.

I’ll create a branch for this change and submit it to the dev team and see what they think.

And there’s the pull request: https://github.com/Homebrew/brew/pull/217

I decided to work a little more independently on my ruby programming chops, and I wanted to be giving a little more back to the free software world, so I signed up for CodeTriage and picked Homebrew as the project I’ll look at. So far:

• Code’s hosted on github, so that’s easy enough.
• There’s a feature request against gist-logs that sounds simple enough to start with (https://github.com/Homebrew/legacy-homebrew/issues/44706), and it’s what codetriage recommended anyway.
• It looks like cloning the remote repo and then just using the brew command in $HOMEBREW_DIR/bin/brew is enough that you can start hacking away on a local copy of the codebase.
• I haven’t reached out to the devs on the google groups channel yet, but I’ll probably try that later today.
• The dispatch chain when looking for a script is $HOMEBREW_DIR/bin/brew –> $HOMEBREW_DIR/Library/brew.sh –> $HOMEBREW_DIR/Library/brew.rb
• rubocop fails the project on its own style guidelines (I think inappropriately, because most of the violations are in formulae not in core, but I’ll ask the dev team)

So, this is kind of fun!