Locked Shared records with missing Gitlab-Runner

I ran into this problem during my first CI/CD functional testing using our GitLab environment. At some point, the system was updated and the AutoDevops configuration for every repository was enabled. It so happens that we were also testing the Kubernetes integration, and created a shared runner to deal with Docker container deployment in our OpenShift cluster. Fast-Forward pass the failures and bad house cleaning and what you find today is approximately 100,000 records of failed automatic deployments headed for a shared runner that doesn’t exist.


I didn’t find a way to auto-select and clear the records and since the runner was deleted some time ago, my only remediation was to select the delete option to the side of each record. Even if I did have time for 10,000 mouse clicks, I refuse to do that on principle.

Note: Do whatever safeguard method you prefer before trying this; snapshots of the virtual machine, application database backups, etc… This is just how I fixed my problem and in no way a guarantee to not break something else.

After some research, in which, I looked for rake tools, migration files, etc, I found that there was a way to get into the Database console.

$ gitlab-rails dbconsole

Inserting this command at the root prompt on your Gitlab instance will drop you into the database. From here, you are basically working in the Gitlab database, so be careful. After getting into the database, I took a look at the tables, using \d, which is standard commands for my chosen PostgreSQL back end.

gitlabhq_production=> \d

                                  List of relations
 Schema |                          Name                          |   Type   | Owner
 public | abuse_reports                                          | table    | gitlab
 public | abuse_reports_id_seq                                   | sequence | gitlab
 public | appearances                                            | table    | gitlab
 public | appearances_id_seq                                     | sequence | gitlab
 public | application_setting_terms                              | table    | gitlab
 public | application_setting_terms_id_seq                       | sequence | gitlab
 public | application_settings                                   | table    | gitlab
 public | application_settings_id_seq                            | sequence | gitlab
 public | approval_merge_request_rule_sources                    | table    | gitlab
 public | approval_merge_request_rule_sources_id_seq             | sequence | gitlab
 public | approval_merge_request_rules                           | table    | gitlab
 public | approval_merge_request_rules_approved_approvers        | table    | gitlab
 public | approval_merge_request_rules_approved_approvers_id_seq | sequence | gitlab
 public | approval_merge_request_rules_groups                    | table    | gitlab
 public | approval_merge_request_rules_groups_id_seq             | sequence | gitlab
 public | approval_merge_request_rules_id_seq                    | sequence | gitlab
 public | approval_merge_request_rules_users                     | table    | gitlab


From there I found a table that was likely relevant to my view; ci_runners, so I ran a simple all records select to verify that the records in the view matched the information in the database table.

gitlabhq_production=> select * from ci_runners;

Bingo!!! Now that I had the table, I wanted to make sure that removing the records didn’t impact the application negatively, so I took a look at all the reference keys contained within that ci_runners.

gitlabhq_production=> \d ci_runners;
    "ci_runners_pkey" PRIMARY KEY, btree (id)
    "index_ci_runners_on_contacted_at" btree (contacted_at)
    "index_ci_runners_on_is_shared" btree (is_shared)
    "index_ci_runners_on_locked" btree (locked)
    "index_ci_runners_on_runner_type" btree (runner_type)
    "index_ci_runners_on_token" btree (token)
    "index_ci_runners_on_token_encrypted" btree (token_encrypted)
Referenced by:
    TABLE "clusters_applications_runners" CONSTRAINT "fk_02de2ded36" FOREIGN KEY (runne
r_id) REFERENCES ci_runners(id) ON DELETE SET NULL
    TABLE "ci_runner_namespaces" CONSTRAINT "fk_rails_8767676b7a" FOREIGN KEY (runner_i

In my case, there was no associated data, so with my snapshot running, I deleted the rows.

gitlabhq_production=> select * from clusters_applications_runners;

gitlabhq_production=> select * from ci_runner_namespaces;

gitlabhq_production=> delete from ci_runners where ip_address = 'XXX.XXX.XXX.XXX';
DELETE 94514

gitlabhq_production=> \q

This cleared up the view and at the point of this post, after deploying a new runner and running some dedicated pipeline jobs, I have seen no negative impacts. This may not be the proper way to approach this solution, but it definitely saved me from clicking a button 94,514 times.

MacBook Environment Setup for Packer using Ansible Remote

So I am late to the Packer party for my organization. I have been watching the products that came out of Hashi Corp and have used some in a past life (Vagrant). In my current role, I had an organization need to create RHEL 7 templates directly from the product iso. This ensures security compliance with minimal historical cruft that our images collect over time due to system updates. I had a need to install the latest packages and tools for team members to manage those systems after they are built. Enter Packer!!! Before, we get started this post will cover some MacBook environment setup.

First, you need to have Homebrew installed. I will make a post about Homebrew when I have some additional time but for now, just check out the site or just run the following command on your MacBook preferred terminal application.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Use some Homebrew commands to verify your Homebrew installation.

$ brew --version
Homebrew 2.1.10
Homebrew/homebrew-core (git revision 13869; last commit 2019-08-22)
Homebrew/homebrew-cask (git revision cb8b32; last commit 2019-08-22)

$ brew update
citus                        goofys                       logtalk                      qemu                         wtfutil
ckan                         goreleaser                   lz4                          qpid-proton                  wxmaxima
clamav                       gradle                       mailutils                    rebar3                       xxhash
clojure                      grafana                      mdds                         rke                          yank
cockroach                    grpc                         minio                        roswell                      ykman
contentful-cli               gtk-doc                      minio-mc                     rust                         youtube-dl
cpl                          h2o                          mkcert                       s-nail                       zelda-roth-se
cromwell                     h3                           mkvtoolnix                   saxon                        zsdx
crowdin                      hapi-fhir-cli                mmark                        scala@2.12                   zstd
crystal                      haproxy                      mosquitto                    schismtracker
deno                         hashcat                      mrboom                       serverless
devspace                     helmfile                     mycli                        skaffold
diffoscope                   helmsman                     n                            sonobuoy

After brew is up and running use it to install Packer.

brew install packer

Verify your packer installation completed successfully using the Packer command.

$ packer --version

Instead of the packer provided binary for creating images to be used in our VMware environment, we are using the packer-builder-vsphere, which is provided by Jetbrains. You can find instructions for plugin installation on the Packer docs site. In my instance, I have stored the plugin in the same directory as my Packer code.

  1. Download the plugin from the release page.
  2. Rename the plugin removing your version of the os. For example the release download will be packer-builder-vsphere-iso.macos modify it to packer-builder-vsphere-clone
  3. Add execute rights to the plugin chmod +x packer-builder-vsphere-clone

Next, you have to install Ansible. The proper way to use python packages is to install them in an isolated virtual space, but that’s a completely different post. Once it’s complete, I will reference it. At this time quickly install it using Homebrew.

$ brew install ansible 

Check your installation using ansible.

$ ansible --version
ansible 2.5.0

So I have recently just committed to blogging regularly, and I don’t want to have ungodly long articles. Given that I am going to make several additional posts on this subject which will review the good bits now that I have a reference point for all the lame environment set up stuff ūüôā

#PGHLittleHack – June 2019

As we bid farewell to our old space, it’s been such a pleasure to have others in the #vCommunity offer up their time and space to help us continue our regular meetups. This month’s #PGHLittleHack will be hosted @Expedient. @iamjohnwhite offered to set us up with a space for our little group. As if that wasn’t enough, he has also offered up a rare opportunity to play with a 3D printer. I have included the location details, how to gain access, start and end time below.

Date and Time:
June 10, 2019, at 6:00 PM until …

Location Details
1 Allegheny Center Mall
Pittsburgh, PA 15212

Please enter Gate 4 of the garage. This entrance is directly up from PNC Park. It’s the first left as soon as you merge from Federal Street onto South Commons.

As a reminder, this is an event in which all are welcome. Everyone in this community provides value. Since the agenda is open on purpose, please bring anything you want to fill the time. Feel free to bring presentations that you want to get some run time on. I love to learn about new stuff, and different ways to attack old problems. If you are having problems with an implementation or a tough time getting through a particular issue, let’s discuss it. If you are playing with new tools, please share your experience. As a bonus, this is the first #PGHLittleHack for @lauriepcapo. Don’t get me in trouble y’all. ūüôā

We will order pizza once everyone makes it through the entrance. This is a BYOB event, so if you have a particular tasty beverage that you want to drink, please bring it for yourself and maybe one or two to share. Our host has also offered up some hacking snacks like chips and soda. I hope to see all the old faces and some new.

PGHVMUG Usercon – BIG #PGHLittleHack

The #PGHLittleHackers are hosting a BIG hack event on May 22, 2019. Starting at 6:00 pm the night before the Pittsburgh VMUG UserCon, we will be hosting the first BIG #PGHLittleHack.

We have had some success with #PGHLittleHack as a consistent meet up for the VMware User community in Pittsburgh, so we wanted to take the opportunity to host a larger audience. Success to me means new friendships, engagement on social media between colleagues, opportunities to teach and learn have increased. We appreciate the contribution from each and every person, who has had an opportunity to join us on our regular recurring Monday meetup. I personally have seen members grow into #vExperts, presenters, and community leaders. It’s not pride that I feel when I see these accomplishments, it’s more affirmation. This city and the surrounding area houses some of the best technologists that I have ever had the pleasure to know. This group has just given them space and a voice to stretch and grow. We welcome any new faces because I know there are a lot more of you out there. Based on the larger communities excitement to learn, collaborate, and grow, we felt that the day before the Pittsburgh VMUG UserCon was a perfect time for you to make some new friends and hang out with some old friends.

Event Details

When: May 22, 2019, at 6 pm (the night before the Pittsburgh VMUG UserCon)

Location: 12 Stanwix St, Pittsburgh PA 15222 The co-working space beside the lobby area.

Plan to attend? To sign up, just hit us up on Twitter using the #PGHLittleHack hashtag and let us know you’re coming. If you’re building a team, let us know and we’ll make sure there’s a table for you (with space for others to join you, of course). All attendees will need a laptop.

What is this event?
This is a team based event. Each team will work to complete tasks, that you will find out about when the event begins. Since some of the more complex tasks, may require a shared working environment, team-leaders can bring additional items for their team to specifically work on. If you have some gear and are willing to share it with teams that are unable to bring their own, you can. For example, @arielsanchezmora will be toting his 64GB NUC with local storage, an SDN switch, and a Raspberry Pi for anyone to use. If you want to play with public APIs there will be wireless internet access. As we get more event details, we will post updates at blogs.arielsanchezmora.com.

Tips for attendees

Parking and Location Tips:
This location is a co-working space, located in downtown Pittsburgh. Since the Food (Pizza) and Drink (Beer) will be provided by @pghvmug and parking downtown can be a PITA, we recommend using your preferred ride-sharing service or carpool with others that are planning to attend.

What to bring:
If you have specific preferences for food and drink (alcoholic or non-alcoholic), you are welcome to bring your own. You will need a laptop. This will be used to complete the tasks, so be sure to bring one. You are not limited to bring only a laptop, you can bring whatever you think your team will need. Although spectators are welcome, we hope that you will join a team and get involved. There will be things that everyone can complete to help their team to VICTORY. We will have some small prizes for the participants and victors.

PGH LittleHack 1.5

Since I suck at a timely post for this community event, it was strongly suggested by some special friends that we just make this a permanent monthly event. #PGHLittleHack is now Monthly.  We will kick off our first monthly meetup on Monday, May 14th and plan to hold this meetup regularly on the 2nd Monday of each month.  The meetup details are below.

Event Details

Date: Monday, May 14th 2018

Time: 6:00 P.M. til Whenever.


Morrow Park City Apartments,

5250 Liberty Ave. Pittsburgh PA.

Parking: Street Parking is free after 6:00 PM

What to bring: All you need to bring is yourself and a laptop.  If you have a Github account, it would help us to share our code with the larger VMUG @myVMUG community.

If you are interested in what this event is about, please check out the first blog post. In short, we are just about getting together and playing with technology.

This is not a funded event, so you can BYO or we can coordinate food and refreshments at the start of the event.

If you have any questions, please contact Carl Capozza @carlcapozza or Ariel Sanchez @arielsanchezmor.

PGH Little Hack 1.1

I would like to thank everyone for their interest in this little project.  The vCommunity, including but not limited to Western Pennsylvania VMUG (@WPVMUG), vBrownBag (@vBrownBag), and the VMware Code () have brought this concept to life.

William Lam (@lamw) and Alan Renouf (@alanrenouf) thank you guys for the #VMworldHackathon, as well as allowing Ariel Sanchez and I to bring this concept home.

Finally,¬†thank you, Ariel Sanchez (, for “All the Things”. ¬†I know that I have told you in person, so I will not go through the list here. ¬†This wouldn’t have even been possible without you.

The “I don’t want to read all this” version:

We had our first #PGHLittleHack. ¬†We didn’t have a lot of attendees, but we had a lot of fun geeking out. ¬†In the months to come, we hope to have more attendees come and join the discussion. All are welcome.¬† Additionally, we promise to give way more notice for the next one ūüôā

The attendees decided to have some fun checking out HUGO. HUGO is an Open Source website development software that can be tied into Github Pages.  Although our focus was HUGO, as in any conversation, we touched base on multiple subjects including Github repository workflows and Xmind for project mind mapping.  We hope that you choose to join us.

The “chatty” version:

As the night began, we started with some whiteboarding and equipment setup.  Yes, we have a new found respect for anyone that has to set up session equipment.

Using a Pi to share internet

Whiteboard Tables are cool






While Ariel hacked apart his home lab to repurpose equipment, I focused on figuring out what to work on.  Keeping in the spirit in which #PGHLittleHack was conceived, we decided to work on HUGO.  Although other items were on the board, namely Ansible configuration management and a Web front end for the vDocumentation project, our interest in blogging drove our decision.  For Ariel, HUGO could provide functionality as a central hub for all of his blogging activity, which I am sure is a fair amount.  For me, I just wanted to play with something new.  Since my recent post is my actual first blog post EVER, I wanted to get a feel for other products in the blogging ecosystem.

We were able to successfully walk through build and deploy HUGO to each of our user spaces in Github.


  • My page runs from a repository in Github.
  • This configuration allows me to leverage the Github SSL service.


  • Ariel’s page runs in the Github Pages space, which allows him to use his custom domain.
  • He is also using a different theme other than the default.

Overall HUGO is really easy to build and deploy.  Some software installation, a couple of commands using the quick start and some understanding of how  Github Pages works pretty much gets your site up and running.  Adding a CNAME pointer to your personal domains DNS record and a quick change of a HUGO configuration file and you have a new web space.  Some of the positives that I see for HUGO include:

  • Markdown language compatibility.
  • Themes are easily installed and quickly deployed.
  • Git integration for “oops I broke that” moments.
  • With HUGO, you can leverage¬†Github Pages SSL for *.github.io pages or HUGO can be quickly integrated into a CDN.

Since this is the beginning of my use with HUGO, I don’t really have any critical feedback. ¬†I just haven’t used it enough. ¬†I guess that would be an expectation from a¬†#PGHLittleHack.

Other topics came up, but that would make this blog post a lot longer.  I hope that it gives a reader an idea of what we are currently doing with this little project.  I hope that it makes you want to join us sometime. All are welcome to join the conversation.  We will keep you posted on future #PGHLittleHack.