Some days ago I was appointed as GitLab Hero - yeah! Yesterday I was on a Twitter search for the other GitLab Heroes on Twitter and during my surf, I recognized that there is no Twitter list of all GitLab Heroes.

My first thought was to create a list for me and to share it - but hey, this would be a manual process to always keep the list up to date. So… automatize it! 😂

The resulting Twitter List is here: Twitter List of GitLab Heroes - feel free to subscribe! For those who want to read how it is done, read on!

tl;dr

Parsing the GitLab Heroes member site

The idea was simple, just parse the GitLab Hero page and parse the Twitter handles. The Twitter handles are located inside a div element with the attribute class="social". Using the Golang tokenizer was my first move, but this was a bad idea because I am not that experienced in programming. But, the second try was better, I found goquery, a JQuery like library to select HTML elements and get their data. With some regex it was easy then to get the Twitter slugs.

var slugs []string
// Find the review items
doc.Find(".social").Each(func(i int, s *goquery.Selection) {
	// For each item found, get the band and title
	res, _ := s.Find("a").Attr("href")
	matched, err := regexp.MatchString(`^https://twitter.com.*`, res)
	if err != nil {
		log.Fatalln(err)
	}
	if matched {
		// only use the last part of the URL = the Twitter handle
		slugs = append(slugs, path.Base(res))
	}
})

Update the Twitter List via API

OAUTH can be cumbersome and therefore I took a look at the recommended Twitter “SDK’s” for Golang. go-twitter fitted my needs best!. It was more complicated to register the needed Twitter developer account than to write the code to update the Twitter list.

resp, err := client.Lists.MembersCreate(&twitter.ListsMembersCreateParams{ListID: 1221901753425309698, ScreenName: slug})

GitLab Pipelining

The last thing needed, is a GitLab pipeline which will start the code on a daily schedule. The secret information is stored inside GitLab variables. Just running the precompiled Golang code is lazy but at first the easiest way.

stages:
  - update-list

update-list:
  stage: update-list
  image: debian:buster-slim
  script:
    - apt update && apt install -y ca-certificates
    - ./gitlab-heroes-twitter-list
  only:
  - tags

Summary

The code of this project is available in this GitLab repository: https://gitlab.com/m4r10k/gitlab-heroes-twitter-list/tree/2020-01-28-04. Feel free to use it! This given Git tag includes the code shown here! The new code uses the GitLab Heroes yaml file from the Git repository and does not parse the website anymore. If you like to see how you can parse any website to extract the content, please look at the tag above! Thanks!

Mario