Gitlab 6 Upgrade To 7
I ran into a few issues while upgrading from gitlab version 6.x to 7.x. The first thing I would like to say when it comes to upgrading gitlab there guides pretty much step by step guide for every version upgrade.
Issue 1: Ruby Gem Error
First error I ran into was in section 5 of the guide: Install libs, migrations, etc.
While running the following command:
sudo -u git -H bundle install --without development test postgres --deployment
Output From The Error:
gem install mysql2 -v '0.3.16' Fetching: mysql2-0.3.16.gem (100%) Building native extensions. This could take a while... ERROR: Error installing mysql2: ERROR: Failed to build gem native extension. /usr/local/rvm/rubies/ruby-2.1.2/bin/ruby extconf.rb checking for ruby/thread.h... yes checking for rb_thread_call_without_gvl() in ruby/thread.h... yes checking for rb_thread_blocking_region()... yes checking for rb_wait_for_single_fd()... yes checking for rb_hash_dup()... yes checking for rb_intern3()... yes checking for mysql_query() in -lmysqlclient... no checking for main() in -lm... yes checking for mysql_query() in -lmysqlclient... no checking for main() in -lz... yes checking for mysql_query() in -lmysqlclient... no checking for main() in -lsocket... no checking for mysql_query() in -lmysqlclient... no checking for main() in -lnsl... yes checking for mysql_query() in -lmysqlclient... no checking for main() in -lmygcc... no checking for mysql_query() in -lmysqlclient... no *** extconf.rb fail Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options.
The error give some good detail hides if you look at it close. Ruby gem is looking for mysqlclient so all you need to is install it. However I use MariaDB so in my case I needed to install libmariadbclient-dev. If you are using MySQL you need to install libmysqlclient-dev.
apt-get install libmysqlclient-dev or apt-get install libmariadbclient-dev
Issue 2: 100% CPU Spikes On Any Calls That Go To Ruby After The Upgrade to 7.x
This issue ran me around for a while, I thought the caused of the 100% CPU spikes were from the ruby upgrade. Version 7.x of gitlab requires you to move from ruby version 1.x to ruby 2.x. I was trying to pin the CPU spikes on the ruby version changes. However this was not the issues after further research. Gitlab was take 4x the time to load even the login screen and the whole time I’m waiting on the browser to load the page there's a 100% CPU spike on the server. Turns out the spike were caused by ruby trying to deploy some more files/links into it ruby deployment location and the pid was being killed off by Unicorn timeout value before all the files/links could be created. This would create a endless cycle and the files/links would never get created.
So if you are seeing similar behaviour try increasing the timeout value of unicorn.rb located in gitlabhome/config/unicorn.rb. I increased my timeout value to 600 seconds from 30 seconds and restarted gitlab for the setting to take affect. After that I tired to login and watched the 100% CPU utilization till it stopped. After that I logged in and clicked through some of the links in gitlab and I did not see any issues with CPU spikes everything was back to normal. Make sure you lower the timeout value back to the original setting before allowing user to access the site again.