Posts on Shyam Jos

Fix - Error: Loading repository 'fedora' has failed

Tue, 16 May 2023 00:00:00 +0000

Recently when I tried to update my Fedora laptop I got the below error message

Error: Loading repository 'fedora' has failed

After some debugging I found out that this is due to a corrupted repo cache, we can easily fix this error by clearing all repo caches by running the below command

sudo dnf clean all

If that doesn’t work, you can run dnf command with debug options to see more details about the error.

dnf update --debuglevel 3

A hand picked list of DevOps/SRE Newsletters

Wed, 21 Sep 2022 00:00:00 +0000

I often get messages from colleagues and friends how I keep myself updated with DevOps/SRE related news and articles, So today I am going to disclose my secret sources 😉 .

Here is a the list of my top newsletters related to DevOps and SRE

Devops Weekly : A weekly slice of devops news brought to you by Gareth Rushgrove.

SRE Weekly : SRE Weekly is a newsletter devoted to everything related to keeping a site or service available as consistently as possible

Monitoring Weekly : Best articles, news, and new tools related to monitoring

LearnK8s : Curated Kubernetes news, article and resources

KubeWeekly : The weekly newsletter for all things Kubernetes and beyond.

eks.news : This newsletter is curated by the Amazon Elastic Kubernetes Service (Amazon EKS) Developer Advocates.

DevOps Bulletin : A hand curated digest of must-read DevOps tutorials, books, podcasts, open-source projects and jobs.

Bonus Content! 🙌 #

r/devops : Subreddit for everything DevOps

r/sre Subreddit for everything site reliability engineering.

r/ExperiencedDevOps : Subreddit for experienced DevOps engineers and those who want to learn from them!

HN digest : Daily email with top Hacker News stories

hackernewsletter : Weekly email with top Hacker News stories

That’s it for now, I will try to keep this post updated with new newsletters as I find them.

HPA - How to gracefully scale down pods with PreStop hooks

Thu, 30 Sep 2021 00:00:00 +0000

One of our application running in kubernetes have a background job feature (which is a bad practice and against stateless architecture) and recently we noticed some of these background jobs were killed in midway due to pod scale-down done by horizontal pod autoscaler. Our assumption was Kubernetes will only terminate pods with less utilization but we were wrong! and its a random selection.

In order to handle this issue we found out that we can use prestop hook feature in kubernetes to check for any background process running in container and wait for it to complete before sending termination signal.

How PreStop hooks works in kubernetes #

Kubernetes provides container lifecycle hook framework to run code triggered by events during their management lifecycle called PostStart and PreStop hooks. A PreStop hook is called immediately before sending termination signal and kubernetes will wait until the PreStop hook to complete or until it exceed the terminationGracePeriodSeconds value, Only after this kubernetes will sent the termination signal to the container.

Our solution #

Our solution was to use a bash script for PreStop hook which will check for .lock files created by the background process, if it finds a .lock file the script will wait for the .lock file to be removed and when the file is removed the script will exit then the container will receive termination signal. If the PreStop hook takes more time than terminationGracePeriodSeconds value then container will be terminated immediately once this value is crossed.

PreStop hook bash script #

#PreStop Script for checking background php tasks 
SECONDS=0
while true
do	
if [ -z "$(ls -A /app/demo-web/public/lock/*.lock 2> /dev/null)" ]; then
 echo "No lock files found!, container is safe to terminate [Time Elapsed: ${SECONDS}s]."
 exit 0
else
 echo "Lock files found!, waiting for background process to complete [Time Elapsed: ${SECONDS}s]."
 sleep 10
fi
done > /proc/1/fd/1 # sent outputs to conatiner's stdout

Final deployment yaml #

apiVersion: apps/v1
kind: Deployment
metadata:
 name: demo-web
 namespace: production
spec:
 progressDeadlineSeconds: 300
 revisionHistoryLimit: 5
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 5
 maxUnavailable: 0
 selector:
 matchLabels:
 app: demo-web
 tier: web
 environment: production 
 template:
 metadata:
 labels:
 app: demo-web
 tier: web
 environment: production
 spec:
 terminationGracePeriodSeconds: 300
 containers:
 - name: demo-web
 image: demo/demoweb:v1
 lifecycle:
 preStop :
 exec:
 command: ["bash", "/app/scripts/prestop.sh"] 
 imagePullPolicy: Always
 resources:
 requests:
 cpu: 50m
 memory: 150Mi
 limits:
 memory: "650Mi" 
 ports:
 - containerPort: 80
 readinessProbe:
 httpGet:
 path: /healthz
 port: 80
 initialDelaySeconds: 2
 periodSeconds: 5
 successThreshold: 1
 failureThreshold: 2
 timeoutSeconds: 2
 env:
 - name: HOST 
 value: "demo.app.com"

How to redirect PreStop hook output to pod logs

Sat, 25 Sep 2021 00:00:00 +0000

You might have noticed that command output of PreStop hook won’t appear in pod logs and only thing you can see is whether the PreStop hook command failed or not is by checking kubernetes events log.

So how can we see the PreStop hook command output in pod logs?

By default container engine only redirects the stdout/err of the main process (running with PID 1) to the logging driver. Since the PreStop hook runs as a separate process the output from this command won’t appear in container logs.

How to redirect PreStop hook command output to container logs ? #

In Linux you can easily redirect stdout/err of one process to another by writing to the other process’s stdout/err file descriptor (/proc/PID/fd/1 or /proc/PID/fd/2). In our case we can redirect the stdout of PreStop hook command to our main process’s stdout using the command echo "Hello from the other side" > /proc/1/fd/1 and this message will be appear in the container logs.

Example PreStop hook command with output redirection #

 lifecycle:
 preStop :
 exec:
 command: ["bash", "uptime > /proc/1/fd/1"]

Redirect CodeIgniter logs to container's STDOUT/ERR

Fri, 20 Aug 2021 00:00:00 +0000

As part of migrating applications from EC2 instance to kubernetes I was assigned with a task of containerizing a CodeIgniter application. once the application was containerized I noticed that the application logs were writing to a local file inside the container. so I thought this can be easily fixed by changing the logging path in CodeIgniter config file to php://stdout.

But it was not that easy!!

strangely the log path config in CodeIgniter is a combination of file path + file extension, so if I set the file path as php://stdout and left the file extension part blank, the application will throw an error 'Use a full server path with trailing slash'.

After some digging on google. I found out that there is no easy way to set custom static log path in CodeIgniter and I also found an open feature request for this issue which is not yet implemented by CodeIgniter team :(

Luckily I was able to fix the issue by modifying CodeIgniter logging module which is located at system/core/Log.php, I am not sure if this is an optimal solution , just in-case if you are stuck with similar issue give it a try!

Firstly I removed the file path and file extension check conditions from below code block (system/core/Log.php)

public function __construct()
	{
		$config =& get_config();

		$this->_log_path = ($config['log_path'] !== '') ? $config['log_path'] : APPPATH.'logs/';
		$this->_file_ext = (isset($config['log_file_extension']) && $config['log_file_extension'] !== '')
			? ltrim($config['log_file_extension'], '.') : 'php';

		file_exists($this->_log_path) OR mkdir($this->_log_path, 0755, TRUE);

		if ( ! is_dir($this->_log_path) OR ! is_really_writable($this->_log_path))
		{
			$this->_enabled = FALSE;
		}

		if (is_numeric($config['log_threshold']))
		{
			$this->_threshold = (int) $config['log_threshold'];
		}
		elseif (is_array($config['log_threshold']))
		{
			$this->_threshold = 0;
			$this->_threshold_array = array_flip($config['log_threshold']);
		}

		if ( ! empty($config['log_date_format']))
		{
			$this->_date_fmt = $config['log_date_format'];
		}

		if ( ! empty($config['log_file_permissions']) && is_int($config['log_file_permissions']))
		{
			$this->_file_permissions = $config['log_file_permissions'];
		}
	}

This is how the code block looks after removing all the checks

	public function __construct()
	{
		$config =& get_config();

			$this->_enabled = TRUE;
	}

Now you need to replace below line

$filepath = $this->_log_path.'log-'.date('Y-m-d').'.'.$this->_file_ext;

with this line (use a static file path instead of a dynamic one)

$filepath = $this->_log_path;

Now find and remove the below block code from the file (used for changing permission of log file, not needed in our case)

		if (isset($newfile) && $newfile === TRUE)
		{
			chmod($filepath, $this->_file_permissions);
		}

And finally make changes to the CodeIgniter main config file (application/config/config.php) as below

/*
|--------------------------------------------------------------------------
| Error Logging Threshold
|--------------------------------------------------------------------------
|
| You can enable error logging by setting a threshold over zero. The
| threshold determines what gets logged. Threshold options are:
|
|	0 = Disables logging, Error logging TURNED OFF
|	1 = Error Messages (including PHP errors)
|	2 = Debug Messages
|	3 = Informational Messages
|	4 = All Messages
|
| You can also pass an array with threshold levels to show individual error types
|
| 	array(2) = Debug Messages, without Error Messages
|
| For a live site you'll usually only enable Errors (1) to be logged otherwise
| your log files will fill up very fast.
|
*/
$config['log_threshold'] = 4;

/*
|--------------------------------------------------------------------------
| Error Logging Directory Path
|--------------------------------------------------------------------------
|
| Leave this BLANK unless you would like to set something other than the default
| application/logs/ directory. Use a full server path with trailing slash.
|
*/
$config['log_path'] = 'php://stdout';

/*
|--------------------------------------------------------------------------
| Log File Extension
|--------------------------------------------------------------------------
|
| The default filename extension for log files. The default 'php' allows for
| protecting the log files via basic scripting, when they are to be stored
| under a publicly accessible directory.
|
| Note: Leaving it blank will default to 'php'.
|
*/
$config['log_file_extension'] = '';

/*
|--------------------------------------------------------------------------
| Log File Permissions
|--------------------------------------------------------------------------
|
| The file system permissions to be applied on newly created log files.
|
| IMPORTANT: This MUST be an integer (no quotes) and you MUST use octal
| integer notation (i.e. 0700, 0644, etc.)
*/
$config['log_file_permissions'] = '';

/*
|--------------------------------------------------------------------------
| Date Format for Logs
|--------------------------------------------------------------------------
|
| Each item that is logged has an associated date. You can use PHP date
| codes to set your own date formatting
|
*/
$config['log_date_format'] = '';

/*
|--------------------------------------------------------------------------
| Error Views Directory Path
|--------------------------------------------------------------------------
|
| Leave this BLANK unless you would like to set something other than the default
| application/views/errors/ directory. Use a full server path with trailing slash.
|
*/
$config['error_views_path'] = '';

Once these changes are deployed you will be able to the see the application logs in container’s STDOUT and STDERR logs (docker/kubernetes logs).

How to batch resize and compress images in linux

Sat, 19 Sep 2020 00:00:00 +0000

In this tutorial I will show you how to batch resize and compress images in Ubuntu or in any other Linux distros(debian,arch and fedora).

‘Convert’ command is a very powerful image manipulation utility which comes preinstalled in almost all Linux distributions (ubuntu,debian,arch and fedora) and it is a part of ImageMagick software suite.

Check if convert command is available on your Linux distro #

 convert -version

output:

If the above command is not found on your Linux distro you can install it by typing the below command.

Install convert tool on Ubuntu / Debian #

 sudo apt-get install imagemagick

Install convert tool on fedora #

 sudo dnf install imagemagick

Install convert tool on arch #

 sudo pacman -S imagemagick

Note: It’s always good idea take backup of your images before running convert command.

Batch resize and compress images with convert command #

Below command will resize and compress all images ending with .jpg extension in a ‘for loop’ and save the compressed/resized image as ‘filename-Optimized.jpg’.

–quality : this option is used to set the image compression level in percentage

-resize : this option is used to resize the image to a given resolution , You can use -resize option with width ( -resize 1600x ) only or both width and height ( -resize 1600×900), In both cases convert command will automatically adjust the given resolution to get optimum aspect ratio so you don’t have to worry about stretched images.

Note: some jpeg image extensions may end with .JPG (in capital letters) ,Since Linux is case-sensitive you may need to adjust the bash script accordingly.

Open terminal and cd into the directory containing your photos and execute below command

 mkdir photos-Optimized;for photos in *.JPG;do convert -verbose "$photos" -quality 85% -resize 1600x900 ./photos-Optimized/"$photos-Optimized.jpg"; done

Voilà! Now you have successfully resized/compressed images, You can see the optimized images in the photos-Optimized directory.

Yay! Received my 2019 hacktoberfest swag

Tue, 03 Dec 2019 00:00:00 +0000

Yay! finally received my 2019 hacktoberfest swag

Finally received my 2018 hacktoberfest swag

Fri, 28 Dec 2018 00:00:00 +0000

Yay! finally received my 2018 hacktoberfest swag

Ripping DVDs in linux with FFmpeg

Wed, 03 Oct 2018 00:00:00 +0000

In this tutorial, I will show you a simple way to RIP DVD into a single file using FFmpeg command line tool in Linux.

Install FFmpeg in Ubuntu/Debian/Linux Mint #

sudo apt-get install ffmpeg

Now copy contents of your DVD to local directory and cd into VIDEO_TS directory

Note: If you want to RIP the DVD directly without copying it to the local directory first , then you need to modify last part of the below commands to : /home/your-username/myvideo/rip.mp4

1. RIP DVD into a single file #

Now enter below command to start ripping process in optimum quality, If you want to control the size (eg: limit size to 700MB) of the ripped file, then use the third command

cat VTS_0*_*VOB | ffmpeg -i - -vcodec h264 -acodec mp2 rip.mp4

2. RIP DVD with deinterlace option #

If you want to enable deinterlace option run below command

cat VTS_0*_*VOB | ffmpeg -i - -vf yadif -vcodec h264 -acodec mp2 rip.mp4

3. RIP DVD with quality option #

Use below command to reduce size and quality of output file, Here the ‘-crf’ option sets the quality of the ripped file. The range of the CRF scale is 0–51, where 0 is lossless , 23 is the default, and 51 is the worst quality possible.

cat VTS_0*_*VOB | ffmpeg -i - -c:v libx264 -crf 23 rip.mp4

Above command will read all video files in a folder and pipe the stream to FFmpeg command, after the successful completion of the command you can find the ripped file with name rip.mp4

How to create or restore efi partition in ubuntu

Thu, 24 Aug 2017 00:00:00 +0000

What Type of file system is used in EFI partition ? #

EFI partition in Ubuntu is basically a FAT partition with label “EFI”

How to create / restore an efi partition ? #

Just reformat or create a normal FAT file system partition with a size of 512MB (can be any size) and label it as “EFI”, You can use fdisk command if you are comfortable with cli or you can use gparted tool from a live media of Ubuntu to create new efi partition.

How can I reinstall Grub efi bootloader after creating or restoring efi partition ? #

After restoring or formatting efi partition your system will not boot since bootloader needs to be reinstalled on the new efi partition, follow this tutorial to reinstall grub efi bootloader in your Ubuntu system.

How to reinstall grub2 efi bootloader on ubuntu

Thu, 24 Aug 2017 00:00:00 +0000

In this tutorial I will show you how to reinstall grub2 bootloader on a ubuntu system with efi partition.

Boot from a live usb/cd #

Boot from the ubuntu live usb/cd and select the option “try ubuntu without installing”

Check if EFI is enabled in bios #

Enter below command in terminal\

[ -d /sys/firmware/efi ] && echo "EFI boot on HDD" || echo "Legacy boot on HDD"

If you see “EFI boot on HDD” Message then you are already running on efi mode or if you see “Legacy boot on HDD” message then your system is not running on efi mode, you must change the boot options to efi before proceeding to next step.

Note: First you have to find the partition name where ubuntu is installed, You can use “sudo fdisk -l” to list all hard disks and partitions in your computer. In my case /dev/sda is the name of my harddisk, /dev/sda2 is the partition where ubuntu is installed (root partition) and /dev/sda1 was the efi partition.

Mount and chroot into local filesystem #

Note: You have to replace the sda with device name of your HDD, In most cases it is sda, assuming that you have only one hard disk installed.

Note: If you get any errors while trying to mount your efi partition (/dev/sda1), You may need to reformat your efi partition, In case if it got corrupted or if you accidentally formatted it. Follow this tutorial to restore or reformat your efi partition.

Open terminal and enter below command\

sudo mount /dev/sda2 /mnt\
sudo mount /dev/sda1 /mnt/boot/efi\
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done\
sudo chroot /mnt\
grub-install /dev/sda\
update-grub\

Important : use blkid command to check UUID of your efi partition, check if it matches the value in your /etc/fstab entry , otherwise ubuntu will not boot , You may need to update the UUID especially if you have formatted EFI partition . (Eg: sudo blkid /dev/sda1)

That’s it! Now reboot to test the bootloader

A better email template for icinga2

Sun, 14 May 2017 00:00:00 +0000

I have created a very simple HTML email alert Template for Icinga 2 to replace the default boring text-based email alert. This template can be easily installed by replacing existing alert scripts inside /etc/icinga2/scripts.

Update: I have replaced mail command with mutt due to an incompatibility of options among different Linux distributions. Also, I have added new color highlight feature for different states (ok=green,warning=orange and critical=red).

Screenshots #

Prerequisite #

Make Sure mutt package is installed on your server

How to install #

Backup existing script files (mail-service-notification.sh and mail-host-notification.sh) inside /etc/icinga2/scripts
Download the scripts from this github repo and edit the “Variables for HTML Template” section in both scripts
Now replace the existing scripts with the new ones and Send a custom notification to test new HTML email template.

If you have any issues, please report it here

Monitoring Linux Server with an Arduino

Sat, 03 Sep 2016 00:00:00 +0000

As usual I was so bored and wanted to build something cool on this weekend, So I ended up building an Arduino based Linux server monitoring solution for monitoring my Linux VPS.

The idea was Simple, use an Arduino Ethernet module to make a get request to the php script hosted on the server and parse the response data (free -m,df -h,uptime) and display it on the LCD screen every one minute.

And here is the demo of Arduino Linux monitor in action

Demo #

If you are interested in this project you can download the Arduino sketch from my github page.

Todo List #

Buzzer Notification upon increase in memory/cpu load..etc
Use json instead of php page

Insecure Airtel wifi routers with MGMNT ssid

Mon, 16 May 2016 00:00:00 +0000

Some of the wifi routers (especially beetel models) provided by airtel broadband is paired with two SSIDs(Access point), One is an actual SSID for internet access and another one called MGMNT, which is used by airtel for maintenance / automatic configuration. The default key for MGMNT is 0987654321, which is hard-coded into the router’s firmware and the stupidest part is there is no option to disable it.

Now lets start hacking!, Connect to MGMNT SSID with the key 0987654321. Then Navigate to 192.168.1.1 in your browser and login with the default credentials username: admin and password: password If you are lucky now you can see a network status page (view only)

In recent models airtel patched a vulnerability that allows an attacker to view/change the password of the main SSID by connecting to MGMNT and browsing to the URL : http://192.168.1.1/basic/home_wlan.htm.Since the router i am testing was running on a patched firmware, it didn’t worked.

So the next option was Telnet, I was able to telnet into the router with the credentials admin/password.

Telnet 192.168.1.1

Airtel router telnet

Bingo! after playing with telnet for sometime I found the password for main SSID by typing the command:

show wlan config

hacking airtel wifi password

I was also able to hack another airtel router in a coffee shop with my andriod phone using terminal emulator app.

airtel patched vulnerability in web interface but forgot to patch telnet service which is accessible through the MGMNT ssid , An attacker can use the default user account which has full privilege to view and modify router configuration via telnet.

How to disable MGMNT ssid ? #

Currently there is a workaround for this problem, by modifying the router configuration backup file and restoring it back.Please see this stackoverflow answer.Also change your router’s default password as soon as possible.

Update Raspberry pi firmware from Kali Linux

Sun, 09 Aug 2015 00:00:00 +0000

We can easily update Raspberry pi firmware in raspbian by running the command rpi-update, But in kali linux we dont have rpi-update command by default.

To install the rpi-update in kali, run the following command:

sudo curl -L --output /usr/bin/rpi-update https://raw.githubusercontent.com/Hexxeh/rpi-update/master/rpi-update && sudo chmod +x /usr/bin/rpi-update

Then, to update your rpi firmware, run the following command:

sudo rpi-update

Now restart your rpi to new firmware to take effect.
For more Expert options visit the rpi-update project page

Installing Jekyll on fedora

Fri, 16 Jan 2015 00:00:00 +0000

Follow below steps to install jekyll on fedora

Install jekyll #

sudo dnf install ruby-devel redhat-rpm-config jekyll

Optional packages #

sudo dnf group install "C Development Tools and Libraries"
sudo dnf install rubygems-devel nodejs bundler

Testing different DIY antennas for RTLSDR

Wed, 02 Apr 2014 00:00:00 +0000

I have been experimenting with different DIY antennas for RTLSDR since 2013,Below is my review on different antennas I have tested so far.

For decoding NOAA weather satellite Signals #

Quadrifilar Helix (QFH) Antenna is the most recommended antenna for APT signal reception, But I was too lazy to build one. So i decided to make a ground plane antenna with coat hanger. In my opinion GP antenna is the simplest and easy to make antenna for decoding APT signals.Check out my NOAA images decoded with coat hanger GP antenna.

For decoding adb signals (tracking aircraft) #

Coaxial Collinear Antenna is the best antenna for long range adb signal reception,On this page there is a good tutorial for making Coaxial Collinear Antenna.

For decoding AIS signals (tracking ships) #

Coaxial Collinear Antenna provides pretty good reception for AIS signals,On this page there is excellent instructions on constructing high gain coax collinear AIS antenna.

For NFM/FM/WFM/AM Reception #

For me a Ground plane antenna worked pretty well for FM/FM/WFM/AM reception.You can use GP antenna calculator to fine tune your GP antenna to a specific frequency for better reception.

Decoding NASA's NOAA weather satellite signals with RTL-SDR

Thu, 03 Oct 2013 00:00:00 +0000

NOAA satellite is a series of weather satellites launched by NASA. It carries a suite of instruments that provides data for weather and climate predictions, everyday multiple NOAA weather satellites pass above us. Each NOAA weather satellite broadcasts an Automatic Picture Transmission (APT) signal (137MHZ), which contains a live weather image of our area. RTL-SDR dongle combined with a good antenna and programs like GQRX, sdrsharp and WXtoImg we can decode this signals.

If you are not familiar with RTL-SDR, RTL-SDR is a very cheap software defined radio that uses a DVB-T TV tuner dongle based on the realtek RTL2832U chipset. Basically , this means that a cheap $20 TV tuner USB dongle with the RTL2832U chipset can be used as a computer controlled radio scanner.

After a long trial and error process with different antennas finally I am surprised with APT signal reception from my simple ground plane antenna which is made from coat hanger.I don’t no more technical stuff about antennas, My GP antenna have 22 inch long radials ( slant down at an angle of 45 degrees) close up photo of antenna.

The 4 radials are connected to metal base plate and the center radial is wrapped in an insulation tape (no contact with metal plate or other 4 radials ) I used a 75 ohm coax cable for the feed line and it is less than 5 meter long, center copper wire of coax is connected to the center radial and silver wires to the metal base plate.

Software setup #

Software used is sdrsharp (RF gain 100% ,40~42khz bandwidth,90% AF gain,Sample Rate: 1,024 MSPS,audio sample rate: 48000) and the audio is piped to WXtoImg via Virtual Audio Cable (48000 sample rate output).