What is load average and why does it matter?
Candidate should be able to identify that load average is a good indication of how busy a server is. A load average of less than 1 per cpu core is alright and as it becomes closer to equal to the number of cpu cores it indicates the server has more processes waiting than resources available. They should also identify that load average can be influenced by other factors than just CPU (I.E. iowait).
What is the difference between LVM and a partition. How do you add a more storage to the existing logical volume?
LVM is logical volume manager, logical volumes sit on top of logical extents which are mapped to physical extents which sit on a physical hard drive. LVM's logical extents can also span multiple hard drives.
Traditional partitions are sequential blocks on a physical disks
You can add more storage space to an LVM by using the extend commands, (vgextend to extend your volume group, and then lvextend to extend your logical volume, if you do not have any extra extents to add to your logical volume then you'll need to run pvcreate and then add the new extents to the volume group first).
When extending LVM, when will you add new disk vs adding more space to the existing disk(Assume that it is VM on hypervisor)
The main problem with adding new disks is that your configurations can become messy over time and you start to get a lot of virtual controllers if people keep adding small hard drives instead of extending existing drives. If you have a bunch of small disks on the same storage volume then you should create one larger disk and then pvmove (move the physical extents) from the smaller disks to the larger disk. Once the move is complete then you can remove the other smaller hdds. Sometimes (for example heavily utilized disk IO) you might need create a new disk on another storage volume to help spread the load, in which case you should add a new disk.
In Linux how does OOM killer work?
I usually try to use this question to gauge how familiar the candidate is with scenarios where the box runs out of memory. I will usually first ask if they know what "OOM" killer is, and if they've never heard of it then explain OOM means "Out Of Memory". It is pretty easy to guess that OOM killer will look for high memory consumption.
OOM killer will look for the highest memory consumption applications and give it score, it will then look at the time the process has been running for and reduce score for longer running processes. It also takes into consideration niceness (higher nice number is safer to kill where lower nice number is in theory more important). It takes all of these factors into consideration and kills the highest scoring application. There is also a flag that can be defined in /proc/<pid>/ to prevent OOM killer from terminating a specific process.
Most candidates will not know all of the factors of OOM killer, but the more the better
Describe a linux package management system and why it is useful.
Hopefully they can describe rpm or debian package manager and explain that it's useful for version control and consistency.
What is difference between hard and soft link, from inode and filesystem perspective?
Soft link is basically a file string pointer that can point to another directory or file (spanning partitions), a soft link is it's own file and has it's own inode number.
Hard link is basically another name for the same file existing on the same partition. It points to the same inode number. If you have two hard links and delete either one of them, the file is still there. If you have a soft link pointing to a file and you delete the file that the soft link is pointing to, the soft link breaks and the file is gone.
(Note: hard links cannot span multiple partitions, if they say they can both point to directories or across multiple partitions; mark it as wrong)
Difference between ctrl+Z and ctrl+D
CTRL + Z pauses the running application and you can choose to throw it in the background with the bg command or bring it back to foreground with fg
CTRL + D sends a signal to indicate that you're done without a harsh kill signal like CTRL + C
What does iowait mean when your looking at your system stats and why does it matter?
Iowait is the time a process is waiting and unable to preform any actions due to a IO block, usually caused by resource contention or because the resource is too slow. This is one of the biggest reason why load averages spike (so they might have touched upon this in the first question)
When is swapping ok and when is it bad?
Swapping is ok when it's high niced applications or when the memory swapped is applications that do not need access to that memory very often. If swap usage is not growing at all and not occurring very often then a little usage of swap is not very concerning.
Swapping is usually bad as it indicates there is not enough available memory for all of the applications and as a system begins swapping it drastically slows down as it starts placing memory that should be stored in RAM onto a hard drive / storage volume.
What is ulimit and why does it matter?
Ulimit allows you to control system settings, for example core file size (if an application crashes and should generate a core dump; what is the maximum file size this should be), max memory allowance, open files… etc (uname –a would show you configurable options and current limitations).
How do you look at open file descriptors?
lsof
How do you check for version of a package? How do you update to new version, if one is available on lets say on CentOS?
Yum search / yum check-update / yum update
If a daemon process is not starting, where would you look to figure out why it isn’t starting?
Good starting location would be to check the logs for the process
Describe how DNS works?
They should be able to touch upon DNS is the resolution of a name to IP address and works like a tree. If their local dns server does not have the answer it has to goto the "root" of the tree, and start going down the name servers until it finds a response
I will also usually ask about the protocol as well, like does it use tcp or udp (it actually uses both, tcp is the only way you can get large dns responses though, udp is by far more commonly used though)
Describe how tcptraceroute is different from traceroute and why it might be helpful (Note: if they've never worked with tcptraceroute, then ask how trace route works from a technical level).
Common misconception / miscommunication: tcpdump is not the same as tcptraceroute, if they say tcpdump in their response try clarifying it's tcptraceroute
Traceroute functions by increasing the TTL value on packets sent, regular trace route usually utilizes ICMP or UDP packets to check the path; but this can be denied on a lot of firewalls.
Tcptraceroute allows you to specify port numbers and will send TCP SYN packets. When troubleshooting connectivity issues to a location where you suspect a firewall might be restricting traffic sending tcptraceroutes with an expected port will usually allow you through the firewall and usually allows you to get a clearer picture of whats going on
How to discard output from a script and print only errors from a script? How do you close a file descriptor?
Discarding output from a script: >/dev/null (they might call > a redirect (correct) or "pipe" (technically not correct, but acceptable))
note: if they do 2>&1 (they might say redirect standard error (or 2) to standard out) ) then they will just put the output to stdout and that would not accomplish the goal, especially if they said the above line in combination with this, that will get rid of all output leaving nothing.
Closing file descriptors: 1>&- or 2>&- or 3>&- (they might say standard out, standard error instead of 1 or 2, that’s fine)
How do you do a simple infinite loop, which finds files created in last hr and have more than 1 hard link, sleep for 1 minute?
while [ true ]; do
find . -type f -mmin -60 -printf "%p %n\n" 2>/dev/null | grep -v ' 1$'
sleep 60 #or sleep 1m
Done
Notes:
they shouldn't use a "For" loop, if they say "while true", or "while 1" or "until false" then that should be good for the infinite loop aspect
a lot of candidates don't know exactly the find parameters off hand, They should at least know "find" "type f" and either "m min" or "m time". And if they don't know the rest I would usually ask them if they were in front of a terminal without internet access what they would do. The answer I'm looking for is "man find" in that case.
How do you preform a search and replace on a file (specifically related to scripting, so what commands opposed to using vi or nano or something else)?
sed -e s/<string>/<replace>/g
What is the purpose of “#!/bin/bash” in a shell script?
The first two bytes of the executable file tells the system what interpreter (or application) and optionally default arguments to run, in this case it tells the system to run bash when it is executed
