I have an Employee Master Excel Sheet with the office locations of all employees. I want to ensure that the employees are at their designated offices only when they are marking their attendance. Is it possible?

Yes, it is possible. 

You can create an app with all the fields as per the excel sheet. Then bulk-upload all the details to that master app. Reference Link47

The next step would be to create another app where you can pull the details from the master app using the "Get data from other apps" block. Now add a GPS Location block where you need to enable the Geofencing option, add the pulled location field in the "Designated Locations" option on the right panel, and set the radius within which they should be able to submit.

Please note that you also have to turn on the option "Default to current location" and turn off the option to manually change the location as shown in the image below.