Reduce initial server response time is so high

Server Response Time (SRT) is the amount of time between when a web client makes a request and the server responds to that request. SRT is measured in a unit known as TTFB (Time to First Byte). TTFB is the time it takes for the browser to receive the first byte in response to the browser request. It is measured in milliseconds. You should reduce your server response time under 200ms (milliseconds). There are multiple strategies to improve your server:

Choose Your Web Server Carefully
Utilize a Content Delivery Network (CDN)
Implementing server-side caching
Optimize Your Database
Minify and combining external files

There are various tools to test and improve server response time, You can use the following:

Google Page Speed Insights
Web Page Test