In discussing this passage, the scholars of Tafsir mention a story which is mostly based upon Isra'iliyat narrations. Nothing has been reported about this from the Infallible Prophet that we could accept as true. But Ibn Abi Hatim narrated a Hadith whose chain of narration cannot be regarded as Sahih because it is reported by Yazid Ar-Raqashi from Anas, may Allah be pleased with him. Although Yazid was one of the righteous, his Hadiths are regarded as weak by the Imams. So, it is better to speak briefly of this story and refer knowledge of it to Allah, may He be exalted. For the Qur'an is true and what it contains is also true.
About Sayyidna Dawud (علیہ السلام) ، it was said in verse 22: فَفَزِعَ مِنْهُمْ (and he was scared of them). The reason of being scared was quite obvious. Two men barging into one's privacy in that manner at an odd hour beating the security cordon can only be because of some bad intention in most cases.
Natural fear is not contrary to the station of a prophet or man of Allah
This tells us that being naturally affected at the sight of something fearful is not contrary to the station of a prophet or man of Allah. Yes, allowing this fear to overtake one's heart and mind and abandoning duties enjoined is certainly bad. Hence, a doubt may arise here. Is it not that the Qur'an portrays the elegant status of the noble prophets by saying: لَا يَخْشَوْنَ أَحَدًا إِلَّا اللَّـهَ. (They do not fear anyone but Allah - AI-Ahzab, 33:39)? Why then, was Sayyidna Dawud (علیہ السلام) in fear on this occasion? For an answer, it could be said that there are two kinds of fear. There is the fear that comes from being hurt by what causes pain. In Arabic, this is called: خَوفُ (khauf: fear). Then there is the other kind of fear. This emerges because of someone great, majestic, awesome. This is called: خَشِیَّۃ (khashyah: awe) (al-Mufradat, Raghib al-Isfahani). There should be no khashiyah for anyone other than Allah - and noble prophets, may peace be on them all, are exactly like that. They are not awed by anyone. Yes, as said earlier, one can have a feeling of fear from naturally hurtful things.
It is said that David had fixed each of the three days for different purposes—one day for holding court and deciding cases, a second day for spending with his family and a third day for remaining in solitude and praying to God. It happened once that on a day fixed for prayers, as he sat engrossed in his devotion, two men who had jumped over the wall of the palace, entered the room and stood beside him. This was an extraordinary occurrence and, therefore, he was somewhat afraid. But the two men allayed his fears and said that they were parties to a dispute and begged him to settle it for them.
Uthmani text of the Madinah mushaf (King Fahd Complex). Translation: Saheeh International. Tafsir via Quran.com.