In my current experience, the slow-down I get from org-agenda when I assign the ~/Documents/org/roam/ directory as the org-agenda-file variable, is less than 10 seconds. I have around 20-30 org-roam docs currently. ALSO, the slow-down that I described as less than 10 secs only occurs during my first time activating org-agenda. In the consequent visits to my org-agenda, it loads instantly. I am guessing, after the first time, org-agenda caches its TODO item finds among the files it has checked.