Discussion about this post

User's avatar
Abhishek Pan's avatar

The FY2022 vs FY2025 mixup can also be fixed on the vector side if ingest pulls fiscal year into chunk metadata and the query filters on it before ranking. The tree walk adds an LLM call or two to every query, and on a high-volume endpoint that latency shows up fast.

Nan Deng's avatar

The number that changes the comparison for me is not the 63/63 tree score by itself. PageIndex's own test on NVIDIA's FY2022 to FY2026 reports says tree retrieval got all 63 right at about $110 per 1,000 queries, while the best vector setup only reached 92.1% by sending the top 100 chunks at about $134, and top-10 stayed at or under 76.2% at about $15. The failure I would eval first is the wrong year: two debt sections look the same to an embedding, and nothing flags it because the chunk really is on topic. I would also version the tree with the filing it was built from, since the post notes an independent run skipped trees after 2 to 5 minutes of LLM calls per document, and a stale summary node is a silent wrong-year answer if the report is amended and the index is not rebuilt.

Ready for more?