I probably wasn’t clear enough in my first reply, but I do the same as you and make symbols and components reusable as early as possible during my UI design process. When I said the design system wasn’t done until near the end, I meant it wasn’t complete until then. It’s started early and in progress all throughout the design process, which means the further along you get, the more economical things become.
For me this issue all boils down to using the right tools and the right time. Wireframing is one of many UX/UI design tools. It’s not a necessity for every project, nor it is a necessity to skip it. If it adds value to your process, use it. Simple as that. There doesn’t need to be (and can’t possibly be) a universal agreement about their value, because every organisation and individual designer has different strength, processes, and communication styles.
Tech industry people often fetishise “process” too much. Process is important because it forms a framework for success that can be consistently repeated. But just like there’s often more than one good solution to a design problem, there’s more than one good process to arriving at that solution. Sometimes we need to step back and forget what tools we use and processes we follow. Those are a means to an end. It doesn’t really matter what tools you use if your process is getting good results.