Part is just experience. You have to have seen enough different kinds of people and systems work together to understand what works and what does not.
I think another part is simply nomenclature. There's a joke about the only two hard problems in computer science: Cache expiry and naming things. It's a joke because cache expiry reduces to naming the keys correctly if you do it right.
Basically, if the different parts of the system don't have sensible names that represent what they actually do (and that there are, indeed, different parts of the system), then there's an architecture problem.
You have to have an effective metaphor for the way that you want the system to work, and the problems you want each part of it to solve, or nobody will be able to understand the entire thing.
I run into this often. People have a functional but struggling system designed with the "big ball of mud" pattern, where you just keep slapping gobs of code onto it until it does what you need, and then they want someone to come in and "do architecture" at it. This predictably fails.
So, there's your heuristic: if it has good names, and the pieces do what they say on the tin, you're on the right path. When something starts to have responsibilities that aren't under that name, you should move those responsibilities somewhere else and name them appropriately.
Edit: There's also a second-order level where you start to recognize the way that cross-cutting concerns happen in different areas and design a piece of the system to handle that concern in an organized way. You can probably think of a bunch yourself, but an easy example you want to handle earlier rather than later is logging, and another significant one is authentication
So a the goal is a bunch of people can have their claws in the code base and avoid this reaction [0], but not necessarily code optimization, just as long as everything is highly modular and works independently of other parts you're kosher.
Right, the lower level stuff you're still looking for (cleanly designed classes and methods, good testing, etc), but you can hit all those micro-level targets without having any architecture to speak of.
(the "pile of classes" design pattern, where everything calls everything else and there's no hierarchy or anisotropy to it)
The other heuristic, besides "names make sense", is ease of modification. When you come back into an area of the system, how painful is it to disentangle that piece from the rest and make a significant change? Is the functioning of the entire system tied together tightly enough to make that difficult? Are there good patterns to extend, or is every change invented from scratch?
Some people just "get" software architecture and some don't.
Part of it is indeed experience, but another part of it is just being able to visualize the entire application (domain experience helps with this) and see it as a series of interconnected modules. Think of the parts individually and what downsides they have and than as a whole.
An example of this is if you have a module that does something very GPU heavy or very bandwidth heavy. On an architecture layer you want want to give it it's own server outside of the main server (in the case of a web app).
In terms of code you need to think about how some modules will need to be updated in the future (want to support multiple payment methods) - and also to think what modules need to be secure or performant. If two modules will always exist together they might be something you can couple if it saves you a large amount of developer time. But if they might be re-used in the future they need their own API's and no close coupling.
By working with people who know how to do it, and learning from them.
There aren't many shortcuts one can take in that area -- the big difference between junior and senior developers is that the senior developers have experienced what can go wrong, often learning from their own mistakes and the mistakes of their employers, so they are better able to make design/architecture decisions.
If you're a junior developer and you don't have good senior people to learn from in your job, it's time to start looking for another job.
The specifics are highly domain-dependent, but at a high level, you need to be able to identify every resource and process in your system, how those processes utilize (or should utilize) those resources, where the bottlenecks might be, and have a plan for how to resolve the bottlenecks given the capabilities of the resources and processes. That is, you need to be thinking at a level of system components, not code.
Unfortunately, most of the above knowledge isn't taught in schools and comes mostly from experience.
So if I'm someone who doesn't really have the patients or attention span to sit down and write lots of code, but I think I'm pretty good at figuring out what all needs to happen from a macro perspective.
An analogy would be I don't have the foggiest on how to play the violin, harp, or flute, but I know how they all need to play together to not sound like a train-wreck.
I don't know if I'm being extremely naive or if I'm actually going to turn out being better at system architecture than actual coding, or if that's even possible.
Is it possible to get good at software architecture without having a lot of personal experience with many subcomponents of said architecture? Sure.
Is it likely? Not at all. This area is even more experience-driven than other senior-orchestration-type roles in other fields, and most of those roles--even the literal example of an orchestra conductor/director you mentioned--very often start with someone with immense personal experience playing a lot of metaphorical "instruments". All the other people in this subthread saying "build things and see how they fail" are right--and you have to be intimately involved of the building of a lot of things to start to see the patterns and move closer to an architect role.
I wouldn't bet the farm on being able to conduct the orchestra without knowing how to play an instrument.
EDIT: I don't think parent should be downvoted. They're highlighting important distinctions and seeking advice early in their career. Just because y'all disagree with their approach doesn't mean their comments lack substance or are bad. And if you think the down arrow will teach some kind of "punishment" for having a disagreeable outlook . . . well, I hope you don't raise kids, 'cause that tends to backfire.
Thanks for the reply and the edit. And you're right, I'm just trying to figure things out before I get to place where the consequences are more real. I could have worded the original comment better, I think it came off as arrogant.
If I can find projects that are open source with public documentation of pain points would it be beneficial to read through their discussions etc, to get a sense of the issues these projects faced with scaling, with the goal being to glean some insight into what went wrong and maybe avoid common pitfalls and save myself time trying to reinvent the wheel.
Writing something that horribly fails to scale, then learning from your mistakes. :)
Seriously, though, come up with a hypothesis about how your system will scale, then devise an experiment to test the hypothesis. If the hypothesis fails, figure out why, fix it and repeat until you get the behavior you want.
The next important piece is monitoring and measuring the performance metrics important to you, then figure out how to change your architecture when the system grows enough to have new bottlenecks.
Having a good load test tool, for example, can be invaluable in learning how your system actually performs.
Nothing can replace many, many years of experience.
Some skill in general optimization-thinking is key. The ability to visualize how the entire system works is key. Aptitude for API and general architectural design is key.
And even when you get it all right, _something will fail._ At which point profiling and having the right metrics in place so you can see what's failing is key.
Mess with stuff in your free time. Most of the time people won't want to hire you to lead implementation/designing things unless you have experience doing it which can be a pretty nasty catch-22. Having some side projects side steps that and puts you ahead of a lot of your peers.
Also have a clear sense of direction of where you want to be going and what you want to be doing. Without that management and mentors won't know how to push you along and might come to expect you to remain in the same position for longer than you expect to.
I'm not actually really suited for programming but I love tech and problem solving, like I do write stuff in my free time, I just take forever to get anything done. I've taken up going to as many hackathons as I can, I've attended 2 this semester and know of at least 2 more I'm going to be at. However when I'm at these things I rarely find myself writing any actual code and more of just making sure everyone is on task and finding resources for everyone to leverage. I find myself acting as a project manager most of the time. I don't know how that's going to look to an employer, but the teams I'm on build stuff that apparently the judges like.
Idk if these events count towards experience, but I feel like I'm getting better and better at getting people to the end goal.
Those are really valuable skills too. There is a slant for engineering here since the site is for, by, of engineers. I recently joined a new team and we have a totally kick-ass technical product manager we work with pretty heavily. He is an absolute boon to the team when it comes to interfacing with different parts of management and being able to walk to the talk of the specifics of certain technologies.
For your last point, I'd be cognizant of being able to measure your value objectively somehow. It can be easy to get left behind if the value you provided didn't get tracked because people were only paying attention to lines of code.
