Anecdote. I swore off spinning disks for my PC years ago, but I've been a bit cheap and recently got fed up with my 500GB "storage" drive always being near full and my work VM as well :/ So I finally went onto Google to look into storage and was absolutely, postively, shocked that I somehow missed the whole m.2 nvme surge. So much so that I got a 1TB 970 EVO for just $170. Nearly 500k random IOPS with qd32, and 3.5GB/s and 2.5GB/s sequential read/write?! Yes please.
I was even able to jump through hoops to get bitlocker working with hardware encryption, so I'm getting full disk encryption at nearly spec speeds. Awesomeness.
EDIT: It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that?
> It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that?
The guys making storage controller ASICs weren't really prepared to start pushing 3M+ IOPS through their entry-level products. If you want to get your money's worth of performance out of NVMe SSDs, you can't have much of an abstraction layer sitting in front of them, especially not one that translates them into SCSI devices. It also doesn't take many drives before the RAID controller's host interface is the bottleneck, because the RAID card can't go into a slot wider than PCIe x16, and most are just x8.
On the other hand, actually using that much storage performance requires a ton of software engineering effort on the application side, and quickly pushes you toward a clustered approach where a traditional RAID controller isn't all that useful. "Software Defined Storage", NVMe over Fabrics, etc. are popular because it's hard to actually keep 24 NVMe SSDs busy with just a 2P 2U server.
Which is a real shame, because 'in the hardware, with no software complexity or overhead' is exactly where encryption should live.
I wish some SSD manufacturers would get third party audits of their firmware which can say "We believe data stored on this SSD is not decryptable without an externally provided key". It ought to be pretty trivial to inspect the source to verify all keys which can derive the master key are never written to persistent storage.
I'd put "it's open-source and simple" somewhere below "it's open source". There are quite a few open source codebases that are very hard to trust due to the sheer complexity.
I'm not sure if I'd put being open source higher than actually being audited. In theory there's no difference between the 2. In practice being open source means that it could be audited while publishing an audit means it was. It's the difference between giving someone money and giving them a lottery ticket for that money.
As many programmers admit themselves, it's also their job/duty/etc. to audit open source code but most actually don't especially for the sprawling, complex code. When everybody's responsible nobody is. It's better than closed source and security through obscurity but potential is only valuable if exploited. OSS without audit could be in some cases just as (in)secure as closed source simply because it may provide its own layer of obscurity through complexity and the expectation that "someone" will check. Someone else.
True, but this is balanced by the fact many audits miss parts, and at least being opensource means it's quite likely to become the target of automated fuzzers and vulnerability scanning tools.
Which is actually cumulative with what you wrote. We currently have a bit of a "trusting trust" issue with open hardware: how do you check that the fab didn't tamper with your ICs before manufacturing them?
Gmail is kind of tricky, I am not sure if you are talking about the web interface or something else. The web interface is a slightly more complex case, as you'd have to patch the browser, and check every piece of javascript. You cannot check the server part yourself, of course. But you wouldn't use SAAS if you were interested in security in the first place, unless you trust Google's servers, in which case HTTPS+HSTS (+cert pinning?) should be enough.
Otherwise (android app? Browser js/html/css?) you separately compile gmail. Assuming the build is reproductible, you can check that its hash corresponds to what was sent to you.
I can answer this! It has nothing to do with raid controller quality so much as NVMe doesn’t go to a disk controller. It lives on the PCIe bus itself.
In IDE, SATA, SAS, you wire up to RAID. In NVMe, you skip the raid controller.
So you can still RAID, but it’s either software or some device with a “private” bus like those PCIe cards that hold 4 m2 sticks. Or IDK how others work.
I just know for Enterprise right now, you get NVMe -OR- RAID. Sucks because I had some screaming drives picked out super cheap but had to ditch them in favor of slower SAS drives. But... those slower and smaller SAS drives in RAID10 will be faster than the RAID1 NMVe plan I had.
Seems like there's enough cores available in most systems you'd want an NVMe system that you probably don't lose much doing software RAID (or pooling, in some more complex FS like ZFS).
I just found some simplistic software RAID benchmarks on Phoronix[1][2], there may be better ones (I would be interested to see how much CPU the software raid processing takes during some of these tests, even if just to confirm to myself it doesn't take much).
Software raid (implemented in os, not the fakeraid/softraid chip on motherboards) in this case is better because it can use as much memory as it needs and also since it is part of the OS it can receive additional information on what kind of workload OS wants from it, i.e. readahead or not and how much. Hardware raid generally doesn't know about those things and tries to infer them so it can't optimize fully.
Xen/KVM market share probably dwarfs VMware if you look at total systems. It's what a lot of the large cloud providers use (AWS/GCE at least). I assume Azure uses HyperV, and also that a lot of shops are using it because it's the supported MS product. Our company has VMware and HyperV deployments for just that reason (Office IT has their own MS cluster, systems uses a VMware cluster we admin. Yes, it's silly, but does illustrate a point).
Using https://old.reddit.com/r/VFIO/comments/eryb71/best_storage_f... setup and truecrypt + zfs
raid with lz4 compression I never see cores even visibly affected by either compression or encryption, the speed seems to hit the actual underlying hardware speeds pretty reliably, and this machine CPU is not anything to be amazed about in current year (Ryzen 7 2700 stock clocks with the performance governor)
Actually I was a bit curious about this so just tried reading some raw images off the zfs pool with no directives against caching or anything to hit the absolute maximum possible read speeds, even with the read speeds getting well above 1GB/sec it still wasn't showing any visible impact on any cores in terms of either truecrypt or zfs. Almost wondering if they're hidden somehow at this stage. If they're not though it seems the performance impact is so low it gets nowhere near restricting throughput compared to uncompressed / unencrypted speeds.
If it's a Ryzen Pro, it has dedicated encryption/decryption hardware which might be being used. The old Via mini-itx systems had this, which allowed them to speed up AES operation massively, to the point that it was basically free to run an encrypted FS (and in their cases, it definitely had a CPU cost, around 10-20% of their single core IIRC, but that was a decade ago, and I'm not sure I could find the blog entry that person did now).
Brief search on Ryzen pro seems to imply it's some kind of mobile integrated graphics / cpu solution? I don't think the Ryzen 7 2700 is one of those as it's just a plain old am4 desktop board CPU of the ryzen2+ generation, that said, it does have AES hardware encryption support in the instruction set according to https://en.wikichip.org/wiki/amd/ryzen_7/2700#Features so that may well be what's going on with the seeming complete lack of overhead there.
On windows, Windows Storage Space sucks. The RAID 5-like write performance is unusable. And Microsoft disabled the old mirroring alternative on most windows SKUs.
My understanding is that it is an implementation of software RAID that enables pools of drives of multiple sizes, plus some tiering for caching. Which is why they disabled the traditional windows software RAID.
> was absolutely, postively, shocked that I somehow missed the whole m.2 nvme surge.
I only stay subscribed to the NewEgg emails to watch retail HDD/SSD prices. The prices always amaze me, I member paying over 100$ for a 32mb usb drive and our first HDD in a home PC was a 200 something megabyte caviar.
It makes me wonder what things will be like in another 25 years.
Does NVMe make a difference over SATA2? Isn't the latter plenty fast anyway? I've been holding off on upgrading because it doesn't seem like I'd see a benefit.
Hmm, maybe my disk is using SATA3 then, I have an Evo and a not-ancient motherboard. However, I'm wondering whether I'll see a practical difference in my day to day workloads, which is mostly web development and no gaming.
NVMe is always electrically PCIe and always M.2 or PCIe card form factor for PC users. If it’s 2.5” it’s 100% SATA. If it’s M.2 then 50:50.
Idk but what I’ve heard is anecdotal small files I/O and boot time speed up stories so webdevs might appreciate it on next builds. The premium is dropping fast anyway.
I was a little annoyed that I didn't notice a big difference, but then I figured that I'm bottlenecked by other parts of my system. It's still a win because it's stuck to the case instead of hanging out in a bay, so my computer is a bit more neat.
I don't understand why anyone still buys anything at Amazon. Their Fulfilled-by-Amazon (FBA) idea was quite a game changer, but the implementation with the inventory co-mingling is something that I'd much rather stay away from as a consumer.
There's been oh-so-many reports that even if you're buying from the official store of any known brand, with FBA, then you might as well still receive a knock-off, due to the inventory co-mingling issue. Have they ever resolved this for good, or is noone really bothers to even pay any attention to this anymore?
>>I don't understand why anyone still buys anything at Amazon. Their Fulfilled-by-Amazon (FBA) idea was quite a game changer, but the implementation with the inventory co-mingling is something that I'd much rather stay away from as a consumer.
As far as I know it doesn't happen in the UK at all. I have placed 200+ orders with Amazon in last year alone and haven't had any issues except for a couple deliveries which were a day late(Amazon extended my prime by a month each time). 99% of my deliveries arrive in 1 day as promised. They have exemplary customer service as well. In a way, I don't understand why you'd buy from anyone else but amazon :P
On a counter point, I regularly order from smaller shops because they offer the same service (next day delivery for 99% of orders) for a lower price (usually about 5-20€ per item). Amazon is quite pricey for some product categories.
I wonder if this knock-off problem I keep reading about is something specific to the US. I've ordered ~200 things over the last 3 years from Amazon, and I've never received a knock-off either. I've never even heard of that happening outside of internet comments. Plus sending things back to Amazon is usually fairly trivial.
No, not US specific. I see some of sellers in my country includes stickers or keychains just to change ASIN.
One aspect is that not everything has counterfeit counterparts, e.g. there is no counter-counterfeit to, specific counterfeits, products that require exotic parts, or foods.
I’ve never seen a 4-cell(Lipo) knockoff battery for Lenovo X200 and I know the reason is because it is not easily possible to source appropriate components.
So there could be specific usage model for Amazon of a person that could completely avoid receiving unexpected items.
Just to add one more data point, I received a fake i7-8700k from Amazon UK, in addition to a couple other fake items over the years. Amazon is my last choice nowadays.
In case anyone's wondering, the fake i7 was some sort of Celeron with an i7 sticker on top - the real one is engraved/printed and has a different shape.
> I don't understand why anyone still buys anything at Amazon.
It is cheaper than almost every other computer store in the UK for the same parts, deliver next day and they don't tend to shaft you on delivery cost (if it costs anything at all).
Unless you are near one of the large online shops and need something delivered next day there is no other option than Amazon.
I use https://uk.pcpartpicker.com/ as a guide when building out a new system. A browse across some parts there seldom shows Amazon as cheapest. It's usually competitive, but in some cases it's quite a ways behind.
It sometimes makes sense to spread the parts buy across a couple of vendors to save costs - of course too many vendors and delivery costs eat into the savings. But it's quite rare that I've ended up buying anything other than monitors and the occasional component from Amazon.
They aren't always cheaper. I often (but not always) find hardware cheaper elsewhere - and I don't have to worry do much about counterfeits/refurbs with other suppliers.
Run your next purchasing decision through Google shopping. You may be surprised.
As a consumer, is there a way to tell when this is the case? It would be nice to filter this way, similar to using "Prime" to filter to just "ships from Amazon warehouse".
co-mingling is mostly opt-in, unless that has changed. And I doubt they are co-mingling high dollar products, nor should anyone be dumb enough to do that. You usually only co-mingle to avoid the effort of putting individual sku stickers on each item which makes sense for very low priced items.
The solution is Amazon Transparency. This is offered through their Brand Services division.
And the solution for the whole industry is the same thing with some distributed ledger or some very light centralized authority (like MAC or UPC vendor assignments) plus vendor-specific databases.
Transparency should be FREE if its the only solution. It really should become a protocol and not Amazon-specific so that other vendors can jump on board. Right now nobody wants to do this unless the majority of their sales are Amazon.
Also, Amazon charges $0.05/UNIT for this!
Source: I am a consultant for Amazon Suppliers. The problem is not as bad as OC exclaims, but its slightly annoying.
About the only thing I buy from Amazon now is books (physical used, digital new) and devices made by Amazon (a new flagship kindle every 2-2.5 years as the PPI and other hardware features get better).
Anything food/supplement related always has reviews mentioning mold, substitutions etc. Cables and electronics are often counterfeit or of questionable quality. I've seen threads here on HN about counterfeit new books. Just not worth the headache of trying to find something that you think is legit and then hoping that co-mingled products alleged to be that don't get shipped to you instead.
It's a story about a hairdryer going awry four years after it was bought. This isn't something I would even bother amazon with, it's on the manufacturer. If it was within the first year I'd expect amazon to get me a new hairdryer and that's about it.
Once there are any customer-complaints on a given SKU then co-mingling is deactivated.
Later, sellers attempting to co-mingle inventory are provided a seller-specific SKU that is matched to the parent and they're required to use. This ties all issues on their orders back to their store.
This still requires lots of human intervention, which sucks.
Well, Amazon is the place you bought them from, they are liable for your warrany, not the manufacturer, so it should not be your problem, Amazon will have to take care of it.
My experiences with Amazon have been amazing; I bought two 1TB 970 pro's when they just came out (hella expensive), and one of them was faulty. After explaining my troubleshoot, they were like: ok, sounds like you know what you are doing, we'll send you a new one, be sure to return the one that's faulty in the next few days. I was blown away, amazing experience for me and certainly a reason for me to look at Amazon first when I need something else.
> NewEgg is a bit better, but don't buy refurbished drives.
Be very careful even buying new drives from party vendors on Newegg. I've gotten quite a few that were advertised as new but report over 30,000 hours of power-on time.
It turns out my experience is far from unique. Basically, very few people will check the SMART data, so the vast majority of the reviews are fine. If you do happen to catch it, Newegg will of course ultimately step in and force the vendor to fix it -- but that's about it. The first vendor that scammed me appears to still be doing brisk business on the site over a year later, with mostly 4 and 5 star reviews.
I bought some "Amazon Renewed" WD/HGST 10TB He drives. They came from a sketchy seller that provides a "warranty," but the provided phone number and website didn't match. I think one was used but a different(?) sketchy used HDD seller in another state.
I'm pretty sure there's no real warranty, I have no idea where these drives even came from, I have no trust in "Amazon Renewed," and I have a fear the drives are actually a bit smaller, but run a hacked firmware. But I saved $120 on the pair, so rolling the dice...
I tried that once a couple months ago. The drive pretty much came with a head crash preinstalled. I plugged it in and it sounded like it's shredding bricks in there.
I assume the seller just sources defective drives from somewhere and directly resells them as refurbished, letting the buyer do the actual testing. I assume this must work because some drives might erroneously been marked defective when it was really a sofstware issue, or other hw problem, like faulty sata cable, mainboard, loose connection, and then obviously there are drives with minor issues, like just a few reallocated sectors, that the naive buyer might not be aware of.
Yeah, I don't know what Amazon Renewed is, but it's not consistent. I bought two renewed phones of the same model a couple weeks apart, and they came with similar but not the same packaging, and one came with a review card from the vendor, even though I bought shipped and sold by Amazon.
I forget why I stopped patronizing Central Computer, probably something to do with selection, price, and service. Newegg is an utterly hostile place to buy things these days.
Honestly, if price isn't a motivating factor CDW still exists (and if you're making large business purchases you'll definitely get better pricing).
Regions like for TV and movies. IE, they sold a drive in Asia and someone sold it to us. We do a warranty check and are told that we can't get a warranty for that drive because we're in the US.
It's worth doing a warranty check with the manufacturer as soon as you receive the drive if there's any possibility at all it might not be up and up, so that you have an opportunity to return the drive if you want to.
Even if the drive is legitimate, a vendor might sell you an OEM drive which means having to go through that vendor for warranty service instead of direct with the manufacturer. That might be too much of a hassle.
Often the drives are sold at different prices in different regions. For instance, I have a few WD Reds (NAS 24/7 grade) cheaper than Blues (consumer grade). They have the words "Not for sale outside Mainland China" in big bold writing on the label. They are usually parallel or grey market goods, in my case they were brought into Hong Kong by parallel traders who probably brought infant formula or medicine back into mainland China
Gray market: The short version is that the exact model that was purchased was intended for a different region due to price segmentation at the manufacturer/brand. The item would be genuine but the manufacturer won't support the item if the customer is outside the intended region.
DVD/BluRay implement a lot of the distribution layer of segmentation in hardware and software, but the notion exists in lots of goods, such as in the marketing of high-end watches to different countries.
I assume they got a drive that wasn't meant for their region of the. When they tried to initiate an RMA with the manufacturer, it appears they wouldn't accept it.
Setting aside the issue of counterfeits on Amazon, this is really nice. Clear, simple, functional and allows you to explore the space not just comparison shop for a specific item.
I've long thought Amazon has terrible search and sort options for things like computer parts. Trying to buy e.g. RAM is a nightmare.
Amazon's sort just flat out doesn't work, I don't understand it, it's obviously not incompetence (and anyone can call a std lib `sort` fn on a given field anyway) so I don't understand why it's an option, why it pretends that sorting by price is something that you can do.
It's not just that it doesn't include shipping (for those which have a fee, which tends to be the cheap tack, presumably to try to catch you out) - even including shipping the order just seems all over the place. I can only assume it's paid-for rankings etc. with no indication of that to the user, but it's a crap site. As a business/seller/provider of goods it's great, and gets too much of my money, but the site really is crap.
I'm glad I'm not the only one for whom sort by price has never worked!
They also have tried to specifically promote their clothes and shoe-selling business within Amazon.com, but those have especially bad interface for figuring out price and all. For example, if there are 20 colours of jeans on the same page (e.g., it's considered the same item, just in different colour), do I really have to click to each one of them to find those that are the cheapest and/or are on sale?
And the sort never working by price is just ridiculous. I think it's been like that for years, really amazing that they've never fixed it. Never works even if you select Amazon.com as the only seller.
I was at an Amazon all-hands years ago (I haven't worked there for years) and I saw an employee ask Bezos what Amazon was going to do about the poor quality of search. Bezos brushed off the criticism and said the search team did good work.
I don't know why. Maybe he earnestly thinks it's good, or just has different priorities. But I don't expect it to get better.
I was there too when someone asked why Google seemed to always have better search results for Amazon.com than Amazon itself, and I remember he said that if you always start searching on Amazon and only fall back to Google when you don't like Amazon's results, you'll notice when Amazon has poor results and Google has good results, but not the reverse. Something of a copout perhaps, but I thought it was an astute observation.
> I wish there was a way to just download the catalog and query/search/sort it myself though.
That would be nice and not entirely without precedent, since you can download some of IMDB's catalogue (owned by Amazon since 1998.) However I suspect this feature is a vestigial remnant of IMDB's earlier days on Usenet.
The best I can figure is that Amazon sort by price is using the lowest price from any new offer, even after you've narrowed to a specific seller. It might be some grades of used as well. It's also likely that the prices are periodically refreshed for the index, not live.
I don't have any real information though, other than a confirmation that it doesn't do what I want either.
How can a site this large get away with such a basic functionality not working as any of their users would want for such a long time? Did it ever work?
I think I've noticed it not working like 5 or 6 years ago when I was shopping for shoes, if not earlier than that. Keep in mind that even without Zappos, they're in the business of selling shoes, too, and have ran some crazy promotions around shoes, too.
Have you ever tried searching for apps on the Google/Android Play store? It's a complete disaster if you're trying to find something specific that's not in the top 20 apps. Given that Search is Google's specialty my personal conclusion is that search is fundamentally broken on the Play Store on purpose.
There are so many apps released constantly since their only hope is to get into "new apps" lists which lasts probably a week or so, if they don't get enough installs then it is usually the end.
So I guess google just heavily filters, limits and shuffles apps so they are somewhat evenly distributed but it does look to end users like a buggy search that can't decide if it has the app or not returning different results for the same query.
Whatever. If I enter the name of an app verbatim at the search field, Google should put that app at least around the first page of results.
Currently, it often doesn't. I don't care much about fairness for the app distributors, but I suspect the current schema is much less fair than a good search. (The phone ecosystems are all so broken, why doesn't anybody create one that works as the user says?)
I don't know if it ever worked; it's certainly not worked for a long time. I recall thinking that amazon is very good at optimizing things, but their targets are different from mine.
They can get away with it, because we still use them, I guess.
Trying to buy any specifically-indicated item, outside possibly books, is a clusterfudge.
Amazon's particularly bad, though few retailers are much better.
An advantage of going directly to a manufacturer or B&M retailer's page is actually the reduced set of possible products. You still get irrelevant results, but vastly fewer.
Finding and buying computer parts is infinitely easier on Newegg than any B&M site, because that's what Newegg does. They have more computer parts than Amazon, not less, but they sort and filter and organize it much much better.
I think Amazon just doesn't care about people who are shopping for items with very exact specificationd in mind. It's an edge case for them. Most people go to Amazon when they want "a tv" or "a teapot".
Whenever I need something specific, I just look up "best X with Y feature review". Almost always I get an Amazon link to the type of thing I was looking for.
Amazon's goal to be the world's retailer, selling everything, creates an incentive for vendors to appear in as many search results as possible. Even if the payoff is small, it's a net positive.
For both Amazon and the shopper, most results subtract from the utility of the experience.
The incentives problem is that:
- For vendor, more hits == benefit.
- For Amazon, irrelevant search == small cost, fix == high effort.
- For shopper, irrelevant search == high cost, switching retailers == low cost
For a shopper, not being able to find what you want (or trust what you find) is a high cost.
The cost of switching to another site, or shopping mode, is low.
More to the point: it's vastly lower than the shopper being able to effectively fix Amazon's search.
This is a general principle of networks and positive- vs. negative-value members or additions.
The naive Metcalf's Law notion, that all members of a network are a net positive, is false, and even the far more useful Tilly-Odlyzko formulation (V = nlog(n)) fails to account for nodes contributing a negative cost. Since all* information imposes an attention cost, you can approximate the actual network value as:
V = n*log(n) - k*n
Where k is some cost constant.
In fact the size of the network is determined by the cost constant. The lower the constant, the larger the sustainable network size.
At some point, adding more members reduces total network value. Worse, since you have high-value and low-value contributors, and quite possibly a higher value-sensitivity of high-value members, as the network value approaches and passes the zero point, high-value members tend to defect. That's what happens as a social network tends to low-quality posts, content, and discussion.
Or a shopping market tends to counterfeit goods, mislabeled content, fraud and the like.
Just as MySpace found itself walking dead, and numerous earlier retail establishments, Amazon could find itself on the wrong side of this line and quickly.
Also: though I represent k as a constant, it's better to think of it at any given point in time as being mostly undifferentiated amongst nodes. But over time that constant might increase or decrease, whether due to the behaviour of nodes, additions or deletions in nodes, or in environmental factors.
It's also a nightmare to search for a HDD of a certain size. Other sizes of HDDs will be mixed in, no matter how you search. Same thing for TVs of a certain size.
Have there been any cases of counterfeit hard drives? I have heard of USB drives rigged to report incorrect free space, but I haven't heard of the same for hard drives.
Hard drives are a little different in that it's not something that countless chinese companies make and is mostly differentiated by brand. There are only a handful of companies that actually make hard drives and prices are all pretty similar.
Hard disks (multi-platter/multi head) can be refurbished in many cases by deactivating a head or a couple (a platter) of course re-labeling it to a lower capacity, I think it was a thing in non first-world markets.
There was a recent thread post about a counterfeit external USB hard disk, though, JFYI:
This is why you have to rely on third party services to perform advanced search and filtering.
Pangoly and PCPartPicker do that exact job with advanced filtering, eg: https://pangoly.com/en/browse/ram
> Setting aside the issue of counterfeits on Amazon, this is really nice. Clear, simple, functional and allows you to explore the space not just comparison shop for a specific item.
yea and totally broken UI on mobile. not their fault though. is it time for browsers to adopt a sensible default CSS for mobile?
I find it interesting how stable HDDs are in terms of price.
I just cross-checked with german "Geizhals"[1], and the 4 TB Seagate IronWolf (ST4000VN008) started at ~160€ in Sep. 2016, quickly fell to 140€ a few days later, and then slowly descended to 105€ in Jan. 2018 - and that's the region it's been sitting there for two years now. The UK price chart looks almost the same.[2]
In 2011 I bought a 2TB WD for around £50[0], and barring some pricing oddities on that particular drive as it went out of production you can still buy 2TB drives for around £50-60[1] with £60 being where the price should be with the rate of inflation. Higher capacity drives are nice for convenience but it is a bit of a shame that the price per gigabyte hasn't really come down much[2], especially as now a single disk failure can cause 3-4 times as much data loss as it would have done around a decade ago.
Nobody puts any R&D into new drive technologies. If demand drops, factories close since they are producing at near zero margin. All R&D costs are already amortized or written off.
End result: Static prices.
The only time prices will change is when only one manufacturer is left and they jack the prices as a monopoly on antique tech that a few big companies still need. We're a long way from that tho.
SSDs are unsuitable for long term storage, technology is simply incapable (cells losing charge, normal usage physically degrading medium, limited number or charges per cell). We dont have anything better than spinning magnetic medium at the moment, and nothing viable on the horizon.
Then we don’t have anything at all. Hard drives are only acceptable at long term storage if they’re unplugged, and even then there is a significant risk that the data won’t be accessible in 50 years. If 5 years is considered long term, then SSDs are acceptable. If 20 years is considered long term, well, make sure you have a robust RAID.
"A key contributor to this reduced reliability is read disturb, where a read to one row of cells impacts the threshold voltages of unread flash cells in different rows of the same block. Such disturbances may shift the threshold voltages of these unread cells to different logical states than originally programmed, leading to read errors that hurt endurance"
The very best SSDs today promise to go into read only mode in case of a failure, and in tests almost none are. Intel server family of drives famously brick itself despite stating "read only" in documentation.
enterprise class SSDs, our system runs 480 370g and we have had drives flagged within two years. granted we have a heavy processing model but we have never loaded the system beyond 60% so far (it is 90tb mirrored protection). I used to think we would never see a failure compared to other systems we have spinning 15k drives on but alas that is not true.
as to the failure rates being similar, too early to tell, but to be up front the sheer performance gain even compared to heavily cached systems is amazing.
At least [edit:] three of the 4TB drives I checked was categorized as "new" but was in fact reburbished ("with a new warranty"). One of them was pretty shady about hiding this fact.
[Life is too short and data is too dear to ever buy one of these refurb units]
If you are trying to build a NAS or similar then my suggestion is go with shucked WD EasyStore externals, you need to either use molex to sata power to bypass the 2.2v issue or tape the pin but they are WD Whites inside and work very well from my experience. I just keep waiting for the 14TB's to fall a little further so I can replace one of my 5TB parity drives with that so I can grow my array.
> you need to either use molex to sata power to bypass the 2.2v issue or tape the pin
You mean the 3.3v issue. In most cases, you can also just clip the 3.3v wire that goes to the sata power connectors from your power supply. It's orange, you won't be missing out by not having it anymore, and it's easy to clip with wire cutters. You should put a dab of electrical tape on the cut ends, but a clean cut isn't likely to short on anything. Consider it updating your power supply to the latest SATA spec.
Some people do have sata enclosures that provide 3.3v, and they may need tape on their drives, or something more drastic.
:headsmack: I couldn’t remember the right voltage and I googled “2.2v” and I found a few reddit threads and assumed that was correct. 3.3v is the right number, thank you for correcting me. Yeah I bought some tape to cover the pin if needed but I found the molex cables to be easier, I didn’t know you could just cut the wire, I feel stupid now lol.
Well, now you know; I found a couple people mentioning it in different forums, but there's a lot more posts on reddit, and I didn't see it mentioned there.
Been there, and it's worked well, but I did some research and used kapton tape. It's thin, and designed for use in electrical applications needing resistance.
Yep, I’ve got a few rolls of Kapton tape sitting next to my desk. I haven’t used it yet since I just used the Molex to Sata cables for all my shucked whites.
I believe this issue is only for certain NAS devices, or maybe only certain versions of the drives. For reference, my Synology DS2015xs didn't require any modifications to the four 10TB WD EasyStore drives I shucked that I grabbed from BestBuy.
> Power Disable feature allows for remote power cycling of SATA drives and a Rebuild Assist function that speeds up the rebuild process to help ease maintenance in the data center.
> The new Power Disable feature (similar to the SAS Power Disable feature) uses Pin 3 of the SATA power connector. Some legacy power supplies that provide 3.3 V power on Pin 3 would force drives with Power Disable feature to get stuck in a hard reset condition preventing them from spinning up. The problem can usually be eliminated by using a simple “Molex to SATA” power adaptor to supply power to these drives.
I don’t think I’ve heard of this. It’s only the power that the cable is providing and I’ve been running for months if not 1-2 years without issue using this setup.
I might not be the best person to ask. I’m running 3 servers at home currently that are all desktop towers. With a 3->4 (3x5.25”->4x3.5” [0]) converter I can fit 12 drives in each tower. Across all 3 I have just over 150TB. There are some nice server chassis that can handle 20 but I’m more comfortable with tower hardware and when I hear “SAS backplate” my eyes glaze over...
I’m a big fan of UnRaid for my server OS but I know it’s not as advanced as some offerings on the market. I enjoy it’s ease of use and management of VM’s/containers.
What country is that if you don’t mind my asking? Unfortunately, in the USA shucked are often cheaper than bare. I wish bare were cheaper b/c you void the warranty when you shuck..
From a Gamers Nexus news video near the start of the year - apparently some of these companies shifted some production to RAM due to high demand from new consoles and high end phones.
Anyone know of one that can display a sorted price per GB _with an offset_? E.g. so you can put in a marginal cost per additional SATA port and get corrected figures?
Saving 5 dollars on drives by using 30 3TB drives instead of 6 15TB drives isn't a win in terms of pricing when you consider the total cost.
Of course. What I meant is I'd like to see an option to weigh in smaller (to be exact: closer to a particular point of available capacity distribution) drive size as a positive feature for the resulting chart.
So, in a sense, the same idea, but an opposite of a GP comment.
It is worth noting the design and density of information presented in this website - we've forgotten how to present lots of data to users because the modern trend is to pad everything with whitespace. Dense design, no frills, just a table - this is what the web needs more of.
Given Amazon's system for packing items, I would not buy hard drives from them. I don't want my hard drive's retail box bouncing around all over inside the outer amazon box for however long it takes to get to me. That's a recipe for issues down the road (or immediately).
That said, this looks like a useful tool. I can accept prices from Amazon and shop around for someone who knows how to pack.
I wish the entire web looked like this once again. Ah the good ol' days.
Please add Newegg and other non-Amazon vendors. Newegg has an API and everything. Around 10 - 15 years ago, I used to use a simple site called pricewatch that wasn't too far off from this.
Apparently it's against Amazon's API terms to compare with other sites. I recall years ago the guy who runs the site explained that he is only ok to compare Amazon with Amazon.
I don't think it's against the rules, but the rules are complex and spread all over multiple places, and they're like Google in that you have no due process if someone there decides they don't like what you're doing.
It is against the rules. If you are an e-commerce company (i.e. a competitor) or you operate a shopping comparison service, you are not allowed to use the API at all.
That appears to be specific to their advertising API, not the affiliate program. The affiliate program policies[0] say:
> if you choose to display prices for any Product on your Site in any “comparison” format (including through the use of any price-comparison tool or engine) together with prices for the same or similar products offered through any web site or other means other than an Amazon Site, you must display both the lowest “new” price and, if we provide it to you, the lowest “used” price at which the Product is available on the Amazon Site.
... which implies that it's not forbidden, but I can't blame someone for not wanting to risk it.
This seems all Amazon links only (with affiliate codes). Given their known counterfeit problems and fake reviews, I would recommend avoiding it for important things like disk drives. Plus, they are not guaranteed to be the cheapest anyway.
If you need to buy a drive, I would recommend local brick and mortar store first, and some sort of proper aggregator (which looks at multiple sites, not just Amazon) second.
Have you ever tried to buy tech at a B&M store? They will have obsolete products that have been on the shelves for years listed at the original MSRP. I'm sure there are exceptions but this is generally terrible advice.
In Asia there are a lot of serious vendor. Sometimes B&M prices and stock availability here is very competitive with online offering.
B&M is still good for me when I can't wait for shipping or risk postal damage. It's also convenient to have a face I can look at whenever I would like to dispute something FAST. Especially disk storage products.
If possible, Micro Center or Frys. Frys is somehow profitable but with dwindling stock at every California/Nevada location I’ve been to recently (maybe an indicator of higher prices?), but Micro Center is great.
Ya, contract issues for a very specific reason. They are trying to switch to a full "consignment" model. So they want the manufacturers to give them all the inventory for free and they'll just return what doesn't sell later. So manufacturers are likely very disinterested in that model.
That's the line they're giving out, anyway. I'm curious to see if it's just a ruse to keep some employees around and keep customers coming in paying full price as they clear the shelves before something else like complete shutdown.
It's weird, the shelves are as empty as you could hope for in a liquidation scenario and there's just crap left. Continuing to pay rent to try get rid of a few hard to sell pieces of old tech or low value items like phone cases is definitely past the point of being sensible.
So I think someone in management genuinely believes suppliers will come around to supplying them goods with payment on sale, despite all evidence to the contrary
I suspect they actually own some of their buildings and have for some time, starting from way back when property values and mortgages were lower. Or at least "own" in the usual sense of being the owner of the mortgage. Which would maybe? give them a few months more comfort zone in that they aren't paying that extra margin of current market rents to a landlord, at least in those cases where they own.
You local Best Buy will also probably have 5 hard disks total. The chances of getting exactly what you want (looking at the list in the linked site) is minimal.
It's really easy for Best Buy to stock Samsung SSDs and therefore offer you something faster than anything else you've ever owned. But that doesn't mean you aren't overpaying by at least $40/TB compared to a good Amazon or Newegg price on an almost-as-fast drive from a smaller brand.
It wasn't a Samsung SSD, I don't really buy Samsung since it seems overpriced and overhyped most of the time, unless I catch a sweet deal. I had bought a Sandisk that was actually more performant than I anticipated, it was noticeable immediately after installing Linux on it.
Disagree about the local stores part, but yeah for specific areas like electronics I always go for a more trustworthy site (like NewEgg, Jet or sometimes Best Buy). Prices are usually the same as Amazon, and if not they will almost always price match.
IDK, I suspect newegg reviews are equally gamed. Fundamentally, I think we need to come to terms with the fact that product reviews are worth the price the consumer paid for them: typically zero.
For electronics at least I can't remember the last time user reviews drove my purchase. I always look at specs/benchmarks, reviews on tech sites etc., and then go to the vendor with a fair idea of what I want.
Buying an internal hard drive from Amazon is extremely fraught, but I'm not aware of similar problems for internal SSDs. And they are definitely one of the most reliable places to get good prices for consumer retail SSD models.
While nobody counterfeits mechanical drives, you can still get screwed buying "refurbs"/"tested only, like new" and similar "deals". SMART data can, and is often reset, making 3 year old >30K hour drives pulled from storage racks report mere couple power on counts and less than hour total lifetime run time.
Sometimes you can also run into situations where someone returned an external hard drive, but they swapped the drive inside with a cheap low capacity drive (like a 500gb drive put into an 8tb drive enclosure)
That's not a particularly scary scenario when buying from, e.g., Amazon.com, as it'd seem like the unit could just be returned for not being the advertised product.
I'm curious, though, why would anyone bother with a scam like that?
I mean, it'd seem like someone with the necessary technical skills could make a larger, steadier income through honest means. So why would they waste their time/effort on something that can backfire when noticed by the returns center or the next buyer?
It certainly wouldn't surprise me if people've done this a few times -- I mean, people do all sorts of weird things -- but is it something that's happened enough to be notable?
You can also get screwed for "new" drives. It depends on whether Amazon gates "new" sales of hard drives and even then, there are likely people gaming the system to get un-gated to sell them. All it takes is the original box and some shrink wrap to make a refurbished drive "new".
I’d buy a lot of things used but never a harddrive or any storage/backup media. Not sure if these are countereit though, I’ve never seen any bootlegs sold as used ever..
I probably wouldn't do it either. Though I recently sold my WD Red NAS HD on ebay. Someone made a really good purchase with it. They got it for half price with the original purchase receipt and 1.5 years warranty left.
Every now and then you find someone with a conscience selling things on ebay and it is just a pleasant experience and restores some faith in humanity. That is what inspired me to sell it instead of having it collect dust in a drawer for 10 years.
I have thought about this as well. I would love someone more knowledgable to chime in - but if I had to guess it is that consumer hard drive sales have been declining a lot in recent years. Desktops are a thing of the past for your average PC user, and as a result hardware/storage upgrades. Laptops, mobile devices, and cloud storage is where it's at. No one is buying hard drives anymore except large enterprises. In order to appeal to the consumer market HDD manufacturers need a value prop to compete with cloud storage. So discounting bulk storage is the play. I am guessing that they are selling "white label" enclosed drives at a lower margin than the enterprise counterparts but still netting more in the long run.
When I shuck drives like these, I frequently find the higher quality models of drives in there. However, I have heard that they bin their drives and the drives that don't pass muster for enterprise end up getting used in these external drives.
I wonder if they are lesser quality or worse specs. External hard drives often don't even show the performance characteristics other than capacity on the boxes. This is different for internal drives from what I recall.
It's actually the opposite. External hard drives are frequently made from over production of enterprise grade hard disks. [1]
Manufacturers would rather over produce high quality drives than risk not being able to fulfill orders. Eventually the surplus gets put in an external housing and rebranded for consumer use.
I do a lot of shucking of external hard drives and I have never come across a 7200rpm drive. They are always 5400. But they are sometimes the higher quality drives in that speed range.
I see a lot of folks talking about the risk of Amazon commingling and used drives. I buy drives off Amazon, but only external/new. I only uses these drives as backups, and everything is stored on multiple drives (not RAID, they're stored in different places for fire/theft security). I've been doing this for the last 20 years and have rarely had any drive errors (usually it's a partition that goes, but after reformatting it's fine). What I've found is that I outgrow the drive before it dies on me. Once I outgrow a drive, I just buy another that is twice the size and copy everything over from the old drive, which I hang onto but never use.
But I wonder, am I doing something wrong here? Or are other people just using drives in different ways (for active files or huge media libraries that they don't want to store redundantly)? I would understand that under those circumstances, failure of a single drive is more catastrophic. I don't mean to be critical here, as I'm far from an expert!
When you let the companies pick which model of disk to embed in their product, they pick the cheapest one available. Probably not what you want.
Personally, I have Zalman ZM-VE300, also no-name m.2 2280 USB 3 enclosures, both work fine. If you'll do the same, don't forget there're 2 sorts of m.2 disks, SATA and NVME, USB enclosures for them aren't compatible.
This guy posted in a thread here a few months ago about side hustles. I was tempted to start my own off of this model. It appears it’s not as easy to get started, and with a new baby I’ve been too distracted. But at the same time with a new baby it’s more tempting for the extra cash.
When my son was small I would put him to bed every night with a book and then sit there in the dark and wait for him to fall asleep. I had a big pillow thing that I would sit on next to the crib (it was big enough that it had back support). Well, that hour of sitting there with my laptop was always one of my most productive hours of the day. Sometimes I would just sit there even after he fell asleep and continue to work.
Also because I was curious -- looks like LTO-8 is about $12.75 on newegg. Which is still keeping below disk - but only just (and ignores the cost of drives)
I have an old 24 tape robot I haven't used in a while. Priced out a used LTO8 drive and 24 LTO7 tapes, and assuming that they're formatted M8 to 9TB, it still costs out to $0.02/GB-- so only just breaking even with the 12TB USB3 external drives.
You need a lot of tapes before LTO starts comparing favorably to current drive prices even using LTO7 tapes. :( like 1PB before it's really attractive.
Hi, this is my site. Thank you for all of the feedback! I can address a few of the issues you've raised...
Missing some products. This is a known bug in how I'm currently importing product data from Amazon's API. Some products are listed with variations that appear on the same product page but are completely separate products in Amazon's catalog. There's no way to get the API to return all of the variations at once, so I have to perform several subrequests to enumerate those. Currently, that doesn't happen. However, I'm in the process of reworking a lot of the data import code to use the new PA-API 5.0 and am planning to make variations work properly with those changes.
Filter by products sold by Amazon. Initially, diskprices.com was set to filter out products not sold by Amazon, but I received quite a bit of feedback asking me to remove that filter as some of the best deals are from resellers. The new PA-API does have a populated Merchant field for most products, so I may try to expose that.
Display prices including shipping costs. This is addressed in the FAQ (https://diskprices.com/faq.html), but it really comes down to privacy. I'd need you to login with your Amazon account or give me your location in order to compute tax and shipping information and I really don't want the burden of handling PII.
Clicking the Back button does weird things. This is a bug. Whenever you change a filter, a bit of JavaScript updates the URL in your address bar so that you can copy/paste a link to the page you're looking at with the current filters and send it to someone. It appears I'm not capturing the Back event and updating the filters to match the address bar. I'll get this fixed soon.
External SSD category is missing. Up until the last few months, there really weren't many external SSDs for sale on Amazon, but it looks like this is definitely becoming it's own product category, so I'll add it.
Add Amazon.co.jp. Funny story, diskprices.com had support for Amazon Japan when I first launched it, but they suspended my account and sent an email telling me why, written in japanese. Google Translate couldn't make any sense of the email and I'd had some significant data quality issues with the filters, so I decided not to pursue it further. This is the first time somebody's asked for Amazon.co.jp support. I'll look into setting it up again, but Amazon's added some new restrictions on API access across all regions since then, so it's a bit more difficult to get new regions added now.
Account for per-port cost in calculating prices. This is something I've been thinking about for a while. I think this feature ends up looking a lot like pcpartpicker, with a constraint solver bolted on the side... Given a set of parameters for total capacity, redundancy, bandwidth, etc, optimize for the best price/performance. I currently don't have enough metadata about most of the drives to implement this properly and it's a big feature to develop, but it's something I want to experiment with eventually.
>I bet that's not enough for them to start to notice.
I got blocked on very first attempt to scrap from cloud. Literally request #1.
...with code that worked fine on residential IP five mins earlier (where I coded it).
I'm sure it can be circumvented, but that experience led me to believe it's better to just save the time and pay for a res proxy (as sketchy as that concept is)
I just spent a few days looking for server hard drives.
IOPs, NVMe doesn’t RAID, throughput, latency, raid controllers, on hard drive DRAM cache, SAS vs SATA vs M2 vs PCI, drive writes per day, mean time between failure, specific server firmware on drive...
I’d buy anything off this list for me personally, but I’m definitely in research mode and there just isn’t enough info here. Buy on price alone!? Savages!
This is the second "Find X on Amazon more easily" sites that I've come across in the last week or so (the last was a USB hub filtering site). Including eBay in the mix, it's the third such site[0].
I love these sites. It's just sad that it's difficult-to-near-impossible to actually use Amazon (or eBay) for the exact same purpose. The perfect example hit me, yesterday. I wanted a new A/C adapter for my Thinkpad. It's a USB-C laptop, so simple, right? Start with a search. OK, there's a lot of A/C adapters, and a lot of USB-C adapters for phones. In the mix is one or two (over-priced, non-OEM) adapters.
Let's try the A/C adapters category, maybe I can click into that and filter based on the metadata? First, find the category. A few minutes later, settle on something somewhat close. This shouldn't be too difficult; this thing can be powered by anything 60W or higher that supports USB-PD and has a USB-C end on it. Nope. Not a single filter and a few thousand results are returned[1].
Product properties/metadata are so important -- why are so many important bits of data completely missing/unavailable to me while trying to find what I need? I feel like Amazon and eBay have managed to recreate the experience of shopping at a Walmart Super Center online -- a store full of products, organized by some magician, in order to make a 2 minute purchase require 20 minutes of browsing. I guess I don't have to roll the dice picking a check-out line that might move quickly, but outside of that, the experience is so bad that I have a solid four or five things I need to buy which I've attempted to find and simply given up.
And it used to be that you could find other shops online that were more narrow in their product categories (but not selection in said categories) and offered better options for filtering and finding the precise item you were looking for. Many of those are gone, now. The ones that remain all use a handful of off-the-shelf cart software and are often worse for finding products[2] or the price is so much worse that you end up finding the product and searching for its specific part number on Amazon.
I have to say, though, I clicked this link because I have been putting off purchasing an NVMe SSD drive. So thank you to whomever is responsible for this site; that was very helpful in getting me to pull the trigger.
In a more perfect world, a site designed to make it possible to filter available products by a set of incredibly important, common properties which is directly tied to the site who's job it is to sell you the actual products wouldn't even exist. In this world, it gets over 250 upvotes and has more than 160 comments after being on HN for about 13 hours.
[1] Unless, of course, I decide to sort by anything other than "Featured" whatever-the-heck-that-means, then there's 150 or so.
[2] NewEgg does a better job than Amazon, but too often a filter will eliminate a set of products because they refer to a common property of a product by a different name; I can tolerate that, but it'd be nice if I didn't have to so frequently.
I was even able to jump through hoops to get bitlocker working with hardware encryption, so I'm getting full disk encryption at nearly spec speeds. Awesomeness.
EDIT: It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that?